Skip to content

feat(parser): concatenate adjacent string literals behind adjacentStringLiterals - #2514

Open
fudianchn wants to merge 3 commits into
JSQLParser:masterfrom
fudianchn:adjacent-literals
Open

feat(parser): concatenate adjacent string literals behind adjacentStringLiterals#2514
fudianchn wants to merge 3 commits into
JSQLParser:masterfrom
fudianchn:adjacent-literals

Conversation

@fudianchn

Copy link
Copy Markdown
Contributor

AI disclosure: this change was prepared with AI coding agents, reviewed and revised line by line by me.

What

Adjacent string literals concatenate behind a mode, item 3 of #2512:

CCJSqlParserUtil.parse("SELECT 'a'\n'b'", p -> p.withAdjacentStringLiterals(AdjacentStringLiterals.NEWLINE))
// -> SELECT 'ab' (one merged StringValue)

Why

Implementing this turned out considerably simpler than the #2512 discussion suggested, so it seems worth having after all: the newline rule needs no lexer work, the token line numbers carry it, and one guarded loop in PrimaryExpression is the whole grammar change. The default is OFF and byte-identical to today, where the second literal stays the MySQL / SQL Server alias.

Three readings, one String valued Feature (the shape Feature.dialect already has):

mode reading dialects
OFF (default) alias in the select list, error in expressions MySQL, SQL Server (today's behavior)
NEWLINE concatenate when the whitespace contains a newline the SQL standard, PostgreSQL
WHITESPACE concatenate across any whitespace GoogleSQL, Spark/Databricks

The result is a single merged StringValue, matching how the engines describe it ("treated as if the string had been written as one constant"), not a Concat expression, so deparsing never invents a ||. The ANSI_SQL and POSTGRESQL presets carry NEWLINE; H2 concatenates too but its separator rule is unverified, so its preset stays off for now. Item 2 (the dollar quoting) is independent and its timing is yours.

How

One guarded loop after the string literal branch in PrimaryExpression; the semantic lookahead reads the mode, checks the next token kind, and for NEWLINE compares getToken(1).beginLine > getToken(0).endLine. Same-line input keeps the alias reading under NEWLINE, exactly the split the engines have. Combined with allowDoubleQuotedStrings, "1" "2" concatenates under WHITESPACE, the GoogleSQL chunking shape. Literals consumed by dedicated productions (interval and friends) concatenate only where they go through ordinary expressions, same as today; a continuation part carrying a prefix (say E'a'\n'b') merges its value and keeps the first part's prefix.

Testing

CCJSqlParserUtilTest: testAdjacentStringLiteralsNewline (off = alias and error unchanged, newline merge incl. three parts and expression positions, same line stays alias, ANSI/Postgres presets on, MySQL preset off), testAdjacentStringLiteralsWhitespace (same-line merge, the BigQuery "1" "2" shape combined with double quoted strings). All verified failing with the loop removed, with the newline gate removed, and with ANSI_SQL missing the preset; full suite green.

Performance

gradle jmh, JSQLParserBenchmark.parseSQLStatements on performance.sql, version=latest, 10 forks × 10 iterations (100 samples) on a 32-core host, interleaved master/branch:

build run 1 run 2
master c86cf6a 3.795 ± 0.027 3.787 ± 0.026
branch (this PR) 3.791 ± 0.029 3.789 ± 0.024

Same-window deltas are -0.1% and +0.05% with fully overlapping CIs: no regression. The lookahead is one mode comparison returning immediately while OFF, paid after a string literal only.

Implements item 3 of #2512.

…ingLiterals

Three readings of two neighboring string literals: alias (MySQL, SQL
Server, the current behavior), concatenated across a newline (the SQL
standard and PostgreSQL), concatenated across any whitespace (GoogleSQL,
Spark/Databricks). The mode lives in a String valued Feature like
Feature.dialect, the merge is one guarded loop in PrimaryExpression
using the token line numbers for the newline rule, producing a single
merged StringValue. ANSI_SQL and POSTGRESQL presets carry NEWLINE.
Implements item 3 of JSQLParser#2512.
The FeatureConfiguration seeds every configurable Feature with its
declared default, so the mode is never null today; the guard keeps the
predicate locally correct regardless.
@manticore-projects

Copy link
Copy Markdown
Contributor

Thank you much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants