parser: ParseFile keeps the trivia the grammar never sees - #6
Merged
Conversation
lexer.LexFile returns the tokens Lex returns today together with the runs of whitespace and comments Lex drops, from the single pass Lex already makes. parser.ParseFile parses a script and carries that trivia on the new File result, so a consumer that needs both — a formatter putting comments back where they came from — no longer lexes the input twice. "Trivia" is Roslyn's term: the C# compiler calls the channel of source text that does not affect syntax "syntax trivia", and swift-syntax and rust-analyzer use the same name. Two corpus-wide properties pin the contract: tokens and trivia tile the consumed input exactly (ordered, no gap, no overlap), and ParseFile agrees with ParseString on every case's statements and errors. Existing entry points are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018MTvpHqNMadH12pTtsgUq2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sqlc fmt's SQLite reprinter needs the comments the grammar never sees, and until now it recovered them by lexing the source a second time with its own scanner. This exposes them from the single lexer pass parsing already makes, so the formatter's statements and comments come from one pass and one lexical truth.API (additive)
lexer.LexFile(src) (tokens, trivia []token.Token)— the tokensLexreturns today, together with the runs of whitespace and commentsLexdrops, asSPACE/COMMENTtokens in source order.Lexis now a thin wrapper over the shared core; its behavior is unchanged.parser.ParseFile(src) (*File, error)and theOptionsform, withFile{Stmts, Trivia}. Same grammar, same token stream, same bail/recover shape as the existing entry points."Trivia" is Roslyn's term — the C# compiler calls the channel of source text that does not affect syntax "syntax trivia", and swift-syntax and rust-analyzer use the same name. The doc comments on
LexFileandFile.Triviacarry the attribution.Tests
Two corpus-wide properties pin the contract, run over every case via
forEachCorpusCase:SPACEandCOMMENTruns.ParseFilematchesParseStringon every case's statements (reflect.DeepEqual) and errors.Plus a focused
lexer.LexFileunit test covering line/block/unterminated comments, quoted regions, and empty input. Full suite passes.Downstream
sqlc's sqlite engine already consumes this from a pseudo-version pin of this branch (
sqlc-dev/sqlcbranchclaude/sqlc-fmt-formatting-pw47ve), deleting its duplicate scanner. After merge, a tag (v0.2.0 fits — additive) lets sqlc pin the release instead.🤖 Generated with Claude Code
https://claude.ai/code/session_018MTvpHqNMadH12pTtsgUq2
Generated by Claude Code