Normalize string literal representation to fix pattern matching

[cknitt]

This issue is the result of an analysis of ReScript’s string-literal pipeline performed by OpenAI Codex.

Summary

String literals currently retain their encoded source representation through most of the compiler instead of being normalized to their semantic runtime value.

A single string * string option representation consequently carries several unrelated meanings:

1. The runtime string value.
2. The original source spelling and escape sequences.
3. Whether the text has already been JavaScript-escaped.
4. Which quotes the emitter should use.
5. In one case, whether the text is raw JavaScript rather than a string.

This representation leaks through the parser, built-in PPX, typedtree, Lambda, JS IR, and emitter. Besides making the pipeline difficult to understand and optimize, it can cause incorrect pattern-matching behavior.

Reproduction

let f = x =>
  switch x {
  | "a" => 1
  | "\x61" => 2
  | _ => 3
  }

Js.log(f("a"))

Both literals have the same JavaScript runtime value: "a".

The compiler currently emits a switch equivalent to:

function f(x) {
  switch (x) {
    case "\x61":
      return 2
    case "a":
      return 1
    default:
      return 3
  }
}

As a result, f("a") returns 2, even though the "a" branch appears first. The compiler also does not report the second pattern as redundant.

The pattern matcher compares and sorts string constants using their stored encoded text while ignoring their delimiter metadata:

• const_compare
• string-switch construction

Sorting "\x61" and "a" by their encoded spelling changes their order, although they are equal at runtime.

Current pipeline

Layer	Representation	Meaning
Scanner	Token.String of string	Mostly raw source contents, including escape syntax
Parser	Pconst_string (text, string option)	The option may be "js", "json", or printer-only metadata
Built-in PPX	Same constructor	Rewrites "js" into private markers "*j" or "bq"
Typedtree	Const_string (text, string option)	Copies the pair unchanged
Lambda	Const_string {s; delim option}	Reparses the string marker into an enum
JS IR	Str {txt; delim}	The delimiter controls escaping, quoting, backticks, or no quotes
Emitter	Four separate branches	Interprets txt differently depending on delim

Scanner

Ordinary quoted strings remain encoded rather than becoming their semantic runtime value. For example, \n, \x61, and \u0061 remain escape sequences.

Decimal escapes are exceptionally rewritten to hexadecimal to avoid JavaScript octal behavior.

See:

• scan_string_escape_sequence
• scan_string

Unknown escapes are deliberately accepted for historical Reason compatibility. This further ties literal semantics to JavaScript’s parser.

Template literal tokens also carry raw text, which is necessary because tagged templates need access to the raw spelling.

Parser and Parsetree

For compilation, an ordinary string becomes:

Pconst_string (contents, Some "js")

For formatting, it becomes:

Pconst_string (contents, None)

Character literals parsed for the formatter are encoded as strings using the sentinel "INTERNAL_RES_CHAR_CONTENTS".

See parse_constant.

The string option field originated as OCaml’s quoted-string delimiter:

Pconst_string

ReScript therefore uses an untyped compatibility field as an internal protocol.

Interpolated strings are lowered in the parser into nested applications of the hidden ++ operator and marked with res.template attributes. Tagged templates become a synthetic application with two arrays and a res.taggedTemplate attribute.

See parse_template_expr.

Built-in PPX

The built-in mapper intercepts strings with a delimiter before ordinary recursive traversal:

expr_mapper

It changes the delimiter protocol again:

• Ordinary "...": "js" → "*j"
• Template segment: "js" → "bq"
• json remains "json"
• Unknown markers pass through unchanged

"*j" means that the contents have already been transformed and should be emitted between double quotes without being escaped again. "bq" means that the contents should be emitted directly between backticks.

See Delim and the string transformations.

This is the main semantic transition, but it is represented only by replacing one marker string with another.

Type checking and typedtree

The type checker performs no normalization:

Pconst_string (s, d) -> Const_string (s, d)

See constant.

Every variant is assigned type string, even though the payload is not always a semantic string value.

Translation then copies the constant directly into upstream Lambda.

Lambda conversion

The backend reparses the marker:

• None → DNone
• "json" → DNoQuotes
• "*j" → DStarJ
• "bq" → DBackQuotes
• Anything else → None

See convert_constant.

The resulting representation is:

Const_string of {
  s: string;
  delim: External_arg_spec.delim option;
}

See Lam_constant.t.

This permits both None and Some DNone, with different provenance but nominally similar meanings.

A particularly revealing symptom is the handling of inline constants. They already contain a typed delimiter, but translation converts it back into its magic-string preimage so that Lambda conversion can parse it again:

lambda_of_inline_const

Optimizations

Because s sometimes contains a semantic string and sometimes encoded JavaScript source, optimizations must inspect the delimiter:

• Constant string length only works for delim = None.
• Constant indexing only works for delim = None.
• Constant concatenation only works for delim = None.
• Let propagation only tracks strings with delim = None.
• JS IR concatenation only merges strings with exactly equal delimiters.

Examples:

• Pstringlength, concatenation, and indexing
• constant propagation

Consequently, ordinary user-written literals—normally represented using DStarJ—miss optimizations available to compiler-generated strings.

JS IR and emission

The JS IR retains the distinction:

Str of {
  txt: string;
  delim: DNone | DStarJ | DNoQuotes | DBackQuotes;
}

See J.expression_desc.

Emission behaves as follows:

• DNone: escape txt, then quote it.
• DStarJ: quote txt without escaping.
• DBackQuotes: wrap txt in backticks without escaping.
• DNoQuotes: emit txt directly as JavaScript code.

See Js_dump.expression.

DNoQuotes is particularly overloaded: it does not describe a string-literal representation, but raw JavaScript syntax used by validated JSON/FFI literals.

Possible direction

1. Establish one semantic representation for ordinary strings

After parsing or frontend normalization, an ordinary string constant could always contain its decoded UTF-8 runtime value:

type constant =
  | String of string
  | ...

Compiler-generated strings and source strings would then use the same representation. The JavaScript emitter would always escape and quote them through one path.

A central helper could define the relevant JavaScript/ReScript string operations:

Js_string.decode_literal
Js_string.utf16_length
Js_string.code_point_at
Js_string.emit

Using OCaml String.length directly would not be sufficient for non-ASCII strings because JavaScript string length counts UTF-16 code units.

2. Represent templates structurally

Templates could use an explicit node rather than strings, attributes, hidden ++ applications, and synthetic calls:

type template = {
  parts: template_part list;
  kind: Interpolated | Tagged of expression;
}

and template_part =
  | Raw of string
  | Expression of expression

Tagged templates must preserve raw spelling until JavaScript emission. Untagged interpolation could initially retain its current concatenation semantics, with lowering moved to a later compiler phase.

3. Separate emission policy from string constants

The current delimiter enum combines distinct concepts. These could instead be represented separately:

• String — semantic runtime string.
• Template_segment — raw template source.
• Json_literal — parsed or validated raw literal used by FFI.
• Raw_js — explicit raw code where supported.

This would make invalid combinations harder to construct and could eliminate option<delim>.

4. Keep source spelling in the syntax/printer layer

The formatter legitimately needs original escape spelling and character-literal contents. That information belongs in scanner tokens or a lossless syntax representation rather than the compilation constant.

The current parser mode changes the AST representation depending on whether it is parsing for type checking or formatting. Ideally, both modes would share the same semantic AST while source spelling is retained separately as token metadata.

5. Preserve PPX compatibility at one boundary

parsetree0.ml must remain frozen. A possible migration strategy is:

1. Continue accepting Pconst_string (string, string option) at the external PPX boundary.
2. Decode and normalize it once when mapping back into the current compiler AST.
3. Do not propagate the magic delimiter strings beyond that boundary.
4. Re-encode only when mapping back to v0.

This would localize historical compatibility instead of carrying it through Lambda and JS IR.

Suggested migration order

1. Fix pattern matching by comparing canonical runtime values before redundancy analysis and sorting.
2. Introduce a shared literal decoder with focused tests for every supported escape form.
3. Normalize ordinary strings immediately after the built-in PPX.
4. Change Lambda Const_string to contain only semantic values.
5. Split JSON/raw emission out of Const_string.
6. Introduce an explicit template node and remove res.template, hidden ++ trees, and synthetic tagged-template calls.
7. Simplify J.Str to contain semantic text only and remove DStarJ and DBackQuotes.

Tests should cover:

• Equivalent escape spellings in patterns.
• Pattern redundancy and source-order preservation.
• Syntax round trips.
• Lambda constant folding.
• Generated JavaScript.
• Tagged-template cooked and .raw values.
• Unicode and UTF-16 length behavior.
• Unknown legacy escapes.
• FFI uses of @as(json…).