Skip to content

DEN-4233: preserve UTF-8 across JSON stream chunk boundaries - #3

Merged
ORESoftware merged 4 commits into
mainfrom
DEN-4233/json-parser-utf8-chunk-hardening
Sep 9, 2026
Merged

ORESoftware merged 4 commits into
mainfrom
DEN-4233/json-parser-utf8-chunk-hardening

Conversation

@ORESoftware

Copy link
Copy Markdown
Owner

Why

The Transform decoded each incoming Buffer independently with String(chunk). A valid UTF-8 code point split across stream chunks could therefore be replaced/corrupted before JSON parsing, including when a multibyte custom delimiter itself crossed a byte boundary.

Changes

  • decode byte streams incrementally with Node StringDecoder('utf8');
  • preserve accurate byte-read accounting from the actual byte buffer;
  • flush decoder tail before parsing the final JSON fragment;
  • test every possible byte split for JSON containing café, emoji, CJK text, and a multibyte ∆∆∆ delimiter;
  • add a pinned Node 22 GitHub Actions regression lane using the existing JS/TS parser tests without requiring the optional native addon build.

Contract note

This parser is an in-process byte-to-object library, so this PR intentionally does not invent a TypeSpec/JSON Schema authority. TJSV remains required where a real serialized cross-runtime contract exists; the gRPC reconciliation repo is the audited boundary target for that enforcement.

Linear: DEN-4233

@linear-code

linear-code Bot commented Sep 9, 2026

Copy link
Copy Markdown

DEN-4233

@ORESoftware
ORESoftware merged commit 872d81a into main Sep 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant