Skip to content

refactor(core): reuse Markdown chunk byte counts - #46079

Merged
kitlangton merged 1 commit into
v2from
markdown-bytes
Aug 31, 2026
Merged

kitlangton merged 1 commit into
v2from
markdown-bytes

Conversation

@kitlangton

Copy link
Copy Markdown
Contributor

Why

HTML-to-Markdown rendering encodes each appended chunk to measure its UTF-8 size, then encodes the same string again when it fits unchanged. The second encoding allocates a redundant byte array for the common path.

What Changes

Reuse the first byte count when the chunk fits. If truncation changes the string, continue measuring the retained prefix as before. Output text, byte budgets, and room reserved for closing syntax are unchanged.

Scope

One production expression and three boundary cases in the existing webfetch tests. No tool contracts or parser chunking changes; this does not address the existing surrogate-pair splitting behavior at parser chunk boundaries.

Verification

cd packages/core
bun run test test/tool-webfetch.test.ts
bun typecheck
cd ../..
bunx prettier --check packages/core/src/tool/html-markdown.ts packages/core/test/tool-webfetch.test.ts
git diff --check

57 tests passed with 102 assertions. Typechecking, formatting, and whitespace checks passed. Added cases cover fitting and truncated ASCII, two-byte, and four-byte UTF-8 content, with surrogate pairs aligned to parser chunks. No throughput benchmark claim.

@kitlangton
kitlangton merged commit e9f7331 into v2 Aug 31, 2026
10 checks passed
@kitlangton
kitlangton deleted the markdown-bytes branch August 31, 2026 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant