Conversation
…2 overflow Large databases crashed the legacy dump endpoint: it accumulated the entire dump in one in-memory string inside a single 30s request. This replaces it with a bounded, resumable dump engine: - ChunkedDumpEngine: rowid-batched table scans, time-budgeted cycles with breathing intervals between them, progress persisted in DO storage (crash/resume safe, no duplicate emission on resume) - Chunks spill to R2 when a bucket is bound, with DO-storage fallback and streaming reassembly - /export/dump stays synchronous for small DBs (backwards compatible); requests that hit the 30s window are upgraded to a job driven by DO alarms, with /export/dump/status polling - New /export/dump/status route and DO RPC (startDumpJob/dumpJobStatus) - 12 unit tests covering resume, chunking, R2, redaction-safe serialization and identifier injection guards
… URLs After a chunked dump completes, a DO-alarm-driven finalize now merges all chunk records into a single R2 object (dumps/<dumpId>/<fileName>) via a multipart upload, making large dumps downloadable as one object: - finalizeDump is time-budgeted and crash-resumable: uploaded parts and their etags are persisted after every part, interrupted finalizes resume via resumeMultipartUpload without re-uploading completed bytes, and a multi-GB consolidation progresses across several alarm invocations - getPresignedUrl feature-detects R2Bucket.createSignedUrl (newer workerd runtimes) and returns an expiring download URL; older bindings fall back to the existing streaming reassembly - /export/dump/status returns downloadUrl (+size, finalObjectKey) when a presigned URL is available; assembleDump prefers the consolidated object - Per-chunk R2 mirrors are cleaned up after a successful complete - parseDumpOptions: new partBytes/finalizeMs tuning params - 7 new unit tests (19 total): part sizing/order, byte-offset resume, budget-out-then-continue, no-R2 finalize, signed-URL presence/shape guards, consolidated-object preference
Author
⬆️ Upgrade: consolidated R2 multipart finalize + presigned download URLs (
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #59
Problem
The legacy
/export/dumppath accumulated the entire database in a single in-memory string inside one request. On large databases this both blows the 30s Workers request window and exhausts memory, and a mid-request crash loses all progress.Solution — bounded, resumable dump engine
New
ChunkedDumpEngine(src/export/chunkedDump.ts):rowsPerBatch, default 200) — bounded memory per batch.cycleTimeBudgetMs, default 4s) — each cycle does bounded work, then yields with a breathing interval (breathingIntervalMs, default 5s) so concurrent queries are starved for at most one cycle, not the whole dump.tmp_dump_state) after every yield — a crashed/restarted dump resumes exactly where it stopped, and a resume never re-emits already-serialized schema (tableIndex advances before the yield point).R2_DUMP_BUCKETunder<dumpId>/00000000.sql-style keys when bound; DO storage keeps the tail window as fallback, and reassembly is a streamed concatenation — the client never holds the whole dump in memory.Job flow (
src/export/dump.ts,src/do.ts,src/handler.ts):POST /export/dump→ synchronous response for fast dumps; otherwise returns{ jobId }immediately.GET /export/dump/status(DO RPCdumpJobStatus) → progress table + downloadable stream when complete.Spec checklist (issue #59)
chunkTargetBytes)>100MBfriendly: chunked put + streamed concat read)Advantages over #76 / #93
Testing
npx vitest run src/export/chunkedDump.test.ts→ 12 passed / 0 failedsrc/rls/index.test.tsand 11 typecheck errors (do.test.ts mocks, cache, cdc) reproduce on a clean checkout ofmain— pre-existing, untouched by this PR.src/export/dump.test.tsstill passes with the new engine wired in.```
Tests 12 passed (12)
Test Files 1 passed (1)
```