Skip to content

feat: chunked, resumable dumps with breathing intervals + R2 overflow - #298

Open
Furox-Art wants to merge 2 commits into
outerbase:mainfrom
Furox-Art:chunked-dump-breathing
Open

Furox-Art wants to merge 2 commits into
outerbase:mainfrom
Furox-Art:chunked-dump-breathing

Conversation

@Furox-Art

Copy link
Copy Markdown

Closes #59

Problem

The legacy /export/dump path accumulated the entire database in a single in-memory string inside one request. On large databases this both blows the 30s Workers request window and exhausts memory, and a mid-request crash loses all progress.

Solution — bounded, resumable dump engine

New ChunkedDumpEngine (src/export/chunkedDump.ts):

  • Rowid-batched table scans (rowsPerBatch, default 200) — bounded memory per batch.
  • Time-budgeted cycles (cycleTimeBudgetMs, default 4s) — each cycle does bounded work, then yields with a breathing interval (breathingIntervalMs, default 5s) so concurrent queries are starved for at most one cycle, not the whole dump.
  • Progress persisted in DO storage (tmp_dump_state) after every yield — a crashed/restarted dump resumes exactly where it stopped, and a resume never re-emits already-serialized schema (tableIndex advances before the yield point).
  • R2 overflow: each chunk spills to R2_DUMP_BUCKET under <dumpId>/00000000.sql-style keys when bound; DO storage keeps the tail window as fallback, and reassembly is a streamed concatenation — the client never holds the whole dump in memory.
  • Injection guard: table names are filtered through an identifier whitelist before being interpolated into any SQL.
  • Backwards compatible: small DBs still get the old synchronous response; only when the 30s window is threatened does the route upgrade to a job.

Job flow (src/export/dump.ts, src/do.ts, src/handler.ts):

  • POST /export/dump → synchronous response for fast dumps; otherwise returns { jobId } immediately.
  • Continuation is driven by DO alarms (self-rescheduling cycles with breathing delays — no external cron).
  • GET /export/dump/status (DO RPC dumpJobStatus) → progress table + downloadable stream when complete.

Spec checklist (issue #59)

  • Chunked dump — rowid batches + byte-sized chunks (chunkTargetBytes)
  • Breathing intervals between chunks (configurable, default 5s)
  • Exceeds 30s → job continues via DO alarms, request returns immediately
  • Large DB → R2 (>100MB friendly: chunked put + streamed concat read)
  • No memory blow-up — nothing accumulates unbounded
  • Status/poll endpoint for progress + download

Advantages over #76 / #93

Aspect This PR
Resumability Crash-safe state machine; resume tested (no duplicate DDL emission)
Memory Bounded at every step: batch, chunk, and reassembly are all streamed
Yield fairness Explicit breathing interval between cycles, driven by alarms
Storage Dual-path: R2 when bound, DO storage fallback, streamed reassembly
Back-compat Small dumps keep the old synchronous contract unchanged
Tests 12 unit tests: resume-mid-schema, resume-mid-data, chunk flush, R2 writes, reassembly order, injection guard, serialization

Testing

  • npx vitest run src/export/chunkedDump.test.ts12 passed / 0 failed
  • Full suite: 163 passed; the 4 failures in src/rls/index.test.ts and 11 typecheck errors (do.test.ts mocks, cache, cdc) reproduce on a clean checkout of main — pre-existing, untouched by this PR.
  • Regression: src/export/dump.test.ts still passes with the new engine wired in.

```
Tests 12 passed (12)
Test Files 1 passed (1)
```

…2 overflow

Large databases crashed the legacy dump endpoint: it accumulated the
entire dump in one in-memory string inside a single 30s request. This
replaces it with a bounded, resumable dump engine:

- ChunkedDumpEngine: rowid-batched table scans, time-budgeted cycles
  with breathing intervals between them, progress persisted in DO
  storage (crash/resume safe, no duplicate emission on resume)
- Chunks spill to R2 when a bucket is bound, with DO-storage fallback
  and streaming reassembly
- /export/dump stays synchronous for small DBs (backwards compatible);
  requests that hit the 30s window are upgraded to a job driven by DO
  alarms, with /export/dump/status polling
- New /export/dump/status route and DO RPC (startDumpJob/dumpJobStatus)
- 12 unit tests covering resume, chunking, R2, redaction-safe
  serialization and identifier injection guards
… URLs

After a chunked dump completes, a DO-alarm-driven finalize now merges all
chunk records into a single R2 object (dumps/<dumpId>/<fileName>) via a
multipart upload, making large dumps downloadable as one object:

- finalizeDump is time-budgeted and crash-resumable: uploaded parts and
  their etags are persisted after every part, interrupted finalizes resume
  via resumeMultipartUpload without re-uploading completed bytes, and a
  multi-GB consolidation progresses across several alarm invocations
- getPresignedUrl feature-detects R2Bucket.createSignedUrl (newer workerd
  runtimes) and returns an expiring download URL; older bindings fall back
  to the existing streaming reassembly
- /export/dump/status returns downloadUrl (+size, finalObjectKey) when a
  presigned URL is available; assembleDump prefers the consolidated object
- Per-chunk R2 mirrors are cleaned up after a successful complete
- parseDumpOptions: new partBytes/finalizeMs tuning params
- 7 new unit tests (19 total): part sizing/order, byte-offset resume,
  budget-out-then-continue, no-R2 finalize, signed-URL presence/shape
  guards, consolidated-object preference
@Furox-Art

Copy link
Copy Markdown
Author

⬆️ Upgrade: consolidated R2 multipart finalize + presigned download URLs (7eed275)

Building on the chunked engine, dumps now land in R2 as one object instead of N chunk keys:

What's new

  • finalizeDump() (alarm-driven, outside the HTTP window): after completion, all chunk records are merged into dumps/<dumpId>/<fileName> via an R2 multipart upload. Time-budgeted per alarm invocation, so even a multi-GB consolidation progresses across several invocations instead of dying in one 30s window.
  • Crash-resumable at the byte level: every uploaded part's partNumber+etag is persisted in DO storage; an interrupted finalize resumes with resumeMultipartUpload and continues from the exact byte offset — no part is ever re-uploaded (unit-tested).
  • Presigned download URLs: /export/dump/status now returns downloadUrl (expiring, 1h) when the runtime exposes R2Bucket.createSignedUrl. Feature-detected — older bindings gracefully fall back to the existing streaming endpoint (also unit-tested for both shapes).
  • Cleanup: per-chunk R2 mirrors are deleted after a successful complete (DO-storage records remain as the streaming fallback). assembleDump prefers the consolidated object.
  • New tuning params: ?partBytes= / ?finalizeMs= (part size defaults to the R2 minimum of 5 MiB for non-final parts).

Evidence

npx vitest run src/export/chunkedDump.test.ts → 19 passed / 0 failed  (12 existing + 7 new)

New tests cover: part sizing & ordering, byte-offset resume, budget-exhaustion mid-upload then continue, no-R2 finalize, createSignedUrl presence/shape guards, and consolidated-object preference in assembleDump. Full suite: 170 passed; the 4 rls failures are pre-existing on main (documented in the PR description).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant