Conversation
On-disk SQLite stores ran with sqlx defaults: rollback journal (`journal_mode=delete`) and `synchronous=FULL`. Every autocommit write paid several fsyncs and blocked readers while it held the lock, so gateway hot paths made of many small writes serialized on disk latency. The clearest case is `openshell forward service`, which mints and revokes an SSH session token around every forwarded TCP connection: two commits per connection, tens of milliseconds each on a virtual disk, wall clock linear in the number of concurrent connections, and enough queueing that bursts hit the per-sandbox connection cap and get refused. Switch on-disk databases to WAL with `synchronous=NORMAL`. The mode change runs once on a single connection before the pool opens: entering WAL needs exclusive access to the file, so doing it up front means pool connections only ever re-apply the pragma to a file already in WAL mode, and a failure surfaces as one clear connect error. The first start after upgrading an existing database therefore needs the file to be otherwise unopened. `synchronous` is applied through the connect options on every pooled connection. In-memory databases keep their defaults. A crash can now roll back the most recent transactions without corrupting the database, which is the standard WAL trade-off and fits the single-node scope of the SQLite backend. Tests cover a fresh store, an existing rollback-journal file that must be switched on connect, sidecar permissions, and concurrent readers under a burst of insert-then-update writes. Architecture, configuration and Helm docs describe the durability trade-off, the sidecar files, and the local filesystem requirement. Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
n1hility
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 22, 2026 10:15
|
All contributors have signed the DCO ✍️ ✅ |
Author
|
I have read the DCO document and I hereby sign the DCO. |
Collaborator
|
/ok to test 5cced0e |
|
Label |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On-disk SQLite stores ran with sqlx defaults (rollback journal,
synchronous=FULL), so every autocommit write paid several fsyncs and blocked readers.openshell forward servicedoes two such writes per forwarded TCP connection (session token minted and revoked), which made connection setup through a forward linear in the number of concurrent connections and pushed bursts into the per-sandbox connection cap. This switches on-disk stores to WAL withsynchronous=NORMAL.Related Issue
Fixes #3494
Changes
SqliteStore::connect: for on-disk URLs, switch the file tojournal_mode=WALonce on a single connection before the pool opens (entering WAL needs exclusive access; done up front so pool connections only ever re-apply the pragma to a file already in WAL mode and a failure surfaces as one clear connect error), then build the pool withjournal_mode=WALandsynchronous=NORMAL. In-memory databases are unchanged. A failed switch reports the file path and the exclusive-access requirement.persistence/tests.rs): fresh on-disk store reportswalandsynchronous=1and its-wal/-shmsidecars are 0600; an existing rollback-journal database is switched on connect; concurrent readers proceed under a burst of insert-then-update writes on a file-backed store; the stale comment about non-WAL production is reworded.architecture/gateway.md(durability trade-off, backup withsqlite3 .backup/VACUUM INTO, local filesystem requirement),docs/reference/gateway-config.mdx,deploy/rpm/CONFIGURATION.md, Helmvalues.yaml/README note that the SQLite volume must be local block storage.Out of scope, noted for follow-up: an explicit store close on gateway shutdown for a final checkpoint; minting one session token per forward process instead of per connection; making the per-sandbox forward connection cap configurable.
Testing
cargo fmt -p openshell-server -- --check,cargo clippy -p openshell-server --all-targets --features test-support -- -D warnings,python3 scripts/update_license_headers.py --checkall clean on the rebased branch (main @ the base of this PR).cargo test -p openshell-server --features test-support persistence: 85 passed, 0 failed, 3 pre-existing ignores; the three new tests were also repeated 40 times with--test-threads=8without a failure.mise run pre-commit/mise run ci: not run locally (no mise on this machine); the equivalent fmt, clippy, license and unit checks above were.Before/after with the same source, gateway on a hosted runner with the Docker driver, a sandbox serving loopback HTTP,
openshell forward servicein front, N simultaneous connections each doing one request (reproducer and scripts: https://github.com/n1hility/OpenShell/tree/forward-sweep-repro, workflowforward-sweep.yml; details on #3494):On a 2 vCPU VM with SQLite on a network block volume the same sweep went from 1.52 s to 0.06 s at 16 connections and from 6.05 s to 0.12 s at 64. Remaining refusals above 20 connections are the fixed per-sandbox cap, independent of this change.
Checklist