perf(server): enable WAL and NORMAL sync for the SQLite store

architecture/gateway.md

@@ -648,11 +648,23 @@ Gateway and Sandbox Protocol token responses follow the same convention: a
 present expiration timestamp carries the absolute deadline, while absence means
 the issued token does not expire.
 
+On-disk SQLite databases run in WAL journal mode with `synchronous=NORMAL`.
+The adapter switches the file to WAL on a single connection before the pool
+opens, then applies both settings to every pooled connection. WAL lets readers
+proceed while a writer commits, and `NORMAL` removes the per-commit `fsync`,
+which matters because gateway hot paths such as SSH session issuance and
+revocation are many small autocommit writes. The trade-off is that a power
+loss or kernel crash can roll back the most recent transactions; the database
+remains consistent. Deployments that need stronger durability or multiple
+replicas use Postgres. WAL requires a local filesystem with working shared
+memory, so the SQLite file must not live on a network mount, and backups must
+use `sqlite3 .backup` or `VACUUM INTO` rather than copying the main file alone.
+
 The SQLite adapter tightens the on-disk database file to mode `0o600` on every
 connect so that provider API keys, SSH session tokens, and sandbox metadata are
 not readable by other local users on shared hosts. The same restriction is
-reapplied to the `<db>-wal` and `<db>-shm` sidecars (created by SQLite's
-default WAL journal mode), which mirror the same sensitive contents.
+reapplied to the `<db>-wal` and `<db>-shm` sidecars that WAL mode creates,
+which mirror the same sensitive contents.
 
 Persisted state includes sandboxes, providers, provider profiles, provider
 credential refresh state, SSH sessions, policy revisions, settings, deployment

crates/openshell-server/src/persistence/sqlite.rs

@@ -15,7 +15,9 @@ use openshell_core::SetResourceVersion;
 use openshell_core::paths::set_file_owner_only;
 use openshell_core::proto::Sandbox;
 use prost::Message;
-use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection, SqlitePoolOptions};
+use sqlx::sqlite::{
+    SqliteConnectOptions, SqliteConnection, SqliteJournalMode, SqlitePoolOptions, SqliteSynchronous,
+};
 use sqlx::{Connection, QueryBuilder, Row, Sqlite, SqlitePool};
 use std::path::{Path, PathBuf};
 use std::str::FromStr;
@@ -70,6 +72,69 @@ pub(super) async fn replace_pool_connection(store: &SqliteStore) -> PersistenceR
     connection.close().await.map_err(|e| map_db_error(&e))
 }
 
+/// Apply the on-disk journal settings and switch the database file to WAL
+/// once, before the pool opens its connections.
+///
+/// The gateway's hot paths (SSH-session tokens minted and revoked around every
+/// forwarded connection, sandbox status updates) are many small autocommit
+/// writes. `SQLite`'s default rollback journal makes each of those commits pay
+/// several `fsync` calls and blocks readers while a writer holds the lock, so
+/// under a burst of forwarded connections the whole store serializes on disk
+/// latency. WAL mode removes the reader/writer exclusion and, combined with
+/// `synchronous=NORMAL`, drops the per-commit `fsync`: a power loss or kernel
+/// crash may roll back the most recent transactions, but the database stays
+/// consistent. That is the standard WAL configuration and matches the
+/// single-node scope of the `SQLite` backend; deployments that need stronger
+/// durability guarantees use the Postgres backend.
+///
+/// `journal_mode=WAL` is persistent in the database file, but switching into
+/// it needs exclusive access: if another connection holds the file open, the
+/// switch waits out `busy_timeout` and then fails. Doing it up front on one
+/// connection means the pool connections only ever re-apply the pragma to a
+/// file that is already in WAL mode, which never blocks, and a failure
+/// surfaces as a single clear connect error instead of a pool error later.
+/// The first start after upgrading a rollback-journal database therefore needs
+/// the file to be otherwise unopened. `synchronous` is a per-connection
+/// setting and is applied through the options on every connection.
+///
+/// In-memory databases are left on their defaults: WAL is meaningless there
+/// and the shared-cache keepalive connection already provides their lifetime
+/// guarantees.
+async fn configure_on_disk_durability(
+    options: SqliteConnectOptions,
+) -> PersistenceResult<SqliteConnectOptions> {
+    let options = options
+        .journal_mode(SqliteJournalMode::Wal)
+        .synchronous(SqliteSynchronous::Normal);
+    let wal_error = |e: &sqlx::Error| {
+        PersistenceError::Database(format!(
+            "failed to switch SQLite database {} to WAL journal mode (the switch needs \
+             exclusive access; close other connections to the file and retry): {}",
+            options.get_filename().display(),
+            map_db_error(e)
+        ))
+    };
+    let connection = SqliteConnection::connect_with(&options)
+        .await
+        .map_err(|e| wal_error(&e))?;
+    connection.close().await.map_err(|e| wal_error(&e))?;
+    Ok(options)
+}
+
+#[cfg(test)]
+pub(super) async fn journal_settings(store: &SqliteStore) -> PersistenceResult<(String, i64)> {
+    let mut connection = store.pool.acquire().await.map_err(|e| map_db_error(&e))?;
+    let journal_mode: String = sqlx::query_scalar("PRAGMA journal_mode")
+        .fetch_one(&mut *connection)
+        .await
+        .map_err(|e| map_db_error(&e))?;
+    let synchronous: i64 = sqlx::query_scalar("PRAGMA synchronous")
+        .fetch_one(&mut *connection)
+        .await
+        .map_err(|e| map_db_error(&e))?;
+    Ok((journal_mode, synchronous))
+}
+
 impl SqliteStore {
     /// Closes the connection pool.
     #[cfg(test)]
@@ -107,6 +172,10 @@ impl SqliteStore {
         // so we can restrict the permissions after the database is connected.
         let db_path = (!is_in_memory).then(|| options.get_filename().to_path_buf());
 
+        if !is_in_memory {
+            options = configure_on_disk_durability(options).await?;
+        }
+
         let in_memory_keepalive = if is_in_memory {
             let connection = SqliteConnection::connect_with(&options)
                 .await

crates/openshell-server/src/persistence/tests.rs

@@ -324,14 +324,217 @@ async fn sqlite_connect_tightens_existing_db_file_permissions() {
     assert_eq!(mode, 0o600, "expected 0600, got {mode:04o}");
 }
 
+fn on_disk_store_url(db_path: &std::path::Path) -> String {
+    format!("sqlite:{}?mode=rwc", db_path.display())
+}
+
+async fn connect_on_disk_sqlite(url: &str) -> super::SqliteStore {
+    match Store::connect(url).await.expect("connect to sqlite") {
+        Store::Sqlite(store) => store,
+        Store::Postgres(_) => unreachable!("sqlite URL must select the SQLite store"),
+    }
+}
+
+async fn file_journal_mode(db_path: &std::path::Path) -> String {
+    use sqlx::{Connection, SqliteConnection};
+
+    let mut connection = SqliteConnection::connect(&format!("sqlite:{}", db_path.display()))
+        .await
+        .expect("open database file directly");
+    let journal_mode: String = sqlx::query_scalar("PRAGMA journal_mode")
+        .fetch_one(&mut connection)
+        .await
+        .expect("read journal_mode");
+    connection.close().await.expect("close direct connection");
+    journal_mode
+}
+
+#[tokio::test]
+async fn sqlite_connect_enables_wal_and_normal_synchronous_on_disk() {
+    let tmp = tempfile::tempdir().expect("tempdir");
+    let db_path = tmp.path().join("openshell.db");
+    let url = on_disk_store_url(&db_path);
+
+    let store = connect_on_disk_sqlite(&url).await;
+
+    let (journal_mode, synchronous) = super::sqlite::journal_settings(&store)
+        .await
+        .expect("read journal settings through the pool");
+    assert_eq!(journal_mode, "wal", "on-disk stores must run in WAL mode");
+    assert_eq!(
+        synchronous, 1,
+        "on-disk stores must run with synchronous=NORMAL (1), got {synchronous}"
+    );
+
+    // Force a write so the WAL sidecars exist on disk, then confirm they are
+    // owner-only like the main file.
+    store
+        .put(
+            "sandbox",
+            "wal-probe",
+            "wal-probe",
+            "default",
+            b"payload",
+            None,
+        )
+        .await
+        .expect("write through the store");
+    let [wal_path, shm_path] = super::sqlite::sqlite_sidecar_paths(&db_path);
+    assert!(wal_path.exists(), "WAL sidecar should exist after a write");
+    assert!(shm_path.exists(), "SHM sidecar should exist after a write");
+    #[cfg(unix)]
+    {
+        use std::os::unix::fs::PermissionsExt;
+        for path in [&db_path, &wal_path, &shm_path] {
+            let mode = std::fs::metadata(path)
+                .expect("sidecar metadata")
+                .permissions()
+                .mode()
+                & 0o777;
+            assert_eq!(
+                mode,
+                0o600,
+                "{}: expected 0600, got {mode:04o}",
+                path.display()
+            );
+        }
+    }
+
+    store.close().await;
+    assert_eq!(
+        file_journal_mode(&db_path).await,
+        "wal",
+        "WAL must persist in the database file after the store closes"
+    );
+}
+
+#[tokio::test]
+async fn sqlite_connect_switches_existing_rollback_journal_database_to_wal() {
+    use sqlx::sqlite::SqliteConnectOptions;
+    use sqlx::{Connection, SqliteConnection};
+    use std::str::FromStr;
+
+    let tmp = tempfile::tempdir().expect("tempdir");
+    let db_path = tmp.path().join("openshell.db");
+    let url = on_disk_store_url(&db_path);
+
+    // A database created by an older gateway, or by any sqlx 0.8 default
+    // connection, is in rollback-journal (`delete`) mode.
+    {
+        let options = SqliteConnectOptions::from_str(&url)
+            .expect("parse url")
+            .create_if_missing(true);
+        let mut connection = SqliteConnection::connect_with(&options)
+            .await
+            .expect("create legacy database");
+        sqlx::query("CREATE TABLE legacy_probe (x INTEGER)")
+            .execute(&mut connection)
+            .await
+            .expect("write legacy database");
+        connection.close().await.expect("close legacy connection");
+    }
+    assert_eq!(file_journal_mode(&db_path).await, "delete");
+
+    let store = connect_on_disk_sqlite(&url).await;
+    let (journal_mode, _) = super::sqlite::journal_settings(&store)
+        .await
+        .expect("read journal settings");
+    assert_eq!(
+        journal_mode, "wal",
+        "connect must switch existing files to WAL"
+    );
+    store.close().await;
+    assert_eq!(file_journal_mode(&db_path).await, "wal");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn sqlite_file_backed_reads_and_writes_proceed_concurrently() {
+    let tmp = tempfile::tempdir().expect("tempdir");
+    let db_path = tmp.path().join("openshell.db");
+    let url = on_disk_store_url(&db_path);
+    let store = std::sync::Arc::new(Store::connect(&url).await.expect("connect to sqlite"));
+
+    store
+        .put("sandbox", "seed", "seed", "default", b"seed", None)
+        .await
+        .expect("seed object");
+
+    // Mirror the forward-service pattern: many independent autocommit writes
+    // (one insert and one update per "connection") while readers keep
+    // fetching. Every operation must complete; no caller may observe a
+    // "database is locked" error even though the writes contend for the
+    // single SQLite writer.
+    let writers = (0..8).map(|writer| {
+        let store = store.clone();
+        tokio::spawn(async move {
+            for index in 0..25 {
+                let id = format!("session-{writer}-{index}");
+                store
+                    .put_if(
+                        "ssh_session",
+                        &id,
+                        &id,
+                        "default",
+                        b"issued",
+                        None,
+                        super::WriteCondition::MustCreate,
+                    )
+                    .await
+                    .expect("insert session");
+                store
+                    .put_if(
+                        "ssh_session",
+                        &id,
+                        &id,
+                        "default",
+                        b"revoked",
+                        None,
+                        super::WriteCondition::MatchResourceVersion(1),
+                    )
+                    .await
+                    .expect("revoke session");
+            }
+        })
+    });
+    let readers = (0..4).map(|_| {
+        let store = store.clone();
+        tokio::spawn(async move {
+            for _ in 0..100 {
+                let record = store
+                    .get("sandbox", "seed")
+                    .await
+                    .expect("read while writers are active")
+                    .expect("seed object present");
+                assert_eq!(record.payload, b"seed");
+            }
+        })
+    });
+
+    for handle in writers.chain(readers) {
+        handle.await.expect("task completed");
+    }
+
+    let sessions = store
+        .list("ssh_session", "default", 1000, 0)
+        .await
+        .expect("list sessions");
+    assert_eq!(sessions.len(), 200, "every session write must be durable");
+    assert!(
+        sessions.iter().all(|record| record.payload == b"revoked"),
+        "every session must have been revoked by its follow-up write"
+    );
+}
+
 // The next three tests cover `restrict_db_file_permissions` against the
 // WAL/SHM sidecars at increasing levels of fidelity:
 //
 // 1. `_tightens_main_and_wal_and_shm_files`: synthetic empty files, proves
 //    the chmod loop walks all three paths.
-// 2. `_skips_missing_sidecars`: proves the `exists()` guard, which is the
-//    actual production path today (sqlx 0.8 doesn't default to WAL and
-//    doesn't accept `journal_mode` as a URL parameter).
+// 2. `_skips_missing_sidecars`: proves the `exists()` guard for databases
+//    that have not been written yet or were created before the adapter
+//    enabled WAL (sqlx 0.8 doesn't default to WAL and doesn't accept
+//    `journal_mode` as a URL parameter; `SqliteStore::connect` opts in
+//    through the builder API).
 // 3. `_handles_real_sqlite_wal_files`: opens a real sqlx pool with
 //    `SqliteJournalMode::Wal` via the builder API so SQLite materializes
 //    real `-wal` and `-shm` files, then checks the helper tightens them.

deploy/helm/openshell/README.md

@@ -281,7 +281,7 @@ discovery endpoint or its TLS CA.
 | server.credentialDrivers.vault.timeoutSecs | string | `""` | HTTP request timeout in seconds. Empty = driver default. |
 | server.credentialDrivers.vault.tokenPath | string | `""` | Mounted token file path when authMethod is token_file. |
 | server.credentialStorage.existingSecret | string | `""` | Name of a pre-existing Secret containing the key-encryption key. When set, the chart does NOT generate a new Secret; it references this one instead. The Secret must contain a key named "key-encryption-key" with a base64-encoded 32-byte value. Required for GitOps workflows that render manifests with `helm template` (where `lookup` is unavailable). |
-| server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). |
+| server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). SQLite runs in WAL mode and needs a local block-backed volume, not NFS or other network filesystems. |
 | server.defaultRuntimeClassName | string | `""` | Default Kubernetes runtimeClassName for sandbox pods. Applied when a CreateSandbox request does not specify one. Empty (default) = omit the field, using the cluster's default RuntimeClass. Set to a RuntimeClass name (e.g. "kata-containers", "nvidia") to apply it to all sandboxes that don't explicitly override it. |
 | server.disableTls | bool | `false` | Disable TLS entirely - the server listens on plaintext HTTP. Set to true when a reverse proxy / tunnel terminates TLS at the edge. |
 | server.drivers.kubernetes.operatorNamespaceFile | string | `""` | Path to a JSON file containing an array of namespace names allowed in operator mode. Hot-reloaded on change. |

deploy/helm/openshell/values.yaml

@@ -206,7 +206,9 @@ server:
   # -- Namespace where sandbox pods are created. Defaults to the Helm release
   # namespace (.Release.Namespace) when left empty.
   sandboxNamespace: ""
-  # -- Gateway database URL (used for the default SQLite backend).
+  # -- Gateway database URL (used for the default SQLite backend). SQLite runs
+  # in WAL mode and needs a local block-backed volume, not NFS or other network
+  # filesystems.
   dbUrl: "sqlite:/var/openshell/openshell.db"
   # -- Name of a pre-existing Opaque Secret containing a PostgreSQL
   # connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL

deploy/rpm/CONFIGURATION.md

@@ -229,6 +229,9 @@ overrides that persist across package upgrades.
 
 The database URL is not accepted in TOML. When `OPENSHELL_DB_URL` is unset,
 the gateway uses `sqlite:$XDG_STATE_HOME/openshell/gateway/openshell.db`.
+The SQLite database runs in WAL mode with `synchronous=NORMAL`, so
+`openshell.db-wal` and `openshell.db-shm` sit next to it and must be kept
+together with it; back it up with `sqlite3 openshell.db ".backup <copy>"`.
 
 ### Driver TOML settings
 

docs/reference/gateway-config.mdx

@@ -18,6 +18,8 @@ Gateway CLI flag  >  gateway OPENSHELL_* env var  >  TOML file  >  built-in defa
 
 `database_url` is env-only. The loader rejects it when it appears in the file. When `OPENSHELL_DB_URL` is unset, the gateway stores its SQLite database under `$XDG_STATE_HOME/openshell/gateway/openshell.db`.
 
+On-disk SQLite databases run in WAL mode with `synchronous=NORMAL`: the gateway keeps `openshell.db-wal` and `openshell.db-shm` next to the database file, the file must live on a local filesystem, and a backup must use `sqlite3 openshell.db ".backup <copy>"` rather than copying `openshell.db` on its own. A power loss can roll back the most recent transactions without corrupting the database.
+
 `name` assigns an operator-facing identity to the gateway installation. Set it with `[openshell.gateway].name`, `--name`, or `OPENSHELL_GATEWAY_NAME`. It defaults to `openshell`; the Helm chart defaults it to the chart fullname so all replicas in one installation share a name. Chart fullnames are only unique within their Kubernetes namespace, so set `server.name` explicitly when one collector receives telemetry from multiple namespaces or clusters. This identity is independent of client-side gateway aliases, TLS names, and `gateway_jwt.gateway_id`.
 
 ## Package-Managed Locations