bug(vm): gateway restart can retain stale supervisor ownership and fail sandbox recovery

[elezar]

User Story

As an operator running VM-backed sandboxes, I want a graceful gateway restart to restore running sandboxes and preserve explicitly stopped sandboxes, so that routine restarts do not leave workloads in Error.

Problem Statement

The gateway can exit before an asynchronous supervisor-session cleanup task deletes its persisted ownership record. After restart, the new supervisor cannot replace the still-fresh record belonging to the previous instance. ConnectSupervisor returns Unavailable; the supervisor's 10-second acceptance timeout expires before the 45-second ownership TTL, and the sandbox enters Error.

Observed in vm_gateway_restart_preserves_running_and_stopped_intent on both attempts of the VM CI job for #3386:

• First failure
• Rerun failure

The race mechanism was subsequently reproduced locally on main at 50230616d51f352954450c527f9dd64218be0b80 using a controlled delay in ownership cleanup. This establishes the failure mechanism, but does not establish the precise scheduling or database delay in CI.

Impact / Why This Matters

A graceful restart can leave a previously healthy, running-intent sandbox unavailable instead of restoring it. The VM restart E2E failed twice in CI, each time exhausting the readiness wait. Retrying CI did not resolve this occurrence, and repeated manual retries are not a reliable recovery strategy for workloads. No reliable operational workaround was validated in this investigation.

Acceptance Criteria

• A graceful gateway restart restores the running VM sandbox with its overlay state intact, while explicitly stopped sandboxes remain stopped.
• Shutdown waits for supervisor-session ownership cleanup to complete before exiting; delayed cleanup does not leave stale ownership blocking the replacement supervisor.
• Cleanup retains session, replica, and resource-version checks so it cannot delete ownership acquired by a replacement session.
• Shutdown remains bounded and reports incomplete cleanup if persistence is unavailable.
• A deterministic regression test covers shutdown while owner deletion is pending, including a session already removed from the in-memory registry.
• The focused VM restart E2E passes in the Linux x86-64 CI lane.

Reproduction Steps

Existing E2E, without instrumentation

On a machine with the VM driver's documented prerequisites (Linux KVM or supported macOS virtualization, repository build dependencies, and container/image tooling), check out 50230616d51f352954450c527f9dd64218be0b80 in a disposable checkout and run:

OPENSHELL_E2E_VM_TEST=vm_gateway_start mise run e2e:vm

The test creates a running VM sandbox, writes and flushes an overlay marker, creates and explicitly stops another sandbox, then stops and restarts the gateway. It expects the first sandbox to execute cat /sandbox/vm-gateway-start-state and return before-restart while the second remains stopped.

This failure is timing-sensitive. Three uninstrumented local runs with the failing PR's ARM64 binaries passed. A control using the gateway built from updated main also passed in 23.25 seconds.

Controlled reproduction used in the investigation

In the disposable checkout, edit crates/openshell-server/src/supervisor_session.rs. Inside the detached task in handle_connect_supervisor, immediately before owner_index.release_if_current(&owner_guard).await, insert:

// Diagnostic scheduling hook only; not a proposed production change.
if state_clone.gateway_shutting_down.load(Ordering::Acquire)
    && let Ok(delay) = std::env::var("DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS")
    && let Ok(delay_ms) = delay.parse::<u64>()
{
    info!(sandbox_id = %sandbox_id_clone, delay_ms, "diagnostic: delaying shutdown owner release");
    tokio::time::sleep(Duration::from_millis(delay_ms)).await;
    info!(sandbox_id = %sandbox_id_clone, "diagnostic: shutdown owner release delay completed");
}

The following reproduces the tested Linux ARM64 binary combination: build the gateway from main and reuse the other executables from the failing CI run. Artifact downloads require GitHub access and are subject to retention; source builds via mise run e2e:vm are the alternative once artifacts expire.

VM_REPRO_ARTIFACTS="$(mktemp -d)"
gh run download 35723405169 --repo NVIDIA/OpenShell \
  --pattern openshell-driver-vm-aarch64-unknown-linux-gnu \
  --pattern openshell-supervisor-aarch64-unknown-linux-gnu \
  --pattern openshell-aarch64-unknown-linux-musl \
  --pattern openshell-conformance-aarch64-unknown-linux-musl \
  --dir "$VM_REPRO_ARTIFACTS"
chmod +x "$VM_REPRO_ARTIFACTS"/*/openshell*
mise exec -- cargo build -p openshell-gateway

export OPENSHELL_GATEWAY_BIN="$PWD/target/debug/openshell-gateway"
export OPENSHELL_BIN="$VM_REPRO_ARTIFACTS/openshell-aarch64-unknown-linux-musl/openshell"
export OPENSHELL_VM_DRIVER_BIN="$VM_REPRO_ARTIFACTS/openshell-driver-vm-aarch64-unknown-linux-gnu/openshell-driver-vm"
export OPENSHELL_VM_SUPERVISOR_BIN="$VM_REPRO_ARTIFACTS/openshell-supervisor-aarch64-unknown-linux-gnu/openshell-supervisor"
export OPENSHELL_CONFORMANCE_BIN="$VM_REPRO_ARTIFACTS/openshell-conformance-aarch64-unknown-linux-musl/openshell-conformance"

DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS=2000 \
  OPENSHELL_E2E_VM_TEST=vm_gateway_start \
  mise run --no-deps --skip-deps e2e:vm

For the control, run the same command without DIAGNOSTIC_VM_OWNER_RELEASE_DELAY_MS. Remove the diagnostic hook and rebuild after investigation.

Observed: the delay-start message appeared, but the delay-completed message did not: the old gateway exited while cleanup was pending. A read-only query of the per-run SQLite objects table, filtered to object_type = 'supervisor_session_owner', confirmed that the previous session and replica still owned the sandbox after restart. New supervisor connections were rejected, and the full test failed at tests/vm_gateway_start.rs:161 after 260.04 seconds, exit code 101.

Environment

• CI: Linux x86-64, VM compute driver, run 35723405169, PR head c1bb9fdc0d9bedb30d2096880dfeabeb4b8f8a69.
• Local: Ubuntu 24.04.5 LTS, Linux aarch64, kernel 6.17.0-1018-nvidia, KVM available.
• Investigated gateway source: main 50230616d51f352954450c527f9dd64218be0b80, after refactor(runtime): retire Community image dependencies #3386 merged.
• Local CLI, VM driver, supervisor, and conformance executables: ARM64 artifacts from run 35723405169. This was not a full fresh-main runtime build or an x86-64 local reproduction.
• Gateway persistence: per-run on-disk SQLite; managed standalone gateway with mTLS.
• VM bootstrap image: nvcr.io/nvidia/base/ubuntu:24.04.
• Workload image: ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:9a59bb7206905ccaae4f7dab222fbac47c125a21e5fc16f43f427cd6c940ade3.
• Latest release checked: v0.0.116; this report concerns the tested commits, and that release was not tested.

Logs

Condensed local diagnostic timeline; identifiers omitted:

12:50:35.127 diagnostic: delaying shutdown owner release (2000 ms)
12:50:35.163 VM driver shutdown begins
12:50:36.141 new gateway starts
12:50:38.190 SQLite still contains previous session/replica owner record
12:50:41–48 ConnectSupervisor returns gRPC Unavailable
12:50:51.987 sandbox transitions Provisioning -> Error
gateway did not accept supervisor session within 10 seconds
test vm_gateway_restart_preserves_running_and_stopped_intent ... FAILED
test result: FAILED; finished in 260.04s

Investigation Notes

• Gateway shutdown awaits listener and compute cleanup, then returns without joining the supervisor-session tasks.
• Session cleanup runs in a detached tokio::spawn task. It removes the in-memory session before awaiting persisted owner deletion. Merely waiting for the registry to become empty would not close this gap.
• Owner replacement rejects a different supervisor instance while the old ownership record is fresh (45-second TTL).
• Supervisor startup waits only 10 seconds for acceptance.

The fix direction is a bounded shutdown drain that tracks cleanup completion through the conditional database deletion, preserving the existing replacement-session protections. A shutdown sleep or longer startup timeout would mask the missing synchronization.

The workload image was identical between the passing control and failing diagnostic run. Image-dependent timing may expose the race, but a direct image incompatibility was not demonstrated. No production fix was applied during this investigation.

[elezar]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — reproduced with delayed ownership cleanup

Summary

Track supervisor connection initialization and session cleanup independently of the live-session registry. Graceful shutdown will close admission, complete compute cleanup, signal remaining session loops, and wait for a bounded ownership-cleanup drain before exiting.

Scope

• crates/openshell-server/src/supervisor_session.rs: lifecycle tracking before owner publication, cancellation-safe setup ownership, explicit shutdown signal, bounded drain, and deterministic regression tests.
• crates/openshell-server/src/lib.rs: shutdown ordering and error handling that still drains sessions if compute cleanup fails.
• architecture/gateway.md: document the shutdown/ownership invariant.

Implementation Steps

1. Add tracking that covers pending handshakes, superseded sessions, and sessions removed from the in-memory registry until their owner release finishes. Use existing Tokio synchronization primitives where practical.
2. Keep connection setup owned by a task so request cancellation cannot abandon cleanup after owner publication. Close admission before stopping compute; signal session exit and drain afterward.
3. Retain conditional owner deletion and report bounded shutdown failures without changing ownership TTLs or supervisor startup timeouts.
4. Add regression coverage and validate the existing VM restart E2E, including the controlled two-second cleanup delay in a disposable checkout.

Test Plan

• Unit/integration: delayed owner deletion after registry removal; pending admission and rejection after shutdown; old-session cleanup preserving replacement ownership; bounded drain timeout; shutdown signal reaching already-active and late-starting session tasks.
• E2E: focused VM gateway restart, with and without diagnostic cleanup delay. ARM64 local validation is available; x86-64 validation comes from CI.
• Run repository tests, pre-commit, and local CI checks before publishing a PR.

Risks & Open Questions

• Existing HTTP connections can submit RPCs after listener shutdown, so admission and lifetime tracking must cover that ordering.
• Do not stop Kubernetes workloads when draining control sessions.
• An empty registry is not proof that database cleanup completed.
• Persistence failure must remain visible and shutdown bounded; preserve session/replica/resource-version protections.

Documentation Impact

Update the gateway architecture overview. No public configuration, driver defaults, process identity, /proc access, or LSM behavior changes are planned.

---

Revision 1 — initial plan, reviewed against current main

[elezar]

🏗️ build-from-issue-agent

Implementation Complete

PR: #3547

The gateway now closes supervisor-session admission, completes compute shutdown, signals remaining control sessions, and waits up to ten seconds for tracked cleanup. Tracking begins before ownership publication and continues through owner release, including initialization and sessions already removed from the live registry. Existing conditional owner-deletion protections remain intact.

Validation passed: five new deterministic shutdown regression tests, mise run pre-commit, and mise run ci (including 1,754 passing server tests with 8 ignored). The existing VM restart test and both CLI conformance scenarios passed on Linux ARM64, both normally and with the same diagnostic two-second cleanup delay that previously reproduced the failure. That instrumentation is excluded from the PR. Linux x86-64 validation remains for CI.

Updated the gateway architecture, public lifecycle documentation, and cluster diagnostic skill. No issue workflow labels were changed. The PR links this issue for closure when merged.