fix(server): drain supervisor ownership cleanup on shutdown

[elezar]

Summary

A graceful gateway restart could interrupt supervisor-session ownership deletion, leaving a fresh owner record that rejected the replacement VM supervisor until its startup timed out. The gateway now closes admission, finishes compute shutdown, and drains tracked session cleanup for up to ten seconds before exiting.

Related Issue

Closes #3546

Follow-up: #3551 investigates structured ownership and shutdown coordination for gateway asynchronous tasks.

Changes

• Track connection setup and session cleanup from before owner publication through completion, including sessions already removed from the live registry.
• Keep admitted setup owned by a task if the RPC caller disconnects. Signal remaining session loops after compute shutdown, including when compute cleanup fails.
• Preserve conditional owner deletion and replacement-session protections. Report drain timeouts without changing ownership TTLs or supervisor startup timeouts.
• Add five regression tests for delayed deletion, admission during shutdown, replacement ownership, stalled cleanup, and empty drains.
• Update the gateway architecture, lifecycle documentation, and cluster diagnostic skill.

Testing

• mise run pre-commit
• mise run ci (includes the full unit-test task; server suite: 1,754 passed, 8 ignored)
• Five new focused shutdown regression tests passed.
• Existing vm_gateway_restart_preserves_running_and_stopped_intent passed locally on Linux ARM64, with CLI smoke and sandbox-lifecycle conformance scenarios also passing.
• Controlled regression experiment: the same two-second shutdown ownership-release delay that reproduced the original failure now passes (25.12 seconds versus the original 260.04-second failure). The diagnostic hook exists only in a disposable checkout and is not part of this PR.
• GitHub Branch Checks and core E2E passed, including managed and external-driver VM E2E on Linux x86-64.

The local VM tests used the gateway built with this fix and the CLI, VM driver, supervisor, and conformance ARM64 artifacts from run 35723405169. No E2E test files were changed.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3547.docs.buildwithfern.com/openshell

[elezar]

🏗️ build-from-issue-agent

Local VM E2E Validation

Implementation commit: d713056b2.

Command: OPENSHELL_E2E_VM_TEST=vm_gateway_start mise run e2e:vm equivalent through mise exec -- bash e2e/rust/e2e-vm.sh, with explicit prebuilt-binary overrides. Gateway built from the fix; CLI, VM driver, supervisor, and conformance executables reused from the Linux ARM64 artifacts of run 35723405169. Host: Ubuntu 24.04.5, aarch64, KVM.

Executed in each run:

• CLI conformance smoke: passed.
• CLI conformance sandbox-lifecycle: passed.
• vm_gateway_restart_preserves_running_and_stopped_intent: passed; no Rust E2E tests skipped.

Normal run: 1 Rust test passed in 25.25 seconds. Controlled diagnostic run: 1 Rust test passed in 25.12 seconds with a two-second delay immediately before shutdown owner release. Before the fix, that experiment failed after 260.04 seconds with a stale owner record and the supervisor acceptance timeout.

The delay hook was applied only in a disposable checkout; it is not committed. The final source-only Clippy correction from _ to () in the shutdown select arm does not change behavior. The controlled-delay build includes that correction, and full local CI passed on the final source. Linux x86-64 VM coverage remains for CI.

[github-actions]

Label test:e2e applied for d713056. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

gator-agent

Maintainer Approval Needed

Gator reviewed the complete gateway shutdown change and found no blocking issues. Thanks @elezar; I checked the controlled-delay validation you reported against the implementation and confirmed that the current-head required E2E and VM jobs are green.

Action required: A maintainer should review and either approve or request changes.

Blocking findings:

• None

Carried findings:

• None

Gator metadata

• Validation: Maintainer-authored, focused fix for the reproducible gateway ownership-cleanup race in linked issue #3546
• Docs: Gateway architecture, published sandbox lifecycle documentation, and the cluster diagnostic skill are updated; existing navigation remains applicable
• Checks: Current-head Branch Checks, Helm Lint, Trivy Changes, DCO, docs preview, and required gates are green
• E2E: test:e2e is applied; OpenShell / E2E, VM managed, and VM external-driver coverage are green
• Head SHA: d713056b2d899aabab60a7607d7ea030e0898110
• Base SHA: 50230616d51f352954450c527f9dd64218be0b80
• Merge base SHA: 50230616d51f352954450c527f9dd64218be0b80
• Patch ID: da509eb7d16fbb21e2e2fdf0a9a8deb4445d7e0f
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:approval-needed

[drew]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator found no blocking review issues, the required checks and E2E coverage passed, and maintainer approval was present before merge.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.