fix(sandbox): reclaim socket descriptors before exhaustion

[pimlock]

Summary

Keep socket-heavy workloads from exhausting the sandbox broker's file descriptors and taking down the sandbox control connection.

Problem

The broker retains a file descriptor for each socket in a pre-connect state. When the workload closes one, the broker does not learn about it immediately. It removes the stale descriptor during cleanup.

The socket registry allows 4,096 metadata entries, but the broker may have a soft RLIMIT_NOFILE of 1,024. Previously, cleanup normally started only when the registry filled or socket(2) returned EMFILE.

For example:

1. A workload repeatedly creates and closes unconnected sockets.
2. Stale broker descriptors accumulate until the process reaches its soft limit.
3. The broker may no longer have a descriptor available to scan /proc or serve the control path.
4. The sandbox remains marked ready, but exec and relay requests stop responding.

Connected sockets are different. The broker releases their source descriptors but keeps their metadata for policy mediation, so the metadata limit must remain separate from the descriptor limit.

Fix

Keep the 4,096-entry metadata limit and add a separate retained-descriptor budget. The budget uses the inherited soft RLIMIT_NOFILE, subtracts descriptors already open at startup, and reserves 64 more for control-path work.

With a soft limit of 1,024 and 24 descriptors already open, the broker can retain up to 936 pre-connect descriptors. Connected socket metadata can still use the full registry because those entries no longer retain broker descriptors.

Before opening another socket, the broker runs cleanup if either limit is full. If cleanup cannot free capacity, the workload receives EMFILE while the broker keeps enough descriptors to scan /proc and serve control traffic.

Related Issue

No issue required. This is a localized reliability bug found while investigating a sandbox that stopped responding under socket churn.

Changes

• Keep separate bounds for socket metadata and retained descriptors.
• Account for descriptors already open when the broker starts.
• Reclaim stale sockets before descriptor exhaustion.
• Test both retained pre-connect sockets and connected metadata.

Testing

• mise run pre-commit
• mise run ci
• mise run e2e

Checklist

• Commit messages follow Conventional Commits.
• Commits include DCO sign-offs.
• Tests cover the new behavior.
• Full repository CI passes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

Label test:e2e applied, but pull-request/3532 is at b74e9d3 while the PR head is 4dbc2cd. A maintainer needs to comment /ok to test 4dbc2cd80bb690a95884d19ee402e37ec27ff199 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[pimlock]

gator-agent

PR Review Status

This focused sandbox reliability fix is project-valid and the initial code review found no blocking issues. The descriptor budget remains separate from socket metadata capacity, preserves control-path headroom, and is covered by targeted regression tests.

Action required: A human maintainer must review and approve the PR.

Blocking findings:

• No blocking findings remain

Carried findings:

• None

Non-blocking suggestions:

• None

Gator metadata

• Validation: Focused, well-motivated sandbox reliability fix with no duplicate candidate; the author is a repository maintainer.
• Docs: Not needed because this changes internal descriptor reclamation without changing a published user workflow or interface.
• Checks: Current-head Branch Checks, Helm Lint, Trivy Changes, DCO, and other required gates are green.
• E2E: test:e2e is applied and OpenShell / E2E is green on the current head.
• Head SHA: 009bbedc4a523b687606c142b35832069010333f
• Base SHA: c8a4ff5f19ded0ad9f70886f902c3881055941e3
• Merge base SHA: c8a4ff5f19ded0ad9f70886f902c3881055941e3
• Patch ID: efb0a0a7815e0e117e01b27888406cb3c804ebea
• Gator payload: 9
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:approval-needed

[pimlock]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: Gator found no blocking review issues, all required checks including E2E passed, and a verified maintainer approved the current head before merge.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.