Skip to content

fix(podman): restore host gateway alias mediation - #3606

Open
grs wants to merge 4 commits into
NVIDIA:mainfrom
grs:internal-host-fix
Open

grs wants to merge 4 commits into
NVIDIA:mainfrom
grs:internal-host-fix

Conversation

@grs

@grs grs commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Restore host.openshell.internal connectivity for Podman-backed sandboxes while preserving OpenShell’s mediated networking boundary.

Related Issue

Fixes #3605

Changes

  • Resolve a trusted Podman host gateway address, with platform-specific defaults.
  • Reject unsafe gateway addresses, including unspecified, multicast, broadcast, and cloud metadata addresses.
  • Pass the resolved host gateway address to the supervisor runtime descriptor.
  • Mount a driver-owned, read-only /etc/resolv.conf that directs workload DNS to the supervisor at 127.0.0.53.
  • Preserve resolver configuration across sandbox restarts.
  • Clean up per-sandbox resolver secrets after creation failures and sandbox deletion.
  • Add a Podman end-to-end test that connects to a host HTTP listener through host.openshell.internal, including after restart.
  • Update Podman networking, sandbox architecture, gateway configuration, and troubleshooting documentation.

The implementation uses a Podman secret for /etc/resolv.conf instead of uploading the file into the workload filesystem. This avoids image-controlled path and symlink behavior and persists naturally across
restarts.

Testing

  • mise run pre-commit passes
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

Signed-off-by: Gordon Sim <gsim@redhat.com>
@grs
grs requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 23, 2026 11:56
@copy-pr-bot

copy-pr-bot Bot commented Sep 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers

Copy link
Copy Markdown
Collaborator

Thanks for restoring this through the mediated DNS path and for adding restart coverage. I found two gaps that should be addressed before merge:

[P1] The new regression test is not run by required PR CI.

Adding podman_host_gateway to e2e/rust/Cargo.toml only registers the test target. The only Podman branch job currently runs mise run e2e:podman:external-driver; that task sets OPENSHELL_E2E_PODMAN_FEATURES="", and e2e/rust/e2e-podman.sh exits after conformance without running Rust E2E tests.

This means the following relevant coverage is still not running in required branch CI:

  • the new podman_host_gateway test;
  • the pre-existing provider_token_exchange test, which also curls host.openshell.internal and was already present when the regression landed;
  • more broadly, every Rust test gated by the e2e-podman feature, because no checked-in branch workflow currently enables that feature or invokes mise run e2e:podman.

Please add a required workflow matrix entry that runs at least podman_host_gateway. Given that this core path already regressed while an applicable test existed but was unscheduled, I strongly recommend restoring the complete e2e:podman suite to required branch CI rather than relying only on a focused test. A manual run reported in the PR is useful, but it does not close the regression gap.

[P2] Add the new resolver secret to emergency cleanup.

The driver now creates openshell-resolver-*, but e2e/with-podman-gateway.sh only removes the token, proxy-auth, and TLS secret prefixes. If the test or gateway terminates before normal sandbox deletion, the resolver secret can remain on the runner. Please add openshell-resolver to that cleanup loop.

@johntmyers johntmyers self-assigned this Sep 23, 2026
@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

PR Review Status

This concentrated Podman regression fix is project-valid through issue #3605. The independent code review found no additional blocking defects; thanks @johntmyers, I checked the required-CI coverage and emergency-cleanup gaps you identified, and both remain unresolved on this head.

Action required: @grs, please update the branch so required PR CI runs at least podman_host_gateway (preferably the complete e2e:podman suite) and add openshell-resolver to the emergency cleanup loop in e2e/with-podman-gateway.sh.

Blocking findings:

  • No additional Gator findings.

Carried findings:

  • None.

Non-blocking suggestions:

  • None.
Gator metadata
  • Validation: Linked issue Podman sandboxes cannot reach host.openshell.internal #3605 documents a reproducible, user-visible Podman networking regression with focused acceptance criteria.
  • Docs: Architecture, Podman networking, Fern gateway configuration, and troubleshooting documentation are updated.
  • Checks: Current-head required checks are not all green, and the requested Podman regression coverage is not dispatched by required CI.
  • E2E: test:e2e is required but deferred until the unresolved review feedback is addressed.
  • Head SHA: ef6b654c16a85e450cfc69865ec953ffb9121ee6
  • Base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Merge base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Patch ID: 449f83722670433384536d0f48f6cbfda21a6252
  • Gator payload: 9
  • Review mode: initial
  • Previous reviewed SHA: none
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

grs added 2 commits September 23, 2026 23:43
Signed-off-by: Gordon Sim <gsim@redhat.com>
Signed-off-by: Gordon Sim <gsim@redhat.com>
@johntmyers johntmyers added the test:e2e Requires end-to-end coverage label Sep 24, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/3606 does not exist yet. A maintainer needs to comment /ok to test ebf8b9dfc0c9e801689fae82f9b8b36e74e24fdf to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test ebf8b9d

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Re-check After Author Update

The latest author update resolves the prior review requests and the independent follow-up review found no new blocking defects. Thanks @johntmyers, I checked the required Podman CI coverage and emergency-cleanup gaps you identified: required branch CI now dispatches the Podman Rust E2E set including podman_host_gateway and provider_token_exchange, and cleanup now removes openshell-resolver-* secrets.

Disposition: resolved.

Remaining items:

  • No blocking review items remain.
Gator metadata
  • Validation: Linked issue Podman sandboxes cannot reach host.openshell.internal #3605 documents a reproducible, user-visible Podman networking regression with focused acceptance criteria.
  • Docs: Architecture, Podman networking, Fern gateway configuration, and troubleshooting documentation are updated.
  • Checks: Required current-head workflows are queued on the mirrored branch.
  • E2E: test:e2e is applied; /ok to test ebf8b9dfc0c9e801689fae82f9b8b36e74e24fdf created the current-head mirror and Branch E2E Checks is queued.
  • Head SHA: ebf8b9dfc0c9e801689fae82f9b8b36e74e24fdf
  • Base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Merge base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Patch ID: 1f0e3a0e83258e754a8759a5c31b5588f32d6928
  • Gator payload: 9
  • Review mode: follow_up
  • Previous reviewed SHA: ef6b654c16a85e450cfc69865ec953ffb9121ee6
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:watch-pipeline

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 24, 2026
Signed-off-by: Gordon Sim <gsim@redhat.com>
@johntmyers

Copy link
Copy Markdown
Collaborator

/ok to test 2194310

@johntmyers

Copy link
Copy Markdown
Collaborator

gator-agent

Re-check After Author Update

Thanks @grs. I reviewed the latest commit, which only makes the macOS gateway test's IP-address parse type explicit, and the independent follow-up review found no blocking defects.

Disposition: resolved.

Remaining items:

  • No blocking review items remain.
Gator metadata
  • Validation: Linked issue Podman sandboxes cannot reach host.openshell.internal #3605 documents a reproducible, user-visible Podman networking regression with focused acceptance criteria.
  • Docs: Architecture, Podman networking, Fern gateway configuration, and troubleshooting documentation are updated.
  • Checks: Current-head required checks are waiting for the copy-pr mirror.
  • E2E: test:e2e is applied; /ok to test 2194310f38cdb0ce68fc587aecf4b1f61cdbeea6 was posted and workflow dispatch must be confirmed.
  • Head SHA: 2194310f38cdb0ce68fc587aecf4b1f61cdbeea6
  • Base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Merge base SHA: d3480d2a7efab3fd0217ab67828655617b9af777
  • Patch ID: 70d4b617ebfc1297ba20b6f31a01e0709b1ccf03
  • Gator payload: 9
  • Review mode: follow_up
  • Previous reviewed SHA: ebf8b9dfc0c9e801689fae82f9b8b36e74e24fdf
  • Review budget exhausted: no
  • Maintainer decision required: no
  • Next state: gator:in-review

@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:blocked Gator is blocked by process or repository gates and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 24, 2026
@johntmyers johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:blocked Gator is blocked by process or repository gates and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 24, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gator:blocked Gator is blocked by process or repository gates test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Podman sandboxes cannot reach host.openshell.internal

2 participants