test(e2e): harden rootful Podman shards against daemon stalls

[skevetter]

Summary

Two consecutive main-branch runs failed in the rootful Podman E2E shards after green PR runs (https://github.com/devsy-org/devsy/actions/runs/35485053108, https://github.com/devsy-org/devsy/actions/runs/35485741862): one wedged daemon stalled cleanup and turned into five cascading BeforeEach health-check failures, and a lifecycle spec hung until Go's 10-minute suite timeout. This PR hardens the harness against that failure shape without masking product regressions.

• e2e/tests/up/podman_rootful.go (new): shared rootful setup with a classified health check (timeout = wedged, socket errors = unavailable, anything else = responsive-but-erroring), bounded per-command daemon diagnostics (podman version/info/ps, systemctl status, bounded journalctl, process list, disk, memory), a single bounded systemctl restart recovery for the wedged/unavailable classes only, and a shard gate that skips remaining specs after the first unrecoverable daemon failure instead of cascading.
• The five rootful provider files now share setupRootfulPodman instead of duplicating wrapper setup.
• e2e/tests/up/helper.go: failed workspace cleanup on rootful shards now collects diagnostics, classifies daemon state, and performs one restart plus one cleanup retry only when the daemon is wedged or missing. A responsive-but-erroring daemon fails without retry, so product bugs are not hidden.
• e2e/framework/exec.go: WaitDelay on framework command execution so a grandchild that survives the process-group kill while holding stdout/stderr pipes cannot block Wait past the spec timeout.
• hack/ci/setup-podman-linux.sh: systemd drop-in keeps the rootful API service resident (system service --time=0) for the whole shard instead of the stock 5s idle exit, removing socket-reactivation churn between test commands.
• Unit tests cover health classification, the recovery decision, and the daemon gate.

What this deliberately does not do

• No blanket timeout increases, no unconditional retries, no flake-attempt bumps, no quarantine.
• Recovery is single-attempt, bounded, and gated on the classified failure class; it only touches infrastructure state, never test assertions.
• The first failure still fails loudly with full diagnostics; only the cascade is skipped.

Verification

• gofmt clean; go build ./e2e/... passes; go test ./e2e/tests/up/ ./e2e/framework/ passes locally.
• golangci-lint and the rootful shards themselves run in CI on this PR.

[netlify]

✅ Deploy Preview for images-devsy-sh canceled.

Name	Link
🔨 Latest commit	3f572b8
🔍 Latest deploy log	https://app.netlify.com/projects/images-devsy-sh/deploys/6aaf7b6f0be3c80008b8ea29

[coderabbitai]

Warning

Review limit reached

Next included review available in 52 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: e672b52f-2b3f-441b-b256-b936a321a7a3

📥 Commits

Reviewing files that changed from the base of the PR and between c6473e4 and 3f572b8.

📒 Files selected for processing (10)

• e2e/framework/exec.go
• e2e/tests/up/helper.go
• e2e/tests/up/podman_rootful.go
• e2e/tests/up/podman_rootful_test.go
• e2e/tests/up/provider_podman_rootful_basic.go
• e2e/tests/up/provider_podman_rootful_config.go
• e2e/tests/up/provider_podman_rootful_features.go
• e2e/tests/up/provider_podman_rootful_lifecycle.go
• e2e/tests/up/provider_podman_rootful_lifecycle_2.go
• hack/ci/setup-podman-linux.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[netlify]

✅ Deploy Preview for devsydev canceled.

Name	Link
🔨 Latest commit	3f572b8
🔍 Latest deploy log	https://app.netlify.com/projects/devsydev/deploys/6aaf7b6facb27f00082a1aa9

[skevetter]

@greptileai review

[greptile-apps]

Confidence Score: 4/5

The PR should not merge until an unrecoverable cleanup-time daemon failure marks the shard gate, otherwise the following spec still produces a redundant cascading infrastructure failure.

Findings

1.  Cleanup Failure Bypasses Gate ▶

Summary

This PR hardens rootful Podman E2E shards by centralizing daemon health checks and recovery, adding bounded diagnostics and command waiting, retrying cleanup after infrastructure failures, and keeping the rootful API service resident.

• Adds classified Podman health checks, bounded diagnostics, restart recovery, and a shard-level failure gate.
• Reuses the shared rootful setup across all five provider suites.
• Adds exec.Cmd.WaitDelay to prevent inherited output pipes from indefinitely blocking command completion.
• Installs a systemd override that disables the Podman API service idle timeout.
• Adds focused unit coverage for health classification, recovery eligibility, and gate state.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Rootful spec setup] --> B{Daemon healthy?}
    B -->|Yes| C[Run spec]
    B -->|No| D[Collect diagnostics]
    D --> E{Wedged or unavailable?}
    E -->|No| F[Fail without restart]
    E -->|Yes| G[Restart and re-probe]
    G -->|Fails| H[Mark gate unhealthy and fail]
    G -->|Succeeds| C
    C --> I[Workspace cleanup]
    I -->|Succeeds| J[Complete]
    I -->|Fails| K[Diagnose and classify]
    K -->|Recoverable| L[Restart and retry cleanup]
    L -->|Succeeds| J
    L -->|Recovery fails| M[Return cleanup failure without marking gate]
    H --> N[Later specs skip]
    M --> O[Next spec runs and fails again]

Loading

_{Reviews (1) · Last reviewed commit: "fix(e2e): cover flagged gosec call line ..."}

[greptile-apps]

Cleanup Failure Bypasses Gate

If the daemon wedges after setup and cleanup recovery fails, this branch returns the cleanup error without marking rootfulDaemonGate unhealthy. The following spec therefore proceeds into setupRootfulPodman and fails against the same unhealthy daemon before the gate is finally marked, producing the cascading infrastructure failure this change is intended to prevent. Mark the gate when cleanup-time daemon recovery fails, as the setup-time recovery path already does.

Suggested change

	if recErr := attemptPodmanRecovery(recoveryCtx, wrapperPath); recErr != nil {
	ginkgo.GinkgoWriter.Printf("[podman-recovery] cleanup recovery failed: %v\n", recErr)
	return cleanupErr
	}
	if recErr := attemptPodmanRecovery(recoveryCtx, wrapperPath); recErr != nil {
	rootfulDaemonGate.markUnhealthy(ginkgo.CurrentSpecReport().FullText())
	ginkgo.GinkgoWriter.Printf("[podman-recovery] cleanup recovery failed: %v\n", recErr)
	return cleanupErr
	}