Cached tasks cannot spawn child processes inside a rootless bubblewrap sandbox (EPERM)

[lifeiscontent]

Summary

Inside a rootless bubblewrap sandbox, a cache-enabled task cannot spawn child processes. Every spawn/spawnSync/execFile fails with EPERM before the child runs. Setting cache: false on the same task, with nothing else changed, makes it work.

The path is absolute and executable in both cases, so this is not PATH resolution.

Reproduction

Sandbox is entered as:

bwrap --die-with-parent --new-session \
  --unshare-user --unshare-pid --unshare-ipc --unshare-uts --unshare-cgroup --unshare-net \
  --cap-drop ALL \
  --clearenv --setenv PATH /runtime/bin \
  --ro-bind <toolchain-image> /runtime \
  --tmpfs /tmp --dev /dev --proc /proc \
  --bind <checkout> /workspace \
  -- /runtime/bin/sh -c 'cd /workspace && vp run -r test'

There is no /usr/bin, /bin or /usr/lib in the sandbox; everything is under /runtime.

Any cached task whose command spawns a child reproduces it:

execFileSync("/tmp/shell/sh", ["-n"], { input: script });

task config	spawn
{ command: 'vp test' }	EPERM
{ command: 'vp test', cache: false }	works

In one vp run -r test over a workspace, the packages I had flipped to cache: false spawned fine while the packages still cached failed in the same run, same sandbox, same commit. Flipping two packages took the spawn failures from 74 to 0.

Observed

Error: spawnSync /tmp/shell/sh EPERM
Error: spawn EPERM

Nothing is printed by the child. git subprocesses fail the same way one level down:

fatal: cannot exec 'git-receive-pack': Operation not permitted

Environment

• vite-plus 0.3.0
• Linux x86_64, glibc
• rootless bubblewrap, seccomp, unprivileged user namespaces

Notes

Not #569/#576 — 0.3.0 has both. LD_PRELOAD is unset going in, so not #340.

Guess: fspy_preload_unix has to inject into each child, and this sandbox is --cap-drop ALL with /usr/lib unmounted, so either the preload object is unreachable in the child's mount namespace or something it does on init is denied — and the exec is refused instead of tracking degrading.

Falling back to caching without file tracking would be better than failing the spawn. Today the only workaround is disabling the cache for the whole package.

Happy to run patches or a debug build against the sandbox.

[lifeiscontent]

Two clarifications after hitting this in more places.

It is not test-harness specific. The original repro used execFileSync from a test. The same EPERM comes from an ordinary application module doing:

const child = spawn(command, args, { cwd, shell: false });
// Error: spawn EPERM

shell: false is explicit there, so the shell path is ruled out as well as PATH resolution.

Scale. Four packages in one workspace now have to opt out of caching to run their suites at all. Two of them are large enough that losing the cache outright was not acceptable, so they instead split into two vitest projects — a cached unit project and an uncached subprocess project holding only the suites that spawn, wired with dependsOn:

"test:subprocess": { command: "vp test --project subprocess", cache: false },
test: { command: "vp test --project unit", dependsOn: ["test:subprocess"], env: [...] },

That keeps 376 of 410 suites cached in the largest one, and measured 1/2 cache hit, 9.37s saved on a second run in another. Workable, but it means the cache boundary has to be drawn around whether a suite spawns, which is not a property anyone would otherwise organise tests by.

Still on 0.3.0, still the same sandbox (--unshare-user --cap-drop ALL --clearenv, no /usr/lib).

[lifeiscontent]

Replacing the guess at the end of the report with a measurement.

vite.config.ts — two tasks, byte-identical commands, differing only in cache:

import { defineConfig } from "vite-plus";

export default defineConfig({
  run: {
    tasks: {
      probe: {
        command:
          "node -e \"console.log('LD_PRELOAD=' + (process.env.LD_PRELOAD ?? '<unset>')); console.log('DYLD_INSERT_LIBRARIES=' + (process.env.DYLD_INSERT_LIBRARIES ?? '<unset>'))\"",
      },
      probeUncached: {
        command:
          "node -e \"console.log('LD_PRELOAD=' + (process.env.LD_PRELOAD ?? '<unset>')); console.log('DYLD_INSERT_LIBRARIES=' + (process.env.DYLD_INSERT_LIBRARIES ?? '<unset>'))\"",
        cache: false,
      },
    },
  },
});

$ vp run probe
LD_PRELOAD=<unset>
DYLD_INSERT_LIBRARIES=/var/folders/…/T/fspy/fspy_preload_5ecf9578902460f99bef8efa68abb9e6.dylib

$ vp run probeUncached
LD_PRELOAD=<unset>
DYLD_INSERT_LIBRARIES=<unset>

(Run on macOS because it was the convenient shell; the Linux sandbox where the EPERM occurs uses the LD_PRELOAD equivalent.)

So the preload is injected into every child of a cache-enabled task and not otherwise, which is the one variable the earlier A/B isolated. That is the mechanism — no longer an inference from the symptom.

Whether the sandbox refuses the execve because of the preload path itself, the seccomp filter, or --cap-drop ALL interacting with it, I cannot say from outside. But the injection is the difference between working and EPERM, and it is injected by the task runner.