Skip to content

Turning on networkPolicy fences the API and the computers and leaves the culler open #276

Description

@zopeVaibhav

Turning networkPolicy.enabled on fences the API and the computers and leaves the culler open. A NetworkPolicy applies only to the pods its selector matches, and a pod nothing selects keeps the cluster default rather than being denied — so the one workload no policy names is the one workload that is not restricted. Here that is the pod holding a cluster token and the whole server environment.

What renders

With networkPolicy.enabled: true and computers.mode: sandbox, exactly two policies come out, and these are their selectors:

ci-openbot-server   -> app.kubernetes.io/component: server
ci-openbot-computer -> app.kubernetes.io/component: computer

The culler CronJob's pod template carries:

app.kubernetes.io/component: culler

Neither selector matches it. networkpolicy.yaml holds two kind: NetworkPolicy blocks and names only "server" and "computer"; the string culler does not appear in the file.

What that pod is carrying

Read from the rendered pod spec rather than assumed:

  • automountServiceAccountToken: true, on the service account the sandbox Role binds to sandboxes with get, list, watch, create, patch, delete.
  • Twenty-two environment variables, because it runs the same image with the same configuration as the API: KEY_ENCRYPTION_KEY, BETTER_AUTH_SECRET, DATABASE_URL, OPENAI_API_KEY, INTELLIGENCE_API_KEY, COMPUTER_TOKEN and GOOGLE_OAUTH_CLIENT_SECRET among them.
  • A schedule of */5 * * * *, so it is a standing surface rather than a rare one.

The chart already noticed half of this. culler-cronjob.yaml carries the postgresql-client label with a comment saying anything that opens the database needs it and that only the API had it — so the pod's database access was thought about. Its network was not.

Reproduction

  1. Render the chart with policy on, as the chart's own CI does:

    helm template ci charts/openbot \
      --values charts/openbot/ci/eks-sandbox-values.yaml \
      --set-string secrets.keyEncryptionKey="$(openssl rand -base64 32)" \
      --api-versions agents.x-k8s.io/v1beta1/Sandbox \
      --api-versions extensions.agents.x-k8s.io/v1beta1/SandboxTemplate
    

    That target sets networkPolicy.enabled: true.

  2. Read the podSelector of every rendered NetworkPolicy, and the labels on the CronJob's pod template. No selector matches component: culler.

  3. grep -n culler charts/openbot/templates/networkpolicy.yaml returns nothing.

Why it matters

The failure is silent and it points the wrong way. An operator who sets networkPolicy.enabled gets a deployment that looks locked down: two policies exist, kubectl get networkpolicy lists them, and the API and the computers genuinely are fenced. Nothing anywhere says a third workload opted out, because opting out is the absence of a resource rather than the presence of one.

The people this reaches are exactly the ones who asked for egress control. It is off by default, so nobody else is affected.

Worth stating plainly: the load-bearing claim here is the Kubernetes rule that an unselected pod is unrestricted rather than denied. I have verified the chart's rendered output, not watched a cluster enforce it. The chart's own computer-policy comment relies on the same rule, which is why the computers got one.

What a fix probably has to do

A third policy selecting component: culler, with egress to DNS, the database and the Kubernetes API server and nothing else. It reads the database and asks the API server which computers to suspend; it never calls Intelligence, an identity provider or a Bot, so it should not have the exception-list rule to the internet that the API needs.

One trap worth naming: the bundled-database egress rule is behind postgresql.enabled, and the two shipped targets that turn policy on use an external database. A new policy that only handles the bundled case would leave the culler unable to reach its own database, which turns a silent gap into a broken sweep.

Whether the better answer is instead a release-wide default-deny with per-component allows is a bigger call and yours to make.

Severity

Moderate, and narrowed by the default. Nothing is broken today, and networkPolicy.enabled is off unless somebody turns it on. What it costs is the guarantee that switch is bought for, on the pod that would matter most.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions