Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -479,8 +479,13 @@ cross the authenticated vsock channel. A gateway-host proxy is addressed as

The Docker driver runs `openshell-supervisor` in a separate companion container.
Its private named volume contains supervisor bootstrap and channel material.
The workload container receives only `openshell-sandbox`, public interception
CA material, and the other sandbox half of the authenticated channel.
The driver bounded-reads operator proxy credentials and CA bundles into that
supervisor-only volume and passes fixed container paths to the supervisor; it
never exposes gateway-host paths to sandbox-controlled configuration. The
corporate CA extends supervisor upstream trust and is folded into the public
combined trust bundle generated for workload processes. The workload container
receives only `openshell-sandbox`, public interception and combined CA
material, and the other sandbox half of the authenticated channel.

For Kubernetes, the operator configures a Secret name and key rather than a
gateway-host file path. Kubernetes projects that Secret only into the separate
Expand Down
26 changes: 26 additions & 0 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,32 @@ through the Docker archive API. No workload launch depends on a host bind
mount or a tool supplied by the workload image, so the same path works with
local, remote, and VM-backed Docker daemons.

## Corporate Proxy Egress

`[openshell.drivers.docker]` accepts the operator-owned corporate proxy fields
`https_proxy`, `no_proxy`, `proxy_auth_file`,
`proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and
`proxy_ca_bundle`. Sandbox environment, image contents, and per-sandbox driver
configuration cannot select or override them.

The driver validates the proxy URL and cross-field relationships at gateway
startup. It bounded-reads proxy credentials and the PEM CA bundle before use.
Missing, unreadable, empty, oversized, malformed, or certificate-free CA files
fail closed. A CA bundle requires `https_proxy`, although the proxy URL may be
`http://` when the proxy intercepts destination TLS.

For each supervisor launch, Docker copies the credential and CA contents into
the existing supervisor-only named volume. It passes only the fixed container
paths `/.openshell/supervisor/upstream-proxy-auth` and
`/.openshell/supervisor/upstream-proxy-ca-bundle.pem` to the supervisor. It
does not bind-mount the gateway-host files, which preserves remote-daemon
support and keeps host paths out of workload container metadata.

The supervisor trusts the configured CA for its TLS connection to an HTTPS
proxy and for destination certificates re-signed by a TLS-intercepting proxy.
It also includes the corporate root in the generated combined trust bundle
used by workload processes.

## Identity and Workspace

Before creating the workload, the driver pins the image ID and reads its
Expand Down
70 changes: 68 additions & 2 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ const BOUNDARY_CERTIFICATE_MOUNT_PATH: &str = "/.openshell/channel/sandbox/serve
const BOUNDARY_PRIVATE_KEY_MOUNT_PATH: &str = "/.openshell/channel/sandbox/server.key";
const SUPERVISOR_STATE_MOUNT_PATH: &str = "/.openshell/supervisor";
const SUPERVISOR_PROXY_AUTH_MOUNT_PATH: &str = "/.openshell/supervisor/upstream-proxy-auth";
const SUPERVISOR_PROXY_CA_BUNDLE_MOUNT_PATH: &str =
"/.openshell/supervisor/upstream-proxy-ca-bundle.pem";
const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR: &str =
openshell_core::driver_utils::PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR;
const DRIVER_ADMITTED_BACKEND: &str = openshell_sandbox_backend::BACKEND_NAME;
Expand Down Expand Up @@ -214,6 +216,10 @@ pub struct DockerComputeConfig {
#[serde(flatten)]
pub upstream_proxy: UpstreamProxyConfig,

/// Gateway-host PEM CA bundle trusted for the corporate proxy and for
/// server certificates re-signed by a TLS-intercepting proxy.
pub proxy_ca_bundle: Option<PathBuf>,

/// Host UNIX socket projected into the supervisor for provider identity.
pub provider_spiffe_workload_api_socket: Option<PathBuf>,

Expand All @@ -237,6 +243,7 @@ impl DockerComputeConfig {
validate_image_pull_policy(self.image_pull_policy)?;
self.upstream_proxy.validate().map_err(Error::config)?;
validate_docker_proxy_auth_file(&self.upstream_proxy)?;
validate_docker_proxy_ca_bundle(self)?;
if let Some(socket) = self.provider_spiffe_workload_api_socket.as_deref() {
openshell_core::driver_utils::validate_provider_spiffe_unix_socket(socket)
.map_err(Error::config)?;
Expand Down Expand Up @@ -268,6 +275,7 @@ impl Default for DockerComputeConfig {
sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(),
enable_bind_mounts: false,
upstream_proxy: UpstreamProxyConfig::default(),
proxy_ca_bundle: None,
provider_spiffe_workload_api_socket: None,
app_armor_profile: None,
}
Expand Down Expand Up @@ -297,6 +305,7 @@ struct DockerDriverRuntimeConfig {
sandbox_pids_limit: Option<std::num::NonZeroI64>,
enable_bind_mounts: bool,
upstream_proxy: UpstreamProxyConfig,
proxy_ca_bundle: Option<PathBuf>,
provider_spiffe_workload_api_socket: Option<PathBuf>,
app_armor_profile: Option<AppArmorProfile>,
}
Expand Down Expand Up @@ -922,6 +931,7 @@ impl DockerComputeDriver {
sandbox_pids_limit: docker_config.sandbox_pids_limit,
enable_bind_mounts: docker_config.enable_bind_mounts,
upstream_proxy: docker_config.upstream_proxy.clone(),
proxy_ca_bundle: docker_config.proxy_ca_bundle.clone(),
provider_spiffe_workload_api_socket: docker_config
.provider_spiffe_workload_api_socket
.clone(),
Expand Down Expand Up @@ -4504,11 +4514,35 @@ async fn docker_supervisor_bundle_archive(
&contents,
)?;
}
append_docker_proxy_ca_bundle(&mut archive, config.proxy_ca_bundle.as_deref())?;
archive
.into_inner()
.map_err(|error| Status::internal(format!("finish Docker supervisor archive: {error}")))
}

fn append_docker_proxy_ca_bundle(
archive: &mut tar::Builder<Vec<u8>>,
path: Option<&Path>,
) -> Result<(), Status> {
let Some(path) = path else {
return Ok(());
};
let path = path
.to_str()
.ok_or_else(|| Status::failed_precondition("proxy_ca_bundle must be valid UTF-8"))?;
let contents =
openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file(path, "proxy_ca_bundle")
.map_err(Status::failed_precondition)?;
append_docker_archive_file(
archive,
"upstream-proxy-ca-bundle.pem",
0o644,
SUPERVISOR_UID,
SUPERVISOR_GID,
contents.as_bytes(),
)
}

async fn refresh_docker_boundary_authentication(
sandbox_id: &str,
config: &DockerDriverRuntimeConfig,
Expand Down Expand Up @@ -4911,7 +4945,10 @@ async fn spawn_docker_control_process(
workspace_root,
format!("--health-socket-path={SUPERVISOR_HEALTH_SOCKET_PATH}"),
];
command.extend(docker_upstream_proxy_cli_args(&config.upstream_proxy));
command.extend(docker_upstream_proxy_cli_args(
&config.upstream_proxy,
config.proxy_ca_bundle.is_some(),
));
let mut supervisor_mounts = vec![
Mount {
target: Some(BOUNDARY_MOUNT_PATH.to_string()),
Expand Down Expand Up @@ -5645,7 +5682,30 @@ fn validate_docker_proxy_auth_file(config: &UpstreamProxyConfig) -> CoreResult<(
Ok(())
}

fn docker_upstream_proxy_cli_args(config: &UpstreamProxyConfig) -> Vec<String> {
fn validate_docker_proxy_ca_bundle(config: &DockerComputeConfig) -> CoreResult<()> {
let Some(path) = config.proxy_ca_bundle.as_ref() else {
return Ok(());
};
if path.as_os_str().is_empty() {
return Err(Error::config("proxy_ca_bundle must not be empty when set"));
}
if config.upstream_proxy.https_proxy.is_none() {
return Err(Error::config(
"proxy_ca_bundle is set but no https_proxy is configured",
));
}
let path = path
.to_str()
.ok_or_else(|| Error::config("proxy_ca_bundle must be valid UTF-8"))?;
openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file(path, "proxy_ca_bundle")
.map_err(Error::config)?;
Ok(())
}

fn docker_upstream_proxy_cli_args(
config: &UpstreamProxyConfig,
proxy_ca_bundle_configured: bool,
) -> Vec<String> {
let mut args = Vec::new();
if let Some(url) = config.https_proxy.as_ref() {
args.extend(["--upstream-proxy".to_string(), url.clone()]);
Expand All @@ -5665,6 +5725,12 @@ fn docker_upstream_proxy_cli_args(config: &UpstreamProxyConfig) -> Vec<String> {
if config.proxy_connect_by_hostname == Some(true) {
args.push("--upstream-proxy-connect-by-hostname".to_string());
}
if proxy_ca_bundle_configured {
args.extend([
"--upstream-proxy-ca-bundle".to_string(),
SUPERVISOR_PROXY_CA_BUNDLE_MOUNT_PATH.to_string(),
]);
}
args
}

Expand Down
144 changes: 144 additions & 0 deletions crates/openshell-driver-docker/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ use openshell_core::proto::compute::v1::{
ResourceRequirements, WorkloadIdentityRequest,
};
use std::fs;
use std::io::Read as _;
use std::sync::Arc;
use tempfile::TempDir;

Expand Down Expand Up @@ -139,11 +140,154 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
sandbox_pids_limit: openshell_core::config::default_sandbox_pids_limit(),
enable_bind_mounts: false,
upstream_proxy: UpstreamProxyConfig::default(),
proxy_ca_bundle: None,
provider_spiffe_workload_api_socket: None,
app_armor_profile: Some(AppArmorProfile::Unconfined),
}
}

fn write_test_proxy_ca_bundle(directory: &TempDir) -> PathBuf {
let tls = generate_sandbox_tls_material(openshell_core::SandboxSessionId::new())
.expect("generate test proxy CA");
let path = directory.path().join("proxy-ca.pem");
fs::write(&path, tls.trust_anchor_pem).expect("write test proxy CA");
path
}

#[test]
fn docker_config_parses_operator_proxy_ca_bundle() {
let config: DockerComputeConfig = toml::from_str(
r#"
https_proxy = "https://proxy.corp.example:8443"
proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem"
"#,
)
.expect("parse Docker proxy CA configuration");

assert_eq!(
config.proxy_ca_bundle,
Some(PathBuf::from("/etc/openshell/tls/proxy-ca.pem"))
);
}

#[test]
fn docker_proxy_ca_bundle_validation_is_fail_closed() {
let directory = TempDir::new().expect("create CA directory");
let ca_bundle = write_test_proxy_ca_bundle(&directory);
let gateway_bind_address = "127.0.0.1:17670".parse().unwrap();

let mut valid = DockerComputeConfig::default();
valid.upstream_proxy.https_proxy = Some("http://proxy.corp.example:8080".to_string());
valid.proxy_ca_bundle = Some(ca_bundle);
valid
.validate_configuration(gateway_bind_address)
.expect("a valid CA bundle is accepted with an HTTP interception proxy");

let mut without_proxy = valid.clone();
without_proxy.upstream_proxy.https_proxy = None;
let error = without_proxy
.validate_configuration(gateway_bind_address)
.expect_err("a CA bundle without a proxy must fail");
assert!(error.to_string().contains("proxy_ca_bundle"), "{error}");
assert!(error.to_string().contains("https_proxy"), "{error}");

let mut empty_path = valid.clone();
empty_path.proxy_ca_bundle = Some(PathBuf::new());
let error = empty_path
.validate_configuration(gateway_bind_address)
.expect_err("an empty CA bundle path must fail");
assert!(error.to_string().contains("must not be empty"), "{error}");

let mut missing = valid.clone();
missing.proxy_ca_bundle = Some(directory.path().join("missing.pem"));
let error = missing
.validate_configuration(gateway_bind_address)
.expect_err("a missing CA bundle must fail");
assert!(error.to_string().contains("could not be read"), "{error}");

let malformed_path = directory.path().join("malformed.pem");
fs::write(&malformed_path, "not a certificate\n").unwrap();
let mut malformed = valid;
malformed.proxy_ca_bundle = Some(malformed_path);
let error = malformed
.validate_configuration(gateway_bind_address)
.expect_err("a certificate-free CA bundle must fail");
assert!(error.to_string().contains("no PEM certificate"), "{error}");
}

#[test]
fn docker_proxy_ca_bundle_uses_fixed_supervisor_path() {
let proxy = UpstreamProxyConfig {
https_proxy: Some("https://proxy.corp.example:8443".to_string()),
..UpstreamProxyConfig::default()
};

let args = docker_upstream_proxy_cli_args(&proxy, true);
let option = args
.iter()
.position(|arg| arg == "--upstream-proxy-ca-bundle")
.expect("proxy CA option");
assert_eq!(
args.get(option + 1).map(String::as_str),
Some(SUPERVISOR_PROXY_CA_BUNDLE_MOUNT_PATH)
);
assert!(
!args.iter().any(|arg| arg.contains("/etc/openshell/tls")),
"gateway-host paths must not appear in supervisor argv: {args:?}"
);

let args = docker_upstream_proxy_cli_args(&proxy, false);
assert!(!args.iter().any(|arg| arg == "--upstream-proxy-ca-bundle"));
}

#[test]
fn docker_proxy_ca_bundle_is_staged_in_supervisor_archive() {
let directory = TempDir::new().expect("create CA directory");
let ca_bundle = write_test_proxy_ca_bundle(&directory);
let expected = fs::read_to_string(&ca_bundle).unwrap();
let mut builder = tar::Builder::new(Vec::new());

append_docker_proxy_ca_bundle(&mut builder, Some(&ca_bundle)).expect("append proxy CA bundle");
let archive = builder.into_inner().expect("finish proxy CA archive");
let mut archive = tar::Archive::new(archive.as_slice());
let mut entries = archive.entries().unwrap();
let mut entry = entries.next().expect("proxy CA entry").unwrap();

assert_eq!(
entry.path().unwrap().as_ref(),
Path::new("upstream-proxy-ca-bundle.pem")
);
assert_eq!(entry.header().uid().unwrap(), u64::from(SUPERVISOR_UID));
assert_eq!(entry.header().gid().unwrap(), u64::from(SUPERVISOR_GID));
assert_eq!(entry.header().mode().unwrap(), 0o644);
let mut actual = String::new();
entry.read_to_string(&mut actual).unwrap();
assert_eq!(actual, expected);
assert!(entries.next().is_none());
}

#[test]
fn sandbox_driver_config_cannot_override_proxy_ca_bundle() {
let config = runtime_config();
let mut sandbox = test_sandbox();
sandbox
.spec
.as_mut()
.unwrap()
.template
.as_mut()
.unwrap()
.driver_config = Some(json_struct(serde_json::json!({
"proxy_ca_bundle": "/workload/controlled-ca.pem"
})));

let error = DockerComputeDriver::validate_sandbox(&sandbox, &config)
.expect_err("sandbox driver config must not accept proxy_ca_bundle");
assert_eq!(error.code(), tonic::Code::InvalidArgument);
assert!(error.message().contains("unknown field"), "{error}");
assert!(error.message().contains("proxy_ca_bundle"), "{error}");
}

fn test_workload_identity() -> ResolvedWorkloadIdentity {
ResolvedWorkloadIdentity::new(
1234,
Expand Down
Loading
Loading