diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 6a9bcd25b9..c2af7f097b 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -74,7 +74,7 @@ Before debugging the compute platform, inspect gateway logs for failures in depe For out-of-tree compute drivers, confirm the custom driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts: ```bash -rg -n 'compute_drivers|socket_path' /etc/openshell/gateway.toml +rg -n 'compute_driver|compute_drivers|socket_path' /etc/openshell/gateway.toml stat /run/openshell/.sock journalctl -u --no-pager --lines=200 journalctl -u openshell-gateway --no-pager --lines=200 diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 831be067ab..6936c314db 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -143,7 +143,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`. | Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). | | Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. | | VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. | -| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = [""]` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | +| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_driver = ""` entry with `[openshell.drivers.].socket_path`, or at launch time by pairing `--drivers ` with `--compute-driver-socket=`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. | Per-sandbox CPU and memory values currently enter the driver layer through template resource limits. Docker and Podman apply them as runtime limits. diff --git a/architecture/gateway.md b/architecture/gateway.md index db8f2508e8..b6e112eb71 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -635,22 +635,27 @@ Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in defa ``` The TOML file is opt-in via `--config ` / `OPENSHELL_GATEWAY_CONFIG`. -Driver implementation settings live in the TOML driver tables. See -`docs/reference/gateway-config.mdx` for worked per-driver examples and RFC -0003 for the full schema. +Driver implementation settings live in the TOML driver tables. The canonical +selector is the singular `[openshell.gateway] compute_driver`; the legacy +`compute_drivers` list remains accepted and normalizes into the existing +exactly-one-driver runtime validation. See `docs/reference/gateway-config.mdx` +for worked per-driver examples and RFC 0003 for the full schema. `database_url` is env-only and rejected when present in the file (`OPENSHELL_DB_URL` / `--db-url`). ### Driver inheritance -`[openshell.gateway]` carries a small set of values (`sandbox_namespace`, -`default_image`, -`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`, -`host_gateway_ip`, `enable_user_namespaces`) that are inherited into each -driver's `[openshell.drivers.]` table when the driver-specific table -does not override them. The allowlist is per-driver so a gateway-wide -default cannot land in a driver that does not understand it (e.g. +`[openshell.gateway]` carries shared defaults such as `default_image`, +`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`, and +`host_gateway_ip`. It also continues to accept the historical +`sandbox_namespace`, `service_account_name`, and `enable_user_namespaces` +locations as compatibility inputs. Canonical Kubernetes configuration places +those values in `[openshell.drivers.kubernetes]` as `namespace`, +`service_account_name`, and `enable_user_namespaces`; canonical Docker +configuration uses `sandbox_label`. Driver-table values take precedence over +compatibility inputs. The allowlist is per-driver so a gateway-wide default +cannot land in a driver that does not understand it (for example, `client_tls_secret_name` is K8s-only). `image_pull_policy` is intentionally **not** inheritable: Kubernetes uses diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index fcbdeb73b6..266a68bd53 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -790,11 +790,31 @@ pub struct GatewayJwtConfig { #[serde(default = "default_gateway_id")] pub gateway_id: String, /// Token lifetime in seconds. A value of 0 disables expiration and is - /// intended only for local single-player deployments. - #[serde(default = "default_sandbox_token_ttl_secs")] + /// intended only for local single-player deployments. Canonical serialized + /// configuration omits the field for that non-expiring behavior; explicit + /// legacy zero remains accepted. + #[serde( + default = "default_sandbox_token_ttl_secs", + skip_serializing_if = "is_default" + )] pub ttl_secs: u64, } +impl GatewayJwtConfig { + /// Effective token lifetime. `None` preserves the established non-expiring + /// behavior represented by an omitted or explicit zero `ttl_secs` value. + pub fn sandbox_token_ttl(&self) -> Option { + (self.ttl_secs != 0).then(|| Duration::from_secs(self.ttl_secs)) + } +} + +fn is_default(value: &T) -> bool +where + T: Default + PartialEq, +{ + value == &T::default() +} + fn default_gateway_id() -> String { "openshell".to_string() } @@ -1217,6 +1237,25 @@ mod tests { .expect("gateway JWT config should deserialize with default ttl"); assert_eq!(cfg.ttl_secs, 0); + assert_eq!(cfg.sandbox_token_ttl(), None); + + let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes"); + assert!(serialized.get("ttl_secs").is_none()); + } + + #[test] + fn gateway_jwt_positive_ttl_serializes_and_has_effective_duration() { + let cfg: GatewayJwtConfig = serde_json::from_value(serde_json::json!({ + "signing_key_path": "/tmp/signing.pem", + "public_key_path": "/tmp/public.pem", + "kid_path": "/tmp/kid", + "ttl_secs": 3600 + })) + .expect("gateway JWT config should deserialize with positive ttl"); + + assert_eq!(cfg.sandbox_token_ttl(), Some(Duration::from_secs(3600))); + let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes"); + assert_eq!(serialized["ttl_secs"], 3600); } #[test] diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index b1fb5ec224..75ffd1eb90 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -98,8 +98,9 @@ pub struct DockerComputeConfig { /// Image pull policy for sandbox images. pub image_pull_policy: String, - /// Namespace label applied to Docker sandboxes. - pub sandbox_namespace: String, + /// Value of the `openshell.sandbox_namespace` label applied to Docker sandboxes. + #[serde(alias = "sandbox_namespace")] + pub sandbox_label: String, /// Gateway gRPC endpoint the sandbox connects back to. pub grpc_endpoint: String, @@ -147,7 +148,7 @@ impl Default for DockerComputeConfig { socket_path: None, default_image: openshell_core::image::default_sandbox_image(), image_pull_policy: String::new(), - sandbox_namespace: "default".to_string(), + sandbox_label: "default".to_string(), grpc_endpoint: String::new(), supervisor_bin: None, supervisor_image: None, @@ -174,7 +175,7 @@ pub(crate) struct DockerGuestTlsPaths { struct DockerDriverRuntimeConfig { default_image: String, image_pull_policy: String, - sandbox_namespace: String, + sandbox_label: String, grpc_endpoint: String, network_name: String, gateway_route: DockerGatewayRoute, @@ -456,7 +457,7 @@ impl DockerComputeDriver { config: DockerDriverRuntimeConfig { default_image: docker_config.default_image.clone(), image_pull_policy: docker_config.image_pull_policy.clone(), - sandbox_namespace: docker_config.sandbox_namespace.clone(), + sandbox_label: docker_config.sandbox_label.clone(), grpc_endpoint, network_name, gateway_route, @@ -732,7 +733,7 @@ impl DockerComputeDriver { ); self.publish_sandbox_snapshot(pending_sandbox_snapshot( sandbox, - &self.config.sandbox_namespace, + &self.config.sandbox_label, provisioning_condition(), false, )); @@ -1132,7 +1133,7 @@ impl DockerComputeDriver { PendingSandboxRecord { sandbox: pending_sandbox_snapshot( sandbox, - &self.config.sandbox_namespace, + &self.config.sandbox_label, provisioning_condition(), false, ), @@ -1187,7 +1188,7 @@ impl DockerComputeDriver { cleanup_sandbox_token_file(sandbox, &self.config); let snapshot = pending_sandbox_snapshot( sandbox, - &self.config.sandbox_namespace, + &self.config.sandbox_label, error_condition(failure.reason, &failure.message), false, ); @@ -1393,7 +1394,7 @@ impl DockerComputeDriver { } async fn list_managed_container_summaries(&self) -> Result, Status> { - let filters = managed_container_label_filters(&self.config.sandbox_namespace, []); + let filters = managed_container_label_filters(&self.config.sandbox_label, []); self.docker .list_containers(Some( ListContainersOptionsBuilder::default() @@ -1418,7 +1419,7 @@ impl DockerComputeDriver { } let filters = - managed_container_label_filters(&self.config.sandbox_namespace, label_filter_values); + managed_container_label_filters(&self.config.sandbox_label, label_filter_values); let containers = self .docker .list_containers(Some( @@ -1436,7 +1437,7 @@ impl DockerComputeDriver { }; let namespace_matches = labels .get(LABEL_SANDBOX_NAMESPACE) - .is_some_and(|value| value == &self.config.sandbox_namespace); + .is_some_and(|value| value == &self.config.sandbox_label); let id_matches = sandbox_id.is_empty() || labels .get(LABEL_SANDBOX_ID) @@ -2314,7 +2315,7 @@ fn sandbox_token_host_path_by_id( ) -> Result { openshell_core::driver_utils::sandbox_token_path( "docker-sandbox-tokens", - Some(&config.sandbox_namespace), + Some(&config.sandbox_label), sandbox_id, ) .map_err(|err| { @@ -2664,13 +2665,13 @@ fn build_container_create_body_for_image( LABEL_SANDBOX_WORKSPACE.to_string(), sandbox.workspace.clone(), ); - // The list/get/find paths filter by `config.sandbox_namespace`, so use + // The list/get/find paths filter by `config.sandbox_label`, so use // the same value here. `DriverSandbox.namespace` is unset on the request // path (the gateway elides it), and using it would produce containers // that the driver itself cannot find afterwards. labels.insert( LABEL_SANDBOX_NAMESPACE.to_string(), - config.sandbox_namespace.clone(), + config.sandbox_label.clone(), ); Ok(ContainerCreateBody { @@ -3256,12 +3257,12 @@ fn label_filters(values: impl IntoIterator) -> HashMap, ) -> HashMap> { let mut values = vec![ format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}"), - format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_namespace}"), + format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_label}"), ]; values.extend(extra_values); label_filters(values) diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index eddfd778bd..dc4586cc6e 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -92,7 +92,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig { DockerDriverRuntimeConfig { default_image: "image:latest".to_string(), image_pull_policy: String::new(), - sandbox_namespace: "default".to_string(), + sandbox_label: "default".to_string(), grpc_endpoint: "https://localhost:8443".to_string(), network_name: DEFAULT_DOCKER_NETWORK_NAME.to_string(), gateway_route: DockerGatewayRoute::Bridge { @@ -123,6 +123,34 @@ fn runtime_config() -> DockerDriverRuntimeConfig { } } +#[test] +fn docker_config_uses_canonical_sandbox_label_name() { + let config: DockerComputeConfig = + serde_json::from_value(serde_json::json!({ "sandbox_label": "tenant-a" })).unwrap(); + assert_eq!(config.sandbox_label, "tenant-a"); + + let serialized = serde_json::to_value(config).unwrap(); + assert_eq!(serialized["sandbox_label"], "tenant-a"); + assert!(serialized.get("sandbox_namespace").is_none()); +} + +#[test] +fn docker_config_accepts_legacy_sandbox_namespace_alias() { + let config: DockerComputeConfig = + serde_json::from_value(serde_json::json!({ "sandbox_namespace": "tenant-a" })).unwrap(); + assert_eq!(config.sandbox_label, "tenant-a"); +} + +#[test] +fn docker_config_rejects_canonical_and_legacy_sandbox_label_names_together() { + let error = serde_json::from_value::(serde_json::json!({ + "sandbox_label": "tenant-a", + "sandbox_namespace": "tenant-b" + })) + .expect_err("canonical and legacy names must not both be accepted"); + assert!(error.to_string().contains("duplicate field")); +} + fn json_struct(value: serde_json::Value) -> prost_types::Struct { let serde_json::Value::Object(object) = value else { panic!("expected JSON object"); @@ -2041,10 +2069,10 @@ fn build_container_create_body_uses_runtime_namespace_label() { // runtime config, not from `DriverSandbox.namespace`. The gateway // does not populate `DriverSandbox.namespace`, so a container created // with that empty value would not match subsequent list/get/find - // queries (which filter on `config.sandbox_namespace`), leaking + // queries (which filter on `config.sandbox_label`), leaking // sandboxes that the driver itself cannot observe. let mut config = runtime_config(); - config.sandbox_namespace = "tenant-a".to_string(); + config.sandbox_label = "tenant-a".to_string(); let mut sandbox = test_sandbox(); sandbox.namespace = "ignored-by-driver".to_string(); diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index 67ba07944e..a89066f29b 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -247,9 +247,9 @@ Podman follows the same end-to-end contract as the Kubernetes and VM drivers for the in-container SSH relay: gateway config to `PodmanComputeConfig` to sandbox environment to supervisor session registration on that path. -1. `openshell-core` `Config::sandbox_ssh_socket_path` is copied into - `PodmanComputeConfig::sandbox_ssh_socket_path` when the gateway builds the - in-process driver. +1. `[openshell.drivers.podman].ssh_socket_path` is deserialized into + `PodmanComputeConfig::ssh_socket_path` when the gateway builds the in-process + driver. The field defaults to `/run/openshell/ssh.sock` when omitted. 2. `build_env()` in `container.rs` sets `OPENSHELL_SSH_SOCKET_PATH` to that value, alongside required vars such as `OPENSHELL_ENDPOINT` and `OPENSHELL_SANDBOX_ID`. These driver-controlled entries overwrite template @@ -430,11 +430,9 @@ matter compared to cluster or rootful runtimes: - Gateway integration: `crates/openshell-server/src/compute/mod.rs` (`new_podman` and `PodmanComputeDriver` wiring). -- Server configuration: `crates/openshell-server/src/lib.rs` - (`ComputeDriverKind::Podman` builds `PodmanComputeConfig` including - `sandbox_ssh_socket_path` from gateway `Config`). -- Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in - `crates/openshell-core/src/config.rs`. +- Server configuration: + `crates/openshell-server/src/compute/driver_config/builtin.rs` builds + `PodmanComputeConfig` from `[openshell.drivers.podman]`. - SSRF mitigation: `crates/openshell-core/src/net.rs`, `crates/openshell-sandbox/src/proxy.rs`, and `crates/openshell-server/src/grpc/policy.rs`. diff --git a/crates/openshell-driver-podman/src/config.rs b/crates/openshell-driver-podman/src/config.rs index 783611e507..2e2bb0a087 100644 --- a/crates/openshell-driver-podman/src/config.rs +++ b/crates/openshell-driver-podman/src/config.rs @@ -90,7 +90,8 @@ pub struct PodmanComputeConfig { /// default. Defaults to [`openshell_core::config::DEFAULT_SERVER_PORT`]. pub gateway_port: u16, /// Unix socket path the in-container supervisor bridges relay traffic to. - pub sandbox_ssh_socket_path: String, + #[serde(alias = "sandbox_ssh_socket_path")] + pub ssh_socket_path: String, /// Name of the Podman bridge network. /// Created automatically if it does not exist. pub network_name: String, @@ -493,7 +494,7 @@ impl Default for PodmanComputeConfig { image_pull_policy: ImagePullPolicy::default(), grpc_endpoint: String::new(), gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, - sandbox_ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), + ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(), network_name: DEFAULT_NETWORK_NAME.to_string(), host_gateway_ip: Self::default_host_gateway_ip(), stop_timeout_secs: DEFAULT_PODMAN_STOP_TIMEOUT_SECS, @@ -524,7 +525,7 @@ impl std::fmt::Debug for PodmanComputeConfig { .field("image_pull_policy", &self.image_pull_policy.as_str()) .field("grpc_endpoint", &self.grpc_endpoint) .field("gateway_port", &self.gateway_port) - .field("sandbox_ssh_socket_path", &self.sandbox_ssh_socket_path) + .field("ssh_socket_path", &self.ssh_socket_path) .field("network_name", &self.network_name) .field("host_gateway_ip", &self.host_gateway_ip) .field("stop_timeout_secs", &self.stop_timeout_secs) @@ -555,6 +556,37 @@ impl std::fmt::Debug for PodmanComputeConfig { mod tests { use super::*; + #[test] + fn config_uses_canonical_ssh_socket_path_name() { + let config: PodmanComputeConfig = + serde_json::from_value(serde_json::json!({ "ssh_socket_path": "/run/test.sock" })) + .unwrap(); + assert_eq!(config.ssh_socket_path, "/run/test.sock"); + + let serialized = serde_json::to_value(config).unwrap(); + assert_eq!(serialized["ssh_socket_path"], "/run/test.sock"); + assert!(serialized.get("sandbox_ssh_socket_path").is_none()); + } + + #[test] + fn config_accepts_legacy_sandbox_ssh_socket_path_alias() { + let config: PodmanComputeConfig = serde_json::from_value(serde_json::json!({ + "sandbox_ssh_socket_path": "/run/test.sock" + })) + .unwrap(); + assert_eq!(config.ssh_socket_path, "/run/test.sock"); + } + + #[test] + fn config_rejects_canonical_and_legacy_ssh_socket_path_names_together() { + let error = serde_json::from_value::(serde_json::json!({ + "ssh_socket_path": "/run/canonical.sock", + "sandbox_ssh_socket_path": "/run/legacy.sock" + })) + .expect_err("canonical and legacy names must not both be accepted"); + assert!(error.to_string().contains("duplicate field")); + } + #[test] fn default_config_sets_health_check_interval() { let cfg = PodmanComputeConfig::default(); diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index cae477618c..02ebbe2b33 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -513,7 +513,7 @@ fn build_env( ); env.insert( openshell_core::sandbox_env::SSH_SOCKET_PATH.into(), - config.sandbox_ssh_socket_path.clone(), + config.ssh_socket_path.clone(), ); env.insert("OPENSHELL_CONTAINER_IMAGE".into(), image.to_string()); env.insert( @@ -1152,7 +1152,7 @@ pub fn build_container_spec_for_image( "CMD-SHELL".into(), format!( "test -e /var/run/openshell-ssh-ready || test -S {} || ss -tlnp | grep -q :{}", - config.sandbox_ssh_socket_path, + config.ssh_socket_path, openshell_core::config::DEFAULT_SSH_PORT ), ], @@ -2231,7 +2231,7 @@ mod tests { default_image: "test-image:latest".to_string(), grpc_endpoint: "http://localhost:50051".to_string(), host_gateway_ip: String::new(), - sandbox_ssh_socket_path: "/run/openshell/test-ssh.sock".to_string(), + ssh_socket_path: "/run/openshell/test-ssh.sock".to_string(), ..PodmanComputeConfig::default() } } diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index 405deb93a6..71a9dcdf3a 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -168,7 +168,7 @@ async fn main() -> Result<()> { host_gateway_ip: args .host_gateway_ip .unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip), - sandbox_ssh_socket_path: args.sandbox_ssh_socket_path, + ssh_socket_path: args.sandbox_ssh_socket_path, network_name: args.network_name, stop_timeout_secs: args.stop_timeout, supervisor_image: args diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index 10ffac2ba1..f86f8de8d2 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -112,7 +112,7 @@ cat > .cache/gateway-vm/gateway.toml <, pub default_image: String, @@ -248,7 +249,7 @@ pub const DEFAULT_SANDBOX_UID: u32 = 10001; impl Default for VmDriverConfig { fn default() -> Self { Self { - openshell_endpoint: String::new(), + grpc_endpoint: String::new(), state_dir: PathBuf::from("target/openshell-vm-driver"), launcher_bin: None, default_image: String::new(), @@ -305,7 +306,7 @@ impl VmDriverConfig { } fn requires_tls_materials(&self) -> bool { - self.openshell_endpoint.starts_with("https://") + self.grpc_endpoint.starts_with("https://") } fn tls_paths(&self) -> Result, String> { @@ -444,10 +445,10 @@ impl VmDriver { .validate() .map_err(|err| err.message().to_string())?; config.validate_sandbox_identity()?; - if config.openshell_endpoint.trim().is_empty() { + if config.grpc_endpoint.trim().is_empty() { return Err("openshell endpoint is required".to_string()); } - validate_openshell_endpoint(&config.openshell_endpoint)?; + validate_openshell_endpoint(&config.grpc_endpoint)?; let _ = config.tls_paths()?; #[cfg(target_os = "linux")] @@ -904,7 +905,7 @@ impl VmDriver { let endpoint_override = if plan.backend == VmBackend::Qemu { plan.host_ip.as_deref().map(|host_ip| { - guest_visible_openshell_endpoint_for_tap(&self.config.openshell_endpoint, host_ip) + guest_visible_openshell_endpoint_for_tap(&self.config.grpc_endpoint, host_ip) }) } else { None @@ -1676,7 +1677,7 @@ impl VmDriver { "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] )); - plan.gateway_port = gateway_port_from_endpoint(&self.config.openshell_endpoint); + plan.gateway_port = gateway_port_from_endpoint(&self.config.grpc_endpoint); Ok(()) } @@ -1808,7 +1809,7 @@ impl VmDriver { mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] ); let tap = tap_device_name(sandbox_id); - let gateway_port = gateway_port_from_endpoint(&self.config.openshell_endpoint); + let gateway_port = gateway_port_from_endpoint(&self.config.grpc_endpoint); let (vcpus, mem_mib) = if is_gpu { (self.config.gpu_vcpus, self.config.gpu_mem_mib) @@ -4332,7 +4333,7 @@ fn merged_environment(sandbox: &Sandbox) -> HashMap { /// Rewrites loopback host references in a gateway URL to a hostname the guest /// can reach via gvproxy. /// -/// The driver receives the gateway endpoint from `--openshell-endpoint`, which +/// The driver receives the gateway endpoint from `--grpc-endpoint`, which /// in local/dev/e2e setups is typically `http://127.0.0.1:`. That URL is /// useless inside the guest because the guest's loopback interface is its own, /// not the host's. Inside the guest we need a name that gvproxy will translate @@ -4396,7 +4397,7 @@ fn build_guest_environment( endpoint_override: Option<&str>, ) -> Vec { let openshell_endpoint = endpoint_override.map_or_else( - || guest_visible_openshell_endpoint(&config.openshell_endpoint), + || guest_visible_openshell_endpoint(&config.grpc_endpoint), String::from, ); // 1. User-supplied environment (lowest priority). @@ -5523,6 +5524,53 @@ mod tests { static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + + #[test] + fn vm_config_uses_canonical_grpc_endpoint_name() { + let config = VmDriverConfig { + grpc_endpoint: "http://127.0.0.1:8080".to_string(), + ..Default::default() + }; + let serialized = serde_json::to_value(&config).unwrap(); + assert_eq!(serialized["grpc_endpoint"], "http://127.0.0.1:8080"); + assert!(serialized.get("openshell_endpoint").is_none()); + + let parsed: VmDriverConfig = serde_json::from_value(serialized).unwrap(); + assert_eq!(parsed.grpc_endpoint, "http://127.0.0.1:8080"); + } + + #[test] + fn vm_config_accepts_legacy_openshell_endpoint_alias() { + let config = VmDriverConfig::default(); + let mut serialized = serde_json::to_value(config).unwrap(); + let fields = serialized.as_object_mut().unwrap(); + fields.remove("grpc_endpoint"); + fields.insert( + "openshell_endpoint".to_string(), + serde_json::json!("http://127.0.0.1:8080"), + ); + + let parsed: VmDriverConfig = serde_json::from_value(serialized).unwrap(); + assert_eq!(parsed.grpc_endpoint, "http://127.0.0.1:8080"); + } + + #[test] + fn vm_config_rejects_canonical_and_legacy_endpoint_names_together() { + let config = VmDriverConfig { + grpc_endpoint: "http://127.0.0.1:8080".to_string(), + ..Default::default() + }; + let mut serialized = serde_json::to_value(config).unwrap(); + serialized.as_object_mut().unwrap().insert( + "openshell_endpoint".to_string(), + serde_json::json!("http://127.0.0.1:9090"), + ); + + let error = serde_json::from_value::(serialized) + .expect_err("canonical and legacy names must not both be accepted"); + assert!(error.to_string().contains("duplicate field")); + } + struct TestTracing { exporter: opentelemetry_sdk::trace::InMemorySpanExporter, _provider: opentelemetry_sdk::trace::SdkTracerProvider, @@ -6928,7 +6976,7 @@ mod tests { #[test] fn build_guest_environment_sets_supervisor_defaults() { let config = VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; let sandbox = Sandbox { @@ -6953,7 +7001,7 @@ mod tests { #[test] fn build_guest_environment_uses_token_file_without_raw_token_env() { let config = VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; let sandbox = Sandbox { @@ -6985,7 +7033,7 @@ mod tests { #[test] fn build_guest_environment_strips_gateway_tls_server_name() { let config = VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; let sandbox = Sandbox { @@ -7022,7 +7070,7 @@ mod tests { )], || { let config = VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; let sandbox = Sandbox { @@ -7061,7 +7109,7 @@ mod tests { #[test] fn build_guest_environment_uses_endpoint_override_for_tap() { let config = VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), ..Default::default() }; let sandbox = Sandbox { @@ -7263,7 +7311,7 @@ mod tests { #[test] fn build_guest_environment_includes_tls_paths_for_https_endpoint() { let config = VmDriverConfig { - openshell_endpoint: "https://127.0.0.1:8443".to_string(), + grpc_endpoint: "https://127.0.0.1:8443".to_string(), guest_tls_ca: Some(PathBuf::from("/host/ca.crt")), guest_tls_cert: Some(PathBuf::from("/host/tls.crt")), guest_tls_key: Some(PathBuf::from("/host/tls.key")), @@ -7285,7 +7333,7 @@ mod tests { #[test] fn vm_driver_config_requires_tls_materials_for_https_endpoint() { let config = VmDriverConfig { - openshell_endpoint: "https://127.0.0.1:8443".to_string(), + grpc_endpoint: "https://127.0.0.1:8443".to_string(), ..Default::default() }; let err = config @@ -7820,7 +7868,7 @@ mod tests { let (events, _) = broadcast::channel(WATCH_BUFFER); VmDriver { config: VmDriverConfig { - openshell_endpoint: "http://127.0.0.1:8080".to_string(), + grpc_endpoint: "http://127.0.0.1:8080".to_string(), vcpus: 2, mem_mib: 2048, gpu_vcpus: 8, diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 949d4ce05c..dc8a8143ff 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -91,8 +91,12 @@ struct Args { #[arg(long, env = "OPENSHELL_OTLP_ENDPOINT")] otlp_endpoint: Option, - #[arg(long, env = "OPENSHELL_GRPC_ENDPOINT")] - openshell_endpoint: Option, + #[arg( + long = "grpc-endpoint", + alias = "openshell-endpoint", + env = "OPENSHELL_GRPC_ENDPOINT" + )] + grpc_endpoint: Option, #[arg(long, env = "OPENSHELL_SANDBOX_IMAGE", default_value = "")] default_image: String, @@ -218,8 +222,8 @@ async fn main() -> Result<()> { } let driver = VmDriver::new(VmDriverConfig { - openshell_endpoint: args - .openshell_endpoint + grpc_endpoint: args + .grpc_endpoint .ok_or_else(|| miette::miette!("OPENSHELL_GRPC_ENDPOINT is required"))?, state_dir: args.state_dir.clone(), launcher_bin: None, @@ -690,6 +694,28 @@ mod tests { assert!(err.contains("--bind-socket is required")); } + #[test] + fn accepts_canonical_grpc_endpoint_flag() { + let args = Args::try_parse_from([ + "openshell-driver-vm", + "--grpc-endpoint", + "http://127.0.0.1:8080", + ]) + .unwrap(); + assert_eq!(args.grpc_endpoint.as_deref(), Some("http://127.0.0.1:8080")); + } + + #[test] + fn accepts_legacy_openshell_endpoint_flag_alias() { + let args = Args::try_parse_from([ + "openshell-driver-vm", + "--openshell-endpoint", + "http://127.0.0.1:8080", + ]) + .unwrap(); + assert_eq!(args.grpc_endpoint.as_deref(), Some("http://127.0.0.1:8080")); + } + #[test] fn accepts_gateway_otlp_endpoint() { let args = Args::try_parse_from([ diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index 2e86c3a1b5..bf23bdb825 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -1785,6 +1785,22 @@ enable_loopback_service_http = false ); } + #[test] + fn canonical_and_legacy_file_driver_selectors_merge_equivalently() { + for input in [ + "[openshell.gateway]\ncompute_driver = \"podman\"\n", + "[openshell.gateway]\ncompute_drivers = [\"podman\"]\n", + ] { + let (mut args, matches) = + parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]); + let file = config_file_from_toml(input); + + merge_file_into_args(&mut args, &file.openshell.gateway, &matches); + + assert_eq!(args.drivers, vec!["podman".to_string()]); + } + } + #[test] fn server_config_preparation_ignores_unselected_driver_tables() { let _lock = ENV_LOCK diff --git a/crates/openshell-server/src/compute/driver_config/builtin.rs b/crates/openshell-server/src/compute/driver_config/builtin.rs index 948c982dd2..8e55ffc66c 100644 --- a/crates/openshell-server/src/compute/driver_config/builtin.rs +++ b/crates/openshell-server/src/compute/driver_config/builtin.rs @@ -194,6 +194,45 @@ service_account_name = "sandbox-sa" assert_eq!(cfg.service_account_name, "sandbox-sa"); } + #[test] + fn kubernetes_canonical_and_legacy_field_locations_are_equivalent() { + let legacy: config_file::ConfigFile = toml::from_str( + r#" +[openshell.gateway] +sandbox_namespace = "sandboxes" +service_account_name = "sandbox-sa" +enable_user_namespaces = true + +[openshell.drivers.kubernetes] +"#, + ) + .expect("legacy config"); + let canonical: config_file::ConfigFile = toml::from_str( + r#" +[openshell.drivers.kubernetes] +namespace = "sandboxes" +service_account_name = "sandbox-sa" +enable_user_namespaces = true +"#, + ) + .expect("canonical config"); + + let legacy_cfg = + kubernetes_config_from_context(test_context(Some(&legacy))).expect("legacy config"); + let canonical_cfg = kubernetes_config_from_context(test_context(Some(&canonical))) + .expect("canonical config"); + + assert_eq!(legacy_cfg.namespace, canonical_cfg.namespace); + assert_eq!( + legacy_cfg.service_account_name, + canonical_cfg.service_account_name + ); + assert_eq!( + legacy_cfg.enable_user_namespaces, + canonical_cfg.enable_user_namespaces + ); + } + #[test] fn podman_config_reads_bind_mount_opt_in_from_driver_table() { let file: config_file::ConfigFile = toml::from_str( @@ -209,6 +248,62 @@ enable_bind_mounts = true assert!(cfg.enable_bind_mounts); } + #[test] + fn docker_config_reads_canonical_sandbox_label_override() { + let file: config_file::ConfigFile = toml::from_str( + r#" +[openshell.gateway] +sandbox_namespace = "gateway-default" + +[openshell.drivers.docker] +sandbox_label = "driver-specific" +"#, + ) + .expect("valid config"); + + let cfg = docker_config_from_context(test_context(Some(&file))).expect("docker config"); + + assert_eq!(cfg.sandbox_label, "driver-specific"); + } + + #[test] + fn docker_config_reads_legacy_sandbox_namespace_override() { + let file: config_file::ConfigFile = toml::from_str( + r#" +[openshell.gateway] +sandbox_namespace = "gateway-default" + +[openshell.drivers.docker] +sandbox_namespace = "driver-specific" +"#, + ) + .expect("valid config"); + + let cfg = docker_config_from_context(test_context(Some(&file))).expect("docker config"); + + assert_eq!(cfg.sandbox_label, "driver-specific"); + } + + #[test] + fn docker_config_rejects_canonical_and_legacy_sandbox_label_names_together() { + let file: config_file::ConfigFile = toml::from_str( + r#" +[openshell.gateway] +sandbox_namespace = "gateway-default" + +[openshell.drivers.docker] +sandbox_label = "canonical" +sandbox_namespace = "legacy" +"#, + ) + .expect("valid config file structure"); + + let error = docker_config_from_context(test_context(Some(&file))) + .expect_err("canonical and legacy names must not both be accepted"); + + assert!(error.to_string().contains("duplicate field")); + } + #[test] fn docker_config_reads_bind_mount_opt_in_from_driver_table() { let file: config_file::ConfigFile = toml::from_str( diff --git a/crates/openshell-server/src/compute/vm.rs b/crates/openshell-server/src/compute/vm.rs index 80b445d201..483571fbff 100644 --- a/crates/openshell-server/src/compute/vm.rs +++ b/crates/openshell-server/src/compute/vm.rs @@ -479,9 +479,7 @@ pub async fn spawn( .arg(std::process::id().to_string()); command.arg("--log-level").arg(&config.log_level); append_otlp_args(&mut command, otlp_config); - command - .arg("--openshell-endpoint") - .arg(&vm_config.grpc_endpoint); + command.arg("--grpc-endpoint").arg(&vm_config.grpc_endpoint); command.arg("--state-dir").arg(&vm_config.state_dir); if !vm_config.default_image.trim().is_empty() { command.arg("--default-image").arg(&vm_config.default_image); diff --git a/crates/openshell-server/src/config_file.rs b/crates/openshell-server/src/config_file.rs index 00b7a2f64d..3c5d30ce9c 100644 --- a/crates/openshell-server/src/config_file.rs +++ b/crates/openshell-server/src/config_file.rs @@ -32,7 +32,8 @@ use openshell_core::{ GatewayAuthConfig, GatewayInterceptorConfig, GatewayJwtConfig, GatewayProviderProfileSourceConfig, MtlsAuthConfig, OidcConfig, TlsConfig, }; -use serde::{Deserialize, Serialize}; +use serde::de::{SeqAccess, Visitor}; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; /// Latest schema version this build understands. pub const SCHEMA_VERSION: u32 = 1; @@ -100,7 +101,18 @@ pub struct GatewayFileSection { pub log_level: Option, // ── Drivers ────────────────────────────────────────────────────────── - #[serde(default)] + /// Canonical TOML uses the singular `compute_driver = "..."`. The legacy + /// `compute_drivers = ["..."]` form remains accepted and is normalized to + /// this existing vector representation so Rust callers and runtime + /// validation retain their current behavior. + #[serde( + default, + rename = "compute_driver", + alias = "compute_drivers", + deserialize_with = "deserialize_compute_drivers", + serialize_with = "serialize_compute_drivers", + skip_serializing_if = "Option::is_none" + )] pub compute_drivers: Option>, #[serde(default)] pub credential_drivers: Option>, @@ -110,6 +122,9 @@ pub struct GatewayFileSection { pub credential_storage: Option, // ── Sandbox / SSH ──────────────────────────────────────────────────── + /// Compatibility input for Kubernetes `namespace` and Docker + /// `sandbox_label`. Canonical configurations set those driver-owned + /// fields in their respective `[openshell.drivers.]` tables. #[serde(default)] pub sandbox_namespace: Option, #[serde(default)] @@ -138,10 +153,12 @@ pub struct GatewayFileSection { pub supervisor_image: Option, #[serde(default)] pub client_tls_secret_name: Option, + /// Compatibility input for Kubernetes `service_account_name`. #[serde(default)] pub service_account_name: Option, #[serde(default)] pub host_gateway_ip: Option, + /// Compatibility input for Kubernetes `enable_user_namespaces`. #[serde(default)] pub enable_user_namespaces: Option, /// Lifetime (seconds) of the projected `ServiceAccount` token kubelet @@ -189,6 +206,62 @@ pub struct GatewayFileSection { pub database_url: Option, } +fn deserialize_compute_drivers<'de, D>(deserializer: D) -> Result>, D::Error> +where + D: Deserializer<'de>, +{ + struct ComputeDriversVisitor; + + impl<'de> Visitor<'de> for ComputeDriversVisitor { + type Value = Option>; + + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a compute driver name or an array of compute driver names") + } + + fn visit_str(self, value: &str) -> Result + where + E: serde::de::Error, + { + Ok(Some(vec![value.to_string()])) + } + + fn visit_string(self, value: String) -> Result + where + E: serde::de::Error, + { + Ok(Some(vec![value])) + } + + fn visit_seq(self, mut sequence: A) -> Result + where + A: SeqAccess<'de>, + { + let mut drivers = Vec::new(); + while let Some(driver) = sequence.next_element::()? { + drivers.push(driver); + } + Ok(Some(drivers)) + } + } + + deserializer.deserialize_any(ComputeDriversVisitor) +} + +fn serialize_compute_drivers( + drivers: &Option>, + serializer: S, +) -> Result +where + S: Serializer, +{ + match drivers { + Some(drivers) if drivers.len() == 1 => serializer.serialize_str(&drivers[0]), + Some(drivers) => drivers.serialize(serializer), + None => serializer.serialize_none(), + } +} + /// `[openshell.gateway.otlp]` section. /// /// Presence of this table enables OTLP export; there is no `enabled` flag. @@ -434,7 +507,7 @@ pub fn driver_table( }; for key in inheritable_keys(driver_name) { - if merged.contains_key(*key) { + if driver_field_is_present(&merged, driver_name, key) { continue; } if let Some(value) = gateway_inherited_value(gateway, key) { @@ -461,7 +534,7 @@ fn inheritable_keys(driver_name: &str) -> &'static [&'static str] { "sa_token_ttl_secs", ], Some(ComputeDriverKind::Docker) => &[ - "sandbox_namespace", + "sandbox_label", "default_image", "supervisor_image", "host_gateway_ip", @@ -487,9 +560,24 @@ fn inheritable_keys(driver_name: &str) -> &'static [&'static str] { } } +fn driver_field_is_present(table: &toml::Table, driver_name: &str, key: &str) -> bool { + if table.contains_key(key) { + return true; + } + + // Docker's legacy alias must count as an explicit driver override. If it + // did not, gateway inheritance would inject `sandbox_label` alongside the + // alias and serde would reject the merged table as a duplicate field. + matches!( + driver_name.parse::().ok(), + Some(ComputeDriverKind::Docker) + ) && key == "sandbox_label" + && table.contains_key("sandbox_namespace") +} + fn gateway_inherited_value(g: &GatewayFileSection, key: &str) -> Option { match key { - "namespace" | "sandbox_namespace" => g.sandbox_namespace.as_deref().map(string_value), + "namespace" | "sandbox_label" => g.sandbox_namespace.as_deref().map(string_value), "default_image" => g.default_image.as_deref().map(string_value), "supervisor_image" => g.supervisor_image.as_deref().map(string_value), "client_tls_secret_name" => g.client_tls_secret_name.as_deref().map(string_value), @@ -535,6 +623,86 @@ mod tests { assert!(file.openshell.drivers.is_empty()); } + #[test] + fn canonical_compute_driver_scalar_normalizes_to_existing_vector() { + let file: ConfigFile = toml::from_str( + r#" +[openshell.gateway] +compute_driver = "docker" +"#, + ) + .expect("canonical compute driver parses"); + + assert_eq!( + file.openshell.gateway.compute_drivers, + Some(vec!["docker".to_string()]) + ); + } + + #[test] + fn legacy_compute_drivers_list_remains_accepted() { + for (input, expected) in [ + ("compute_drivers = []", Vec::::new()), + ("compute_drivers = [\"docker\"]", vec!["docker".to_string()]), + ( + "compute_drivers = [\"docker\", \"podman\"]", + vec!["docker".to_string(), "podman".to_string()], + ), + ] { + let file: ConfigFile = toml::from_str(&format!("[openshell.gateway]\n{input}\n")) + .expect("legacy compute drivers parse"); + assert_eq!(file.openshell.gateway.compute_drivers, Some(expected)); + } + } + + #[test] + fn compute_driver_rejects_non_string_values_with_a_clear_error() { + let error = toml::from_str::( + r" +[openshell.gateway] +compute_driver = 42 +", + ) + .expect_err("compute driver must be a string or string array"); + + assert!( + error + .to_string() + .contains("a compute driver name or an array of compute driver names") + ); + } + + #[test] + fn canonical_and_legacy_compute_driver_names_are_rejected_together() { + let error = toml::from_str::( + r#" +[openshell.gateway] +compute_driver = "docker" +compute_drivers = ["docker"] +"#, + ) + .expect_err("canonical and legacy names must not both be accepted"); + + assert!(error.to_string().contains("duplicate field")); + } + + #[test] + fn compute_driver_serialization_uses_canonical_scalar_name() { + let file = ConfigFile { + openshell: OpenShellRoot { + gateway: GatewayFileSection { + compute_drivers: Some(vec!["docker".to_string()]), + ..Default::default() + }, + ..Default::default() + }, + }; + + let serialized = toml::to_string(&file).expect("config serializes"); + assert!(serialized.contains("compute_driver = \"docker\"")); + assert!(!serialized.contains("compute_drivers")); + } + #[test] fn parses_full_example() { let toml = r#" @@ -545,7 +713,7 @@ version = 1 bind_address = "0.0.0.0:8080" health_bind_address = "0.0.0.0:8081" log_level = "info" -compute_drivers = ["kubernetes"] +compute_driver = "kubernetes" credential_drivers = ["kubernetes-secrets"] sandbox_namespace = "agents" grpc_rate_limit_requests = 120 @@ -998,6 +1166,73 @@ version = 2 ); } + #[test] + fn kubernetes_driver_fields_override_legacy_gateway_compatibility_values() { + let gateway = GatewayFileSection { + sandbox_namespace: Some("legacy-namespace".to_string()), + service_account_name: Some("legacy-service-account".to_string()), + enable_user_namespaces: Some(true), + ..Default::default() + }; + let raw = toml::toml! { + namespace = "canonical-namespace" + service_account_name = "canonical-service-account" + enable_user_namespaces = false + }; + let merged = driver_table( + ComputeDriverKind::Kubernetes.as_str(), + &gateway, + Some(&toml::Value::Table(raw)), + ); + let table = merged.as_table().expect("table"); + + assert_eq!( + table.get("namespace").and_then(toml::Value::as_str), + Some("canonical-namespace") + ); + assert_eq!( + table + .get("service_account_name") + .and_then(toml::Value::as_str), + Some("canonical-service-account") + ); + assert_eq!( + table + .get("enable_user_namespaces") + .and_then(toml::Value::as_bool), + Some(false) + ); + } + + #[test] + fn kubernetes_driver_inherits_legacy_gateway_compatibility_values() { + let gateway = GatewayFileSection { + sandbox_namespace: Some("legacy-namespace".to_string()), + service_account_name: Some("legacy-service-account".to_string()), + enable_user_namespaces: Some(true), + ..Default::default() + }; + let merged = driver_table(ComputeDriverKind::Kubernetes.as_str(), &gateway, None); + let table = merged.as_table().expect("table"); + + assert_eq!( + table.get("namespace").and_then(toml::Value::as_str), + Some("legacy-namespace") + ); + assert_eq!( + table + .get("service_account_name") + .and_then(toml::Value::as_str), + Some("legacy-service-account") + ); + assert_eq!( + table + .get("enable_user_namespaces") + .and_then(toml::Value::as_bool), + Some(true) + ); + } + #[test] fn docker_driver_table_inherits_gateway_defaults() { let gateway = GatewayFileSection { @@ -1009,7 +1244,7 @@ version = 2 let merged = driver_table(ComputeDriverKind::Docker.as_str(), &gateway, None); let table = merged.as_table().expect("table"); assert_eq!( - table.get("sandbox_namespace").and_then(|v| v.as_str()), + table.get("sandbox_label").and_then(|v| v.as_str()), Some("agents") ); assert_eq!( @@ -1022,6 +1257,52 @@ version = 2 ); } + #[test] + fn docker_driver_canonical_sandbox_label_overrides_gateway_default() { + let gateway = GatewayFileSection { + sandbox_namespace: Some("gateway-default".to_string()), + ..Default::default() + }; + let raw = toml::toml! { + sandbox_label = "driver-specific" + }; + let merged = driver_table( + ComputeDriverKind::Docker.as_str(), + &gateway, + Some(&toml::Value::Table(raw)), + ); + let table = merged.as_table().expect("table"); + assert_eq!( + table.get("sandbox_label").and_then(|value| value.as_str()), + Some("driver-specific") + ); + assert!(!table.contains_key("sandbox_namespace")); + } + + #[test] + fn docker_driver_legacy_sandbox_namespace_overrides_gateway_default() { + let gateway = GatewayFileSection { + sandbox_namespace: Some("gateway-default".to_string()), + ..Default::default() + }; + let raw = toml::toml! { + sandbox_namespace = "driver-specific" + }; + let merged = driver_table( + ComputeDriverKind::Docker.as_str(), + &gateway, + Some(&toml::Value::Table(raw)), + ); + let table = merged.as_table().expect("table"); + assert_eq!( + table + .get("sandbox_namespace") + .and_then(|value| value.as_str()), + Some("driver-specific") + ); + assert!(!table.contains_key("sandbox_label")); + } + #[test] fn podman_driver_table_inherits_gateway_host_gateway_ip() { let gateway = GatewayFileSection { @@ -1119,7 +1400,7 @@ version = 2 /// - template corruption or unknown fields (`deny_unknown_fields`) /// - schema drift (version bump or field renames) /// - accidental addition of a wildcard bind-address override - /// - accidental changes to the compute driver list + /// - accidental changes to the configured compute driver #[test] fn rpm_default_config_parses_and_has_podman_defaults() { let path = @@ -1138,11 +1419,11 @@ version = 2 let drivers = gw .compute_drivers .as_ref() - .expect("compute_drivers must be explicitly set in the RPM default config"); + .expect("compute_driver must be explicitly set in the RPM default config"); assert_eq!( drivers, &["podman".to_string()], - "RPM default must pin compute_drivers to [podman] to prevent unexpected \ + "RPM default must pin compute_driver to podman to prevent unexpected \ driver selection when Docker is also installed" ); } diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index a6031a9bc6..46f5c9f25a 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -490,7 +490,7 @@ pub(crate) async fn run_server( &signing_pem, kid.clone(), &jwt.gateway_id, - Duration::from_secs(jwt.ttl_secs), + jwt.sandbox_token_ttl().unwrap_or_default(), ) .map_err(Error::config)?, ); @@ -1271,13 +1271,13 @@ fn kubernetes_sandbox_jwt_expiry_disabled(config: &Config) -> bool { config .gateway_jwt .as_ref() - .is_some_and(|jwt| jwt.ttl_secs == 0) + .is_some_and(|jwt| jwt.sandbox_token_ttl().is_none()) } fn warn_if_kubernetes_sandbox_jwt_expiry_disabled(config: &Config) { if kubernetes_sandbox_jwt_expiry_disabled(config) { warn!( - "Kubernetes gateway configured with non-expiring sandbox JWTs (gateway_jwt.ttl_secs = 0); set ttl_secs > 0 for shared Kubernetes deployments" + "Kubernetes gateway configured with non-expiring sandbox JWTs (gateway_jwt.ttl_secs is omitted or zero); set ttl_secs > 0 for shared Kubernetes deployments" ); } } diff --git a/deploy/docker/gateway.toml b/deploy/docker/gateway.toml index 4fe84d633a..da8ef72873 100644 --- a/deploy/docker/gateway.toml +++ b/deploy/docker/gateway.toml @@ -30,7 +30,7 @@ version = 1 bind_address = "127.0.0.1:8080" health_bind_address = "127.0.0.1:8081" log_level = "info" -compute_drivers = ["docker"] +compute_driver = "docker" disable_tls = true [openshell.drivers.docker] @@ -41,8 +41,8 @@ default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest" # Only pull images that are not already cached locally. image_pull_policy = "IfNotPresent" -# Prefix applied to sandbox container names. -sandbox_namespace = "openshell" +# Value assigned to the openshell.sandbox_namespace label on sandbox containers. +sandbox_label = "openshell" # Address sandbox containers use to call back to the gateway. # The Docker driver replaces the host with host.openshell.internal and the # port with the gateway's own bind port (8080). Only the scheme survives. diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 9d24dbd917..f844b5ac77 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -45,7 +45,6 @@ data: {{- if $credentialDrivers }} credential_drivers = [{{- range $i, $driver := $credentialDrivers }}{{ if $i }}, {{ end }}{{ $driver | quote }}{{- end }}] {{- end }} - sandbox_namespace = {{ include "openshell.sandboxNamespace" . | quote }} {{- $policyValidationFailureMode := .Values.server.policyValidationFailureMode }} {{- if not (has $policyValidationFailureMode (list "fail_closed" "retain_last_valid")) }} {{- fail "server.policyValidationFailureMode must be fail_closed or retain_last_valid" }} @@ -58,9 +57,6 @@ data: {{- if .Values.server.hostGatewayIP }} host_gateway_ip = {{ .Values.server.hostGatewayIP | quote }} {{- end }} - {{- if .Values.server.enableUserNamespaces }} - enable_user_namespaces = true - {{- end }} {{- if .Values.server.disableTls }} disable_tls = true {{- else }} @@ -135,10 +131,14 @@ data: {{- end }} [openshell.drivers.kubernetes] + namespace = {{ include "openshell.sandboxNamespace" . | quote }} workspace_mode = {{ .Values.server.drivers.kubernetes.workspaceMode | default "shared" | quote }} gateway_id = {{ .Values.server.sandboxJwt.gatewayId | default (include "openshell.fullname" .) | quote }} grpc_endpoint = {{ include "openshell.grpcEndpoint" . | quote }} service_account_name = {{ include "openshell.sandboxServiceAccountName" . | quote }} + {{- if .Values.server.enableUserNamespaces }} + enable_user_namespaces = true + {{- end }} {{- if .Values.server.drivers.kubernetes.operatorNamespaceLabel }} operator_namespace_label = {{ .Values.server.drivers.kubernetes.operatorNamespaceLabel | quote }} {{- end }} diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index afacd01eb4..8db2e4699b 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -103,6 +103,18 @@ tests: path: data["gateway.toml"] pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?service_account_name\s*=\s*"openshell-sandbox"' + - it: renders user namespace enablement under [openshell.drivers.kubernetes] + template: templates/gateway-config.yaml + set: + server.enableUserNamespaces: true + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?enable_user_namespaces\s*=\s*true' + - notMatchRegex: + path: data["gateway.toml"] + pattern: '(?ms)\[openshell\.gateway\][^\[]*?enable_user_namespaces' + - it: renders combined supervisor topology by default under [openshell.drivers.kubernetes] template: templates/gateway-config.yaml asserts: diff --git a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml index ee89fce53d..0576bbec46 100644 --- a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml +++ b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml @@ -13,21 +13,27 @@ release: namespace: my-namespace tests: - - it: defaults sandbox_namespace to release namespace in the TOML config + - it: defaults the Kubernetes driver namespace to release namespace template: templates/gateway-config.yaml asserts: - matchRegex: path: data["gateway.toml"] - pattern: 'sandbox_namespace\s*=\s*"my-namespace"' + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?namespace\s*=\s*"my-namespace"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'sandbox_namespace\s*=' - - it: uses explicit sandboxNamespace when set + - it: uses explicit sandboxNamespace for the Kubernetes driver template: templates/gateway-config.yaml set: server.sandboxNamespace: other-ns asserts: - matchRegex: path: data["gateway.toml"] - pattern: 'sandbox_namespace\s*=\s*"other-ns"' + pattern: '(?ms)\[openshell\.drivers\.kubernetes\].*?namespace\s*=\s*"other-ns"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'sandbox_namespace\s*=' - it: defaults NetworkPolicy namespace to release namespace template: templates/networkpolicy.yaml diff --git a/deploy/rpm/CONFIGURATION.md b/deploy/rpm/CONFIGURATION.md index 4fc18e6215..aaa97d08d0 100644 --- a/deploy/rpm/CONFIGURATION.md +++ b/deploy/rpm/CONFIGURATION.md @@ -20,7 +20,7 @@ The defaults are tuned for rootless Podman use: version = 1 [openshell.gateway] -compute_drivers = ["podman"] +compute_driver = "podman" ``` The RPM does not override `bind_address`. The primary listener uses the @@ -28,7 +28,7 @@ built-in `127.0.0.1:17670` default. The Podman driver reports the callback interface it needs, and the gateway adds a separate listener scoped to that interface. This keeps the general API off unrelated host interfaces. -`compute_drivers = ["podman"]` pins the compute driver to Podman. Without +`compute_driver = "podman"` pins the compute driver to Podman. Without this, the gateway auto-detects in order: Kubernetes, Podman, Docker. Pinning prevents unexpected driver selection if Docker is also installed on the host. @@ -215,7 +215,7 @@ overrides that persist across package upgrades. | TOML option | Default | Description | |-------------|---------|-------------| | `bind_address` | `127.0.0.1:17670` (gateway default) | Address for the primary gRPC/HTTP API listener. | -| `compute_drivers` | `["podman"]` (RPM default) | When unset, the gateway auto-detects Kubernetes, then Podman, then Docker. The RPM default pins to Podman. | +| `compute_driver` | `"podman"` (RPM default) | When unset, the gateway auto-detects Kubernetes, then Podman, then Docker. The RPM default pins to Podman. The legacy `compute_drivers` list remains accepted. | | `default_image` | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` | Default sandbox image. | | `supervisor_image` | `ghcr.io/nvidia/openshell/supervisor:latest` | Supervisor image mounted into Podman sandboxes. | | `guest_tls_ca`, `guest_tls_cert`, `guest_tls_key` | auto-generated paths | Client TLS material bind-mounted into sandbox containers. | @@ -235,7 +235,7 @@ settings: version = 1 [openshell.gateway] -compute_drivers = ["podman"] +compute_driver = "podman" default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" [openshell.drivers.podman] diff --git a/deploy/rpm/TROUBLESHOOTING.md b/deploy/rpm/TROUBLESHOOTING.md index 103ce3bf9d..f67b69149b 100644 --- a/deploy/rpm/TROUBLESHOOTING.md +++ b/deploy/rpm/TROUBLESHOOTING.md @@ -255,7 +255,7 @@ and map the relevant variables: | Environment variable | TOML equivalent | |---|---| | `OPENSHELL_BIND_ADDRESS=A` + `OPENSHELL_SERVER_PORT=P` | `bind_address = "A:P"` under `[openshell.gateway]` | -| `OPENSHELL_DRIVERS=podman` | `compute_drivers = ["podman"]` under `[openshell.gateway]` | +| `OPENSHELL_DRIVERS=podman` | `compute_driver = "podman"` under `[openshell.gateway]` | | `OPENSHELL_DISABLE_TLS=true` | `disable_tls = true` under `[openshell.gateway]` | | `OPENSHELL_TLS_CERT=PATH` | `cert_path = "PATH"` under `[openshell.gateway.tls]` | | `OPENSHELL_TLS_KEY=PATH` | `key_path = "PATH"` under `[openshell.gateway.tls]` | diff --git a/deploy/rpm/gateway.toml.default b/deploy/rpm/gateway.toml.default index cd7e0d99c3..ba76f873b2 100644 --- a/deploy/rpm/gateway.toml.default +++ b/deploy/rpm/gateway.toml.default @@ -25,4 +25,4 @@ version = 1 # Pin to the Podman compute driver. Without this, the gateway auto-detects # in order: Kubernetes, Podman, Docker. Pinning prevents unexpected driver # selection if Docker is also installed on the host. -compute_drivers = ["podman"] +compute_driver = "podman" diff --git a/docs/about/installation.mdx b/docs/about/installation.mdx index e464d3da42..ab101c6701 100644 --- a/docs/about/installation.mdx +++ b/docs/about/installation.mdx @@ -24,7 +24,7 @@ Use `openshell status` to confirm the CLI can reach the gateway. ## Supported Compute Drivers -OpenShell supports several local compute drivers. Package-managed gateways leave the driver unset by default so the gateway can auto-detect an available driver. Set `compute_drivers` in the gateway TOML when you need to pin a specific driver. +OpenShell supports several local compute drivers. Package-managed gateways leave the driver unset by default so the gateway can auto-detect an available driver. Set `compute_driver` in the gateway TOML when you need to pin a specific driver. | Compute Driver | How It Is Configured | System Requirements | |---|---|---| diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 8d74b3d44f..02878029b2 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -57,6 +57,8 @@ version = 1 # ... credential-driver-specific settings ... ``` +The canonical gateway selector is `compute_driver = ""`. The legacy `compute_drivers = [""]` list remains accepted for compatibility. An omitted selector or an empty legacy list retains auto-detection; a legacy list with multiple entries retains the existing startup error because only one compute driver can be active. + ## Full Example A complete gateway configuration covering every section. Trim to the fields you need. @@ -75,15 +77,14 @@ metrics_bind_address = "0.0.0.0:9090" log_level = "info" -# When empty, the gateway auto-detects Kubernetes, then Podman, then Docker. +# When omitted, the gateway auto-detects Kubernetes, then Podman, then Docker. # VM is never auto-detected and requires an explicit entry here. -compute_drivers = ["kubernetes"] +compute_driver = "kubernetes" # Optional external provider credential storage backend. Omit this key to use # the gateway's default encrypted database credential storage. credential_drivers = ["kubernetes-secrets"] -sandbox_namespace = "openshell" ssh_session_ttl_secs = 3600 # Reject invalid policy generations securely by default. Set @@ -106,9 +107,7 @@ default_image = "ghcr.io/nvidia/openshell/sandbox:latest" # Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" client_tls_secret_name = "openshell-client-tls" -service_account_name = "openshell-sandbox" host_gateway_ip = "10.0.0.1" -enable_user_namespaces = false sa_token_ttl_secs = 3600 guest_tls_ca = "/etc/openshell/certs/ca.pem" guest_tls_cert = "/etc/openshell/certs/client.pem" @@ -195,6 +194,11 @@ failure_policy = "fail_closed" rpc = "openshell.v1.OpenShell/UpdateConfig" phases = ["validate"] +[openshell.drivers.kubernetes] +namespace = "openshell" +service_account_name = "openshell-sandbox" +enable_user_namespaces = false + [openshell.credential_drivers.kubernetes-secrets] namespace = "openshell" allow_reference_namespace = false @@ -418,6 +422,8 @@ args = [ Each example is a complete TOML file for one compute driver. The examples repeat `[openshell]` and `[openshell.gateway]` so they stay copyable, and the driver tables list the accepted driver-specific keys. Driver-specific values override inherited gateway defaults. The gateway rejects unknown driver fields after inheritance is merged. +Canonical Kubernetes configurations set `namespace`, `service_account_name`, and `enable_user_namespaces` in `[openshell.drivers.kubernetes]`. Their historical gateway-level locations remain accepted as compatibility inputs and retain the same lower precedence. Gateway-level `sandbox_namespace` also remains a compatibility default for Docker `sandbox_label`. + ### Kubernetes The gateway runs as a Pod and creates sandbox Pods in another namespace. mTLS material for sandboxes is delivered through a Kubernetes Secret rather than host-side file paths. @@ -431,7 +437,7 @@ bind_address = "0.0.0.0:8080" health_bind_address = "0.0.0.0:8081" metrics_bind_address = "0.0.0.0:9090" log_level = "info" -compute_drivers = ["kubernetes"] +compute_driver = "kubernetes" [openshell.gateway.tls] cert_path = "/etc/openshell-tls/server/tls.crt" @@ -562,14 +568,15 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" -compute_drivers = ["docker"] +compute_driver = "docker" [openshell.drivers.docker] socket_path = "/var/run/docker.sock" default_image = "ghcr.io/nvidia/openshell/sandbox:latest" # Docker vocabulary: Always | IfNotPresent | Never. Empty behaves like IfNotPresent. image_pull_policy = "IfNotPresent" -sandbox_namespace = "docker-dev" +# Value assigned to the openshell.sandbox_namespace label on sandbox containers. +sandbox_label = "docker-dev" # Empty auto-detects https://host.openshell.internal: when guest TLS is set. grpc_endpoint = "https://host.openshell.internal:17670" # Skip the image-pull-and-extract step by pointing at a locally built binary. @@ -591,6 +598,10 @@ enable_bind_mounts = false sandbox_pids_limit = 2048 ``` +Use `sandbox_label` for new Docker configurations. The legacy +`sandbox_namespace` key remains accepted as a compatibility alias. Do not set +both keys in the same driver table. + ### Podman Sandboxes run as Podman containers on a user-mode bridge network. The supervisor image is mounted read-only via Podman's `type=image` mount; guest mTLS material is supplied as host paths. @@ -602,7 +613,7 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" -compute_drivers = ["podman"] +compute_driver = "podman" [openshell.drivers.podman] # Rootless socket path. For root Podman use /run/podman/podman.sock. @@ -618,7 +629,7 @@ network_name = "openshell" # Omit for the platform default: empty on Linux, 192.168.127.254 on macOS Podman machine. # Set "" to force Podman's host-gateway resolver. # host_gateway_ip = "192.168.127.254" -sandbox_ssh_socket_path = "/run/openshell/ssh.sock" +ssh_socket_path = "/run/openshell/ssh.sock" stop_timeout_secs = 45 # Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" @@ -706,6 +717,10 @@ health_check_interval_secs = 10 # proxy_connect_by_hostname = true ``` +Use `ssh_socket_path` for new Podman configurations. The legacy +`sandbox_ssh_socket_path` key remains accepted as a compatibility alias. Do not +set both keys in the same driver table. + ### MicroVM Each sandbox runs inside its own libkrun microVM managed by the standalone `openshell-driver-vm` subprocess. Use this driver when you want stronger isolation than container namespaces alone. @@ -718,7 +733,7 @@ version = 1 bind_address = "127.0.0.1:17670" log_level = "info" # VM is never auto-detected; an explicit entry here is required. -compute_drivers = ["vm"] +compute_driver = "vm" [openshell.drivers.vm] state_dir = "/var/lib/openshell/vm" @@ -755,7 +770,7 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:17670" log_level = "info" -compute_drivers = ["kyma"] +compute_driver = "kyma" [openshell.drivers.kyma] socket_path = "/run/openshell/kyma-compute-driver.sock" diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 897e780c39..df2791937b 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -20,24 +20,26 @@ remain unavailable. ## Configure a Compute Driver -Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set `compute_drivers` in the gateway TOML file: +Configure the compute driver on the gateway. Current releases accept one driver per gateway. Set the singular `compute_driver` key in the gateway TOML file: ```toml [openshell.gateway] -compute_drivers = ["docker"] +compute_driver = "docker" ``` Reserved built-in values are `docker`, `podman`, `kubernetes`, and `vm`. Non-reserved names select an extension driver and require a `socket_path` in `[openshell.drivers.]`. -When `compute_drivers` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Local container runtimes must respond to an API probe before the gateway selects them. The VM driver is never auto-detected; configure it explicitly with `compute_drivers = ["vm"]` or set `OPENSHELL_DRIVERS=vm` in the launch environment. +When `compute_driver` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Local container runtimes must respond to an API probe before the gateway selects them. The VM driver is never auto-detected; configure it explicitly with `compute_driver = "vm"` or set `OPENSHELL_DRIVERS=vm` in the launch environment. + +The legacy `compute_drivers = [""]` list remains accepted for compatibility. Empty legacy lists retain auto-detection, and lists with more than one entry retain the existing startup error because a gateway supports exactly one active compute driver. Common gateway options: | Gateway TOML option | Description | |---|---| -| `compute_drivers = [""]` | Select the compute driver. Built-in values are `docker`, `podman`, `kubernetes`, and `vm`; custom names require `[openshell.drivers.].socket_path`. | +| `compute_driver = ""` | Select the compute driver. Built-in values are `docker`, `podman`, `kubernetes`, and `vm`; custom names require `[openshell.drivers.].socket_path`. | Set driver-specific values such as sandbox images, callback endpoints, network names, TLS material, and VM sizing in the gateway TOML file. See the [Gateway Configuration File](./gateway-config) reference for the full `[openshell.drivers.]` schema. @@ -47,7 +49,7 @@ the gateway at the Unix socket the operator has already provisioned: ```toml [openshell.gateway] -compute_drivers = ["kyma"] +compute_driver = "kyma" [openshell.drivers.kyma] socket_path = "/run/openshell/kyma.sock" @@ -136,7 +138,7 @@ that already covers loopback. Otherwise, the Docker driver requests a separate For maintainer-level implementation details, refer to the [Docker driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-docker/README.md). -Select Docker with `compute_drivers = ["docker"]` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. +Select Docker with `compute_driver = "docker"` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `sandbox_label`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. Stop stops the existing Docker container without removing its writable layer or attached volumes. Start starts that same container. A durably @@ -207,7 +209,7 @@ The gateway talks to the Podman API socket. The Podman driver requires Podman 5. For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md). -Select Podman with `compute_drivers = ["podman"]` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `sandbox_ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`. +Select Podman with `compute_driver = "podman"` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`. Podman sandboxes default to a 45-second graceful stop window before Podman escalates from `SIGTERM` to `SIGKILL`. Set `stop_timeout_secs` in gateway config, or `OPENSHELL_STOP_TIMEOUT` for the standalone driver, when a local runtime needs a different teardown window. @@ -297,11 +299,11 @@ For maintainer-level implementation details, refer to the [VM driver README](htt The VM driver is opt-in. Release packages can install `openshell-driver-vm`, but the gateway does not select it unless you configure the driver explicitly. -Enable VM by setting `compute_drivers = ["vm"]` in the gateway TOML file: +Enable VM by setting `compute_driver = "vm"` in the gateway TOML file: ```toml [openshell.gateway] -compute_drivers = ["vm"] +compute_driver = "vm" ``` For a launch-time override, set `OPENSHELL_DRIVERS=vm` in the gateway environment and restart the service. @@ -339,15 +341,16 @@ owner references or use the sandbox ServiceAccount. The operator namespace allowlist is a trust grant, not a tenant isolation mechanism. -Helm deployments set Kubernetes driver values through the chart. +Helm deployments set Kubernetes driver values through the chart. Canonical TOML places `namespace`, `service_account_name`, and `enable_user_namespaces` in `[openshell.drivers.kubernetes]`. Their historical `[openshell.gateway]` locations remain accepted as lower-precedence compatibility inputs. For maintainer-level implementation details, refer to the [Kubernetes driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-kubernetes/README.md). | Gateway configuration | Helm value | Description | |---|---|---| -| `compute_drivers = ["kubernetes"]` | Not applicable | Select the Kubernetes compute driver. | +| `compute_driver = "kubernetes"` | Not applicable | Select the Kubernetes compute driver. | | `[openshell.drivers.kubernetes].namespace` | `server.sandboxNamespace` | Set the namespace for sandbox resources. The Helm chart defaults to the release namespace when left empty. | -| `service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the gateway TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | +| `[openshell.drivers.kubernetes].service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the gateway TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | +| `[openshell.drivers.kubernetes].enable_user_namespaces` | `server.enableUserNamespaces` | Enable Kubernetes user namespaces for sandbox pods. | | `default_image` | `server.sandboxImage` | Set the default sandbox image. | | `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the Kubernetes image pull policy for sandbox pods. | | `image_pull_secrets` | `server.sandboxImagePullSecrets` | Attach Kubernetes image-pull Secrets to sandbox pods. Managed mode copies these explicitly named Secrets from the configured source namespace into each workspace namespace. In shared and operator modes, the Secrets must already exist in the sandbox namespace. | diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 29f095d6f8..63cc16730f 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -71,7 +71,7 @@ This provides defense-in-depth: even if a container escape vulnerability exists, | Aspect | Detail | |---|---| -| Default | Disabled. Set `server.enableUserNamespaces: true` in Helm values or `enable_user_namespaces = true` in the gateway config to enable cluster-wide. | +| Default | Disabled. Set `server.enableUserNamespaces: true` in Helm values or `enable_user_namespaces = true` in `[openshell.drivers.kubernetes]` to enable cluster-wide. | | What you can change | Enable cluster-wide through Helm or gateway config. Override per-sandbox through the `user_namespaces` field on `SandboxTemplate` in the API. | | Prerequisites | Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), a container runtime that supports user namespaces (containerd 2.0+, CRI-O 1.25+), and Linux 5.12+ for ID-mapped mounts. | | Risk if enabled with GPU | NVIDIA device plugin compatibility with user namespaces is unverified. OpenShell logs a warning when both GPU and user namespaces are active on the same sandbox. | diff --git a/e2e/configs/gateway/docker.toml b/e2e/configs/gateway/docker.toml index 59baed1d7d..878aee677c 100644 --- a/e2e/configs/gateway/docker.toml +++ b/e2e/configs/gateway/docker.toml @@ -7,7 +7,7 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:8080" log_level = "info" -compute_drivers = ["docker"] +compute_driver = "docker" disable_tls = true [openshell.gateway.auth] @@ -23,5 +23,5 @@ ttl_secs = 0 [openshell.drivers.docker] default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" image_pull_policy = "IfNotPresent" -sandbox_namespace = "openshell-e2e" +sandbox_label = "openshell-e2e" supervisor_image = "localhost/openshell/supervisor:e2e-vm" diff --git a/e2e/configs/gateway/podman.toml b/e2e/configs/gateway/podman.toml index c1549cd933..2064a081f5 100644 --- a/e2e/configs/gateway/podman.toml +++ b/e2e/configs/gateway/podman.toml @@ -7,7 +7,7 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:8080" log_level = "info" -compute_drivers = ["podman"] +compute_driver = "podman" disable_tls = true [openshell.gateway.auth] @@ -25,4 +25,5 @@ default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" image_pull_policy = "missing" network_name = "openshell-e2e" grpc_endpoint = "http://host.containers.internal:8080" +ssh_socket_path = "/run/openshell/ssh.sock" supervisor_image = "localhost/openshell/supervisor:e2e-vm" diff --git a/e2e/run.sh b/e2e/run.sh index 0505730f05..b904469730 100755 --- a/e2e/run.sh +++ b/e2e/run.sh @@ -142,7 +142,14 @@ if ! gateway_config="$(resolve_file "${gateway_config_source}")"; then fi gateway_driver="$(python3 -c ' import sys, tomllib -print(tomllib.load(open(sys.argv[1], "rb"))["openshell"]["gateway"]["compute_drivers"][0]) +gateway = tomllib.load(open(sys.argv[1], "rb"))["openshell"]["gateway"] +driver = gateway.get("compute_driver") +if driver is None: + drivers = gateway.get("compute_drivers", []) + driver = drivers[0] if drivers else None +if not driver: + raise SystemExit("gateway config must explicitly select a compute driver") +print(driver) ' "${gateway_config}")" if [[ ! ${suite_name} =~ ^[a-z0-9][a-z0-9-]*$ ]]; then die "suite name must contain only lowercase letters, digits, and hyphens: ${suite_name}" diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index 26671ccb86..2cdc0d27c0 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -241,7 +241,7 @@ version = 1 [openshell.gateway] bind_address = "127.0.0.1:${HOST_PORT}" -compute_drivers = ["vm"] +compute_driver = "vm" [openshell.gateway.tls] cert_path = "${PKI_DIR}/server/tls.crt" diff --git a/e2e/with-docker-gateway.sh b/e2e/with-docker-gateway.sh index 15e9d3466e..461c4ce30b 100755 --- a/e2e/with-docker-gateway.sh +++ b/e2e/with-docker-gateway.sh @@ -495,7 +495,7 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" fi fi printf '[openshell.drivers.docker]\n' - printf 'sandbox_namespace = %s\n' "$(toml_string "${E2E_NAMESPACE}")" + printf 'sandbox_label = %s\n' "$(toml_string "${E2E_NAMESPACE}")" printf 'network_name = %s\n' "$(toml_string "${DOCKER_NETWORK_NAME}")" printf 'grpc_endpoint = %s\n' "$(toml_string "${GATEWAY_ENDPOINT}")" printf 'default_image = %s\n' "$(toml_string "${SANDBOX_IMAGE}")" diff --git a/e2e/with-podman-gateway.sh b/e2e/with-podman-gateway.sh index cd52e007ab..6e5143e09b 100755 --- a/e2e/with-podman-gateway.sh +++ b/e2e/with-podman-gateway.sh @@ -427,7 +427,7 @@ GATEWAY_CONFIG="${STATE_DIR}/gateway.toml" # Start from the RPM default template so this e2e test exercises the same TOML # config path that RPM users get on first start. The template leaves -# bind_address unset and sets compute_drivers = ["podman"]. On Podman Machine, +# bind_address unset and sets compute_driver = "podman". On Podman Machine, # the driver reserves IPv4 loopback for its callback-only listener, so the # primary listener uses IPv6 loopback. Native Linux keeps the IPv4 default. # @@ -468,7 +468,7 @@ cp "${ROOT}/deploy/rpm/gateway.toml.default" "${GATEWAY_CONFIG}" GATEWAY_ARGS=( --config "${GATEWAY_CONFIG}" - # compute_drivers comes from the RPM template. Override the loopback address + # compute_driver comes from the RPM template. Override the loopback address # and port so Podman Machine can keep its IPv4 callback listener distinct. --bind-address "${PRIMARY_BIND_IP}" --port "${HOST_PORT}" diff --git a/examples/aws-s3-sts.md b/examples/aws-s3-sts.md index e622e71117..5a7acfe5f1 100644 --- a/examples/aws-s3-sts.md +++ b/examples/aws-s3-sts.md @@ -90,7 +90,7 @@ if your gateway cache directory differs): version = 1 [openshell.gateway] -compute_drivers = ["podman"] +compute_driver = "podman" default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" disable_tls = true supervisor_image = "localhost/openshell/supervisor:dev" diff --git a/rfc/0003-gateway-configuration/README.md b/rfc/0003-gateway-configuration/README.md index 2b7c095065..df71cc1f5b 100644 --- a/rfc/0003-gateway-configuration/README.md +++ b/rfc/0003-gateway-configuration/README.md @@ -72,10 +72,9 @@ metrics_bind_address = "0.0.0.0:9090" # optional; omit to disable # Logging log_level = "info" -# Compute drivers — list of driver names whose [openshell.drivers.] -# tables should be activated. When empty, the gateway auto-detects a driver -# (kubernetes → podman → docker). VM is never auto-detected. -compute_drivers = ["kubernetes"] +# Compute driver — exactly one driver may be active. When omitted, the gateway +# auto-detects a driver (kubernetes → podman → docker). VM is never auto-detected. +compute_driver = "kubernetes" # Note: database_url is a secret and must be supplied via OPENSHELL_DB_URL # (or --db-url) — it is NOT permitted in the file. @@ -113,7 +112,7 @@ scopes_claim = "" # empty disables scope enforcement # ────────────────────────────────────────────────────────────────────────────── # Compute drivers — each table is owned and parsed by its driver crate. -# Only tables for drivers listed in compute_drivers are activated. +# Only the selected or auto-detected driver's table is activated. # ────────────────────────────────────────────────────────────────────────────── [openshell.drivers.kubernetes] @@ -130,7 +129,7 @@ ssh_socket_path = "/run/openshell/ssh.sock" [openshell.drivers.docker] default_image = "ghcr.io/nvidia/openshell/sandbox:latest" image_pull_policy = "IfNotPresent" -sandbox_namespace = "docker-dev" +sandbox_label = "docker-dev" grpc_endpoint = "https://host.openshell.internal:8080" network_name = "openshell" supervisor_bin = "/usr/local/libexec/openshell/openshell-sandbox" # optional override @@ -172,7 +171,7 @@ Each `[openshell.drivers.]` table is extracted from the parsed file and ha Driver authors define and own their config schema. Adding a new driver does not require changes to the gateway's core `Config` struct or to this RFC. -`[openshell.drivers.]` tables for drivers not listed in `compute_drivers` (and not the auto-detected driver) are parsed for syntax but not activated. +`[openshell.drivers.]` tables for drivers other than the selected or auto-detected driver are parsed for syntax but not activated. ### Merge semantics @@ -209,11 +208,11 @@ The following cross-field validations are applied after merging file + env + CLI - `bind_address`, `health_bind_address`, and `metrics_bind_address` must all use distinct ports when set. - When `[openshell.gateway.tls]` is present, all three of `cert_path`, `key_path`, and `client_ca_path` must be present (either from the file or from CLI/env). Partial TLS configuration is an error. - `database_url` must be non-empty after merging env + CLI — every supported driver requires it. The field is not accepted from the file (see Secrets above). -- `compute_drivers` may be empty; in that case the gateway falls back to auto-detection. If the list contains a driver name with no matching `[openshell.drivers.]` table, the driver runs with its built-in defaults. +- `compute_driver` selects exactly one driver. When omitted, the gateway falls back to auto-detection. A custom driver name with no matching `[openshell.drivers.]` table runs with its built-in defaults. The legacy `compute_drivers` list remains accepted: an empty list auto-detects, a singleton selects that driver, and multiple entries retain the existing startup error. ### Backwards compatibility -The existing CLI interface is fully preserved. All flags continue to work exactly as before. The `--config` flag is new and additive. `OPENSHELL_DB_URL` remains a required process input (it is not accepted from the file). +The existing CLI interface is fully preserved. All flags continue to work exactly as before. The `--config` flag is new and additive. `OPENSHELL_DB_URL` remains a required process input (it is not accepted from the file). Legacy `compute_drivers = [""]` TOML remains accepted, while canonical configurations use the singular `compute_driver = ""`. ### Example: minimal Kubernetes deployment @@ -222,8 +221,8 @@ The existing CLI interface is fully preserved. All flags continue to work exactl version = 1 [openshell.gateway] -bind_address = "0.0.0.0:8080" -compute_drivers = ["kubernetes"] +bind_address = "0.0.0.0:8080" +compute_driver = "kubernetes" # database_url comes from env (e.g. valueFrom.secretKeyRef). # No [openshell.gateway.tls] → plaintext listener (gateway runs behind Envoy / ingress). @@ -250,7 +249,7 @@ gateway: bind_address: "0.0.0.0:8080" health_bind_address: "0.0.0.0:8081" metrics_bind_address: "0.0.0.0:9090" - compute_drivers: ["kubernetes"] + compute_driver: "kubernetes" drivers: kubernetes: namespace: agents @@ -298,5 +297,5 @@ No part of this RFC has shipped yet. The work breaks down as: 1. **Schema versioning** — the `version` field is reserved but not acted on. Should the parser reject files with `version > 1`, or just warn? Define this before the first stable release. 2. **Directory-based config (`conf.d` pattern)** — a `--config-dir` flag that globs all `*.toml` files in a directory, sorts them alphabetically, and deep-merges them in order (later files win per key). CLI/env overrides still sit above everything. This maps cleanly to Kubernetes: a base `ConfigMap` as `10-base.toml`, driver config as `20-kubernetes.toml`, and credentials from a projected `Secret` as `90-credentials.toml` — all mounted into the same directory without a monolithic file. This is the approach taken by cri-o and kubelet, inspired by systemd's `conf.d` convention. - Deferred to a follow-on: the single `--config` file is sufficient for v1, and the directory loader can be added without any schema changes. Before implementing, three design decisions must be settled: (a) whether `--config` and `--config-dir` are mutually exclusive or composable (and if so which takes lower precedence); (b) whether a later file's array value (e.g. `compute_drivers`) replaces or appends — replace is simpler and less surprising; (c) `deny_unknown_fields` validation must apply to the final merged result rather than each individual file, since partial drop-in files won't contain all sections. + Deferred to a follow-on: the single `--config` file is sufficient for v1, and the directory loader can be added without any schema changes. Before implementing, three design decisions must be settled: (a) whether `--config` and `--config-dir` are mutually exclusive or composable (and if so which takes lower precedence); (b) whether a later file's array value (for example `credential_drivers`) replaces or appends — replace is simpler and less surprising; (c) `deny_unknown_fields` validation must apply to the final merged result rather than each individual file, since partial drop-in files won't contain all sections. 3. **OIDC secret hygiene (revisit)** — `database_url` is excluded from the file schema (resolved). OIDC settings are allowed for v1 since the listed fields are identifiers, not credentials. If we add OIDC fields that *are* credentials in the future (e.g. a client secret for confidential-client flows), they should join the env-only list at that point. Re-evaluate once the OIDC surface stabilises. diff --git a/rfc/0011-multi-player-design/README.md b/rfc/0011-multi-player-design/README.md index 32a2f584e1..5fcb67fe51 100644 --- a/rfc/0011-multi-player-design/README.md +++ b/rfc/0011-multi-player-design/README.md @@ -755,13 +755,14 @@ use the workspace to select the target Kubernetes namespace instead of encoding it in the resource name. The label-based lookup and annotation patterns established here carry over unchanged. -**Docker and Podman drivers.** The Docker driver's `sandbox_namespace` label -provides a foundation for workspace mapping, but the driver currently uses a -single configured namespace rather than per-sandbox values. The driver contract -must be updated so that workspace flows through `DriverSandbox` and the driver -applies it as the container label filter. The same applies to Podman and other -local drivers — workspace isolation is enforced at the gateway level and does -not require Kubernetes. +**Docker and Podman drivers.** The Docker driver's `sandbox_label` +configuration value is stored in the `openshell.sandbox_namespace` container +label and provides a foundation for workspace mapping, but the driver currently +uses a single configured value rather than per-sandbox values. The driver +contract must be updated so that workspace flows through `DriverSandbox` and +the driver applies it as the container label filter. The same applies to Podman +and other local drivers — workspace isolation is enforced at the gateway level +and does not require Kubernetes. ### Compute Driver Trust Model diff --git a/tasks/scripts/gateway-docker.sh b/tasks/scripts/gateway-docker.sh index 895d5a62fc..8621e60af6 100644 --- a/tasks/scripts/gateway-docker.sh +++ b/tasks/scripts/gateway-docker.sh @@ -172,7 +172,7 @@ cat >"${CONFIG_PATH}" <"${CONFIG_PATH}" <"${CONFIG_PATH}" <