Helm Validation Policy for BYOC clusters

[estroz]

Is this related to a problem? Please describe.

Helm ReVal renders and validates customer Helm charts before they are deployed to BYOC
clusters as NVCF functions or NVCT tasks. Today every cluster gets the same fixed
validation policy, and ReVal has no information about which cluster a chart is destined
for. This causes two problems:

• Clusters with different security postures (single-tenant vs multi-tenant workloads)
  cannot get different levels of Helm chart restriction.
• Clusters that run operators providing extra CRDs (for example, the LeaderWorkerSet
  API used by NIM Factory) cannot accept Helm charts that reference those custom types,
  because ReVal rejects unknown Kubernetes types.

NVIDIA Cloud Partners with complex function/task use cases and specialized Helm charts
are blocked by this today.

Describe the solution you'd like

Introduce a per-cluster "Helm validation policy" made up of:

• a name field, one of Default (current ReVal validation rules apply) or
  Unrestricted (only image validation is performed, no template validation), and
• an allowedExtraKubernetesTypes list (group/version/kind/resource) describing
  additional CRD-backed types the cluster's charts may include.

Clusters register with a policy at creation time (immutable after registration, since
loosening it later could be a security or compatibility risk). Users deploying a
function or task can request required extra types and/or a policy name as cluster
targeting fields; only clusters matching are selected, and Unrestricted clusters are
only selected when explicitly requested. The Helm ReVal service accepts a list of
policies to evaluate in a single request (instead of one request per cluster/policy) so
network- and validation-bound work is done once, and returns per-policy validity plus
errors. On the compute plane, the NVCA Operator receives the cluster's policy, updates
the miniservice controller's RBAC (Role) to allow the extra types, and passes the policy
to NVCA via a ConfigMap so installed CRD/policy mismatches can be detected at NVCA
startup instead of failing mid-deployment.

Default behavior for all existing clusters remains unchanged (Default policy, no
extra types) so this is fully backward compatible and opt-in.

Describe alternatives you've considered

• Per-request Helm chart validation caching instead of single-policy-list requests to
  Helm ReVal: rejected for now because it adds complexity and requires security review
  since customer data may include registry credentials; batching multiple policies into
  one ReVal request avoids duplicate chart downloads without caching.
• Automatically granting operators all cluster-allowed types via the cluster policy
  itself: rejected because Kubernetes Roles must be created at operator install/upgrade
  time, so allowedExtraKubernetesTypes must be explicit operator Helm chart
  configuration rather than derived automatically.
• Account-level (rather than cluster-level) validation policies: out of scope for this
  effort, tracked separately.

Additional context

Implementation has already started based on an internal design document and is at
different stages of completion per service. TODO list below, organized by service,
with completed items checked off based on the current state of this repository.

Helm ReVal

• Extend /v1/validate to accept a validationPolicies list and return per-policy
  valid/invalid + errors, keyed by request-supplied UUID
• Extend /v1/render to accept a single validationPolicy
• Skip validation (pass through as-is) for objects matching an allowed extra
  Kubernetes type
• Deploy Helm ReVal as a standalone control-plane service for Self-Hosted, with
  JWT-based authn/z via the control-plane auth provider
• Unified caching for chart render/validation across app pods (latency mitigation
  for ICMS instance scheduling path)

ICMS (formerly SIS)

• Accept helmValidationPolicy on cluster registration/update endpoints; persist as
  helm_validation_policy_name + allowed_extra_kubernetes_types in the cluster
  Cassandra tables; default all clusters to Default with no extra types
• Return helmValidationPolicy on all endpoints that return cluster data
• Reject cluster policy name changes after registration; only allow
  allowedExtraKubernetesTypes updates
• Move the Helm ReVal /v1/validate call from NVCF/NVCT into ICMS, called once per
  request with the deduplicated set of policies across filtered clusters
• Filter clusters using LaunchSpecification.HelmValidationPolicy (required types
  must be a subset of the cluster's allowed types; only select non-Default
  clusters when explicitly requested)
• Add a warnings response field for clusters filtered out due to policy mismatch
• Add the helmreval:validate token scope to the ICMS OIDC client

NVCF (cloud-functions)

• Accept helmValidationPolicy (name + extraKubernetesTypes) on the function
  deployment specification, reject it for container-based functions
• Base64-encode and pass helmValidationPolicy to ICMS via the launch specification

NVCT (cloud-tasks)

• Accept helmValidationPolicy (name + extraKubernetesTypes) on the task GPU
  specification, reject it for container-based tasks
• Base64-encode and pass helmValidationPolicy to ICMS via the launch specification

NVCA / NVCA Operator

• Add allowedExtraKubernetesTypes to NVCA Operator Helm values; validate against
  the configured cluster policy
• Add allowed extra types to the nvca Role and to the mini-service-restrictions
  Role embedded in the nvca-miniservice-rbac ConfigMap
• Store the cluster policy on the NVCFBackend object
  (spec.clusterConfig.helmValidationPolicy)
• Pass the policy to NVCA via the existing agent-config ConfigMap (config.yaml
  key), triggering an NVCA rollout on change - reuses the existing agent config
  delivery mechanism rather than a new ConfigMap
• On NVCA startup, verify each allowedExtraKubernetesTypes API resource is
  present on the cluster; report unhealthy and exit if missing (CRD/policy mismatch
  prevention)

CLI (nvcf-cli)

• Add a discrete --validation-policy flag to nvcf-cli task create
  (cmd/task.go, internal/client/tasks.go); today helmValidationPolicy is only
  reachable via the --input-file JSON payload, not a CLI flag
• Add a discrete --validation-policy flag to nvcf-cli deploy create /
  deploy update (cmd/deploy.go) and the corresponding deployment-spec client DTO

Security

• Default policy for all clusters remains Default with no allowed extra types;
  existing deployments are unaffected; opt-in required from both cluster
  owners/operators (to loosen a cluster) and workload deployers (to target an
  Unrestricted cluster)

---

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.