Skip to main content
Version: Weekly Build

Confidential virtual clusters

note

Not every Codesphere instance offers confidential compute. Ask your Codesphere administrator whether confidential compute is available on your instance. See Prerequisites.

For confidential workspaces and managed services, Codesphere protects your data in use, in transit and at rest automatically. A Virtual Cluster has no confidential switch. You apply each protection yourself, per pod. This page shows how to apply and verify each one.

warning

Codesphere does not enforce confidential mode inside a Virtual Cluster. Every pod without the annotations below runs in regular mode, next to your confidential pods. Check every pod template of your manifests and Helm charts, including the ones of dependencies.

Prerequisites​

In use: run pods in a micro-VM​

Add the following annotations to the pod. For a Deployment, StatefulSet, Job or Helm chart, add them to the pod template.

AnnotationValuePurpose
dyneemes.enclaive.io/runtimeThe confidential RuntimeClass of your instance, for example dyneemes-qemu-sev-snp. Run kubectl get runtimeclass to see it.Runs the pod in its own confidential micro-VM on a confidential compute node.
nitride.enclaive.io/vhsmThe address of your trustee service.The micro-VM proves its integrity to this trustee service when it boots.
nitride.enclaive.io/workloadYour workload ID.Selects the trustee policy of your workload, which decides which secrets the pod receives.
nitride.enclaive.io/providersev-snp-rawThe attestation type of the hardware.
nitride.enclaive.io/mountnitrideWhere the trustee service accepts attestations.

The following pod runs in a micro-VM:

apiVersion: v1
kind: Pod
metadata:
name: confidential-alpine
annotations:
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: alpine:3.22
command: ["sleep", "infinity"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 256Mi

Pods in a Virtual Cluster must meet the restricted Pod Security Standard, confidential ones included. The security context in the example meets it.

The platform schedules the pod onto a confidential compute node based on the dyneemes.enclaive.io/runtime annotation. Do not add node selectors or tolerations for confidential nodes yourself.

warning

The micro-VM needs resources on top of the limits of your containers. The platform adds a fixed overhead to every confidential pod, for example 1 CPU and 2Gi of memory. This overhead counts against the resource quota of your Virtual Cluster, but it does not show up inside your Virtual Cluster. Ask your Codesphere administrator for the overhead on your instance.

Images can come from public or private registries. The micro-VM pulls the image itself, so the host cannot tamper with it before starting the container. Reference images by digest.

warning

The pull credentials of a private registry are not confidential. You store them in a regular Kubernetes Secret, referenced by imagePullSecrets, and the platform can read them. Use credentials that only grant read access to the images you need.

Verify​

Check that the pod is running:

kubectl get pod confidential-alpine
# NAME READY STATUS RESTARTS AGE
# confidential-alpine 1/1 Running 0 30s

A pod in the Running state has passed attestation and runs your application inside the micro-VM. If attestation fails, the pod does not start.

Then check the RuntimeClass that the platform set on the pod. It is the confidential RuntimeClass from the dyneemes.enclaive.io/runtime annotation:

kubectl get pod confidential-alpine -o jsonpath='{.spec.runtimeClassName}'
# dyneemes-qemu-sev-snp

In transit: encrypt traffic between pods​

Confidential pods in a Virtual Cluster can join the service mesh. The mesh encrypts the traffic between them with mutual TLS and only accepts peers that passed attestation.

Add the following annotations to the pod, next to the ones from In use:

AnnotationValuePurpose
linkerd.io/injectenabledAdds the mesh proxy to the pod.
config.linkerd.io/default-inbound-policyall-authenticatedAccepts incoming connections only from pods in the mesh.
config.linkerd.io/default-outbound-policyall-authenticatedSends outgoing traffic to other pods only through the mesh.
config.linkerd.io/skip-outbound-ports"443,8200,8443"Keeps the connections of the micro-VM to the trustee service and the Kubernetes API out of the mesh.

Your application does not need to change. It connects to other services as usual, and the mesh encrypts the connection.

Allow external traffic​

With all-authenticated, a pod only accepts connections from other pods in the mesh. A pod that you make reachable from outside the Virtual Cluster through a load balancer must accept connections from outside the mesh. For such a pod, set the inbound policy to all-unauthenticated, next to the label from Allowing load balancer ingress:

metadata:
labels:
codesphere.com/allow-ingress-traffic: "true"
annotations:
config.linkerd.io/default-inbound-policy: all-unauthenticated

Keep all-authenticated for every pod that only needs traffic from inside the mesh.

Traffic to workspaces and managed services​

Confidential pods can also talk to workspaces and managed services outside the Virtual Cluster through the mesh. This only works if both sides are confidential. A confidential pod cannot reach a regular workspace or managed service, and a regular one cannot reach a confidential pod.

Example: server and client​

The following manifest runs a server pod with a Service, and a client pod that calls the server every 5 seconds. Both pods are confidential and in the mesh.

apiVersion: v1
kind: Pod
metadata:
name: mesh-server
labels:
app: mesh-server
annotations:
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
linkerd.io/inject: enabled
config.linkerd.io/default-inbound-policy: all-authenticated
config.linkerd.io/default-outbound-policy: all-authenticated
config.linkerd.io/skip-outbound-ports: "443,8200,8443"
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "mkdir -p /tmp/www && echo hello-from-mesh-server > /tmp/www/index.html && httpd -f -p 8080 -h /tmp/www"]
ports:
- containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
name: mesh-server
spec:
selector:
app: mesh-server
ports:
- port: 80
targetPort: 8080
---
apiVersion: v1
kind: Pod
metadata:
name: mesh-client
annotations:
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
linkerd.io/inject: enabled
config.linkerd.io/default-inbound-policy: all-authenticated
config.linkerd.io/default-outbound-policy: all-authenticated
config.linkerd.io/skip-outbound-ports: "443,8200,8443"
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "while true; do wget -qO- -T 5 http://mesh-server/ || echo FAILED; sleep 5; done"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 128Mi

Verify​

The client reaches the server through the mesh:

kubectl logs mesh-client
# hello-from-mesh-server

Now remove the linkerd.io/inject annotation from the client and recreate it. The client is no longer in the mesh, so the server refuses its requests:

kubectl logs mesh-client
# wget: server returned error: HTTP/1.1 403 Forbidden
# FAILED

At rest: encrypt persistent volumes​

Use the codesphere-rbd-confidential StorageClass for persistent volumes of confidential pods. The micro-VM encrypts the volume itself, so the node and the storage backend only see encrypted data.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: confidential-data
spec:
storageClassName: codesphere-rbd-confidential
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: confidential-storage
annotations:
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: alpine:3.22
command: ["sh", "-c", "df -h /data && cat /sys/block/dm-*/dm/uuid && sleep infinity"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 256Mi
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: confidential-data

Confidential volumes only support the ReadWriteOnce access mode. Each volume is unlocked inside a single micro-VM, so pods cannot share it.

Verify​

The example pod prints the mounted file system and the identifier of its device-mapper device:

kubectl logs confidential-storage
# Filesystem Size Used Available Use% Mounted on
# /dev/mapper/pvc-<id> 973.7M 276.0K 957.4M 0% /data
# CRYPT-LUKS2-<uuid>-pvc-<id>

The volume is mounted from /dev/mapper/pvc-<id> instead of a plain disk. The CRYPT-LUKS2- prefix of its identifier shows that it is a LUKS2 encrypted device, unlocked inside the micro-VM.

Inject application secrets​

After the micro-VM passed attestation, it reads secrets from the vault of your trustee service and hands them to a container. A container can receive secrets as environment variables, as files, or both. The secrets never pass through the platform, so they don't show up in the pod spec.

warning

Container logs leave the micro-VM and pass through the platform. Never log secret values.

Templates​

You define what a container receives with templates. Each template is an annotation dyneemes.enclaive.io/template-<name>, where <name> is a name of your choice. The annotations for environment variables and files reference templates by this name. Several containers of a pod can use the same template.

Templates use Go template syntax and the following functions:

  • buckypaperWorkload "app-secrets/<key>" "dynamic=env" reads the secret <key> from the trustee service vault of your workload, at buckypaper/<workloadId>/app-secrets/<key>. Its .value is the secret value.
  • env "<NAME>" <value> sets the environment variable <NAME>. Only use it in templates for environment variables.
  • printf combines several secrets and plain text into one value. Write a literal % as %%.

The "dynamic=env" argument tells the trustee service to generate a random value when the secret does not exist yet. Without it, a missing secret stops the container from starting. See the Buckypaper secrets engine for details. To set your own values instead, see Manage secrets in the trustee service vault.

The container receives its secrets once, when it starts. To pick up a changed secret, recreate the pod.

As environment variables​

The annotation dyneemes.enclaive.io/environment-<container> names the template that sets the environment variables of <container>.

In the simplest case, each secret becomes one environment variable. This template sets DB_USER and DB_PASSWORD in the container app:

metadata:
annotations:
dyneemes.enclaive.io/environment-app: app-env
dyneemes.enclaive.io/template-app-env: |
{{ env "DB_USER" (buckypaperWorkload "app-secrets/DB_USER" "dynamic=env").value }}
{{ env "DB_PASSWORD" (buckypaperWorkload "app-secrets/DB_PASSWORD" "dynamic=env").value }}

A template can also combine several secrets into one value. The following pod extends the previous example and the pod from In use. It builds DATABASE_URL from the secrets DB_USER and DB_PASSWORD in the trustee service vault, at buckypaper/<workloadId>/app-secrets/DB_USER and buckypaper/<workloadId>/app-secrets/DB_PASSWORD. At startup, it prints the value with the password masked.

apiVersion: v1
kind: Pod
metadata:
name: confidential-secrets
annotations:
dyneemes.enclaive.io/environment-app: app-env
dyneemes.enclaive.io/template-app-env: |
{{- $user := (buckypaperWorkload "app-secrets/DB_USER" "dynamic=env").value -}}
{{- $password := (buckypaperWorkload "app-secrets/DB_PASSWORD" "dynamic=env").value -}}
{{ env "DATABASE_URL" (printf "postgres://%s:%s@db:5432/app" $user $password) }}
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: alpine:3.22
command: ["sh", "-c", "echo \"DATABASE_URL=$DATABASE_URL\" | sed 's/:[^:@]*@/:***@/' && sleep infinity"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 256Mi

The injected environment variables are only set for the process that the container starts with its command, and for the child processes of that process. They are not set for the container in general. For example, a shell opened with kubectl exec does not see them:

kubectl exec confidential-secrets -- printenv DATABASE_URL
# (no output)

Verify​

The pod prints the composed value with the password masked:

kubectl logs confidential-secrets
# DATABASE_URL=postgres://<DB_USER>:***@db:5432/app

As files​

The annotation dyneemes.enclaive.io/injector-<container> maps file paths in <container> to templates, as a JSON object. The micro-VM writes the output of each template to its file.

Each file must be inside a volume that meets the following rules:

  • The volume is an emptyDir with medium: Memory, so secrets never touch a disk.
  • The volume is mounted in the container, and the file path is inside its mount path. Paths with ../ are not allowed.
  • Each container has its own volume. Containers cannot share a volume for secret files.

The following pod writes the secrets DB_USER and DB_PASSWORD into the file /secrets/db.properties of the container app. At startup, it lists the files in /secrets.

apiVersion: v1
kind: Pod
metadata:
name: confidential-secret-files
annotations:
dyneemes.enclaive.io/injector-app: |
{ "/secrets/db.properties": "db-properties" }
dyneemes.enclaive.io/template-db-properties: |
user={{ (buckypaperWorkload "app-secrets/DB_USER" "dynamic=env").value }}
password={{ (buckypaperWorkload "app-secrets/DB_PASSWORD" "dynamic=env").value }}
dyneemes.enclaive.io/runtime: dyneemes-qemu-sev-snp
nitride.enclaive.io/provider: sev-snp-raw
nitride.enclaive.io/mount: nitride
nitride.enclaive.io/vhsm: https://<trustee-service-address>/
nitride.enclaive.io/workload: <workload-id>
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: alpine:3.22
command: ["sh", "-c", "ls /secrets && sleep infinity"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: 250m
memory: 256Mi
volumeMounts:
- name: secrets
mountPath: /secrets
volumes:
- name: secrets
emptyDir:
medium: Memory

A memory-backed volume counts against the memory limit of the container. Keep secret files small.

Verify​

The pod lists the injected file:

kubectl logs confidential-secret-files
# db.properties

Secrets that pods create​

Some pods write Kubernetes Secrets themselves, for example operators that generate credentials. When a confidential pod writes a Secret, the Secret is encrypted before it is sent to the Kubernetes API server. kubectl get secret only shows the encrypted values.

Confidential pods of the same workload use these Secrets as usual. They can mount them as a volume, set them as environment variables, or read them through the Kubernetes API.

warning

A Secret that you create yourself, for example with kubectl create secret or kubectl apply, is stored in plaintext and the platform can read it. Never store confidential values this way. Always inject secrets from outside the Virtual Cluster through the trustee service, as described in Inject application secrets.

The annotations of all sections combine. One pod can join the mesh, use a confidential volume and get injected secrets at the same time.

Troubleshooting​

Run kubectl describe pod <pod> to see why a pod does not start. Events from the node, such as FailedCreatePodSandBox, and sync errors (SyncError) show up in your Virtual Cluster. Events of the storage backend for a PersistentVolumeClaim do not.

IssueRoot CauseSolution
Pod is stuck in ContainerCreating with vhsm-addr is requiredThe nitride.enclaive.io/* annotations are missingAdd all annotations listed in In use: run pods in a micro-VM
Pod is stuck in ContainerCreating with container creation requires successful attestationThe pod failed attestation, for example because the workload ID is wrongCheck the nitride.enclaive.io/workload and nitride.enclaive.io/vhsm annotations. If they are correct, ask your trustee provider to check the trustee policy of your workload
Pod is rejected with requests an unsupported Dyneemes runtimeThe value of the dyneemes.enclaive.io/runtime annotation is wrong, or the instance does not offer confidential computeCheck the name with kubectl get runtimeclass, or ask your Codesphere administrator
Pod stays Pending with a SyncError event exceeded quota: vcluster-resource-quotaThe container limits plus the confidential overhead exceed the resource quota of your Virtual Cluster. The requested value in the event includes the overheadFree up quota or reduce the limits of other pods. See In use: run pods in a micro-VM
PersistentVolumeClaim stays Pending without eventsThe claim requests ReadWriteMany or ReadOnlyMany. The storage backend rejects it, but the error is not shown inside your Virtual ClusterUse ReadWriteOnce. See At rest: encrypt persistent volumes
Pod of a StatefulSet fails with RunContainerErrorThe Rewrite Hosts option of your Virtual Cluster is enabled. It changes the pods of StatefulSets in a way that confidential pods do not supportDisable the Rewrite Hosts option of your Virtual Cluster
Pod is rejected by the Pod Security StandardThe image or pod runs as root or requests privilegesUse a non-root image and the security context from In use: run pods in a micro-VM

Limitations​

  • Confidential mode is opt-in per pod and not enforced. See the warning at the top of this page.
  • Pull credentials of private registries are not confidential.
  • Secrets that you create yourself are stored in plaintext. See Secrets that pods create.
  • For limitations of confidential services in general, see Limitations and gotchas.