Confidential Inferencing

Experimenting with encrypted model delivery into a confidential inferencing environment.

Approach

Use an init container to copy the model image from a registry into ramdisk inside the confidential container.

Steps

1. Run make kind to start a kind cluster
2. Run make init-container to build the init-container image
3. Run make run to run the inference pod

Usage

% make
fetch                Fetch image from Huggingface
oci                  Build OCI artifact for model
push                 Push OCI artifact to registry
gen-keys             Generate crypto keypair
encrypt              Encrypt the model (registry -> registry)
decrypt              Decrypt the modek (registry -> registry)
pull                 Pull and extract model from registry
kind                 Start a kind cluster
init-container       Build the init container
run                  Run the inferencing pod
clean                Clean everything up, also deleting the kind cluster
configure-machine    Configure podman machine to use insecure registry
help                 This help

Note: when using a podman machine it needs space for an 8GB ramdisk in addition to the kubernetes pods.

podman machine init --cpus 8 --memory 16384

Model packaging as OCI artifact

Fetch model and package as OCI

huggingface-cli download Qwen/Qwen3-0.6B --local-dir model

modctl modelfile generate model
modctl build -t tosh.lan:5000/qwen/qwen3-0.6b:latest -f Modelfile model

Extract model from modctl cache dir

rm -rf staging
mkdir staging
REGDIR=~/.modctl/content.v1/docker/registry/v2
REPODIR=${REGDIR}/repositories/tosh.lan:5000/qwen/qwen3-0.6b
BLOBDIR=${REGDIR}/blobs/sha256
TAGFILE=${REPODIR}/_manifests/tags/latest/current/link
MID=$(cat $TAGFILE)
MANIFEST=${BLOBDIR}/${MID:7:2}/${MID:7}/data
cp ${MANIFEST} staging/manifest.json
for DIGEST in `jq -M -r '.layers[] | .digest' ${MANIFEST}`
do
    cp ${BLOBDIR}/${DIGEST:7:2}/${DIGEST:7}/data staging/${DIGEST:7}
done
CONFIG=`jq -r .config.digest ${MANIFEST}`
cp ${BLOBDIR}/${CONFIG:7:2}/${CONFIG:7}/data staging/${CONFIG:7}

Encrypt model with skopeo

openssl genpkey -algorithm RSA -out private.pem -pkeyopt rsa_keygen_bits:4096
openssl rsa -pubout -in private.pem -out public.pem

skopeo copy --encryption-key jwe:public.pem \
	dir:staging \
	docker://tosh.lan:5000/qwen/qwen3-0.6b:encrypted

Decrypt model with skopeo

skopeo copy --decryption-key private.pem \
	docker://tosh.lan:5000/qwen/qwen3-0.6b:encrypted \
	dir:decrypted

Unpack OCI into usable model

mkdir ramdisk
for layer in `jq -r '.layers[] | .digest' decrypted/manifest.json`
do
    tar xvf decrypted/${layer:7} -C ramdisk
done

Serve model with vLLM

vllm serve /ramdisk