An air-gapped deployment is one where the machines running the models have no route to the internet. Not a firewall rule. No route. It is the strictest shape of sovereign deployment, and it is more common than people expect: defence, critical infrastructure, some ministries, some banks.
Running models in that setting is easy. Models do not need the internet to run. Updating them is where the work is, because every habit the AI ecosystem has, from pip install to from_pretrained to "the container pulls the weights on start", assumes a connection. This is how we do it without one.
What the inside cannot do
Write these down, because each one is a place a well-meaning engineer will try to punch a hole.
- No package index. No PyPI, no npm, no apt mirror unless you bring one.
- No container registry. No
docker pull. - No model hub. No
from_pretrained("org/model"), and no tokeniser or config fetched on first use. - No telemetry, no licence checks, no "call home to see if there is a newer version".
- No time sync from the outside, which matters more than it sounds for certificate checks.
The bundle
Everything a release needs crosses the gap once, as one artefact, with a manifest. Ours looks like this.
| Item | Format | Why it is there |
|---|---|---|
| Runtime image | OCI image, saved as a tarball | The serving stack, pinned, with every dependency baked in |
| Weights | Safetensors, sharded, with checksums | The model itself; never inside the image, so it can be updated alone |
| Tokeniser and config | Files alongside the weights | So nothing is fetched at load time |
| Eval report | JSON plus a human summary | Proof the candidate passed the gate before it left our side |
| SBOM | SPDX or CycloneDX | So the receiving security team can review what is inside |
| Signatures | Sigstore bundle, offline key | So the inside can verify the bundle without us or the internet |
| Runbook | Markdown | Install, verify, roll out, roll back, in that order |
The image never contains the weights. Images change rarely; weights change with every model version. Keeping them separate makes the common update small and the review focused.
Build and sign, on the outside
The build script produces the bundle and signs it with a key whose public half was installed inside the network when the deployment was first set up. That first exchange happens in person. After that, every update can be verified inside without any further trust in the transport.
#!/usr/bin/env bash
set -euo pipefail
VERSION="$1" # e.g. 2026.05.30-tooluse-v7
OUT="bundle-${VERSION}"
mkdir -p "$OUT"
# 1. Runtime image, pinned by digest, saved as a tarball.
docker pull "registry.example.com/serve@${IMAGE_DIGEST}"
docker save "registry.example.com/serve@${IMAGE_DIGEST}" -o "$OUT/serve.tar"
# 2. Weights, tokeniser, config: everything the loader touches.
cp -r "/models/${VERSION}/" "$OUT/model/"
# 3. Evidence: the eval report this exact version passed.
cp "evals/reports/${VERSION}.json" "$OUT/eval-report.json"
syft "registry.example.com/serve@${IMAGE_DIGEST}" -o spdx-json > "$OUT/sbom.spdx.json"
# 4. Checksums over every file, then a signature over the checksum file.
(cd "$OUT" && find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 sha256sum > SHA256SUMS)
cosign sign-blob --key "$SIGNING_KEY" --bundle "$OUT/SHA256SUMS.sigstore" "$OUT/SHA256SUMS"
tar -cf "${OUT}.tar" "$OUT"
sha256sum "${OUT}.tar" > "${OUT}.tar.sha256"
echo "bundle ready: ${OUT}.tar"The bundle then moves across the gap by whatever the site allows: approved removable media, a data diode, a one-way transfer appliance. That step belongs to the client's security team, and the runbook says so.
Verify, on the inside
Nothing is loaded until the bundle verifies. The verification is a script, not a checklist, so it is run the same way every time.
#!/usr/bin/env bash
set -euo pipefail
BUNDLE="$1"
sha256sum -c "${BUNDLE}.sha256" # transport integrity
tar -xf "$BUNDLE" && cd "${BUNDLE%.tar}"
cosign verify-blob --key /etc/zingaro/release.pub \
--bundle SHA256SUMS.sigstore SHA256SUMS # authenticity, offline
sha256sum -c SHA256SUMS # every file, unchanged
jq -e '.gate.passed == true' eval-report.json > /dev/null # it passed the gate
docker load -i serve.tar
echo "verified: $(basename "$PWD")"If any line fails, the update stops and the previous version keeps running. There is no "verify later".
Roll out in stages
Air-gapped does not mean brave. The new version goes through the same stages as anywhere else, just with the bundle as the source of truth.
- Shadow. The new model receives a copy of live traffic and its outputs are recorded, not used. Compare against the running version for a day.
- Canary. A small share of real traffic, with the review queue watching the exceptions closely.
- Full. The rest, and the previous version stays installed.
Two versions are always present on disk. Rollback is a symlink change and a restart, and it is rehearsed during the install, not discovered during an incident.
Logs stay inside
Nothing in the runtime image is allowed to open an outbound connection, and the network confirms it, but the discipline matters even so: no crash reporters, no usage pings, no "anonymous statistics". Logs go to the client's own collector. When we need to debug something, the client exports what they choose to export, reviewed, across the gap in the other direction.
Things that go wrong
- Lazy fetches. A library that quietly downloads a tokeniser, a config or a small auxiliary model on first use. Every load path is exercised on a disconnected test machine before the bundle is built, and the error you want is a loud one.
- Cached credentials and hub tokens left in an image. The SBOM review catches some of this; a grep of the image layers catches the rest.
- Clock drift. Certificate and signature checks depend on time. An air-gapped network needs its own time source, and the runbook checks it first.
- Bundle size. Weights are large and removable media is slow. Shipping a weight delta helps when only an adapter changed; a full bundle is the default because it is simpler to verify.
- Version confusion. The running version must be printed on every dashboard and in every log line. When there is no internet, there is also no "check the registry".
None of this is exotic. It is the ordinary discipline of releasing software, applied with no shortcuts available. That constraint turns out to be clarifying: the process that works with no connection is the process that should have been running all along.