Airgapped Execution¶

The docker-airgapped runner executes workflows you do not trust — above all, model-authored (dynamic) workflows — and workflows over data that must not leave the box. Each execution runs in its own container whose only capability channel is the stdio protocol to the parent worker: checkpoints, secrets, configs, and approval-gate operations all flow through the worker, where every request is permission-checked. Everything else is closed.

The locked profile¶

Surface	Enforcement
Network	`--network=none` — no egress, no DNS, no `pip install`
Filesystem	`--read-only` rootfs; size-capped tmpfs `/tmp`
Privileges	`--cap-drop=ALL`, `--security-opt=no-new-privileges`
Resources	memory / cpus / pids limits (required, never unlimited)
Time	wall-clock ceiling; on expiry the container is killed and the execution fails terminally for both durabilities
Credentials	none — the child holds no worker or fleet credentials
Core dumps	on Linux the child entrypoint makes itself undumpable (`PR_SET_DUMPABLE=0`), and the profile pins the opt-out closed

The profile is emitted from code after any operator extra_args, so docker's last-wins flag parsing keeps it authoritative; extra_args that would re-open a closed surface are rejected at worker startup.

The core-dump surface deserves a word, because no container flag covers it: when the host's kernel.core_pattern is a pipe (apport on Ubuntu, systemd-coredump on most other distros — i.e. the typical default), a crashing process's entire address space is handed to a collector running on the host, outside the read-only rootfs, the dropped capabilities, and even RLIMIT_CORE (which core(5) documents as ignored for piped dumps). Every runner child therefore disables dumpability in-kernel at startup on Linux — where both the exposure and prctl exist. The call is skipped on other platforms, so a subprocess runner on macOS or Windows is unaffected either way.

Setting FLUX_CHILD_ALLOW_COREDUMP=1 in the child's environment re-enables dumps for debugging on the subprocess and plain docker runners; the airgapped profile forces the variable empty after operator extra_args, so the escape hatch does not exist on this runner.

Enabling the runner¶

[flux.workers]
runners = ["docker-airgapped"]
airgapped_image = "edurdias/flux:<version>-slim"  # falls back to docker_image

airgapped_memory = "512m"          # required non-empty
airgapped_cpus = 1.0
airgapped_pids_limit = 256
airgapped_tmp_size = "64m"
airgapped_execution_timeout = 900  # seconds; 0 disables (discouraged)

The image contract is minimal: it must run python -m flux.runners.child, i.e. have flux-core installed at a worker-compatible version. The official image satisfies this when its tag matches the worker (see DOCKER.md).

Pin workflows to the sealed runner with @workflow.with_options(runner="docker-airgapped") — this also constrains dispatch, so the workflow only reaches workers advertising the runner:

@workflow.with_options(runner="docker-airgapped")
async def sealed_keyword_count(ctx: ExecutionContext[str]):
    ...

See examples/airgapped.py for runnable examples (sealed text processing with read-only asset mounts and graceful fallback, and privacy-preserving redaction of sensitive data).

Capability knobs¶

Capabilities are granted only through named config keys — the raw flags (--gpus, --shm-size, and all mount flags) are rejected in airgapped_extra_args — so a grep of flux.toml for airgapped_ is the complete audit trail of opened surfaces:

Key	Grants	Why it's safe to grant
`airgapped_gpus`	`--gpus all` / `"device=0"`	a compute device; no data path out
`airgapped_mounts`	read-only bind mounts	an input channel — data can enter, results still leave only via the stdio protocol
`airgapped_shm_size`	`/dev/shm` sizing	RAM allocation (large inter-process buffers); accounted next to `airgapped_memory`
`airgapped_service_sockets`	UDS access to local sidecars	point-to-point to loopback-bound trusted infrastructure; no network stack, no lateral channel, no egress

airgapped_mounts entries are "/host/path:/container/path". Read-only is forced by the runner regardless of what the entry says; rw, relative paths, missing host paths, and duplicate targets fail worker startup. Mounted content is readable by every airgapped workflow on the worker — mount reference datasets and static assets, never directories containing secrets.

Capability channels are never knobs: network, DNS, published ports, privileges, host namespaces, and writable mounts cannot be enabled on this runner. Operators who need them switch to the plain docker runner — changing the runner name, and therefore the guarantee that dispatch and the dynamic-workflows server rely on.

Per-workflow narrowing: `runner_options`¶

A workflow can ask for a container narrower than the worker's configuration, never wider:

@workflow.with_options(runner="docker", runner_options={"cpus": 0.5, "memory": "256m"})
async def small_job(ctx: ExecutionContext):
    ...

Option	Effect
`cpus`, `memory`, `timeout`	applied only if lower than the worker's configured value
`network`	only `"none"` — dropping the network, never naming one
`read_only`	only `True`

Widening is not rejected at dispatch, it simply does not happen: the runner takes the stricter of the configured value and the requested one, so {"cpus": 64} on a worker configured for 1 CPU still yields 1. Options travel with the workflow, so their values are author-controlled — that guarantee is what makes accepting them safe.

Unknown keys, loosening values (network: "host", read_only: False), and non-positive numbers fail at registration, so a typo is an error rather than a setting that silently does nothing.

Only container runners accept them (docker, docker-airgapped). The subprocess runner ignores a request entirely and keeps its configured execution_timeout.

On this runner the locked profile is still emitted last, so nothing here can reopen a surface the profile closed — --network=none, --cap-drop=ALL, --read-only and the rest hold regardless of what was asked for. The profile's cpus/memory limits are the one part that merges: a narrower request replaces them, a wider one does not.

Service sockets — warm runtimes for sealed executions¶

A runtime whose startup is expensive (state loaded into memory over minutes) cannot live inside a single-use container. airgapped_service_sockets lets it live outside as a long-lived, operator-managed sidecar on the worker host, reached over a Unix domain socket — no network stack anywhere, point-to-point by construction:

[flux.workers]
runners = ["docker-airgapped"]
airgapped_service_sockets = { inference = "/run/flux-services/inference" }

Each value is a host directory containing the service's socket at <dir>/service.sock; it is bind-mounted into containers at /run/flux/services/<name>. The directory contract keeps the mount write-proof: mode 0555 (validated at worker startup; created if absent), sockets 0666. The mount is rw — connecting to a UDS requires write permission on the socket inode — but --cap-drop=ALL strips CAP_DAC_OVERRIDE, so the write-less directory binds every container uid, root included: executions can connect, never create files. A missing socket at startup is only a warning (the sidecar may start later); a down sidecar surfaces to workflows as a normal connect error.

Granted services are advertised as flux.service.<name> worker labels (the flux. label prefix is reserved — user labels can't spoof grants), so workflows target service-bearing workers with the existing affinity mechanism. In workflow code:

from flux.tasks import service_client

@task
async def generate(prompt: str) -> dict:
    async with service_client("inference") as client:   # httpx over UDS
        response = await client.post("/v1/completions", json={"prompt": prompt})
        return response.json()

@workflow.with_options(
    runner="docker-airgapped",
    affinity={"flux.service.inference": "true"},
)
async def sealed_generate(ctx: ExecutionContext[str]):
    return await generate(ctx.input)

Streaming is native HTTP over the socket (SSE/chunked responses work as-is; tee tokens into progress() for live visibility), and because service calls live inside ordinary tasks, replay short-circuits from the event log without re-contacting the sidecar.

The honest tradeoff, documented deliberately: socket traffic is the one channel the worker does not mediate — no per-call allowlist, audit, or rate limit (the grant granularity is the service, and the sidecar enforces its own limits). The sidecar receives data, not code, from processes with no other capabilities; run it as its own hardened unit (dedicated user, no egress, GPU pinned) — it is trusted infrastructure, like a database. Sidecar recipes (llama-server, vLLM) are in DOCKER.md.

Sizing for data- and compute-heavy workloads¶

The profile's defaults are sized for untrusted glue code. Heavier sealed workloads — numeric simulation, media processing, large dataset transforms — raise the limits and grant what they need (image recipe, sizing guidance, and cache environment variables for the read-only rootfs are documented in DOCKER.md under "Sizing the airgapped runner for heavy workloads"):

[flux.workers]
runners = ["docker-airgapped"]
airgapped_image = "my-registry/flux-compute:0.60.0"
airgapped_gpus = "all"                        # GPU-accelerated compute
airgapped_mounts = ["/srv/datasets:/data"]    # reference data, read-only
airgapped_shm_size = "8g"                     # large inter-process buffers
airgapped_memory = "32g"
airgapped_tmp_size = "8g"
airgapped_execution_timeout = 3600

--network=none keeps the container's own loopback, so a workflow can spawn helper processes and talk to them on 127.0.0.1 inside the sandbox — nothing is reachable from outside.

Dynamic workflows run here by default¶

Workflows registered by agents through the dynamic-registration endpoint get runner="docker-airgapped" stamped server-side ([flux.dynamic_workflows] require_runner), overriding anything the authored source declares. The author is the adversary in that model; the sealed runner is what makes accepting model-authored code tenable. Relax require_runner only in development.

Failure semantics¶

Crash (OOM kill, segfault, hard exit): durable executions are released for re-dispatch and deterministic replay resumes from the last checkpoint; transient executions fail terminally.
Wall-clock timeout (ExecutionTimedOut): terminal FAILED for both durabilities — a deterministic timeout would re-dispatch forever, so it is a policy violation, not a crash.
Cancellation: SIGTERM is forwarded into the container (docker run sig-proxying), with a grace period before the container is killed.