Deploy a model

A deployment tells CapyCTL which model to run, with which engine, and the name clients ask for. You write it once as a YAML file.

The deployment file

Three fields are enough:

name: my-model
engine: vllm
model: Qwen3-4B

name: the name clients send as model in their requests.
engine: the engine profile, as capyctl engine list shows it (Add an engine).
model: where the weights are. A directory under your models directory (~/models unless you set another), an absolute path (a Hugging Face cache snapshot works; its directory must not be writable by other users), or a Hugging Face repository:

name: qwen3-4b
engine: vllm
model: {hf: Qwen/Qwen3-4B-Instruct-2507}

CapyCTL pins a Hugging Face repository to the commit it points at when you deploy (write Qwen/Qwen3-4B-Instruct-2507@<commit> to pick one yourself), then downloads it into ~/models/sources on the machine that runs it. It checks free disk space first, and all downloads together are capped at 500 GiB. See settings to change the cap or turn downloads off.

CapyCTL fills in the rest: the GPU, the memory the engine may use (sized from the checkpoint and the GPU), and how the model parks. To see what it fills in:

capyctl validate config --file my-model.yaml

validate works offline, so it refuses an hf: reference without a commit; deploy pins it for you. Without --host it checks the document, its timeouts, a declared memory.startup and a declared resources block, and lists what needs a host:

my-model.yaml is a valid deployment document

Not checked
  resolution against a host: pass --host <host.yaml> to check the runtime profile, placement, the host's devices and capacity, and timeouts, and to see the host's defaults; a TensorFold profile under another name than tensorfold is recognised only there
  whether each allowed host is enrolled, online and has published (host_unpublished)
  which runtime profiles the host's role accepted and published after measuring each installation (profile_not_published)
  the host's current resource policy as the server stores it (resource_policy_unavailable)
  route and deployment-name conflicts with existing deployments (route_conflict)
  a new checkpoint's digest, measured on the host after acceptance (checkpoint_digest_pending)

Add a field to choose something yourself, for example:

residency: restart_only to stop the model instead of parking it, or deep to drop its weights when parked (Parking and switching).
devices: [{id: gpu1}] to pin a GPU on a machine with several.
host: gpu-box to choose the machine, with several machines.
engine_config: {memory: {request: 16GiB}} to set the GPU memory it may use: weights, KV cache and a margin for the engine’s own memory. On a unified-memory machine such as a GB10 the margin is 8 GiB, or the weights x 0.15 plus 4 GiB for checkpoints above 26.7 GiB; on a discrete GPU it is the weights x 0.10.
engine_config: {cuda_graphs: false} on SGLang to turn its CUDA graphs off. They are on by default; on a 16 GB card FrogNano-4B decoded 61 tokens/s with them and 35 without, and kept about 0.7 GiB more of the card while parked. After the first park CapyCTL charges a parked model what it measured it holding (parked in capyctl status deployment <name> --json).

Every other field is in Configuration files.

Decision models

A decision model answers typed questions about an input with a probability for every option, instead of generating text. It is served at POST /v1/systemone rather than at chat completions. Say so with api:

name: decider
engine: llamacpp
model: Clef-Flash-GGUF
api: decisions
resources: {gpu: 16GiB, ram: 2GiB}
engine_config:
  context_length: 16384
  llamacpp: {gguf_file: Clef-Flash-Q8_0.gguf}
  accept_extra_args: true
  extra_args: [--batch-size, "16384", --ubatch-size, "16384"]

llama.cpp reads a decision request’s whole input in one batch, so set both batch sizes to the context. With llama.cpp’s defaults (2,048 and 512 tokens) a longer input is refused with 502 engine_error and llama.cpp’s message, “increase the physical batch size”. The batch costs memory: state it in resources, as above, from the peak CapyCTL measures (capyctl status deployment decider --json, startup.measured). CapyCTL’s recipes have measured files for Clef-Flash, Clef, Kev-4B and Julia-1 on llama.cpp and Clef-Flash and Clef on SGLang, and a comparison of six decision models on one GB10.

api is chat when you leave it out. It is part of the deployment’s revision: the same weights served as chat and as decisions are two revisions. In this release a decisions deployment:

needs llama.cpp 0.7.0 or newer, or SGLang 0.5.22 or newer, registered with capyctl engine add (which records the engine’s version), or a custom engine whose manifest says api: decisions. vLLM and TensorFold are refused with capability_missing:decisions (Exit codes and errors). vLLM 0.32.0 serves POST /v1/systemone, but it answers Clef and Clef-Flash from their text output instead of their decision head: questions with more than 26 options are refused and accuracy and calibration fall well below SGLang’s and llama.cpp’s. CapyCTL admits vLLM once a release uses these models’ decision heads;
runs on one machine: a topology is refused;
restarts instead of parking: residency is restart_only, and deep or host_backed is refused;
has no tool-call or reasoning parser: tool_call_parser, reasoning_parser and those options in extra_args are refused;
runs only on machines whose CapyCTL serves decision models; an older one is never chosen, and naming it is refused with host_capability_missing:decisions.

capyctl status deployment <name> shows an API line for it ("api": "decisions" with --json); a chat deployment has neither.

A decision model generates no text, so CapyCTL proves it ready with a decision instead: it asks one fixed question (which team handles a ticket about a double charge, with three options) and waits for a complete answer, a probability for every option that sums to 1. It does not have to pick the obvious team; it has to answer. A model that answers malformed, or not at all, never becomes ready, and CapyCTL never sends it a chat request in place of the decision. The same check runs when a machine reconnects or CapyCTL restarts with the model still running. The first answer is recorded for the wake check (Parking).

The readiness question

This is the exact request CapyCTL sends, with model set to the name the engine serves the model under:

{
  "model": "<the engine's served model name>",
  "state": "The ticket says: my card was charged twice.",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Which team should handle this ticket?",
      "criteria": {
        "billing": "payments, charges and refunds",
        "shipping": "deliveries, parcels and tracking",
        "technical": "errors, crashes and sign-in problems"
      }
    }
  }
}

The model is ready when it answers route as a choice with a probability for each of the three options and no other, each between 0 and 1, summing to 1 within 0.001, and picks one of the three. Any answer that meets this counts, whichever team it picks. Any change to the request is a new version of the check, and a wake reference recorded with an older version is never compared with the new answers.

Deploy and start

capyctl deploy model --file my-model.yaml --activate --wait

Request identity: 01M3R78B47ANFBFVFY40HATPYJ (reuse --request-id 01M3R78B47ANFBFVFY40HATPYJ to recover this command)
Waiting for the checkpoint digest of my-model to be measured (at most 900s)
Deployed my-model: ready

  Revision    1
  Hosts       gpu-box
  Ready       1/1
  Startup     28.5 GiB
  Context     26752 tokens
  Operation   initialize succeeded

This saves the deployment, starts it and returns when the model answers. The first time CapyCTL sees a checkpoint it reads the files once to fingerprint them, so the first start takes longer.

Without --activate, deploy only saves it. For a second model, other-model.yaml:

capyctl deploy model --file other-model.yaml

Request identity: 01M3R7A6Y7JQV9A5X404JZC93X (reuse --request-id 01M3R7A6Y7JQV9A5X404JZC93X to recover this command)
Deployment other-model created (revision 1)

  Deployment ID       01M3R7A6YJW402N962HH17A66W
  Operation           01M3R7A6YJFV3HDFWQNZTWQTN0
  Checkpoint digest   being measured
the checkpoint digest of other-model is being measured; `capyctl start deployment other-model --wait` waits for it and starts the deployment

Then start it when you want it:

capyctl start deployment other-model --wait

Check on it

capyctl list deployments

NAME       STATE   READY   REVISION   HOSTS
my-model   ready   1/1     1          gpu-box

capyctl status deployment my-model

NAME       STATE   READY   REVISION   STARTUP    INITIALIZE   LAST OPERATION
my-model   ready   1/1     1          28.5 GiB   210s         initialize succeeded

INSTANCE   HOST      STATE   LIFECYCLE   DEVICES   LAST ERROR
0          gpu-box   ready   active      gpu0      -

To take a model off the GPU without stopping it, park it. The deployment shows parking and then parked while its memory is freed:

capyctl park deployment my-model

Request identity: 01M3R7ADN3J81JRNH2RDCVD12W (reuse --request-id 01M3R7ADN3J81JRNH2RDCVD12W to recover this command)
Park requested for my-model

  Operation   01M3R7ADNMD6JTTDDFJ9DKFXVQ

capyctl status deployment my-model

NAME       STATE    READY   REVISION   STARTUP    INITIALIZE   LAST OPERATION
my-model   parked   0/1     1          28.5 GiB   210s         park succeeded

INSTANCE   HOST      STATE    LIFECYCLE   DEVICES   LAST ERROR
0          gpu-box   parked   active      gpu0      -

STARTUP is the memory CapyCTL set aside to start the model. On a discrete GPU it is the card’s memory, with the host RAM the engine process takes beside it. A first start measures the peak (on a discrete GPU the card and host RAM apart) and later starts set that aside instead of the estimate; --json says where each figure came from (startup.provenance, and on a discrete GPU startup.device_provenance and startup.host_provenance: measured, or default for the estimate). Before a first start has measured it, a model whose estimated startup peak is above the machine’s managed limit shows that limit: its first start runs alone on the machine and sets aside all of it (--json names this whole_host). When a start fails, LAST OPERATION says so and the instance’s LAST ERROR gives the reason:

NAME           STATE    READY   REVISION   STARTUP                   INITIALIZE   LAST OPERATION
broken-model   failed   0/1     1          13.2 GiB (+4.0 GiB RAM)   130s         initialize failed (launch_failed)

INSTANCE   HOST      STATE    LIFECYCLE   DEVICES   LAST ERROR
0          gpu-box   failed   active      gpu0      launch_failed: launch failed: engine launch failed: the engine exited before readiness

Exit codes and errors says where to look next. While parking is on, these commands also print a warning that the engine runs with its sleep controls enabled. Those controls listen on loopback only. Add --json to any list or status command for the full record.

Stop, start, delete

capyctl stop deployment my-model               # stops the engine, keeps the deployment
capyctl start deployment my-model --wait       # starts it again
capyctl delete deployment my-model --stop      # stops it and removes it

Request identity: 01M3R7BZ886JYFSRCB9K915YSJ (reuse --request-id 01M3R7BZ886JYFSRCB9K915YSJ to recover this command)
Stop requested for my-model

  Operation   01M3R7BZ8SSE6F0WGNSV09WNHV

stop returns once the stop is accepted, while the engine is still going away. A start right after it is refused, exit status 25, and nothing is started:

$ capyctl start deployment my-model
Request identity: 01M3R7BZ98AKHFBBTZXBCCX1BC (reuse --request-id 01M3R7BZ98AKHFBBTZXBCCX1BC to recover this command)
error [still_stopping]: my-model is still stopping; nothing was started. Retry in a moment, or run `capyctl start deployment my-model --wait`, which waits for the stop to finish and then starts

With --wait, start waits for the stop to finish, starts the model and returns when it answers:

$ capyctl start deployment my-model --wait
Request identity: 01M3R7BZA67CC0V5C8F8BFF7BR (reuse --request-id 01M3R7BZA67CC0V5C8F8BFF7BR to recover this command)
Waiting for the stop of my-model to finish (at most 160s)
Started my-model: ready

  Ready       1/1
  Hosts       gpu-box
  Operation   initialize succeeded

A request for a stopped deployment is refused; it does not start it:

{"code":"deployment_stopped","message":"deployment 01M3R7A6YJW402N962HH17A66W was stopped by an operator; inference does not start it; start it with `capyctl start deployment 01M3R7A6YJW402N962HH17A66W`"}

A plain delete deployment right after stop is refused the same way (delete_requires_cleanup) until the stop finishes; delete deployment --stop waits for it. Deleting never touches the model files.

To change a deployment, edit the file and deploy it again with its current revision, the REVISION column of capyctl list deployments:

capyctl deploy model --file my-model.yaml --revision 1

Request identity: 01M3R7DDQH4E287PVNR4CBR38Y (reuse --request-id 01M3R7DDQH4E287PVNR4CBR38Y to recover this command)
Deployment my-model updated (revision 2)

  Deployment ID       01M3R7A6YJW402N962HH17A66W
  Operation           01M3R7DDR0S1GTBW8DGQ55SPMG
  Checkpoint digest   being measured

A change restarts the model’s engine; changing only the instance count does not. Afterwards:

NAME       STATE   READY   REVISION   HOSTS
my-model   ready   1/1     2          gpu-box

Next: Make a request.