Overview

CapyCTL runs vLLM, SGLang, TensorFold and llama.cpp models on GPU machines you own, and any other inference server as a custom engine. It parks the models nobody is using, which frees their GPU memory, and wakes the one a request asks for. Clients use one OpenAI-compatible endpoint, on your network, with an API key.

Start here:

1. How it works: one picture of the parts, and what parks and wakes mean.
2. Install CapyCTL with one command.
3. Run on one machine: add an engine, start, deploy, send a request.
4. Run on several machines: one server, several GPU machines.

Tasks:

• Install an engine: vLLM, SGLang or TensorFold in a virtual environment, or a llama.cpp build, step by step.
• Add an engine: register the vLLM, SGLang, TensorFold or llama.cpp you have, or another program as a custom engine.
• Deploy a model: the deployment file; start, stop, delete.
• Make a request: curl, streaming, the Python client, other machines, decision requests, and request IDs and metrics.
• Parking and switching: more models than your GPU holds.
• How requests wait and models switch: two apps, two models, one GPU.
• Driving CapyCTL from a controller: the management API from a program: create, start, watch, retry.
• Run in a container: a host or standalone role as the main process of a container.

Reference: CLI, configuration files, settings, engine options, network access, data retention, installer options, exit codes and errors.

What changed in the latest release: release notes for 0.1.6.

CapyCTL does not install engines or GPU drivers. Bring your own vLLM, SGLang or TensorFold installation, llama.cpp build or other server; Install an engine shows how. Models come from a directory on your machine or from Hugging Face.