Serving an AI Model

Fuzzball serves large language models from the workflow catalog. One command starts a model on a cluster GPU and gives you a base URL that speaks the OpenAI API, so any client written for OpenAI can use it: a coding agent, an IDE plugin, a notebook, or plain curl.

You do not need to write a Fuzzfile, choose a container image, or know how vLLM, the server that runs the model on the GPU, is configured. Each model has its own catalog entry, already carrying the settings that model needs.

Choosing a model

The only thing you have to match is GPU memory. A model needs enough of it to hold its weights plus the context it is serving:

Two numbers move those figures. Each entry assumes its own default context size, which is large, so lowering MaxContextSize lets a model run on less memory.

The memory also does not have to come from one card. Each entry defaults to as many 80GB-class GPUs as its total requires, which you change with GpusPerNode. Qwen3-Coder 30B defaults to two GPUs to reach 160GB, but runs on a single 160GB card, or on one 80GB card with a smaller context. Those GPUs all come from a single node; if no node in your cluster has enough of them, see Spanning several nodes.

To see everything your cluster offers:

$ fuzzball workflow catalog list --category ML_AND_AI
$ fuzzball workflow catalog list --tag LLM

The first lists every AI and ML entry; the second narrows to language models.

GPT-OSS 20B is the smallest and the usual starting point. If you are unsure what GPUs your cluster has, run fuzzball node list or ask your administrator.

Starting the model

Every per-model entry except Llama 4 Scout runs without parameters:

$ fuzzball workflow catalog start "GPT-OSS 20B"

This prints a workflow ID. Copy it: the commands below need it, and fuzzball workflow list finds it again later. Add --watch to follow the workflow until it is running:

$ fuzzball workflow catalog start "GPT-OSS 20B" --watch

A gated model additionally needs a Hugging Face token in a Fuzzball secret. Llama 4 Scout is the one gated entry today:

$ fuzzball secret create secret://user/hf-token --type hf '{"token":"hf_..."}'

$ fuzzball workflow catalog start "Llama 4 Scout" --values HfTokenSecret=secret://user/hf-token

The secret URI you create is the value you pass to HfTokenSecret.

Getting the URL and credentials

Calling the model takes two things: the base URL of its endpoint, and a Fuzzball access token for that endpoint.

The workflow is ready when its service is running and its health check has passed:

$ fuzzball workflow describe WORKFLOW_ID

If you started the workflow with --watch, this has already happened.

Then find the endpoint the workflow published. The list covers every endpoint you have access to, so match the WORKFLOW ID column against the ID you copied:

$ fuzzball endpoint list

The URL column is the endpoint root, and the ENDPOINT ID column is the ENDPOINT_ID used below. OpenAI clients want the root plus /v1 as their base URL, for example https://ENDPOINT_URL/v1.

Every endpoint has a Scope controlling who may call it. It defaults to user, so requests need a Fuzzball token bound to the endpoint:

$ fuzzball endpoint generate-token ENDPOINT_ID --expiration 8h

That prints a bearer token valid for the requested period.

On a non-public endpoint the Fuzzball token is the only credential you send. The endpoint proxy consumes the Authorization header to authenticate you, then tells the model service who you are, and the LiteLLM in front of the model admits you on that alone. Ask for the model list to confirm the whole path works:

$ curl -H "Authorization: Bearer YOUR-FUZZBALL-TOKEN" \
    https://ENDPOINT_URL/v1/models

A healthy response names the model:

{"data":[{"id":"openai/gpt-oss-20b","object":"model","created":1677610602,"owned_by":"openai"}],"object":"list"}

When you need the model's API key

Two cases still call for the LiteLLM key rather than caller identity: a request without a Fuzzball token to an endpoint started with Scope=public, which serves such requests anonymously (as it does any caller outside the endpoint's organization), and a cluster whose nodes have not picked up the caller-identity signing keys. Send it in the x-litellm-api-key header, which the endpoint proxy leaves alone:

$ curl -H "x-litellm-api-key: sk-YOUR-API-KEY" https://ENDPOINT_URL/v1/models

A key you send decides the request. The auth hook steps aside and LiteLLM's own key check answers instead of your caller identity, so a wrong or stale key is refused even when your Fuzzball token is valid -- see When you still need a LiteLLM key.

The entry generates a key at start time unless you supply one. Prefer ApiKeySecret, which names a user-scoped Fuzzball secret, over ApiKey, which stores the key in plaintext in the workflow definition. A generated key is in the definition:

$ fuzzball workflow get WORKFLOW_ID | grep -oE 'sk-[A-Za-z0-9_-]+' | head -1

That prints the key. It returns nothing when ApiKeySecret is in use, since the key is in the secret rather than the definition.

Spanning several nodes

Everything above puts one copy of the model on one node. When no single node has the memory, Nodes splits a copy across several. Fuzzball starts and stops those nodes together, and the first one serves the endpoint:

$ fuzzball workflow catalog start "Nemotron 3 Super" --values Nodes=2,GpusPerNode=4

A copy then uses Nodes times GpusPerNode GPUs, and no two of its ranks, the per-node processes of one copy, share a node, so the cluster needs that many nodes free. When they are merely busy the workflow waits for them and reports a scheduling_blocked event. It is rejected at submission only when the copy asks for more nodes than the cluster is configured to give it, which waiting cannot change.

You do not need to choose how the model is split across the nodes. Each per-model entry pins ExpertParallelism to false, the tensor-parallel layout its settings were tested with. For a mixture-of-experts model you can set ExpertParallelism=auto or =true to switch to expert parallelism instead; the generic vLLM entry defaults to auto.

What you may need to set is the network. The nodes talk to each other directly, and on a cluster with more than one network interface you must tell NCCL, the library the GPUs use to exchange data, which interface to use. That goes in ExtraEnv as space-separated NAME=VALUE pairs:

$ fuzzball workflow catalog start "Nemotron 3 Super" \
    --values 'Nodes=2,GpusPerNode=4,ExtraEnv=NCCL_SOCKET_IFNAME=eth0'

The interface name is specific to your cluster, so ask your administrator which one carries node-to-node traffic. A fabric with RDMA may need more than one variable.

A copy spanning nodes fails as a unit. If any of its ranks dies, the whole copy is stopped and returned for re-placement, rather than left serving with part of the model missing, and the error names the rank, its node, and the exit status. A later scale-up replaces the copy if the replica pool (see Serving several models behind one gateway) is still above its minimum; losing a copy at the minimum ends the workflow.

Serving several models behind one gateway

If one model is all you need, the defaults already serve it on one stable URL; skip to See also. Read on when you want several models behind one URL.

Each model entry runs a pool of replicas, meaning copies of the model that the cluster scales with demand. The Proxy parameter controls how that pool is published, and decides whether clients talk to the model directly or through a gateway. Either way the URL you are given stays stable for the life of the workflow.

Proxy=true, the default. The workflow runs its own LiteLLM proxy, which holds the endpoint and load-balances across whichever replicas are ready. The ApiKey is enforced there, and the request rate through it is what triggers autoscaling. This is the behaviour described above.

Proxy=false. Choose this when you plan to put several models behind one gateway. No LiteLLM proxy is started, and the pool carries the endpoint itself, still on one stable URL. It additionally publishes one address per replica, annotated so a gateway can discover the replicas and balance across them directly. Two things change:

• The ApiKey is no longer enforced, so the endpoint's scope becomes the only access control.
• Request-rate autoscaling stops, since it is the proxy that measures the rate.

$ fuzzball workflow catalog start "GPT-OSS 20B" --values Proxy=false

Start the models that way, then start the gateway. The LiteLLM Model Gateway entry polls the endpoints API, picks up every annotated replica you have access to, and serves them all from a single OpenAI-compatible base URL, adding and dropping them as pools scale:

$ fuzzball workflow catalog start "LiteLLM Model Gateway"

Point your clients at the gateway's URL. They ask for a model by name and never learn which workflow serves it. A request for a model whose pool has scaled to zero wakes the pool. That first request fails; retry it once the replica is ready.

See also

• Using OpenCode with a Fuzzball-Hosted Model for connecting a terminal coding agent to the endpoint you just created, including trusting a private cluster certificate from your workstation.
• Workflow Endpoints for endpoint scopes and access tokens in full.
• Service Autoscaling for the replica pool the model runs in.
• Connecting to Workflow Services for reaching the served model from your workstation, and for the browser and client-script connection modes.