Service Autoscaling

A workflow service normally runs as a single, long-lived container. With an autoscaler block you can instead run a service as a pool of replicas whose size grows and shrinks automatically based on metrics the service itself exposes. This is useful for workloads whose demand varies over time — inference servers, queue workers, or any service that benefits from running more (or fewer) copies as load changes.

The autoscaler decides when to add or remove a replica by evaluating PromQL expressions against the service's own metrics. Replicas of an autoscaled service are reached through a single DNS name that always resolves to the currently-ready replicas, so clients do not need to know how many replicas exist at any moment.

How it works

When a service declares an autoscaler, Fuzzball:

1. Pre-allocates capacity for up to replicas.max copies of the service, starting replicas.min of them (which may be 0).
2. Scrapes the service's metrics endpoint (when metrics.enabled is set) at the configured interval.
3. Evaluates each scale-up trigger and the scale-down policy against those metrics. When a scale-up trigger fires, the next pending replica is started; when scale-down fires, a running replica is stopped.
4. Maintains a DNS record for the service that contains one address per ready replica, adding a replica's address once it passes its readiness probe and removing it when the replica is scaled down.

Each scaling action is rate-limited by its cooldown-period, and the replica count is always bounded by replicas.min and replicas.max.

If a replica above replicas.min fails to start or dies, it is treated as optional capacity: the replica is returned to the pending pool (so a later scale-up can retry it) and the workflow and its other replicas keep running. Failures of the guaranteed baseline replicas (those at or below replicas.min) still fail the workflow.

Pool capacity

The provisioner pool a replica pool lands on must be large enough to actually host the replicas. At submission time Fuzzball checks the selected provisioner definition's capacity -- how many replicas of the requested size fit per node (flooring independently on cores, memory, and devices) multiplied by the pool's node count (the usable nodes for a static pool, or the definition's maxNodes for a dynamic pool that can still provision):

• replicas.min must fit. If the pool cannot host the guaranteed baseline, the workflow submission is rejected with an error naming the service, the definition, and the shortfall. Reduce replicas.min or select a larger pool.
• replicas.max above capacity is capped. If the baseline fits but the pool cannot host the full replicas.max, the submission is accepted, a warning is logged, and a service_capacity_capped event records the effective maximum. The service runs at the capacity the pool allows.

At runtime, if a scale-up fires but the pool has no room for another replica, the scale-up is skipped and a scale_up_capacity_exhausted event is emitted rather than leaving the replica pending forever. fuzzball workflow why reports such a replica as blocked on pool capacity.

This check is evaluated per service against the pool's nominal capacity -- it is not admission control across tenants. If several autoscaled services (or a service and other jobs) share the same provisioner definition, each is validated against the full pool independently, so they can pass individually while jointly exceeding the pool. Size shared pools for their combined peak demand.

Metrics

The autoscaler evaluates PromQL expressions against metrics collected from your services. Fuzzball supports two types of metrics:

Service-exposed metrics

When you enable metrics.enabled: true on a service, Fuzzball scrapes a Prometheus-compatible metrics endpoint from each replica at the configured interval. These are metrics that your service application exposes — for example, vllm:num_requests_waiting from vLLM, or custom business metrics from your application.

Configure the scrape with:

• metrics.path — HTTP path to the metrics endpoint (default /metrics)
• metrics.port — TCP port where metrics are served
• metrics.interval — Scrape interval in seconds (default 15)

All scraped metrics are tagged with service and replica labels, so your PromQL queries can aggregate across the pool (e.g., sum(my_metric) by (service)) or target specific replicas.

Default container metrics

For every autoscaled service (whether or not it exposes its own metrics), Fuzzball automatically collects synthetic container-level metrics from the underlying runtime. These metrics are available immediately without requiring your service to expose a /metrics endpoint, making them ideal for simple CPU- or memory-based scaling policies.

The default metrics follow the cAdvisor naming convention:

These metrics are collected from the container runtime at the same interval as service-exposed metrics (or every 15 seconds if no scrape is configured) and are tagged with the same service and replica labels.

Example: Scale up when average CPU usage across all replicas exceeds 80% of one core for one minute:

autoscaler:
  replicas:
    min: 1
    max: 10
  scale-up:
    triggers:
      - metrics-query: rate(container_cpu_usage_seconds_total[1m]) >= bool 0.8
        cooldown-period: 60

Example: Scale down when a replica's memory usage falls below 50% of its limit:

autoscaler:
  scale-down:
    metrics-query: |
      (container_memory_usage_bytes / container_memory_limit_bytes) <= bool 0.5
    cooldown-period: 300

Reaching the replicas

Replicas of a self-scaling service share a single autoscaler DNS name of the form:

<service>.autoscaler.<workflow-id>.fuzzball

Resolving this name returns one A record for every ready replica. Because the replicas are reached through this shared record, a self-scaling service cannot declare per-replica network endpoints.

Each replica serves its declared port on a randomly assigned host port (see the workflow syntax reference), so an IP address alone is not enough to connect — the port differs per replica. Two mechanisms carry the full host:port information:

• DNS SRV records named _<port-name>._tcp.<service>.autoscaler.<workflow-id>.fuzzball, one per ready replica, each resolving to a replica's hostname and its assigned port. SRV-capable clients and load balancers can consume these directly.
• Dynamic configuration files, which expose per-port ip:port arrays to a consumer service's rendered config. This is the recommended pattern for fronting a replica pool with a proxy or load balancer that takes a static backend list.

Self-scaling a service

The most common pattern is a service that scales its own replicas. Define replicas and one or more scale-up triggers (with no services field) plus an optional scale-down policy:

version: v1
services:
  inference:
    image:
      uri: docker://vllm/vllm-openai:latest
    command: ["--model", "facebook/opt-125m", "--port", "8000"]
    resource:
      cpu:
        cores: 8
      memory:
        size: 16GB
    network:
      ports:
        - name: http
          port: 8000
          protocol: tcp
    readiness-probe:
      http-get:
        path: /health
        port: 8000
      period-seconds: 5
    autoscaler:
      replicas:
        min: 1
        max: 8
      metrics:
        enabled: true
        path: /metrics
        port: 8000
        interval: 15
      scale-up:
        triggers:
          - metrics-query: vllm:num_requests_waiting >= bool 2
            cooldown-period: 60
      scale-down:
        metrics-query: vllm:num_requests_running == bool 0
        cooldown-period: 300

This service starts with one replica, scales up (one replica at a time, at most once per minute) whenever two or more requests are waiting, and scales back down — no more often than every five minutes — when a replica has no requests running.

Scale-to-zero

Setting replicas.min: 0 lets a service idle with no running replicas and start its first replica only when a trigger fires. This is well suited to bursty or on-demand workloads where you do not want to hold capacity while the service is idle.

Cross-service scaling

Instead of scaling its own replicas, a service may act as a source that scales one or more other services. This is useful for bringing a pool of workers up from zero in response to demand observed by a frontend.

To do this, give the source service's scale-up triggers a services list naming the target services. The targets define the replicas pool; the source does not:

version: v1
services:
  frontend:
    image:
      uri: docker://mycompany/frontend:latest
    network:
      ports:
        - name: metrics
          port: 9090
          protocol: tcp
    autoscaler:
      metrics:
        enabled: true
        path: /metrics
        port: 9090
        interval: 15
      scale-up:
        triggers:
          - metrics-query: queue_depth >= bool 10
            cooldown-period: 30
            services: [worker]

  worker:
    image:
      uri: docker://mycompany/worker:latest
    autoscaler:
      replicas:
        min: 0
        max: 20
      scale-down:
        metrics-query: worker_idle == bool 1
        cooldown-period: 120

Here the frontend watches its queue depth and, when it grows, scales the worker pool up from zero. Each worker owns its own scale-down policy, so it removes idle replicas independently of the source.

Dynamic configuration files

A service in a workflow can render a configuration file from the live addresses of other services and have it injected into its container, then re-rendered automatically as those services' replicas scale up and down. This lets a service (such as a load balancer or proxy) keep an up-to-date list of its backends without restarting.

Add a dynamic-config block naming the services to watch, the destination path inside the container, and a script that emits the file contents. Each watched service is exposed to the script as bash arrays (names uppercased, with any character that is not a valid identifier replaced by _):

• SERVICES_<NAME> — the ready replicas' IP addresses.
• SERVICES_<NAME>_<PORT-NAME> — one array per named TCP port, holding ip:port pairs that carry each replica's randomly assigned host port. Use these to address the replicas: the assigned host port differs per replica, so an IP plus the declared port is not reachable.

services:
  proxy:
    image:
      uri: docker://haproxy:latest
    dynamic-config:
      path: /usr/local/etc/haproxy/backends.cfg
      services: [worker]
      script: |
        for addr in "${SERVICES_WORKER_HTTP[@]}"; do
          cat <<EOF
        server ${addr} ${addr} check
        EOF
        done

Here worker declares a port named http; each rendered server line points at one replica's ip:port.

Validation rules

Fuzzball rejects a workflow at submission time if its autoscaler configuration is inconsistent. The most common rules are:

• A self-scaling service (no scale-up trigger targets other services) must define replicas and must not define network endpoints.
• replicas.max must be at least 1 and not less than replicas.min.
• A cross-service source (any scale-up trigger sets services) must not define its own replicas or a scale-down policy; each target service owns those.
• Every target named in a services list must exist, must define autoscaler.replicas, and must not define its own scale-up triggers (a pool is scaled up by exactly one source).
• All of a service's scale-up triggers must agree: either every trigger targets other services, or none do. The two modes cannot be mixed within one service.
• A given service may be targeted by at most one scale-up trigger.
• scale-down may never target other services.
• dynamic-config may only reference services that exist in the workflow.

See also

• Workflow syntax reference: services
• Workflow Endpoints
• Connecting to Workflow Services