# The dista command

> The dista command: a generic head, status and status --json, drain, scale --once, timeline and stop; Prometheus metrics on the head and workers; and the task timeline in Chrome's trace format.

[magmalake](https://magmalake.dev/) [Dista](https://magmalake.dev/s/dista)

`dista` is a head for any Dista program, and a look at a running cluster. It depends on Dista's core alone, so your program's own binary stays the driver and the worker. Beside it, the head and every worker serve Prometheus metrics, and a driver can write its tasks' timeline.

## Commands

Build it from the repository with `pixi run cli`, to `build/dista`; the container image has it at `/opt/dista/bin/dista`. Every command but `head` connects to the head as an observer, which owns no objects and isn't counted as a driver. Its head is `--head host:port`, else `DISTA_HEAD`, else `localhost:6390`.

| Command | What it does |
| --- | --- |
| dista head \[--port 6390\] \[--pipeline 2\] | a head for any program, serving several at once |
| dista status \[--json\] | the head's workers, drivers and actors; --json, its demand on one line, for an autoscaler |
| dista drain WORKER | drains the worker at ip:port, exactly as SIGTERM would |
| dista scale --once | what an autoscaler should do now, from the head's demand |
| dista timeline -o FILE | every driver's finished tasks, as a Chrome trace |
| dista stop | shuts the head and its workers down |

`--pipeline` sets how many tasks the head keeps in flight per worker; see [Tuning](https://magmalake.dev/s/dista/docs/tuning#pipeline). `stop`'s workers exit instead of rejoining.

## status

`dista status` prints the head's view: workers, with the program each was built from and what it's running or hosting; drivers, with the objects each holds; and actors, with their worker, calls and last checkpoint.

`dista status --json` is the head's demand, on one line: tasks ready, waiting for inputs and running, actors waiting for a worker, the pipeline depth, whether there's a checkpoint store, the objects each driver holds (`driver_objects`), and per worker its address, node, tasks in flight, seconds idle, bytes of results held, actor and whether it's draining.

## scale

`dista scale --once` reads the demand and prints `dista.scale.decide`'s decision, a line each: `want N`, `start N`, `drain ADDRESS` for each worker to drain, and `short_since T` to pass back as `--short-since` next time. Its flags are the policy's, and what the loop running it knows:

| Flag | Sets | Default |
| --- | --- | --- |
| \--min | min\_workers | 1 |
| \--max | max\_workers | 4 |
| \--per | tasks\_per\_worker; 0, the head's pipeline depth | 0 |
| \--up-after | up\_after, seconds | 5 |
| \--idle | idle\_before\_down, seconds | 60 |
| \--step | max\_step | 4 |
| \--pending | workers started that haven't joined yet | 0 |
| \--short-since | the short\_since the last decision printed | 0 |

```
dista scale --once --min 0 --max 6 --step 4 --up-after 1 --idle 2 \
  --pending 0 --short-since 0
```

[Autoscaling](https://magmalake.dev/s/dista/docs/cluster#autoscaling) says how the policy decides.

## Metrics

With `DISTA_METRICS_PORT` set, the head and every worker serve Prometheus metrics at `/metrics`. The kind manifests turn it on at port 9090, for Kubernetes monitoring to scrape. The head's include:

- tasks: `dista_tasks_submitted_total`, `dista_tasks_finished_total` by outcome, `dista_tasks_retried_total`, `dista_tasks_timed_out_total`, and the `dista_task_duration_seconds` histogram
- demand: `dista_queue_depth`, `dista_tasks_ready`, `dista_tasks_waiting`, `dista_tasks_running`, `dista_workers`, `dista_workers_idle`, `dista_workers_draining`
- memory: `dista_objects`, `dista_head_result_bytes`, `dista_stored_results`, `dista_queued_bytes`, `dista_driver_held_objects_max`
- backpressure: `dista_admission_paused`, `dista_admission_pauses_total`, `dista_drivers_behind`
- actors: `dista_actors`, `dista_actors_waiting`, `dista_actor_restarts_total`, `dista_actors_moved_total`, `dista_checkpoints_total`, and `dista_results_moved_total` for drains

A worker's metrics count what it runs, stores, spills, copies and adopts: `dista_worker_tasks_finished_total`, `dista_worker_store_bytes`, `dista_worker_results_spilled_total`, `dista_worker_bytes_copied_total`, `dista_worker_fetch_buffer_bytes`, `dista_worker_results_adopted_total` and `dista_worker_draining`, among others.

## The timeline

The head stamps each task's submission, dispatch and completion, and the worker its start and end. `rt.timeline(path)` writes every finished task, in Chrome's trace format, which [Perfetto](https://ui.perfetto.dev) and the browser's trace viewer open, and returns how many it wrote. Each worker is a track, with a second track for time spent waiting for inputs. The driver can add its own work, so a gap between tasks says what it was:

```
var t = rt.clock()
var tasks = scan.plan_files()
rt.span("plan", t)          # from t to now, on the driver's track
...
_ = rt.timeline("trace.json")
```

`dista timeline -o trace.json` writes every driver's finished tasks, without taking them from the drivers. It names tasks by their wire ID, since a generic head doesn't know their names. A head restart loses the spans it held.

[A magmalake project](https://magmalake.dev/)

Apache 2.0 · © 2026 Marius Seritan