Docs

Tuning

The defaults suit most jobs. These are the settings that change how much work is in flight, where results are kept, and how much memory the head and workers use. Measured runs against Ray and Rust are on the performance page.

Tasks in flight

The head keeps pipeline tasks in flight per worker, default 2: Runtime[...](pipeline=n) for a head built into the program, or dista head --pipeline n. Deeper hides the round trip to the head for small tasks. On an M4 MacBook, 10,000 tasks of square on 4 local workers took 11.0 µs each with no pipelining, 6.2 µs at depth 2 and 5.3 µs at depth 4. The head's single loop handles about 160,000 empty tasks a second, flat from 4 to 192 workers. pixi run bench runs both, from bench/fanout.mojo.

Backpressure

While the tasks queued or waiting for inputs reach DISTA_MAX_QUEUED (default 100,000), or their arguments DISTA_MAX_QUEUED_BYTES (default 1 GiB), the head stops reading drivers, and remote blocks until workers catch up. Past DISTA_MAX_DRIVER_BUFFER bytes (default 256 MiB) of results waiting for a driver that isn't reading, that driver's queued tasks wait. 0 turns a limit off. Workers, observers and clients inside tasks are never paused. The head's /metrics show dista_queued_bytes, dista_admission_paused and dista_drivers_behind.

The node store and spilling

A worker writes each large result into a file in the node store, which readers on the same host map without a socket. The store is /dev/shm on Linux, $TMPDIR elsewhere, or DISTA_STORE_DIR. DISTA_SHM=0 turns it off, keeping large results in the worker's memory. Past DISTA_STORE_BYTES of results (no limit when unset), or when a store write fails, results go to DISTA_SPILL_DIR on disk (default $TMPDIR, else /tmp), where readers map or fetch them as before.

On Kubernetes each pod has its own /dev/shm, so pods use sockets unless they share a volume for DISTA_STORE_DIR. A memory-backed /dev/shm evicts a pod that passes its size limit, so set DISTA_STORE_BYTES below it; the AWS manifests use 1 GiB per worker.

Memory held for drivers

A result stays in the driver, at the head (if it's small) and on its worker until the driver releases it. Read results with rt.take unless they're read again, and rt.release the refs you only passed to other tasks; see Freeing results. The head logs a driver holding DISTA_WARN_HELD objects (default 100,000; 0, never), and again at each doubling, and dista_driver_held_objects_max shows the largest.

Large results

A large result is copied once per hop. A worker serving a fetch sends from its file's mapping, or from memory, 1 MiB at a time, so many fetchers of one result don't cost a copy each. A result over 64 MiB comes in parts, which the fetcher assembles into one buffer, so results past 1 GiB cross hosts. A frame has to fit 1 GiB, so a task's argument can't be larger: keep large data in results. In bench/fetch.mojo (pixi run bench-fetch), 12 readers of a 256 MB result over sockets took about half as long once results were sent this way.

Debugging inputs

DISTA_DEBUG=1 has a worker log each task's inputs as it assembles them. For where time goes, write the timeline.