Docs

Failure and recovery

Dista runs a dead worker's tasks again and rebuilds a restarted head from its peers. It doesn't rebuild results lost with a worker: a job that needs one fails, naming it, and is run again, by Orca or by hand. Tasks may run more than once, so they should be safe to repeat, as a pure function is.

A task that raises

A task that raises fails its ObjectRef with the error, which get and take raise in the driver, and every task that depends on it fails with the same error. The worker carries on.

A worker that dies

The tasks a dead worker was running go back in the queue for another worker, up to DISTA_TASK_RETRIES times (default 3). Past that they fail with worker lost while running this task, as they do at once if an input they read from that worker went with it, or if a TaskMany had answered some of its results.

The results it held are lost. Dista doesn't rebuild them by running again the tasks that made them, so a task that needs one fails, saying the result was lost with its worker and that the job should be run again. A task on another worker that was fetching one fails with input lost: the worker holding it (<ip:port>) went away, and its worker drops the connection and carries on. In local mode, where nothing replaces a dead worker, tasks fail with no workers left once every worker is gone, rather than waiting.

A worker that's stopped, rather than killed, drains first: it loses nothing if it finishes within DISTA_DRAIN_SECONDS.

A silent peer

A host that freezes, loses its network or is hard-stopped never closes its connections. Every Dista socket has TCP keepalive: after DISTA_KEEPALIVE_SECONDS of silence (default 10; 0 turns it off) the kernel probes, and after 3 unanswered probes, about 25 s in all, the connection fails. The head then drops the worker and its tasks run again; a worker whose head went away rejoins DISTA_HEAD, a replacement head included; and a driver resumes with the new head. The kernel answers probes for a process that's only busy, so a long task doesn't trip it. With data unacknowledged on the connection, the kernel's retransmission limit applies instead, about 15 minutes on Linux.

An unreachable peer

A worker connects to the workers it fetches inputs from inside its event loop, so it keeps running tasks and serving fetches meanwhile. After DISTA_PEER_CONNECT_TIMEOUT seconds (default 5, fractions allowed) it fails the tasks waiting on that peer, with input lost: cannot reach the worker holding it. A driver's fetch waits the same time. A closed port fails at once; the timeout bounds an address that never answers, which the kernel would otherwise retry for about two minutes.

A task that runs too long

DISTA_TASK_TIMEOUT (seconds; unset, no limit) covers what keepalive can't see, such as a hung task or a stopped process. The head fails a task still running that long after it sent it, with timed out: ran longer than DISTA_TASK_TIMEOUT. It isn't retried, its worker isn't stopped, and a late result is dropped. dista_tasks_timed_out_total counts them. To stop waiting without failing the task, use get with a timeout.

A driver that leaves

When a driver's connection drops, the head deletes its tasks still queued or waiting for inputs, rather than run them for nobody. Tasks already running finish, and their results are dropped.

A head that restarts

Nothing the head holds is written to disk; a new head is rebuilt from its peers. Workers rejoin and report what they hold, and drivers reconnect under their old number and resubmit what they still need. A task running at that moment may run twice. Actors come back too, from their checkpoints.

A head numbers new drivers from a random start and gives an epoch in its welcome, so only a driver resuming under its old number claims the results workers report, and a new driver can't be handed another job's. Results no driver comes back for are freed after DISTA_ORPHAN_SECONDS (default 120). That covers a head restart, not a head lost for good together with its driver.

How it's tested

A chaos test kills the head at random under a random task graph and checks every answer. Fuzzers cover the head's frames, the codec and the scaling policy. Other tests kill a worker while two others fetch from it, fetch from a worker that stopped answering, drain a worker mid-job, and restart the head once an actor has a checkpoint. On kind, a worker pod is killed during the taxi queries and the actor's pod is deleted mid-run, and every answer and the actor's count must come out exact.