Docs

Actors and checkpoints

An actor is state that lives on one worker between calls. Its methods run one at a time, in the order they were made, from the driver, from tasks or from other actors. Checkpoints let it come back with its state when its worker goes away.

Actors

An actor is a state type, the function that builds it, and methods that take it mut. Actors and methods go in Runtime[...] with the tasks:

from dista import Actor, Method, Runtime

@fieldwise_init
struct Counter(Movable):
    var total: Int

def new_counter(start: Int) -> Counter:
    return Counter(start)

def add(mut c: Counter, n: Int) -> Int:
    c.total += n
    return c.total

comptime CounterActor = Actor[Counter, Int, new_counter]
comptime Add = Method[CounterActor, Int, Int, add]

var rt = Runtime[CounterActor, Add, Square](workers=2)
var c = rt.create[CounterActor](0)
var n = rt.call[Add](c, 5)            # ObjectRef[Int]
print(rt.take(rt.remote[Square](n)))  # 25
rt.kill(c)                            # after its earlier calls; frees the worker

create returns at once; calls made before the actor is up wait for it, and if its constructor raises, every call fails with that error. Each actor gets a worker of its own, which runs no tasks while it hosts it. An actor created when every worker already hosts one waits for a worker to join or another actor to be killed. A method's result is an ObjectRef like a task's, so a task can take it as input. Calling a method on the wrong actor's handle is a compile error.

Calling from tasks

A handle is a value, so a task or another actor can call the actor too. Declare the task with CtxTask, or the method with CtxMethod, and its function gets a Client to call through. A CtxMethod's function takes the state, then the client, then its argument.

from dista import ActorHandle, Client, CtxTask

def bump(mut c: Client, h: ActorHandle[CounterActor]) raises -> Int:
    return c.take(c.call[Add](h, 1))

comptime Bump = CtxTask[ActorHandle[CounterActor], Int, bump]
var n = rt.remote[Bump](c)   # a task calling the actor

Calls from tasks go through the head, which keeps their order. A call blocks the task or actor making it until the answer comes. Tasks never run on an actor's worker, so a task can't wait on itself, but a cycle of actor calls, one actor calling another that calls it back before answering, deadlocks.

Restarts

When an actor's worker goes away, the head places the actor on another worker, builds it again from its last checkpoint, or its constructor if it has none, and runs every call since then again, in order. Each call is answered once. A call may run twice on the actor's state, so a method should be safe to repeat; that's the price of not losing calls.

Checkpoints

A checkpoint is the actor's state through the codec, written to DISTA_CHECKPOINT_URI: a local directory, a file:// one, or anything objectstore writes to, such as s3://bucket/prefix. A local directory only helps if the node that restarts the actor sees it too, so a cluster needs a shared volume or object storage. Without the variable, checkpoints fail. Ask for one every so many calls, or when it suits:

var c = rt.create[CounterActor](0, checkpoint_every=100)
...
var calls = rt.take(rt.checkpoint(c))   # written; covers this many calls

A restart then replays only the calls since the checkpoint, and the head stops keeping the ones it covers. A state type that implements Codec decides what it saves: a cache can save nothing, a loaded model its path. Checkpoints also carry an actor across a head restart: the new head restores it from its newest checkpoint, and its callers send again the calls no checkpoint includes. Killing an actor deletes its checkpoints.

Draining an actor's worker

A worker that drains moves its actor rather than losing it. The head holds the actor's new calls, asks its worker for a checkpoint, restores the actor on another worker from it, and sends it the held calls: no call runs twice, and it doesn't count as a restart. Without DISTA_CHECKPOINT_URI that checkpoint fails, and the actor restarts when the worker leaves, replaying its calls, as when a worker dies.