magmalake Dista

Tins · Dista

API reference

The dista package, from Dista 0.2.1, as published on mojoshelf (commit 9e4839937d82), by mojo doc from Mojo 1.2.0. Source.

dista: distributed tasks for Mojo.

Declare a remote function as a task, start a runtime with it registered, and call it:

from dista import Runtime, Task

def square(x: Int) -> Int:
    return x * x

comptime Square = Task[Int, Int, square]

def main() raises:
    var rt = Runtime[Square](workers=4)
    var r = rt.remote[Square](7)
    print(rt.get(r))  # 49

An actor keeps state between calls on a worker of its own:

comptime CounterActor = Actor[Counter, Int, new_counter]
comptime Add = Method[CounterActor, Int, Int, add]

var rt = Runtime[CounterActor, Add](workers=2)
var c = rt.create[CounterActor](0)
var total = rt.call[Add](c, 5)  # an ObjectRef[Int]
rt.kill(c)

See DESIGN.md for how it works.

Modules

  • dista.checkpoint Where actor checkpoints live: objects under DISTA_CHECKPOINT_URI. 7 functions
  • dista.client The driver's side of the protocol: submitting tasks, calling actors and getting results back from the head. 9 structs · 2 traits · 4 functions
  • dista.codec Turning task arguments and results into bytes and back. 1 trait · 4 functions
  • dista.conn A non-blocking framed connection, for the head's event loop. 1 struct
  • dista.frame Length-prefixed messages between drivers, the head and workers. 1 struct · 8 functions
  • dista.head The head: one event loop that schedules every driver's tasks onto workers. 1 struct · 1 function
  • dista.metrics Prometheus metrics: counters and a histogram, and the /metrics page. 1 struct · 5 functions
  • dista.runtime The runtime: one binary that runs as driver, head or worker. 1 struct
  • dista.scale How many workers a running job wants: a policy over the head's demand. 4 structs · 1 function
  • dista.seq Seq[T]: a list that goes through the codec. 1 struct
  • dista.signals SIGTERM: the head exits, and a worker drains, even as a container's PID 1. 3 functions
  • dista.socket IPv4 TCP and process control over libc. 2 structs · 13 functions
  • dista.store The node-local object store: large results in shared mappings. 1 struct · 8 functions
  • dista.task Tasks: remote functions, their IDs, and how a worker runs one. 7 structs · 6 traits · 2 functions
  • dista.worker A worker: runs tasks for the head and serves its results to peers. 3 structs · 1 function