# Orca API reference

> The `orca` package, from Orca 0.3.2, as published on mojoshelf (commit 1db5de9aab67), by `mojo doc` from Mojo 1.2.0.

## Modules

- [orca.actions](https://magmalake.dev/s/orca/api/actions.md): What `orca ui` does to runs: each action is rows in Postgres that the scheduler, or the run's own process, acts on.
- [orca.auth](https://magmalake.dev/s/orca/api/auth.md): Who may act in `orca ui`: users with passwords, their sessions, and a record of what each did.
- [orca.aws](https://magmalake.dev/s/orca/api/aws.md): `orca aws`: a project's orca on AWS, with the `aws` and `docker` commands its user already has (`aws login`, or `AWS_PROFILE` for a named profile).
- [orca.aws_assets](https://magmalake.dev/s/orca/api/aws_assets.md): The stack template and the project Dockerfile `orca aws` writes out: deploy/aws/orca.yaml and deploy/project.Dockerfile, embedded by scripts/embed-aws.py. Edit those, then run it; `pixi run test-aws-assets` fails while this is out of date.
- [orca.awsec2](https://magmalake.dev/s/orca/api/awsec2.md): The few EC2 calls the `ec2` runner makes (`orca.ec2`), over objectstore's SigV4 signing and HTTP.
- [orca.chart](https://magmalake.dev/s/orca/api/chart.md): Runs per day, drawn as SVG on the server for the home page: one bar a day, stacked by how the runs ended.
- [orca.civil](https://magmalake.dev/s/orca/api/civil.md): Calendar arithmetic: dates to days since 1970-01-01 and back.
- [orca.cli](https://magmalake.dev/s/orca/api/cli.md): `orca_main`: the command line every pipeline binary gets.
- [orca.command](https://magmalake.dev/s/orca/api/command.md): The `orca` command: the scheduler, the UI and registration, in a binary that links no pipeline. A project builds its own from two lines:
- [orca.cron](https://magmalake.dev/s/orca/api/cron.md): Cron expressions, and when they next fire in a time zone.
- [orca.ctx](https://magmalake.dev/s/orca/api/ctx.md): `Ctx`: what a step gets besides its input.
- [orca.dag](https://magmalake.dev/s/orca/api/dag.md): A pipeline's DAG drawn as SVG, on the server.
- [orca.day](https://magmalake.dev/s/orca/api/day.md): `Day`: the partition key of a daily pipeline.
- [orca.ec2](https://magmalake.dev/s/orca/api/ec2.md): Runs on EC2: a head instance and worker instances per run, started and ended by the scheduler (DESIGN.md, phase 5, PR 5).
- [orca.graph](https://magmalake.dev/s/orca/api/graph.md): A directed graph drawn as SVG, on the server: the pipelines and datasets that feed each other, and a snapshot's lineage.
- [orca.heartbeat](https://magmalake.dev/s/orca/api/heartbeat.md): A run's heartbeat, kept fresh from a thread while the run's steps work.
- [orca.html](https://magmalake.dev/s/orca/api/html.md): Writing HTML: every value escaped, markup only where it's said to be.
- [orca.json](https://magmalake.dev/s/orca/api/json.md): The little JSON orca writes: quoted strings.
- [orca.key](https://magmalake.dev/s/orca/api/key.md): `Key`: what a partition key is, whatever it counts.
- [orca.keyed](https://magmalake.dev/s/orca/api/keyed.md): Keyed partitions: keys that are names, read from a column of a dataset.
- [orca.kube](https://magmalake.dev/s/orca/api/kube.md): Runs on Kubernetes: a dista cluster per run, made and removed by the scheduler (K8S.md).
- [orca.lineage](https://magmalake.dev/s/orca/api/lineage.md): Lineage: which snapshots a dataset's snapshot was made from, and what was made from it, from the reads and commits every attempt records.
- [orca.locate](https://magmalake.dev/s/orca/api/locate.md): Where a pipeline's step functions and dista tasks are defined in its source, so `orca ui` can link each one to its line at the commit a run ran.
- [orca.logs](https://magmalake.dev/s/orca/api/logs.md): Where attempts' logs, drivers' logs and runs' timelines go.
- [orca.month](https://magmalake.dev/s/orca/api/month.md): `Month`: the partition key of a monthly pipeline.
- [orca.name](https://magmalake.dev/s/orca/api/name.md): `Name`: the partition key of a keyed pipeline (`orca.keyed`).
- [orca.partitions](https://magmalake.dev/s/orca/api/partitions.md): Partition sets: which keys a pipeline is partitioned by, from when.
- [orca.pipeline](https://magmalake.dev/s/orca/api/pipeline.md): Pipelines: a named list of steps, checked and run in order.
- [orca.proc](https://magmalake.dev/s/orca/api/proc.md): Starting processes: what the scheduler needs to run pipeline binaries.
- [orca.schedule](https://magmalake.dev/s/orca/api/schedule.md): Schedules: when a pipeline runs by itself, and which partition it fills.
- [orca.scheduler](https://magmalake.dev/s/orca/api/scheduler.md): The scheduler: fires schedules, senses datasets, queues backfills and rebuilds, and starts runs.
- [orca.secrets](https://magmalake.dev/s/orca/api/secrets.md): Secrets a pipeline's runs get: `orca register --secret NAME=ID`.
- [orca.sensors](https://magmalake.dev/s/orca/api/sensors.md): Runs started by datasets: sensors, and rebuilds downstream of a dataset.
- [orca.source](https://magmalake.dev/s/orca/api/source.md): Where a pipeline binary's code came from: its git repository and commit, given at build time.
- [orca.state](https://magmalake.dev/s/orca/api/state.md): orca's state in Postgres: pipelines, runs, steps and attempts.
- [orca.step](https://magmalake.dev/s/orca/api/step.md): Steps: the functions a pipeline runs, and the edges between them.
- [orca.store](https://magmalake.dev/s/orca/api/store.md): Where the executor records what it does.
- [orca.tables](https://magmalake.dev/s/orca/api/tables.md): Iceberg tables from steps: pinned reads, and commits a retry recognises.
- [orca.tasks](https://magmalake.dev/s/orca/api/tasks.md): The dista tasks a pipeline's steps call, and the runtime a run gets.
- [orca.timeline](https://magmalake.dev/s/orca/api/timeline.md): A run's Dista timeline and its workers over time, drawn as SVG on the server for the run's page.
- [orca.tz](https://magmalake.dev/s/orca/api/tz.md): Time zones, from the system's TZif files (RFC 8536).
- [orca.ui](https://magmalake.dev/s/orca/api/ui.md): `orca ui`: pipelines and runs as web pages, and the same as JSON.
- [orca.ui_style](https://magmalake.dev/s/orca/api/ui_style.md): The stylesheet `orca ui` serves at `/static/orca.css`: magmalake.dev's colours and type, light and dark, and system fonts (nothing is fetched).
- [orca.warehouse](https://magmalake.dev/s/orca/api/warehouse.md): A dataset's current snapshot, read from its table's metadata file without iceberg, for `<binary> sense`.

orca: orchestration for Mojo data pipelines.

Declare steps and the pipeline they make, and give the binary orca's
command line:

```mojo
from dista import Seq
from orca import Ctx, Gather, Mapped, Month, Pipeline, Step, orca_main

def size(mut ctx: Ctx, m: Month) raises -> Int:
    return m.month * 100

def total(mut ctx: Ctx, sizes: Seq[Int]) raises -> Int:
    var t = 0
    for i in range(len(sizes)):
        t += sizes[i]
    return t

comptime Size = Step["size", Month, Int, size, Mapped]
comptime Total = Step["total", Seq[Int], Int, total, Gather, after=Size]
comptime Sizes = Pipeline["sizes", Size, Total]

def main() raises:
    orca_main[Sizes]()
```

See DESIGN.md for how it works and where it's going.