Docs

Scheduling and running

The scheduler starts runs when they're due, retries the ones that fail, and starts each on this machine or on a cluster of its own. Every run is recorded in Postgres, so any of them can be resumed, and its tables say what made them.

The scheduler

orca scheduler links no pipeline. It knows one by the describe that orca register recorded, and starts its runs as <binary> resume <run>, so a run records its own progress exactly as a run started by hand does. It needs ORCA_DATABASE_URL; the command line lists its commands.

./build/orca register ./build/gharchive
./build/orca backfill gharchive 2025-01-01..2025-01-31
./build/orca scheduler

Each fire is a row with a unique (pipeline, time), inserted with its queued run in one transaction, so two schedulers ticking at once, or one killed mid-tick, never fire twice. Fires missed while it was down are made on its next tick, once each. A queued run starts only while fewer than max_runs of the pipeline's runs are live, counted under a lock on the pipeline's row, and none start while it's paused. A scheduled, backfilled or sensed run that fails is started again after ORCA_RETRY_SECONDS, doubled with each start, plus up to a tenth as jitter, up to run_retries times; one whose process died is started again as soon as its heartbeat is stale. A run you started by hand, or from the UI, isn't retried. The scheduler holds nothing in memory, so it can be stopped and restarted at any time. Registering a new build moves the pipeline's queued runs to the new code.

On Kubernetes and EC2

By default the scheduler starts a run as a process on its own machine, and the run forks its Dista workers there. ORCA_RUNNER starts each run on a cluster of its own instead, from the image the pipeline was registered with:

./build/orca register --image 123456789012.dkr.ecr.us-east-2.amazonaws.com/orca:data \
    --head-type c7g.large --worker-type c7g.2xlarge --workers 3 ./build/taxi
ORCA_RUNNER=ec2 ./build/orca scheduler

ORCA_RUNNER=ec2 starts a head instance, on demand, running the Dista head and the driver (<binary> resume <run>), and --workers worker instances pointed at it; --spot makes the workers Spot instances, where the account allows them, and --workers MIN..MAX starts MIN and adds or drains workers up to MAX as the head's demand changes. Each instance is tagged with its run, told to terminate when it shuts down, and recorded with its type, when it started and ended, and why it ended (it shut itself down, it was terminated, or a Spot interruption). ORCA_RUNNER=kubernetes creates a driver Job, a head Pod and Service and the worker Pods in ORCA_RUNS_NAMESPACE, all labelled with the run and owned by the Job. A pipeline without Dista tasks gets only its driver.

Both clean up in layers. A run's instances or pods end when it does; every tick the scheduler removes those of runs that aren't running any more and those older than ORCA_RUN_DEADLINE, so nothing a killed scheduler left behind keeps running. A run whose head goes is started again on new instances. Before making a cluster, the scheduler adds up the vCPUs of every live run's pods or instances, and a run that would take them past ORCA_VCPU_BUDGET waits.

orca aws up makes all of this on AWS as one stack: a control instance running the scheduler with ORCA_RUNNER=ec2, Postgres and the UI, and what run instances need (orca aws, local or AWS?).

Runs in Postgres

With ORCA_DATABASE_URL set, the binary creates and migrates an orca schema at start-up and records every run, every step of every partition, and every attempt, with the inputs and outputs of each. Each change of state is one transaction.

export ORCA_DATABASE_URL=postgresql:///orca
./build/gharchive run --partitions 2025-01-01..2025-01-03
./build/gharchive status 7        # run 7's steps and attempts
./build/gharchive resume 7

resume reruns every step that hasn't succeeded, in order, with its recorded input; steps that succeeded are read back, not run again. A run started by a binary with a different registry hash is refused unless --allow-new-code is passed.

One process owns a run at a time. It holds the run with a heartbeat, written from a thread with its own connection, and resume refuses a run whose heartbeat is younger than ORCA_STALE_SECONDS. A run whose process was killed goes stale and can be resumed; a run that ended can be resumed at once. Each attempt also checks that its process still owns the run, so a process that lost its run stops at its next step rather than racing the new owner. tests/kill.sh kills a run at a random step with kill -9, resumes it, and checks every step ran to success exactly once.

Postgres must keep synchronous_commit on: with it off, an immediate restart of the server can lose commits it already acknowledged. The chaos test from PR #6 found this when a run's row vanished.

Logs

With ORCA_LOGS set to a directory, each attempt writes <pipeline>/<run>/<step>/<partition>/<attempt>.log: its start, its end, its error, and its ctx.log lines with UTC timestamps. The path is recorded with the attempt and shown by status <run>. A run with Dista tasks writes the runtime's timeline beside them as timeline.json, in Chrome's trace format, which Perfetto opens. The scheduler keeps a run's driver output as driver.log, and an EC2 run's head and workers upload theirs to cluster/<role>-<instance>.log every ORCA_CLUSTER_LOG_SECONDS while they run, as an instance can go without notice.

ORCA_LOGS and ORCA_WAREHOUSE can be locations in object storage, s3://bucket/prefix, configured from the AWS_* environment or the instance's role. An object can't be appended to, so a log is written to a file under ORCA_LOG_STAGE and uploaded when it's complete: an attempt's when the attempt ends, a timeline when the run does. The UI reads them back from wherever they are.

Machine metrics

Beside every process doing a run's work, the runner starts <binary> sample, which writes a JSON line of the machine's CPU, memory, disk, disk I/O and network every ORCA_SAMPLE_SECONDS to <pipeline>/<run>/metrics/<instance>.jsonl under ORCA_LOGS. A run's page draws them as its Machines section, on the same time axis as its steps, and marks the stretches where memory use or a disk was over 90% or CPU steal over 20%, which explain most slow or failed runs.

Lineage

Every read and commit a step makes is recorded with its dataset, so lineage is two walks over them. Back from a snapshot: the run, step, partition and code that made it, the snapshots that attempt read, what made each of those, and so on, to sources no Orca run made. Forward: everything made from it. Both stop after 20 steps.

$ ./build/orca lineage lake.report
lake.report@3141592653
made from:
  0 lake.report@3141592653 by report run 9 report code 0x5e1f... from lake.clean@2718281828
  1 lake.clean@2718281828 by clean run 8 clean 2024-02 code 0x9a0c... from lake.raw@1618033988
  2 lake.raw@1618033988 by ingest run 6 load 2024-02 code 0x77d2...
made from it:
  nothing

Rebuilds

$ ./build/orca rebuild raw.events 2024-03
rebuild 1
  level 0 clean 2024-03
  level 1 report 2024-03

A rebuild runs every pipeline downstream of a dataset, a sensor or not, over the partitions given. Each pipeline is placed at its longest distance from the dataset, so one that reads two rebuilt datasets runs after both, and the scheduler queues a level once every run before it has succeeded. A run that fails stops the rebuild; resuming it from the UI carries on. The rebuild's own commits don't also start the sensors it reruns, so each pipeline runs once.

Environment

When things fail

A step that raises is retried up to its retries, then fails. The steps downstream of it are skipped, while other partitions' steps that don't depend on it carry on, so one bad month doesn't stop the others; the run ends failed and can be resumed. A process that dies leaves its run running with a stale heartbeat; resume marks the interrupted attempt failed and continues from there. One that crashed has its attempts failed with its stack dump's first frames as soon as its runner sees it end (Debugging failures). A commit that landed before the crash is found by the retry rather than made again. Fuzzing, and a chaos test that kills runs, restarts Postgres and races two resumes over many seeds, run in CI.