magmalake Orca

Docs

Using Orca

Orca turns a Mojo file of steps into a pipeline binary with a command line, and a scheduler runs those binaries on this machine, on Kubernetes or on EC2, recording every run in Postgres. This page installs it; the pages below cover the rest. The plan is in the repository's DESIGN.md.

The docs

  • Writing a pipeline: Steps and pipelines, partitions by date or by name, schedules, Iceberg tables and datasets, sensors, Dista inside steps, and building the binaries.
  • The command line: The commands you type, on a pipeline binary and on orca, the ones orca runs for you, and how the same binary acts locally or on AWS.
  • Scheduling and running: The scheduler, runs on Kubernetes and EC2, runs in Postgres, logs and machine metrics, lineage and rebuilds, the environment, and what happens when things fail.
  • The UI: The pages and their JSON, the actions, logins, timelines and links to the code.
  • Debugging failures: Where a failed run's errors and logs are, adding context to errors, symbolized crash dumps, reproducing a partition locally, and resuming it.

Install it

Orca is the mojoshelf tin orca-mojo. With pixi and mojoshelf's pixi shelf installed, a new project adds it, and the tins it needs come with it: 

pixi init -c https://conda.modular.com/max-nightly -c conda-forge lake && cd lake
sed -i.bak 's/^\[workspace\]/[workspace]\npreview = ["pixi-build"]/' pixi.toml   # tins build with pixi-build
pixi add "mojo-compiler==1.2.0"
pixi shelf add orca-mojo

A pipeline is one Mojo file whose main hands it to orca_main, compiled into a binary of its own. Build one and run it in memory to try it: 

pixi run mojo build -I src pipelines/lake.mojo -o build/lake
./build/lake run --partitions 2024-01..2024-03

Writing a pipeline says what goes in the file, and how to build the orca command that schedules it.

The playground

To see the UI before running anything real, the repository's playground starts it on this machine with a throwaway Postgres, the example pipelines and their runs, two of them on a stand-in EC2: 

git clone https://github.com/magmalake/orca.mojo && cd orca.mojo
git clone https://github.com/magmalake/dista.mojo ../dista.mojo
git clone https://github.com/magmalake/k8s.mojo ../k8s.mojo
pixi run playground          # then http://127.0.0.1:8270, log in as demo / demo-password

From source

Building from source works the same way: the repository compiles Dista and k8s.mojo from those checkouts beside it, pixi run test runs the tests, and pixi run orca builds the scheduler to build/orca. It targets Mojo 1.2.0, with a nightly environment alongside. Its state goes through postgres-mojo, which loads libpq; the tests start a throwaway Postgres server of their own. orca.tables, the Iceberg part, has its tests in the data environment (pixi run -e data test-tables).