magmalake

Getting started

A lake of GitHub activity

The example is mojo-census: it imports every day of the GH Archive into an Iceberg table, finds the Mojo repositories in it, reads each one's git history, and serves a web console that shows how Mojo libraries have grown. Everything in it is Mojo, in one pixi project. An agent built it from a prompt, with only the skills as documentation.

See it running

The example runs on AWS, deployed with orca aws as one image of 71.1 MB compressed, which holds the scheduler, the three pipelines and the console. Both sites are public to read.

Orca's UI orca.mlake.dev The three pipelines, their runs and logs, the Dista timeline of a run's tasks, the lineage of each table, and links to the code that made them. Mojo census census.mlake.dev How Mojo libraries grew: their size and age, the Mojo versions and tooling they moved to over time, and a page per repository.

How it fits together

Three Orca pipelines write five Iceberg tables. gharchive runs on a schedule; each day it commits starts mojo_repos, and each batch of new repositories starts mojo_history. The console reads the last three tables.

The pipelines and the tables they write gharchive imports each day of data.gharchive.org into gharchive.events. mojo_repos, a sensor on it, checks candidates with GitHub's API and writes mojo.repos and mojo.checked. mojo_history, a sensor keyed by owner/name on mojo.repos, clones each repository in a dista task and writes mojo.commits and mojo.migrations. Mojo census, the console at census.mlake.dev, reads those tables, and Orca's UI at orca.mlake.dev shows the three pipelines. data.gharchive.org an hourly .json.gz of GitHub's public events gharchive daily at 03:00 UTC · a dista task per hour gharchive.events every event, partitioned by day mojo_repos sensor · checks candidates with GitHub's API mojo.repos the Mojo repositories mojo.checked every answer, asked once mojo_history sensor · keyed by owner/name, 50 a run a dista task clones each repository mojo.commits Mojo lines, tooling, version mojo.migrations each change of either read by a flare app runs, logs, lineage console census.mlake.dev Orca's UI orca.mlake.dev

Build it yourself

Two ways, one project. Each page stands on its own.

Let an agent build it Install the skills for Mojo, magmalake, mojoshelf and AWS, then give a coding agent the prompt that built mojo-census. Step by step Install pixi and mojoshelf, create the project, read the code the agent wrote, run it locally, and deploy it to AWS. Or clone the example and run it as it is.