Let an agent build it

A coding agent can write all of mojo-census if it knows current Mojo, the magmalake libraries and AWS. Each of those comes as a skill, and AWS also as an MCP server. The commands below are for Claude Code; any agent that reads Agent Skills can use the npx skills lines instead, which need Node.js.

Install the skills

The Mojo skills come in one install. magmalake's magmalake plugin holds orca, dista, magmalake-apps (a whole project, its lake, a flare app and orca aws) and mojo-data-performance (a pipeline's per-row loops and the readers, writers and codecs under them, and how to take a benchmark number that survives being checked). It also brings two skills written by others: Modular's mojo-syntax, which keeps the agent writing the Mojo of today rather than the Mojo it was trained on, and mojoshelf's mojoshelf-consume, which tells it how to find tins and add them to a project. In Claude Code 2.1.275 or later:

/plugin install magmalake --marketplace magmalake/skills

In an older Claude Code:

/plugin marketplace add magmalake/skills
/plugin install magmalake@magmalake

In any other agent, such as Codex or Cursor, this installs all six:

npx skills add magmalake/skills

AWS publishes the Agent Toolkit for AWS: the AWS MCP Server, through which the agent reads AWS documentation and runs AWS calls with your credentials, and a set of AWS skills. It needs uv, and the AWS CLI signed in (aws login) for anything beyond reading documentation. In Claude Code one plugin brings both:

/plugin install aws-core@claude-plugins-official
/reload-plugins

Elsewhere, add the server to the agent's MCP configuration and install the skills. This is AWS's line for Claude Code with your own credentials; the setup guide has the other agents', and the options for sign-in in the browser and for more than one profile. Set AWS_REGION to your project's Region:

claude mcp add-json aws-mcp '{"type":"stdio","command":"uvx","args":["mcp-proxy-for-aws-cli@latest","https://aws-mcp.us-east-1.api.aws/mcp","--metadata","AWS_REGION=us-east-2"],"env":{}}'
npx skills add aws/agent-toolkit-for-aws/skills

The prompt

Start the agent in an empty directory and give it this. It names the skills to use, the three pipelines and their tables, the console, and the order to work in: locally on a few hours of the archive first, then on AWS.

Build me a lake of GitHub activity with magmalake, and an app that shows how
Mojo libraries have grown.

- Make a new pixi project with orca (`pixi shelf add orca-mojo`). Use the
  magmalake skills (orca, dista, magmalake-apps), mojoshelf's skills and
  Modular's mojo-syntax.
- gharchive: a daily pipeline that imports each day of the GitHub Archive
  (data.gharchive.org, 24 hourly .json.gz files) into an Iceberg table
  gharchive.events (id, type, created_at, repo, actor, the event's payload
  as JSON), partitioned by day, from 2023-01-01, every day at 03:00 UTC for
  the day before.
- mojo_repos: a sensor on gharchive.events that finds Mojo repositories
  (pushes touching .mojo or .🔥 files, and projects whose pixi.toml or
  mojoproject.toml names a Mojo package) and keeps them in mojo.repos (when
  GitHub says each was created, when it was first seen, how it was found,
  stars, forks, license, default branch, description).
- mojo_history: keyed by repository, 50 a run, started by new repositories:
  clone each one in a dista task and read its history into mojo.commits
  (lines of Mojo at each commit, which tooling the project used: none, then
  magic, then pixi, and the Mojo version its lock file or manifest named)
  and mojo.migrations (each change of tooling or Mojo version). Versions
  sort in one order across Modular's renumbering (24.x, 25.x, 0.25.x,
  0.26.x, 1.x); test that.
- The console: a flare app over those tables: the libraries by size and
  age, the Mojo versions they moved to over time, a page per repository.
- Run it locally first on a few hours of the archive. Then deploy it to AWS
  with `orca aws` (on-demand workers, autoscaling from 0), backfill one
  month to see what it costs, then the rest, and serve the console on its
  own hostname. The tables go in the stack's public bucket.

The agent that wrote mojo-census from this prompt ran it locally on two hours of 2024-06-12: 476,227 events (154 MB of Parquet), 23 candidate repositories, 4 of them Mojo, and 3 histories read, with 372 commits and 27 changes of tooling or version. It also kept a list of where the skills and libraries were wrong or silent, GAPS.md, and the skills and libraries have since been corrected from it. Expect your agent to take its own route to the same tables; Step by step walks through the route this one took.