magmalake

The tins

Every tin, layer by layer

At the top, execution: Orca schedules pipelines, Dista runs Mojo functions across a cluster, and restate makes a flow durable. Beneath it, Iceberg tables and their catalogs; then the file formats a table is made of, and the storage and databases it lives in; then the codecs, hashes and threads they all share, and small utilities beside them. I kept every piece separate, so you can take just the one you need.

magmalake's own tins are Apache-2.0 and CI-tested on stable Mojo 1.2.0 and the current nightly, across macOS and Linux; the three republished ones keep upstream's MIT licence. Every tin but xml is published on mojoshelf. Each of magmalake's own carries a correctness oracle — an outside implementation I made it agree with. Its page says which, with its numbers and its install line.

Execution
Orca 0.3.2 pipelines as DAGs · cron schedules · backfills Dista 0.2.1 distributed tasks · actors · results that stay on workers restate 0.4.1 durable execution
Tables & catalogs
iceberg 0.8.6 metadata · scan planning · read · write · REST, SQL and filesystem catalogs
File formats
parquet 0.11.7 read · write · every encoding · Arrow out parquet-full 0.1.8 parquet · plus ZSTD, BROTLI and LZ4 avro 0.4.4 object container files, both ways arrow-mlake 0.2.1 Arrow layout · C Data Interface · IPC roaring 0.1.3 32/64-bit · deletion vectors
Storage & databases
objectstore 0.4.1 local · HTTP · S3 · GCS · Azure · concurrent reads sqlite 0.3.5 embedded SQL · the catalog store postgres 0.3.3 libpq · params · COPY
Primitives
thrift 0.1.3 compact + binary protocol zstd 0.1.5 libzstd lz4 0.1.4 liblz4 brotli 0.1.4 libbrotli snappy 0.1.4 pure Mojo deflate 0.1.3 DEFLATE · gzip · pure Mojo hashes 0.1.3 CRC-32 · murmur3 · XXH64 memory-region 0.1.1 bytes addressed by offset · shared mappings threads 0.6.1 pthreads · atomics · parallel_for
Utilities
xml from source pull parser · path queries lint 0.1.4 mojolint · origin and threading mistakes k8s 0.1.2 the Kubernetes API from Mojo
Cross-implementation oracle
iceberg-rs 0.1.3 iceberg-rust behind a C ABI
Republished from upstream
json_mlake 0.4.3 simdjson-backed JSON · ehsanmok/json flare_mlake 0.13.2 TCP · TLS · HTTP/1–3 · gRPC · ehsanmok/flare mozz_mlake 0.2.2 fuzzing and property tests · ehsanmok/mozz
Each box is a tin with its own page, its own tests and its own release cadence, and each layer uses only the ones below it; the number beside its name is the version on mojoshelf. Outside the diagram there is only libzstd, liblz4, libbrotli, libcurl, libpq, SQLite, the Restate Rust SDK and the system pthreads, plus OpenSSL and simdjson under the republished packages. A box marked from source works today but isn't on mojoshelf yet. Beside the stack are a second Iceberg implementation kept only to check the native one, and three upstream libraries republished so the tins can depend on them.

What this is

From a table format to a platform

magmalake began when Mojo reached 1.0 as an exploration in building a meaningfully large data library in it? — and I picked Apache Iceberg as the target. What came out is a stack rather than a demo. Parquet, Avro, Thrift, Iceberg itself, Roaring bitmaps, Snappy and the hashes are implemented in Mojo. A few very mature C libraries are linked rather than reimplemented — libzstd, liblz4 and libbrotli for those three codecs, libcurl for HTTP and TLS, and pthreads for the threads the language does not yet provide. What is gone is the Python and JVM layer that every other route from Mojo to Iceberg has to go through.

Then it grew upward. Orca is the orchestration layer: pipelines as DAGs in Mojo, checked by the compiler, partitioned by time or arbitrary keys and run on cron schedules, with a scheduler that backfills and retries. It loads every month of NYC taxi trips since 2011, 1.44 billion rows, four runs at a time. Dista, beneath it, runs Mojo functions distributed across a cluster. On those taxi rows in S3 it beats both Ray and a Rust implementation of the same queries.

Take a tin

Each of the 27 repositories is independently usable and independently installable. Start with whichever layer you need — you do not have to take the stack.

github.com/magmalake mojoshelf.org