# Every tin, layer by layer > Every magmalake tin, layer by layer — execution, tables and catalogs, file formats, storage, primitives and utilities — each linking to its own page: what it is, how to install it, what proves it and how fast it is. At the top, execution: Orca schedules pipelines, Dista runs Mojo functions across a cluster, and restate makes a flow durable. Beneath it, Iceberg tables and their catalogs; then the file formats a table is made of, and the storage and databases it lives in; then the codecs, hashes and threads they all share, and small utilities beside them. I kept every piece separate, so you can take just the one you need. magmalake's own tins are Apache-2.0 and CI-tested on stable Mojo 1.2.0 and the current nightly, across macOS and Linux; the three republished ones keep upstream's MIT licence. Every tin but xml is published on mojoshelf. Each of magmalake's own carries a correctness oracle — an outside implementation I made it agree with. Its page says which, with its numbers and its install line. Execution - [Orca 0.3.2 pipelines as DAGs · cron schedules · backfills](https://magmalake.dev/s/orca) - [Dista 0.2.1 distributed tasks · actors · results that stay on workers](https://magmalake.dev/s/dista) - [restate 0.4.1 durable execution](https://magmalake.dev/s/restate) Tables & catalogs - [iceberg 0.8.6 metadata · scan planning · read · write · REST, SQL and filesystem catalogs](https://magmalake.dev/s/iceberg) File formats - [parquet 0.11.7 read · write · every encoding · Arrow out](https://magmalake.dev/s/parquet) - [parquet-full 0.1.8 parquet · plus ZSTD, BROTLI and LZ4](https://magmalake.dev/s/parquet-full) - [avro 0.4.4 object container files, both ways](https://magmalake.dev/s/avro) - [arrow-mlake 0.2.1 Arrow layout · C Data Interface · IPC](https://magmalake.dev/s/arrow-mlake) - [roaring 0.1.3 32/64-bit · deletion vectors](https://magmalake.dev/s/roaring) Storage & databases - [objectstore 0.4.1 local · HTTP · S3 · GCS · Azure · concurrent reads](https://magmalake.dev/s/objectstore) - [sqlite 0.3.5 embedded SQL · the catalog store](https://magmalake.dev/s/sqlite) - [postgres 0.3.3 libpq · params · COPY](https://magmalake.dev/s/postgres) Primitives - [thrift 0.1.3 compact + binary protocol](https://magmalake.dev/s/thrift) - [zstd 0.1.5 libzstd](https://magmalake.dev/s/zstd) - [lz4 0.1.4 liblz4](https://magmalake.dev/s/lz4) - [brotli 0.1.4 libbrotli](https://magmalake.dev/s/brotli) - [snappy 0.1.4 pure Mojo](https://magmalake.dev/s/snappy) - [deflate 0.1.3 DEFLATE · gzip · pure Mojo](https://magmalake.dev/s/deflate) - [hashes 0.1.3 CRC-32 · murmur3 · XXH64](https://magmalake.dev/s/hashes) - [memory-region 0.1.1 bytes addressed by offset · shared mappings](https://magmalake.dev/s/memory-region) - [threads 0.6.1 pthreads · atomics · parallel\_for](https://magmalake.dev/s/threads) Utilities - [xml from source pull parser · path queries](https://magmalake.dev/s/xml) - [lint 0.1.4 mojolint · origin and threading mistakes](https://magmalake.dev/s/lint) - [k8s 0.1.2 the Kubernetes API from Mojo](https://magmalake.dev/s/k8s) Cross-implementation oracle - [iceberg-rs 0.1.3 iceberg-rust behind a C ABI](https://magmalake.dev/s/iceberg-rs) Republished from upstream - [json\_mlake 0.4.3 simdjson-backed JSON · ehsanmok/json](https://magmalake.dev/s/json%5Fmlake) - [flare\_mlake 0.13.2 TCP · TLS · HTTP/1–3 · gRPC · ehsanmok/flare](https://magmalake.dev/s/flare%5Fmlake) - [mozz\_mlake 0.2.2 fuzzing and property tests · ehsanmok/mozz](https://magmalake.dev/s/mozz%5Fmlake) Each box is a tin with its own page, its own tests and its own release cadence, and each layer uses only the ones below it; the number beside its name is the version on mojoshelf. Outside the diagram there is only libzstd, liblz4, libbrotli, libcurl, libpq, SQLite, the Restate Rust SDK and the system pthreads, plus OpenSSL and simdjson under the republished packages. A box marked _from source_ works today but isn't on mojoshelf yet. Beside the stack are a second Iceberg implementation kept only to check the native one, and three upstream libraries republished so the tins can depend on them. ## From a table format to a platform magmalake began when Mojo reached 1.0 as an exploration in building a meaningfully large data library in it? — and I picked [Apache Iceberg](https://iceberg.apache.org) as the target. What came out is a stack rather than a demo. Parquet, Avro, Thrift, Iceberg itself, Roaring bitmaps, Snappy and the hashes are implemented in Mojo. A few very mature C libraries are linked rather than reimplemented — **libzstd**, **liblz4** and **libbrotli** for those three codecs, **libcurl** for HTTP and TLS, and **pthreads** for the threads the language does not yet provide. What is gone is the Python and JVM layer that every other route from Mojo to Iceberg has to go through. Then it grew upward. [Orca](https://magmalake.dev/s/orca) is the orchestration layer: pipelines as DAGs in Mojo, checked by the compiler, partitioned by time or arbitrary keys and run on cron schedules, with a scheduler that backfills and retries. It loads every month of NYC taxi trips since 2011, 1.44 billion rows, four runs at a time. [Dista](https://magmalake.dev/s/dista), beneath it, runs Mojo functions distributed across a cluster. On those taxi rows in S3 it beats both Ray and a Rust implementation of the same queries. ### Packaged on mojoshelf Magmalake leans on the packaging infrastructure defined by [mojoshelf.org](https://mojoshelf.org) — a registry for Mojo _tins_ on top of pixi. Nearly every tin here is published there and installable with one command. [mojoshelf.org](https://mojoshelf.org) ### Skills for coding agents What magmalake learned making its code fast is packaged as an agent skill, so an agent writing a data pipeline in Mojo starts with it. Each rule carries the measurement behind it. ``` npx skills add magmalake/skills ``` That installs it. In Claude Code you can instead run `/plugin marketplace add magmalake/skills` and then `/plugin install skills@magmalake`. [mojo-data-performance](https://github.com/magmalake/skills/tree/main/mojo-data-performance) covers a pipeline's per-row loops and the readers, writers and codecs under them, how to tell the time spent decoding from the time spent on your own work without a second implementation to compare against, and how to take a benchmark number that survives being checked. ### Take a tin Each of the 27 repositories is independently usable and independently installable. Start with whichever layer you need — you do not have to take the stack. [github.com/magmalake](https://github.com/magmalake) [mojoshelf.org](https://mojoshelf.org)