Performance

Measured, including where it loses

Most of the rows on this site were measured on one machine, an Apple M4 with ten cores, four of them performance, with nothing else building on it. Same files, and every row reproducible from its own repository. The few below say the most, in both directions: magmalake beats pyarrow on flat columnar Parquet reads, on one core and on four; it loses on nested and mixed data; and an Iceberg scan is still behind pyarrow at matched thread counts. Each tin's page has the rest of its table.

Highlights

The strongest numbers and the clearest losses. Every row links to the tin it belongs to, where the rest of its table is.

Operation	magmalake	Reference
Parquet read, 1M rows — 1 core int64, double and two dictionary columns, uncompressed, four row groups parquet →	3.77 ms p50 / 3.83 p90 — 265M rows/s	pyarrow, one thread: 8.0 ms — we are 2.1× faster
Parquet read, 1M rows — 4 workers the same file, ParquetReader.num_workers = 4 parquet →	1.96 ms p50 / 2.07 p90 — 510M rows/s	pyarrow, threaded across all 10 CPUs: 2.57 ms — we are 1.3× faster while using four threads to its ten
Parquet read, 100k rows, nested and mixed int64, double, string, bool and a list<int32>, snappy — 1 core / 4 workers parquet →	2.63 ms / 0.98 ms — 0.78 ms at eight workers	pyarrow 2.24 ms one thread, 0.66 ms threaded — 1.17× and 1.18× faster than us. Still the file we lose, but by a fifth rather than by half.
Iceberg scan, 1M rows — 1 core / 4 workers zstd, six columns; best-of-three warm, not the percentile harness iceberg →	30.3 ms / 9.9 ms	pyarrow on the same four data files: 27.3 ms one thread, 8.8 ms threaded — about 1.1× ahead of us at either thread count. PyIceberg 0.11.1 does the whole scan in 8.0 ms on pyarrow's pool.
Avro decode, manifest-shaped records the schema-compiled cursor, 1 core avro →	18.1M records/s — 25.9M with field selection	fastavro 1.12.2 on the same file: 1.70M — 10.7× faster
SHA-256 pure Mojo with ARMv8 crypto intrinsics; a single timed pass, median of three objectstore →	2.70 GB/s — 590 MB/s scalar fallback	OpenSSL 3.17 GB/s on the same 64 MiB, in the same process
The same 200 ranges, coalesced adjacent spans asked for at once — the shape a Parquet scan actually has objectstore →	0.022 ms local / 0.029 ms signed S3, per range	6.7× and 17× cheaper per range than asking one at a time
Dista: eight taxi queries over 1.4 billion rows in S3 3 × AWS c7g.4xlarge (16 Graviton3 vCPUs, median pass, 1 November 2026 — measured on AWS, not on the M4 Dista →	12.0 s a pass	Rust 28.7 s, Ray 38.6 s on the same queries, every answer matching

scroll the table sideways →

Each tin's numbers

Crossing a process boundary — Arrow Flight, the C Data Interface and shared memory — is on the memory-region page.

How these were measured

Almost every number is a p50 over real iterations, with the p90 beside it where the spread is the point — not a mean, which one slow sample can move by 30%, and not a best-of-N. The two repos whose benchmarks are not on that harness yet, iceberg and objectstore, say so in their own rows. Each benchmark is compiled and then run: JIT-executing a suite with mojo run inflated one of these figures by 1.9×, and comparing a JIT number against a compiled one produced two confident wrong diagnoses before that was found. Rows are taken per benchmark rather than from one pass over a whole suite, and the harness times a reference kernel either side of every benchmark and reads the load average, so a run taken on a busy machine says so instead of quietly reporting inflated numbers. Where a row measures a C library rather than Mojo, it says so.

“Single core” is no longer a property of this stack. ParquetReader.num_workers and ScanOptions.num_workers both exist now, so every row carries its own thread count — and so does every pyarrow reference, which is quoted either as one thread (set_cpu_count(1), set_io_thread_count(1), use_threads=False) or as threaded, pyarrow's own default. On the 1M-row file those two legs are 2.9× apart, so a comparison that does not name the leg is not a comparison.

They were measured on stable Mojo 1.0.0. The toolchain has since moved to 1.1.0 and 1.2.0, and these numbers were not re-taken for either.

Benchmark history

The tables are hand-measured; these are not. Those numbers were taken in one sitting on the M4, which is why they carry a date rather than a build number. Every merge also runs the same benchmarks on a GitHub runner and appends to a history each repository publishes itself:

Expect those to be slower and noisier than the tables — a shared runner is not a workstation. They are worth reading as trends, not as absolute figures: each history is keyed by machine, so a run on a laptop and a run on CI stay separate series and are never averaged, and a benchmark is only flagged as regressed when it falls further than its own recent variance.

Reproducing any of this

Every tin carries its own benchmarks and runs them through the same harness, bench.mojo. Clone the repo a row belongs to and run:

git clone https://github.com/magmalake/parquet.mojo
cd parquet.mojo

pixi run -e bench bench                    # the table, with spread
pixi run -e bench bench -- --json          # every repetition, machine-readable
pixi run -e bench bench -- --list          # what this repo measures
pixi run -e bench bench -- --only bench_read_wide

The bench environment is stable Mojo 1.2.0: a precompiled Mojo package is stamped with the compiler version that built it, and every tin builds with 1.2.0. The libraries themselves test on stable and nightly.

One caveat on reproducing the numbers exactly: the worker ladders contend with each other if you run them together, so take them one at a time: --only bench_read_wide_w4.

A few benchmarks need a fixture that is generated rather than committed — parquet wants python tools/bench_pyarrow.py --make first, iceberg builds a warehouse, objectstore needs MinIO. Each repo's README says which, and the bench stops with instructions rather than measuring the wrong thing. Expect your absolute numbers to differ from an M4; the ratios are what travel.