magmalake parquet

Tins · File formats

parquet

The first native Apache Parquet reader and writer in Mojo. It decodes the footer, the page headers, the levels and the values itself, and hands the result back in Arrow memory layout, exportable over the Arrow C Data Interface. That interface is the boundary, not a full Arrow library: there are no compute kernels, no IPC and no Flight.

Install 
pixi shelf add parquet-mojo
Version

0.11.7 on mojoshelf

Source

GitHub README Benchmark history

What it does

Works today

  • Every physical and logical type, every encoding, v1 and v2 pages
  • UNCOMPRESSED, SNAPPY and GZIP with no FFI; ZSTD, BROTLI and LZ4 with parquet-full
  • Nested lists, maps and structs reconstructed from definition and repetition levels
  • Row-group, page-index and bloom-filter pruning; field-id projection for Iceberg
  • Arrow C Data Interface export, verified by pyarrow importing the arrays

Not yet

  • More than 2 GiB of string data in one Parquet column chunk

Correctness

Checked against pyarrow, value-exact on 33 fixtures; pyarrow reads back every file it writes

Value-exact against pyarrow on 33 fixtures — every value of every column — and pyarrow reads back every file magmalake writes.

The Arrow export imports through pyarrow.Array._import_from_c.

Building it found bugs upstream

Holding three implementations against each other surfaces disagreements, and not all of them turned out to be my fault. I cut each one below down to a standalone script or unit test with no magmalake code in it, then either reported it upstream or — where someone had got there first — matched it to the report that already existed.

  • Apache Arrow apache/arrow#51097 reported The Parquet writer undercounts nulls for a fixed-width leaf under list<struct> when some lists are null or empty. The BYTE_ARRAY leaf beside it in the same struct is right, so two leaves disagree about the same level records.

Also in File formats