parquet · API

parquet.codec

Page compression codecs.

ParquetReader is parametrised on a CodecSet, a compile-time table of page decompressors, exactly like avro.mojo's block codecs. The default set covers everything that needs no FFI:

codec where it comes from
UNCOMPRESSED —
SNAPPY snappy.mojo, pure Mojo
GZIP deflate.mojo, pure Mojo, with the gzip wrapper parsed here

ZSTD, BROTLI, LZ4_RAW and the legacy Hadoop-framed LZ4 need zstd.mojo, brotli.mojo and lz4.mojo, which dlopen libzstd, libbrotli and liblz4, so they live in parquet_full and its AllCodecs:

from parquet import ParquetReader
# -I ../zstd.mojo/src -I ../lz4.mojo/src -I ../brotli.mojo/src
from parquet_full import AllCodecs

var r = ParquetReader[AllCodecs].open("part-0.parquet")

Between the two sets that is every codec the Parquet spec defines. A reader on DefaultCodecs that meets one of the four FFI codecs raises a message naming it and pointing at AllCodecs; every other column of that file still reads.

Structs

DefaultCodecs

struct DefaultCodecs

UNCOMPRESSED, SNAPPY and GZIP — no FFI, no shared libraries.

Implements CodecSet, Movable

supports

def supports(codec: Int32) -> Bool

decompress

def decompress[page_origin: ImmOrigin, scratch_origin: MutOrigin](codec: Int32, data: Span[UInt8, page_origin], uncompressed_size: Int, ref[scratch_origin] scratch: List[UInt8]) -> Span[UInt8, origin_of(page_origin, scratch_origin)].Immutable

compress

def compress(codec: Int32, data: Span[UInt8]) -> List[UInt8]

Traits

CodecSet

A compile-time table of Parquet page decompressors.

supports

def supports(codec: Int32) -> Bool

Whether this set can decompress CompressionCodec value codec.

decompress

def decompress[page_origin: ImmOrigin, scratch_origin: MutOrigin](codec: Int32, data: Span[UInt8, page_origin], uncompressed_size: Int, ref[page_origin] scratch: List[UInt8]) -> Span[UInt8, origin_of(page_origin, scratch_origin)].Immutable

Decompress one page body, borrowing rather than allocating.

The result is a view, never a fresh buffer. For UNCOMPRESSED it is data itself — the page's own bytes inside the file, handed back untouched, which is what SerializedPageReader::DecompressIfNeeded does in parquet-cpp. For every other codec it is scratch, the caller's one decompression buffer per column chunk, grown in place and never shrunk — parquet-cpp's decompression_buffer_, resized with shrink_to_fit=false (column_reader.cc:564).

Lifetime contract — the span is valid only until the next call that passes the same scratch. That call overwrites the buffer and may reallocate it, so a span held across a page boundary reads another page's bytes. parquet-cpp documents the same rule at column_reader.cc:251: the previous page is invalidated by NextPage(). Copy out anything that has to outlive the page — decode_plain_into and its neighbours already do, which is why a dictionary page survives the data pages decompressed after it.

Mojo's origins cannot make the compiler enforce that. The returned origin is the union of the file's and the scratch buffer's, so both are kept alive for as long as the span is and it can never dangle, but Mojo does not treat an outstanding immutable view as blocking a later mutation the way an exclusivity checker would — a held span is a wrong answer, not a compile error. So the rule is a contract, and test_a_page_span_is_only_valid_until_the_next_page is what states it.

uncompressed_size is the page header's declared decompressed length, which for Brotli and raw LZ4 is the only size there is.

compress

def compress(codec: Int32, data: Span[UInt8]) -> List[UInt8]

Compress one page body. Used by parquet.writer.

Functions

copy_of

def copy_of(data: Span[UInt8]) -> List[UInt8]

page_span

def page_span[origin: ImmOrigin](data: Span[UInt8]) -> Span[UInt8, origin]

Re-label data with the origin CodecSet.decompress returns.

Its two answers live in different places — the file for an uncompressed page, the caller's scratch buffer for a compressed one — and the declared return type is the union of those two origins, which keeps both alive for as long as the span is. Neither branch can produce that union type on its own, hence this one narrow re-labelling, and it is a no-op at run time: the same pointer and length.

Sound because the union outlives either input, so the origin stamped on is never longer-lived than the one the bytes actually have.

sized_scratch

def sized_scratch[origin: MutOrigin](ref[origin] scratch: List[UInt8], n: Int) -> Span[UInt8, origin]

scratch, resized to exactly n bytes and ready to be written over.

Grows in place and never shrinks the allocation, which is what Resize(uncompressed_len, /*shrink_to_fit=*/false) buys parquet-cpp: after the first page of a chunk, a page no larger than the biggest one so far costs no allocation at all. The bytes are left uninitialised rather than zeroed — a decompressor writes every one of them, and a memset per page would be most of what this commit is trying to remove.

codec_name

def codec_name(codec: Int32) -> String

unsupported_codec

def unsupported_codec(codec: Int32) -> Error

gunzip

def gunzip(data: Span[UInt8]) -> List[UInt8]

Strip a gzip (RFC 1952) wrapper and inflate the DEFLATE stream inside.

Parquet's GZIP codec is gzip-framed, not a bare DEFLATE stream, so the 10-byte header and its optional extra/name/comment/CRC fields have to come off first. A zlib (RFC 1950) stream is accepted too — some writers emit one — and so is a bare DEFLATE stream.

A gzip stream may be several members concatenated (RFC 1952 §2.2), and a page written that way decodes to the members joined in order. Stopping at the first one silently loses the rest, so the members are looped over here.

gunzip_into

def gunzip_into(data: Span[UInt8], mut out: List[UInt8])

gunzip into a buffer the caller already has, replacing its contents.

The gzip-framed path — every GZIP page a Parquet writer emits — inflates each member onto the end of out, so a decompression buffer reused across a chunk keeps whatever capacity it has already grown. The zlib and bare DEFLATE fallbacks hand out a buffer inflate allocated: neither shape comes out of a Parquet writer, and neither is worth a second copy to keep the growth in place.

gzip_wrap

def gzip_wrap(data: Span[UInt8]) -> List[UInt8]

Wrap a DEFLATE stream in the gzip (RFC 1952) framing Parquet's GZIP codec expects: a 10-byte header, then the deflate bits, then the CRC32 of the uncompressed data and its length modulo 2^32.