parquet · API

parquet.convert

Physical Parquet values to Arrow buffers.

A Parquet leaf's physical type and its Arrow type are often different widths: an INT32 annotated Int(8, true) is one byte in Arrow, a DECIMAL over FIXED_LEN_BYTE_ARRAY(7) is a sign-extended little-endian 16-byte integer, and INT96 is a nanosecond timestamp built from a Julian day and a time of day. append_value and append_null do that translation one value at a time, straight into the destination ArrayData's buffers.

Functions

int96_to_nanos

def int96_to_nanos(src: Span[UInt8]) -> Int64

INT96: 8 little-endian bytes of nanoseconds within the day, then a little-endian Julian day number.

append_null

def append_null(mut out: ArrayData, leaf: LeafColumn)

Append a null slot, keeping every buffer the right length.

Validity is the caller's business — parquet.assemble maintains it, and only materialises a bitmap once a null actually turns up.

append_value

def append_value(mut out: ArrayData, leaf: LeafColumn, vals: PhysBuffer, vi: Int)

Append value vi of vals, translated to out's Arrow type.

Aliases

comptime JULIAN_UNIX_EPOCH = 2440588
comptime NANOS_PER_DAY = 86400000000000