magmalake parquet

parquet · API

parquet.encoding

Value encodings: PLAIN, dictionaries, the DELTA family, BYTE_STREAM_SPLIT.

Everything decodes into a PhysBuffer — values still in their physical Parquet form (an INT32 is four little-endian bytes whatever logical type annotates it). parquet.convert is what turns a PhysBuffer plus definition levels into an Arrow buffer.

Three physical shapes cover all eight Parquet types:

  • PK_BOOL — a bit-packed bitmap, LSB first, one bit per value.
  • PK_FIXED — count * width bytes; width is 4, 8 or 12 for the numeric types and type_length for FIXED_LEN_BYTE_ARRAY.
  • PK_VAR — count + 1 offsets into a data buffer, for BYTE_ARRAY.

Structs

PhysBuffer

struct PhysBuffer

Decoded values still in their physical Parquet representation.

Implements Copyable, Defaultable, Movable

Fields
kind: Int
width: Int
count: Int
bytes: List[UInt8]
offsets: List[Int]

__init__

def __init__(out self)
def __init__(out self, kind: Int, width: Int)

bool_at

def bool_at(self, i: Int) -> Bool

append_bool

def append_bool(mut self, v: Bool)

append_bytes

def append_bytes(mut self, src: Span[UInt8])

Append one PK_VAR value.

reserve_values

def reserve_values(mut self, count: Int, bytes: Int)

Size the buffers for count more values and bytes more bytes, so that a chunk of many pages does not grow its buffer log-many times.

extend

def extend(mut self, other: Self)

Append every value of other, which must have the same shape.

value_span

def value_span(self, i: Int) -> Span[UInt8, origin_of(self.bytes)]

The bytes of value i — PK_VAR or PK_FIXED only.

Functions

physical_width

def physical_width(phys: Int32, type_length: Int) -> Int

Bytes per value for a fixed-width physical type; 0 for the others.

physical_kind

def physical_kind(phys: Int32) -> Int

decode_plain

def decode_plain(phys: Int32, type_length: Int, data: Span[UInt8], count: Int) -> PhysBuffer

PLAIN: fixed-width values little-endian, byte arrays length-prefixed.

decode_plain_into

def decode_plain_into(mut out: PhysBuffer, phys: Int32, type_length: Int, data: Span[UInt8], count: Int)

PLAIN-decode count values onto the end of out.

Decoding straight into the chunk's buffer is what keeps a many-page column from copying every byte twice: once into the page's own buffer and again when the page is concatenated.

gather

def gather(dict: PhysBuffer, indices: List[UInt32]) -> PhysBuffer

Materialise dictionary-encoded values from a dictionary page.

gather_into

def gather_into(mut out: PhysBuffer, dict: PhysBuffer, indices: List[UInt32])

Gather indices out of dict onto the end of out.

Fixed-width entries are a straight gather — one load and one store of the whole element per index. Byte arrays take two passes: the lengths into the offsets, then the bytes.

decode_dict_indices

def decode_dict_indices(data: Span[UInt8], count: Int) -> List[UInt32]

RLE_DICTIONARY / PLAIN_DICTIONARY data: a bit width byte, then a hybrid RLE run of that many indices.

gather_dict_into

def gather_dict_into(mut out: PhysBuffer, dict: PhysBuffer, data: Span[UInt8], count: Int)

Decode count dictionary indices out of data and gather them out of dict onto the end of out, a block at a time.

decode_dict_indices followed by gather_into is the same work in three passes over a page-sized List[UInt32]: one to write it, one to bounds-check it, one to gather it — four bytes per value written to memory and read back twice, plus the allocation. This walks the hybrid runs into 4 KiB of scratch instead, and each block is checked and gathered while it is still in L1. parquet-cpp's GetBatchWithDict and arrow-rs's RleDecoder::get_batch_with_dict are both this shape.

The bounds check is the same check in the same order — blocks come off the stream in order and _check_dict_block reports the first offending index in the first offending block — so a corrupt file raises exactly what it raised before. Unlike gather_into the check is no longer complete before the first value is written, so out may hold part of the page when this raises; the caller is abandoning the read either way.

decode_rle_bool

def decode_rle_bool(data: Span[UInt8], count: Int) -> PhysBuffer

The RLE value encoding for booleans: a 4-byte length, then a hybrid run of 1-bit values.

decode_delta_ints

def decode_delta_ints(data: Span[UInt8], count: Int, mut end: Int) -> List[Int64]

DELTA_BINARY_PACKED, the block / miniblock delta encoding.

header    := block_size_in_values, miniblocks_per_block, total_count, first_value
block     := min_delta, list-of-bitwidths-of-miniblocks, miniblocks

All four header fields are ULEB128, first_value and min_delta zigzag. Values wrap around at 64 bits, exactly as the spec says.

decode_delta_binary_packed

def decode_delta_binary_packed(data: Span[UInt8], count: Int, width: Int) -> PhysBuffer

decode_delta_length_byte_array

def decode_delta_length_byte_array(data: Span[UInt8], count: Int) -> PhysBuffer

Lengths as DELTA_BINARY_PACKED, then the bytes back to back.

decode_delta_byte_array

def decode_delta_byte_array(data: Span[UInt8], count: Int) -> PhysBuffer

Prefix lengths, suffix lengths, then the suffix bytes: each value is the first prefix bytes of the value before it plus its own suffix.

decode_byte_stream_split

def decode_byte_stream_split(data: Span[UInt8], count: Int, width: Int) -> PhysBuffer

BYTE_STREAM_SPLIT: width streams of count bytes, the k-th stream holding byte k of every value.

Aliases

comptime PK_BOOL = 0
comptime PK_FIXED = 1
comptime PK_VAR = 2
comptime MAX_ARROW_VAR_BYTES = 2147483647

How many bytes of BYTE_ARRAY data one Arrow array can address.

Arrow's binary/string layout addresses value bytes with 32-bit offsets, so this is the ceiling on a record batch, not on a column chunk: PhysBuffer.offsets is List[Int], wide enough for anything a chunk can hold, and the reader cuts a batch short before its value bytes reach this. Keeping the physical buffer wide is what makes that cut possible at all — an overflow that has already happened during decode cannot be split away afterwards.