# parquet.encoding > A module of parquet's API. Value encodings: PLAIN, dictionaries, the DELTA family, BYTE_STREAM_SPLIT. Everything decodes into a `PhysBuffer` — values still in their *physical* Parquet form (an `INT32` is four little-endian bytes whatever logical type annotates it). `parquet.convert` is what turns a `PhysBuffer` plus definition levels into an Arrow buffer. Three physical shapes cover all eight Parquet types: * `PK_BOOL` — a bit-packed bitmap, LSB first, one bit per value. * `PK_FIXED` — `count * width` bytes; `width` is 4, 8 or 12 for the numeric types and `type_length` for `FIXED_LEN_BYTE_ARRAY`. * `PK_VAR` — `count + 1` offsets into a data buffer, for `BYTE_ARRAY`. ## Structs ### `PhysBuffer` ```mojo struct PhysBuffer ``` Decoded values still in their physical Parquet representation. Implements: `Copyable`, `Defaultable`, `Movable` Fields: - `kind: Int` - `width: Int` - `count: Int` - `bytes: List[UInt8]` - `offsets: List[Int]` #### `__init__` ```mojo def __init__(out self) ``` ```mojo def __init__(out self, kind: Int, width: Int) ``` #### `bool_at` ```mojo def bool_at(self, i: Int) -> Bool ``` #### `append_bool` ```mojo def append_bool(mut self, v: Bool) ``` #### `append_bytes` ```mojo def append_bytes(mut self, src: Span[UInt8]) ``` Append one `PK_VAR` value. #### `reserve_values` ```mojo def reserve_values(mut self, count: Int, bytes: Int) ``` Size the buffers for `count` more values and `bytes` more bytes, so that a chunk of many pages does not grow its buffer log-many times. #### `extend` ```mojo def extend(mut self, other: Self) ``` Append every value of `other`, which must have the same shape. #### `value_span` ```mojo def value_span(self, i: Int) -> Span[UInt8, origin_of(self.bytes)] ``` The bytes of value `i` — `PK_VAR` or `PK_FIXED` only. ## Functions ### `physical_width` ```mojo def physical_width(phys: Int32, type_length: Int) -> Int ``` Bytes per value for a fixed-width physical type; 0 for the others. ### `physical_kind` ```mojo def physical_kind(phys: Int32) -> Int ``` ### `decode_plain` ```mojo def decode_plain(phys: Int32, type_length: Int, data: Span[UInt8], count: Int) -> PhysBuffer ``` PLAIN: fixed-width values little-endian, byte arrays length-prefixed. ### `decode_plain_into` ```mojo def decode_plain_into(mut out: PhysBuffer, phys: Int32, type_length: Int, data: Span[UInt8], count: Int) ``` PLAIN-decode `count` values onto the end of `out`. Decoding straight into the chunk's buffer is what keeps a many-page column from copying every byte twice: once into the page's own buffer and again when the page is concatenated. ### `gather` ```mojo def gather(dict: PhysBuffer, indices: List[UInt32]) -> PhysBuffer ``` Materialise dictionary-encoded values from a dictionary page. ### `gather_into` ```mojo def gather_into(mut out: PhysBuffer, dict: PhysBuffer, indices: List[UInt32]) ``` Gather `indices` out of `dict` onto the end of `out`. Fixed-width entries are a straight gather — one load and one store of the whole element per index. Byte arrays take two passes: the lengths into the offsets, then the bytes. ### `decode_dict_indices` ```mojo def decode_dict_indices(data: Span[UInt8], count: Int) -> List[UInt32] ``` `RLE_DICTIONARY` / `PLAIN_DICTIONARY` data: a bit width byte, then a hybrid RLE run of that many indices. ### `gather_dict_into` ```mojo def gather_dict_into(mut out: PhysBuffer, dict: PhysBuffer, data: Span[UInt8], count: Int) ``` Decode `count` dictionary indices out of `data` and gather them out of `dict` onto the end of `out`, a block at a time. `decode_dict_indices` followed by `gather_into` is the same work in three passes over a page-sized `List[UInt32]`: one to write it, one to bounds-check it, one to gather it — four bytes per value written to memory and read back twice, plus the allocation. This walks the hybrid runs into 4 KiB of scratch instead, and each block is checked and gathered while it is still in L1. parquet-cpp's `GetBatchWithDict` and arrow-rs's `RleDecoder::get_batch_with_dict` are both this shape. The bounds check is the same check in the same order — blocks come off the stream in order and `_check_dict_block` reports the first offending index in the first offending block — so a corrupt file raises exactly what it raised before. Unlike `gather_into` the check is no longer complete *before* the first value is written, so `out` may hold part of the page when this raises; the caller is abandoning the read either way. ### `decode_rle_bool` ```mojo def decode_rle_bool(data: Span[UInt8], count: Int) -> PhysBuffer ``` The `RLE` value encoding for booleans: a 4-byte length, then a hybrid run of 1-bit values. ### `decode_delta_ints` ```mojo def decode_delta_ints(data: Span[UInt8], count: Int, mut end: Int) -> List[Int64] ``` `DELTA_BINARY_PACKED`, the block / miniblock delta encoding. ```text header := block_size_in_values, miniblocks_per_block, total_count, first_value block := min_delta, list-of-bitwidths-of-miniblocks, miniblocks ``` All four header fields are ULEB128, `first_value` and `min_delta` zigzag. Values wrap around at 64 bits, exactly as the spec says. ### `decode_delta_binary_packed` ```mojo def decode_delta_binary_packed(data: Span[UInt8], count: Int, width: Int) -> PhysBuffer ``` ### `decode_delta_length_byte_array` ```mojo def decode_delta_length_byte_array(data: Span[UInt8], count: Int) -> PhysBuffer ``` Lengths as `DELTA_BINARY_PACKED`, then the bytes back to back. ### `decode_delta_byte_array` ```mojo def decode_delta_byte_array(data: Span[UInt8], count: Int) -> PhysBuffer ``` Prefix lengths, suffix lengths, then the suffix bytes: each value is the first `prefix` bytes of the value before it plus its own suffix. ### `decode_byte_stream_split` ```mojo def decode_byte_stream_split(data: Span[UInt8], count: Int, width: Int) -> PhysBuffer ``` `BYTE_STREAM_SPLIT`: `width` streams of `count` bytes, the k-th stream holding byte k of every value. ## Aliases ### `PK_BOOL` ```mojo comptime PK_BOOL = 0 ``` ### `PK_FIXED` ```mojo comptime PK_FIXED = 1 ``` ### `PK_VAR` ```mojo comptime PK_VAR = 2 ``` ### `MAX_ARROW_VAR_BYTES` ```mojo comptime MAX_ARROW_VAR_BYTES = 2147483647 ``` How many bytes of `BYTE_ARRAY` data one Arrow array can address. Arrow's `binary`/`string` layout addresses value bytes with **32-bit** offsets, so this is the ceiling on a *record batch*, not on a column chunk: `PhysBuffer.offsets` is `List[Int]`, wide enough for anything a chunk can hold, and the reader cuts a batch short before its value bytes reach this. Keeping the physical buffer wide is what makes that cut possible at all — an overflow that has already happened during decode cannot be split away afterwards.