# parquet.page > A module of parquet's API. Page decoding: headers, compression, levels, and the values inside. `read_column_chunk` walks every page of one column chunk — the optional dictionary page and then the data pages, v1 or v2 — and returns the whole chunk as three parallel things: * `defs` — one definition level per *slot*, absent when the leaf's maximum definition level is 0; * `reps` — one repetition level per slot, absent when the maximum repetition level is 0; * `values` — the non-null values, back to back, still physical. Page CRC32s are verified when the writer stored them. Every length in a page header is checked against the bytes that are actually there, so a truncated or doctored file raises instead of reading past the end of the buffer. ## Structs ### `PageWindow` ```mojo struct PageWindow ``` Which pages of a column chunk `read_column_chunk` has to decode. Default-constructed — no page rows, an empty row range — it means *all of them*, which is what every read without page pruning asks for and is the path this reader always had. `first_rows` is `OffsetIndex.page_locations[k].first_row_index`, one entry per **data** page in file order: the number of records of the row group that are complete before page `k` starts. With `first_rows[npages]` taken to be `group_rows`, page `k` can hold values for rows [first_rows[k], first_rows[k + 1]] — closed at the top, because a record can still be *in flight* when a page ends and then has values in both pages. That one row of overhang is the whole difference between a repeated column and a flat one here, and getting it wrong drops list elements at a page seam. A flat column's pages hold `[first_rows[k], first_rows[k + 1])` exactly, one slot per row. Implements: `Copyable`, `Defaultable`, `Movable` Fields: - `first_rows: List[Int]`: Per data page, the first row of the row group it can hold. Empty means no page index, so nothing can be skipped. - `row_lo: Int` - `row_hi: Int`: The rows the caller still wants, as one outer hull. `row_hi <= row_lo` means no restriction. - `group_rows: Int`: Rows in the whole row group — the implied `first_rows[npages]`. #### `__init__` ```mojo def __init__(out self) ``` #### `restricts` ```mojo def restricts(self) -> Bool ``` Is there anything here that could let a page be skipped? #### `page_first_row` ```mojo def page_first_row(self, k: Int) -> Int ``` `first_rows[k]`, with the one-past-the-end entry spelled out. #### `page_range` ```mojo def page_range(self, flat: Bool) -> Tuple[Int, Int] ``` The half-open range of data pages that can hold the wanted rows. A page is in it when the rows it can hold meet `[row_lo, row_hi)`. For a flat column that is `first_rows[k] < row_hi` and `first_rows[k + 1] > row_lo`; a repeated column widens the second test to `>=`, which is the in-flight record the struct's own note describes. ### `ColumnData` ```mojo struct ColumnData ``` One decoded column chunk: levels and non-null values. Implements: `Copyable`, `Defaultable`, `Movable` Fields: - `defs: List[UInt16]` - `reps: List[UInt16]` - `values: PhysBuffer` - `num_slots: Int`: Level slots — the number of values including nulls. - `first_row: Int`: The row group row this chunk's first record is. Non-zero only when page pruning let whole leading pages be skipped. - `num_rows: Int`: Records this chunk covers, or -1 when it covers the whole row group and the row group's own count is the answer. - `pages_read: Int`: Data pages actually decompressed and decoded. - `all_present: Bool`: Every slot is at the leaf's maximum definition level, so `defs` was never materialised. This is the usual case — a column with no nulls — and skipping the per-slot level array is worth a great deal on a wide read. - `page_slot: List[Int]`: The first slot of each data page from the first one with a null in it, and one final entry equal to `num_slots`. Empty until then, and so empty altogether for a chunk with no nulls. - `page_value: List[Int]`: The first value of each data page `page_slot` covers, and one final entry equal to the number of values in `values`. - `packed: Bool`: This leaf's definition levels *are* a validity bitmap — `max_def == 1` and `max_rep == 0`, so a level is one bit — and `mask`, not `defs`, is where they land. Set by `read_column_chunk` from the leaf's descriptor before a single page is read, so it is a property of the column and not of what any one page happened to contain. - `mask: List[UInt8]`: Packed validity when `packed`: bit `i`, LSB-first, is 1 when slot `i` is at `max_def`. Empty when the chunk turned out to have no nulls at all, which is what `all_present` then says. #### `__init__` ```mojo def __init__(out self) ``` #### `masked` ```mojo def masked(self) -> Bool ``` Validity for this chunk is a bitmap, not an array of levels. #### `value_at` ```mojo def value_at(self, row: Int, max_def: Int) -> Int ``` How many of the first `row` slots hold a value. The answer for a slot that is a page boundary is stored; for anything else it is that answer plus a scan of at most one page — a popcount over the validity mask where there is one. Nothing here is per row, which is the point: the caller wants this at a handful of batch boundaries, not at every row. #### `def_at` ```mojo def def_at(self, i: Int, max_def: Int) -> Int ``` The definition level of slot `i`. #### `rep_at` ```mojo def rep_at(self, i: Int) -> Int ``` ## Functions ### `copy_mask_bits` ```mojo def copy_mask_bits(src: List[UInt8], at: Int, mut dst: List[UInt8], n: Int) -> Int ``` Copy `n` bits from bit `at` of `src` to bit 0 of `dst`, and return how many of them are set. `dst` is already sized to `(n + 7) // 8` and zeroed. A batch that starts on a byte boundary — every batch whose size is a multiple of 8, which is every default one — makes this a `memcpy`; otherwise it is one shift per byte. This is how a leaf's Arrow validity buffer gets built when the chunk's definition levels were decoded as a bitmap in the first place: no pass over levels, and the null count comes out of the same popcount. ### `chunk_start` ```mojo def chunk_start(cm: ColumnMetaData) -> Int ``` Where a column chunk's pages begin — the dictionary page if it has one. ### `read_column_chunk` ```mojo def read_column_chunk[Codecs: CodecSet](file: Span[UInt8], cm: ColumnMetaData, leaf: LeafColumn, verify_crc: Bool, window: PageWindow) -> ColumnData ``` Decode the pages of one column chunk that `window` still wants. A default `PageWindow` — what every read without page pruning passes — is every page, and then this is the whole-chunk decode it always was: nothing below branches on the window until there is one. With a window, the pages outside it are found from the page index and never decompressed. Their headers are still read, because that is how the next page's offset is known, so what is saved is decompression, level decoding and value decoding — everything after `Codecs.decompress` — which is where a column chunk's time is. **A record can span a page boundary.** The Parquet spec has pages change on record boundaries when there is an `OffsetIndex`, but nothing in a file proves it, and a window that starts inside a record would drop that record's leading values and silently shorten a list. That is exactly what the first slot's repetition level says: a chunk that begins a record has `reps[0] == 0`. When it does not, this falls back to decoding the whole chunk rather than trusting the seam — a rare, self-correcting cost, and the only alternative that cannot be wrong.