# parquet.codec

> A module of parquet's API.

Page compression codecs.

`ParquetReader` is parametrised on a `CodecSet`, a compile-time table of page
decompressors, exactly like `avro.mojo`'s block codecs. The default set covers
everything that needs no FFI:

| codec | where it comes from |
|---|---|
| `UNCOMPRESSED` | — |
| `SNAPPY` | `snappy.mojo`, pure Mojo |
| `GZIP` | `deflate.mojo`, pure Mojo, with the gzip wrapper parsed here |

`ZSTD`, `BROTLI`, `LZ4_RAW` and the legacy Hadoop-framed `LZ4` need `zstd.mojo`,
`brotli.mojo` and `lz4.mojo`, which dlopen libzstd, libbrotli and liblz4, so
they live in `parquet_full` and its `AllCodecs`:

```mojo
from parquet import ParquetReader
# -I ../zstd.mojo/src -I ../lz4.mojo/src -I ../brotli.mojo/src
from parquet_full import AllCodecs

var r = ParquetReader[AllCodecs].open("part-0.parquet")
```

Between the two sets that is every codec the Parquet spec defines. A reader on
`DefaultCodecs` that meets one of the four FFI codecs raises a message naming
it and pointing at `AllCodecs`; every other column of that file still reads.

## Structs

### `DefaultCodecs`

```mojo
struct DefaultCodecs
```

`UNCOMPRESSED`, `SNAPPY` and `GZIP` — no FFI, no shared libraries.

Implements: `CodecSet`, `Movable`

#### `supports`

```mojo
def supports(codec: Int32) -> Bool
```

#### `decompress`

```mojo
def decompress[page_origin: ImmOrigin, scratch_origin: MutOrigin](codec: Int32, data: Span[UInt8, page_origin], uncompressed_size: Int, ref[scratch_origin] scratch: List[UInt8]) -> Span[UInt8, origin_of(page_origin, scratch_origin)].Immutable
```

#### `compress`

```mojo
def compress(codec: Int32, data: Span[UInt8]) -> List[UInt8]
```

## Traits

### `CodecSet`

```mojo
undefined
```

A compile-time table of Parquet page decompressors.

#### `supports`

```mojo
def supports(codec: Int32) -> Bool
```

Whether this set can decompress `CompressionCodec` value `codec`.

#### `decompress`

```mojo
def decompress[page_origin: ImmOrigin, scratch_origin: MutOrigin](codec: Int32, data: Span[UInt8, page_origin], uncompressed_size: Int, ref[page_origin] scratch: List[UInt8]) -> Span[UInt8, origin_of(page_origin, scratch_origin)].Immutable
```

Decompress one page body, borrowing rather than allocating.

The result is *a view*, never a fresh buffer. For `UNCOMPRESSED` it is
`data` itself — the page's own bytes inside the file, handed back
untouched, which is what `SerializedPageReader::DecompressIfNeeded`
does in parquet-cpp. For every other codec it is `scratch`, the
caller's one decompression buffer per column chunk, grown in place and
never shrunk — parquet-cpp's `decompression_buffer_`, resized with
`shrink_to_fit=false` (`column_reader.cc:564`).

**Lifetime contract — the span is valid only until the next call that
passes the same `scratch`.** That call overwrites the buffer and may
reallocate it, so a span held across a page boundary reads another
page's bytes. parquet-cpp documents the same rule at
`column_reader.cc:251`: the previous page is invalidated by
`NextPage()`. Copy out anything that has to outlive the page —
`decode_plain_into` and its neighbours already do, which is why a
dictionary page survives the data pages decompressed after it.

Mojo's origins cannot make the compiler enforce that. The returned
origin is the union of the file's and the scratch buffer's, so both are
kept alive for as long as the span is and it can never dangle, but Mojo
does not treat an outstanding immutable view as blocking a later
mutation the way an exclusivity checker would — a held span is a wrong
answer, not a compile error. So the rule is a contract, and
`test_a_page_span_is_only_valid_until_the_next_page` is what states it.

`uncompressed_size` is the page header's declared decompressed length,
which for Brotli and raw LZ4 is the only size there is.

#### `compress`

```mojo
def compress(codec: Int32, data: Span[UInt8]) -> List[UInt8]
```

Compress one page body. Used by `parquet.writer`.

## Functions

### `copy_of`

```mojo
def copy_of(data: Span[UInt8]) -> List[UInt8]
```

### `page_span`

```mojo
def page_span[origin: ImmOrigin](data: Span[UInt8]) -> Span[UInt8, origin]
```

Re-label `data` with the origin `CodecSet.decompress` returns.

Its two answers live in different places — the file for an uncompressed
page, the caller's scratch buffer for a compressed one — and the declared
return type is the union of those two origins, which keeps both alive for
as long as the span is. Neither branch can produce that union type on its
own, hence this one narrow re-labelling, and it is a no-op at run time: the
same pointer and length.

Sound because the union outlives either input, so the origin stamped on is
never longer-lived than the one the bytes actually have.

### `sized_scratch`

```mojo
def sized_scratch[origin: MutOrigin](ref[origin] scratch: List[UInt8], n: Int) -> Span[UInt8, origin]
```

`scratch`, resized to exactly `n` bytes and ready to be written over.

Grows in place and never shrinks the allocation, which is what
`Resize(uncompressed_len, /*shrink_to_fit=*/false)` buys parquet-cpp: after
the first page of a chunk, a page no larger than the biggest one so far
costs no allocation at all. The bytes are left uninitialised rather than
zeroed — a decompressor writes every one of them, and a memset per page
would be most of what this commit is trying to remove.

### `codec_name`

```mojo
def codec_name(codec: Int32) -> String
```

### `unsupported_codec`

```mojo
def unsupported_codec(codec: Int32) -> Error
```

### `gunzip`

```mojo
def gunzip(data: Span[UInt8]) -> List[UInt8]
```

Strip a gzip (RFC 1952) wrapper and inflate the DEFLATE stream inside.

Parquet's `GZIP` codec is gzip-framed, not a bare DEFLATE stream, so the
10-byte header and its optional extra/name/comment/CRC fields have to come
off first. A zlib (RFC 1950) stream is accepted too — some writers emit
one — and so is a bare DEFLATE stream.

A gzip stream may be several members concatenated (RFC 1952 §2.2), and a
page written that way decodes to the members joined in order. Stopping at
the first one silently loses the rest, so the members are looped over here.

### `gunzip_into`

```mojo
def gunzip_into(data: Span[UInt8], mut out: List[UInt8])
```

`gunzip` into a buffer the caller already has, replacing its contents.

The gzip-framed path — every `GZIP` page a Parquet writer emits — inflates
each member onto the end of `out`, so a decompression buffer reused across
a chunk keeps whatever capacity it has already grown. The zlib and bare
DEFLATE fallbacks hand `out` a buffer `inflate` allocated: neither shape
comes out of a Parquet writer, and neither is worth a second copy to keep
the growth in place.

### `gzip_wrap`

```mojo
def gzip_wrap(data: Span[UInt8]) -> List[UInt8]
```

Wrap a DEFLATE stream in the gzip (RFC 1952) framing Parquet's `GZIP` codec expects: a 10-byte header, then the deflate bits, then the CRC32 of the *uncompressed* data and its length modulo 2^32.