parquet · API

parquet.rle_encode

Encoders for the two bit-level formats a Parquet writer needs.

Structs

BitPacker

struct BitPacker

Packs values of a fixed bit width, least-significant bit first.

Bits go into a 64-bit accumulator and leave it eight at a time, so a value costs a shift, an or and at most a few byte stores — not a loop over its bits with a bounds-checked list grow inside it.

Implements Copyable, Defaultable, Movable

Fields
out: List[UInt8]
acc: UInt64
bits: Int

__init__

def __init__(out self)

reserve

def reserve(mut self, count: Int, width: Int)

put

def put(mut self, value: UInt64, width: Int)

align

def align(mut self)

Functions

write_uleb128

def write_uleb128(mut out: List[UInt8], value: UInt64)

encode_hybrid

def encode_hybrid(values: Span[UInt16], width: Int) -> List[UInt8]

The RLE / bit-packed hybrid: repeated runs of eight or more values, and bit-packed groups for everything else.

encode_levels

def encode_levels(values: Span[UInt16], width: Int) -> List[UInt8]

Levels as a v1 data page wants them: a 4-byte length, then the hybrid.