magmalake parquet

parquet · API

parquet.schema

The Parquet schema tree, its leaf columns, and the Arrow schema it means.

A Parquet file's schema is a flat list of SchemaElements in depth-first order, each group carrying its child count. ParquetSchema turns that back into a tree, computes the maximum definition and repetition level of every leaf, and then derives the Arrow field tree by applying the LogicalTypes rules — including the backward-compatibility rules for two-level lists that old writers produced.

Three views come out of it:

  • nodes — the Parquet tree itself, one node per SchemaElement.
  • leaves — the columns that actually hold data, in column-chunk order.
  • fields / roots — the Arrow field tree, which is what a RecordBatch looks like. A LIST field collapses the annotated group and its repeated child into one node; a MAP field collapses the annotated group and keeps the key_value group as its single STRUCT child, exactly as Arrow wants.

Structs

SchemaNode

struct SchemaNode

One node of the Parquet schema tree.

Implements Copyable, Defaultable, Movable

Fields
name: String
elem: Int: Index into FileMetaData.schema.
parent: Int
children: List[Int]
repetition: Int
max_def: Int
max_rep: Int
is_leaf: Bool
leaf: Int: Index into ParquetSchema.leaves, or -1.
path: List[String]
field_id: Int32: The Parquet field id, or -1 when the writer did not set one.

__init__

def __init__(out self)

LeafColumn

struct LeafColumn

A column that holds data — one column chunk per row group.

Implements Copyable, Defaultable, Movable

Fields
node: Int
path: List[String]
physical: Int32
type_length: Int
max_def: Int
max_rep: Int
arrow: ArrowType
field_id: Int32
converted: Int32: The ConvertedType, or -1.
scale: Int
precision: Int

__init__

def __init__(out self)

dotted

def dotted(self) -> String

ArrowField

struct ArrowField

One node of the Arrow field tree derived from the Parquet schema.

Implements Copyable, Defaultable, Movable

Fields
name: String
type: ArrowType
nullable: Bool
children: List[Int]
leaf: Int: Index into ParquetSchema.leaves for a primitive field, else -1.
field_id: Int32
node: Int: The Parquet node this field came from.
def_level: Int: This field is null in a record whose definition level is below this.
rep_level: Int
elem_def_level: Int: LIST/MAP only: at or above this level the container has an element.
elem_rep_level: Int: LIST/MAP only: a repetition level of exactly this continues a list.

__init__

def __init__(out self)

ParquetSchema

struct ParquetSchema

The schema tree, its leaves, and the Arrow fields they add up to.

Implements Copyable, Defaultable, Movable

Fields
nodes: List[SchemaNode]
leaves: List[LeafColumn]
fields: List[ArrowField]
roots: List[Int]

__init__

def __init__(out self)

num_leaves

def num_leaves(self) -> Int

leaf_by_path

def leaf_by_path(self, path: StringSpan) -> Int

The leaf whose dotted path is path, or -1.

field_by_name

def field_by_name(self, name: StringSpan) -> Int

field_by_id

def field_by_id(self, id: Int32) -> Int

Depth-first search of the whole tree for a Parquet field id.

Functions

arrow_type_of

def arrow_type_of(el: SchemaElement) -> ArrowType

The Arrow type of one primitive SchemaElement.

LogicalType wins when present; ConvertedType is the fallback for files written before Parquet 2.4. Anything unrecognised falls back to the plain physical type, which is what every other reader does.

build_schema

def build_schema(elements: List[SchemaElement]) -> ParquetSchema

Turn the flat depth-first SchemaElement list into a ParquetSchema.

Aliases

comptime REP_REQUIRED = 0
comptime REP_OPTIONAL = 1
comptime REP_REPEATED = 2