magmalake iceberg

iceberg · API

iceberg.write

Writing data files: Arrow in, Parquet out, a DataFile record back.

write_data_files takes Arrow RecordBatches, aligns their columns to the table's schema by name (falling back to the field id when the batch carries one), partitions each batch by the spec's transforms, writes one Parquet file per partition per batch through parquet.mojo's writer, and returns the data_file records a manifest needs — record count, file size, split offsets, and the per-column sizes, value counts, null counts and Appendix-D bounds that InclusiveMetricsEvaluator prunes on.

The statistics come from the Parquet footer the writer just produced rather than from a second pass over the values: whatever the writer decided the min and max were is what the manifest reports, which is the only way the two can agree. Bounds are truncated the way write.metadata.metrics.default's truncate(16) says — a lower bound is cut, an upper bound is cut and then incremented so it stays an upper bound, and dropped when it cannot be.

Structs

WriteOptions

struct WriteOptions

How to write data files.

Implements Copyable, Defaultable, Movable

Fields
codec: String
row_group_rows: Int: Rows per row group, which is what parquet.mojo's writer splits on.
page_size: Int: Target uncompressed bytes of values per data page.
truncate: Int: write.metadata.metrics.default = truncate(N); 0 disables truncation.

__init__

def __init__(out self)

from_properties

def from_properties(properties: Dict[String, String]) -> Self

Functions

codec_value

def codec_value(name: String) -> Int32

truncate_lower

def truncate_lower(raw: List[UInt8], kind: UInt8, n: Int) -> List[UInt8]

A lower bound cut to n units — code points for a string, bytes otherwise. Cutting a lower bound can only lower it, which is safe.

truncate_upper

def truncate_upper(raw: List[UInt8], kind: UInt8, n: Int) -> Tuple[List[UInt8], Bool]

An upper bound cut to n units and then incremented.

Truncation lowers a value, so an upper bound has to be nudged back up or it stops bounding. For a string that means incrementing the last code point that is not the maximum; for binary, the last byte that is not 0xFF. When every unit is at its maximum the bound cannot be truncated at all and the second element of the result is False — the spec's answer is to omit it.

bound_bytes

def bound_bytes(v: ScalarValue, kind: UInt8, precision: Int, scale: Int) -> Tuple[List[UInt8], Bool]

human_partition_value

def human_partition_value(d: Datum, transform_kind: UInt8) -> String

The Hive-style rendering of one partition value.

escape_path

def escape_path(s: String) -> String

Percent-escape everything a Hive path may not contain.

partition_path

def partition_path(spec: PartitionSpec, values: List[Datum]) -> String

align_batch

def align_batch(batch: RecordBatch, schema: Schema) -> List[ColumnTree]

The batch's columns in the table's schema order, at the table's types.

A nested column is matched child by child by field id, so a batch whose struct has its fields in another order, under other names, or one field short still lines up with the table — the same resolution the read path does, run the other way.

write_parquet

def write_parquet(columns: List[ColumnTree], schema: Schema, options: WriteOptions) -> List[UInt8]

data_file_from_parquet

def data_file_from_parquet(data: List[UInt8], path: String, schema: Schema, spec: PartitionSpec, partition: List[Datum], sort_order_id: Int, options: WriteOptions) -> DataFile

Read back the footer we just wrote and turn it into a data_file.

write_data_files

def write_data_files(io: FileIO, location: String, batches: List[RecordBatch], schema: Schema, spec: PartitionSpec, sort_order_id: Int, options: WriteOptions) -> List[DataFile]

Write every batch out, one Parquet file per partition per batch.

Aliases

comptime DEFAULT_METRICS_TRUNCATE = 16
comptime HIVE_NULL = String("__HIVE_DEFAULT_PARTITION__")
comptime PROP_COMPRESSION = String("write.parquet.compression-codec")
comptime PROP_PAGE_SIZE = String("write.parquet.page-size-bytes")
comptime PROP_ROW_GROUP_ROWS = String("write.parquet.row-group-size-rows")

Not an Iceberg property.

Iceberg spells the row-group target in bytes (write.parquet.row-group-size-bytes), and parquet.mojo's writer splits a batch by a row count. Converting between them means guessing a compression ratio before compressing, which would make the setting mean something different from what it says, so the byte property is ignored and this one — under a name no reader will mistake for the spec's — is honoured instead.