iceberg · API

iceberg.append

Fast append: new data files, a new manifest, a new snapshot.

A fast append is the one write operation that never rewrites anything. It adds a manifest holding the files it wrote, carries every manifest of the parent snapshot over into the new manifest list by reference — same path, same sequence numbers, same first_row_id — and commits a snapshot whose operation is append.

What that produces, in order:

  1. data files (iceberg.write), one Parquet file per partition per batch;
  2. one manifest of ADDED entries, with null sequence numbers so that the manifest list's own numbers are inherited, and — under v3 — null first_row_ids for the same reason;
  3. a manifest list: the new manifest first, then the parent's, each keeping the values it was committed with;
  4. a Snapshot with the standard summary keys, and the metadata edits a commit implies: last-sequence-number, last-updated-ms, snapshot-log, metadata-log, refs.main, and under v3 next-row-id.

Row lineage (v3). The snapshot's first-row-id is the table's next-row-id before the commit; the new manifest is assigned that same id in the manifest list, and every data file in it inherits an id from it in order. The table's new next-row-id is first-row-id + added-rows. Existing manifests keep the ids they were assigned, so no two ranges overlap. That is the rule the v3 fixture generator encodes and the reader already implements for the other direction.

Overwrite, delete, replace and compaction are not here: they need a merge of manifests rather than an append to them, and the iceberg-rs.mojo bridge covers them.

Structs

AppendResult

struct AppendResult

A prepared commit: the new metadata, and where its files went.

Implements Copyable, Movable

Fields
metadata: TableMetadata
snapshot_id: Int64
manifest_list: String
manifest_paths: List[String]
data_files: List[DataFile]

__init__

def __init__(out self, var metadata: TableMetadata, snapshot_id: Int64, var manifest_list: String, var manifest_paths: List[String], var data_files: List[DataFile])

added_records

def added_records(self) -> Int64

Functions

prepare_append

def prepare_append(io: FileIO, metadata: TableMetadata, var data_files: List[DataFile], var extra_summary: Dict[String, String]) -> AppendResult

Write the manifest and manifest list for an append, and build the new table metadata. Nothing is committed: the caller persists the metadata.

write_and_prepare_append

def write_and_prepare_append(io: FileIO, metadata: TableMetadata, batches: List[RecordBatch]) -> AppendResult

Write batches as data files, then prepare the commit.

next_metadata_version

def next_metadata_version(metadata: TableMetadata) -> Int

The <V> of the next <V>-<uuid>.metadata.json.

Java derives it from the current file's own name, falling back to the length of the metadata log — which is what a table written by something else, or a brand new one, needs.

metadata_file_name

def metadata_file_name(version: Int) -> String

Aliases

comptime OP_APPEND = String("append")