magmalake iceberg

iceberg · API

iceberg.manifest

Manifest lists and manifests, read through avro.mojo.

Two Avro files, both manifest_file/manifest_entry records:

  • the manifest list (snap-<id>-<n>-<uuid>.avro) names every manifest in a snapshot, with the counts and per-partition-field summaries a scan uses to skip manifests without opening them;
  • each manifest lists data or delete files with their partition tuple, per-column metrics, and the entry status that drives inheritance.

Three inheritance rules are implemented here, because getting them wrong silently mis-plans a scan:

  • Sequence numbers. A null sequence_number / file_sequence_number is inherited from the manifest's entry in the manifest list — but only for ADDED entries. EXISTING and DELETED entries must carry both explicitly, and a v1 manifest with no sequence column defaults every file to 0.
  • Snapshot id. A null snapshot_id is inherited from the manifest's added_snapshot_id.
  • first_row_id (v3). A null value is the manifest's first_row_id plus the record counts of every preceding data file in that manifest that also had a null first_row_id.

Fields this build does not recognise are ignored rather than fatal, and the reserved id 141 and the v4-only content_stats (146) are simply not read.

Both files are read with avro.mojo's RecordCursor rather than its Value API. Manifests sit on the scan-planning critical path and a Value costs roughly sixty heap allocations per entry, plus a string comparison per field lookup. The cursor compiles the file's own schema into a decode plan once, and every field this module wants is then a slot number resolved once per file (_ListSlots / _EntrySlots) — the per-entry work is integer indexing and, for a path or a bound, a span into the block buffer. A field the manifest's schema does not have resolves to slot -1, which is how the version differences below stay expressible.

Once per file is still once too many. A snapshot's five hundred manifests were written by one writer and carry five hundred byte-identical copies of the same avro.schema, the same Iceberg schema and the same partition-spec. ManifestCache keys everything those three determine — the compiled decode plan, the parsed schema and spec, the partition type and the resolved slot table — on the raw metadata bytes, before any of it is parsed. Byte equality is the cheap sufficient condition, and the hash over those bytes is only a filter: a hit is confirmed by comparing them. A scan owns one cache and drops it when it is done; there is no global state, and a plan handed out stays alive on its own because a cursor holds its own reference to it.

Structs

ManifestFile

struct ManifestFile

One entry of a manifest list (manifest_file, field ids 500-520).

Implements Copyable, Movable, Writable

Fields
manifest_path: String
manifest_length: Int64
partition_spec_id: Int
content: Int: 0 = data, 1 = deletes. Always 0 for a v1 manifest list.
sequence_number: Int64
min_sequence_number: Int64
added_snapshot_id: Int64
added_files_count: Int
has_added_files_count: Bool
existing_files_count: Int
has_existing_files_count: Bool
deleted_files_count: Int
has_deleted_files_count: Bool
added_rows_count: Int64
has_added_rows_count: Bool
existing_rows_count: Int64
has_existing_rows_count: Bool
deleted_rows_count: Int64
has_deleted_rows_count: Bool
partitions: List[FieldSummary]
has_partitions: Bool
key_metadata: List[UInt8]
first_row_id: Int64
has_first_row_id: Bool

is_delete_manifest

def is_delete_manifest(self) -> Bool

live_files

def live_files(self) -> Int

Entries that are not DELETED, as far as the counts can say.

write_to

def write_to(self, mut writer: T)

DataFile

struct DataFile

A data_file struct (field ids 100-145).

Implements Copyable, Movable, Writable

Fields
content: Int
file_path: String
file_format: String
partition: List[Datum]: Positional, matching the writing spec's field order.
record_count: Int64
file_size_in_bytes: Int64
column_sizes: List[ColumnMetrics]: Reused as the carrier for all the per-column metric maps.
metrics: List[ColumnMetrics]
key_metadata: List[UInt8]
has_key_metadata: Bool
split_offsets: List[Int64]
equality_ids: List[Int]
sort_order_id: Int
has_sort_order_id: Bool
first_row_id: Int64
has_first_row_id: Bool
referenced_data_file: String
has_referenced_data_file: Bool
content_offset: Int64
has_content_offset: Bool
content_size_in_bytes: Int64
has_content_size_in_bytes: Bool

is_data

def is_data(self) -> Bool

is_position_delete

def is_position_delete(self) -> Bool

is_equality_delete

def is_equality_delete(self) -> Bool

is_deletion_vector

def is_deletion_vector(self) -> Bool

A DV is a position-delete entry in a Puffin blob with a referent.

write_to

def write_to(self, mut writer: T)

ManifestEntry

struct ManifestEntry

A manifest_entry, with every inherited value already filled in.

Implements Copyable, Movable

Fields
status: Int
snapshot_id: Int64
has_snapshot_id: Bool
sequence_number: Int64: The data sequence number, after inheritance.
file_sequence_number: Int64
data_file: DataFile

is_live

def is_live(self) -> Bool

Manifest

struct Manifest

A decoded manifest file: its metadata plus its entries.

Implements Copyable, Movable

Fields
entries: List[ManifestEntry]
partition_spec: PartitionSpec
partition_spec_id: Int
schema: Schema
format_version: Int
content: Int

__init__

def __init__(out self, var entries: List[ManifestEntry], var spec: PartitionSpec, spec_id: Int, var schema: Schema, format_version: Int, content: Int)

live_entries

def live_entries(self) -> List[ManifestEntry]

ManifestCache

struct ManifestCache

One scan's worth of parsed manifest metadata.

Three caches that share a lifetime, all keyed on raw bytes:

  • plans — avro.mojo's own PlanCache, keyed on the file's avro.schema, holding the compiled decode plan;
  • shapes — keyed on every metadata entry a _ManifestShape is derived from, so two manifests written under different specs cannot share one;
  • lists — the manifest list's _ListSlots, keyed on its avro.schema.

disabled() turns all three off without changing the code path, which is how the tests check that the cache is transparent.

It also keeps manifest and manifest-list bytes, by location, up to max_file_bytes (64 MiB). Iceberg never rewrites one of those files, so a plan of another query or a later snapshot reads only the manifests it hasn't seen: on object storage, a plan of an unchanged table makes no requests at all. A caller planning many queries holds one cache and passes it to TableScan.plan_files_with, as iceberg-rust's object cache does for its table.

Implements Movable

Fields
plans: PlanCache
shape_hashes: List[UInt64]
shape_keys: List[List[UInt8]]
shapes: List[_ManifestShape]
list_hashes: List[UInt64]
list_keys: List[List[UInt8]]
lists: List[_ListSlots]
enabled: Bool
collide: Bool: A test seam: force every key to hash to zero, so the byte compare is the only thing telling two manifest shapes apart.
hits: Int
misses: Int
files: Dict[String, List[UInt8]]: Manifest and manifest-list bytes, by location.
file_bytes: Int
max_file_bytes: Int
file_hits: Int

__init__

def __init__(out self)

disabled

def disabled() -> Self

Every lookup a miss, nothing stored — the escape hatch.

has_file

def has_file(self, location: String) -> Bool

file

def file(mut self, location: String) -> List[UInt8]

A copy of a cached file's bytes; has_file first.

keep_file

def keep_file(mut self, location: String, data: Span[UInt8])

Keeps a manifest's or manifest list's bytes, while under budget.

collide_all_hashes

def collide_all_hashes(mut self)

Force every key — plans, shapes and lists — to hash to zero. A test seam: it leaves the byte compare as the only thing telling two different schemas apart.

hash_of

def hash_of(self, key: Span[UInt8]) -> UInt64

find_shape

def find_shape(self, key: Span[UInt8]) -> Int

add_shape

def add_shape(mut self, var key: List[UInt8], var shape: _ManifestShape) -> Int

Store shape and return where it landed. A disabled cache keeps only the newest one — the caller still reads it back by index, but find_shape will never hand it out again.

find_list

def find_list(self, key: Span[UInt8]) -> Int

add_list

def add_list(mut self, var key: List[UInt8], var slots: _ListSlots) -> Int

Functions

read_manifest_list

def read_manifest_list(path: String) -> List[ManifestFile]

Read a snapshot's manifest list from a local path.

read_manifest_list_io

def read_manifest_list_io(io: FileIO, location: String) -> List[ManifestFile]

Read a snapshot's manifest list through a FileIO.

A manifest list is small and read whole; there is nothing to seek to.

read_local

def read_local(path: String) -> List[UInt8]

read_manifest_list_io_cached

def read_manifest_list_io_cached(io: FileIO, location: String, mut cache: ManifestCache) -> List[ManifestFile]

Read a manifest list, reusing whatever cache already parsed.

One scan reads one list, so this pays for itself only across snapshots — time travel, expire_snapshots, a diff between two refs.

read_manifest_list_bytes

def read_manifest_list_bytes(var data: List[UInt8]) -> List[ManifestFile]

read_manifest_list_bytes_cached

def read_manifest_list_bytes_cached(var data: List[UInt8], mut cache: ManifestCache) -> List[ManifestFile]

read_manifest

def read_manifest(mf: ManifestFile) -> Manifest

Read a manifest, inheriting everything the manifest list supplies.

read_manifest_at

def read_manifest_at(path: String, mf: ManifestFile) -> Manifest

Read a manifest from a local path.

read_manifest_io

def read_manifest_io(io: FileIO, location: String, mf: ManifestFile) -> Manifest

Read a manifest through a FileIO.

read_manifest_io_cached

def read_manifest_io_cached(io: FileIO, location: String, mf: ManifestFile, mut cache: ManifestCache) -> Manifest

Read a manifest, reusing whatever cache already parsed.

read_manifest_bytes

def read_manifest_bytes(var data: List[UInt8], mf: ManifestFile) -> Manifest

Read a manifest with nothing cached — every schema parsed afresh.

read_manifest_bytes_cached

def read_manifest_bytes_cached(var data: List[UInt8], mf: ManifestFile, mut cache: ManifestCache) -> Manifest

Aliases

comptime MANIFEST_CONTENT_DATA = Int(0)
comptime MANIFEST_CONTENT_DELETES = Int(1)
comptime CONTENT_DATA = Int(0)
comptime CONTENT_POSITION_DELETES = Int(1)
comptime CONTENT_EQUALITY_DELETES = Int(2)
comptime STATUS_EXISTING = Int(0)
comptime STATUS_ADDED = Int(1)
comptime STATUS_DELETED = Int(2)