# iceberg.manifest > A module of iceberg's API. Manifest lists and manifests, read through avro.mojo. Two Avro files, both `manifest_file`/`manifest_entry` records: * the **manifest list** (`snap---.avro`) names every manifest in a snapshot, with the counts and per-partition-field summaries a scan uses to skip manifests without opening them; * each **manifest** lists data or delete files with their partition tuple, per-column metrics, and the entry status that drives inheritance. Three inheritance rules are implemented here, because getting them wrong silently mis-plans a scan: * **Sequence numbers.** A `null` `sequence_number` / `file_sequence_number` is inherited from the manifest's entry in the manifest list — but *only* for ADDED entries. EXISTING and DELETED entries must carry both explicitly, and a v1 manifest with no sequence column defaults every file to 0. * **Snapshot id.** A `null` `snapshot_id` is inherited from the manifest's `added_snapshot_id`. * **`first_row_id`** (v3). A `null` value is the manifest's `first_row_id` plus the record counts of every preceding data file in that manifest that also had a null `first_row_id`. Fields this build does not recognise are ignored rather than fatal, and the reserved id 141 and the v4-only `content_stats` (146) are simply not read. Both files are read with avro.mojo's `RecordCursor` rather than its `Value` API. Manifests sit on the scan-planning critical path and a `Value` costs roughly sixty heap allocations per entry, plus a string comparison per field lookup. The cursor compiles the file's own schema into a decode plan once, and every field this module wants is then a slot number resolved once per file (`_ListSlots` / `_EntrySlots`) — the per-entry work is integer indexing and, for a path or a bound, a span into the block buffer. A field the manifest's schema does not have resolves to slot -1, which is how the version differences below stay expressible. **Once per file is still once too many.** A snapshot's five hundred manifests were written by one writer and carry five hundred byte-identical copies of the same `avro.schema`, the same Iceberg `schema` and the same `partition-spec`. `ManifestCache` keys everything those three determine — the compiled decode plan, the parsed schema and spec, the partition type and the resolved slot table — on the raw metadata bytes, before any of it is parsed. Byte equality is the cheap sufficient condition, and the hash over those bytes is only a filter: a hit is confirmed by comparing them. A scan owns one cache and drops it when it is done; there is no global state, and a plan handed out stays alive on its own because a cursor holds its own reference to it. ## Structs ### `ManifestFile` ```mojo struct ManifestFile ``` One entry of a manifest list (`manifest_file`, field ids 500-520). Implements: `Copyable`, `Movable`, `Writable` Fields: - `manifest_path: String` - `manifest_length: Int64` - `partition_spec_id: Int` - `content: Int`: 0 = data, 1 = deletes. Always 0 for a v1 manifest list. - `sequence_number: Int64` - `min_sequence_number: Int64` - `added_snapshot_id: Int64` - `added_files_count: Int` - `has_added_files_count: Bool` - `existing_files_count: Int` - `has_existing_files_count: Bool` - `deleted_files_count: Int` - `has_deleted_files_count: Bool` - `added_rows_count: Int64` - `has_added_rows_count: Bool` - `existing_rows_count: Int64` - `has_existing_rows_count: Bool` - `deleted_rows_count: Int64` - `has_deleted_rows_count: Bool` - `partitions: List[FieldSummary]` - `has_partitions: Bool` - `key_metadata: List[UInt8]` - `first_row_id: Int64` - `has_first_row_id: Bool` #### `is_delete_manifest` ```mojo def is_delete_manifest(self) -> Bool ``` #### `live_files` ```mojo def live_files(self) -> Int ``` Entries that are not DELETED, as far as the counts can say. #### `write_to` ```mojo def write_to(self, mut writer: T) ``` ### `DataFile` ```mojo struct DataFile ``` A `data_file` struct (field ids 100-145). Implements: `Copyable`, `Movable`, `Writable` Fields: - `content: Int` - `file_path: String` - `file_format: String` - `partition: List[Datum]`: Positional, matching the writing spec's field order. - `record_count: Int64` - `file_size_in_bytes: Int64` - `column_sizes: List[ColumnMetrics]`: Reused as the carrier for all the per-column metric maps. - `metrics: List[ColumnMetrics]` - `key_metadata: List[UInt8]` - `has_key_metadata: Bool` - `split_offsets: List[Int64]` - `equality_ids: List[Int]` - `sort_order_id: Int` - `has_sort_order_id: Bool` - `first_row_id: Int64` - `has_first_row_id: Bool` - `referenced_data_file: String` - `has_referenced_data_file: Bool` - `content_offset: Int64` - `has_content_offset: Bool` - `content_size_in_bytes: Int64` - `has_content_size_in_bytes: Bool` #### `is_data` ```mojo def is_data(self) -> Bool ``` #### `is_position_delete` ```mojo def is_position_delete(self) -> Bool ``` #### `is_equality_delete` ```mojo def is_equality_delete(self) -> Bool ``` #### `is_deletion_vector` ```mojo def is_deletion_vector(self) -> Bool ``` A DV is a position-delete entry in a Puffin blob with a referent. #### `write_to` ```mojo def write_to(self, mut writer: T) ``` ### `ManifestEntry` ```mojo struct ManifestEntry ``` A `manifest_entry`, with every inherited value already filled in. Implements: `Copyable`, `Movable` Fields: - `status: Int` - `snapshot_id: Int64` - `has_snapshot_id: Bool` - `sequence_number: Int64`: The *data* sequence number, after inheritance. - `file_sequence_number: Int64` - `data_file: DataFile` #### `is_live` ```mojo def is_live(self) -> Bool ``` ### `Manifest` ```mojo struct Manifest ``` A decoded manifest file: its metadata plus its entries. Implements: `Copyable`, `Movable` Fields: - `entries: List[ManifestEntry]` - `partition_spec: PartitionSpec` - `partition_spec_id: Int` - `schema: Schema` - `format_version: Int` - `content: Int` #### `__init__` ```mojo def __init__(out self, var entries: List[ManifestEntry], var spec: PartitionSpec, spec_id: Int, var schema: Schema, format_version: Int, content: Int) ``` #### `live_entries` ```mojo def live_entries(self) -> List[ManifestEntry] ``` ### `ManifestCache` ```mojo struct ManifestCache ``` One scan's worth of parsed manifest metadata. Three caches that share a lifetime, all keyed on raw bytes: * `plans` — avro.mojo's own `PlanCache`, keyed on the file's `avro.schema`, holding the compiled decode plan; * `shapes` — keyed on every metadata entry a `_ManifestShape` is derived from, so two manifests written under different specs cannot share one; * `lists` — the manifest list's `_ListSlots`, keyed on its `avro.schema`. `disabled()` turns all three off without changing the code path, which is how the tests check that the cache is transparent. It also keeps manifest and manifest-list **bytes**, by location, up to `max_file_bytes` (64 MiB). Iceberg never rewrites one of those files, so a plan of another query or a later snapshot reads only the manifests it hasn't seen: on object storage, a plan of an unchanged table makes no requests at all. A caller planning many queries holds one cache and passes it to `TableScan.plan_files_with`, as iceberg-rust's object cache does for its table. Implements: `Movable` Fields: - `plans: PlanCache` - `shape_hashes: List[UInt64]` - `shape_keys: List[List[UInt8]]` - `shapes: List[_ManifestShape]` - `list_hashes: List[UInt64]` - `list_keys: List[List[UInt8]]` - `lists: List[_ListSlots]` - `enabled: Bool` - `collide: Bool`: A test seam: force every key to hash to zero, so the byte compare is the only thing telling two manifest shapes apart. - `hits: Int` - `misses: Int` - `files: Dict[String, List[UInt8]]`: Manifest and manifest-list bytes, by location. - `file_bytes: Int` - `max_file_bytes: Int` - `file_hits: Int` #### `__init__` ```mojo def __init__(out self) ``` #### `disabled` ```mojo def disabled() -> Self ``` Every lookup a miss, nothing stored — the escape hatch. #### `has_file` ```mojo def has_file(self, location: String) -> Bool ``` #### `file` ```mojo def file(mut self, location: String) -> List[UInt8] ``` A copy of a cached file's bytes; `has_file` first. #### `keep_file` ```mojo def keep_file(mut self, location: String, data: Span[UInt8]) ``` Keeps a manifest's or manifest list's bytes, while under budget. #### `collide_all_hashes` ```mojo def collide_all_hashes(mut self) ``` Force every key — plans, shapes and lists — to hash to zero. A test seam: it leaves the byte compare as the only thing telling two different schemas apart. #### `hash_of` ```mojo def hash_of(self, key: Span[UInt8]) -> UInt64 ``` #### `find_shape` ```mojo def find_shape(self, key: Span[UInt8]) -> Int ``` #### `add_shape` ```mojo def add_shape(mut self, var key: List[UInt8], var shape: _ManifestShape) -> Int ``` Store `shape` and return where it landed. A disabled cache keeps only the newest one — the caller still reads it back by index, but `find_shape` will never hand it out again. #### `find_list` ```mojo def find_list(self, key: Span[UInt8]) -> Int ``` #### `add_list` ```mojo def add_list(mut self, var key: List[UInt8], var slots: _ListSlots) -> Int ``` ## Functions ### `read_manifest_list` ```mojo def read_manifest_list(path: String) -> List[ManifestFile] ``` Read a snapshot's manifest list from a local path. ### `read_manifest_list_io` ```mojo def read_manifest_list_io(io: FileIO, location: String) -> List[ManifestFile] ``` Read a snapshot's manifest list through a `FileIO`. A manifest list is small and read whole; there is nothing to seek to. ### `read_local` ```mojo def read_local(path: String) -> List[UInt8] ``` ### `read_manifest_list_io_cached` ```mojo def read_manifest_list_io_cached(io: FileIO, location: String, mut cache: ManifestCache) -> List[ManifestFile] ``` Read a manifest list, reusing whatever `cache` already parsed. One scan reads one list, so this pays for itself only across snapshots — time travel, `expire_snapshots`, a diff between two refs. ### `read_manifest_list_bytes` ```mojo def read_manifest_list_bytes(var data: List[UInt8]) -> List[ManifestFile] ``` ### `read_manifest_list_bytes_cached` ```mojo def read_manifest_list_bytes_cached(var data: List[UInt8], mut cache: ManifestCache) -> List[ManifestFile] ``` ### `read_manifest` ```mojo def read_manifest(mf: ManifestFile) -> Manifest ``` Read a manifest, inheriting everything the manifest list supplies. ### `read_manifest_at` ```mojo def read_manifest_at(path: String, mf: ManifestFile) -> Manifest ``` Read a manifest from a local path. ### `read_manifest_io` ```mojo def read_manifest_io(io: FileIO, location: String, mf: ManifestFile) -> Manifest ``` Read a manifest through a `FileIO`. ### `read_manifest_io_cached` ```mojo def read_manifest_io_cached(io: FileIO, location: String, mf: ManifestFile, mut cache: ManifestCache) -> Manifest ``` Read a manifest, reusing whatever `cache` already parsed. ### `read_manifest_bytes` ```mojo def read_manifest_bytes(var data: List[UInt8], mf: ManifestFile) -> Manifest ``` Read a manifest with nothing cached — every schema parsed afresh. ### `read_manifest_bytes_cached` ```mojo def read_manifest_bytes_cached(var data: List[UInt8], mf: ManifestFile, mut cache: ManifestCache) -> Manifest ``` ## Aliases ### `MANIFEST_CONTENT_DATA` ```mojo comptime MANIFEST_CONTENT_DATA = Int(0) ``` ### `MANIFEST_CONTENT_DELETES` ```mojo comptime MANIFEST_CONTENT_DELETES = Int(1) ``` ### `CONTENT_DATA` ```mojo comptime CONTENT_DATA = Int(0) ``` ### `CONTENT_POSITION_DELETES` ```mojo comptime CONTENT_POSITION_DELETES = Int(1) ``` ### `CONTENT_EQUALITY_DELETES` ```mojo comptime CONTENT_EQUALITY_DELETES = Int(2) ``` ### `STATUS_EXISTING` ```mojo comptime STATUS_EXISTING = Int(0) ``` ### `STATUS_ADDED` ```mojo comptime STATUS_ADDED = Int(1) ``` ### `STATUS_DELETED` ```mojo comptime STATUS_DELETED = Int(2) ```