magmalake iceberg

iceberg · API

iceberg.delete

Row-level delete and overwrite: which rows go, and how they are recorded.

Two strategies, chosen by the table property write.delete.mode:

  • copy-on-write (the default, as in Java, and the only thing PyIceberg can do) — every data file with a matching row is rewritten without those rows and the original is marked DELETED. The result needs no delete files at all, so every reader can read it, at the cost of rewriting whole files to remove one row.
  • merge-on-read — the rows stay where they are and a delete file records their positions. Under v3 that is a deletion vector: one Puffin deletion-vector-v1 blob per data file, referenced by a manifest entry with content=1, file_format=puffin, referenced_data_file and the content_offset/content_size_in_bytes that must match the blob's footer entry exactly. Under v2 it is a position delete file: a Parquet file of (file_path, pos) pairs — the two reserved field ids 2147483546 and 2147483545 — sorted by path then position, one per partition.

Whichever strategy is in force, a data file whose rows are all deleted is removed rather than emptied. That is the metadata-delete fast path: when the scan's residual for a file is true — the partition predicate alone already proves every row matches — the file is not read at all, and the delete costs one manifest rewrite.

Merging. A deletion vector must contain every position deleted from its data file, not just the new ones: the spec makes a new vector absorb the ones it replaces, and a reader that finds a vector ignores every position delete file for that data file. So a second delete against a file that already has a vector reads the old vector, unions the new positions in, writes a fresh blob, and marks the old vector's manifest entry DELETED. Position delete files are left alone even when their positions are absorbed: one of them can cover several data files, and removing it would resurrect rows in the others.

Overwrite is a delete and an append in one snapshot, always copy-on-write, with operation=overwrite. dynamic_partition_overwrite derives the filter from the data instead of taking one: it replaces exactly the partitions the new rows land in, which needs no row-level work at all — a partition is replaced by removing every data file in it.

Structs

RowDelete

struct RowDelete

One data file, and which of its rows this delete takes out.

Implements Copyable, Movable

Fields
task: FileScanTask
positions: List[UInt64]: The positions the filter matched, ascending. Live rows only: a row a previous delete already removed is not matched again.
existing: List[UInt64]: Positions already deleted by the delete files that apply to this data file — what a new deletion vector has to absorb.
live_rows: Int64: Rows still alive in the file before this delete.
whole_file: Bool: Every live row matched, so the file goes rather than being marked up.
superseded: List[DataFile]: Delete files this commit replaces — the deletion vectors a new vector absorbs.

__init__

def __init__(out self, var task: FileScanTask, var positions: List[UInt64], var existing: List[UInt64], live_rows: Int64, whole_file: Bool, var superseded: List[DataFile])

Functions

delete_mode_of

def delete_mode_of(properties: Dict[String, String]) -> String

write.delete.mode, defaulting to copy-on-write as Java does.

plan_row_deletes

def plan_row_deletes(scan: TableScan, filter_dsl: String, options: ScanOptions = ScanOptions()) -> List[RowDelete]

Which rows of which data files a DELETE ... WHERE takes out.

existing_complement

def existing_complement(record_count: Int64, already: List[Bool]) -> List[UInt64]

Every position of a data file that is not already deleted.

deleted_row_count

def deleted_row_count(plans: List[RowDelete]) -> Int64

write_deletion_vectors

def write_deletion_vectors(io: FileIO, location: String, plans: List[RowDelete]) -> List[DataFile]

One Puffin file holding a vector per partially-deleted data file.

Every vector carries the file's whole delete set — the positions this commit adds, unioned with the ones already there — because a reader that finds a vector uses it alone.

write_position_deletes

def write_position_deletes(io: FileIO, location: String, plans: List[RowDelete], spec: PartitionSpec, options: WriteOptions) -> List[DataFile]

One Parquet file of (file_path, pos) per partition, sorted.

The two columns are written under the reserved ids the spec assigns them, and the rows are sorted by path then position, which is what makes a reader able to merge them without buffering the whole file.

surviving_batch

def surviving_batch(io: FileIO, plan: RowDelete, schema: Schema, spec: PartitionSpec, mapping: NameMapping, options: ScanOptions) -> RecordBatch

Everything left of one data file once this delete has taken its rows.

Read with no filter and with _pos, then dropped by position, rather than read with the negation of the filter: a three-valued NOT does not keep the rows a two-valued one drops, and "everything the delete did not match" is the only definition that cannot disagree with the delete itself.

rewrite_files

def rewrite_files(io: FileIO, metadata: TableMetadata, plans: List[RowDelete], options: ScanOptions) -> List[DataFile]

The replacement data files for every partially-deleted file.

prepare_delete

def prepare_delete(io: FileIO, metadata: TableMetadata, filter_dsl: String, mode: String = "", options: ScanOptions = ScanOptions()) -> AppendResult

DELETE FROM t WHERE <filter>, as one prepared snapshot.

Raises when the filter matches nothing, so a caller cannot commit an empty snapshot by accident.

prepare_delete_from

def prepare_delete_from(io: FileIO, metadata: TableMetadata, plans: List[RowDelete], mode: String, options: ScanOptions = ScanOptions()) -> AppendResult

prepare_delete with the planning already done — which is what a caller that needs the row count before it commits wants.

prepare_overwrite

def prepare_overwrite(io: FileIO, metadata: TableMetadata, batches: List[RecordBatch], filter_dsl: String = "[\22true\22]", options: ScanOptions = ScanOptions()) -> AppendResult

Delete every row the filter matches and add batches, in one snapshot.

Copy-on-write, always: an overwrite that left merge-on-read deletes behind would need the added rows to sort after the deletes it wrote in the same commit, and the spec's scope rules do not let a delete from one commit skip data added by the same commit.

prepare_dynamic_partition_overwrite

def prepare_dynamic_partition_overwrite(io: FileIO, metadata: TableMetadata, batches: List[RecordBatch]) -> AppendResult

Replace exactly the partitions the new rows land in.

No row is examined: the new data files say which partitions they belong to, and every existing data file in one of those partitions is removed whole. A partition the new data does not touch is left alone, which is the difference between this and an unfiltered overwrite.

write_equality_deletes

def write_equality_deletes(io: FileIO, location: String, rows: RecordBatch, schema: Schema, equality_ids: List[Int], options: WriteOptions) -> DataFile

One equality delete file: the values, and the ids they match on.

A row is deleted when every one of the equality_ids columns equals the delete row's value for it, null included — the spec's "a null value in a delete column matches a row if the row's value is null". The file holds only those columns, under the table's own field ids, and the manifest entry carries equality_ids so a reader knows which they are.

Written with the unpartitioned spec, which is what makes the file apply to the whole table: a partitioned equality delete only reaches its own partition, and pairing one with the right partition tuple is a job for a caller that knows the data. Format version 2 only — v3 replaced these with deletion vectors for positions and v4 is deprecating writing them.

prepare_equality_delete

def prepare_equality_delete(io: FileIO, metadata: TableMetadata, rows: RecordBatch, equality_ids: List[Int]) -> AppendResult

DELETE FROM t WHERE (a, b) IN (<rows>), as one prepared snapshot.

Aliases

comptime PROP_DELETE_MODE = String("write.delete.mode")
comptime MODE_COPY_ON_WRITE = String("copy-on-write")
comptime MODE_MERGE_ON_READ = String("merge-on-read")
comptime POS_DELETE_SCHEMA_JSON = String("{\22schema-id\22:0,\22type\22:\22struct\22,\22fields\22:[{\22id\22:2147483546,\22name\22:\22file_path\22,\22required\22:true,\22type\22:\22string\22},{\22id\22:2147483545,\22name\22:\22pos\22,\22required\22:true,\22type\22:\22long\22}]}")