magmalake iceberg

iceberg · API

iceberg.puffin

Puffin files: the sidecar format Iceberg keeps deletion vectors and sketches in.

Magic Blob1 Blob2 ... BlobN Footer Footer := Magic FooterPayload FooterPayloadSize Flags Magic

Magic is PFA1 (0x50 0x46 0x41 0x31). FooterPayloadSize and Flags are little-endian 4-byte integers; bit 0 of the first flag byte says the payload is a single LZ4 frame rather than plain UTF-8 JSON. The payload is a FileMetadata object: a list of BlobMetadata (type, fields, snapshot-id, sequence-number, offset, length, compression-codec, properties) plus file-level properties.

Two ways in, and the difference matters:

  • PuffinFile.open reads the footer — the trailing 12 bytes first, then just the payload — which is four small ranged reads instead of downloading the file. Use it to discover what a Puffin file holds.
  • read_dv_blob skips the footer entirely. A delete manifest entry already records content_offset and content_size_in_bytes for its deletion vector, and the spec requires them to match the footer exactly, so a scan reads the blob directly and never parses the footer at all.

PuffinWriter is the other direction: blobs in, a complete file out, with the footer written as plain JSON or as a single LZ4 frame. It is what writes the deletion vectors a merge-on-read delete produces.

Only deletion-vector-v1 is decoded, by roaring.mojo, which verifies the blob's magic and CRC-32. apache-datasketches-theta-v1 blobs are listed with their metadata (including the ndv property) but their sketches are not decoded: that needs the DataSketches Theta format, which nothing in Mojo implements and which no reader needs.

Structs

BlobMetadata

struct BlobMetadata

One entry of a Puffin footer's blobs list.

Implements Copyable, Movable, Writable

Fields
type: String
fields: List[Int]
snapshot_id: Int64
sequence_number: Int64
offset: Int64
length: Int64
compression_codec: String: Empty when the blob is uncompressed, as the spec's absent field means.
property_keys: List[String]
property_values: List[String]

property

def property(self, key: String) -> String

has_property

def has_property(self, key: String) -> Bool

is_deletion_vector

def is_deletion_vector(self) -> Bool

referenced_data_file

def referenced_data_file(self) -> String

A deletion vector's referenced-data-file property.

cardinality

def cardinality(self) -> Int64

A deletion vector's declared number of deleted rows, or -1.

write_to

def write_to(self, mut writer: T)

PuffinFile

struct PuffinFile

A Puffin file's footer, and the location it came from.

Implements Copyable, Movable

Fields
location: String
blobs: List[BlobMetadata]
property_keys: List[String]
property_values: List[String]
footer_compressed: Bool

__init__

def __init__(out self, var location: String, var blobs: List[BlobMetadata], var property_keys: List[String], var property_values: List[String], footer_compressed: Bool)

property

def property(self, key: String) -> String

created_by

def created_by(self) -> String

open

def open(io: FileIO, location: String) -> Self

Read a Puffin footer without reading the blobs.

Four ranged reads: the trailing 12 bytes, then the payload, then the two magics that bracket it. On a local file that is one seek each; on S3 it is a HEAD plus two GETs with Range.

from_payload

def from_payload(var location: String, payload: Span[UInt8], compressed: Bool) -> Self

Parse a FileMetadata JSON payload.

PuffinWriter

struct PuffinWriter

Builds a Puffin file blob by blob, footer last.

var w = PuffinWriter()
_ = w.add_deletion_vector(data_file_path, bitmap)
var bytes = w^.finish()

The blobs go into the body as they arrive, each recording the offset and length its BlobMetadata will claim; finish appends the footer, whose payload is the FileMetadata JSON — optionally as a single LZ4 frame, which is the only compression the format allows there.

A deletion-vector-v1 blob is written the way the spec requires and no other way: never compressed, snapshot-id and sequence-number both -1 (a DV is not owned by a snapshot; the manifest entry that references it is), fields empty, and the two properties a reader needs — referenced-data-file and cardinality.

Implements Movable

Fields
body: List[UInt8]: Magic, then every blob written so far.
blobs: List[BlobMetadata]
property_keys: List[String]
property_values: List[String]

__init__

def __init__(out self, var created_by: String = PUFFIN_CREATED_BY)

set_property

def set_property(mut self, var key: String, var value: String)

Set a file-level property, replacing any previous value.

add_blob

def add_blob(mut self, var type: String, var fields: List[Int], snapshot_id: Int64, sequence_number: Int64, data: Span[UInt8], var compression_codec: String = "", var property_keys: List[String] = List(), var property_values: List[String] = List()) -> Int

Append one blob; returns its index in blobs.

compression_codec is applied here: lz4 means a single LZ4 frame, which is the codec Puffin defines for blob payloads. Anything else is refused rather than written under a name a reader would trust.

add_deletion_vector

def add_deletion_vector(mut self, referenced_data_file: String, bitmap: Bitmap64) -> Int

One deletion-vector-v1 blob for one data file.

The bitmap is framed by roaring.mojo — big-endian length, the D1 D3 39 64 magic, the portable serialisation and a CRC-32 over both — which is byte-for-byte what a reader verifies.

blob

def blob(self, i: Int) -> BlobMetadata

footer_json

def footer_json(self) -> String

The FileMetadata payload: every blob, then the file properties.

finish

def finish(self, compress_footer: Bool = False) -> List[UInt8]

The complete file: body, then Magic payload size flags Magic.

compress_footer writes the payload as one LZ4 frame and sets bit 0 of the first flag byte, which is the only flag the format defines. LZ4 is also the only footer codec Puffin allows — there is no ZSTD footer — so this is a Bool rather than a codec name.

Functions

read_blob_bytes

def read_blob_bytes(io: FileIO, location: String, offset: Int64, length: Int64) -> List[UInt8]

The raw bytes of one blob, by the offset and length a manifest records.

decompress_blob

def decompress_blob(var data: List[UInt8], codec: String) -> List[UInt8]

Undo a blob's compression-codec. Only lz4 is reachable here: zstd is legal in the format but no blob type Iceberg defines uses it, and deletion vectors must omit the codec entirely.

read_deletion_vector

def read_deletion_vector(io: FileIO, location: String, offset: Int64, length: Int64) -> Bitmap64

Read and decode one deletion-vector-v1 blob.

decode_iceberg_dv checks the blob's own framing — the big-endian length, the D1 D3 39 64 magic and the CRC-32 over both — so a truncated or corrupted vector is an error here rather than silently missing deletes.

deleted_positions

def deleted_positions(io: FileIO, location: String, offset: Int64, length: Int64) -> List[UInt64]

The deleted row positions of one deletion vector, ascending.

Aliases

comptime PUFFIN_MAGIC_0 = UInt8(80)
comptime PUFFIN_MAGIC_1 = UInt8(70)
comptime PUFFIN_MAGIC_2 = UInt8(65)
comptime PUFFIN_MAGIC_3 = UInt8(49)
comptime PUFFIN_MAGIC_LEN = 4
comptime PUFFIN_FOOTER_TAIL = 12

FooterPayloadSize (4) + Flags (4) + trailing Magic (4).

comptime BLOB_DELETION_VECTOR_V1 = String("deletion-vector-v1")
comptime BLOB_THETA_V1 = String("apache-datasketches-theta-v1")
comptime PUFFIN_CREATED_BY = String("iceberg.mojo")

The created-by file property, so a Puffin file says who made it.