magmalake iceberg-rs

Tins · iceberg-rs

API reference

The iceberg_rs package, from iceberg-rs 0.1.3, as published on mojoshelf (commit 096434e40a0d), by mojo doc from Mojo 1.2.0. Source.

iceberg_rs — Apache Iceberg for Mojo, over a Rust cdylib wrapping iceberg-rust.

Mirrors lancedb.mojo's FFI pattern: one relocatable library (ffi/src/*.rs -> $CONDA_PREFIX/lib/libicebergrsmojo.{dylib,so}, built by the iceberg-shim pixi package) loaded through an OwnedDLHandle. The handle is passed as a BORROWED read param to every worker so Mojo's ASAP destruction can't dlclose the library mid-call (the flare gotcha) — even though each struct also owns a handle as a field, the borrow keeps it pinned across the C call.

Nothing but integers, raw buffers and UTF-8 bytes crosses FFI. Iceberg handles (catalog, table, scan, batch) are opaque Int pointers; results that don't fit in a register come back as heap C strings which _take_string copies out through ib_string_len / ib_string_copy and frees — so no Mojo code ever dereferences a foreign pointer.

from iceberg_rs import Catalog

var cat = Catalog.sql("sqlite:/tmp/cat.db?mode=rwc", "file:///tmp/warehouse")
cat.create_namespace("sales")
var t = cat.create_table("sales", "orders", SCHEMA_JSON, SPEC_JSON)

var b = t.builder()
b.int_col("id", [Int64(1), Int64(2)])
b.str_col("region", ["eu", "us"])
t.append(b.build())
var snapshot = t.commit()

var scan = t.scan(columns="id,region", filter='["=","region","eu"]')
for batch in scan.batches():
    print(batch.int_col("id"))

Structs

Batch

struct Batch

One Arrow record batch owned by the Rust side.

Columns are copied out on demand and cast on the way: int_col reads any integer, date, time, timestamp or boolean column as Int64 (timestamps in microseconds since the epoch, dates in days), float_col reads any numeric column as Float64, str_col reads any column as UTF-8.

Implements Movable

Fields
lib: OwnedDLHandle
ptr: Int

__init__

def __init__(out self, var lib: OwnedDLHandle, ptr: Int)

__deinit__

def __deinit__(deinit self)

num_rows

def num_rows(self) -> Int

num_columns

def num_columns(self) -> Int

schema_json

def schema_json(self) -> String

JSON array of {"name", "type", "nullable"} for the batch's columns.

int_col

def int_col(self, name: String) -> List[Int64]

Column name as Int64. Nulls read back as 0 — use validity to tell them apart.

float_col

def float_col(self, name: String) -> List[Float64]

Column name as Float64. Nulls read back as 0.0.

bool_col

def bool_col(self, name: String) -> List[Bool]

Column name as Bool. Nulls read back as False.

str_col

def str_col(self, name: String) -> List[String]

Column name as UTF-8 strings. Nulls read back as empty strings.

validity

def validity(self, name: String) -> List[Bool]

One flag per row of column name: True when the value is present.

export_c_data

def export_c_data(self) -> ArrowCData

Export this batch through the Arrow C Data Interface as a struct array (one child per column) — the shape marrow's c_data.mojo imports.

ArrowCData

struct ArrowCData

Storage for one Arrow C Data Interface struct pair.

The two structs live in Int64-backed Mojo buffers (guaranteed 8-byte aligned, sized from the shim's own sizeof), so a consumer such as marrow can read them straight from array_ptr() / schema_ptr(). Releasing is automatic: the destructor calls the structs' release callbacks through ib_arrow_release.

Implements Movable

Fields
lib: OwnedDLHandle
array_buf: List[Int64]
schema_buf: List[Int64]
live: Bool

__init__

def __init__(out self, var lib: OwnedDLHandle, sizer: OwnedDLHandle)

__deinit__

def __deinit__(deinit self)

array_ptr

def array_ptr(self) -> Int

Address of the ArrowArray struct.

schema_ptr

def schema_ptr(self) -> Int

Address of the ArrowSchema struct.

Scan

struct Scan

A planned table scan. Pull batches with next() until it returns an empty Optional, or inspect the plan with plan_files() without reading any data.

Implements Movable

Fields
lib: OwnedDLHandle
ptr: Int

__init__

def __init__(out self, var lib: OwnedDLHandle, ptr: Int)

__deinit__

def __deinit__(deinit self)

plan_files

def plan_files(self) -> String

JSON array of the file scan tasks: data file path, format, record count, byte range, and any delete files that apply to each.

next

def next(mut self) -> Optional[Batch]

The next record batch, or None at the end of the stream.

next_c_data

def next_c_data(mut self) -> Optional[ArrowCData]

The next record batch straight through the Arrow C Data Interface, skipping the opaque handle entirely.

batches

def batches(mut self) -> List[Batch]

Drain the scan into a list of batches. Convenient for the small results a parity check produces; stream with next() for large tables.

BatchBuilder

struct BatchBuilder

Builds one record batch column by column against a table's schema.

Each *_col call names a column of the table; the values are cast to the column's declared type on the Rust side, so int_col feeds int, long, date (days) and timestamp (microseconds) columns alike. Columns you leave out must be optional in the schema; they are filled with nulls.

Implements Movable

Fields
lib: OwnedDLHandle
ptr: Int

__init__

def __init__(out self, var lib: OwnedDLHandle, ptr: Int)

__deinit__

def __deinit__(deinit self)

int_col

def int_col(mut self, name: String, values: List[Int64])
def int_col(mut self, name: String, values: List[Int64], valid: List[Bool])

float_col

def float_col(mut self, name: String, values: List[Float64])
def float_col(mut self, name: String, values: List[Float64], valid: List[Bool])

bool_col

def bool_col(mut self, name: String, values: List[Bool])
def bool_col(mut self, name: String, values: List[Bool], valid: List[Bool])

str_col

def str_col(mut self, name: String, values: List[String])
def str_col(mut self, name: String, values: List[String], valid: List[Bool])

build

def build(self) -> Batch

Assemble the staged columns into a Batch.

Table

struct Table

A loaded Iceberg table: metadata, scans, and append/commit.

iceberg-rust 0.10 is append-only. append writes Parquet data files right away and stages them; commit turns everything staged into exactly one new snapshot and returns its id.

Implements Movable

Fields
lib: OwnedDLHandle
ptr: Int

__init__

def __init__(out self, var lib: OwnedDLHandle, ptr: Int)

__deinit__

def __deinit__(deinit self)

refresh

def refresh(mut self)

Re-load from the catalog, picking up snapshots committed elsewhere.

metadata_json

def metadata_json(self) -> String

The table's full current metadata.json text.

schema_json

def schema_json(self) -> String

The current schema in Iceberg metadata JSON form.

partition_spec_json

def partition_spec_json(self) -> String

The default partition spec as JSON.

snapshots_json

def snapshots_json(self) -> String

JSON array of the table's snapshots, oldest first.

location

def location(self) -> String

The table's base location.

current_snapshot_id

def current_snapshot_id(self) -> Int64

The current snapshot id, or 0 when the table has no snapshot yet.

builder

def builder(self) -> BatchBuilder

A BatchBuilder bound to this table's schema.

append

def append(mut self, batch: Batch)

Stage batch for append: writes its Parquet data file(s) now and remembers them until commit.

append_c_data

def append_c_data(mut self, mut data: ArrowCData)

Stage an Arrow C Data Interface struct pair for append. Consumes the pair — data is inert afterwards.

pending_files

def pending_files(self) -> Int

Data files staged but not yet committed.

commit

def commit(mut self) -> Int64

Commit every staged append as one new snapshot; returns its id. With nothing staged this is a no-op returning the current snapshot id.

rollback

def rollback(mut self)

Drop every staged (uncommitted) append.

scan

def scan(self) -> Scan

Scan every column of the current snapshot.

def scan(self, columns: String) -> Scan

Scan a comma-separated column projection of the current snapshot.

def scan(self, columns: String, filter: String) -> Scan

Scan a projection of the current snapshot under a filter (see the filter DSL in the README).

def scan(self, columns: String, snapshot_id: Int64, filter: String) -> Scan

Scan columns (empty for all) of snapshot_id (0 for current) under filter (empty for none).

Catalog

struct Catalog

An Iceberg catalog: REST, or SQL over sqlite/postgres/mysql.

Implements Movable

Fields
lib: OwnedDLHandle
ptr: Int

__init__

def __init__(out self, var lib: OwnedDLHandle, ptr: Int)

__deinit__

def __deinit__(deinit self)

sql

def sql(uri: String, warehouse: String) -> Self

A SQL catalog. uri is an sqlx connection string (sqlite:/abs/path/catalog.db?mode=rwc, postgres://…, mysql://…) and warehouse is the data root (file:///…, s3://…). The sqlite flavour needs no infrastructure at all — it is what the test-suite uses.

rest

def rest(uri: String) -> Self

A REST catalog with no warehouse hint and no extra properties.

def rest(uri: String, warehouse: String, props_json: String) -> Self

A REST catalog. props_json is a JSON object of extra properties passed to the REST client — OAuth2 (credential, oauth2-server-uri, scope, token) and header.* entries go here. See examples/rest_polaris.md.

list_namespaces

def list_namespaces(self) -> String

JSON array of top-level namespaces, each a dotted string.

def list_namespaces(self, parent: String) -> String

JSON array of the namespaces under parent (empty for top level).

create_namespace

def create_namespace(mut self, ns: String)
def create_namespace(mut self, ns: String, props_json: String)

namespace_exists

def namespace_exists(self, ns: String) -> Bool

drop_namespace

def drop_namespace(mut self, ns: String)

list_tables

def list_tables(self, ns: String) -> String

JSON array of the table names in ns.

table_exists

def table_exists(self, ns: String, name: String) -> Bool

load_table

def load_table(self, ns: String, name: String) -> Table

drop_table

def drop_table(mut self, ns: String, name: String)

create_table

def create_table(mut self, ns: String, name: String, schema_json: String) -> Table

Create an unpartitioned table. schema_json is Iceberg's metadata JSON schema form.

def create_table(mut self, ns: String, name: String, schema_json: String, partition_spec_json: String) -> Table

Create a table with a partition spec (unbound partition spec JSON).

def create_table(mut self, ns: String, name: String, schema_json: String, partition_spec_json: String, props_json: String) -> Table

Functions

version

def version() -> String

The version of the iceberg Rust crate the shim was built against.