arrow_mlake.carrow_import

arrow_mlake.carrow is the outbound half: an ArrayData tree becomes a C ArrowSchema/ArrowArray pair that pyarrow, DuckDB or a Rust crate can consume. This module is its inverse. import_c turns that same pair back into an ArrayData in an ArrayArena, and ImportedStream drives the third struct of the interface, ArrowArrayStream, which is how a producer hands over a sequence of batches:

struct ArrowArrayStream {
  int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out);
  int (*get_next)(struct ArrowArrayStream*, struct ArrowArray* out);
  const char* (*get_last_error)(struct ArrowArrayStream*);
  void (*release)(struct ArrowArrayStream*);
  void* private_data;
};

That is what a scan returns, so it is the entry point a LanceDB or ADBC binding actually needs.

Importing copies. ArrayData owns its buffers as List[UInt8], so every buffer a producer hands over is copied into Mojo memory. The alternative — an ArrayData that borrows a foreign pointer and keeps the producer's release callback alive behind it — would be zero-copy but changes the type every consumer of ArrayData reads, so it is deliberately not what this does. On a 16-column, 65k-row batch the copy runs at memory-copy speed, which is a rounding error next to decoding the same data out of a file format.

Who releases what. The C Data Interface makes the consumer responsible for the structs it is handed: it calls the root's release, which frees the whole tree, and a released struct is recognised by a null release pointer. Children are never released on their own — the root owns them. This module implements exactly that side of the convention:

• import_c borrows. It reads the pair and copies out of it, and it does not release anything. A caller that owns the pair — because a producer moved it into caller storage — decides when to release it.
• ImportedArray owns. Constructing one asserts "this pair is mine now"; it releases the root array and the root schema exactly once, on release() or on destruction, and release() is idempotent because it checks for the null release pointer first.
• ImportedStream owns the stream, the schema it pulled out of get_schema, and every array get_next moves into it. Each array is released as soon as its batch has been copied, so a long scan holds one batch of producer memory at a time and not the whole stream.

Nothing about a producer is trusted. n_buffers is checked against the format string, n_children against the type, offsets for monotonicity and against the child length they index into, and every format string this module cannot name is an error carrying the string itself. The interface does not carry buffer sizes — a const void** and nothing more — so a truncated utf8 data buffer is undetectable by construction; what can be cross-checked is checked, and the limit is stated rather than papered over.

var arena = ArrayArena()
var root = import_c(arena, array_addr, schema_addr)   # borrowing
var batch = ImportedArray(array_addr, schema_addr).into_batch()  # owning

struct ImportedArray

def __init__(out self, array: Int, schema: Int)

def __deinit__(deinit self)

def into_arena(self, mut arena: ArrayArena) -> Int

def into_batch(self) -> RecordBatch

def release(mut self)

struct CArrowArrayStream

struct ImportedStream

def __init__(out self, stream: Int)

def __deinit__(deinit self)

def format(self) -> String

def schema_address(self) -> Int

def next(mut self) -> Optional[RecordBatch]

def release(mut self)

def release_c_array(addr: Int)

def release_c_schema(addr: Int)

def parse_format(fmt: StringSpan) -> ArrowType

def n_children_for_type(id: Int) -> Int

def extension_name(metadata: Int) -> String

def parquet_field_id(metadata: Int) -> Int32

def import_c(mut arena: ArrayArena, array: Int, schema: Int) -> Int

def import_batch_c(array: Int, schema: Int) -> RecordBatch

comptime MAX_IMPORT_DEPTH = 64

comptime StreamGetSchemaFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin], Pointer[CArrowSchema, MutUntrackedOrigin]) abi("C") thin -> Int32

comptime StreamGetNextFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin], Pointer[CArrowArray, MutUntrackedOrigin]) abi("C") thin -> Int32

comptime StreamGetLastErrorFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin]) abi("C") thin -> Int

comptime StreamReleaseFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin]) abi("C") thin -> None