`arrow_mlake.carrow_stream`

Producing an ArrowArrayStream: the C Data Interface, outbound, in batches.

carrow.export_c hands one array across the boundary. This hands a sequence of them, which is the shape every consumer wants for a scan — DuckDB, pyarrow and polars all take an ArrowArrayStream and pull batches until it ends.

Why this is the local answer

Flight moves Arrow between processes. In one process there is nothing to move: the consumer reads our buffers where they already are, and the only thing that crosses is a struct of pointers. A DuckDB extension loading a Mojo shared library is exactly that case, and reaching for a wire protocol there would add an encode and a decode to a problem with neither.

The contract, and who frees what

A stream is five fields: four C function pointers and a private_data the producer owns. The consumer calls get_next until it receives an array whose release is NULL — that, not an error, is how a stream ends — and then calls release on the stream itself.

Ownership is strict and worth stating because getting it wrong is a double-free rather than a wrong answer:

Every array handed out becomes the consumer's to release. This exports each batch up front and then moves it out on get_next, clearing our copy so nothing is released twice.
The schema is handed out the same way, once per get_schema call, so a consumer that asks twice gets two independently-releasable copies.
Whatever the consumer never asked for is freed by release.

The callbacks are abi("C") and take raw addresses because that is what the other side of the boundary can call. Everything they touch is heap-allocated and outlives the Mojo frame that built it — a stream handed to C and then pointing at a stack is the mistake this file exists to not make.

Functions

`export_stream_of`

def export_stream_of(var exported: List[ExportedArray]) -> Int

Turn already-exported arrays into a stream; returns its address.

The general form. Each ExportedArray has copied its buffers out of its arena already, so the batches need not come from the same arena — which is what a scan produces, one arena per batch. Ownership passes here: the stream releases everything the consumer does not take.

Every array is exported before the consumer sees the first one. A lazier stream that read inside get_next would hold one batch instead of all of them; that needs a scan that can be resumed from inside a C callback, and this Mojo has no way to keep such an object alive across the boundary.

`export_stream`

def export_stream(arena: ArrayArena, roots: List[Int]) -> Int

Export one arena's arrays as a stream; returns its address.

The caller hands the address on and then forgets it: the stream owns everything from here, and the consumer's release frees it — including the stream struct, which is why this returns an address rather than a value it would have to outlive.