# arrow_mlake.carrow_stream

> A module of arrow-mlake's API.

Producing an `ArrowArrayStream`: the C Data Interface, outbound, in batches.

`carrow.export_c` hands one array across the boundary. This hands a *sequence*
of them, which is the shape every consumer wants for a scan — DuckDB, pyarrow
and polars all take an `ArrowArrayStream` and pull batches until it ends.

## Why this is the local answer

Flight moves Arrow between processes. In one process there is nothing to move:
the consumer reads our buffers where they already are, and the only thing that
crosses is a struct of pointers. A DuckDB extension loading a Mojo shared
library is exactly that case, and reaching for a wire protocol there would add
an encode and a decode to a problem with neither.

## The contract, and who frees what

A stream is five fields: four C function pointers and a `private_data` the
producer owns. The consumer calls `get_next` until it receives an array whose
`release` is NULL — that, not an error, is how a stream ends — and then calls
`release` on the stream itself.

Ownership is strict and worth stating because getting it wrong is a
double-free rather than a wrong answer:

* Every array handed out becomes **the consumer's** to release. This exports
  each batch up front and then *moves* it out on `get_next`, clearing our copy
  so nothing is released twice.
* The schema is handed out the same way, once per `get_schema` call, so a
  consumer that asks twice gets two independently-releasable copies.
* Whatever the consumer never asked for is freed by `release`.

The callbacks are `abi("C")` and take raw addresses because that is what the
other side of the boundary can call. Everything they touch is heap-allocated
and outlives the Mojo frame that built it — a stream handed to C and then
pointing at a stack is the mistake this file exists to not make.

## Functions

### `export_stream_of`

```mojo
def export_stream_of(var exported: List[ExportedArray]) -> Int
```

Turn already-exported arrays into a stream; returns its address.

The general form. Each `ExportedArray` has copied its buffers out of its
arena already, so the batches need not come from the same arena — which is
what a scan produces, one arena per batch. Ownership passes here: the
stream releases everything the consumer does not take.

Every array is exported before the consumer sees the first one. A lazier
stream that read inside `get_next` would hold one batch instead of all of
them; that needs a scan that can be resumed from inside a C callback, and
this Mojo has no way to keep such an object alive across the boundary.

### `export_stream`

```mojo
def export_stream(arena: ArrayArena, roots: List[Int]) -> Int
```

Export one arena's arrays as a stream; returns its address.

The caller hands the address on and then forgets it: the stream owns
everything from here, and the consumer's `release` frees it — including
the stream struct, which is why this returns an address rather than a
value it would have to outlive.