# arrow_mlake.carrow_import

> A module of arrow-mlake's API.

The Arrow C Data Interface, inbound — another runtime's arrays read into this one.

`arrow_mlake.carrow` is the outbound half: an `ArrayData` tree becomes a C
`ArrowSchema`/`ArrowArray` pair that pyarrow, DuckDB or a Rust crate can
consume. This module is its inverse. `import_c` turns that same pair back into
an `ArrayData` in an `ArrayArena`, and `ImportedStream` drives the third struct
of the interface, `ArrowArrayStream`, which is how a producer hands over a
*sequence* of batches:

```c
struct ArrowArrayStream {
  int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out);
  int (*get_next)(struct ArrowArrayStream*, struct ArrowArray* out);
  const char* (*get_last_error)(struct ArrowArrayStream*);
  void (*release)(struct ArrowArrayStream*);
  void* private_data;
};
```

That is what a scan returns, so it is the entry point a LanceDB or ADBC
binding actually needs.

**Importing copies.** `ArrayData` owns its buffers as `List[UInt8]`, so every
buffer a producer hands over is copied into Mojo memory. The alternative — an
`ArrayData` that borrows a foreign pointer and keeps the producer's release
callback alive behind it — would be zero-copy but changes the type every
consumer of `ArrayData` reads, so it is deliberately not what this does. On a
16-column, 65k-row batch the copy runs at memory-copy speed, which is a
rounding error next to decoding the same data out of a file format.

**Who releases what.** The C Data Interface makes the *consumer* responsible
for the structs it is handed: it calls the root's `release`, which frees the
whole tree, and a released struct is recognised by a null `release` pointer.
Children are never released on their own — the root owns them. This module
implements exactly that side of the convention:

* `import_c` **borrows**. It reads the pair and copies out of it, and it does
  not release anything. A caller that owns the pair — because a producer moved
  it into caller storage — decides when to release it.
* `ImportedArray` **owns**. Constructing one asserts "this pair is mine now";
  it releases the root array and the root schema exactly once, on `release()`
  or on destruction, and `release()` is idempotent because it checks for the
  null `release` pointer first.
* `ImportedStream` owns the stream, the schema it pulled out of `get_schema`,
  and every array `get_next` moves into it. Each array is released as soon as
  its batch has been copied, so a long scan holds one batch of producer memory
  at a time and not the whole stream.

**Nothing about a producer is trusted.** `n_buffers` is checked against the
format string, `n_children` against the type, offsets for monotonicity and
against the child length they index into, and every format string this module
cannot name is an error carrying the string itself. The interface does *not*
carry buffer sizes — a `const void**` and nothing more — so a truncated
`utf8` data buffer is undetectable by construction; what can be cross-checked
is checked, and the limit is stated rather than papered over.

```mojo
var arena = ArrayArena()
var root = import_c(arena, array_addr, schema_addr)   # borrowing
var batch = ImportedArray(array_addr, schema_addr).into_batch()  # owning
```

## Structs

### `ImportedArray`

```mojo
struct ImportedArray
```

One C `ArrowArray`/`ArrowSchema` pair, owned by us until released.

Construct this when a producer has *moved* the two structs into storage
you hold — which is what "consuming" means in the C Data Interface — and
it will call the root release callbacks exactly once, on `release()` or at
destruction. The addresses stay valid afterwards, but their contents do
not, so import before releasing.

Implements: `Movable`

Fields:

- `array: Int`: Address of the `ArrowArray`.
- `schema: Int`: Address of the `ArrowSchema`.

#### `__init__`

```mojo
def __init__(out self, array: Int, schema: Int)
```

#### `__deinit__`

```mojo
def __deinit__(deinit self)
```

#### `into_arena`

```mojo
def into_arena(self, mut arena: ArrayArena) -> Int
```

#### `into_batch`

```mojo
def into_batch(self) -> RecordBatch
```

#### `release`

```mojo
def release(mut self)
```

### `CArrowArrayStream`

```mojo
struct CArrowArrayStream
```

The C `ArrowArrayStream`, field for field. Pointers are addresses.

Implements: `Copyable`, `Movable`

Fields:

- `get_schema: Int`
- `get_next: Int`
- `get_last_error: Int`
- `release: Int`
- `private_data: Int`

### `ImportedStream`

```mojo
struct ImportedStream
```

A producer's `ArrowArrayStream`, consumed batch by batch.

This is the shape a scan arrives in: one schema up front, then arrays
until the producer signals the end by handing back a *released* array —
a null `release` pointer and nothing else. `get_next` returning non-zero
is a real error, and `get_last_error` says what it was; the two are
distinguished here because conflating them turns a failed scan into a
short one.

Ownership runs three ways and all three are ours. The stream is released
when this object dies. The schema is moved out of `get_schema` into
storage this object allocates, and released with it. Each array `get_next`
moves into us is released as soon as its batch has been copied, which is
what keeps a long scan to one batch of producer memory at a time rather
than the whole stream.

Implements: `Movable`

Fields:

- `stream: Int`: Address of the caller's `ArrowArrayStream`.

#### `__init__`

```mojo
def __init__(out self, stream: Int)
```

Take ownership of `stream` and pull its schema.

#### `__deinit__`

```mojo
def __deinit__(deinit self)
```

#### `format`

```mojo
def format(self) -> String
```

The root format string of the stream's schema, usually `+s`.

#### `schema_address`

```mojo
def schema_address(self) -> Int
```

#### `next`

```mojo
def next(mut self) -> Optional[RecordBatch]
```

The next batch, or nothing once the stream has ended.

#### `release`

```mojo
def release(mut self)
```

## Functions

### `release_c_array`

```mojo
def release_c_array(addr: Int)
```

Call a producer's `release` on an `ArrowArray`, once.

A released struct carries a null `release`, which is both how the producer
tells us it is gone and how this stays idempotent. Only ever called on a
*root*: the interface says children are freed by their parent, and calling
a child's release does nothing but mark it.

### `release_c_schema`

```mojo
def release_c_schema(addr: Int)
```

Call a producer's `release` on an `ArrowSchema`, once.

### `parse_format`

```mojo
def parse_format(fmt: StringSpan) -> ArrowType
```

One Arrow C Data Interface format string as an `ArrowType`.

Every string this library can *write* round-trips, plus the temporal and
decimal spellings a foreign producer will send that our writer never
emits. Everything else raises with the string in the message: guessing at
an unrecognised format would mean reinterpreting somebody else's buffer,
which is the one failure mode that produces wrong numbers silently rather
than an error.

### `n_children_for_type`

```mojo
def n_children_for_type(id: Int) -> Int
```

How many children a type must have, or -1 where any number is legal.

### `extension_name`

```mojo
def extension_name(metadata: Int) -> String
```

`ARROW:extension:name` out of a C schema's metadata block, if present.

The encoding is Arrow's own: an `int32` count, then that many
(`int32` length, bytes) pairs alternating key and value, native-endian and
unaligned. Nothing in the interface bounds the block, so the counts are
sanity-checked before they are used as loop bounds — a garbage `n` here
would otherwise be an unbounded read of somebody else's address space.

### `parquet_field_id`

```mojo
def parquet_field_id(metadata: Int) -> Int32
```

`PARQUET:field_id` out of a C schema's metadata, or -1.

Arrow's Parquet writers put the Iceberg/Parquet field id here as decimal
text, and this library's `ArrayData` has a slot for it. Our own exporter
does not write the key, so a round trip through it drops the id; a foreign
producer that does write it keeps it.

### `import_c`

```mojo
def import_c(mut arena: ArrayArena, array: Int, schema: Int) -> Int
```

Copy a foreign C array into `arena`; return the index of its root.

Borrowing: the pair is read, never released. The caller owns it — see
`ImportedArray` for the owning form.

### `import_batch_c`

```mojo
def import_batch_c(array: Int, schema: Int) -> RecordBatch
```

A foreign C array as a `RecordBatch`.

A stream's arrays are struct arrays by definition — the struct's fields
are the batch's columns — so a `+s` root is unwrapped into one root per
field. Anything else becomes a one-column batch, which is what a caller
importing a bare column wants.

A struct root with nulls of its own has no representation here, because a
`RecordBatch` is a list of columns and not an array; that raises rather
than dropping the row-level nulls on the floor.

## Aliases

### `MAX_IMPORT_DEPTH`

```mojo
comptime MAX_IMPORT_DEPTH = 64
```

How deep a foreign type may nest before we call it a malformed producer.

Children are reached through pointers a producer wrote, so a cycle is possible
and would otherwise be an unbounded walk. Real Arrow schemas are a handful of
levels deep; anything past this is a bug on the other side of the interface.

### `StreamGetSchemaFn`

```mojo
comptime StreamGetSchemaFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin], Pointer[CArrowSchema, MutUntrackedOrigin]) abi("C") thin -> Int32
```

### `StreamGetNextFn`

```mojo
comptime StreamGetNextFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin], Pointer[CArrowArray, MutUntrackedOrigin]) abi("C") thin -> Int32
```

### `StreamGetLastErrorFn`

```mojo
comptime StreamGetLastErrorFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin]) abi("C") thin -> Int
```

### `StreamReleaseFn`

```mojo
comptime StreamReleaseFn = def(Pointer[CArrowArrayStream, MutUntrackedOrigin]) abi("C") thin -> None
```