# arrow-mlake API reference

> The `arrow_mlake` package, from arrow-mlake 0.2.1, as published on mojoshelf (commit 178aae6e0da0), by `mojo doc` from Mojo 1.2.0.

## Modules

- [arrow_mlake.arrow](https://magmalake.dev/s/arrow-mlake/api/arrow.md): Arrow types and Arrow array data, in the layout the C Data Interface wants.
- [arrow_mlake.batch](https://magmalake.dev/s/arrow-mlake/api/batch.md): `RecordBatch` — a run of rows as Arrow arrays, one per column.
- [arrow_mlake.carrow](https://magmalake.dev/s/arrow-mlake/api/carrow.md): The Arrow C Data Interface — this library's public output contract.
- [arrow_mlake.carrow_import](https://magmalake.dev/s/arrow-mlake/api/carrow_import.md): The Arrow C Data Interface, inbound — another runtime's arrays read into this one.
- [arrow_mlake.carrow_shared](https://magmalake.dev/s/arrow-mlake/api/carrow_shared.md): Publish a batch into a shared mapping, for a consumer in another process.
- [arrow_mlake.carrow_stream](https://magmalake.dev/s/arrow-mlake/api/carrow_stream.md): Producing an `ArrowArrayStream`: the C Data Interface, outbound, in batches.
- [arrow_mlake.flatbuf](https://magmalake.dev/s/arrow-mlake/api/flatbuf.md): FlatBuffers, as much as Arrow IPC metadata needs: a builder and a reader.
- [arrow_mlake.ipc](https://magmalake.dev/s/arrow-mlake/api/ipc.md): The Arrow IPC stream format: `RecordBatch`es to bytes and back.
- [arrow_mlake.memory_shim](https://magmalake.dev/s/arrow-mlake/api/memory_shim.md): Opening a region, for a caller that does not want the allocator's types.

`arrow-mlake.mojo` — the Arrow memory layout and the C Data Interface, in pure Mojo.

Part of magmalake: data lake building blocks in Mojo.

```mojo
from arrow_mlake import ImportedStream

var s = ImportedStream(stream_address)   # a producer's ArrowArrayStream
while True:
    var got = s.next()
    if not got:
        break
    ref batch = got.value()
    print(batch.num_rows, "rows,", batch.num_columns(), "columns")
```

Seven modules, and no dependencies on anything but `std`:

* `arrow_mlake.arrow` — `ArrowType`, `ArrayData` and `ArrayArena`: one Arrow
  array as the columnar spec lays it out, with nesting held in an arena
  because Mojo does not allow a struct to contain a `List` of itself.
* `arrow_mlake.carrow` — the C Data Interface outbound. `export_c` turns an
  array into an `ArrowSchema`/`ArrowArray` pair any Arrow implementation can
  consume.
* `arrow_mlake.carrow_import` — the same interface inbound, including
  `ArrowArrayStream`, which is the shape a scan arrives in.
* `arrow_mlake.carrow_stream` — `export_stream`, the outbound half of that:
  a sequence of batches as one `ArrowArrayStream` a consumer pulls from.
* `arrow_mlake.batch` — `RecordBatch`, a run of rows as one array per column.
* `arrow_mlake.ipc` — the Arrow IPC stream format: `IpcWriter` and
  `read_ipc_stream`, batches to bytes and back, for files, sockets and shared
  mappings. `arrow_mlake.flatbuf` is the FlatBuffers builder and reader under
  it.

**This tin is a stopgap.** The community Arrow implementation for Mojo is
[marrow](https://github.com/kszucs/marrow), which is more complete than this
and which this should be replaced by as soon as it builds on Mojo 1.x. See the
README.