API reference

Part of magmalake: data lake building blocks in Mojo.

from arrow_mlake import ImportedStream

var s = ImportedStream(stream_address)   # a producer's ArrowArrayStream
while True:
    var got = s.next()
    if not got:
        break
    ref batch = got.value()
    print(batch.num_rows, "rows,", batch.num_columns(), "columns")

Seven modules, and no dependencies on anything but std:

• arrow_mlake.arrow — ArrowType, ArrayData and ArrayArena: one Arrow array as the columnar spec lays it out, with nesting held in an arena because Mojo does not allow a struct to contain a List of itself.
• arrow_mlake.carrow — the C Data Interface outbound. export_c turns an array into an ArrowSchema/ArrowArray pair any Arrow implementation can consume.
• arrow_mlake.carrow_import — the same interface inbound, including ArrowArrayStream, which is the shape a scan arrives in.
• arrow_mlake.carrow_stream — export_stream, the outbound half of that: a sequence of batches as one ArrowArrayStream a consumer pulls from.
• arrow_mlake.batch — RecordBatch, a run of rows as one array per column.
• arrow_mlake.ipc — the Arrow IPC stream format: IpcWriter and read_ipc_stream, batches to bytes and back, for files, sockets and shared mappings. arrow_mlake.flatbuf is the FlatBuffers builder and reader under it.

This tin is a stopgap. The community Arrow implementation for Mojo is marrow, which is more complete than this and which this should be replaced by as soon as it builds on Mojo 1.x. See the README.