magmalake arrow-mlake

arrow-mlake · API

arrow_mlake.batch

RecordBatch — a run of rows as Arrow arrays, one per column.

A batch is {arena, roots, num_rows} and nothing else: the arrays live in one ArrayArena, roots names the column at the top of each tree, and num_rows is the length every root shares. That is the shape a Parquet reader hands back, the shape import_batch_c unwraps a +s root into, and the shape a scan over the C Data Interface arrives in, so it lives here rather than in any one of them.

The column_* accessors widen a column to a Mojo List — Int64 for any integer, date, time or timestamp, Float64 for any float, String for utf8 and binary — paired with a per-element validity list. They copy, which is the point: they are for reading a column out into ordinary Mojo values, not for the columnar path, where ArrayData's buffers are read in place.

var b = import_batch_c(array_addr, schema_addr)
for i in range(b.num_columns()):
    print(b.name(i), String(b.type(i)))
var ids = b.column_i64(0)      # (values, validity)

Structs

RecordBatch

struct RecordBatch

A contiguous run of rows as Arrow arrays, one per selected column.

Implements Copyable, Defaultable, Movable

Fields
arena: ArrayArena
roots: List[Int]
num_rows: Int

__init__

def __init__(out self)

num_columns

def num_columns(self) -> Int

column

def column(ref self, i: Int) -> ref[self.arena.nodes["element"]] ArrayData

child

def child(ref self, node: Int, k: Int) -> ref[self.arena.nodes["element"]] ArrayData

name

def name(self, i: Int) -> String

type

def type(self, i: Int) -> ArrowType

export_c

def export_c(self, i: Int) -> ExportedArray

Column i over the Arrow C Data Interface. The result owns copies of every buffer, so it outlives this batch.

column_i64

def column_i64(self, i: Int) -> Tuple[List[Int64], List[Bool]]

column_f64

def column_f64(self, i: Int) -> Tuple[List[Float64], List[Bool]]

column_bool

def column_bool(self, i: Int) -> Tuple[List[Bool], List[Bool]]

column_str

def column_str(self, i: Int) -> Tuple[List[String], List[Bool]]

Functions

array_i64_into

def array_i64_into(a: ArrayData, mut vals: List[Int64], mut valid: List[Bool])

Widen any integer, date, time or timestamp array to Int64.

array_f64_into

def array_f64_into(a: ArrayData, mut vals: List[Float64], mut valid: List[Bool])

array_bool_into

def array_bool_into(a: ArrayData, mut vals: List[Bool], mut valid: List[Bool])

array_str_into

def array_str_into(a: ArrayData, mut vals: List[String], mut valid: List[Bool])

array_i64

def array_i64(a: ArrayData) -> Tuple[List[Int64], List[Bool]]

array_f64

def array_f64(a: ArrayData) -> Tuple[List[Float64], List[Bool]]

array_bool

def array_bool(a: ArrayData) -> Tuple[List[Bool], List[Bool]]

array_str

def array_str(a: ArrayData) -> Tuple[List[String], List[Bool]]