magmalake parquet

Tins · parquet

API reference

The parquet package, from parquet 0.11.7, as published on mojoshelf (commit c15f51aee06c), by mojo doc from Mojo 1.2.0. Source.

parquet.mojo — Apache Parquet in pure Mojo.

Part of magmalake: data lake building blocks in Mojo.

from parquet import ParquetReader

var r = ParquetReader.open("part-0.parquet")
var t = r.read_table()
for i in range(t.num_columns()):
    print(t.name(i), String(t.type(i)))

The Arrow side — ArrayData, ArrayArena, RecordBatch and the C Data Interface — is arrow-mlake.mojo, and is re-exported here so that from parquet.arrow import ArrayData and from parquet import RecordBatch keep resolving. Metadata comes from thrift.mojo, page CRC32s from hashes.mojo, Snappy from snappy.mojo and the DEFLATE half of GZIP from deflate.mojo — all pure Mojo and all consumed by source path. ZSTD and LZ4 need parquet_full, which pulls in the two FFI tins.

Modules

  • parquet.alp ALP — Adaptive Lossless floating-Point, Parquet encoding 10. 1 function
  • parquet.arrow Compatibility re-export. The Arrow layer lives in arrow-mlake.mojo now.
  • parquet.assemble Dremel record assembly: definition and repetition levels back into Arrow. 1 struct · 3 functions
  • parquet.bitio Bit-level readers: little-endian bit packing and the RLE/bit-packed hybrid. 3 structs · 10 functions
  • parquet.bloom Split-block bloom filters — might_contain for equality predicates. 1 struct · 1 function
  • parquet.carrow Compatibility re-export. The C Data Interface export lives in arrow-mlake.mojo now — see parquet.arrow for why.
  • parquet.carrow_import Compatibility re-export. The C Data Interface import lives in arrow-mlake.mojo now — see parquet.arrow for why.
  • parquet.codec Page compression codecs. 1 struct · 1 trait · 8 functions
  • parquet.convert Physical Parquet values to Arrow buffers. 3 functions
  • parquet.display Human- and JSON-readable rendering of decoded Arrow arrays, for the CLI. 6 functions
  • parquet.encoding Value encodings: PLAIN, dictionaries, the DELTA family, BYTE_STREAM_SPLIT. 1 struct · 14 functions
  • parquet.page Page decoding: headers, compression, levels, and the values inside. 2 structs · 3 functions
  • parquet.reader ParquetReader — open a file, pick columns and row groups, read batches. 3 structs · 4 functions
  • parquet.rle_encode Encoders for the two bit-level formats a Parquet writer needs. 1 struct · 3 functions
  • parquet.schema The Parquet schema tree, its leaf columns, and the Arrow schema it means. 4 structs · 2 functions
  • parquet.stats Column statistics, decoded from their raw bytes to typed values. 2 structs · 4 functions
  • parquet.writer ParquetWriter — write Arrow arrays back out as a Parquet file. 7 structs · 10 functions