# avro.datafile > A module of avro's API. Avro Object Container Files — the reader and the writer. An OCF is `Obj\x01`, a `map` of file metadata (carrying at least `avro.schema`, and usually `avro.codec`), a 16-byte sync marker, and then a run of blocks, each `count · size · data · sync`. Apache Iceberg stores its own keys in that same metadata map — `schema`, `partition-spec`, `format-version`, `content` — so the metadata is exposed both as raw bytes (`metadata_keys` / `metadata_vals`) and as text (`metadata_string`), alongside the parsed `avro.schema`. The metadata is kept as two parallel lists rather than a `Dict`, in the order the file wrote them. A file has a handful of entries, so a linear scan over `StringSlice`s beats hashing — and, unlike a `Dict`, looking one up does not have to build a `String` first. `metadata_map()` still hands back a `Dict` for callers that want one. ## Structs ### `DataFileReader` ```mojo struct DataFileReader[C: CodecSet = DefaultCodecs] ``` Reads an Avro Object Container File held in memory. Implements: `Copyable`, `Movable` Fields: - `data: List[UInt8]` - `schema: Schema`: The parsed `avro.schema`. Empty until `ensure_schema()` when the reader was opened with `defer_schema=True`. - `schema_ready: Bool`: False only between a `defer_schema=True` open and `ensure_schema()`. - `metadata_keys: List[String]`: Metadata keys in the order the file wrote them. - `metadata_vals: List[List[UInt8]]`: Metadata values, parallel to `metadata_keys`. - `codec: String` - `sync_marker: List[UInt8]` - `pos: Int`: Offset of the next block header in `data`. - `block: List[UInt8]` - `block_pos: Int` - `block_left: Int` - `block_index: Int` #### `__init__` ```mojo def __init__(out self, var data: List[UInt8], *, defer_schema: Bool = False) ``` Read the header. `defer_schema` skips parsing the `avro.schema` JSON, which is the expensive half of opening a small file. It is for a caller that already holds a compiled `DecodePlan` for this exact schema — see `avro.cursor.PlanCache`. Nothing else changes: the schema is parsed on demand the moment anything needs it, so `next()` and `read_all()` behave the same either way. #### `open` ```mojo def open(path: StringSpan) -> Self ``` Read the whole file, then parse its header. #### `from_bytes` ```mojo def from_bytes(data: Span[UInt8]) -> Self ``` #### `ensure_schema` ```mojo def ensure_schema(mut self) ``` Parse `avro.schema` if a `defer_schema=True` open left it unparsed. #### `metadata_index` ```mojo def metadata_index(self, key: StringSpan) -> Int ``` Where `key` sits in `metadata_keys`/`metadata_vals`, or -1. `Span(reader.metadata_vals[i])` then reads the entry's raw bytes without copying them. A linear scan, and deliberately so: a file has a handful of metadata entries, and this compares `StringSlice`s rather than allocating a `String` to hash the way a `Dict` lookup would. #### `metadata_string` ```mojo def metadata_string(self, key: StringSpan) -> String ``` A metadata entry decoded as UTF-8; "" when the key is absent. #### `has_metadata` ```mojo def has_metadata(self, key: StringSpan) -> Bool ``` #### `metadata_map` ```mojo def metadata_map(self) -> Dict[String, List[UInt8]] ``` The metadata as a `Dict`, for callers that want that shape. #### `schema_json` ```mojo def schema_json(self) -> String ``` The schema exactly as the file spelled it. #### `has_next` ```mojo def has_next(mut self) -> Bool ``` #### `next` ```mojo def next(mut self) -> Value ``` Decode the next record; raises when the file is exhausted. #### `read_all` ```mojo def read_all(mut self) -> List[Value] ``` #### `count_blocks` ```mojo def count_blocks(self) -> Int ``` Walk every block header, without decompressing anything. Independent of how far iteration has got: it re-reads the header and then strides over the blocks. ### `DataFileWriter` ```mojo struct DataFileWriter[C: CodecSet = DefaultCodecs] ``` Builds an Avro Object Container File in memory. Implements: `Copyable`, `Movable` Fields: - `schema: Schema` - `codec: String` - `sync_marker: List[UInt8]` - `sync_interval: Int`: Approximate uncompressed bytes per block before an automatic flush. - `meta_keys: List[String]` - `meta_vals: List[List[UInt8]]` - `out: List[UInt8]` - `buf: Encoder` - `block_count: Int` - `started: Bool` - `closed: Bool` - `schema_json: String`: What to write as `avro.schema`; the printed schema unless overridden. #### `__init__` ```mojo def __init__(out self, var schema: Schema, codec: StringSpan = "null", sync_interval: Int = Int(64000)) ``` #### `set_sync_marker` ```mojo def set_sync_marker(mut self, marker: Span[UInt8]) ``` Pin the sync marker — the tests need reproducible files. #### `set_metadata` ```mojo def set_metadata(mut self, key: StringSpan, value: Span[UInt8]) ``` Add a file-level metadata entry. `avro.*` keys are reserved. #### `set_metadata_string` ```mojo def set_metadata_string(mut self, key: StringSpan, value: StringSpan) ``` #### `set_schema_json` ```mojo def set_schema_json(mut self, text: StringSpan) ``` Write `text` verbatim as `avro.schema`. `to_json()` prints an equivalent schema, not a byte-identical one. Apache Iceberg writers that want their manifests to match another implementation's bytes can hand over the exact JSON here; it is the caller's job to keep it equivalent to the schema being written. #### `append` ```mojo def append(mut self, value: Value) ``` #### `flush` ```mojo def flush(mut self) ``` Close off the current block, if it has anything in it. #### `close` ```mojo def close(mut self) ``` #### `bytes` ```mojo def bytes(mut self) -> List[UInt8] ``` Finish the file and hand back its bytes. #### `save` ```mojo def save(mut self, path: StringSpan) ``` ## Functions ### `read_file_bytes` ```mojo def read_file_bytes(path: StringSpan) -> List[UInt8] ``` ### `write_file_bytes` ```mojo def write_file_bytes(path: StringSpan, data: Span[UInt8]) ``` ### `random_sync_marker` ```mojo def random_sync_marker() -> List[UInt8] ``` ## Aliases ### `MAGIC0` ```mojo comptime MAGIC0 = UInt8(79) ``` ### `MAGIC1` ```mojo comptime MAGIC1 = UInt8(98) ``` ### `MAGIC2` ```mojo comptime MAGIC2 = UInt8(106) ``` ### `MAGIC3` ```mojo comptime MAGIC3 = UInt8(1) ``` ### `SYNC_SIZE` ```mojo comptime SYNC_SIZE = Int(16) ``` ### `DEFAULT_SYNC_INTERVAL` ```mojo comptime DEFAULT_SYNC_INTERVAL = Int(64000) ```