# iceberg.io

> A module of iceberg's API.

File access: locations in, bytes out — over local files, S3, GCS, Azure or plain HTTP.

Iceberg metadata stores **absolute** locations — a manifest list names its
manifests by full URI, a manifest names its data files by full URI. That is
fine in place and awkward everywhere else: a table copied to another path, a
warehouse mounted somewhere new, or a test fixture checked into a repo all need
those locations redirected without touching the metadata. `FileIO` therefore
resolves every location through an ordered list of prefix rewrites before
opening it. `FileIO.local()` does none; `rebase(old, new)` adds one.

The backends come from [objectstore.mojo](https://github.com/magmalake/objectstore.mojo):
`FileIOResolver` picks one by URI scheme, configures it from Iceberg property
names (`s3.endpoint`, `s3.access-key-id`, `gcs.oauth2.token`,
`adls.sas-token.<account>`, …) and from the `storage-credentials` a REST
catalog vends, and falls back to the `AWS_*` environment. Everything above this
line only ever sees a location and a byte range.

`InputFile` is objectstore's trait, a superset of the one this module used to
declare: `location`, `exists`, `read_all` as before, plus `length` and
`read_range`, which is what a Parquet reader needs in order to fetch a footer
without downloading the whole file.

## Structs

### `FileIO`

```mojo
struct FileIO
```

Resolves Iceberg locations to bytes.

Rewrites are applied in the order they were added; the first prefix that
matches wins. Properties and vended credentials configure the object-store
backends, and are ignored for `file://`.

Implements: `Copyable`, `Movable`

Fields:

- `resolver: FileIOResolver`
- `base_location: String`: The table's own location, used to resolve *relative* locations.

#### `__init__`

```mojo
def __init__(out self)
```

```mojo
def __init__(out self, var resolver: FileIOResolver)
```

#### `local`

```mojo
def local() -> Self
```

#### `with_properties`

```mojo
def with_properties(var properties: Dict[String, String]) -> Self
```

#### `rebase`

```mojo
def rebase(mut self, var old_prefix: String, var new_prefix: String)
```

Redirect every location under `old_prefix` to `new_prefix`.

#### `with_base`

```mojo
def with_base(mut self, var location: String)
```

Set the table root that relative locations resolve against.

#### `set`

```mojo
def set(mut self, var key: String, var value: String)
```

Set one storage property, using Iceberg's own property names.

#### `add_storage_credential`

```mojo
def add_storage_credential(mut self, var prefix: String, var config: Dict[String, String])
```

Absorb one `storage-credentials` entry from a `loadTable` response.

#### `properties_for`

```mojo
def properties_for(self, location: String) -> Dict[String, String]
```

#### `resolve`

```mojo
def resolve(self, location: String) -> String
```

The location this one should actually be read from.

Local locations come back as bare filesystem paths, which is what the
callers that hand a path to `open()` or `listdir()` expect; every other
scheme comes back as a URI.

#### `absolute`

```mojo
def absolute(self, location: String) -> String
```

A relative location joined to the table root; anything else as is.

#### `local_path`

```mojo
def local_path(self, location: String) -> String
```

The filesystem path for a location that must be local.

#### `is_local`

```mojo
def is_local(self, location: String) -> Bool
```

#### `new_input`

```mojo
def new_input(self, location: String) -> AnyInputFile
```

#### `read_all`

```mojo
def read_all(self, location: String) -> List[UInt8]
```

#### `read_range`

```mojo
def read_range(self, location: String, offset: Int, length: Int) -> List[UInt8]
```

#### `read_ranges`

```mojo
def read_ranges(self, location: String, ranges: List[ByteRange]) -> List[List[UInt8]]
```

Several ranges of one file, in order. On object storage nearby ranges are coalesced and the rest fetched at once.

#### `read_ranges_many`

```mojo
def read_ranges_many(self, locations: List[String], ranges: List[List[ByteRange]]) -> List[List[List[UInt8]]]
```

`ranges[i]` of `locations[i]` for every `i`; on S3 every file's ranges are fetched in one batch.

#### `read_all_many`

```mojo
def read_all_many(self, locations: List[String]) -> List[List[UInt8]]
```

Several whole files, in order; on S3 fetched at once.

#### `length`

```mojo
def length(self, location: String) -> Int
```

#### `read_text`

```mojo
def read_text(self, location: String) -> String
```

#### `exists`

```mojo
def exists(self, location: String) -> Bool
```

#### `new_output`

```mojo
def new_output(self, location: String) -> AnyOutputFile
```

#### `write_all`

```mojo
def write_all(self, location: String, data: Span[UInt8])
```

Overwrite `location` with `data`, creating parents as needed.

#### `write_new`

```mojo
def write_new(self, location: String, data: Span[UInt8])
```

Write `location`, failing if something is already there.

This is the primitive an optimistic commit needs: two writers racing on
the same metadata version must not both believe they won. Only a local
filesystem really provides it — S3 has no atomic create, which the spec
itself calls out — so `FilesystemCatalog` refuses to commit over an
object store without being told to.

#### `write_text`

```mojo
def write_text(self, location: String, text: String)
```

#### `delete`

```mojo
def delete(self, location: String)
```

Remove one object. A missing one is not an error.

The location is resolved to a bare filesystem path first when it is
local: `list()` hands back `file://` plus whatever path it was given,
and `file://build/x` is not a well-formed URI — a URI parser reads
`build` as the host — so a *relative* local location has to lose its
scheme before an object store's URI parsing gets hold of it.

#### `list`

```mojo
def list(self, prefix: String) -> List[String]
```

Every location under `prefix`, as full URIs.

#### `list_names`

```mojo
def list_names(self, directory: String) -> List[String]
```

The immediate children of a directory, by name.

Object stores have no directories, so the names are derived from the
keys: everything under the prefix, cut at the first `/`, deduplicated.
A local directory is listed directly.

## Functions

### `strip_scheme`

```mojo
def strip_scheme(location: String) -> String
```

`file:///a/b` -> `/a/b`. Other schemes are returned unchanged.

### `is_local_location`

```mojo
def is_local_location(location: String) -> Bool
```

True when this location names an ordinary filesystem path.

### `basename`

```mojo
def basename(path: String) -> String
```

### `dirname`

```mojo
def dirname(path: String) -> String
```

### `join_path`

```mojo
def join_path(a: String, b: String) -> String
```

## Aliases

### `FILE_SCHEME`

```mojo
comptime FILE_SCHEME = String("file://")
```