`orca.logs`

Where attempts' logs, drivers' logs and runs' timelines go.

With ORCA_LOGS set, each attempt logs to <ORCA_LOGS>/<pipeline>/<run>/<step>/<partition>/<attempt>.log (the partition is _run for a gather step, and a keyed one's / is written %2F, orca.name.path_segment), a run of a pipeline with dista tasks leaves its timeline at <ORCA_LOGS>/<pipeline>/<run>/timeline.json, and the scheduler keeps a run's output at .../<run>/driver.log. Without it, logs go to stdout only.

ORCA_LOGS is a directory, or a location in object storage such as s3://bucket/orca/logs (objectstore's FileIOResolver, configured from the AWS_* environment). An object can't be appended to, so for a remote root a log is written to a staging file under ORCA_LOG_STAGE (default $TMPDIR/orca-logs) and published when it's complete: an attempt's when the attempt ends, a timeline when the run does. attempts.log records where the log ends up, which read_location reads either way.

Secrets are masked in every log written here: ORCA_SECRETS names the environment variables that hold a run's secrets (orca register --secret), and mask replaces their values with ***.

Functions

`mask`

def mask(text: String) -> String

text with every secret's value replaced by ***.

`mask_bytes`

def mask_bytes(data: Span[UInt8]) -> List[UInt8]

mask, on bytes that may not be UTF-8.

`logs_dir`

def logs_dir() -> String

`is_remote`

def is_remote(location: String) -> Bool

Whether location is in object storage, not on this machine.

`run_dir`

def run_dir(pipeline: String, run: Int) -> String

The run's directory under ORCA_LOGS, or "" without it.

`stage_of`

def stage_of(location: String) -> String

Where a log bound for location is written first: location itself on this machine, else its path under ORCA_LOG_STAGE.

`attempt_log`

def attempt_log(pipeline: String, run: Int, step: String, partition: String, attempt: Int) -> String

Where the attempt's log ends up, stage_of it ready to be written; "" without ORCA_LOGS.

`publish`

def publish(location: String)

Copies a remote location's staging file to it; nothing for a local one, which was written in place.

`write_location`

def write_location(location: String, data: Span[UInt8])

Writes data to location, local or remote, replacing it, with secrets masked.

`read_location`

def read_location(location: String) -> List[UInt8]

What's at location, local or remote; raises if there's nothing.

`location_exists`

def location_exists(location: String) -> Bool

`utc_now`

def utc_now() -> String

The time now, to the second, as YYYY-MM-DDTHH:MM:SSZ.