magmalake postgres

postgres · API

postgres.copyfmt

postgres.copyfmt — encode and parse rows for PostgreSQL's COPY text and CSV sub-formats.

COPY ... TO/FROM STDOUT/STDIN streams raw bytes over PQputCopyData/PQgetCopyData (wired up separately, in copy.mojo). This module is the pure-Mojo codec for what flows through that pipe: CopyEncoder builds row bytes a caller hands to PQputCopyData in chunks via take(); CopyDecoder (and the free decode_row/split_rows functions) turn bytes delivered by PQgetCopyData back into typed fields. Neither type touches libpq — see https://www.postgresql.org/docs/current/sql-copy.html ("Text Format" and "CSV Format") for the wire formats implemented here.

A COPY row is a list of fields, each either NULL or a text value; this module represents that as List[Optional[String]] — None is NULL, Optional[String]("") is a real empty string, which text format and (quoted) CSV can both express distinctly from NULL.

Structs

CopyEncoder

struct CopyEncoder

Builds COPY row bytes in format, ready for PQputCopyData in chunks via take().

Field values are escaped/quoted as they are appended, so the buffer is always valid COPY data up to the last completed row — size()/take() can be called mid-row, though a caller normally flushes only after end_row().

Implements Movable

Fields
format: Int
delimiter: String
null_string: String

__init__

def __init__(out self, format: Int = Int(0), delimiter: Optional[String] = None, null: Optional[String] = None)

Configure the encoder.

Args
format: COPY_TEXT or COPY_CSV.
delimiter: One-byte field separator. Defaults to tab (text) or comma (CSV).
null: The NULL marker. Defaults to \N (text) or the empty string (CSV) — PostgreSQL's own defaults for each sub-format.

Raises Error: If format is neither sub-format, or delimiter is not exactly one byte.

field

def field(mut self, v: StringSpan)

Append a non-NULL field, escaping/quoting it for self.format.

null

def null(mut self)

Append a NULL field (the raw, unescaped null marker).

end_row

def end_row(mut self)

Terminate the current row with \n and reset for the next one.

row

def row(mut self, values: List[Optional[String]])

Convenience: append a whole row (None entries as NULL) and terminate it.

size

def size(self) -> Int

Bytes currently buffered.

take

def take(mut self) -> List[UInt8]

Hand back the buffered bytes and reset the buffer, so a caller can flush in chunks without losing anything already appended.

bytes

def bytes(ref self) -> Span[UInt8, origin_of(self._buf)]

A read-only view of the currently buffered bytes, without taking them.

CopyDecoder

struct CopyDecoder

decode_row bound to a fixed format/delimiter/null configuration, for callers decoding many rows of the same COPY stream.

Implements Movable

Fields
format: Int
delimiter: String
null_string: String

__init__

def __init__(out self, format: Int = Int(0), delimiter: Optional[String] = None, null: Optional[String] = None)

decode_row

def decode_row(self, line: StringSpan) -> List[Optional[String]]

Functions

decode_row

def decode_row(line: StringSpan, format: Int, delimiter: String, null: String) -> List[Optional[String]]

Parse one line of COPY ... TO STDOUT output into its fields.

line must not include the trailing newline (split_rows strips it).

split_rows

def split_rows(buffer: Span[UInt8]) -> Tuple[List[String], String]

Split a buffer of concatenated COPY row bytes on \n, returning the complete lines (newline stripped) and any trailing partial line still waiting for more bytes.

This is a byte-oriented batching helper for the common case where PQgetCopyData results have been concatenated before parsing — it does not track CSV quote state, so a row containing a raw embedded newline (a CSV field with a real newline inside quotes) is indistinguishable here from a row boundary. PQgetCopyData itself always delivers one complete row per call regardless of embedded newlines; feed decode_row from those calls directly if embedded raw newlines are possible.

Aliases

comptime COPY_TEXT = 0

PostgreSQL's default COPY sub-format: tab-delimited, \N for NULL, backslash escapes.

comptime COPY_CSV = 1

COPY ... (FORMAT CSV): comma-delimited, double-quote quoting, empty unquoted string for NULL.