File I/O

File I/O connectors read data from local files into the pipeline and write results back to disk.

Operations

Operation	Direction	Description
file.read	Source	Reads a file and produces a Table or Record.
file.write	Sink	Writes a Table or Record to a file.

The format is determined by the format field or inferred from the file extension.

Supported formats

Format	Extensions	Notes
CSV	.csv, .tsv	Configurable delimiter, header row, encoding.
JSON	.json	Array of objects or single object.
NDJSON	.ndjson, .jsonl	One JSON object per line. Native interchange format.
Parquet	.parquet	Columnar binary format. Read via DuckDB.

Reading files

# CSV with options
read-leads:
  type: source
  op: file.read
  params:
    path: data/leads.csv
    format: csv
    options:
      delimiter: ","
      header: true
      encoding: utf-8
  outputs:
    rows:
      type: Table
      schema:
        name: { type: string }
        email: { type: string }
        score: { type: number }

# JSON → Record
read-config:
  type: source
  op: file.read
  params:
    path: config/settings.json
    format: json
  outputs:
    settings: { type: Record }

# NDJSON → Table
read-events:
  type: source
  op: file.read
  params:
    path: logs/events.ndjson
  outputs:
    events: { type: Table }

# Parquet → Table
read-warehouse:
  type: source
  op: file.read
  params:
    path: warehouse/orders.parquet
  outputs:
    orders: { type: Table }

Writing files

write-results:
  type: deterministic
  op: file.write
  params:
    path: output/qualified.csv
    format: csv
    options: { delimiter: ",", header: true }
  inputs:
    data: { type: Table, from: ref(filter-leads.qualified) }

Specify format: json or format: ndjson for other output formats. The same options fields apply.

Config reference

Field	Required	Default	Description
path	Yes	—	File path, relative to project root.
format	No	Inferred from extension	One of csv, json, ndjson, parquet.
options.delimiter	No	,	Column separator for CSV.
options.header	No	true	First row contains column names (CSV).
options.encoding	No	utf-8	File encoding (CSV).

Paths are resolved relative to the project root. Absolute paths are rejected.