CLI Reference
load
Load the selected tables of a project, reload them from scratch, follow a Postgres CDC source, and print the result as JSON.
sqlload load [SELECTOR ...] [OPTIONS]sqlload load loads the selected tables, creating any that don’t exist yet. It prints the
plan’s closing line and any errors and warnings, then a line per table as it
finishes. It takes the same options as plan.
Loading
Section titled “Loading”$ sqlload load✓ Plan complete 2 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0sloaded raw.app__orders 1,000 rows 0.0s
✓ Completed 2 loaded, 1,050 rows, 0.1s├── slowest raw.app__orders 0.0s└── peak memory 85.4 MiBEach table prints created, altered, loaded or failed; a loaded line ends with the
table’s new cursor position when it keeps one. A failed table doesn’t stop the others. The summary
counts them, names the slowest table, gives the run’s peak memory, and
after failures says what a rerun does. The command exits
1, or 75 when every failure may pass on a rerun (see
Retrying failed runs). A table the plan skips, such as a files
table with no new file, isn’t read.
On a terminal, a live block under these lines shows how many tables are done, each running table
with its rows so far, the tables waiting, and any rate_limit in use. It’s redrawn in place and
cleared at the end. See Events for --progress.
Stopping at the first failure
Section titled “Stopping at the first failure”--fail-fast stops the load once a table fails: no other table starts, and tables
already loading finish and keep what they committed. The summary names the tables that never
started under not started, and the command exits as it would have otherwise: 1, or 75 when every
table that failed, including any still loading when the first failed, may pass on a rerun. A rerun
loads the tables that didn’t start as usual.
$ sqlload load --fail-fast --concurrency 2…loaded raw.shop__customers 400,000 rows 0.8sloaded raw.shop__refunds 200,000 rows 0.4sfailed raw.shop__returns checks.min_rows failed: 50,000 rows staged, fewer than the 100,000 required; nothing was publishedloaded raw.shop__orders 3,000,000 rows 2.4s
✗ Completed with failures 3 loaded, 1 failed, 1 not started, 3,600,000 rows, 2.5s├── not started raw.shop__shipments (--fail-fast)├── slowest raw.shop__orders 2.4s├── peak memory 314 MiB└── next the 2 unchanged tables load; the 3 that committed carry on from their positions--fail-fast and --memory-per-table don’t combine with --follow.
Large loads commit along the way, so a failure keeps what committed and the next run carries on from
there; a delete_insert, append or merge into ClickHouse read in windows or cursor order
publishes once per run instead (a merge also every 10 million rows). See
Batches. To read a range instead of continuing from the
destination’s maximum, give --start-cursor and --end-cursor, or their typed forms
--start-cursor-ts/--end-cursor-ts and --start-cursor-int/--end-cursor-int; see Backfilling a
range.
Reloading
Section titled “Reloading”--reload rebuilds tables from scratch. On a terminal it shows the plan and asks; in scripts, add
--yes:
$ sqlload load --reload --yes app__customers✓ Plan complete 1 object to create, 1 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0s
✓ Completed 1 loaded, 50 rows, 0.1s├── slowest raw.app__customers 0.0s└── peak memory 75.1 MiB--reload takes --start-cursor and --end-cursor, and rebuilds the table with only
the rows in that range, as SQLBuild’s
--full-refresh does with an interval. Rows
outside the range are dropped, and plan warns which:
$ sqlload plan -v --reload --start-cursor 100000 --end-cursor 200000 shop__orders… ├── reads order_id from 100000 (--start-cursor), below 200000 (--end-cursor)… └── commits 2 windows of 50000, read one after another into a new table, swapped in once at the end
Warnings (1)└── shop__orders └── --reload rebuilds the table with only the rows from 100000 (--start-cursor), below 200000 (--end-cursor): rows before 100000 and from 200000 on are droppedWithout --start-cursor, a reload starts from cursor_start, or from the first row. An integer
batch_size on a source that can’t report its maximum needs --end-cursor, with or without
--reload; see Batches.
A postgres_cdc table is copied again with --resnapshot instead; --reload refuses it. See
Reloading. --reload is also refused on
Postgres targets.
On a files source, --reload reads every file again; a merge table merges them one
at a time into the new table, and after_load = { move_to = ... } reads the moved files too. See
Reloading a files table.
Peak memory
Section titled “Peak memory”Every load reports the most memory the process held at once: under peak memory in
the summary, as peak_memory_bytes in the JSON report and in the
run_finished event. It’s the peak for the whole run, all tables together: ODBC
drivers, DuckDB and Python sources count too. Under the Python package,
it includes the Python interpreter. It’s the peak resident set (ru_maxrss) on Linux and macOS,
and the peak working set on Windows.
On Linux, --memory-per-table also measures each table. It loads one table at a time, whatever
--concurrency or concurrency say, resets the process’s high-water mark before each table and
reads it after:
$ sqlload load --memory-per-table✓ Plan complete 2 to load (sqlload plan shows the tree)
loaded raw.app__customers 50 rows 0.0s peak memory 80.7 MiBloaded raw.app__orders 1 row 0.0s peak memory 84.7 MiB
✓ Completed 2 loaded, 51 rows, 0.1s├── slowest raw.app__orders 0.0s└── peak memory 84.7 MiBThe JSON report and the events give the figures in bytes:
$ sqlload load --json | jq -c .summary{"created":0,"altered":0,"recreated":0,"loaded":2,"failed":0,"skipped":0,"rows":51,"peak_memory_bytes":90210304}$ sqlload load --memory-per-table --events jsonl 2>&1 >/dev/null | grep -E 'table_done|run_finished'{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.244Z","table":"raw.app__customers","event":"table_done","rows":50,"elapsed_ms":22,"peak_memory_bytes":84353024}{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.266Z","table":"raw.app__orders","event":"table_done","rows":1,"elapsed_ms":22,"peak_memory_bytes":89116672}{"schema_version":0,"run_id":"2e96ce9d-…","at":"2026-11-15T02:06:38.267Z","event":"run_finished","loaded":2,"failed":0,"not_started":0,"rows":51,"elapsed_ms":113,"exit_code":0,"peak_memory_bytes":89116672}A table’s figure is the most the process held while it loaded, so it includes what the process
already held when the table started, such as memory an earlier table left allocated. Compare the
tables of one run: the table that uses too much stands out. On other systems, plan, apply and
load refuse the flag: --memory-per-table needs Linux; the run's peak is in every report. See
Finding a source that uses too much memory.
Memory budget
Section titled “Memory budget”A run stays under one memory budget, and plan -v shows it first, with where it came
from and how it’s shared:
$ sqlload plan -v --memory-limit 1200MBMemory 1.1 GiB budget (--memory-limit); 39.8 MiB held before loading; DuckDB 488 MiB (memory_limit 244 MiB, 2 threads); 154 MiB per table, 4 at once
Objects (2) sqlload apply├── raw.customers create table, 4 columns└── raw.documents create table, 6 columns
Loads (2) into DuckDB dev sqlload load└── app postgres into raw ├── customers replace └── documents replace
✓ Plan complete 2 objects to create, 2 to load- The budget is
memory_limitor--memory-limit. Without either, it’s 80% of the limit of the container (the cgroup’smemory.max) or the Windows job object SQLLoad runs in, else 50% of the machine’s memory, which other programs likely share. Where the memory can’t be read, the line saysno budgetand why, and the run has no budget. - Held before loading is what the process holds once it has connected, which comes off first.
- DuckDB (and MotherDuck) targets take 30% of the rest together. Each one’s
memory_limitis half its part, because DuckDB holds up to about twice itsmemory_limit, and never under 256MB. It runs a statement on one thread per 128 MiB of itsmemory_limit, between two and the machine’s cores. - Per table: the tables share what remains, as many at once as get at least 64 MiB each, up to
concurrency. When fewer fit, the line says so:1 at once (concurrency 8 doesn't fit). A table’s sources keep a quarter of its share each for what they read ahead: an ODBC fetch buffer, the pages ansl.httpjob requests ahead (read_aheadsends fewer at once when its pages are large), and a Postgres source’s batches.
memory_limit = "off" (or --memory-limit off) turns the budget off. The run then
loads with no watchdog, no pieces (each commit is whole), DuckDB with its own
memory_limit (80% of the machine’s memory) and threads and its own checkpoints, and an ODBC
fetch buffer of 512 MiB per table. Only the
row-group check stays, against DuckDB’s own memory_limit, since a
checkpoint that can’t rewrite a row group fails whatever the budget. plan always shows the line,
and load still reports its peak memory:
Memory off (memory_limit): no budget or watchdog, and DuckDB keeps its own memory_limit and threadsplan fails when the budget can’t hold what SQLLoad holds already, or leaves no table 64 MiB,
naming the budget the run needs, and load stops there, before it creates or loads anything:
the 600 MiB memory budget leaves the tables 11.7 MiB after DuckDB's 488 MiB, under the 64 MiB one table loads in; raise memory_limit to at least 653MiBLarge rows in DuckDB
Section titled “Large rows in DuckDB”DuckDB keeps values of about 85 KiB and more in memory from their insert until a checkpoint
writes them to the file. So when the large values of a table’s open commit reach a quarter of
DuckDB’s memory_limit, SQLLoad commits what it has written as a piece and carries on. Rows that
go through a staging table first, as a merge or delete_insert does, count three times. Under
send = "at_commit", the default on MotherDuck, the rows held for a commit are sent a piece at a
time as well. How the pieces commit depends on the table:
- A table the load builds and swaps in at the end (a first load,
replace,--reload) commits its pieces into the build table, so nothing changes for readers. appendandmergecommit each piece into the table. Adelete_insertcuts its pieces only between values of itsreplace_key, or of its cursor for a window, so each piece replaces whole keys or a whole part of the window; its deletes keep the rows this load committed in earlier pieces. Until the last piece, a reader may see part of the load, as with commits everybatch_size.- An
appendthat stops between its pieces, even by a crash, leaves a marker; the nextloadof that table removes that load’s rows before it loads, so it reads them again. See Unfinished appends.
SQLLoad turns DuckDB’s automatic checkpoint off during a load, since one that runs out of memory leaves the database unusable. Instead it checkpoints after a piece as needed, and after each table that held large values.
Some rows can’t be cut small enough. Then only their table fails, naming the fix, while the other tables load:
-
one
replace_keyvalue, cursor value or partition whose rows are larger than a piece:failed raw.attachments attachments: the rows of one k value hold 300 MiB of values over 85 KiB, which a delete_insert replaces in one commit, past the 61 MiB of them DuckDB's 244 MiB memory_limit commits at once (through a staging table, they count three times); raise memory_limit to at least 23.5GiB -
a table whose open row group, the rows past its last full 122,880, holds more large values than DuckDB can rewrite at a checkpoint. The message also gives what a full row group of such rows needs. This doesn’t apply to MotherDuck, which checkpoints on its own servers:
failed raw.documents documents: its last row group would hold 366 MiB of strings, 366 MiB of them values over 85 KiB from this load (rows of about 1 MiB each), more than DuckDB rewrites at a checkpoint within its 244 MiB memory_limit; raise memory_limit to at least 2GiB (a row group fills at 122,880 rows, and rows this wide need a memory_limit of at least 640.1GiB by then)
Unnested child tables, Postgres CDC changes, the key scan of --sync-deletes, a merge that
refuses repeated keys (on_duplicate_keys = "fail") and a table with
checks that isn’t rebuilt aren’t cut into pieces: a refusing merge and
checks must see every row of a commit before any of it is written. They fail the same way when one
commit is too large, writing nothing of it. A CDC table then stops at that commit
until memory_limit is raised, and its replication slot keeps the source’s write-ahead log
meanwhile.
The watchdog
Section titled “The watchdog”While a run loads, SQLLoad reads the memory of the whole process, and of a MotherDuck target’s bridge process, every 10 ms. From 80% of the budget, no table starts until it falls again, and pieces are cut four times smaller. From 90%, the table holding the most large values in memory stops: its running DuckDB statement is interrupted and rolled back, and it fails while the other tables carry on. A table holding none isn’t stopped.
failed raw.attachments raw.attachments stopped: the load held 1 GiB of its 1.1 GiB memory budget (90%); from 90% the table holding the most large values in memory stops, here this one with 39 MiB; raise memory_limit, or lower batch_sizeFrom 95%, when no table holds large values to stop (memory taken by a Python function building a
list, or an ODBC driver’s own buffers), or the table stopped at 90% didn’t bring memory back under
90% within half a second, the whole run stops. Every running DuckDB and MotherDuck statement is
interrupted and rolled back, the running tables fail with the reason below, and the others don’t
start. Tables that committed before stay committed, and an append stopped between its pieces is
cleaned up by the next load. SQLLoad then writes its summary, the --json report and the
run_finished event, and exits with 1 (retryable: false, since the same run would likely run
out again) without waiting for a table busy in its source, such as a Python function that hasn’t
yielded. A MotherDuck bridge ends with the process.
failed raw.events stopped: the run held 3.9 GiB of its 4 GiB memory budget (97%), and from 95% it stops as no table held large values to stop; raise memory_limit, or lower concurrencyThe watchdog reads memory every 10 ms, so an allocation that grows past the limit the OS or a service manager sets within one reading is only caught by that limit.
Following a Postgres CDC source
Section titled “Following a Postgres CDC source”sqlload load SOURCE --follow keeps streaming one postgres_cdc source after it catches up,
writing a batch every batch_interval, until --max-duration passes or it gets Ctrl-C or
SIGTERM. A second Ctrl-C or SIGTERM stops it at once, with exit code 130. See
Keep streaming.
sqlload load app --follow --max-duration 10mIt prints the same plan line, table lines and summary as load, with a
written line for each table in each batch, giving its rows and how far it is behind the server.
On a terminal, a live block shows the rows and batches written so far.
Results as JSON
Section titled “Results as JSON”apply --json and load --json print one JSON document on
standard output when they finish, for scripts and orchestrators. Logs stay on standard error. The
exit code is the same as without --json.
$ sqlload load --json{ "schema_version": 1, "command": "load", "ok": false, "retryable": true, "started_at": "2026-11-15T02:24:02.691Z", "finished_at": "2026-11-15T02:24:04.729Z", "duration_seconds": 2.039, "summary": { "created": 1, "altered": 0, "recreated": 0, "loaded": 1, "failed": 1, "skipped": 0, "rows": 1200, "peak_memory_bytes": 101089280 }, "plan": { "ok": true, "summary": "2 selected, 2 objects to create, 2 to load", … }, "objects": [ { "action": "created", "name": "raw.shop__orders", "detail": "4 columns" } ], "tables": [ { "table": "raw.shop__orders", "source": "shop", "target": "dev", "write_strategy": "replace", "status": "loaded", "rows": 1200, "duration_seconds": 0.043, "note": null, "error": null, "retryable": false }, { "table": "raw.shop__refunds", "source": "shop", "target": "dev", "write_strategy": "replace", "status": "failed", "rows": null, "duration_seconds": 1.999, "note": null, "error": "GET http://api.example.com/v1/refunds?page=3: HTTP 503: unavailable (after 4 attempts)", "retryable": true } ]}| Field | Meaning |
|---|---|
schema_version |
1. New fields can appear without a change; it changes only when a field changes meaning or goes away. |
command |
apply or load. |
started_at, finished_at, duration_seconds |
When the command started and finished, as RFC 3339 UTC timestamps, and the seconds between. |
ok |
Whether every selected table planned and loaded: true exactly when the exit code is 0. |
retryable |
Whether running the command again unchanged may fix every failure: true exactly when the exit code is 75. It’s false when ok is true, whenever a table failed for good, and whenever a plan error other than an unreachable source left one out. See Retrying failed runs. |
summary |
How many objects were created, altered (each table once, however many changes it had) and recreated; tables loaded and failed, and skipped; rows loaded. peak_memory_bytes: the most memory the whole process held at once during the run, in bytes; see Peak memory. |
plan |
The same document plan --json prints, with its own ok and retryable. Tables a plan error left out are in its errors, and tables with nothing to load in skipped. Its phases give each connection’s and discovery’s time: see Connecting and discovering. Its memory is the plan’s memory line, such as "4 GiB budget (memory_limit); …", or "off (memory_limit): …" with the budget off; see Memory budget. |
objects |
One entry per change to a table or stage, by apply or during a load: its action (created, altered or recreated), its name and the detail. A table that gains two columns has two altered entries. |
tables |
Each table loaded: its status (loaded or failed, or skipped for a table --fail-fast didn’t start, its error naming the table that stopped the run and retryable true; also for a table the watchdog’s run stop didn’t start, with retryable false), rows, duration_seconds, a note (the cursor window it read, or that its table waits for first data), and the error when it failed. retryable says whether that failure may pass on a rerun. source, target and write_strategy are null only for a load --follow that failed before any table. A files source with after_load also has after_load: its action (delete or move), status (done, or failed when any file stayed), files handled, and failed, each file left with its error. A table that keeps its files has no such field. A delete_insert table with a replace_key also has replaced_keys: its columns, the values its commits replaced (sorted, each as its columns’ text joined by , , NULL for a NULL, at most 1,000) and truncated, true when there were more. Under --memory-per-table each table has peak_memory_bytes, the most memory the process held while it loaded. A table with a transform also has transforms: for each file or range, its file, the milliseconds the function took, the rows it was given and dicts, whether it returned dicts. A table with a column rename named or numbering numbered also has names: a line for each, as plan shows them, such as "Qty (column 4) -> qty_1 (numbered)". A table whose load warned, such as of rows outside a cursor window, has warnings, one line each. apply loads nothing, so its list is empty. |
When the run can’t start (the configuration doesn’t parse, a connection fails), standard output
holds one error document instead, the same as plan --json’s:
{"error": {"code": "config", "message": "…", "help": "…", "retryable": false}}. code is
config for the configuration, else the command’s name.
retryable is true, and the exit code 75, when the run couldn’t start for a reason a rerun may
fix, such as a target that refused the connection or timed out.
--reload and --resnapshot need --yes with --json, as there is no prompt to answer.
Under --json, standard output holds only the document. Everything a project with
Python sources writes to standard output while it runs goes to standard error instead: a source’s
print(), a child process that inherits standard output, and os.write(1, …).
Events
Section titled “Events”--events jsonl writes each load event to standard error as one line of JSON, next to
the logs, while it happens. --events-file PATH writes them to a file instead. Neither combines with
--follow. Prefect and Dagster shows how to send them to an
orchestrator’s log.
$ sqlload load --events jsonl 2>&1 >/dev/null{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.303Z","event":"run_started","selected":2,"to_load":2}{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.307Z","table":"raw.items","event":"table_started","strategy":"replace"}…{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","table":"raw.items","event":"table_failed","reason":"checks.min_rows failed: 1 row staged, fewer than the 1,000 required; nothing was published","retryable":false,"elapsed_ms":18}{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","table":"raw.orders","event":"table_done","rows":3,"elapsed_ms":18}{"schema_version":0,"run_id":"6c3e27d9-…","at":"2026-11-15T02:24:32.326Z","event":"run_finished","loaded":1,"failed":1,"not_started":0,"rows":3,"elapsed_ms":71,"exit_code":1,"peak_memory_bytes":86921216}The events are version 0: names and fields can still change in a minor release until
schema_version is 1. Every event has schema_version, run_id, at (RFC 3339, UTC) and
event, and a table’s events have table, named as plan names it.
event |
Fields |
|---|---|
connected |
Before run_started: a target’s or source’s connection opened, or failed. role (target or source), name, label (what it reached, such as OPENEDGE through ODBC), ok and elapsed_ms. No table. |
discovered |
Before run_started: a source’s tables were found, or that failed. source, tables (how many), ok and elapsed_ms, timed from its connection when it has one. No table. |
run_started |
selected, to_load: the tables selected and the tables the load starts. |
table_started |
strategy: the write strategy, as plan shows it. |
table_progress |
unit (page, batch, window or file), count of them read so far, rows so far. |
rate_limited |
connection, its rate_limit as rate, and wait_ms, how long a request waited. No table. |
table_committing |
rows: the table’s rows are read and its last commit starts. |
table_done |
rows, elapsed_ms, and position, the cursor value it reached, when it keeps one. changes lists the created and altered lines of a table that made or widened its columns while loading. transforms lists each file or range a transform ran on, as in the JSON report, and under --memory-per-table peak_memory_bytes gives the table’s peak memory. names gives a line per column rename named or numbering numbered, as in the JSON report, and warnings what the load warns of, such as rows outside a cursor window. |
table_failed |
reason, retryable (whether a rerun may pass), elapsed_ms, and peak_memory_bytes under --memory-per-table. |
table_not_started |
reason, such as the table that stopped a --fail-fast load. |
run_finished |
loaded, failed, not_started, rows, elapsed_ms and exit_code; peak_memory_bytes, the run’s peak memory. |
On a terminal, load also draws a live block of running tables on standard error. The default,
--progress auto, leaves it out under --json or --log-format json; --progress always draws it
on a terminal anyway, and --progress never leaves it out. Output that isn’t a terminal never has
the block or colours.