Skip to main content
INCREMENTAL uses a stream cursor and saved checkpoint to read newer or changed source rows after each successful execution.

​
Requirements

  • a connector/stream that advertises Incremental support;
  • a stable, populated cursor such as updated_at, created_at, or an increasing ID;
  • published destination-owned SQL;
  • an existing destination table; and
  • a stable destination Upsert key.

​
Configure Incremental

  1. Open Source.
  2. Discover the catalog and enable the stream.
  3. Choose Incremental in the stream’s Sync mode column.
  4. Enter the cursor in the Cursor column.
  5. Preview and click Save stream settings.
  6. Configure and publish the destination’s transformation.
  7. Assign its output table and Upsert keys in the destination editor.
  8. Validate in Overview, click Run all, and inspect Runs.
  9. Add an interval or UTC cron schedule in Settings after the manual run succeeds.

​
How each run reads

For database sources, each run:
  • records the largest cursor value when it starts and reads up to that value. Rows written during the run are picked up by the next run.
  • re-reads rows equal to the saved cursor, so rows that share a timestamp are never skipped. Re-read rows are merged by the Upsert key.
  • skips rows whose cursor is empty (NULL).
The first run copies the table in primary-key order, in chunks of 500,000 rows. Larger tables continue in follow-up runs that start automatically, and a failed chunk resumes where it stopped.

​
Stream options

Open Options on an Incremental stream:
  • Check cursor reports the column type, empty values, index coverage, and out-of-order recent rows. These are warnings only.
  • Lookback re-reads rows this far behind the saved cursor, in seconds for timestamps or units for integers. Use it when transactions commit late or clocks differ.
  • Deletes: cursor sync does not see deleted rows. See Delete handling.
Click Apply, then Save stream settings.

​
Checkpoint state

After a successful run, Arcyria saves checkpoint state for each stream that was delivered. Failed runs, and streams that were not delivered in a partial run, keep their previous checkpoint. To reload a stream or re-read from a value, see Sync state. Use cursor columns that move forward predictably and are populated on every row. Index timestamp cursors for large database tables.