Skip to main content
MongoDB is available as an Available Now source connector. Arcyria reads the selected collections, stages them in an isolated per-run workspace, runs your published SQL models, and delivers the result to a supported destination. The connector accepts both mongodb:// and mongodb+srv:// connection URIs. Standalone MongoDB has been verified end to end. Atlas, replica-set, sharded, and live TLS deployments should be validated in your environment before production use.

​
Required access

Use a dedicated read-only user scoped to the databases that Arcyria needs. For a self-managed deployment, create the user in the authentication database:
For Atlas, create a database user with read access to the selected database and allow network access from the Arcyria runtime. Do not place a real URI in source control, support tickets, screenshots, or pipeline SQL.

​
Connection fields

The complete URI is encrypted as one secret. It is masked after saving and is not returned to the browser. On edit, leave the masked URI unchanged unless you intend to rotate it. Common URI options include authSource, replicaSet, readPreference, tls=true, and provider certificate options. Arcyria passes supported URI options to PyMongo rather than splitting them into separate fields.

​
Create the connection

  1. Open Connections.
  2. Click + New Connection.
  3. Set the role to Source.
  4. Choose MongoDB.
  5. Enter the complete URI and database.
  6. Click Test Connection.
  7. Save only after the test succeeds.
If the deployment uses an IP allowlist or private network, follow Private database access first.

​
Discover and preview

After creating a pipeline:
  1. Open the pipeline Source tab.
  2. Click Discover catalog.
  3. Select the required database.collection streams.
  4. Choose Full Table or Incremental for each collection.
  5. For Incremental, select a stable cursor such as updated_at or _id.
  6. Preview representative documents and confirm the inferred field types.
  7. Click Save stream settings.
Discovery is dynamic and returns collections visible to the configured user. It supports empty collections and identifies normal collections, views, and time-series collections. Schema inference reads a bounded sample; it does not scan or materialize the entire collection. MongoDB documents do not require a fixed schema. Discovery reports observed types, mixed fields, nullability, missing fields, arrays, nested objects, indexes, and _id primary-key semantics.

​
Full Table mode

Use Full Table for an initial backfill, small reference collection, or deliberate snapshot. Documents are streamed from MongoDB in bounded batches rather than loaded into memory as one collection.

​
Incremental mode

Use Incremental when the collection has a stable cursor:
  • updated_at for inserts and updates;
  • created_at for append-only collections; or
  • _id when ObjectId creation order matches the required extraction order.
How a run reads a collection:
  • The first run loads the whole collection in _id order, in chunks (500,000 documents by default). Large collections continue in follow-up runs, and the load resumes where it stopped after a failure. Documents without a cursor value are included in this first load.
  • After that, each run first notes the highest cursor value in the collection, then reads documents from the last saved value up to that one. Documents written during the run are read by the next run.
  • Documents that share the saved value are read again and matched by the destination’s Upsert key, so ties are never skipped. A run with no new documents reads nothing.
  • An optional lookback re-reads a window before the saved value, for writes that arrive late. It works for date, number, ObjectId, and timestamp cursors.
A cursor must be a date, number, ObjectId, or BSON timestamp field (a string also works but compares as text). MongoDB only compares values of the same type, so the first run fixes the cursor’s type. Documents whose cursor is missing, null, or of another type are not read by later runs; Validate cursor in the stream’s options counts them for you. A cursor that is set only when a document is created, such as created_at or _id, does not see updates. Use a field your application sets on every update. Arcyria advances checkpoint state only after a successful delivery. A failed run does not replace the previous successful checkpoint. Checkpoints from earlier versions continue from their saved value without reloading the collection.

​
Deleted documents

Incremental runs do not see deleted documents. Turn on delete detection in the stream’s options to compare _id values on a schedule. It needs the destination key to be _id.

​
BSON handling

The source handles common BSON values, including:
  • ObjectId and Decimal128;
  • dates and BSON timestamps;
  • binary values;
  • arrays and nested documents;
  • regular expressions and DBRef values; and
  • null, missing, and mixed-schema fields.
ObjectIds are represented as stable strings during SQL staging. Decimal128 is kept as an exact decimal value rather than converted to a floating-point value. Special BSON values use loss-aware Extended JSON-compatible structures.

​
Current limitations

  • Change Streams and log-based CDC are not enabled.
  • Arbitrary multi-field incremental checkpoints are not exposed in the UI.
  • Atlas, replica sets, sharded clusters, and live TLS certificate scenarios have not completed the production certification matrix.
  • MongoDB-to-PostgreSQL and MongoDB-to-MongoDB have automated evidence; other destination pairings and manual UI validation remain pending owner checks.

​
Troubleshooting

Continue with MongoDB destination or the MongoDB and PostgreSQL verified paths.