LogoLogo
OverviewQuickstartsHow To GuidesReferenceArticlesSupport
Quickstarts
Quickstarts
  • Quickstarts
  • DATA INGESTION WIZARD
    • Using the Wizard
      • Source Set-up
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Cloud
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
      • Target Set-up
        • Amazon Redshift
        • AWS Glue Data Catalog
        • ClickHouse
        • Polaris Catalog
        • Snowflake
      • Job Configuration
        • Job Configuration
        • Job Configuration for CDC
      • Review and Run Job
  • CONNECTORS
    • Connectors
      • Amazon Kinesis
      • Amazon Redshift
      • Amazon S3
      • Apache Kafka
      • AWS Glue Data Catalog
      • ClickHouse
      • Confluent Cloud
      • Elasticsearch
      • Microsoft SQL Server
      • MongoDB
      • MySQL
      • Polaris Catalog
      • PostgreSQL
      • Snowflake
  • JOBS
    • Ingestion
      • Job Basics
        • Ingest to a Staging Table
        • Output to a Target Table
      • Stream and File Sources
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Kafka
      • CDC Sources
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
    • Transformation
      • Updating Data
        • Upsert Data to a Target Table
        • Delete Data from a Target Table
        • Aggregate and Output Data
        • Join Two Data Streams
      • Data Targets
        • Output to Amazon Athena
        • Output to Amazon Redshift
        • Output to Amazon S3
        • Output to Elasticsearch
        • Output to Snowflake
  • APACHE ICEBERG
    • Optimize Your Iceberg Tables
    • Install the Iceberg Table Analyzer
Powered by GitBook
On this page
  • Prerequisite - Configure MongoDB for CDC
  • Step 1 - Connect to your MongoDB
  • Create a new connection
  • Use an existing connection
  • Step 2 - Select databases and collections
  1. DATA INGESTION WIZARD
  2. Using the Wizard
  3. Source Set-up

MongoDB

Follow these steps to use MongoDB as your source.

Last updated 11 months ago

Prerequisite - Configure MongoDB for CDC

Upsolver supports ingesting CDC data from relational databases including MongoDB. Upsolver provides CDC capabilities by running a Debezium Engine under the hood. Connectors detect and ingest changes automatically.

Before you ingest data into Upsolver, you must enable change data capture on your MongoDB database. If you are using a managed MongoDB service such as Atlas, CDC is most likely to be enabled, if not, please refer to the guide to for more information

Step 1 - Connect to your MongoDB

Create a new connection

Click Create a new connection, if it is not already selected.

Enter your Connection String in the following format:

mongodb+[srv]://HOST

where:

  • HOST - the hostname or IP address of the MongoDB Server instance you want to connect to, for example, my-mongo.mongodb.net.

Optionally, provide a username and password to authenticate to the database, and a Comment.

In the Name your connection field, type in the name for this connection. Please note this connection will be available to other users in your organization.

Use an existing connection

By default, if you have already created a connection, Upsolver selects Use an existing connection, and your MongoDB connection is populated in the list.

For organizations with multiple connections, select the source connection you want to use.

Step 2 - Select databases and collections

There are two ways to replicate objects to the target:

  1. Manually select the databases and collections.

  2. Use regular expressions to specify which databases and collections are included and excluded.

The following examples show how to use a regular expression to include specific collections:

Expression
Results

db_name.*

Select all collections in the db_name database.

db_name.users,

db_name.items

Select users and items collections in the db_name database.

db_name.items_.*

Select all collections in the db_name database that start with items_.

"my.db".*

Select all collections in the my.db database.

If the name of the database or collection that you want to include contains a dot (.) in the name of the database or collection, you must escape each part of the name in double quotes.

Note: the following databases are internal MongoDB system databases and therefore they will not be replicated:

  • admin

  • local

  • config

Schema evolution

By default, new databases, collections, and fields added after the job is launched, will be replicated to the target.

You can overwrite the default behavior and set specific behavior for newly created databases or collections in a specific database.

Deploy a Replica Set
Create a new connection to MongoDB to ingest your CDC data.
Select your existing MongoDB connection to use for your ingestion job.