Microsoft SQL Server
This page describes how to create a replication job to ingest data from Microsoft SQL Server.
Syntax
See Replication jobs for a full list of job options.
Example
This example shows you how to create a replication job that ingests to Snowflake. To use a different target, replace the value in the REPLICATION_TARGET
option with the name of your your target connection.
Ingest CDC data to Snowflake example
The following example creates a job named mssql_replication_to_snowflake that writes to two replication groups in Snowflake: replicate_to_snowflake_prod for production use and replicate_to_snowflake_dev for development:
Each group has its own set of options that can be configured differently while streaming from the same data source.
Both groups replicate all tables in the orders schema, as specified in the INCLUDED_TABLES_REGEX
option. However, all creditcard columns are excluded from the production target in the EXCLUDED_COLUMNS_REGEX
option to remove PII.
The production replication group has a COMMIT_INTERVAL
of 5 MINUTES to keep the target frequently updated, whereas the development group has an interval of 1 HOUR to update less often. Furthermore, the production group will MERGE
the data, using the column is_deleted to denote if a row is deleted in the source, and the development group will APPEND
the data.
The REPLICATION_TARGET
is set to my_snowflake_connection for both groups, but this can be any Upsolver-supported target.
In the development group, the TARGET_SCHEMA_NAME_EXPRESSION
option includes the prefix value of history_. This means that Upsolver will create the target table using the name of the source table with the prefix so it is clear that the target tables in this group are used for development.
Last updated