Microsoft SQL Server
This page describes how to ingest your data from Microsoft SQL Server.
Prerequisites
Create a CDC job that reads from your database
You can create a job to ingest your data from SQL Server into a staging table in the data lake.
Jump to
Ingest to the data lake
After completing the prerequisites, you can create your staging tables. The example below creates a table without defining columns or data types, as these will be inferred automatically by Upsolver, though you can define columns if required:
Upsolver recommends partitioning by the system column $event_date
or another date column within the data in order to optimize your query performance.
Next, create an ingestion job as follows:
Job options
The example above only uses a small subset of all job options available when reading from SQL Server. Depending on your use case, you may want to configure a different set of options.
Transformations can be applied to your ingestion job, for example, to exclude columns, correct issues, or mask data, before it lands in the target. Furthermore, you can use expectations to define data quality rules on your data stream and take appropriate action.
Alter a job
You can alter some of the options of an existing job. For example, if you want to keep the job as is, but only change the cluster that is running the job, execute the following command:
Note that some options such as COMPRESSION
cannot be altered once the job has been created.
Drop a job that reads from SQL Server
If you no longer need a job, you can easily drop it using the following SQL command:
Learn More
Last updated