CREATE JOB
The section explains the different types of jobs and the options you can configure to ingest and transform your data.
The
CREATE JOB
command enables you to create three types of jobs in Upsolver: ingestion, replication, and transformation. Using familiar SQL syntax, you can use the available options to customize the job to suit your requirements.- Ingestion jobs use the
COPY FROM
command to copy your data from their source to a table within your metastore connection, or directly to a supported target. Upsolver ingestion jobs can automatically infer the schema and populate the column names and types in the table. - Replication jobs are a type of ingestion job that enables you to replicate your change data capture (CDC) tables to multiple targets using a single entity. Each replication group shares the same source but is configured to write to a different target. Operating independently from other groups, each group can use its own set of options to perform an individual purpose.
- Transformation jobs use
INSERT
andMERGE
statements, enabling you to insert and update the data in your table. Using aMERGE
statement in your jobs provides the additional capability of deleting data if specified conditions are met.
When writing to data lake tables, if the
ADD_MISSING_COLUMNS
option is enabled, transformation jobs automatically infer the schema and populate the column names and types in the table. Note that the table's underlying files are stored as Parquet when using jobs to write to data lake tables.
Last modified 1mo ago