CREATE JOB
The section explains the different types of jobs and the options you can configure to ingest and transform your data.
The
CREATE JOB command enables you to create three types of jobs in Upsolver: ingestion, replication, and transformation. Using familiar SQL syntax, you can use the available options to customize the job to suit your requirements.- Ingestion jobs use the
COPY FROMcommand to copy your data from their source to a table within your metastore connection, or directly to a supported target. Upsolver ingestion jobs can automatically infer the schema and populate the column names and types in the table. - Replication jobs are a type of ingestion job that enables you to replicate your change data capture (CDC) tables to multiple targets using a single entity. Each replication group shares the same source but is configured to write to a different target. Operating independently from other groups, each group can use its own set of options to perform an individual purpose.
- Transformation jobs use
INSERTandMERGEstatements, enabling you to insert and update the data in your table. Using aMERGEstatement in your jobs provides the additional capability of deleting data if specified conditions are met.
When writing to data lake tables, if the
ADD_MISSING_COLUMNS option is enabled, transformation jobs automatically infer the schema and populate the column names and types in the table. Note that the table's underlying files are stored as Parquet when using jobs to write to data lake tables.
Last modified 1mo ago