CREATE JOB

The CREATE JOB command enables you to create ingestion and transformation jobs, while the CREATE REPLICATION JOB command provides the ability to create replication groups based on a CDC source.

Using familiar SQL syntax, you can use the available options to customize the job to suit your requirements:

• Ingestion jobs use the COPY FROM command to copy your data from their source to a table within your metastore connection, or directly to a supported target. Upsolver ingestion jobs can automatically infer the schema and populate the column names and types in the table.

• Replication jobs are a variation of ingestion job that enables you to replicate your change data capture (CDC) tables to multiple targets using a single entity. Each replication group shares the same source but is configured to write to a different target. Operating independently, each group can use its own set of options to perform an individual purpose.

• Transformation jobs use INSERT and MERGE statements, enabling you to insert and update the data in your table. Using a MERGE statement in your jobs provides the additional ability to delete data if specified conditions are met.

• Monitoring jobs use similar syntax and, using an INSERT statement, you can export job metrics to your monitoring dashboard.

When writing to data lake tables, if the ADD_MISSING_COLUMNS option is enabled, transformation jobs automatically infer the schema and populate the column names and types in the table. Note that the table's underlying files are stored as Parquet when using jobs that write to data lake tables.