Amazon Redshift
Last updated
Reference
Last updated
The following job options are used when writing to Amazon Redshift tables. Upsolver will create missing tables in Redshift.
Amazon Redshift job options:
General job options:
AGGREGATION_PARALLELISM — editableType: integer
Default: 1
(Optional) Only supported when the query contains aggregations. Formally known as "output sharding."
FAIL_ON_WRITE_ERRORType: Boolean
Default: false
(Optional) When true, the job will fail when an on-write error occurs.
SKIP_FAILED_FILESType: Boolean
Default: true
(Optional) When true, the job will skip any files in which the job is unsuccessful and continue with the rest of the files.
TRUNCATE_COLUMNSType: Boolean
Default: false
(Optional) When true, data is truncated when inserting into an existing table containing VARCHAR or CHARACTER VARYING columns smaller than the incoming data size, to prevent the insert from failing.
When false, the insert fails if the incoming data is larger than the target column and SKIP_FAILED_FILES is also false.
ADD_MISSING_COLUMNSType: Boolean
Default: false
(Optional) When true, columns that don't exist in the target table are added automatically when encountered.
When false, you cannot do SELECT * within the SELECT statement of your transformation job.
ON_COLUMN_TYPE_MISMATCHType: String
Default: None
Possible values: Add Column ,None
This option is applicable only if ADD_MISSING_COLUMNS=true.
It determines how to handle cases where the datatype of the source data does not match the datatype of the corresponding column in the target table.
If set to Add Column, Upsolver will attempt to cast the incoming data to the original column's datatype. If the cast fails, a new column with the format <originalColumnName>_newDataType will be created, and the mismatched data will be written to this new column. For example, if the CLIENT_ID column is a number in the target table and a VARCHAR arrives in that column, a new column called CLIENT_ID_VARCHAR will be added for the string data. The original column will continue to be populated if the data can be cast successfully.
If set to None, no new columns will be added, and only valid casts will be written to the original column.