Job Options
Last updated
Last updated
The following job options can be applied to all targets. See the links below for .
COMMENT — editableType: text
(Optional) A description or comment regarding this job.
COMPUTE_CLUSTER — editableType: identifier
Default: The sole cluster in your environment
(Optional) The compute cluster to run this job.
This option can only be omitted when there is just one cluster in your environment.
Once you have more than one compute cluster, you are required to provide which one to use through this option.
END_AT — editableValues: { NOW | timestamp }
Default: Never
(Optional) Configures the time to stop inserting data. Data after the specified time is ignored.
RUN_INTERVALValue: <integer> { MINUTE[S] | HOUR[S] | DAY[S] }
Default: 1 MINUTE
(Optional) How often the job runs.
The runs take place over a set period of time defined by this interval and they must be divisible by the number of hours in a day.
For example, you can set RUN_INTERVAL to 2 hours (the job runs 12 times per day), but trying to set RUN_INTERVAL to 5 hours would fail since 24 hours is not evenly divisible by 5.
RUN_PARALLELISM — editableType: integer
Default: 1
(Optional) Controls how many jobs run in parallel to process a single minute of data from the source table.
Increasing this can lower the end-to-end latency if you have lots of data per minute.
START_FROMValues: { NOW | BEGINNING | timestamp }
Default: BEGINNING
(Optional) Configures the time to start inserting data from. Data before the specified time is ignored.
Visit the pages below for target-specific job options:
If set as a timestamp, it should be aligned to the .
For example, if is set to 5 minutes, then you can set an end time of 12:05 PM but not 12:03 PM. Additionally, the timestamp should be based in UTC and in the following format: TIMESTAMP 'YYYY-MM-DD HH:MM:SS'.
If set to NOW, the job runs up until the previous full period. For example, if the current time is 12:03 PM, creating the job with a of 5 minutes ending at NOW means that the last task executed by the job ends at 12:00 PM.
If set as a timestamp, it should be aligned to the .
For example, if is set to 5 minutes, then you can set a start time of 12:05 PM but not 12:03 PM. Additionally, the timestamp should be based in UTC and in the following format: TIMESTAMP 'YYYY-MM-DD HH:MM:SS'.
If set to NOW or BEGINNING, the job runs from the previous full period. For example, if the current time is 12:03 PM, creating the job with a of 5 minutes starting from NOW means that the first task executed by the job starts from 12:00 PM.
