Job Execution Status

These metrics provide information about your job, enabling you to monitor performance and check for issues.

Job executions currently running

More information

A job is scheduled to run every defined time interval (i.e every minute, hour, day etc). A job execution is defined as an execution of a specific time interval. As long as the job is up-to-date, a single job execution will be running according to the expected running schedule.

In case there's a backlog, multiple job executions can run concurrently. Each execution will handle a different time interval. This can happen, for example, due to historical running of the job or a spike in the amount of data which causes the job execution duration to exceed the time between intervals.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', running_executions) AS     running_executions, 'OK' AS running_executions_severity 
FROM system.monitoring.jobs 
WHERE job_id = '<job_id>';

Job executions currently in queue

More information

This metric is the count of runnable jobs not running. The value for an up-to-date job should be 0. If you are performing a replay, this number could be very high, potentially in the thousands. However, as the replay runs, the number should steadily decrease at the rate at which work is being done. If it is not a replay, it can mean that the cluster is not big enough to handle the workload. A value below 10 is considered acceptable, but you may have a problem if it is increasing, and the cluster is not managing to keep up with the workload.

Be aware that an increasing queue will cause latency issues with your data. However, if the number is constant over time and the latency is acceptable, then this may not be an issue for you. If the queue increases, it should be investigated.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', queued_executions) AS queued_executions, IF_ELSE(queued_executions > 0, 'WARNING', 'OK') AS queued_executions_severity 
FROM system.monitoring.jobs 
WHERE job_id = '<job_id>';

Job executions completed - today

More information

The total number of job executions completed today. A job is scheduled to run every defined time interval (i.e. every minute, hour, day etc). A job execution is defined as an execution of a single time interval.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', completed_executions_today) AS completed_executions_today, 'OK' AS completed_executions_today_severity 
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';

Job executions completed - lifetime

More information

The total number of job executions completed over the lifetime of the job. A job is scheduled to run every defined time interval (i.e. every minute, hour, day etc). A job execution is defined as an execution of a single time interval.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', total_completed_executions) AS total_completed_executions, 'OK' AS total_completed_executions_severity
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';

Job executions currently waiting for dependencies

More information

The number of job executions currently pending upon other jobs to complete their work before this execution can run.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', executions_waiting_for_dependencies) AS executions_waiting_for_dependencies, 'OK' AS executions_waiting_for_dependencies_severity
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';

Job executions currently retrying after failure

More information

The number of job executions that encountered an error and are currently retrying. Ideally, this should be 0. If a job encounters a transient error, the value will disappear after the retry is successful, otherwise, investigation is required to fix the issue. The retry will continue as long as the issue occurs and will stop only once it is resolved.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT STRING_FORMAT('{0,number,#,###}', executions_retrying_after_failure) AS executions_retrying_after_failure, IF_ELSE(executions_retrying_after_failure > 0, 'ERROR', 'OK') AS executions_retrying_after_failure_severity 
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';

See thelogs.tasks.task_executions system table for details and error messages.

Run the following query to return tasks that have encountered an error:

SELECT job_id, job_name, stage_name, job_type, 
        task_name, task_start_time, task_end_time, task_error_message
FROM logs.tasks.task_executions 
WHERE task_error_message IS NOT NULL
ORDER BY job_id;

Execution Failure Reason

More information

The error message detailing why the job failed, which will be unique to your job. You can adapt the query under the See All Events (SQL Syntax) tab to view the full error for the message relevant to your job.

Run the following SQL command in a query window in Upsolver, replacing <job_id> with the Id for your job. The Id for your job can be found in the Details section under the Settings tab.

For additional columns, alter this statement and use SELECT *.

SELECT execution_failure_reason
FROM system.monitoring.jobs 
WHERE job_id '<job_id>';

Last updated 4 months ago