Jobs

The jobs system table includes information about the jobs that run within your organization.

From a worksheet within Upsolver, run the following query:

SELECT * 
FROM system.monitoring.jobs;

The system.monitoring.jobs table includes the following columns:

ColumnTypeDescription

avg_file_size_today

bigint

The average size of the files discovered by the job today.

avg_rows_scanned_per_execution_today

bigint

The average number of rows scanned per job execution.

bytes_written_today

bigint

The size of the data written by the job.

catalog_error

string

Errors returned by the catalog today.

columns_written_to_today

bigint

The number of columns written to by the job. This value can change over time if the query uses * in the select clause.

completed_executions_today

bigint

The number of job executions completed today.

discovered_bytes_today

bigint

The number of bytes to load discovered in the source stream.

discovered_files_today

bigint

The number of files to load discovered by the job today.

discovered_records_today

bigint

The number of records to load discovered by the job.

drop_expectations_triggered_today

bigint

The number of expectations with a DROP action that was triggered by the job today.

execution_failure_reason

string

The error message detailing why the job failed.

executions_retrying_after_failure

bigint

The number of job executions that encountered an error and are currently retrying.

executions_waiting_for_dependencies

bigint

The number of job executions that are waiting for a dependency to complete.

input_id

string

The unique identifier for the ingestion job input.

job_id

string

The unique identifier for the job.

job_name

string

The text name of the job.

load_failure_reason

string

The error message as to why the data failed to load.

max_bytes_written_to_single_partition_today

bigint

The maximum number of bytes written to a single partition by the job today.

max_rows_scanned_in_execution_today

bigint

The maximum number of rows scanned in a single job execution today.

next_run_time

string

The date and time when the job will next run.

parse_errors_today

bigint

The number of items that failed to parse. This value represents a lower bound as malformed items may corrupt subsequent items in the same file as well.

parquet_files_written_today

bigint

The number of Parquet files written to by the job today.

partitions_written_to_today

bigint

The number of partitions written to by the job today.

queued_executions

bigint

The number of queued job executions pending.

running_executions

bigint

The number of currently running job executions.

rows_filtered_by_dedup_clause_today

bigint

The number of rows filtered out by a deduplication clause in the job today.

rows_filtered_by_expectations_today

bigint

The number of rows filtered out by an expectation today.

rows_filtered_by_having_clause_today

bigint

The number of rows that were filtered out because they didn’t pass the HAVING clause predicate defined in the job.

rows_filtered_by_missing_partition_today

bigint

The number of rows that were filtered out because some or all of the partition columns were NULL or empty string.

rows_filtered_by_missing_primary_key_today

bigint

The number of rows that were filtered out because some or all of the primary key columns were NULL.

rows_pending_processing

bigint

The number of rows in the source table that have not been processed yet. Only rows that have been committed to the source table are included.

rows_scanned_by_cancelled_tasks_today

bigint

The number of rows scanned by tasks that were cancelled today.

rows_scanned_by_completed_tasks_today

bigint

The total number of rows scanned by completed executions today. This is a measure of rows that were processed successfully.

rows_scanned_by_failed_tasks_today

bigint

The number of rows scanned by tasks that failed in the job today.

rows_filtered_by_where_clause_today

bigint

The number of rows that were filtered out because they didn’t pass the WHERE clause predicate defined in the job.

rows_scanned_today

bigint

The number of rows scanned by the job today.

rows_written_today

bigint

The number of rows written to the target by the job today.

running_executions

bigint

The number of currently running job executions.

sparse_columns_written_to_today

bigint

The number of sparse columns written to today. A sparse column is a column that appears in less than 0.01% of all rows.

tasks_failing_to_load

bigint

The number of tasks in the job that failed to load today.

total_completed_executions

bigint

The number of job executions completed over the lifetime of the job.

total_tasks_to_load

bigint

The total number of tasks that were loaded.

warn_expectations_triggered_today

bigint

The number of expectations with a WARN action that was triggered in the job today.

Last updated