Task executions table
This page describes how to use the task executions table to understand and troubleshoot your jobs in Upsolver.
The system information in Upsolver is designed to help you to monitor and troubleshoot your jobs, by providing internal insights. Jobs are divided into various tasks, with each task responsible for working with data, performing maintenance work, and more. This section describes the task execution table to help understand your jobs.
The task executions table enables you to monitor the execution of the tasks that run your jobs and maintain your tables. To monitor and troubleshoot your jobs, run the following query:
SELECT * FROM logs.tasks.task_executions;
The following three sections describe the task executions table:
- 1.Task execution records: This section includes a list of fields in the
task_executions
table. It includes the field name, data type, as well as a short description of how to interpret each value. - 2.Stage names: SQLake operations comprise multiple stages that execute tasks to complete a job. This section describes each of these stages that can be found in the
stage_name
field. This can help you to better understand the progress of your jobs and identify the status of each stage. - 3.Task event types: Each stage is a logical grouping of one or more tasks. This section describes the types of tasks that can be executed, along with a description of the task event types found in the
task_event_types
field.
Each record within the
task_executions
table describes a task being executed. The following is a list of the fields for each task:Field name | Data type | Description |
---|---|---|
cluster_name | string | The name of the cluster that processed this task. |
cluster_id | string | The unique ID of the cluster that processed this task. |
cloud_server_name | string | The ID of the cloud instance this job is running on. |
stage_name | string | Describes the type of task being executed.
For descriptions of the different stage names, see Stage names. |
job_name | string | The name of the job that the task belongs to. |
job_id | string | The unique ID of the job that the task belongs to. |
task_name | string | The name of the task formatted as the job_id with a prefix or suffix descriptor attached. |
task_start_time | timestamp | The start time of the window of data being processed. In a transformation job, this corresponds to the value of run_start_time() .
The difference between the task_start_time and task_end_time corresponds to the RUN_INTERVAL configured within the job options for transformation jobs.
For data ingestion jobs, this defaults to 1 minute. |
task_end_time | timestamp | The end time of the window of data being processed. This corresponds to the value of run_end_time() within transformation jobs.
The difference between the task_start_time and task_end_time corresponds to the RUN_INTERVAL configured within the job options for transformation jobs.
For data ingestion jobs, this defaults to 1 minute. |
shard | bigint | The shard number corresponding to this task. |
total_shards | bigint | The total number of shards used to process the job for this execution.
This corresponds to the value configured by the EXECUTION_PARALLELISM job option. If the value of EXECUTION_PARALLELISM is altered at any point, the total_shards for future tasks belonging to that job are updated to match. |
task_start_processing_time | timestamp | The time the task started being processed. |
task_end_processing_time | timestamp | The time the task finished being processed. |
task_items_read | bigint | The total number of records read. |
bytes_read | bigint | The total bytes ingested from the source data in its original form, including header information. |
bytes_read_as_json | bigint | The total bytes ingested from the source data if it were in a JSON format.
This is the number used to determine the volume of data scanned for billing purposes. |
duration | bigint | The time in milliseconds it took to process this task.
This is equivalent to the difference between the task_start_processing_time and task_end_processing_time . |
task_delay_from_start | bigint | The delay in milliseconds between the end of the data window and when the task began processing.
This is equivalent to the difference between the task_end_time and task_start_processing_time . |
task_classification | string | The classification of the task as user , system , input , or metadata based on the type of task being executed. |
task_error_message | string | The error message, if an error is encountered. |
task_event_type | string | Classifies the task into event types. |
organization_name | string | The name of the organization that the task belongs to. |
log_processing_time | timestamp | The time the log record was processed. |
organization_id | string | The unique ID of your organization (the same as the organization name). |
partition_date_str | string | The partition date as a string. |
partition_date | date | The date column that the table is partitioned by.
Always qualify a partition_date filter in your queries to avoid full scans. |
upsolver_schema_version | bigint | The system table's schema version. It changes when the user edits the output job that is written to this table. |
This section describes each of these stages that can be found in the
stage_name
field. Stage name | Description |
---|---|
file discovery | Discovers the files within a file-based data source such as Amazon S3, Azure Blob Storage, or Google Cloud Storage. |
data ingestion | Pulls data from the data source. |
parse data | Parses the data discovered during file discovery or data ingestion stage. |
Ingestion state maintenance | Performs maintenance work when data is being ingested. |
write to storage | Writes output to object store. |
write to target | Writes the data to the target location. |
cleanup | Deletes old files that are unnecessary.
This can be cleaning up unneeded files after compaction or removing other temporary files such as deleting batch files once the data has been parsed. |
table state maintenance | Collects and maintains metadata about files as they are written to tables.
This metadata is later used to perform tasks such as maintaining the file system, running compactions, running queries, and more. |
retention | Deletes old data and metadata that have passed the retention period as configured when the table was created. |
build indices | Builds indices for materialized views by reading the raw data and creating small files for the data that are then compacted and merged together. |
compact indices | Compacts indices for materialized views after they have been built. |
aggregation | Builds and compacts indices to perform aggregation for aggregated outputs. |
collect statistics | Gathers metadata from the ingestion or output job by generating indexes. |
compact statistics | Compacts and merges the metadata index. |
partition metadata | Processes metadata for partition management and maintenance. |
partition maintenance | Creates new partitions and deletes old ones. |
partition management | Creates new partitions and deletes old ones. |
count distinct metadata | Collects the number of distinct values for a field. |
event type metadata | Builds the metadata index for a field when an event type is set in Upsolver Classic. This allows us to filter by event type and show statistics per event type. |
upsert metadata | Maintains metadata about primary keys in order to know how and where to perform updates when they arrive as events. |
monitoring metadata | Ensures metadata is written successfully. |
dedup index | Builds the deduplication index. This index is used to run IS_DUPLICATE calculations. |
coordinate compaction | Coordinates partition compactions by checking available files. Simultaneously maintains other table metadata. |
compaction | Compacts smaller files into larger ones to optimize query performance when writing to a data lake output. |
upsert compaction | Compacts data from multiple files to delete old rows that have a newer update. |
compaction state maintenance | Performs maintenance work to ensure compaction state is healthy. |
maintenance | Performs general maintenance tasks. |
internal task | Performs tasks for working with connections to external environments. |
The following table describes the types of tasks that can be executed, along with a description of the task event types found in the
task_event_types
field.Event type | Description |
---|---|
started | The task has started. |
finished | The task has completed successfully . |
heartbeat | An indicator that the task is still running. This is sent every 5 minutes to determine if a task is long-running and the current state of the task (so it has the current duration, read bytes and etc). |
canceled | The task was canceled. |
no-resources | Indicates a lack of resources to start a task. This is usually due to a connection limitation. |
failed | The task has failed. Check task_error_message to better understand the error encountered. |
failed-build | Failed to build a task. |
failed-recoverable | An intermittent error has occurred (e.g. reading a file that was modified while reading it). The task will retry and recover from the error and the resulting data will be consistent. |
dry-run-failed | The task from Upsolver's automated testing process; the testing of a new version has failed. |
ignored-dry-run-failure | The dry run is ignored due to false positives. |
Last modified 2mo ago