Job monitoring

All jobs overview

Job monitoring summary page

1. Jobs

Access to job monitoring's summary page. From this page, you can access each job to monitor or debug.

2. Job

Name of the job that you've created. Click on each job to access a detailed job page.

3. Status

Status of each job. Whether the job is running or in other phases.

4. Delay

Backlog of events being processed.

5. Errors Last Day

Number of errors in the last 24 hours.

6. Events Over Time

A graph of the events processed since the job started. Hover over the graph for exact number of events processed at a certain point in time.

7. Created At

How long ago the job was created.

8. From... To

Where the data is being read from and loaded into.

9. Filter

Filter the jobs summary table by a value within a column.

Job monitoring summary

COPY job monitoring (page 1)

1. From... To...

Shows where the data is being copied from and where it is being loaded into. The example shows reading from S3 and loading into an Athena table.

2. Name of the job

The name of the job defined when it was created.

3. Status

Whether the job is running or not.

4. Cluster

The compute cluster that the job is running on.

5. Unsolved Errors (Last Hour)

Whether there are unsolved errors within the past hour. You can click on it to get more details on the errors.

6. Events Processed (Last Hour)

How many events were parsed in the past hour.

7. Processing Rate (Last Hour)

The rate of events being processed in the past hour.

8. Processing Backlog (Last Hour)

The timestamp waiting to be processed relative to the timestamp now.
For example, seeing a processing backlog of 3 hours means the events that are being processed have a timestamp that's 3 hours earlier than the current timestamp.

9. Time picker

Allows graphs below to change based on the time range that you pick.

10. Delay

Shows the backlog being processed over time based on the time picker range.

11. Name... Current

Name of the current work a job is performing and the backlog in relation to that work.

Job events and cluster utilization

COPY job monitoring (page 2)

1. Handled Events

Number of events processed over time based on the time picker.

2. Cluster Utilization

CPU utilization of a cluster. You can see the resources the job is consuming relative to the entire cluster utilization.

3. Errors Over Time

How many errors occurred over time based on the time picker.

Job errors

Job monitoring (page 3)

1. Error Details

This table gives you a view of the errors.

2. Filter

You may filter the table by Unresolved Errors, Resolved Errors, or All Errors over a period of time.

3. Error table

Details of each error that were generated by your job. This can help you to troubleshoot your job.
For example, if there are any permission errors, it will show up in this table and you will know to adjust your permissions accordingly.

4. System tables

You may click on Query System Table to start writing SQL against the system table. The system table is very powerful and can help you to debug your jobs and their related tasks.

Parsing errors

The COPY FROM jobs have an additional panel to help you debug parsing errors.
For example, if your JSON has formatting errors, the object will show up on the parsing error panel. Only COPY FROM jobs have the parsing panel on the bottom of the monitoring page.