Monitoring

This section describes the Monitoring tab for stream and file data sources.

Metric groups

Some metrics may not apply to the job type you created or only be relevant to particular data sources or target destinations.

• Summary: Displays the most relevant metrics to your job.

• All: All metrics applicable to your job.

Job metrics are categorized into three groups:

1. Job Execution Status

2. Data Scanned

3. Cluster

Job Execution Status

The Job Execution Status metrics include all jobs running on the cluster:

Metric	Description
Job executions currently running	The number of currently running job executions.
Job executions currently in queue	The number of queued job executions pending.
Job executions completed - today	The number of job executions completed today.
Job executions completed - lifetime	The number of job executions completed over the lifetime of the job.
Job executions currently waiting for dependencies	The number of job executions that are waiting for a dependency to complete.
Job executions currently retrying after failure	The number of job executions that encountered an error and are currently retrying.
Execution Failure Reason	The error message detailing why the job failed.

Data Scanned

The following metrics provide monitoring information regarding the data processed by your job:

Metric	Description
Rows read (completed executions)	The total number of rows scanned by completed executions today. This is a measure of rows that were processed successfully.
Rows filtered by WHERE clause	The number of rows that were filtered out because they didn’t pass the WHERE clause predicate defined in the job.
Average rows scanned per execution	The average number of rows scanned per job execution.
Maximum rows scanned in an execution	The maximum number of rows scanned in a single job execution today.
Rows Pending Processing	The number of rows in the source table that have not been processed yet.
Discovered Files	The number of files to load discovered by the job.
Discovered Bytes	The number of bytes to load discovered in the source stream.
Parse Errors (for ingestion jobs)	The number of items that failed to parse. This value represents a lower bound as malformed items may corrupt subsequent items in the same file as well.
Rows written (completed executions)	The number of rows written to the target by the job.
Rows filtered by HAVING clause	The number of rows that were filtered out because they did not pass the HAVING clause predicate defined in the job.
Rows filtered due to missing partition	The number of rows that were filtered out because some or all of the partition columns were NULL or empty string.
Rows filtered due to missing Primary Key	The number of rows that were filtered out because some or all of the primary key columns were NULL.
Bytes written (completed executions)	The size of the data written by the job.
Columns written	The number of columns written to by the job. This value can change over time if the query uses * in the SELECT clause.
Columns written - sparse	The number of sparse columns written to today. A sparse column is a column that appears in less than 0.01% of all rows.

Cluster

The following metrics provide information about the cluster running your job:

Metric	Description
Utilization Percent	Represents how much of the server's processing capacity is in use.
Tasks in Queue	The number of job tasks pending execution in the cluster queue.
Memory Load Index (%GC)	The percentage of time that the server is doing garbage collection rather than working.
Reload from disk percent	The percent of bytes re-loaded into memory from disk.
Cluster Crashes	How many server crashes happened in the job’s cluster today.