Monitoring metrics

This article provides a list of data source monitoring metrics and descriptions.

Data source metrics

Metric	Description	Unit
data_source_failed-items	Data source parse errors	Number
data_source_handled-bytes	Number of bytes handled by the data source	Number
data_source_handled-items	The number of items (records) handled by the data source	Number
data_source.ingest_delay	Data source ingestion delay	Seconds
data_source.parse_delay	Data source parse delay	Seconds
data_source.parse_min-delay	Data source minimum delay. This can be used as a filter for the delay metric	Seconds
data_source.errors_errors	Errors encountered when ingesting data	Number
binlog-delay	Read delay from the binlog of the database for CDC inputs	Seconds

Lookup table metrics

Metric	Description	Unit
materialized_view_estimated_rows	Number of rows currently in the lookup table (available if the lookup table runs on a query cluster)	Number
materialized_view.delay_delay	Lookup table delay	Seconds
materialized_view.query_hits	The number of queries that resulted in a lookup table hit (available if the lookup table runs on a query cluster)	Number
materialized_view.query_misses	The number of queries that resulted in a lookup table miss (available if the lookup table runs on a query cluster)	Number
materialized_view.query_queries	Number of queries sent to the lookup table (available if the lookup table runs on a query cluster)	Number
materialized_view.errors_errors	Errors encountered while running the lookup table	Number

Output metrics

Metric	Description	Unit
output_handled-bytes	Number of bytes handled by the output	Bytes
output_handled-items	Number of items (records) handled by the data source	Number
output_written-items	Number of items written to the output destination	Number
output_written-bytes	The number of bytes written to the output destination	Bytes
output.delay_delay	Output delay	Seconds
output.delay_min-delay	Output minimum delay. This can be used as a filter for the delay metric	Seconds
output.errors_errors	Errors encountered when outputting data	Number
output.data-transformation_handled-items	Number of items (records) handled during output processing (only relevant to database and stream outputs)	Number
output.data-transformation_handled-bytes	Number of bytes handled during output processing (only relevant to database and stream outputs)	Bytes
output.data-transformation-delay_delay	Delay during output processing (only relevant to database and stream outputs)	Seconds
output.hive.partition_manager_last-success-diff	The amount of time passed since the last successful execution of the partition manager in Athena/Qubole/Redshift Spectrum outputs	Seconds
output.hive.partition_manager_partitions-count	The number of partitions in the table (for Athena/Qubole/Redshift Spectrum outputs)	Number
output.hive.partition_manager_partitions-delay	When using PARTITION_TIME, this metric calculates the delay between now and the latest partition time (for Athena/Qubole/Redshift Spectrum outputs)	Seconds
output.hive.partition_manager_data-loading-delay	When creating partitions, this metric shows the delay of the data loading used to calculate the partitions (for Athena/Qubole/Redshift Spectrum outputs)	Seconds
output.hive.partition_manager_wal-size	When using Upserts, this shows the number of rows in the WAL (Write Ahead Log) partition (for Athena/Qubole/Redshift Spectrum outputs)	Number
output.hive.partition_manager_wal-files-count	When using Upserts, this shows the number of files in the WAL (Write Ahead Log) partition (for Athena/Qubole/Redshift Spectrum outputs)	Number
output.hive.partition_manager_compaction-delay	Shows the delay of the compaction process of Data Lake outputs (Athena/Spectrum/etc). Delayed compactions affect query speeds but they do not affect data freshness.	Seconds

Compute cluster and environment metrics

Metric	Description	Unit
upsolver_compute_units_value	The amount of Upsolver compute units used in a cluster	Number
environment_utilization_value	The average CPU load of all the servers in a cluster	Percent (0-100)
memory_memory	Memory usage per server	Percent (0-100)
tasks_in_queue	The current number of tasks pending execution in the cluster queue	Number