Export Metrics to a Third-Party System
This article shows you how to export your job metrics to a third-party system for continuous monitoring.
Monitoring data plays a crucial role in ensuring the reliability and performance of your data pipelines. Some of the key advantages of exporting your monitoring data to a third-party system include:
Centralized visibility: by integrating with third-party monitoring systems, you can centralize your monitoring data in one location. This simplifies the monitoring and troubleshooting process because you don't need to switch between different tools and dashboards to gain insights into your data pipelines and systems.
Customization: third-party monitoring systems regularly offer advanced visualization and reporting capabilities. You can customize dashboards and alerts to suit your specific monitoring needs, enabling you to focus on the most critical metrics and anomalies for your business.
Alerting and notifications: many third-party monitoring tools provide robust alerting and notification features. You can create alerts based on predefined thresholds or anomalies in your data, ensuring you are promptly informed of issues requiring attention.
Upsolver exposes numerous metrics within the system monitoring tables that you can export to a supported platform.
Supported Integrations
Step 1
Connect to your target
The first step is to create a connection to your target monitoring platform where you will send the metrics data for your Upsolver account.
Here's the code:
This script creates a new CloudWatch connection named my_cloudwatch_connection, which will use to send our metric data. Like any other connection you create in Upsolver, the connection is persistent, so you need only create this once.
Step 2
Create a job to export job metrics
Next, let's create a job named send_job_metrics_to_cloudwatch that will send data from the system.monitoring.jobs table to the upsolver namespace in the CloudWatch account that we created above:
The send_job_metrics_to_cloudwatch includes the job_id and job_name tags to ensure the names and ids of your jobs are sent to CloudWatch, which is essential when you have multiple jobs running in your organization.
Upsolver creates the send_job_metrics_to_cloudwatch job, and immediately begins sending metrics on your jobs to CloudWatch.
If the upsolver namespace does not exist, Upsolver will create it for us.
Step 3
Create a job to export cluster metrics
Now, let's create another job named send_cluster_metrics_to_cloudwatch that will send data from the system.monitoring.clusters table to the same upsolver namespace:
The send_cluster_metrics_to_cloudwatch job includes the cluster_id and cluster_name tags to ensure the name and id of each cluster are sent to CloudWatch, which is helpful if you have more than one cluster in your organization.
As above, Upsolver starts sending the cluster metrics to CloudWatch immediately after the job is created, and will run continuously until the job is deleted.
System Tables
In these example, we used the system.monitoring.jobs and system.monitoring.clusters tables, but you can make use of further system tables to build a query to feed your monitoring requirements.
Conclusion
In this how-to guide, you learned how to create jobs within Upsolver to send metric data to your CloudWatch account to monitor your jobs and clusters. By integrating metrics from Upsolver into CloudWatch, you can use a central location to gain visibility into multiple operations within your organization, and ensure optimal performance and up-time.
Try it yourself
To monitor your Upsolver jobs from a third-party monitoring system:
Create a connection to your monitoring system.
Create one or more jobs to select metrics from the system tables, and write this to your monitoring system.
Last updated