Datasets
Real-Time Data Observability
The Datasets tab in Upsolver provides essential insights into your data, enabling you easily to uncover performance problems, and troubleshoot and diagnose data quality issues. These insights are available to everyone in your organization, meaning anyone can drill deep into the data statistics and observe the health of your data.
Whether you're a data engineer responding to end-user queries about the data lineage in your pipelines, or a consumer investigating the freshness of your data, the Datasets tab is your go-to location for data observability.
Using Datasets, you can drill into source data stored in your staging tables in your data lake, and view the data in your analytics targets (if you have created a direct ingestion job, you will only see the target schema). Datasets make it easy to compare the results in your target with the data from your source, so you can quickly trace back to uncover where problems first appeared. The written rows charts deliver an instant insight into the volume of data flowing to your target, making spikes and dips in your data easy to identify.
Viewing Datasets
To open your datasets, click on the Datasets link in the sidebar menu in Upsolver. You may want to expand the menu if it is collapsed by clicking on the arrow icon at the bottom of the menu. The entities tree then displays your datasets.
Expand a connection in the tree to view the schemas and tables. The tree will only display schemas and tables that are ingestion targets for jobs created in Upsolver. Alternatively, use the Search box to find an object: you can search by schema or table name. Click the cross icon in the search box to clear your results and return to the default view.
From the entities tree, you can click on a schema name to view the details for the full dataset, or click on a column name to drill through to view the column level data. The system columns are included to provide you with full observability.
Each dataset provides the following tabs:
Properties (applicable only to staging tables in your data lake)
Last updated
