Schema
Table Last Update
The timestamp of the last update by jobs writing to the table, indicating when the table was last modified. This only applies to datasets in your data lake:
Overview
The Overview card provides instant visibility into the following:
Number of Fields: the total count of all fields in the dataset, including system columns that are automatically added by Upsolver.
Number of Keys: the count of fields that appear unique.
A field will be considered as unique if all values in the sample data of the column are distinct.
Number of Arrays: The count of fields which are an array or sub-fields within an array. This number corresponds to the 'Array' column in the Written Data Statistics table.
Written Rows Over Time
This graph is a visual display to show the number of rows ingested over the selected timespan, enabling you to quickly discover spikes or drops in your pipeline and troubleshoot unexpected volumes. By default, the chart shows written rows from the initial ingestion through to now., however you can change this. Click on the Time View button, which displays Lifetime by default, to change the timespan displayed in the dataset report:
Ingested Data Statistics
This table provides powerful metadata about your dataset. You can click on a column name in the table to drill deeper into the Column details, which includes stats for the Upsolver system columns appended to your dataset. To filter the columns in the dataset, type your search string into the Search box. Click the x icon in the Search box to clear the results.
The system columns that Upsolver appends to your datasets are included in the column list:
The Ingested Data Statistics table provides the following information:
| Measurement | Description |
|---|---|
Column | The name of the column. Click on the column name to drill through to the Column page. |
Type | The column data type, e.g. |
Density | The percentage of records that have a non-null value. |
Top Values | The most common data values within this column. |
Is Uniqe | True if all values in the sample data of the column are distinct and false if the field has repeated values.(Note: this value might be true despite the field containing repeating values as the calculation is based on sample data). |
Distinct Values | The count of distinct values within the column. Values written before the METADATA_RETENTION period will not be counted. |
Array | True if the field is an array or a sub-field of an array. |
First Seen | The first date and time that a non-null value was seen in this column. |
Last Seen | The last date and time that a non-null value was seen in this column. |
Min | The minimum data value in this column. |
Max | The maximum data value in this column. |
Furthermore, if your table is in the data lake, you can download the dataset statistics to a table inspection report in CSV format for your own consumption. Click the Download icon to the right of the Search box to download the contents of the Ingested Data Statistics table to CSV. You can then perform your own discovery using this data.
Last updated
