Expire Snapshots
Snapshots in Iceberg tables represent consistent views of the table's data at a specific point in time. Each snapshot is created whenever changes are made to the table, such as adding, updating, or deleting data. These snapshots enable features like time travel, allowing users to query the table as it existed at a previous state.
Over time, as new snapshots are created, old ones may become irrelevant or redundant, consuming unnecessary storage. Snapshot expiration is the process of removing these outdated snapshots based on predefined policies, freeing up storage while maintaining the table's integrity and performance.
The Expire Snapshots tab lets you monitor the status of snapshot expiration in the table. It tracks the expiration process using the following parameters:
history.expire.max-snapshot-age-ms: The maximum age of snapshots to retain on the table and its branches during expiration.
history.expire.min-snapshots-to-keep: The minimum number of snapshots to retain on the table and its branches during expiration.
When the expire snapshots job runs, it checks for snapshots eligible for expiration based on these parameters. Once the job completes, the expired snapshots are deleted, and their storage is reclaimed.
Per Snapshot Expiration Job run, you will be able to track the following KPIs:
Job Start time
The timestamp indicating when the snapshot expiration job started.
Status
The current status of the snapshot expiration job. Possible values include: "Running", "Completed", "Failed (Retrying)", "Failed".
Duration
The total run time duration of the job.
Snapshots Expired
The total number of snapshots that were expired during the job execution.
Latest Expired Snapshot
The timestamp of the most recent snapshot that was expired during the job.
Data Files Deleted
The total number of data files that were deleted as a result of expiring the snapshots.
Metadata files Deleted
The total number of metadata files that were deleted during the snapshot expiration process.
Storage Size Deleted
The total amount of storage space (in bytes) freed by deleting the expired snapshots, data files, and metadata files.
Errors
Errors text in case errors were detected.
Last updated