LogoLogo
OverviewQuickstartsHow To GuidesReferenceArticlesSupport
Overview
Overview
  • Welcome to Upsolver
  • GET STARTED
    • What is Upsolver?
    • Schedule a Demo
    • Start Your Free Trial
    • Apache Iceberg
  • RESOURCES
    • Reference
    • Iceberg Academy
    • Blog
    • Chill Data Summit
    • Community
    • Videos
  • RELEASE NOTES
    • March 2025
    • February 2025
    • January 2025
    • Earlier Releases
      • 2024
        • December 2024
        • November 2024
        • October 2024
        • September 2024
        • August 2024
        • July 2024
        • June 2024
        • May 2024
        • April 2024
        • March 2024
        • February 2024
        • January 2024
      • 2023
        • December 2023
        • November 2023
          • Deprecated Job Option
        • October 2023
        • September 2023
        • August 2023
        • July 2023
        • June 2023
        • May 2023
  • Legal
Powered by GitBook
On this page
  1. RELEASE NOTES
  2. Earlier Releases
  3. 2024

December 2024

Upsolver new features, enhancements, and bug fixes for December 2024.

Last updated 4 months ago

2024.11.28-08.49

⬆️ Enhancements

  • Datasets

    • Enhanced Iceberg Statistics Page

      We enhanced the Iceberg table page, located in the module, with new metrics and insights to improve your monitoring experience. Track key metrics including storage, scan time, row count, partition stats, and details on time travel, snapshots, compactions, data lifecycle, and orphan file clean-up. The page provides a high-level summary, while the tab offers deeper insights into specific processes such as compaction, snapshot expiration, and orphan file clean-up.

    • Snapshot Expiration Tracking

      Monitor Iceberg table snapshot expiration using the new tab, located in > . Snapshots are created with each table change, but outdated snapshots can accumulate and consume storage. The expiration process removes these snapshots based on predefined policies, freeing up storage while maintaining table integrity.

    • Orphan Files Clean-up Tracking

      Monitor Iceberg table orphan files clean-up with the new tab, located in > . In distributed processing environments, tasks or jobs sometimes fail, leaving behind files that are not referenced in the table metadata. These files, referred to as orphan files, can accumulate over time and consume significant storage space. A file is considered an orphan if it is not associated with any valid snapshot in the table metadata. Regular clean-up of these files is essential for optimizing storage and maintaining efficient table operations. The tab enables you to monitor clean-up jobs that remove these files.

    • New Partitions Tab

      Partition information has been moved to a dedicated tab (previously in the tab).

    • New Columns Tab

      Find the columns in your table as defined in the Iceberg specification in the new tab.

  • New Metadata retention option for jobs writing to Iceberg tables:

    • We introduced a new configuration option, , for jobs writing to Iceberg tables. This feature enables you to define the retention period for the statistical data collected during job execution. By managing the retention of this metadata, you can ensure that relevant statistics are available for data analysis while managing storage costs associated with keeping this information. For more details, please refer to the page.

  • We’ve introduced retention policies for Iceberg tables, enabling you to define and enforce data retention directly within Upsolver:

    • Two new options are now available when defining your Iceberg tables:

      • : Specify the column to determine data retention. Compatible column types include DATE, TIMESTAMP, TIMESTAMPTZ, LONG, or INT. Optimal performance is achieved when the retention column is part of the table's partition columns.

      • : Set the number of days data should be retained. Values range from 1 to 9999 days, and data older than this duration will be scheduled for deletion, ensuring efficient storage cost management. For detailed information, please see .

  • Iceberg:

    • Reduced the number of snapshots created on compactions.

    • Upgraded library version to 1.6.1.

    • Improved the performance of the commit operation when processing large backlogs.

  • Support via ALTER CONNECTION <connection_name> RENAME TO <new_name>.

  • Ingestion Wizard:

    • We enhanced the experience on the Jobs page for creating new jobs. When Iceberg is selected as the target, the wizard now includes an additional step that guides you through defining key table properties. Now you can easily configure the target table’s name, columns, partitions, retention policies, sorting, and snapshot expiration in a streamlined and intuitive process.

🔧 Bug Fixes

  • Iceberg:

    • Fixed the incorrect update to the partition spec of compacted files, after partition evolution.

  • Fixed a bug that caused some internal files not to be deleted when dropping an Iceberg table.

Columns
Apache Iceberg job options
renaming connections
UI Wizard
Statistics
Datasets
Maintenance
Expire Snapshots
Datasets
Maintenance
Orphan Files
Datasets
Maintenance
Orphan Files
Partitions
Statistics
METADATA_RETENTION
RETENTION_COLUMN
RETENTION_DURATION
CREATE ICEBERG TABLE