January 2025
Upsolver new features, enhancements, and bug fixes for January 2024.
Last updated
Overview
Upsolver new features, enhancements, and bug fixes for January 2024.
Last updated
⬆️ Enhancements
API: Excluded statistics of deleted jobs from table ingested data statistics.
Kafka: Added support for using regex in the TOPIC property of Kafka jobs, enclosed in /regex/ syntax, alongside existing string and glob patterns.
Iceberg: New cleanup task to get rid of unused deleted files
Monitoring - introduced a new system table, system.monitoring.retention_deletions, that shows the retention executions of Iceberg tables.
Replication Job: Added support for configuring the primary key column name for each replication group using the new PRIMARY_KEY_COLUMN property
🔧 Bug Fixes
Iceberg:
Fixed high memory usage in compaction of tables with a large number of position delete files.
Improved performance and reduced memory usage when optimizing and collecting statistics on tables with a large number of files.
Fixed a rare issue where compaction could duplicate data
Fixed a bug where deleted jobs failed to remove intermediate files that were no longer needed.
⬆️ Enhancements
: Newly discovered tables in the source database will be created immediately in the target. Additionally, empty tables and columns within the target table will be created. Currently, this feature is limited to source tables, and and target tables.
You can now decide what should be the schema strategy on column type mismatch, using the ON_COLUMN_TYPE_MISMATCH key - ignore or add a new column for the new type
Iceberg: Now you can configure Iceberg table properties in replication jobs:
We added support for converting equality deletes to position deletes files, instead of fully re-writing data files during compactions
When the partition field is missing or null, the row will be written into a null partition
🔧 Bug Fixes
Hive Tables: Fixed the issue when setting an end time for a sync job to a point earlier than the last processed time and then dropping the job, could cause delays in other jobs reading from the same table.