July 2023

Upsolver new features, enhancements, and bug fixes for July 2023.

Weekly Software Updates

The following software updates were released in July:


⬆️ Enhancements

  • Snowflake Jobs:

    • Support setting COMMIT_INTERVAL. This allows configuring different intervals for processing the job and for writing to Snowflake.

  • CDC Jobs:

    • (Beta) Support replication data from Microsoft SQL Server using change data capture.


⬆️ Enhancements

  • Elasticsearch Jobs:

    • Support setting routing (_routing) by using the new property ROUTING_FIELD_NAME.

    • Added a new option to the INDEX_PARTITION_SIZE property: NONE. This allows us to write to a single index name.

  • CDC Jobs:

    • Add the ability to snapshot multiple tables at once (Microsoft SQL Server, MySQL, PostgreSQL).

🔧 Bug Fixes

  • Snowflake Jobs:

    • Fixed an issue with a custom insert/update expression causing the job to fail if the field is also mapped in the select statement.

    • On auto-managed tables, Upsolver will not create an extra column if the following conversion happens:

      • Original column is double and has a value of type long.

      • Original column is a timestamp and has a value of type date.

      • Original column is a varchar.

      • Original column is Variant In all other cases, we will create an extra column with the new type as the column name suffix. For example: if a column col was of type bigint and got a Double value, we will create a column COL_DOUBLE in the Snowflake table.

    • Fixed a delay in Materialized View on Job List/Index page.


🔧 Bug Fixes

  • Fixed tree on fields containing dots, e.g. turning {"a\.b": 1} to {"a.b": 1}.

  • Snowflake Jobs:

    • Changed the file format to copy from Avro to JSON. This fixed an issue when ingesting records with sub-fields that have special characters.


⬆️ Enhancements

  • Added VALUE_INDEX_IN_ROW() - this function receives an element of an array of records and returns the 1-based index of the element position (incrementing regardless of whether the array is nested). Null values are not counted.

  • Added VALUE_INDEX_IN_ARRAY() - this function receives an element of an array of records and returns the 1-based index of the element position (index resets to 1 for each sub-array). Null values are not counted.

  • Ingestion wizard:

    • Support was added for creating a PostgreSQL heartbeat table within the wizard.

🔧 Bug Fixes

  • Reduce the frequency of metadata queries to Snowflake in order to reduce the cost of COMPUTE SERVICES charged by Snowflake.

  • [BREAKING CHANGE] Fixed RECORD_TO_JSON on fields containing dots, e.g. turning {"a.b": 1} to {"a\.b": 1}.


New Features

  • New UUID() function returns a unique identifier (UUID) string.

  • Sign-out is now available from the main screen.

⬆️ Enhancements

  • PostgreSQL CDC: ignore rows from the heartbeat table.

  • Upgraded Debezium version from 2.1.3 to 2.2.1.

  • Ingestion wizard:

    • Supports compute cluster input (in case the organization has more than one compute cluster).

    • Supports basic expectation.

  • The cluster version appears in the UI on the Clusters page.

  • Snowflake table: show variant column statistics on field level.

🔧 Bug Fixes

  • Fixed the conversion of float to double to preserve the perceived semantic value in CDC sources and in data sources that get Avro or Parquet.