July 2023

Upsolver new features, enhancements, and bug fixes for July 2023.

Weekly Software Updates

The following software updates were released in July:

2023.07.31-02.02

⬆️ Enhancements

• Snowflake Jobs:

  • Support setting COMMIT_INTERVAL. This allows configuring different intervals for processing the job and for writing to Snowflake.

• CDC Jobs:

  • (Beta) Support replication data from Microsoft SQL Server using change data capture.

---

2023.07.27-01.59

⬆️ Enhancements

• Elasticsearch Jobs:

  • Support setting routing (_routing) by using the new property ROUTING_FIELD_NAME.

  • Added a new option to the INDEX_PARTITION_SIZE property: NONE. This allows us to write to a single index name.

• CDC Jobs:

  • Add the ability to snapshot multiple tables at once:

    • Microsoft SQL Server

    • MySQL

    • PostgreSQL

🔧 Bug Fixes

• Snowflake Jobs:

  • Fixed an issue with a custom insert/update expression causing the job to fail if the field is also mapped in the select statement.

  • On auto-managed tables, Upsolver will not create an extra column if the following conversion happens:

    • Original column is double and has a value of type long.

    • Original column is a timestamp and has a value of type date.

    • Original column is a varchar.

    • Original column is Variant In all other cases, we will create an extra column with the new type as the column name suffix. For example: if a column col was of type bigint and got a Double value, we will create a column COL_DOUBLE in the Snowflake table.

  • Fixed a delay in Materialized View on Job List/Index page.

---

2023.07.19-02.34

🔧 Bug Fixes

• Fixed tree on fields containing dots, e.g. turning {"a\.b": 1} to {"a.b": 1}.

• Snowflake Jobs:

  • Changed the file format to copy from Avro to JSON. This fixed an issue when ingesting records with sub-fields that have special characters.

---

2023.07.13-02.20

⬆️ Enhancements

• Added VALUE_INDEX_IN_ROW() - this function receives an element of an array of records and returns the 1-based index of the element position (incrementing regardless of whether the array is nested). Null values are not counted.
• Added VALUE_INDEX_IN_ARRAY() - this function receives an element of an array of records and returns the 1-based index of the element position (index resets to 1 for each sub-array). Null values are not counted.
• Ingestion wizard:

  • Support was added for creating a PostgreSQL heartbeat table within the wizard.

🔧 Bug Fixes

• Reduce the frequency of metadata queries to Snowflake in order to reduce the cost of COMPUTE SERVICES charged by Snowflake.

• [BREAKING CHANGE] Fixed RECORD_TO_JSON on fields containing dots, e.g. turning {"a.b": 1} to {"a\.b": 1}.

---

2023.07.04-14.42

✨ New Features

• New UUID() function returns a unique identifier (UUID) string.

• Sign-out is now available from the main screen.

⬆️ Enhancements

• PostgreSQL CDC: ignore rows from the heartbeat table.

• Upgraded Debezium version from 2.1.3 to 2.2.1.

• Ingestion wizard:

  • Supports compute cluster input (in case the organization has more than one compute cluster).

  • Supports basic expectation.

• The cluster version appears in the UI on the Clusters page.

• Snowflake table: show variant column statistics on field level.

🔧 Bug Fixes

• Fixed the conversion of float to double to preserve the perceived semantic value in CDC sources and in data sources that get Avro or Parquet.