Change log
Change log for SQLake (sqlake.upsolver.com)
Bug Fixes
- Jdbc Outputs: delete intermediate files after being written to the DB
Enhancements
- The cluster catalog is now visible in the tree
- Gather all system entities in the tree under a catalog named "System"
- Improved AS OF syntax
- Auto complete on Jobs
- Daily usage graph and report are available
- Ability to decide which query engine to use to run a select statement (Athena/Upsolver)
- Put all System entities under a catalog named "System"
- Support AS of syntax
- Information Schema: Add a table for columns
Bug Fixes
- JDBC Outputs: delete intermediate files after being written to the DB
Enhancements
- Cluster catalog is now visible in the tree
- Gather all system entities in the tree under a catalog named "System"
- Improved AS OF syntax
- Auto complete on Jobs
Bug Fixes
- Fixed Kafka batcher tasks getting stuck when reading with a wildcard topic and deleting all the topics in Kafka
Enhancements
- Show information schema catalog on the tree
- Auto Complete on Information Schema tables and columns
- Allow creating jobs from Information Schema tables
- Add support for Timestamp, Date, and Decimal types in CDC and AVRO sources
- Added support for bigserial in postgres outputs
- Support IGNORE_FIELDS for a copy from job
- Memory allocation optimizations in Lookup Table Query servers
Bug fixes
- Fixed an issue when creating a Kafka Data Source with glob pattern that doesn't match any topics would cause no response in the API.
Enhancements
- Upgrade debezium to V2.1.2
- Support transformation job to PostgreSQL
- Expose security information within the app to allow easier AWS configuration to connect your own data
- Memory allocation optimizations in Lookup Table Query servers
- Allow to alter Materialized View COMMENT.
- Display managed entities in the tree even when can't connect to Athena.
- Support ignoring fields in COPY FROM jobs by specifying IGNORE_FIELDS option.
Bug fixes
- Fixed a rare issue that can cause duplicate data to be loaded into Redshift after copy failures
Enhancements
- Use regional STS endpoints if available
- Indication on an executed statement in the editor, successs/failed
- Column appears immediately on the tree on creating transformation job
- Support
CASTexpression in the language - Renamed function TO_LONG to TO_BIGINT
- SELECT * now returns columns from joined Materialized View
- Enhancements
- Add support for
information_schemaqueries - Add support for
SKIP_VALIDATIONandSKIP_ALL_VALIDATIONoptions.- DEPRECATION:
ALLOW_EMPTY_SOURCESwill be deprecated in favor of the new options.
- Added validation to prevent explicitly mapping fields with different data type to the defined output table columns.
- Bug Fixes
- Add support for hierarchical system columns in the tree
- Enhancements
- Support using non fully qualified names for tables and materialized views
- Improve error message when trying to create a table with the same name as existing one
- Support querying without WHERE statement (Infinite Window)
- S3 Output Job: Support split files to folders
- Home page redesign
- Cluster management tab in the UI
- Bug Fixes
- Alter Cluster: Fix Alter to null not set default values
- Transformation Jobs: Fix missing columns mapping validations for partition and key columns
- API: Support join materialized view with array
- Enhancements
- Jobs monitoring will now show materialized views
- API: Join with materialized view is no longer requiring alias for the mv
- Auto completion for CREATE TABLE options
- Added a new System Table
jobs.transform_job_statethat provides a summary status of all running transform jobs. - Remove RETENTION property from all transformation jobs
- Improved query results tab
- Bug Fixes
- Fixed slow loading of Schemas under Athena connections in the tree
- Job SQL Statement in Inspection Pane doesn't omit parenthesis when they're required
- Monitoring: Fixed the 'job_name' of aggregation stages to be the original 'job_name' instead of "Output Aggregation". This means logs in the System Table 'logs.task_executions' will now have a correct 'job_name' for aggregation stages.
- Fixed
MAP_COLUMNS_BY_NAMEis not needed for S3, ElasticSearch targets
- Enhancements
- New system columns added: $source_id , $shard_number , $row_number
- Support Time Travel in joins!
- Support running a SELECT query without a FROM clause
- Support running a SELECT query reading from an Upsolver Classic Data Source
- Support running UNNEST queries
- Support selecting columns by their fully qualified name (e.g. catalog.schema.table.column)
- Support select System Columns with glob patterns
- Bug Fixes
- Fixed slow replay progress for Snowflake and PostgreSQL outputs
- Enhancements
- Event log improvements + Present informative diagram for copy/transform jobs
- Pipeline Monitoring: Expose filtered rows due to missing PK or Partition Column
- CLI: Show only message on DDL commands success
- Support using Classic Data Sources
- Enhancements
- Inviting a user to an organization is now supported
- Expose the column data type in the tree
- Improved CLI experience
- New SQLake templates added (CDC Mysql, CDC PostgreSQL, Elastic Search, Snowflake, Redshift)
- Enhancements
- Support Transformation Jobs to Elastic Search
- Enhancements
- PostgreSQL CDC: Moved TABLE_INCLUDE_LIST and COLUMN_EXCLUDE_LIST from job options to source definition
- MySQL CDC: Moved TABLE_INCLUDE_LIST and COLUMN_EXCLUDE_LIST from job options to source definition
- API token management
- Enhancements
- Support Transformation Jobs to S3
- Support Copy From PostgreSQL Jobs
- New Home Page
- Support private VPC integration
- Enhancements
- New system tables: running_tasks, failing_tasks
- Support Copy From MYSQL Jobs
- Support Transformation Jobs to Redshift
- Features
- New system tables were added:
- running_tasks
- failing_tasks
- copy_from_job_status
- Changes
- Preview is now limited to a fixed amount of input rows. Queries that are too large for preview will be aborted
- Bug Fixes
- Jobs: transformation jobs with an interval larger than one minute did not hanbdle cases where the start time or end time of the job were not fully aligned with that interval
Last modified 2d ago