Data lake tables

This page describes the job options for writing data to the data lake.

These job options are used when writing to data lake tables or Upsolver-managed tables.
Tables created within SQLake using your metastore connection are considered Upsolver-managed tables. Note that these tables can still be queried externally. For example, you can create a Glue Catalog table within SQLake, and this table can be queried within SQLake itself or when you go to your Athena console.
Syntax
[ ADD_MISSING_COLUMNS = { TRUE | FALSE } ]
[ AGGREGATION_PARALLELISM = <integer> ]
[ COMMENT = '<comment>' ]
[ COMPUTE_CLUSTER = <cluster_identifier> ]
[ END_AT = { NOW | timestamp } ]
[ FLATTEN_PATHS = (<array_path> [, ...]) ]
[ RUN_INTERVAL = <integer> { MINUTE[S] | HOUR[S] | DAY[S] } ]
[ RUN_PARALLELISM = <integer> ]
[ START_FROM = { NOW | BEGINNING | timestamp } ]
Jump to
Data lake table options:
​ADD_MISSING_COLUMNS​
​FLATTEN_PATHS​
General job options:
​AGGREGATION_PARALLELISM​
​COMMENT​
​COMPUTE_CLUSTER​
​END_AT​
​RUN_INTERVAL​
​RUN_PARALLELISM​
​START_FROM​
Job options
ADD_MISSING_COLUMNS
Type: Boolean
Default: false
(Optional) When true, columns that don't exist in the target table are added automatically when encountered.
When false, you cannot do SELECT * within the SELECT statement of your transformation job.
FLATTEN_PATHS
Type: Array<String>
Default: ()
(Optional) Allows specifying arrays that will be used to flatten the output rows. Please see the Flattening arrays guide for details and examples.

Amazon S3

Elasticsearch

Last modified 15d ago

Data lake tables

Syntax

Jump to

Job options

`ADD_MISSING_COLUMNS`

`FLATTEN_PATHS`