General output properties

Properties have been sorted as:
​General 
properties most outputs have
​Storage​
e.g. Amazon S3, Google Storage, etc.
​Hive Metastores​
e.g. Amazon Athena, Qubole, etc.
​Databases​
e.g. Snowflake, MySQL, etc.
​Lookup Tables​
​Kinesis​
​Elasticsearch​
​Kafka​
Note: Properties listed below may not apply to all outputs.
General
Property
Description
Data Sources
The required data sources to base the output on.
Override Window Size
If enabled, this feature can be set to minutes, hours, or days, or Infinite Override Size.
For example, if you wish to create an hourly output that will aggregate data for the last day, then the output interval should be 1 hour, but override window size should be 1 day.
This option is useful when used in combination with upserts.
For example, if you want the latest total count of events per user, set the Override Window to infinite; in this case, every time an event arises for a user, the updated total count for the user is upserted, and the old record is deleted. 
Reporting Tags
Tags that can be sent to external monitoring systems.
Path
The desired path for data to be exported to.
If left empty, a system generated unique path will be used.
Storage
Property
Description
Date Format
How output will be partitioned by time (e.g. yyyy/MM/dd).
Output Format
File format for output.
<Storage> Connection
Connection string for storage location.
Tabular
Whether or not the output is tabular.
Hive Metastores
Property
Description
Partition Size
The output partition size (Hourly, Daily, Weekly, or Yearly).
Compression
The compression applied to the output (None, GZip, or Snappy).
Run Compactions
Whether to run compactions on the output.
Excluded Partitions
The partitions to exclude. You can import, export, or edit the list.
S3 Storage
S3 storage location where Upsolver stores the intermediate bulk files before uploading.
Databases
Property
Description
Skip Failed Files
If enabled, when a load fails the manifest of that load will be skipped. The skipped manifest will be saved aside for manual re-processing once the copy error has been fixed.
Fail on Write Error
If enabled, any error while copying to the target will cause the load to fail. If disabled the same behavior will occur after 100K errors (The max allowed by Redshift)
<Database> Connection
Connection string of database connected to this output.
Schema
Schema of this output.
Table Name
Name of output table.
Intermediate Storage Location
Where Upsolver stores the intermediate bulk files before uploading.
Lookup Tables
Property
Description
Window
Window of time to keep in lookup table.
Omit Key Columns
Whether or not to store the key columns in the lookup table. This saves space if enabled, but prevents iteration from returning the key columns.
Real Time
Load data into this lookup table directly from the input stream using a real time cluster if one is deployed.
Kinesis
Property
Description
Kinesis Connection
Connection string for Kinesis stream.
Stream Name
Name of stream output is written to.
Intermediate Storage
Location where Upsolver stores the intermediate bulk files before uploading.
Tabular
Whether or not the output is tabular.
Elasticsearch
Property
Description
Delete Indices
Delete indices from ElasticSearch based on retention period.
Connection
Elasticsearch connection string.
Index Name
Name of index output is written to.
Index Partition Size
Size of index partition.
S3 Connection
S3 storage location where Upsolver stores the intermediate bulk files before uploading.
Kafka
Property
Description
Kafka Hosts
Comma-separated list of host:port pairs required to connect to Kafka cluster.
Topic Name
Name of topic this output is written to.
Additional Kafka Properties
Tags that can be sent to external monitoring systems.
Intermediate Storage
Location where Upsolver stores the intermediate bulk files before uploading.
Tabular
Whether or not the output is tabular.

Output properties

Run an output

Last modified 1yr ago

Copy link

Outline