General output properties

This page provides a list of the properties that can be found under the general section of output properties.

Properties have been sorted as:

Note: Properties listed below may not apply to all outputs.

General

Property

Description

Data Sources

The required data sources to base the output on.

Override Window Size

If enabled, this feature can be set to minutes, hours, or days, or Infinite Override Size.

For example, if you wish to create an hourly output that will aggregate data for the last day, then the output interval should be 1 hour, but override window size should be 1 day.

This option is useful when used in combination with upserts. For example, if you want the latest total count of events per user, set the Override Window to infinite; in this case, every time an event arises for a user, the updated total count for the user is upserted, and the old record is deleted.

Reporting Tags

Tags that can be sent to external monitoring systems.

Path

The desired path for data to be exported to.

If left empty, a system generated unique path will be used.

Storage

Property

Description

Date Format

How output will be partitioned by time (e.g. yyyy/MM/dd).

Output Format

File format for output.

<Storage> Connection

Connection string for storage location.

Tabular

Whether or not the output is tabular.

Hive Metastores

Property

Description

Partition Size

The output partition size (Hourly, Daily, Weekly, or Yearly).

Compression

The compression applied to the output (None, GZip, or Snappy).

Run Compactions

Whether to run compactions on the output.

Excluded Partitions

The partitions to exclude. You can import, export, or edit the list.

S3 Storage

S3 storage location where Upsolver stores the intermediate bulk files before uploading.

Databases

Property

Description

Skip Failed Files

If enabled, when a load fails the manifest of that load will be skipped. The skipped manifest will be saved aside for manual re-processing once the copy error has been fixed.

Fail on Write Error

If enabled, any error while copying to the target will cause the load to fail. If disabled the same behavior will occur after 100K errors (The max allowed by Redshift)

<Database> Connection

Connection string of database connected to this output.

Schema

Schema of this output.

Table Name

Name of output table.

Intermediate Storage Location

Where Upsolver stores the intermediate bulk files before uploading.

Lookup Tables

Property

Description

Window

Window of time to keep in lookup table.

Omit Key Columns

Whether or not to store the key columns in the lookup table. This saves space if enabled, but prevents iteration from returning the key columns.

Real Time

Load data into this lookup table directly from the input stream using a real time cluster if one is deployed.

Kinesis

Property

Description

Kinesis Connection

Connection string for Kinesis stream.

Stream Name

Name of stream output is written to.

Intermediate Storage

Location where Upsolver stores the intermediate bulk files before uploading.

Tabular

Whether or not the output is tabular.

Elasticsearch

Property

Description

Delete Indices

Delete indices from ElasticSearch based on retention period.

Connection

Elasticsearch connection string.

Index Name

Name of index output is written to.

Index Partition Size

Size of index partition.

S3 Connection

S3 storage location where Upsolver stores the intermediate bulk files before uploading.

Kafka

Property

Description

Kafka Hosts

Comma-separated list of host:port pairs required to connect to Kafka cluster.

Topic Name

Name of topic this output is written to.

Additional Kafka Properties

Tags that can be sent to external monitoring systems.

Intermediate Storage

Location where Upsolver stores the intermediate bulk files before uploading.

Tabular

Whether or not the output is tabular.