Data formats

This page goes over the different data formats available in Upsolver and, if applicable, their corresponding configuration options.

When creating a data source, the content format is typically auto detected, but you can manually select a format and specify additional details, as required.

The following content formats are supported:

• Avro

• Parquet

• ORC

• JSON

• CSV

• TSV

• x-www-form-urlencoded

• Protobuf

• Avro-record

  • For Amazon S3, Azure Blob Storage, Google Cloud Storage or File Upload data sources.

• Avro-record

  • For Amazon Kinesis, Kafka, S3 Over SQS.

• Avro with Schema Registry

• XML

Avro

The schema is auto detected.

Parquet

The schema is auto detected.

ORC

The schema is auto detected.

JSON

The schema is auto detected. The body should contain the message itself, which should not be url-encoded.

Store JSON as String

(Optional) Whether to store the JSON in native format in a separate field.

CSV

Infer Types

(Optional) Select whether to auto detect the types. If not selected, Upsolver will read all fields as strings.

Header

(Optional) The content header.

If you only add details for one column, additional columns will be labeled as overflow columns.

Delimiter

(Optional) The delimiter between columns of data.

Null Value

(Optional) The value to be interpreted as a null value.

TSV

Infer Types

(Optional) Whether to auto detect the types.

Header

(Optional) The content header.

If you only add details for one column, additional columns will be labeled as overflow columns.

x-www-form-urlencoded

The body should contain the message itself, which should not be url-encoded.

Protobuf

Schema files

(Optional) Click Select to choose the required schema files.

Main File

(Optional) The main file from the list of selected schema files.

Message Type

(Optional) The message type.

Avro-record

For Amazon S3, Azure Blob Storage, Google Cloud Storage or File Upload data sources.

Avro Record Schema

(Optional) The Avro record schema.

Avro-record

For Amazon Kinesis, Kafka, S3 Over SQS.

Avro Record Schema

The Avro record schema.

Bytes Parsers

(Optional) A parser for JSON-format schemas.

Additional Schemas

(Optional) Additional Avro record schemas.

Avro with Schema Registry

Schema Registry URL

(Optional) The URL to the schema registry.

XML

The schema is auto-detected.