API content formats

This page provides a guide on how to configure different content formats in Upsolver using API calls.

Auto-Detect
Example
{
	"clazz" : "AutoDetectContentType"
}
Avro
Example
{
	"clazz" : "AvroContentType"
}
Parquet
Example
{
	"clazz" : "ParquetContentType"
}
ORC
Example
{
	"clazz" : "OrcContentType"
}
JSON
JSON data. Multiple JSONs can be read from a single file/record by appending them with optional whitespace in between.
Fields
Field
Name
Type
Description
Optional
nestedJsonPaths
Nested Json Paths
[][]
Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.
+
nestedJsons
Nested Jsons
NestedPath[]
Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.
+
splitRootArray
Split Root Array
Boolean
If the root object is an array, it can either be parsed as separate events, or as a single event which contains only an array.
+
keepOriginalNestedJsonString
Keep Original Nested Json String
Boolean
When using the nestedJsonPaths parameter, the original JSON string can optionally be kept in addition to the parsed value. If this is selected, the parsed data is stored in a record with the suffix _parsed.
+
storeJsonAsString
Store Json As String
Boolean
Whether to store the JSON in native format in a separate field.
+
Example
{
	"clazz" : "JsonContentType"
}
CSV
Fields
Field
Name
Type
Description
Optional
inferTypes
Infer Types
Boolean
Whether or not to infer types. If not selected, Upsolver will read all fields as strings.
​
header
Header
String
If applicable, the header of the file. If you only add details for one column, additional columns will be labeled as overflow columns.
​
delimiter
Delimiter
Char
The delimiter between columns of data.
+
nullValue
Null Value
String
If applicable, the default null value in the data.
+
nestedJsons
Nested Jsons
NestedPath[]
Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.
+
keepOriginalNestedJsonString
Keep Original Nested Json String
Boolean
When using the nestedJsonPaths parameter, the original JSON string can optionally be kept in addition to the parsed value. If this is selected, the parsed data will be stored in a record with the suffix _parsed.
+
Example
{
	"clazz" : "CsvContentType",
	"inferTypes" : true,
	"header" : "header1,header2,header2"
}
TSV
Fields
Field
Name
Type
Description
Optional
inferTypes
Infer Types
Boolean
Whether or not to infer types. If not selected, Upsolver will read all fields as strings.
​
header
Header
String
If applicable, the header of the file. If you only add details for one column, additional columns will be labeled as overflow columns.
​
nestedJsons
Nested Jsons
NestedPath[]
Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.
+
keepOriginalNestedJsonString
Keep Original Nested Json String
Boolean
When using the nestedJsonPaths parameter, the original JSON string can optionally be kept in addition to the parsed value. If this is selected, the parsed data will be stored in a record with the suffix _parsed.
+
Example
{
	"clazz" : "TsvContentType",
	"inferTypes" : true,
	"header" : "header1,header2,header2"
}
x-www-form-urlencoded
Fields
Field
Name
Type
Description
Optional
inferTypes
Infer Types
Boolean
Whether or not to infer types. If not selected, Upsolver will read all fields as strings.
​
Example
{
	"clazz" : "WWWFormUrlEncodedType",
	"inferTypes" : true,
}
Protobuf
Fields
Field
Name
Type
Description
Optional
schemaFiles
Schema Files
SchemaFile[]
​
​
mainFile
Main File
String
The main file from the list of selected schema files.
​
messageType
Message Type
String
The message type.
​
bytesParsers
Bytes Parsers
BytesParser[]
BytesParsers can be used to define special parsing behavior for 'bytes' fields in the schema.
​
bytesParsers.path
Path
String
The path to the field that should use this parser.
​
bytesParsers.parserSchema
Parser Schema
String
The schema that the parser will use. 
In CSV and TSV formats, it should be a comma-delimited list of the field names in the CSV/TSV rows. 
In the Avro format, it should be the Avro schema used to decode the bytes of the inner field.
+
bytesParsers.schemaType
Schema Type
String
The type of parser to use. Supported types are JSON, CSv, TSF, or Avro.
+
Example
{
	"clazz" : "ProtobufContentType",
	"schemaFiles" : "schemaFiles",
	"mainFile" : "mainFile",
	"messageType" : "messageType",
	"bytesParsers" : "bytesParsers"
}
Avro-record
Individual Avro records without the framing or schema.
Fields
Field
Name
Type
Description
Optional
schema
Schema
String
The Avro schema used to decode the messages. Note that the behavior is undefined if the schema does not match the data.
​
bytesParsers
Bytes Parsers
BytesParser[]
BytesParsers can be used to define special parsing behavior for 'bytes' fields in the schema.
+
bytesParsers.path
Path
String
The path to the field that should use this parser.
​
bytesParsers.parserSchema
Parser Schema
String
The schema that the parser will use. In CSv and TSV formats, it should be a comma-delimited list of the field names in the CSV/TSV rows. 
In the Avro format, it should be the Avro schema used to decode the bytes of the inner field.
+
bytesParsers.schemaType
Schema Type
String
The type of parser to use. Supported types are JSON, CSV, TSV, or Avro.
+
Example
{
	"clazz" : "AvroRecordContentType",
	"schema" : "{ \"type\": \"record\", \"name\": 
\"root\", \"fields\": [ {\"name\": \"value\", \"type\": 
\"string\" } ] }"
}
Avro w/ Schema Registry
Individual Avro records with schema provided by Schema Registry.
Fields
Field
Name
Type
Description
Optional
schemaRegistryUrl
Schema Registry Url
String
The URL of getting schema where {id} is the ID of the schema.
​
schemaRegistryFormat
Schema Registry Format
SchemaRegistryFormat
The strategy used to encode the schema version into every message.
+
bytesParsers
Bytes Parsers
BytesParser[]
BytesParsers can be used to define special parsing behavior for 'bytes' fields in the schema.
​
bytesParsers.path
Path
String
The path to the field that should use this parser.
​
bytesParsers.parserSchema
Parser Schema
String
The schema that the parser will use. 
In CSV and TSV formats, it should be a comma-delimited list of the field names in the CSV/TSV rows. In the Avro format, it should be the Avro schema used to decode the bytes of the inner field.
+
bytesParsers.schemaType
Schema Type
String
The type of parser to use. Supported types are JSON, CSV, TSV, or Avro.
+
Example
{
	"clazz" : "AvroSchemaRegistryContentType",
	"schemaRegistryUrl" : "schemaRegistryUrl"
}
XML
XML data. Multiple XMLs can be read from a single file/record by appending them with optional whitespace in between.
Fields
Field
Name
Type
Description
Optional
storeRootAsString
Store Root As String
Boolean
Whether to store the XML root in native format in a separate field.
+
Example
{
	"clazz" : "XmlContentType",
}

Modify a data source

Next - Guide for developers

CI/CD on Upsolver

Last updated 5 months ago

Field	Name	Type	Description	Optional
nestedJsonPaths	Nested Json Paths	[][]	Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.	+
nestedJsons	Nested Jsons	NestedPath[]	Paths to string fields that contain a "stringified" JSON that should be parsed into the schema. Each such path is represented as an array of the path parts.	+
splitRootArray	Split Root Array	Boolean	If the root object is an array, it can either be parsed as separate events, or as a single event which contains only an array.	+
keepOriginalNestedJsonString	Keep Original Nested Json String	Boolean	When using the nestedJsonPaths parameter, the original JSON string can optionally be kept in addition to the parsed value. If this is selected, the parsed data is stored in a record with the suffix `_parsed`.	+
storeJsonAsString	Store Json As String	Boolean	Whether to store the JSON in native format in a separate field.	+