Create a data source

This article provides a guide on how to create different types of data sources using an API call.

This API enables you to create a new data source. All API calls require an API token.

POST https://api.upsolver.com/inputs

Amazon S3 (Quick)

Connect to your AWS S3 Bucket.

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. See: S3 connection

Fields

Field

Name

Type

Description

Optional

bucket

Bucket

String

The Amazon S3 bucket to read from.

globFilePattern

Glob File Pattern

String

The pattern for files to ingest.

datePattern

Date Pattern

String

The date pattern in the file name/folder structure (e.g. yyyy/MM/dd/HH/mm). The date format specification must be set according to Java DateTimeFormatter format.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

prefix

Folder

String

If the data resides in a sub folder within the defined cloud storage, specify this folder.

+

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickS3StorageInputRequest",
"bucket" : "bucket",
"globFilePattern" : "globFilePattern",
"datePattern" : "yyyy/MM/dd/HH/mm",
"contentType" : {
"type" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Amazon S3 Data Source",
"description" : "Description of first Amazon
S3 data source"
},
"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Amazon S3 (Advanced)

Connect to your AWS S3 Bucket.

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. See: S3 connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

sourceStorage

S3 Connection

String

The cloud storage to ingest files from.

datePattern

Date Pattern

String

The date pattern in the file name/folder structure. For example: yyyy/MM/dd/HH/mm. The date format specification must be set according to Java DateTimeFormatter format.

fileMatchPattern

File Name Pattern

FileNameMatcher

The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from the bucket specified.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See Content Formats.

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See Adding a Compute Cluster.

destinationStorage

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

interval

Interval

Int (Minutes)

The sliding interval to wait for data.

prefix

Folder

String

If the data resides in a sub folder within the defined cloud storage, specify this folder.

+

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.

+

retention

Retention

Int (Minutes)

The retention period for the data.

+

dataDumpDate

Data Dump Date

String (ISO-8601)

The date that the data starts.

+

maxDelay

Max Delay

Int (Minutes)

The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "CloudStorageInputRequest",
"displayData" : {
"name" : "First Amazon S3 Data Source",
"description" : "Description of first Amazon
S3 data source"
},
"sourceStorage" : "aa302f0a-e6ee-44aa-aa38-e28f1ff455f7",
"datePattern" : "yyyy/MM/dd/HH/mm",
"fileMatchPattern" : {
"clazz" : "AllMatcher"
},
"contentType" : {
"type" : "JsonContentType"
,
"computeEnvironment" : "53b314af-ffab-419a-9c2c-56032c6ef4c0",
"destinationStorage" : "e019c0fe-bb80-4cf1-bc7b-aee579d8672e",
"compression" : {
"clazz" : "AutoDetectCompression"
},
"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"

Amazon Kinesis Stream (Quick)

Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.

A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream. See: Kinesis connection

Fields

Field

Name

Type

Description

Optional

region

Region

Region

Your AWS region.

streamName

Stream

String

The name of the relevant Kinesis stream.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickKinesisInputRequest",
"region" : "region",
"streamName" : "streamName",
"contentType" : {
"type" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Amazon Kinesis Stream Data
Source",
"description" : "Description of first Amazon
Kinesis Stream data source"
},
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"

Amazon Kinesis Stream (Advanced)

Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.

A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream. See: Kinesis connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

kinesisConnection

Kinesis Connection

String

The AWS credentials to connect to Kinesis.

streamName

Stream

String

The name of the relevant Kinesis stream.

readFromStart

Read From Start

String

The time from which to ingest the data from.

Messages from before this time will be ignored. If you leave this field empty all messages are ingested.

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

connectionPointer

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

isOnline

Real Time Statistics

Boolean

Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed.

shards

Shards

Int

How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.

parallelism

Parallelism

Int

The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.

+

endExecutionAt

End Read At

String (ISO-8601)

If configured, stop reading after this date.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KinesisInputRequest",
"displayData" : {
"name" : "First Amazon Kinesis Stream Data
Source",
"description" : "Description of first Amazon
Kinesis Stream data source"
},
"contentType" : {
"type" : "JsonContentType"
},
"kinesisConnection" : "dcfc39c1-c458-4ec5-87f1-0c7f437ea17e",
"streamName" : "streamName",
"readFromStart" : true,
"computeEnvironment" : "fc2a356d-c3ae-4756-a4a3-1b15158df5e7",
"connectionPointer" : "4e3787cd-daac-42ea-8dd4-642c516a4a31",
"isOnline" : true,
"shards" : 1,
"parallelism" : 1,
"compression" : {
"clazz" : "AutoDetectCompression"
}
}' "https://api.upsolver.com/api/v1/data-source/"

Amazon S3 over SQS

Connect to your AWS S3 Bucket using SQS Notifications.

You will need to configure SQS Notifications from your S3 Bucket and open permissions to read and delete messages from the SQS Queue to the same access key and secret key you entered to give Upsolver permissions to read from the S3 Bucket. See: S3 over SQS connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

sourceStorage

Source Storage

String

The cloud storage to ingest files from.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

destinationStorage

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

executionParallelism

Parallelism

Int

The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.

prefix

Prefix

String

The prefix of the files or directories. To filter a specific directory, add a trailing /.

+

suffix

Suffix

String

The suffix of the files to read.

+

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Messages from before this time will be ignored. If you leave this field empty all messages are ingested.

+

endExecutionAt

End Read At

String (ISO-8601)

If configured, stop reading after this date.

+

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "S3OverSQSInputRequest",
"displayData" : {
"name" : "First S3 Over SQS Data Source",
"description" : "Description of first S3
Over SQS data source"
},
"sourceStorage" : "6ee598b6-4928-4b07-b532-83a79464e5bb",
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "46c2945e-5a46-4d93-9e21-5a85653c28c5",
"destinationStorage" : "2a08601c-52d2-425e-8d2c-324dbcea5858",
"compression" : {
"clazz" : "AutoDetectCompression"
},
"softRetention" : true,
"executionParallelism" : 1
}' "https://api.upsolver.com/api/v1/data-source/"

Apache Kafka (Quick)

Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.

A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster. See: Kafka connection

Fields

Field

Name

Type

Description

Optional

kafkaHosts

Kafka Hosts

String

The Kafka hosts separated with commas (e.g. foo:9092,bar:9092)

topicName

Kafka Topic

String

The Kafka topic to ingest the data from.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

maybekafkaVersion

Kafka Version

KafkaVersion

The version of the Kafka Servers. If unsure, use 0.10.x.x.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickKafkaInputRequest",
"kafkaHosts" : "kafkaHosts",
"topicName" : "topicName",
"contentType" : {
"type" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Kafka Data Source",
"description" : "Description of first
Kafka data source"
},
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"

Apache Kafka (Advanced)

Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.

A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster. See: Kafka connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

kafkaVersion

Kafka Version

KafkaVersion

The version of the Kafka Servers. If unsure, use 0.10.x.x.

kafkaHosts

Kafka Hosts

String

The Kafka hosts separated with commas. For example: foo:9092,bar:9092

topicName

Kafka Topic

String

The Kafka topic to ingest the data from.

readFromStart

Read From Start

Boolean

Whether to read the data from the start of the topic or to begin from the end.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

connectionPointer

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

shards

Shards

Int

How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.

executionParallelism

Execution Parallelism

Int

The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source.

isOnline

Real Time Statistics

Boolean

Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed.

useSsl

Use SSL

Boolean

Set this to true if your connection requires SSL. Contact us to ensure that your SSL certificate is supported.

storeRawData

Store Raw Data

Boolean

Store an additional copy of the data in its original format.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

consumerProperties

Kafka Consumer Properties

String

Extra properties for Kafka Consumer.

+

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever.

+

endExecutionAt

End Read At

String (ISO-8601)

If configured, stop reading after this date.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json"
-H "Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KafkaInputRequest",
"displayData" : {
"name" : "First Kafka Data Source",
"description" : "Description of first
Kafka data source"
},
"kafkaVersion" : "0.10.x.x",
"kafkaHosts" : "kafkaHosts",
"topicName" : "topicName",
"readFromStart" : true,
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "5d8b6e27-6004-48b2-8dc1-1db9f25880cb",
"connectionPointer" : "36a1d237-1ff0-4574-972b-b071482f3d08",
"softRetention" : true,
"shards" : 1,
"executionParallelism" : 1,
"isOnline" : true,
"useSsl" : true,
"storeRawData" : true,
"compression" : {
"clazz" : "AutoDetectCompression"
}
}' "https://api.upsolver.com/api/v1/data-source/"

Azure Blob storage

Connect to your Azure Blob storage container.

In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).

A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. See: Azure Blob storage connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

sourceStorage

Azure Blob Storage Connection

String

The cloud storage to ingest files from.

datePattern

Date Pattern

String

The date pattern in the file name/folder structure (e.g. yyyy/MM/dd/HH/mm). The date format specification must be set according to Java DateTimeFormatter format.

fileMatchPattern

File Name Pattern

FileNameMatcher

The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from the storage container specified.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

destinationStorage

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

interval

Interval

Int (Minutes)

The sliding interval to wait for data.

prefix

Folder

String

If the data resides in a sub folder within the defined cloud storage, specify this folder.

+

initialLoadConfiguration

Initial Load Configuration

InitialLoadConfiguration

If you have initial data, enter in a prefix and regex pattern to list the relevant data and select the required files.

+

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.

+

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.

+

dataDumpDate

Data Dump Date

String (ISO-8601)

The date that the data starts.

+

maxDelay

Max Delay

Int (Minutes)

The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "AzureBlobStorageInputRequest",
"displayData" : {
"name" : "First Azure Blob Storage Data
Source",
"description" : "Description of first Azure
Blob Storage data source"
},
"sourceStorage" : "2da1c848-9c44-4b4e-a226-4ffebf0d9c49",
"datePattern" : "yyyy/MM/dd/HH/mm",
"fileMatchPattern" : {
"clazz" : "AllMatcher"
},
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "1bce245a-0c76-4fe7-acb5-6f2bf4b64d8f",
"destinationStorage" : "004974e0-e384-466e-8ac4-7429c30614e3",
"softRetention" : true,
"compression" : {
"clazz" : "AutoDetectCompression"
},
"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"

Google Cloud Storage

Connect to your Google Storage Bucket.

In order for Upsolver to read events directly from your cloud strage, files should be partitioned by date and time (which defines the folder structure in the cloud storage)

A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage. See: Google Storage connection

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

sourceStorage

Google Storage Connection

String

The cloud storage to ingest files from.

datePattern

Date Pattern

String

The date pattern in the file name/folder structure (e.g. yyyy/MM/dd/HH/mm). The date format specification must be set according to Java DateTimeFormatter format.

fileMatchPattern

File Name Pattern

FileNameMatcher

The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from storage source specified.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

destinationStorage

Target Storage

String

The data and metadata files for this data source will be stored in this storage.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

compression

Compression

Compression

The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None).

interval

Interval

Int (Minutes)

The sliding interval to wait for data.

prefix

Folder

String

If the data resides in a sub folder within the defined cloud storage, specify this folder.

+

initialLoadConfiguration

Initial Load Configuration

InitialLoadConfiguration

If you have initial data, enter in a prefix and regex pattern to list the relevant data and select the required files.

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.

+

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.

+

dataDumpDate

Data Dump Date

String (ISO-8601)

The date that the data starts.

+

maxDelay

Max Delay

Int (Minutes)

The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "GoogleCloudStorageInputRequest",
"displayData" : {
"name" : "First Google Cloud Storage
Data Source",
"description" : "Description of first Google
Cloud Storage data source"
},
"sourceStorage" : "0a617e70-c2b2-4870-a908-f4c864c4961f",
"datePattern" : "yyyy/MM/dd/HH/mm",
"fileMatchPattern" : {
"clazz" : "AllMatcher"
},
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "d4f6ee48-0782-4789-b661-4e19679a541a",
"destinationStorage" : "8ddd7397-e1ed-4b21-a9cb-91aa923ea165",
"softRetention" : true,
"compression" : {
"clazz" : "AutoDetectCompression"
},
"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"

Kinesis-backed HTTP

Connect your stream using HTTP requests from any source.

Once you create the connection, you will be provided with an HTTP endpoint. Upsolver receives the data as POST, with the data in the body, and the data is stored in a Kinesis Stream until processed by Upsolver.

Headers sent with the request are also ingested as part of the stream, so metadata can be added to the request header.

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

ingestionEnvironment

Ingestion Cluster

String

storageConnection

Target Storage

String

The data and metadata files for this Data Source will be stored in this storage.

kinesisConnection

Kinesis Connection

String

The data and metadata files for this Data Source will be stored in this storage.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.

+

startExecutionFrom

Start Ingestion From

String (ISO-8601)

The time from which to ingest the data from.

Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested.

+

endExecutionAt

End Read At

String (ISO-8601)

If configured, stop reading after this date.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KinesisHttpInputRequest",
"displayData" : {
"name" : "First Kinesis backed HTTP Data
Source",
"description" : "Description of first Kinesis
backed HTTP data source"
},
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "bdb99c82-4874-4e41-b735-090cc53af71e",
"ingestionEnvironment" : "c71ba158-8022-494e-84e1-e0623533ef1c",
"storageConnection" : "f4376c08-1b13-495b-904f-768effa818da",
"kinesisConnection" : "44a93a4b-5083-4f26-90cb-c3ea7284e3d3",
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"

HTTP

Connect your stream using HTTP requests from any source.

Once you create the connection, you will be provided with an HTTP endpoint. Upsolver receives the data as POST, with the data in the body.

Headers sent with the request are also ingested as part of the stream, so metadata can be added to the request header.

Fields

Field

Name

Type

Description

Optional

displayData.name

Name

String

The data source name.

displayData.description

Description

String

The data source description.

contentType

Content Format

ContentType

The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.

For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.

Messages can be compressed, Upsolver automatically detects the compression type.

Supported compression types are: Zip, GZip, Snappy and None.

See: Content formats

computeEnvironment

Compute Cluster

String

The compute cluster to run the calculation on. See: Compute cluster

connectionPointer

Target Storage

String

The data and metadata files for this Data Source are stored in this storage.

softRetention

Soft Retention

Boolean

A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

shards

Shards

Int

How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic.

retention

Retention

Int (Minutes)

A retention period for the data in Upsolver. After this amount of time elapsed the data will be deleted forever.

+

endExecutionAt

End Read At

String (ISO-8601)

If configured, stop reading after this date.

+

workspaces

Workspaces

String[]

The workspaces attached to this data source.

+

Example

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KinesisHttpInputRequest",
"displayData" : {
"name" : "First Kinesis backed HTTP Data
Source",
"description" : "Description of first
Kinesis backed HTTP data source"
},
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "bdb99c82-4874-4e41-b735-090cc53af71e",
"ingestionEnvironment" : "c71ba158-8022-494e-84e1-e0623533ef1c",
"storageConnection" : "f4376c08-1b13-495b-904f-768effa818da",
"kinesisConnection" : "44a93a4b-5083-4f26-90cb-c3ea7284e3d3",
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"