Create a data source
This article provides a guide on how to create different types of data sources using an API call.
POST https://api.upsolver.com/inputs
Connect to your AWS S3 Bucket.
In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage.
See: S3 connection
Field | Name | Type | Description | Optional |
bucket | Bucket | String | The Amazon S3 bucket to read from. | |
globFilePattern | Glob File Pattern | String | The pattern for files to ingest. | |
datePattern | Date Pattern | String | The date pattern in the file name/folder structure (e.g. yyyy/MM/dd/HH/mm). The date format specification must be set according to Java DateTimeFormatter format. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
softRetention | Soft Retention | Boolean | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
prefix | Folder | String | If the data resides in a sub folder within the defined cloud storage, specify this folder. | + |
startExecutionFrom | Start Ingestion From | String (ISO-8601) | The time from which to ingest the data from. Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested. | + |
workspaces | Workspaces | String[] | The workspaces attached to this data source. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickS3StorageInputRequest",
"bucket" : "bucket",
"globFilePattern" : "globFilePattern",
"datePattern" : "yyyy/MM/dd/HH/mm",
"contentType" : {
"clazz" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Amazon S3 Data Source",
"description" : "Description of first Amazon
S3 data source"
},
"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
curl
-X POST
-H "content-type: application/json"
-H "Authorization: token"
-d '{"clazz":"QuickS3StorageInputRequest",
"bucket": "upsolver-tutorials-orders",
"globFilePattern": "*",
"datePattern": "yyyy/MM/dd/HH",
"prefix": "data/",
"contentType":
{
"clazz": "JsonContentType"
},
"compression":
{
"clazz": "AutoDetectCompression"
},
"displayData":
{
"name": "API test data source",
"description": "Description of first Amazon S3 data source"},
"softRetention":false
}'
"https://your-api.upsolver.com/inputs/"
Connect to your AWS S3 Bucket.
In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage.
See: S3 connection
Field | Name | Type | Description | Optional |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
sourceStorage | S3 Connection | String | The cloud storage to ingest files from. | |
datePattern | Date Pattern | String | The date pattern in the file name/folder structure. For example: yyyy/MM/dd/HH/mm. The date format specification must be set according to Java DateTimeFormatter format. | |
fileMatchPattern | File Name Pattern | FileNameMatcher | The file name pattern for the files to ingest. If all the files in the specified folders are relevant, specify All. The pattern given is matched against the file path starting from the bucket specified. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
computeEnvironment | Compute Cluster | String | | |
destinationStorage | Target Storage | String | The data and metadata files for this data source will be stored in this storage. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
interval | Interval | Int (Minutes) | The sliding interval to wait for data. | |
prefix | Folder | String | If the data resides in a sub folder within the defined cloud storage, specify this folder. | + |
startExecutionFrom | Start Ingestion From | String (ISO-8601) | The time from which to ingest the data from. Files from before this time (based on the provided date pattern) are ignored. If you leave this field empty all files are ingested. | + |
retention | Retention | Int (Minutes) | The retention period for the data. | + |
dataDumpDate | Data Dump Date | String (ISO-8601) | The date that the data starts. | + |
maxDelay | Max Delay | Int (Minutes) | The maximum delay to consider the data, that is, any data that arrives delayed by more than the max delay is filtered out. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "CloudStorageInputRequest",
"displayData" : {
"name" : "First Amazon S3 Data Source",
"description" : "Description of first Amazon
S3 data source"
},
"sourceStorage" : "aa302f0a-e6ee-44aa-aa38-e28f1ff455f7",
"datePattern" : "yyyy/MM/dd/HH/mm",
"fileMatchPattern" : {
"clazz" : "AllMatcher"
},
"contentType" : {
"type" : "JsonContentType"
,
"computeEnvironment" : "53b314af-ffab-419a-9c2c-56032c6ef4c0",
"destinationStorage" : "e019c0fe-bb80-4cf1-bc7b-aee579d8672e",
"compression" : {
"clazz" : "AutoDetectCompression"
},
"interval" : 120
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.
A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream.
See: Kinesis connection
Field | Name | Type | Description | Optional |
region | Region | Region | Your AWS region. | |
streamName | Stream | String | The name of the relevant Kinesis stream. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
softRetention | Soft Retention | Boolean | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
workspaces | Workspaces | String[] | The workspaces attached to this data source. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickKinesisInputRequest",
"region" : "region",
"streamName" : "streamName",
"contentType" : {
"type" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Amazon Kinesis Stream Data
Source",
"description" : "Description of first Amazon
Kinesis Stream data source"
},
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to your Amazon Kinesis. Upsolver can read events from your Amazon Kinesis, according to the stream you define.
A prerequisite for defining an Amazon Kinesis stream connection is providing Upsolver with the appropriate credentials for reading from your Amazon Kinesis stream.
See: Kinesis connection
Field | Name | Type | Description | Optional |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
kinesisConnection | Kinesis Connection | String | The AWS credentials to connect to Kinesis. | |
streamName | Stream | String | The name of the relevant Kinesis stream. | |
readFromStart | Read From Start | String | The time from which to ingest the data from. Messages from before this time will be ignored. If you leave this field empty all messages are ingested. | |
computeEnvironment | Compute Cluster | String | | |
connectionPointer | Target Storage | String | The data and metadata files for this data source will be stored in this storage. | |
isOnline | Real Time Statistics | Boolean | Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed. | |
shards | Shards | Int | How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic. | |
parallelism | Parallelism | Int | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
retention | Retention | Int (Minutes) | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever. | + |
endExecutionAt | End Read At | String (ISO-8601) | If configured, stop reading after this date. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KinesisInputRequest",
"displayData" : {
"name" : "First Amazon Kinesis Stream Data
Source",
"description" : "Description of first Amazon
Kinesis Stream data source"
},
"contentType" : {
"type" : "JsonContentType"
},
"kinesisConnection" : "dcfc39c1-c458-4ec5-87f1-0c7f437ea17e",
"streamName" : "streamName",
"readFromStart" : true,
"computeEnvironment" : "fc2a356d-c3ae-4756-a4a3-1b15158df5e7",
"connectionPointer" : "4e3787cd-daac-42ea-8dd4-642c516a4a31",
"isOnline" : true,
"shards" : 1,
"parallelism" : 1,
"compression" : {
"clazz" : "AutoDetectCompression"
}
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to your AWS S3 Bucket using SQS Notifications.
You will need to configure SQS Notifications from your S3 Bucket and open permissions to read and delete messages from the SQS Queue to the same access key and secret key you entered to give Upsolver permissions to read from the S3 Bucket.
See: S3 over SQS connection
Field | Name | Type | Description | Optional |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
sourceStorage | Source Storage | String | The cloud storage to ingest files from. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
computeEnvironment | Compute Cluster | String | | |
destinationStorage | Target Storage | String | The data and metadata files for this data source will be stored in this storage. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
softRetention | Soft Retention | Boolean | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
executionParallelism | Parallelism | Int | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source. | |
prefix | Prefix | String | The prefix of the files or directories. To filter a specific directory, add a trailing /. | + |
suffix | Suffix | String | The suffix of the files to read. | + |
startExecutionFrom | Start Ingestion From | String (ISO-8601) | The time from which to ingest the data from. Messages from before this time will be ignored. If you leave this field empty all messages are ingested. | + |
endExecutionAt | End Read At | String (ISO-8601) | If configured, stop reading after this date. | + |
retention | Retention | Int (Minutes) | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "S3OverSQSInputRequest",
"displayData" : {
"name" : "First S3 Over SQS Data Source",
"description" : "Description of first S3
Over SQS data source"
},
"sourceStorage" : "6ee598b6-4928-4b07-b532-83a79464e5bb",
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "46c2945e-5a46-4d93-9e21-5a85653c28c5",
"destinationStorage" : "2a08601c-52d2-425e-8d2c-324dbcea5858",
"compression" : {
"clazz" : "AutoDetectCompression"
},
"softRetention" : true,
"executionParallelism" : 1
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.
A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster.
See: Kafka connection
Field | Name | Type | Description | Optional |
kafkaHosts | Kafka Hosts | String | The Kafka hosts separated with commas (e.g. foo:9092,bar:9092) | |
topicName | Kafka Topic | String | The Kafka topic to ingest the data from. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
softRetention | Soft Retention | Boolean | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
maybekafkaVersion | Kafka Version | KafkaVersion | The version of the Kafka Servers. If unsure, use 0.10.x.x. | + |
workspaces | Workspaces | String[] | The workspaces attached to this data source. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "QuickKafkaInputRequest",
"kafkaHosts" : "kafkaHosts",
"topicName" : "topicName",
"contentType" : {
"type" : "JsonContentType"
},
"compression" : {
"clazz" : "AutoDetectCompression"
},
"displayData" : {
"name" : "First Kafka Data Source",
"description" : "Description of first
Kafka data source"
},
"softRetention" : true
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to any topic on your Kafka Servers. Upsolver can read events from your Kafka cluster from the specified Kafka topic.
A prerequisite for defining a Kafka stream connection is providing Upsolver with the appropriate credentials for reading from your Kafka cluster.
See: Kafka connection
Field | Name | Type | Description | Optional |
displayData.name | Name | String | The data source name. | |
displayData.description | Description | String | The data source description. | |
kafkaVersion | Kafka Version | KafkaVersion | The version of the Kafka Servers. If unsure, use 0.10.x.x. | |
kafkaHosts | Kafka Hosts | String | The Kafka hosts separated with commas. For example: foo:9092,bar:9092 | |
topicName | Kafka Topic | String | The Kafka topic to ingest the data from. | |
readFromStart | Read From Start | Boolean | Whether to read the data from the start of the topic or to begin from the end. | |
contentType | Content Format | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
computeEnvironment | Compute Cluster | String | | |
connectionPointer | Target Storage | String | The data and metadata files for this data source will be stored in this storage. | |
softRetention | Soft Retention | Boolean | A setting that prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
shards | Shards | Int | How many readers to use in parallel to read the stream. A recommended value would be to increase it by 1 for every 70 MB/s sent to your topic. | |
executionParallelism | Execution Parallelism | Int | The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases and be no more than the number of shards used to read the data from the source. | |
isOnline | Real Time Statistics | Boolean | Calculate this data source's statistics in real time directly from the input stream if a real time cluster is deployed. | |
useSsl | Use SSL | Boolean | Set this to true if your connection requires SSL. Contact us to ensure that your SSL certificate is supported. | |
storeRawData | Store Raw Data | Boolean | Store an additional copy of the data in its original format. | |
compression | Compression | Compression | The compression in the data (Zip, GZip, Snappy, SnappyUnframed, Tar, or None). | |
consumerProperties | Kafka Consumer Properties | String | Extra properties for Kafka Consumer. | + |
retention | Retention | Int (Minutes) | A retention period for the data in Upsolver. After this period of time passes, the data is deleted forever. | + |
endExecutionAt | End Read At | String (ISO-8601) | If configured, stop reading after this date. | + |
workspaces | Workspaces | String[] | The workspaces attached to this data source. | + |
curl -X POST -H "content-type: application/json"
-H "Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "KafkaInputRequest",
"displayData" : {
"name" : "First Kafka Data Source",
"description" : "Description of first
Kafka data source"
},
"kafkaVersion" : "0.10.x.x",
"kafkaHosts" : "kafkaHosts",
"topicName" : "topicName",
"readFromStart" : true,
"contentType" : {
"type" : "JsonContentType"
},
"computeEnvironment" : "5d8b6e27-6004-48b2-8dc1-1db9f25880cb",
"connectionPointer" : "36a1d237-1ff0-4574-972b-b071482f3d08",
"softRetention" : true,
"shards" : 1,
"executionParallelism" : 1,
"isOnline" : true,
"useSsl" : true,
"storeRawData" : true,
"compression" : {
"clazz" : "AutoDetectCompression"
}
}' "https://api.upsolver.com/api/v1/data-source/"
Connect to your Azure Blob storage container.
In order for Upsolver to read events directly from your cloud storage, files should be partitioned by date and time (which defines the folder structure in the cloud storage).
A prerequisite for defining a cloud storage data source is providing Upsolver with the appropriate credentials for reading from your cloud storage.
See: Azure Blob storage connection
Field | Name | Type | Description |