The Amazon Kinesis Data Source is a streaming data source which will ingest data directly from the provided Amazon Kinesis Stream into Upsolver. For the purposes of this guide we will assume you already have a Kinesis stream up and running. If you don't, please read Amazon's documentation for Kinesis Streams and set one up.
Creating A New Amazon Kinesis Data Source
- On the Data Sources page click Add New Data Source.
- Select Amazon Kinesis from the list.
- Fill in a name for your Data Source.
- In the Content Format drop down menu select the format of the messages inside the Kinesis stream.
- Fill in the Name of your Kinesis stream in the Stream field.
- Create an IAM user with permission to read from the Kinesis stream you would like to ingest.
- Fill in the Access Key and Secret Key fields with the respective values from this IAM user.
- Select the AWS region in which the Kinesis stream exists in the Region drop down menu.
- In the Start Execution From time field, select the time you would like Upsolver to start ingesting events in the stream from. Events which were added to the stream before this time will be ignored.
- Select the Compute Cluster you would like this Data Source to run on. If you have not created a Compute Cluster yet, you can create one from the drop down menu.
- You can choose to change where the ingested data from this source will be saved by changing the Storage field in the Advanced section.
- Click Create to finish creating your data source.
When you are done creating your input you will be taken to the Schema tab of the Data Source Discovery page. Here you are able to inspect the data that was ingested from your Kinesis stream.