LogoLogo
OverviewQuickstartsHow To GuidesReferenceArticlesSupport
Quickstarts
Quickstarts
  • Quickstarts
  • DATA INGESTION WIZARD
    • Using the Wizard
      • Source Set-up
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Cloud
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
      • Target Set-up
        • Amazon Redshift
        • AWS Glue Data Catalog
        • ClickHouse
        • Polaris Catalog
        • Snowflake
      • Job Configuration
        • Job Configuration
        • Job Configuration for CDC
      • Review and Run Job
  • CONNECTORS
    • Connectors
      • Amazon Kinesis
      • Amazon Redshift
      • Amazon S3
      • Apache Kafka
      • AWS Glue Data Catalog
      • ClickHouse
      • Confluent Cloud
      • Elasticsearch
      • Microsoft SQL Server
      • MongoDB
      • MySQL
      • Polaris Catalog
      • PostgreSQL
      • Snowflake
  • JOBS
    • Ingestion
      • Job Basics
        • Ingest to a Staging Table
        • Output to a Target Table
      • Stream and File Sources
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Kafka
      • CDC Sources
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
    • Transformation
      • Updating Data
        • Upsert Data to a Target Table
        • Delete Data from a Target Table
        • Aggregate and Output Data
        • Join Two Data Streams
      • Data Targets
        • Output to Amazon Athena
        • Output to Amazon Redshift
        • Output to Amazon S3
        • Output to Elasticsearch
        • Output to Snowflake
  • APACHE ICEBERG
    • Optimize Your Iceberg Tables
    • Install the Iceberg Table Analyzer
Powered by GitBook
On this page
  • Step 1 - Connect to Amazon Kinesis
  • Create a new connection
  • Use an existing connection
  • Step 2 - Select a stream to ingest
  • Step 3 - Check that events are read successfully
  1. DATA INGESTION WIZARD
  2. Using the Wizard
  3. Source Set-up

Amazon Kinesis

Follow these steps to use Amazon Kinesis as your source.

Last updated 11 months ago

Step 1 - Connect to Amazon Kinesis

Create a new connection

Click Create a new connection, if it is not already selected. Then select the AWS region where your Kinesis is hosted from the pre-populated list.

For the Authentication Method, select either the Role-based or the AccessKey/SecretKey option:

Role-based

Upsolver recommends that you use Role-based access.

  • To define the correct permissions for the role, follow the guide to create an IAM policy

  • If your Kinesis runs on a different AWS account than the one running Upsolver, you need to create trust between the role and the account running Upsolver. To create a trusted AWS ROLE and find your External Id, follow the guide.

AccessKey/SecretKey

To create your Access key id and Secret access key, follow the .

When you have entered your authentication information, click Test Connection.

Use an existing connection

By default, if you have already created a connection, Upsolver selects Use an existing connection, and your Kinesis connection is populated in the list.

For organizations with multiple connections, select the source connection you want to use.

Step 2 - Select a stream to ingest

When the connection is established, a list of streams will appear in the select list, Select a stream to ingest.

Then choose the file type you want to ingest in the Select the source events content type list, or leave it on the default value of Automatic.

Step 3 - Check that events are read successfully

As soon as you select a stream, Upsolver will attempt to load a sample of the events.

If Upsolver did not load any sample events, try the following:

  1. Verify that Kinesis has events.

Select a that matches the type of your stream.

content type
AWS Account and Access Keys guide
Kinesis access configuration
Role-Based AWS Credentials
Create a new connection to your Amazon Kinesis source to ingest your streaming data.
Select an existing Amazon Kinesis connection to use as your source to ingest data from.