LogoLogo
OverviewQuickstartsHow To GuidesReferenceArticlesSupport
How To Guides
How To Guides
  • How To Guides
  • SETUP
    • Deploy Upsolver on AWS
      • Deployment Guide
      • AWS Role Permissions
      • VPC Peering Guide
      • Role-Based AWS Credentials
    • Enable API Integration
    • Install the Upsolver CLI
  • CONNECTORS
    • Create Connections
      • Amazon Kinesis
      • Amazon Redshift
      • Amazon S3
      • Apache Kafka
      • AWS Glue Data Catalog
      • ClickHouse
      • Confluent Cloud
      • Elasticsearch
      • Microsoft SQL Server
      • MongoDB
      • MySQL
      • PostgreSQL
      • Snowflake
      • Tabular
    • Configure Access
      • Amazon Kinesis
      • Amazon S3
      • Apache Kafka
      • AWS Glue Data Catalog
      • Confluent Kafka
    • Enable CDC
      • Microsoft SQL Server
      • MongoDB
      • MySQL
      • PostgreSQL
  • JOBS
    • Basics
      • Real-time Data Ingestion — Amazon Kinesis to ClickHouse
      • Real-time Data Ingestion — Amazon S3 to Amazon Athena
      • Real-time Data Ingestion — Apache Kafka to Amazon Athena
      • Real-time Data Ingestion — Apache Kafka to Snowflake
    • Advanced Use Cases
      • Build a Data Lakehouse
      • Enriching Data - Amazon S3 to ClickHouse
      • Joining Data — Amazon S3 to Amazon Athena
      • Upserting Data — Amazon S3 to Amazon Athena
      • Aggregating Data — Amazon S3 to Amazon Athena
      • Managing Data Quality - Ingesting Data with Expectations
    • Database Replication
      • Replicate CDC Data into Snowflake
      • Replicate CDC Data to Multiple Targets in Snowflake
      • Ingest Your Microsoft SQL Server CDC Data to Snowflake
      • Ingest Your MongoDB CDC Data to Snowflake
      • Handle PostgreSQL TOAST Values
    • VPC Flow Logs
      • Data Ingestion — VPC Flow Logs
      • Data Analytics — VPC Flow Logs
    • Job Monitoring
      • Export Metrics to a Third-Party System
    • Data Observability
      • Observe Data with Datasets
  • DATA
    • Query Upsolver Iceberg Tables from Snowflake
  • APACHE ICEBERG
    • Analyze Your Iceberg Tables Using the Upsolver CLI
    • Optimize Your Iceberg Tables
Powered by GitBook
On this page
  1. CONNECTORS
  2. Configure Access

Amazon Kinesis

This section covers how to configure an Amazon Kinesis connection in Upsolver to read from an Amazon Kinesis stream managed by another AWS account.

In order to create an IAM role and a trust relationship, please visit the documentation, and use the following documentation to create the IAM policy with the required Amazon Kinesis permissions.

Create an IAM policy with Amazon Kinesis permissions

Upsolver requires the following permissions:

ListStreams
ListShards
GetShardIterator
GetRecords
DescribeStream

When creating an Amazon Kinesis connection in Upsolver, you can include the STREAM_DISPLAY_FILTERS property, which allows you to restrict the Amazon Kinesis streams that users can see in the Upsolver navigation tree. However, this does not limit the user’s ability to read objects; that is still managed by the permissions in the IAM role attached to the connection. This property is not to be used to restrict access to data.

If the STREAMS_DISPLAY_FILTERS property is omitted, Upsolver attempts to list all streams in the account. The available streams are listed in the Upsolver navigation tree to make it easier for users to discover datasets. For this to function correctly, Upsolver requires the IAM policy to include kinesis:ListStreams.

If STREAMS_DISPLAY_FILTERS is included when creating the Amazon Kinesis connection, you do not need to add the kinesis:ListStreams permission.

When creating the IAM policy, add the policy statements that allow Upsolver to access the data in your Amazon Kinesis:

{
	"Version": "2012-10-17",
	"Statement": [
    	{
        	"Effect": "Allow",
        	"Action": [
            	"kinesis:ListStreams"
        	],
        	"Resource": "*"
    	},
    	{
        	"Effect": "Allow",
        	"Action": [
                "kinesis:Get*",
                "kinesis:DescribeStream"
            ],
            "Resource": [
                "arn:aws:kinesis:us-east-1:111122223333:stream/stream1"
            ]
    	}
	]
}

Learn More

To learn more about setting permissions for Amazon Kinesis, please read the guide from AWS.

Role Based AWS Credentials
Controlling Access to Amazon Kinesis Data Streams Resources Using IAM