Amazon Kinesis

This page describes how to create and maintain connections to your Amazon Kinesis stream.

In order to read and work with your Kinesis data in SQLake, you should first create a connection to your Kinesis stream that provides Upsolver with the necessary credentials to access your data.

Create a Kinesis connection

Simple example

A Kinesis connection can be created as simply as follows:

CREATE KINESIS CONNECTION my_kinesis_connection
    REGION = 'us-east-1';

Note that the connection in this example is created based on the default credentials derived from Upsolver's integration with your AWS account.

Full example

The following example also creates a Kinesis connection but additionally configures credentials by providing a specific role as well as establishes the connection as read-only:

CREATE KINESIS CONNECTION my_kinesis_connection
    AWS_ROLE = 'arn:aws:iam::123456789012:role/upsolver-sqlake-role'
    REGION = 'us-east-1'
    READ_ONLY = true
    MAX_WRITERS = 22
    STREAM_DISPLAY_FILTERS = ('stream1', 'stream2')
    COMMENT = 'kinesis connection example';

To establish a connection with specific permissions, you can configure the AWS_ROLE and EXTERNAL_ID options like in the example above or you can configure theAWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY options to provide the credentials to read from your stream.

Additionally, all connections have read and write permissions by default but you can easily create a connection with only read access by setting READ_ONLY to true.

You can also limit the list of streams displayed within your catalog by providing the list of streams to display using STREAM_DISPLAY_FILTER[S].

For the full list of connection options with syntax and detailed descriptions, see Kinesis connection with SQL

Once you've created your connection, you are ready to move onto the next step of building your data pipeline: reading your data into SQLake with an ingestion job.

Alter a Kinesis connection

Certain connection options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing Kinesis connection rather than creating a new one.

For example, take the Kinesis connection we created previously based on default credentials:

CREATE KINESIS CONNECTION my_kinesis_connection
    REGION = 'us-east-1';

To change the connection's permissions and keep everything else the same without having to create an entirely new connection, you can run the following command:

ALTER KINESIS CONNECTION my_kinesis_connection
    SET AWS_ROLE = 'arn:aws:iam::123456789012:role/new-sqlake-role'; 

Note that some options such as READ_ONLY and REGION cannot be altered once the connection has been created.

To check which specific connection options are mutable, see: Kinesis connection with SQL

Drop a Kinesis connection

If you no longer need a certain connection, you can easily drop it with the following SQL command:

DROP CONNECTION my_kinesis_connection; 

However, note that if there are existing tables or jobs that are dependent upon the connection in question, the connection cannot be deleted.

For more details, see: DROP CONNECTION