Amazon Kinesis
This page describes how to create and maintain connections to your Amazon Kinesis stream.
In order to read and work with your Kinesis data in SQLake, you should first create a connection to your Kinesis stream that provides Upsolver with the necessary credentials to access your data.
A Kinesis connection can be created as simply as follows:
CREATE KINESIS CONNECTION my_kinesis_connection
REGION = 'us-east-1';
Note that the connection in this example is created based on the default credentials derived from Upsolver's integration with your AWS account.
The following example also creates a Kinesis connection but additionally configures credentials by providing a specific role as well as establishes the connection as read-only:
CREATE KINESIS CONNECTION my_kinesis_connection
AWS_ROLE = 'arn:aws:iam::123456789012:role/upsolver-sqlake-role'
REGION = 'us-east-1'
READ_ONLY = true
MAX_WRITERS = 22
STREAM_DISPLAY_FILTERS = ('stream1', 'stream2')
COMMENT = 'kinesis connection example';
To establish a connection with specific permissions, you can configure the
AWS_ROLE and EXTERNAL_ID options like in the example above or you can configure theAWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY options to provide the credentials to read from your stream.Additionally, all connections have read and write permissions by default but you can easily create a connection with only read access by setting
READ_ONLY to true.You can also limit the list of streams displayed within your catalog by providing the list of streams to display using
STREAM_DISPLAY_FILTER[S].For the full list of connection options with syntax and detailed descriptions, see Kinesis connection with SQL
Once you've created your connection, you are ready to move onto the next step of building your data pipeline: reading your data into SQLake with an ingestion job.
Certain connection options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing Kinesis connection rather than creating a new one.
For example, take the Kinesis connection we created previously based on default credentials:
CREATE KINESIS CONNECTION my_kinesis_connection
REGION = 'us-east-1';
To change the connection's permissions and keep everything else the same without having to create an entirely new connection, you can run the following command:
ALTER KINESIS CONNECTION my_kinesis_connection
SET AWS_ROLE = 'arn:aws:iam::123456789012:role/new-sqlake-role';
Note that some options such as
READ_ONLY and REGION cannot be altered once the connection has been created.If you no longer need a certain connection, you can easily drop it with the following SQL command:
DROP CONNECTION my_kinesis_connection;
However, note that if there are existing tables or jobs that are dependent upon the connection in question, the connection cannot be deleted.
Last modified 14d ago