Configure SSL for your Kafka connection

In order for Upsolver to successfully connect to your Kafka cluster using SSL, the key and certificate files generated when SSL was deployed should be provided as part of your Upsolver cluster configuration.
This means you need to upload your self-signed certificates to our servers by running a patch HTTP request for each cluster you wish to use with your connection.
Prerequisites
Before beginning, ensure that you have SSL authentication configured within your Kafka cluster.
See: Encryption and authentication with SSL​
Note that the instructions below only work in a Linux-based system. 
If you are working with Windows, you can use a Linux mimicking program or file a ticket via the Upsolver support portal and send the certificates for us to update it for you.
Configure your Upsolver cluster
Step 1: Upload your files
First run this request for the API server:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "kafka.client.keystore.jks", "path": "/opt/kafka.client.keystore.jks", "content": $file1 }, { name: "kafka.client.truststore.jks", "path": "/opt/kafka.client.truststore.jks", "content": $file2 } ]  }' --arg file1 $(cat /<FILE_PATH>/kafka.client.keystore.jks | base64) --arg file2 $(cat /<FILE_PATH>/kafka.client.truststore.jks | base64) |
http PATCH "https://api.upsolver.com/environments/<API_SERVER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"
Then run this request for the cluster you wish to upload the files to:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "kafka.client.keystore.jks", "path": "/opt/kafka.client.keystore.jks", "content": $file1 }, { name: "kafka.client.truststore.jks", "path": "/opt/kafka.client.truststore.jks", "content": $file2 } ]  }' --arg file1 $(cat /<FILE_PATH>/kafka.client.keystore.jks | base64) --arg file2 $(cat /<FILE_PATH>/kafka.client.truststore.jks | base64) |
http PATCH "https://api.upsolver.com/environments/<CLUSTER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"
Note that the two requests only differ in the IDs provided within the URLs for each request.
The first line of the request creates a JSON array serverFiles which contains the path and content of the file you are uploading.
The path referenced within the array itself is the path the file is written to within the server; it is also the path that should be provided when using this file to establish a connection.
The content of the file is passed through with as an argument with --arg. Here <FILE_PATH> represents the path to the file you are uploading on your local computer.
This example uploads two files to the server, but the serverFiles array elements can be adjusted to upload either one or more files.
Finally, you should also provide your <API_SERVER_ID> and <CLUSTER_ID>, as well as your<API_TOKEN> and your <ORG_ID>.
To learn how to generate an API token, see: Upsolver REST API​
How to find your <API_SERVER_ID>
1.Go to the Clusters page and click on the PrivateAPI cluster.
2.Click on Copy API endpoint in the upper right corner.
3.The API server ID can be found within the endpoint as follows: https://api-<API_SERVER_ID>.upsolver.com/
How to find your <CLUSTER_ID>
1.Go to the Clusters page and click on the cluster you wish to upload your files to.
2.Once you are on that specific cluster's page, the cluster ID can be found within the page's URL as follows: https://app.upsolver.com/environments/view/<CLUSTER_ID>
How to find your <ORG_ID>
1.Navigate to the SAML Integration page by clicking More > SAML.
2.Your org id can be found at the end of the Audience URI as upsolver://organization/<ORG_ID>
Note that running this request overrides any files that may have been uploaded previously.
To upload a single file without overriding any existing ones, run these request instead:
echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 }  }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/<API_SERVER_ID>/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: <ORG_ID>"
echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 }  }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/<CLUSTER_ID>/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: <ORG_ID>"
Step 2: Roll your cluster
Once the certificates have been uploaded, roll the modified cluster to apply the changes.
How to roll a cluster
1.Go to the Clusters page and select the cluster you would like to roll.
2.In the upper righthand corner, click on the three dots next to Stop.
3.Select Roll from the list of options that appears.
Create your connection
To use your key and certificate files to connect Upsolver to your Kafka cluster, you should provide the paths to your uploaded files as part of the properties in the CREATE KAFKA CONNECTION command.
To allow the connection to be used for reading data, the key store and trust store locations should be configured as CONSUMER_PROPERTIES. 
Example
CREATE KAFKA CONNECTION my_kafka_connection
       HOSTS = (('<bootstrap_server_1>:<port_number>','<bootstrap_server_2>:<port_number>'))
       CONSUMER_PROPERTIES = 'security.protocol=SSL
                              ssl.truststore.location=/opt/kafka.client.truststore.jks
                              ssl.keystore.location=/opt/kafka.client.keystore.jks
                              ssl.keystore.password=<PASSWORD>
                              ssl.key.password=<PASSWORD>';
Connect to your Kafka cluster
Connect to your Kinesis stream
Last modified 4mo ago