Configure SSL for your Kafka connection
In order for Upsolver to successfully connect to your Kafka cluster using SSL, the key and certificate files generated when SSL was deployed should be provided as part of your Upsolver cluster configuration.
This means you need to upload your self-signed certificates to our servers by running a patch HTTP request for each cluster you wish to use with your connection.
Before beginning, ensure that you have SSL authentication configured within your Kafka cluster.
Note that the instructions below only work in a Linux-based system.
If you are working with Windows, you can use a Linux mimicking program or file a ticket via the Upsolver support portal and send the certificates for us to update it for you.
First run this request for the API server:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "kafka.client.keystore.jks", "path": "/opt/kafka.client.keystore.jks", "content": $file1 }, { name: "kafka.client.truststore.jks", "path": "/opt/kafka.client.truststore.jks", "content": $file2 } ] }' --arg file1 $(cat /<FILE_PATH>/kafka.client.keystore.jks | base64) --arg file2 $(cat /<FILE_PATH>/kafka.client.truststore.jks | base64) |
http PATCH "https://api.upsolver.com/environments/<API_SERVER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"
Then run this request for the cluster you wish to upload the files to:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "kafka.client.keystore.jks", "path": "/opt/kafka.client.keystore.jks", "content": $file1 }, { name: "kafka.client.truststore.jks", "path": "/opt/kafka.client.truststore.jks", "content": $file2 } ] }' --arg file1 $(cat /<FILE_PATH>/kafka.client.keystore.jks | base64) --arg file2 $(cat /<FILE_PATH>/kafka.client.truststore.jks | base64) |
http PATCH "https://api.upsolver.com/environments/<CLUSTER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"
Note that the two requests only differ in the IDs provided within the URLs for each request.
The first line of the request creates a JSON array
serverFiles
which contains the path
and content
of the file you are uploading.The
path
referenced within the array itself is the path the file is written to within the server; it is also the path that should be provided when using this file to establish a connection.The
content
of the file is passed through with as an argument with --arg
. Here <FILE_PATH>
represents the path to the file you are uploading on your local computer.This example uploads two files to the server, but the
serverFiles
array elements can be adjusted to upload either one or more files.Finally, you should also provide your
<API_SERVER_ID>
and <CLUSTER_ID>
, as well as your<API_TOKEN>
and your <ORG_ID>
.Note that running this request overrides any files that may have been uploaded previously.
To upload a single file without overriding any existing ones, run these request instead:
echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 } }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/<API_SERVER_ID>/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: <ORG_ID>"
echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 } }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/<CLUSTER_ID>/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: <ORG_ID>"
Once the certificates have been uploaded, roll the modified cluster to apply the changes.
To use your key and certificate files to connect Upsolver to your Kafka cluster, you should provide the paths to your uploaded files as part of the properties in the
CREATE KAFKA CONNECTION
command.To allow the connection to be used for reading data, the key store and trust store locations should be configured as
CONSUMER_PROPERTIES
. CREATE KAFKA CONNECTION my_kafka_connection
HOSTS = (('<bootstrap_server_1>:<port_number>','<bootstrap_server_2>:<port_number>'))
CONSUMER_PROPERTIES = 'security.protocol=SSL
ssl.truststore.location=/opt/kafka.client.truststore.jks
ssl.keystore.location=/opt/kafka.client.keystore.jks
ssl.keystore.password=<PASSWORD>
ssl.key.password=<PASSWORD>';
Last modified 4mo ago