Uploading user-provided certificates
In order to successfully connect to Kafka or other third parties, you may need to upload the required certificates to the Upsolver servers by running a patch HTTP request for each cluster you wish to use with your connection.
Note that the instructions below were build for a Linux-based system with HTTPie installed.
If you are working with Windows, you can use a Linux mimicking program to use the below examples, adjust the example to your current system or reach out to Upsolver support and send the certificates for us to update it for you.
The following is an example of how to upload your keystore and truststore files to connect to Kafka, but it can be easily adjusted to upload other files as well. The files you require and the Kafka properties you need to use might be different than the ones shown in this example.
The files need to be uploaded to the API cluster and the Data updates that the data source will be using, it's recommended to upload the certificates to all clusters.
The API cluster will use the certificates when testing the connection when creating a new connection. The data updates cluster will use the certificate to run the reading tasks.
There are two request types to upload the files:
1.
ModifyServerFiles - Can be used to upload multiple files at once. Replaces existing files, only the files in the latest upload will be saved, and all previously uploaded files will be removed.
2. ModifyServerFile - Can be used to upload a single file with each request. Does not remove the previously uploaded files.The name must be Unique, uploading a new file with an existing name will replace the previously uploaded file using both scripts.
First, run one of the following requests for the API and Data Updates servers:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "<Unique Name>", "path": "<Unique path>", "content": $file1 }, { name: "<Unique Name>", "path": "<Unique path>", "content": $file2 } ] }' --arg file1 $(cat <Local file path>/<file name> | base64) --arg file2 $(cat <Local file path>/<file name> | base64) |
http PATCH "https://api.upsolver.com/environments/<SERVER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"
The first line of the request creates a JSON array
serverFiles which contains the name, path, and content of the files you are uploading.The
name and the path referenced are the name of the certificate and the path the file is written to within the server, it is also the path that should be provided when using this file to establish a connection. Replace <Unique Name> with a unique name for the certificate and the <Unique path> with a unique path for the file within the server. The
content of the file is passed through with as an argument with --arg. The <Local file path> represents the path to the file you are uploading on your local computer, The <File name> represents the name of the file you'd like to upload, note to use the full path to the file including the file's extension.This example uploads two files to the server, but the
serverFiles array elements can be adjusted to upload either one or more files.Finally, provide the
<SERVER_ID> , as well as the <API_TOKEN> and the <ORG_ID>.An example for a final request:
echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 } }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/12345678-1234-1234-1234-12345678/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: 12345678-1234-1234-1234-12345678"
Note that running the ModifyServerFiles request overrides any files that may have been previously uploaded to the server.
To upload a single file without overriding any existing ones, run the requests instead:
echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "<Unique Name>", "path": "<Unique path>", "content": $file1 } }' --arg file1 $(cat <Local file path>/<File name> | base64) |
http PATCH "https://api.upsolver.com/environments/<SERVER_ID>/" "Authorization: <API_TOKEN>" "X-Api-Impersonate-Organization: <ORG_ID>"
The first line of the request creates a JSON array
serverFile which contains the name, path, and content of the file you are uploading.The
name and the path referenced are the name of the certificate and the path the file is written to within the server, it is also the path that should be provided when using this file to establish a connection. Replace <Unique Name> with a unique name for the certificate and the <Unique path> with a unique path for the file within the server. The
content of the file is passed through as an argument with --arg. The <Local file path> represents the path to the file you are uploading on your local computer, The <File name> represents the name of the file you'd like to upload, note to use the full path to the file including the file's extension.Note: The name parameter must be unique across different files. Using the same name will replace the existing file with that name.
Finally, provide the
<SERVER_ID> , as well as the <API_TOKEN> and the <ORG_ID>.Once the certificates have been uploaded, roll the modified cluster to apply the changes.
If you followed this example using Kafka, you should now set the following as your consumer properties for your Kafka data sources:
security.protocol=SSL
ssl.truststore.location=<Unique path>
ssl.keystore.location=<Unique path>
ssl.keystore.password=<PASSWORD>
ssl.truststore.password=<PASSWORD>
Note: the above Kafka properties configuration is an example, the final configuration will depend on the security setting you are using in Kafka. More information cam be found in the Kafka connection documentation.
For existing Kafka data sources, you can update the consumer properties by going to Properties > Advanced > Kafka Consumer Properties.
Last modified 2mo ago