Uploading user-provided certificates

In order to successfully connect to Kafka or other third parties, you may need to upload the required certificates to the Upsolver servers by running a patch HTTP request for each cluster you wish to use with your connection.

Note that the instructions below were build for a Linux-based system with HTTPie installed.

If you are working with Windows, you can use a Linux mimicking program to use the below examples, adjust the example to your current system or reach out to Upsolver support and send the certificates for us to update it for you.

How to upload files

The following is an example of how to upload your keystore and truststore files to connect to Kafka, but it can be easily adjusted to upload other files as well. The files you require and the Kafka properties you need to use might be different than the ones shown in this example.

The files need to be uploaded to the API cluster and the Data updates that the data source will be using, it's recommended to upload the certificates to all clusters. The API cluster will use the certificates when testing the connection when creating a new connection. The data updates cluster will use the certificate to run the reading tasks.

There are two request types to upload the files: 1. ModifyServerFiles - Can be used to upload multiple files at once. Replaces existing files, only the files in the latest upload will be saved, and all previously uploaded files will be removed. 2. ModifyServerFile - Can be used to upload a single file with each request. Does not remove the previously uploaded files.

The name must be Unique, uploading a new file with an existing name will replace the previously uploaded file using both scripts.

Step 1

First, run one of the following requests for the API and Data Updates servers:

echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: [ { name: "<Unique Name>", "path": "<Unique path>", "content": $file1 }, { name: "<Unique Name>", "path": "<Unique path>", "content": $file2 } ]  }' --arg file1 $(cat <Local file path>/<file name> | base64) --arg file2 $(cat <Local file path>/<file name> | base64) |
http PATCH "https://api.upsolver.com/environments/<SERVER_ID>/" "Authorization: <API_TOKEN>" "x-user-organization: <ORG_ID>"

The first line of the request creates a JSON array serverFiles which contains the name, path, and content of the files you are uploading.

The name and the path referenced are the name of the certificate and the path the file is written to within the server, it is also the path that should be provided when using this file to establish a connection. Replace <Unique Name> with a unique name for the certificate and the <Unique path> with a unique path for the file within the server.

The content of the file is passed through with as an argument with --arg. The <Local file path> represents the path to the file you are uploading on your local computer, The <File name> represents the name of the file you'd like to upload, note to use the full path to the file including the file's extension.

This example uploads two files to the server, but the serverFiles array elements can be adjusted to upload either one or more files.

Finally, provide the <SERVER_ID> , as well as the <API_TOKEN> and the <ORG_ID>.

An example for a final request:

echo {} | jq '{ clazz: "ModifyServerFiles", serverFiles: { name: "cert.pem", "path": "/opt/cert.pem", "content": $file1 }  }' --arg file1 $(cat ~/Downloads/cert.pem | base64) |
http PATCH "https://api.upsolver.com/environments/12345678-1234-1234-1234-12345678/" "Authorization: $(cat ~/.upsolver/token)" "X-Api-Impersonate-Organization: 12345678-1234-1234-1234-12345678"

To learn how to generate an API token, see: Upsolver REST API

How to find your <API_SERVER_ID>

1. Go to the Clusters page and click on the PrivateAPI cluster.

2. Click on Copy API endpoint in the upper right corner.

3. The API server ID can be found within the endpoint as follows: https://api-<API_SERVER_ID>.upsolver.com/

How to find your <CLUSTER_ID>

1. Go to the Clusters page and click on the cluster you wish to upload your files to.

2. Once you are on that specific cluster's page, the cluster ID can be found within the page's URL as follows: https://app.upsolver.com/environments/view/<CLUSTER_ID>

How to find your <ORG_ID>

1. Navigate to the SAML Integration page by clicking More > SAML.

2. Your org id can be found at the end of the Audience URI as upsolver://organization/<ORG_ID>

Note that running the ModifyServerFiles request overrides any files that may have been previously uploaded to the server.

To upload a single file without overriding any existing ones, run the requests instead:

echo {} | jq '{ clazz: "ModifyServerFile", serverFile: { name: "<Unique Name>", "path": "<Unique path>", "content": $file1 }  }' --arg file1 $(cat <Local file path>/<File name> | base64) |
http PATCH "https://api.upsolver.com/environments/<SERVER_ID>/" "Authorization: <API_TOKEN>" "X-Api-Impersonate-Organization: <ORG_ID>"

The first line of the request creates a JSON array serverFile which contains the name, path, and content of the file you are uploading.

The name and the path referenced are the name of the certificate and the path the file is written to within the server, it is also the path that should be provided when using this file to establish a connection. Replace <Unique Name> with a unique name for the certificate and the <Unique path> with a unique path for the file within the server.

The content of the file is passed through as an argument with --arg. The <Local file path> represents the path to the file you are uploading on your local computer, The <File name> represents the name of the file you'd like to upload, note to use the full path to the file including the file's extension.

Note: The name parameter must be unique across different files. Using the same name will replace the existing file with that name.

Finally, provide the <SERVER_ID> , as well as the <API_TOKEN> and the <ORG_ID>.

Step 2

Once the certificates have been uploaded, roll the modified cluster to apply the changes.

How to roll a cluster

1. Go to the Clusters page and select the cluster you would like to roll.

2. In the upper righthand corner, click on the three dots next to Stop.

3. Select Roll from the list of options that appears.

Step 3

If you followed this example using Kafka, you should now set the following as your consumer properties for your Kafka data sources:

security.protocol=SSL
ssl.truststore.location=<Unique path>
ssl.keystore.location=<Unique path>
ssl.keystore.password=<PASSWORD>
ssl.truststore.password=<PASSWORD>

Note: the above Kafka properties configuration is an example, the final configuration will depend on the security setting you are using in Kafka. More information cam be found in the Kafka connection documentation.

For existing Kafka data sources, you can update the consumer properties by going to Properties > Advanced > Kafka Consumer Properties.

For information on creating a Kafka connection see: Kafka connection