Apache Kafka

This page describes how to create and maintain connections to your Apache Kafka cluster.

To read and work with data from your Apache Kafka topic in Upsolver, you should first create a connection to your Kafka cluster that provides Upsolver with the necessary credentials to access your data. This guides shows you how.

Create a Kafka connection

Simple example

A Kafka connection can be created as simply as follows:

CREATE KAFKA CONNECTION my_kafka_connection
    HOSTS = ('foo:9092', 'bar:9092');

However, you may find that additional connection options may need to be configured in order to provide the proper credentials to read from your Kafka cluster.

Full example

The following example also creates a Kafka connection but sets additional options such as CONSUMER_PROPERTIES to ensure the configurations necessary to ingest data from this example cluster are provided:

CREATE KAFKA CONNECTION my_kafka_connection
    HOSTS = ('foo:9092', 'bar:9092')
    CONSUMER_PROPERTIES = 
       'bootstrap.servers = HOST:PORT
       security.protocol = SASL_SSL
       sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule   
             required username = "API_KEY"   password = "SECRET";
       ssl.endpoint.identification.algorithm = https
       sasl.mechanism = PLAIN'
    VERSION = CURRENT
    REQUIRE_STATIC_IP = true
    SSL = false
    TOPIC_DISPLAY_FILTERS = ('topic1', 'topic2')
    COMMENT = 'My new Kafka connection';

Alter a Kafka connection

Some connection options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing Kafka connection rather than create a new one.

For example, take the simple Kafka connection we created previously:

CREATE KAFKA CONNECTION my_kafka_connection
    HOSTS = ('foo:9092', 'bar:9092');

If it turns out certain consumer properties need to be configured in order from our cluster, instead of having to create an entirely new connection, you can run the following command:

ALTER KAFKA CONNECTION my_kafka_connection
    SET CONSUMER_PROPERTIES = 
       'bootstrap.servers = HOST:PORT
       security.protocol = SASL_SSL
       sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule   
             required username = "API_KEY"   password = "SECRET";
       ssl.endpoint.identification.algorithm = https
       sasl.mechanism = PLAIN'; 

Note that some options such as VERSION cannot be altered once the connection has been created.

Drop a Kafka connection

If you no longer need a connection, you can easily drop it with the following SQL command:

DROP CONNECTION my_kafka_connection; 

However, note that if there are existing tables or jobs that are dependent upon the connection in question, the connection cannot be deleted.

---

Learn More

To discover which connection options are mutable, and to learn more about the options, please see the SQL command reference for Apache Kafka.