LogoLogo
OverviewQuickstartsHow To GuidesReferenceArticlesSupport
Quickstarts
Quickstarts
  • Quickstarts
  • DATA INGESTION WIZARD
    • Using the Wizard
      • Source Set-up
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Cloud
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
      • Target Set-up
        • Amazon Redshift
        • AWS Glue Data Catalog
        • ClickHouse
        • Polaris Catalog
        • Snowflake
      • Job Configuration
        • Job Configuration
        • Job Configuration for CDC
      • Review and Run Job
  • CONNECTORS
    • Connectors
      • Amazon Kinesis
      • Amazon Redshift
      • Amazon S3
      • Apache Kafka
      • AWS Glue Data Catalog
      • ClickHouse
      • Confluent Cloud
      • Elasticsearch
      • Microsoft SQL Server
      • MongoDB
      • MySQL
      • Polaris Catalog
      • PostgreSQL
      • Snowflake
  • JOBS
    • Ingestion
      • Job Basics
        • Ingest to a Staging Table
        • Output to a Target Table
      • Stream and File Sources
        • Amazon Kinesis
        • Amazon S3
        • Apache Kafka
        • Confluent Kafka
      • CDC Sources
        • Microsoft SQL Server
        • MongoDB
        • MySQL
        • PostgreSQL
    • Transformation
      • Updating Data
        • Upsert Data to a Target Table
        • Delete Data from a Target Table
        • Aggregate and Output Data
        • Join Two Data Streams
      • Data Targets
        • Output to Amazon Athena
        • Output to Amazon Redshift
        • Output to Amazon S3
        • Output to Elasticsearch
        • Output to Snowflake
  • APACHE ICEBERG
    • Optimize Your Iceberg Tables
    • Install the Iceberg Table Analyzer
Powered by GitBook
On this page
  • Step 1 - Connect to Kafka
  • Create a new connection
  • Use an existing connection
  • Step 2 - Select a topic to ingest
  • Step 3 - Check that events are read successfully
  1. DATA INGESTION WIZARD
  2. Using the Wizard
  3. Source Set-up

Apache Kafka

Follow these steps to use Kafka as your source.

Last updated 11 months ago

Step 1 - Connect to Kafka

Create a new connection

Click Create a new connection, if it is not already selected. In Kafka host[s], enter a single host in the format of hostname:port, or a list of Kafka hosts in the format of hostname1:port, hostname2:port.

Ensure the host address is accessible to Upsolver.

When using Upsolver's sandbox cloud, Kafka needs public access. If you have already integrated Upsolver into your environment and you want to avoid exposing the Kafka cluster and save AWS transfer costs, you can .

Additional connection options may require configuration in order to provide the correct credentials to read from your Kafka cluster.

For a standard connection, in Consumer Properties, use the following format:

bootstrap.servers = HOST:PORT
security.protocol = SASL_SSL
sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule
     required username = "API_KEY"   password = "SECRET";
ssl.endpoint.identification.algorithm = https
sasl.mechanism = PLAIN

where:

  • The the boostrap.servers value is same value as Kafka host[s] you entered above.

  • The username and password values are the API_KEY, along with the corresponding secret configured on Kafka to allow access to the Kafka cluster.

To learn more about consumer properties visit .

In the Name your connection, type in the name you want to give to this connection.

Use an existing connection

By default, if you have already created a connection, Upsolver selects Use an existing connection, and your Apache Kafka connection is populated in the list.

For organizations with multiple connections, select the source connection you want to use.

Step 2 - Select a topic to ingest

When the connection is established, use the Select a topic for ingestion list to choose the topic you want to ingest. Optionally, you can then Select the source events content type to change the default from Automatic, to a specific file type such as CSV, JSON, or Avro.

Step 3 - Check that events are read successfully

As soon as you select a topic, Upsolver will attempt to load a sample of the events.

If Upsolver did not load any sample events, try the following:

  1. Verify that Kafka has events.

In order for Upsolver to connect to your Kafka cluster using SSL, follow these steps to .

Select a that matches the content type of your topic.

content type
Consumer Configuration
Create a new connection to Apache Kafka and select a topic to ingest.
Select your existing Apache Kafka connection to use for your ingestion job. Then select your topic.
peer the VPC where the Kafka cluster runs to the VPC running Upsolver
Configure SSL for your Kafka connection