Elasticsearch

This article describes how to create and maintain connections to your Elasticsearch cluster.

Before you can write your transformed data to an index in Elasticsearch, you should first establish a connection to your Elasticseach cluster.

Create an Elasticsearch connection

Simple example

An Elasticsearch connection can be created as simply as follows:

CREATE ELASTICSEARCH CONNECTION my_elasticsearch
    CONNECTION_STRING = 'elasticsearch://search-elastic1-tiq6hb7ltwvcu2vwp3tbixyzey.us-east-1.es.amazonaws.com:443?cluster.name=elastic_main&ssl=true'
    USER_NAME = 'your_user'
    PASSWORD = 'your_password'
    COMMENT = 'My new Elasticsearch connection';

Note that an Elasticsearch connection must specify the cluster name it is connecting to within the connection string.

This means that in order to connect to multiple clusters within your account, you need to create at least one connection per cluster.

For the full list of connection options with syntax and detailed descriptions, see: Elasticsearch connection with SQL

Once you've created your connection, you are ready to move on to the next step of building your data pipeline: Output to an Elasticsearch index

Alter an Elasticsearch connection

Certain connection options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing Elasticseach connection rather than create a new one.

For example, take the Snowflake connection we created previously:

CREATE ELASTICSEARCH CONNECTION my_elasticsearch
    CONNECTION_STRING = 'elasticsearch://search-elastic1-tiq6hb7ltwvcu2vwp3tbixyzey.us-east-1.es.amazonaws.com:443?cluster.name=elastic_main&ssl=true'
    USER_NAME = 'your_user'
    PASSWORD = 'your_password'
    COMMENT = 'My new Elasticsearch connection';

To change the host you are connecting to but keep everything else the same without having to create an entirely new connection, you can run the following command:

ALTER ELASTICSEARCH CONNECTION my_elasticsearch
    SET CONNECTION_STRING = 'elasticsearch://your_new_host:443?cluster.name=elastic1&ssl=true'; 

To check which specific connection options are mutable, see: Elasticsearch connection with SQL

Drop an Elasticsearch connection

If you no longer need a certain connection, you can easily drop it with the following SQL command:

DROP CONNECTION my_elasticsearch; 

However, note that if there are existing jobs that are dependent upon the connection in question, the connection cannot be deleted.

For more details, see: DROP CONNECTION