Output to Elasticsearch

Prerequisites

Ensure that you have an Elasticsearch connection with the correct permissions to write to your target bucket.

Create a job writing into Elasticsearch

After you have fulfilled the prerequisites, you can create an INSERT job as follows:

CREATE SYNC JOB insert_into_elasticsearch_index
   RUN_INTERVAL = 1 MINUTE
   START_FROM = BEGINNING
   COMMENT = 'insert data into Elasticsearch index'
AS INSERT INTO ELASTICSEARCH your_elasticsearch_connection 
   PREFIX = 'orders'
      SELECT customer_id,
             COUNT(DISTINCT order_id) AS num_orders,
             SUM(net_total) AS total_spent,
             MIN(order_date) AS first_purchase,
             MAX(order_date) AS last_purchase
      FROM default_glue_catalog.your_database.staging_table
      WHERE time_filter()
      GROUP BY customer_id;

This example only demonstrates an example of all job options available when writing to Elasticsearch. Depending on your use case, you may want to configure different options.

Alter a job writing to Elasticsearch

Certain job options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing transformation job rather than having to create a new one.

For example, take the job we created as an example earlier:

CREATE SYNC JOB insert_into_elasticsearch_index
   RUN_INTERVAL = 1 MINUTE
   START_FROM = BEGINNING
   COMMENT = 'insert data into Elasticsearch index'
AS INSERT INTO ELASTICSEARCH your_elasticsearch_connection 
   PREFIX = 'orders'
      SELECT customer_id,
             COUNT(DISTINCT order_id) AS num_orders,
             SUM(net_total) AS total_spent,
             MIN(order_date) AS first_purchase,
             MAX(order_date) AS last_purchase
      FROM default_glue_catalog.your_database.staging_table
      WHERE time_filter()
      GROUP BY customer_id;

If you wanted to keep the job as is but just change the cluster that is running the job, you can run the following command:

ALTER JOB insert_into_elasticsearch_index
    SET COMPUTE_CLUSTER = high_memory_cluster;

Drop a job writing to Elasticsearch

If you no longer need a certain job, you can easily drop it with the following SQL command:

DROP JOB insert_into_elasticsearch_index;

---

Learn More

For the full list of job options with syntax and detailed descriptions, see the transformation job options for Elasticsearch.

See the INSERT SQL command reference for more details and examples.