Links
Comment on page

Output to Elasticsearch index

Prerequisites

Ensure that you have an Elasticsearch connection with the correct permissions to write to your target bucket.

Create a job writing into Elasticsearch

Once you have fulfilled the prerequisites, you can create an INSERT job as follows:
CREATE SYNC JOB insert_into_elasticsearch_index
RUN_INTERVAL = 1 MINUTE
START_FROM = BEGINNING
COMMENT = 'insert data into Elasticsearch index'
AS INSERT INTO ELASTICSEARCH your_elasticsearch_connection PREFIX = 'orders'
SELECT customer_id,
COUNT(DISTINCT order_id) AS num_orders,
SUM(net_total) AS total_spent,
MIN(order_date) AS first_purchase,
MAX(order_date) AS last_purchase
FROM default_glue_catalog.your_database.staging_table
WHERE time_filter()
GROUP BY customer_id;
This example only demonstrates an example of all job options available when writing to Elasticsearch. Depending on your use case, you may want to configure a different set of options.
For the full list of job options with syntax and detailed descriptions, see: Elasticsearch transformation job options
For more details regarding the INSERT command syntax, see: INSERT

Alter a job writing to Elasticsearch

Certain job options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing transformation job rather than having to create a new one.
For example, take the job we created as an example earlier:
CREATE SYNC JOB insert_into_elasticsearch_index
RUN_INTERVAL = 1 MINUTE
START_FROM = BEGINNING
COMMENT = 'insert data into Elasticsearch index'
AS INSERT INTO ELASTICSEARCH your_elasticsearch_connection PREFIX = 'orders'
SELECT customer_id,
COUNT(DISTINCT order_id) AS num_orders,
SUM(net_total) AS total_spent,
MIN(order_date) AS first_purchase,
MAX(order_date) AS last_purchase
FROM default_glue_catalog.your_database.staging_table
WHERE time_filter()
GROUP BY customer_id;
If you wanted to keep the job as is but just change the cluster that is running the job, you can run the following command:
ALTER JOB insert_into_elasticsearch_index
SET COMPUTE_CLUSTER = high_memory_cluster;
To check which specific job options are mutable, see: Elasticsearch transformation job options

Drop a job writing to Elasticsearch

If you no longer need a certain job, you can easily drop it with the following SQL command:
DROP JOB insert_into_elasticsearch_index;
For the full list of job options with descriptions, see: DROP JOB