Output to Amazon Athena
Prerequisites
Writing to an Amazon Athena table is equivalent to writing to an Upsolver-managed table.
This means that you can create the target table within Upsolver itself, rather than having to do it in Athena beforehand, and it also gives you the option of automatically adding columns in your job that are missing from your target table.
Furthermore, you need a storage connection that has access to the bucket you would like the job to use to store the intermediate files used while running the job as well as a storage connection that has access to the bucket to store your target table's underlying files. These do not necessarily need to be separate connections. Finally, you should also have a staging table already created that contains your ingested data.
Create a job writing to Athena
After you have fulfilled the prerequisites and created your target table, you can create an INSERT
job that writes to that table as follows:
This example only uses a subset of all job options available when writing to a data lake table. Depending on your use case, you may want to configure different options. For instance, if you'd like to insert data up to a specific point in time, you should configure the END_AT
option.
Alter a job writing to Athena
Some job options are considered mutable, meaning that in some cases, you can run a SQL command to alter an existing transformation job rather than having to create a new one.
For example, take the job we created as an example earlier:
If you want to keep the job as is but just change the cluster that is running the job, you can run the following command:
Note that some options such as RUN_INTERVAL
cannot be altered once the connection has been created.
Drop a job writing to Athena
If you no longer need a certain job, you can easily drop it with the following SQL command:
Learn More
Last updated