---
description: >-
  This article provides a guide on how to use Upsolver to index less data into
  Splunk.
---

# Use Upsolver to index less data into Splunk

{% embed url="https://www.youtube.com/watch?v=54ZLgyvddzo&feature=youtu.be" %}

This guide provides an example of how to index less data into Splunk and thereby reduce your Splunk cost.

{% hint style="info" %}
Before we begin, you should have already [deployed Upsolver](../../start-using-upsolver/upsolver-deployment-guide) and[ created data sources](../../../connecting-data-sources/amazon-aws-data-sources/amazon-s3-data-source/quick-guide-s3-data-source-1).
{% endhint %}

## Upsolver architecture for various data structures

The Upsolver architecture liberates your data from vendor lock-in; it allows many ways of analyzing data including SQL engine, Machine Learning, and Searching. Many Upsolver users utilize Athena to run SQL on log data.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkPyyj-mr4XtWSCXMe%2Fimage.png?alt=media\&token=a8d59adf-2b41-4853-b7ee-1091a9039f12)

## Create an Amazon S3 data output

1\. Click on **Outputs** on the left and then click **New** on the right upper corner.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGLei8MFRCauNsZZpsj%2F-MGOmSG3kSczb8BvJ7SR%2FScreen%20Shot%202020-09-04%20at%208.41.17%20AM.png?alt=media\&token=080a6197-2a43-4bf2-9337-9db98da77f9b)

2\. **Select** Amazon S3 as your data output.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGU_WEbGS2sFJaJs1kF%2F-MGUahpxZD4Nws1kOg-y%2FScreen%20Shot%202020-09-05%20at%2011.49.18%20AM.png?alt=media\&token=1974c44c-03aa-429f-ae41-f0b21b9e74f1)

3\. Give the data output a **name** and define your output format (**tabular** or **hierarchical**).&#x20;

4\. Select your **data sources**. This guide uses [AWS VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html).

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQ4L0KpxyEKpH4VGS%2Fimage.png?alt=media\&token=18f8dfba-cdfa-439d-b14e-54b10b5af0a3)

5\. Click **Next** to continue.

## Use the UI or SQL to aggregate data before sending to Splunk

1\. Select the SQL window from the upper right hand corner. Keep in mind that everything that you do on the UI will be reflected in SQL and vice versa.&#x20;

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGUTWKzAw-G9z4OvOZd%2F-MGUZFBWigQJYawLy3o_%2FScreen%20Shot%202020-09-05%20at%2011.38.24%20AM.png?alt=media\&token=0fb6815d-ac85-49bd-ad30-ab7871b6952b)

2\. The sample SQL below aggregates multiple values together for a given period of time, reducing the amount of data being sent to Splunk.

{% code title="Sample SQL" %}
```sql
SELECT data."account-id" AS ACCOUNT_ID, 
    data.action AS action, 
    SUM(TO_NUMBER(data.bytes)) AS SUM_BYTES, 
    SUM(TO_NUMBER(data.packets)) AS SUM_PACKETS, 
    COUNT(*) AS count
FROM “bhopp-vpc-flowlogs”
GROUP BY data."account-id", data.action
```
{% endcode %}

3\. Click on **Properties** on the upper right hand corner.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGU_WEbGS2sFJaJs1kF%2F-MGUcYKcPgJnht_apihs%2FScreen%20Shot%202020-09-05%20at%2011.57.12%20AM.png?alt=media\&token=49ea26f5-6bb8-40ab-93e0-8845300813d9)

4\. Under **Scheduling**, change the **Output Interval** to your desired length.&#x20;

{% hint style="info" %}
This property defines how frequently Upsolver outputs the aggregated data, with the default being 1 minute.
{% endhint %}

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQPl1qIzbgdhyueeg%2Fimage.png?alt=media\&token=574370f7-425d-48c6-bcee-367c0b9029de)

5\. Click **Run** on the upper right hand corner.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGU_WEbGS2sFJaJs1kF%2F-MGUdCiAl-fKmHNj9o_T%2FScreen%20Shot%202020-09-05%20at%2012.00.11%20PM.png?alt=media\&token=61998e46-2786-4e3f-bd84-603368a0ad34)

## Define Amazon S3 output parameters

1\. Define the **Output Format** and the **S3 Connection** information; then click **Next**.&#x20;

{% hint style="info" %}
Keep in mind that Upsolver supports all file types.
{% endhint %}

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQLB2Fpe6VTKmugIe%2Fimage.png?alt=media\&token=fce40d66-a8e9-4a26-8b1c-bbdc3ae5f254)

2\. Define the **compute cluster** that you would like to use and the **time range** of the data you would like to output.&#x20;

{% hint style="info" %}
Keep in mind that setting **Ending At** to **Never** means the output will be a continuous stream.&#x20;
{% endhint %}

3\. Click **Deploy**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQWtRLZFa3t3X752j%2Fimage.png?alt=media\&token=7e0dbab4-bcf7-4b1c-b1f8-fb2b359d590b)

## Configure Splunk environment to read data from S3

While waiting for the data to write to the output, configure the Splunk environment to read from S3; this guide uses a size `t2.large` Splunk instance.&#x20;

{% hint style="info" %}
If you don’t have a Splunk environment, you can easily [start up a Splunk instance](https://aws.amazon.com/marketplace/pp/B00PUXWXNE) in the same environment in which Upsolver is deployed.
{% endhint %}

1\. After logging in, click on **Find More Apps**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQ_-KzXDWIS_A3k1i%2Fimage.png?alt=media\&token=853b3ab0-37f3-4dfc-bddf-15696339ca45)

2\. Find the **Splunk Add-on for Amazon Web Services** app and click **Install**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQbq8i7aopJxXU3P6%2Fimage.png?alt=media\&token=476bc5f5-730b-440f-8052-52e48427d4bc)

3\. Fill out your login information for [Splunk.com](http://www.splunk.com/). Check the license and agreement box and click **Login and Install.**

{% hint style="info" %}
If you don’t have an account, click on [FREE SPLUNK](http://www.splunk.com/) on the upper right hand corner and sign up for a free account.
{% endhint %}

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQewyTkm70Cp55_6M%2Fimage.png?alt=media\&token=e7f5bd2d-968a-4ed3-b60b-0035ccdb1461)

4\. The installation might take a few seconds and Splunk will prompt you to restart. Click **Restart Now**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQisKnwAJzSQkUqWr%2Fimage.png?alt=media\&token=47121e21-b240-460e-ab99-e6f8bf0abbe5)

5\. Login to your Splunk environment again and click on the Splunk Enterprise logo. Then click **Splunk Add-on for AWS**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQl_ehxin9ynRq1uC%2Fimage.png?alt=media\&token=8f22c8c7-28be-4580-a6f1-0dcd87c53c55)

6\. Click on the **Configuration** tab and then click **Add** on the right.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQoBa9qQQDDXioVHR%2Fimage.png?alt=media\&token=e0af9f5a-8440-445a-a58b-37e92c8991e6)

7\. Give your Account a name (make sure to remember this name, we will use it for the data input next). Fill out your **AWS Access Key (Key ID)** and **Secret Key** information then click **Add**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQrG78kfn5qCmHGEy%2Fimage.png?alt=media\&token=0884b370-6637-4f18-88b5-6b8e82a61a04)

8\. Click on **Settings** **>** **Data inputs** on your Splunk UI’s upper right hand corner.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQvMFlnNcDo0WHYhM%2Fimage.png?alt=media\&token=5b88081a-bfe3-4f14-a48d-b513548ea932)

9\. Find and select **AWS S3** data input (most likely on page 2).

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkQxvZIkJa822xzx8J%2Fimage.png?alt=media\&token=b1fe4103-bf22-494b-aff5-2d38299ddf95)

10\. Give the data input a **name** and fill out your **AWS Account** information. It should be the same **Account Name from step 5**.&#x20;

11\. Give it a **bucket name**. This must match the bucket name on your AWS account where the output data is being stored.&#x20;

12\. Change the **Polling interval** to **10**. Define **Key prefix** as your S3 folder path.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkR1jjG4ED_7czLIAd%2Fimage.png?alt=media\&token=43f808c0-e4ec-4351-845a-5f8098ff4809)

13\. Scroll down and check **More settings** to configure additional setting options.&#x20;

14\. Change **Set sourcetype** to **From list**, and select **json\_no\_timestamp** from the **Select sourcetype from list** dropdown. Then click **Next**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkR5Vd5Aouph_SONLh%2Fimage.png?alt=media\&token=8b898773-2d48-42f7-a570-0ea3dfddc155)

15\. Click **Start searching**.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkR9-Ag-J4NQxPxF8r%2Fimage.png?alt=media\&token=4d5bdb6a-ecca-4d8d-8d51-3b2f30942643)

## Verify data in Splunk

1\. Click on **Data Summary** under What to Search.

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkRBs3OyXd08pvYv3M%2Fimage.png?alt=media\&token=6fcbc559-784c-46a5-811b-d3120f196178)

2\. Click on **Sourcetype** and **json\_no\_timestamp.**

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkREbaAIBT5SLobKAk%2Fimage.png?alt=media\&token=c29b4bf3-e9e6-4ab5-ab15-3bed5a57a32f)

3\. Verify your indexed data is the same as the aggregated data from Upsolver. Success!

![](https://1488427404-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MAbbZ6XL3iXMHb3pjqP%2F-MGkMlMAhvQHAJz17esC%2F-MGkRHfn3ZwqCD8fsIJG%2Fimage.png?alt=media\&token=56d68033-a6fb-4bf8-9fc3-9e07448b0296)

## What’s next?

{% content-ref url="../../../data-outputs-and-data-transformation/data-outputs/database-data-outputs/snowflake-output/upsert-data-to-snowflake-using-upsolver" %}
[upsert-data-to-snowflake-using-upsolver](../../../data-outputs-and-data-transformation/data-outputs/database-data-outputs/snowflake-output/upsert-data-to-snowflake-using-upsolver)
{% endcontent-ref %}
