Search results

    Edit on GitHub


    The HTTP Data Source is a fully managed Data Source provided by Upsolver. With this simple to set up Data Source you can ingest data in real-time into Upsolver from any source or device around the world without any extra processing. All data sent to an HTTP Data Source is stored on Amazon S3, so there is no need to worry about losing any of your data. Once set up data can be ingested simply by making HTTP Post requests to the HTTP Data Source's ingestion URL.

    Creating A New HTTP Data Source

    1. On the Data Sources page click Add New Data Source.
    2. Select HTTP from the list of available Data Sources.
    3. Give your Data Source a name in the Name field.
    4. In the Content Type drop down menu select the type of events you are going to be sending to this input. This can be JSON or x-www-url-encoded form data.
    5. Select the Compute Cluster you would like this Data Source to run on. If you have not created a Compute Cluster yet, you can create one from the drop down menu.
    6. You can choose to change where the ingested data from this source will be saved by changing the Storage field in the Advanced section.
    7. Click Create to create your new data source.

    Ingest And Test URLs

    Once you create your data source you will be taken to the Data Source Discovery page. Select the Properties tab and you will see the following information:

    There are two URLs listed here, the first is the Ingest URL, which is the URL you will use to send data to this input from your sources. The second is the Test URL, which is the URL you can use to test that you are sending data correctly and that Upsolver is parsing the data as expected.

    Testing Your HTTP Data Source

    Once you create your data source you should test that you are able to send it information correctly and that Upsolver is parsing it as you expect. To do this follow these steps:

    1. Copy the Test URL from the Properties tab, as described above.
    2. Use curl or any other equivalent tool (such as Postman) to send some test data.

    For example with curl we would do:

    curl -H "Content-Type: application/json" -X POST -d '{"key1":"value","key2":"value2"}' "YOUR_TEST_URL"

    Your response should be a json string with the parsed data such as:

    "results" : [ {
    "data" : {
      "key2" : "value2",
      "key1" : "value"
    "headers" : {
      "head" : {
        "Content-Type" : "application/json",
        "Content-Length" : "32",
        "Host" : "",
        "X-Forwarded-Proto" : "https",
        "User-Agent" : "curl/7.47.0",
        "Accept" : "\*/\*"
    "time" : 1504701629989
    } ]

    Note: the headers of the HTTP request you made and the time of the request will be added to the parsed message. Your original request data will be inside the "data" key.

    Sending Data To Your HTTP Data Source

    Once you have tested your data source you can start ingesting data from your actual sources by sending data in the same format you used when testing to the Ingest URL described in Ingest And Test URLs. Once you start sending data to the Ingest URL you can go to the Schema tab in the Data Source Discovery page for your input to inspect the data being ingested.