Upsolver can read events directly from your Azure Blob storage, assuming events are partitioned by event date (this partitioning defines the folder structure in the cloud storage).
Upsolver auto-detects the file format's date pattern and shows you a preview of the top files that will be ingested with their corresponding dates. It also auto-detects when to start the ingestion from.
1. From the Data Sources, click New.
2. Select Azure Blob Storage.
3. Select the Azure Blob storage connection to read from (or create a new one).
4. (Optional) Enter the path to the data folder in Azure Blob storage (e.g. billing data). If this is not specified, the data is assumed to be in the top-level of the hierarchy.
5. Select or enter in the date pattern of the files to be ingested. This is autodetected but can be modified if required.
6. (Optional) Select a time to start ingesting files from. This is usually auto detected, but if there is no preview, the date cannot be established and this defaults to today's date. In this case, set the required start date.
7. (Optional) Under File Name Pattern, select the type of pattern to use (All, Starts With, Ends With, Regular Expression, or Glob Expression) and then enter the file name pattern that specifies the file set to ingest.
8. Select the content format. This is typically auto-detected, but you can manually select a format.
9. From the dropdown, select a compute cluster (or create a new one) to run the calculation on.
10. Select a target storage connection (or create a new one) where the data read will be stored (output storage).
11. (Optional) In addition to the streaming data, you may have initial data. In this case, to list the relevant data, enter a file name prefix (e.g. for DMS loads, enter LOAD) under Initial Load Configuration.
12. (Optional) Under Initial Load Configuration, enter a regex pattern to select the required files.
13. Name this data source.
14. Click Continue. A preview of the data will appear.
15. For CSV, select a Header.
16. Click Continue again.
17. (Optional) If there are any errors, click Back to change the settings as required.
18. Click Create.