Modify a data source
This article provides a guide on how to modify data sources using an API call.
PATCH https://api.upsolver.com/inputs/DATA-SOURCE-ID
Field | Name | Type | Description | Optional |
softRetention | Soft Retention | Boolean | This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "ToggleSoftRetention",
"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
retention | Retention | Int (Minutes) | The retention period for the data in Upsolver. | + |
curl -X POST -H "content-type: application/json" -H "Authorization:
YOUR_TOKEN" \
-d '{
"clazz" : "UpdateRetention",
"retention" : 1440
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
endExecutionAt | End Read At | String (ISO-8601) | Stop reading after this date. | + |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateEndExecutionAt",
"endExecutionAt" : "2018-11-04T08:00:00Z"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
environment | Ingestion Cluster | String | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "SetIngestionEnvironment",
"environment" : "09267786-8d84-4644-adac-52601b9609bb"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
environment | Environment | String | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "SetComputeEnvironment",
"environment" : "12ba7222-5764-4d37-8581-95ca5be4b3f7"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
name | Name | String | The data source name. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "Rename",
"name" : "NEW_NAME"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
parallelism | Parallelism | Int | The number of independent shards to parse data, to increase parallelism and reduce latency.
This should remain 1 in most cases. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateParallelism",
"parallelism" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
parallelism | Parallelism | Int | The number of independent shards to parse data, to increase parallelism and reduce latency.
This should remain 1 in most cases. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateShardParallelism",
"shards" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
contentType | Content Type | ContentType | The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded. For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded. Messages can be compressed, Upsolver automatically detects the compression type. Supported compression types are: Zip, GZip, Snappy and None. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateContentType",
"contentType" : {
"type" : "JsonContentType"
}
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
description | Description | String | The description of the data source. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateDescription",
"description" : "description"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
workspace | Workspace | String | The workspace to attach to this data source. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "AttachWorkspace",
"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
workspace | Workspace | String | The workspace to detach the data source from. | |
curl -X POST -H "content-type: application/json"
-H "Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "DetachWorkspace",
"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
workspaces | Workspaces | String[] | The workspaces to attach this data source to. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "AttachManyWorkspaces",
"workspaces" : "workspaces"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
connection | Connection | String | Updating the storage location will cause new objects to be written to the new location. | |
moveData | Move existing objects | Boolean | Copy existing objects to the new storage location and then delete from the original location. Dependencies will continue working normally during this process. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateConnection",
"connection" : "00dbf0e2-7c11-47f8-a672-
cacb537af190",
"moveData" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "Stop"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Field | Name | Type | Description | Optional |
reportingTags | Reporting Tags | String | Update the tags that can be sent to external monitoring systems. | |
curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateReportingTags",
"reportingTags" : "reportingTags"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Last modified 2yr ago