Upsolver
Search…
Modify a data source
This article provides a guide on how to modify data sources using an API call.
This API enables you to modify an existing data source. All API calls require an API token.
PATCH https://api.upsolver.com/inputs/DATA-SOURCE-ID

Field
Name
Type
Description
Optional
softRetention
Soft Retention
Boolean
This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "ToggleSoftRetention",
"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
retention
Retention
Int (Minutes)
The retention period for the data in Upsolver.
+

curl -X POST -H "content-type: application/json" -H "Authorization:
YOUR_TOKEN" \
-d '{
"clazz" : "UpdateRetention",
"retention" : 1440
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
endExecutionAt
End Read At
String (ISO-8601)
Stop reading after this date.
+

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateEndExecutionAt",
"endExecutionAt" : "2018-11-04T08:00:00Z"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
environment
Ingestion Cluster
String
The compute cluster to run the calculation on. See: Compute cluster

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "SetIngestionEnvironment",
"environment" : "09267786-8d84-4644-adac-52601b9609bb"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
environment
Environment
String
The compute cluster to run the calculation on. See: Compute cluster

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "SetComputeEnvironment",
"environment" : "12ba7222-5764-4d37-8581-95ca5be4b3f7"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
name
Name
String
The data source name.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "Rename",
"name" : "NEW_NAME"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateParallelism",
"parallelism" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateShardParallelism",
"shards" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
contentType
Content Type
ContentType
The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.
For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.
Messages can be compressed, Upsolver automatically detects the compression type.
Supported compression types are: Zip, GZip, Snappy and None.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateContentType",
"contentType" : {
"type" : "JsonContentType"
}
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
description
Description
String
The description of the data source.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateDescription",
"description" : "description"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to attach to this data source.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "AttachWorkspace",
"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to detach the data source from.

curl -X POST -H "content-type: application/json"
-H "Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "DetachWorkspace",
"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
workspaces
Workspaces
String[]
The workspaces to attach this data source to.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "AttachManyWorkspaces",
"workspaces" : "workspaces"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
connection
Connection
String
Updating the storage location will cause new objects to be written to the new location.
moveData
Move existing objects
Boolean
Copy existing objects to the new storage location and then delete from the original location. Dependencies will continue working normally during this process.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateConnection",
"connection" : "00dbf0e2-7c11-47f8-a672-
cacb537af190",
"moveData" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "Stop"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Field
Name
Type
Description
Optional
reportingTags
Reporting Tags
String
Update the tags that can be sent to external monitoring systems.

curl -X POST -H "content-type: application/json" -H
"Authorization: YOUR_TOKEN" \
-d '{
"clazz" : "UpdateReportingTags",
"reportingTags" : "reportingTags"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copy link
On this page
Toggle soft retention
Update retention
Update end execution at
Set ingestion environment
Set compute environment
Rename
Update parallelism
Update shard parallelism
Update content type
Update description
Attach workspace
Detach workspace
Attach many workspaces
Update connection
Stop
Update reporting tags