Modify a data source

This article provides a guide on how to modify data sources using an API call.

This API enables you to modify an existing data source. All API calls require an API token.
PATCH https://api.upsolver.com/inputs/DATA-SOURCE-ID
Toggle soft retention
Fields
Field
Name
Type
Description
Optional
softRetention
Soft Retention
Boolean
This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "ToggleSoftRetention",
	"softRetention" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update retention
Fields
Field
Name
Type
Description
Optional
retention
Retention
Int (Minutes)
The retention period for the data in Upsolver.
+
Example
curl -X POST -H "content-type: application/json" -H "Authorization: 
YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateRetention",
	"retention" : 1440
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update end execution at
Fields
Field
Name
Type
Description
Optional
endExecutionAt
End Read At
String (ISO-8601)
Stop reading after this date.
+
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateEndExecutionAt",
	"endExecutionAt" : "2018-11-04T08:00:00Z"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Set ingestion environment
Fields
Field
Name
Type
Description
Optional
environment
Ingestion Cluster
String
The compute cluster to run the calculation on. 
See: Compute cluster​
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
 "clazz" : "SetIngestionEnvironment",
 "environment" : "09267786-8d84-4644-adac-52601b9609bb"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Set compute environment
Fields
Field
Name
Type
Description
Optional
environment
Environment
String
The compute cluster to run the calculation on. 
See: Compute cluster​
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
  "clazz" : "SetComputeEnvironment",
  "environment" : "12ba7222-5764-4d37-8581-95ca5be4b3f7"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Rename
Fields
Field
Name
Type
Description
Optional
name
Name
String
The data source name.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "Rename",
	"name" : "NEW_NAME"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update parallelism
Fields
Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. 
This should remain 1 in most cases.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateParallelism",
	"parallelism" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update shard parallelism
Fields
Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. 
This should remain 1 in most cases.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateShardParallelism",
	"shards" : 1
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update content type
Fields
Field
Name
Type
Description
Optional
contentType
Content Type
ContentType
The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.
For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.
Messages can be compressed, Upsolver automatically detects the compression type.
Supported compression types are: Zip, GZip, Snappy and None.
See: Content formats​
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateContentType",
	"contentType" : {
		"type" : "JsonContentType"
	}
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update description
Fields
Field
Name
Type
Description
Optional
description
Description
String
The description of the data source.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateDescription",
	"description" : "description"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Attach workspace
Fields
Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to attach to this data source.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "AttachWorkspace",
	"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Detach workspace
Fields
Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to detach the data source from.
​
Example
curl -X POST -H "content-type: application/json" 
-H "Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "DetachWorkspace",
	"workspace" : "workspace"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Attach many workspaces
Fields
Field
Name
Type
Description
Optional
workspaces
Workspaces
String[]
The workspaces to attach this data source to.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "AttachManyWorkspaces",
	"workspaces" : "workspaces"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update connection
Fields
Field
Name
Type
Description
Optional
connection
Connection
String
Updating the storage location will cause new objects to be written to the new location.
​
moveData
Move existing objects
Boolean
Copy existing objects to the new storage location and then delete from the original location. Dependencies will continue working normally during this process.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateConnection",
	"connection" : "00dbf0e2-7c11-47f8-a672-
cacb537af190",
	"moveData" : true
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Stop
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "Stop"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Update reporting tags
Fields
Field
Name
Type
Description
Optional
reportingTags
Reporting Tags
String
Update the tags that can be sent to external monitoring systems.
​
Example
curl -X POST -H "content-type: application/json" -H 
"Authorization: YOUR_TOKEN" \
-d '{
	"clazz" : "UpdateReportingTags",
	"reportingTags" : "reportingTags"
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"

Create a data source

API content formats

Last modified 3yr ago

Field	Name	Type	Description	Optional
softRetention	Soft Retention	Boolean	This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.