Upsolver
Search…
Modify a data source
This article provides a guide on how to modify data sources using an API call.
This API enables you to modify an existing data source. All API calls require an API token.
1
PATCH https://api.upsolver.com/inputs/DATA-SOURCE-ID
Copied!

Toggle soft retention

Fields

Field
Name
Type
Description
Optional
softRetention
Soft Retention
Boolean
This setting prevents data deletion when the retention policy in Upsolver activates. When enabled, the metadata is purged but the underlying data (e.g. S3 object) is not deleted.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "ToggleSoftRetention",
5
"softRetention" : true
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update retention

Fields

Field
Name
Type
Description
Optional
retention
Retention
Int (Minutes)
The retention period for the data in Upsolver.
+

Example

1
curl -X POST -H "content-type: application/json" -H "Authorization:
2
YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateRetention",
5
"retention" : 1440
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update end execution at

Fields

Field
Name
Type
Description
Optional
endExecutionAt
End Read At
String (ISO-8601)
Stop reading after this date.
+

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateEndExecutionAt",
5
"endExecutionAt" : "2018-11-04T08:00:00Z"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Set ingestion environment

Fields

Field
Name
Type
Description
Optional
environment
Ingestion Cluster
String
The compute cluster to run the calculation on. See: Compute cluster

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "SetIngestionEnvironment",
5
"environment" : "09267786-8d84-4644-adac-52601b9609bb"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Set compute environment

Fields

Field
Name
Type
Description
Optional
environment
Environment
String
The compute cluster to run the calculation on. See: Compute cluster

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "SetComputeEnvironment",
5
"environment" : "12ba7222-5764-4d37-8581-95ca5be4b3f7"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Rename

Fields

Field
Name
Type
Description
Optional
name
Name
String
The data source name.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "Rename",
5
"name" : "NEW_NAME"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update parallelism

Fields

Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateParallelism",
5
"parallelism" : 1
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update shard parallelism

Fields

Field
Name
Type
Description
Optional
parallelism
Parallelism
Int
The number of independent shards to parse data, to increase parallelism and reduce latency. This should remain 1 in most cases.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateShardParallelism",
5
"shards" : 1
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update content type

Fields

Field
Name
Type
Description
Optional
contentType
Content Type
ContentType
The format of the messages. Supported formats are: JSON, AVRO, CSV, TSV, ORC, Protobuf and x-www-form-urlencoded.
For self-describing formats like JSON, the schema is auto-detected. The body should contain of the message should contain the message itself, which should not be url-encoded.
Messages can be compressed, Upsolver automatically detects the compression type.
Supported compression types are: Zip, GZip, Snappy and None.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateContentType",
5
"contentType" : {
6
"type" : "JsonContentType"
7
}
8
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update description

Fields

Field
Name
Type
Description
Optional
description
Description
String
The description of the data source.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateDescription",
5
"description" : "description"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Attach workspace

Fields

Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to attach to this data source.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "AttachWorkspace",
5
"workspace" : "workspace"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Detach workspace

Fields

Field
Name
Type
Description
Optional
workspace
Workspace
String
The workspace to detach the data source from.

Example

1
curl -X POST -H "content-type: application/json"
2
-H "Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "DetachWorkspace",
5
"workspace" : "workspace"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Attach many workspaces

Fields

Field
Name
Type
Description
Optional
workspaces
Workspaces
String[]
The workspaces to attach this data source to.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "AttachManyWorkspaces",
5
"workspaces" : "workspaces"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update connection

Fields

Field
Name
Type
Description
Optional
connection
Connection
String
Updating the storage location will cause new objects to be written to the new location.
moveData
Move existing objects
Boolean
Copy existing objects to the new storage location and then delete from the original location. Dependencies will continue working normally during this process.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateConnection",
5
"connection" : "00dbf0e2-7c11-47f8-a672-
6
cacb537af190",
7
"moveData" : true
8
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Stop

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "Stop"
5
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!

Update reporting tags

Fields

Field
Name
Type
Description
Optional
reportingTags
Reporting Tags
String
Update the tags that can be sent to external monitoring systems.

Example

1
curl -X POST -H "content-type: application/json" -H
2
"Authorization: YOUR_TOKEN" \
3
-d '{
4
"clazz" : "UpdateReportingTags",
5
"reportingTags" : "reportingTags"
6
}' "https://api.upsolver.com/inputs/DATA-SOURCE-ID"
Copied!
Last modified 1yr ago