CREATE CLUSTER

Cluster Types

Compute Cluster

A compute cluster is a group of servers that is used to process and maintain the data that is held in tables and transformed or loaded in jobs.

Jobs are executed on compute clusters. You can create multiple compute clusters with different settings to separate different use cases and workloads.

Query Cluster

A query cluster is used for querying materialized views in real time. Query clusters host the materialized views in memory for millisecond response times.

Syntax

CREATE { COMPUTE | QUERY } CLUSTER <cluster_name>
  MIN_INSTANCES = <min_instance_count>
  MAX_INSTANCES = <max_instance_count>
  [ ALLOW_MAINTENANCE_ACCESS = { TRUE | FALSE } ]
  [ COMMENT = '<comment' ]
  [ INSTANCE_TYPE_FAMILY = '<instance_type_family>' ]
  [ ON_DEMAND_INSTANCES = <on_demand_instance_count> ]
  [ MAX_REPLAY_INSTANCES = <max_replay_instance_count> ]
  [ SCALING_STRATEGY = { 
      LOW_COST | LOW_LATENCY | CONSISTENT_LOW_LATENCY | NO_SCALING } ]
  [ STATIC_PUBLIC_IPS = <number_of_static_ips> ]
  [ VPC_CONNECTION = <connection_name> ]

Cluster Options

• MIN_INSTANCES

• MAX_INSTANCES

• ALLOW_MAINTENANCE_ACCESS

• COMMENT

• INSTANT_TYPE_FAMILY

• MAX_REPLAY_INSTANCES

• ON_DEMAND_INSTANCES

• SCALING_STRATEGY

• STATIC_PUBLIC_IPS

• VPC_CONNECTION

MIN_INSTANCES — editable

Type: int

The lower limit for the number of instances to use when autoscaling the cluster.

MAX_INSTANCES — editable

Type: int

The upper limit for the number of instances to use when autoscaling the cluster.

ALLOW_MAINTENANCE_ACCESS — editable

Type: Boolean

(Optional) When enabled it will allow Upsolver Support access to the instances for debugging purposes. This only enables SSH access. Network access is still controlled by the network settings of the VPC in which the cluster is running.

Defaults to FALSE.

COMMENT — editable

Type: text

(Optional) Adds a comment or description to the cluster for documentation or clarification purposes.

INSTANCE_TYPE_FAMILY — editable

Type: int

(Optional) The instance type family to use. Upsolver uses similar instance types from the same family and size to achieve better spot instance availability and server uptime. For example, if r5.xlarge is configured then both r5.xlarge and r5d.xlarge will be used.

Defaults to 'r5.xlarge'

MAX_REPLAY_INSTANCES — editable

Type: int

(Optional) Replay Instances are instances that are started automatically when new jobs are added to the cluster. They are used to process the backlog of data without interfering with the latency of production workloads already running on the cluster. After the new jobs are up-to-date, the replay instances will automatically be shut down and the job moved back to the main instances.

Replay cluster instances operate independently and have different IP addresses than the main cluster. Usually, tasks that write to external resources such as databases are executed on the main cluster to maintain IP whitelisting. If you run into access issues, consider either turning off the replay cluster or using a separate, dedicated cluster.

ON_DEMAND_INSTANCES — editable

Type: int

(Optional) Specifies the minimum number of guaranteed on-demand instances allocated to the cluster. While this sets a floor, there may sometimes be more on-demand instances if there's low spot availability.

By default, Upsolver uses ephemeral Spot Instances for data processing. Setting this value allows you to configure how many on-demand Instances are within the cluster.

SCALING_STRATEGY — editable

Type: Enum

(Optional) Determines the scaling strategy of the cluster. Options are:

• LOW_COST: Prioritize cost savings.

• LOW_LATENCY: Prioritize reducing latency.

• CONSISTENT_LOW_LATENCY: Maintain consistently low latency.

• NO_SCALING: No scaling, static number of instances.

Defaults to LOW_LATENCY.

STATIC_PUBLIC_IPS — editable

Type: int

(Optional) The number of static IPs to create and associate with the cluster's instances. This parameter is useful when the cluster needs to connect to external systems that are behind a firewall. Configuring static IPs will allow the opening of those IPs in the target system's firewall.

The value is the number of static IPs to use and can be lower than the total number of instances in the cluster. If configured this way, the instances that get the static IP will be used to make requests to external resources.

VPC_CONNECTION

Type: identifier

(Optional) Use the VPC_CONNECTION option when you have multiple VPC connections for the same organization, otherwise running this command results in an error.