setup-cluster
Description
Deploy a SingleStoreDB cluster across multiple hosts
This command conveniently runs various sdb-toolbox-config, sdb-deploy, and sdb-admin commands as a single command to produce a functional SingleStoreDB cluster that can be connected to and queried.
The --password flag is required and specifies the SingleStoreDB root password on each node. Note that the MEMSQL_PASSWORD environment variable is a safer alternative option for setting the password.
If you would instead like to set up a local SingleStoreDB "cluster" for testing purposes, see sdb-deploy cluster-in-a-box.
Examples
sdb-deploy setup-cluster \ --master-host user@host1:33 \ --aggregator-hosts host2 \ --leaf-hosts host3,host4 \ --identity-file ~/my_key.pem \ --license AAAAAAa/Aaa/AA== \ --password testpass
This will install the latest released version of singlestoredb-server on each host and deploy a Master Aggregator on host1; a child aggregator on host2; and leaf nodes on host3 and host4. These hosts must be accessible via SSH using the current user and the key ~/my_key.pem. A valid license and password for the SingleStoreDB root user are also required. If you set a MEMSQL_LICENSE environment variable, you can omit the --license flag.
"High Availability" mode is enabled by default. To disable it, set --high-availability to false:
sdb-deploy setup-cluster --master-host host1 ... --high-availability=false
The command will deploy SingleStoreDB on port 3306 on each node by default. To configure it to a different port, set --memsql-port:
sdb-deploy setup-cluster --master-host host1 ... --memsql-port 3307
If you want to install a version other than the latest released version, you can specify a different released version, or the path to a singlestoredb-server package:
sdb-deploy setup-cluster --master-host host1 ... --version 6.7.0
sdb-deploy setup-cluster --master-host host1 ... --file-path path/to/singlestoredb-server.deb
sdb-deploy setup-cluster --master-host host1 ... --file-path path/to/singlestoredb-server.rpm
If you need to specify a host's SSH user or SSH port, you can add the information to the host descriptor:
sdb-deploy setup-cluster --master-host admin@host1:10022 --leaf-hosts admin@host2:10022,admin@host3:10022 ...
If you want to use a YAML file to define your cluster you can do so by using the --cluster-file flag which takes in the path to a YAML file as input. Note that you have to give all the configurations as part of the cluster file and cannot use any other flag along with --cluster-file. If you want to learn about creating a cluster file please refer to YAML File Configuration.
Usage of the command with --cluster-file flag:
sdb-deploy setup-cluster --cluster-file path/to/cluster-file.yml
YAML File Configuration
As of SingleStoreDB Toolbox 1.3.0, the sdb-deploy setup-cluster command now accepts YAML configuration files, and validates the configuration file format before attempting to set up the specified cluster.
The command is designed to be consistent, where re-running the sdb-deploy setup-cluster command with the same cluster file will always produce the same cluster. It is also resilient, allowing errors encountered at any stage of the cluster construction process to be corrected, and sdb-deploy setup-cluster re-run, in order to generate the desired cluster.
Steps for using a Configuration File
Install SingleStoreDB Toolbox onto the main deployment host.
Using the template below, create a YAML file with your desired cluster configuration.
license: <license-from-portal.singlestore.com> memsql_server_version: <version> package_type: <type> ← rpm, deb, or tar hosts: - hostname: <ip-address> localhost: true nodes: - register: false role: Master config: password: <secure-password> port: 3306 - register: false role: Leaf config: password: <secure-password> port: 3307Run the following with the path to the YAML file as input.
sdb-deploy setup-cluster --cluster-file <path-to-cluster-file>
Note: By default, the sdb-deploy setup-cluster command configures the host's operating system per SingleStore's recommendations. To disable this auto-configuration, add the --skip-auto-config=true option to the hosts field in the YAML configuration file.
Complete Cluster File Template
license: <LICENSE | /path/to/LICENSE-file> [Required to bootstrap Master Aggregator]
high_availability: <true | false>
memsql_server_version: <the version of memsql you want to install (6.7+)>
memsql_server_file_path: <path to the downloaded memsql server file>
memsql_server_preinstalled_path: <equivalent to using the '--preinstalled-path' option;
the path to the unpacked singlestoredb-server file
where the unpacked folder name must be of the form
'singlestoredb-server-<version>*' or
'memsql-server-<version>*'>
skip_install: <true | false> [ADVANCED]
skip_validate_env: <true | false> [ADVANCED]
allow_duplicate_host_fingerprints: <true | false> [ADVANCED]
assert_clean_state: <true | false> [ADVANCED]
package_type: <deb | rpm | tar> [Required if multiple package present]
root_password: <default password to be used for all nodes>
optimize: <true | false>
optimize_config:
memory_percentage: <percentage of memory you want memsql to use>
no_numa: <true | false>
sync_variables: [ADVANCED]
<variable's name>: <variable's value>
hosts:
- hostname: <host-name> [Required]
localhost: <true | false>
skip_auto_config: <true | false>
memsqlctl_path: <path to memsqlctl> [ADVANCED]
memsqlctl_config_path: <path to memsqlctl config> [ADVANCED]
tar_install_dir: <path to tar install dir> [ADVANCED]
tar_install_state: <path to tar install state> [ADVANCED]
ssh: [Required for remote Hosts]
host: <ssh host name>
port: <ssh port>
user: <ssh user>
private_key: <path to your identity key>
nodes:
- register: <true | false>
force_registration: <true | false> [ADVANCED]
role: <Unknown | Master | Leaf | Aggregator> (case sensitive) [Required]
availability_group: <availability group>
no_start: <true | false>
config:
auditlogsdir: <path to auditlogs directory> [ADVANCED]
baseinstalldir: <path to base install directory> [ADVANCED]
configpath: <path to configuration path> [ADVANCED] [Required if register is true]
datadir: <path to data directory> [ADVANCED]
disable_auto_restart: <true | false>
password: <password>
plancachedir: <path to plancache directory> [ADVANCED]
port: <port number> [Required for node creation]
tracelogsdir: <path to tracelogs directory> [ADVANCED]
bind_address: <bind address> [ADVANCED]
ssl_fips_mode: <true | false > [ADVANCED]
variables:
<variable's name>: <variable's value>Example Cluster Files
The following cluster file template examples can be used to set up a cluster.
Copy an example into an example.yaml file and pass this file to the sdb-deploy setup-cluster command.
sdb-deploy setup-cluster --cluster-file example.yaml
Note: Depending on how you installed SingleStoreDB Toolbox (for example, via tarball-based install, you may need to include the full path to the cluster file.
./sdb-deploy setup-cluster --cluster-file /full-path/example.yaml
For Example 2, Example 3, and Example 4, you will need multiple hosts and the ability to ssh into each host from the main deployment host.
Example 1
license: <license-from-portal.singlestore.com>
memsql_server_version: 7.3.10
package_type: rpm
hosts:
- hostname: 127.0.0.1
localhost: true
nodes:
- register: false
role: Master
config:
password: <secure-password>
port: 3306
- register: false
role: Leaf
config:
password: <secure-password>
port: 3307
Using this cluster file, sdb-deploy setup-cluster:
Registers a local host to the cluster.
Installs
singlestoredb-serverv7.3.10 on this local host.Creates a Master Aggregator node on port
3306.Creates a leaf node on port
3307.
Example 2
license: <license-from-portal.singlestore.com>
memsql_server_version: 7.3.10
package_type: rpm
hosts:
- hostname: <hostname1>
localhost: true
- hostname: <hostname2>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
Using this cluster file, sdb-deploy setup-cluster:
Registers a local host to the cluster.
Registers a remote host to the cluster using the provided
sshcredentials.Installs
singlestoredb-serverv7.3.10 on both hosts.
Example 3
license: <license-from-portal.singlestore.com>
memsql_server_version: 7.3.10
package_type: rpm
hosts:
- hostname: <hostname1>
localhost: true
nodes:
- register: false
role: Master
config:
password: <secure-password>
port: 3306
- register: false
role: Leaf
config:
password: <secure-password>
port: 3307
- hostname: <hostname2>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- register: false
role: Aggregator
config:
password: <secure-password>
port: 3306
- register: false
role: Leaf
config:
password: <secure-password>
port: 3307
Using this cluster file, sdb-deploy setup-cluster:
Registers a local host to the cluster.
Register a remote host to the cluster using the provided
sshcredentials.Installs
singlestoredb-serverv7.3.10 on both the hosts.On Host 1, creates a node on port
3306, bootstraps it as the Master Aggregator using the specified license, and sets the password to the value provided in the cluster file.On Host 1, creates a node on port
3307,assign it as a leaf node, and sets the password to the value provided in the cluster file.On Host 2, creates a node on port
3306, assign it as a Child Aggregator, and sets the password to the value provided in the cluster file.On Host 2, creates a node on port
3307, assign it as a leaf node, and sets the password to the value provided in the cluster file.
Example 4
license: <license-from-portal.singlestore.com>
high_availability: true
package_type: rpm
memsql_server_file_path: /home/ec2-user/singlestoredb-server-7.3.10-5aa7127018.x86_64.rpm
hosts:
- hostname: <hostname1>
localhost: true
nodes:
- role: Master
config:
password: <secure-password>
port: 3306
- hostname: <hostname2>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Aggregator
config:
password: <secure-password>
- hostname: <hostname3>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Aggregator
config:
password: <secure-password>
- hostname: <hostname4>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Leaf
availability_group: 1
config:
password: <secure-password>
- hostname: <hostname5>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Leaf
availability_group: 1
config:
password: <secure-password>
- hostname: <hostname6>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Leaf
availability_group: 2
config:
password: <secure-password>
- hostname: <hostname7>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- role: Leaf
availability_group: 2
config:
password: <secure-password>
Using this cluster file, sdb-deploy setup-cluster:
Enables high availability.
Specifies the use of RPM packages.
Specifies the location of the
singlestoredb-serverpackage which will be used directly instead of downloaded.Installs
singlestoredb-serverv7.3.10 on all hosts.On Host 1, creates a node on port
3306, bootstraps it as the Master Aggregator using the specified license, and sets the password to the value provided in the cluster file.On Host 2, creates a node on the default port (
3306), assign it as a Child Aggregator, and sets the password to the value provided in the cluster file.On Host 3, creates a node on the default port (
3306), assign it as a Child Aggregator, and sets the password to the value provided in the cluster file.On Host 4, creates a node on the default port (
3306), assign it as a leaf node, sets the node’s availability group to1, and sets the password to the value provided in the cluster file.On Host 5, creates a node on the default port (
3306), assign it as a leaf node, sets the node’s availability group to1, and sets the password to the value provided in the cluster file.On Host 6, creates a node on the default port (
3306), assign it as a leaf node, sets the node’s availability group to2, and sets the password to the value provided in the cluster file.On Host 7, creates a node on the default port (
3306), assign it as a leaf node, sets the node’s availability group to2, and sets the password to the value provided in the cluster file.
Example 5
license: <license-from-portal.singlestore.com>
memsql_server_version: 8.0.5
package_type: deb
sync_variables:
geo_sphere_radius: 10000.66
cluster_name: <cluster-name>
hosts:
- hostname: <hostname1>
localhost: true
nodes:
- register: false
role: Master
config:
password: <secure-password>
port: 3306
- register: false
role: Leaf
config:
password: <secure-password>
port: 3307
- hostname: <hostname2>
ssh:
host: <cluster-addressable-hostname>
private_key: <path/to/private-key>
nodes:
- register: false
role: Aggregator
config:
password: <secure-password>
port: 3306
- register: false
role: Leaf
config:
password: <secure-password>
port: 3307
variables:
http_api_pool_capacity: 4048
allow_unsupported_filesystem: ONUsing this cluster file, sdb-deploy setup-cluster:
Registers a local host to the cluster.
Register a remote host to the cluster using the provided
sshcredentials.Installs
singlestoredb-serverv8.0.5 on both hosts.On Host 1, creates a node on port
3306, bootstraps it as the Master Aggregator using the specified license, and sets the password to the value provided in the cluster file.On Host 1, creates a node on port
3307, assign it as a leaf node, and sets the password to the value provided in the cluster file.On Host 2, creates a node on port
3306, assign it as a Child Aggregator, and sets the password to the value provided in the cluster file.On Host 2, creates a node on port
3307, assign it as a leaf node, setshttp_api_pool_capacityandallow_unsupported_filesystemvariables to values provided in the cluster file, and sets the password to the value provided in the cluster file.Sets
geo_sphere_radiusandcluster_namesync variables to values provided in the cluster file.
Definitions
The following is a detailed explanation of each field that can be configured in the cluster file.
CLUSTER
Option | Description |
|---|---|
| When set to |
| When set to |
| This enables high availability on your cluster. This will only work if there are an even number of leaves on your cluster. The default value is |
| The license that will be used for the cluster. You can obtain a free or existing license from the SingleStore Customer Portal. Your license will be used to bootstrap a Master Aggregator (MA). |
| When set to |
| Includes two fields that can change how the cluster is optimized. The |
| The default password for all the nodes if a password is not specified in the node password field. Required if a node-level password is not specified. |
| When set to |
| When set to |
| The list of sync variables that will be set on the Master Aggregator node. |
The following two options are related to the version of SingleStoreDB that will be installed for your cluster. Either option can be specified. If neither option is specified, the latest version of singlestoredb-server is installed.
Option | Description |
|---|---|
| The desired version of |
| Installs version of |
| The package type to use for the installation: |
HOST
Option | Description |
|---|---|
| The cluster-addressable hostname of this host. If only one, local host is available, use |
| Use this option to specify that the target host is |
| The path to the |
| The path to the |
| When set to |
| The directory to use for tarball-based installs on the target host. The default value is |
| The path to the |
The following ssh-related fields are required for remote hosts:
Option | Description |
|---|---|
| The |
| The |
| The identity file for the host (typically found in |
| The |
NODE
Option | Description |
|---|---|
| The availability group to assign to the node when |
| When set to |
| If set to |
| Set the value of this field to |
| The role of the current node, which is one of |
| The list of local variables that will be set on the node before the node starts. |
CONFIG
Option | Description |
|---|---|
| The absolute path to the |
| The absolute path to the base install directory. This option is only available when creating a node. |
| The absolute path to the node’s configuration file. If |
| The absolute path to the node’s |
| When a node is created, its associated data and directories will be tied to a base install directory known as the “default install dir.” The directory’s name is a randomly-generated UUID. Run |
| If set to |
| The SingleStoreDB password for the node. |
| The absolute path to the |
| The port that this node uses on its host. Ensure that all the nodes on a host are set to different ports. The default port is |
| When set to |
| The absolute path to the |
Host File Structure
For large clusters, you can specify all of your hosts' SSH information through an INI file that you reference using the --host-file flag. The format is below.
[master] [ssh_user@]<master_host_IP_address|DNS_name>[:ssh_port] [aggregators] [ssh_user@]<agg_host1_IP_address|DNS_name>[:ssh_port] ... [leaves] [ssh_user@]<leaf_host1_IP_address|DNS_name>[:ssh_port] ...
Usage
Usage:
sdb-deploy setup-cluster [flags]
Flags:
--aggregator-hosts strings SSH addresses of the child aggregator hosts
--allow-duplicate-host-fingerprints Whether to allow hosts with identical SSH host keys. (ADVANCED)
--allow-ipv6 Sets allow_ipv6 to true for all nodes in cluster
--base-install-dir ABSOLUTE_PATH The absolute path to the base install directory for all the nodes
--cluster-file FILE_PATH The path to the cluster definition file
--data-dir ABSOLUTE_PATH The absolute path to the data directory for all the nodes
--file-path ABSOLUTE_PATH The full path, including the filename, of the singlestoredb-server package to install
--force-package-format {deb, rpm, tar} Specify 'deb', 'rpm', or 'tar' package format. Required if you have both dpkg and rpm package managers found on the host
-h, --help Help for setup-cluster
--high-availability Indicate whether to deploy the cluster in high-availability mode (default true)
--hosts-file ABSOLUTE_PATH The file specifying the host addresses/hostnames for the Master Aggregator, child aggregators, and leaf nodes, in INI format with [master], [aggregators], and [leaves] sections
-i, --identity-file ABSOLUTE_PATH The identity file to SSH to each host
--leaf-hosts strings SSH addresses/hostnames of each leaf node host. At least one must be specified
--license LICENSE The license for the cluster
--load-balanced Use load_balanced mode for partitioning leaf nodes when high availability is enabled
--master-host string SSH address/hostname of the Master Aggregator host
--memsql-port PORT The port each node will run on
--password STRING The database root user's password. If a password is specified on the command line, it must not contain an unescaped '$' character as it will be replaced by the shell
--preinstalled-path ABSOLUTE_PATH The path to the folder (must be of the form 'singlestoredb-server-<version>*' or 'memsql-server-<version>*') that contains the singlestoredb-server binaries. Depending on the mode of cluster deployment, the folder contains either the unpacked tarball file or the installed package file (Debian or RPM) of singlestoredb-server. (ADVANCED)
--skip-validate-env Skip environment validation
--ssl-fips-mode string Whether the cluster operates in OpenSSL/FIPS mode
--version VERSION The version of singlestoredb-server to install (default production:latest)
Global Flags:
--backup-cache FILE_PATH File path for the backup cache
--cache-file FILE_PATH File path for the Toolbox node cache
-c, --config FILE_PATH File path for the Toolbox configuration
--disable-colors Disable color output in console, which some terminal sessions/environments may have difficulty with
--disable-spinner Disable the progress spinner, which some terminal sessions/environments may have issues with
-j, --json Enable JSON output
--parallelism POSITIVE_INTEGER Maximum number of operations to run in parallel
--runtime-dir DIRECTORY_PATH Where to store Toolbox runtime data
--ssh-max-sessions POSITIVE_INTEGER Maximum number of SSH sessions to open per host, must be at least 3
--ssh-strict-host-key-checking Enable strict host key checking for SSH connections
--ssh-user-known-hosts-file FILE_PATH Path to the user known_hosts file for SSH connections. If not set, /dev/null will be used
--state-file FILE_PATH Toolbox state file path
-v, --verbosity count Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
-y, --yes Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default
Remarks
This command is interactive unless you use either --yes or --json flag to override interactive behavior.
Output
The following example uses the setup-cluster command to set up a simple cluster on two different hosts using a hosts file.
hosts file
[master] 172.18.1.4 [leaves] 172.18.1.5
Setting up a cluster with a hosts file
sdb-deploy setup-cluster -i clusterkey --license <license> --hosts-file hosts --password SecureKey
****
✓ Registered hosts
sdb-deploy will perform the following actions:
· Install singlestoredb-server 7.3.10 on hosts
- 172.18.1.4
- 172.18.1.5
· Deploy a master aggregator on 172.18.1.4:3306
- Enable high availability mode
· Deploy a leaf node on 172.18.1.5:3306
· Set root password on all nodes
Would you like to continue? [y/N]: y
✓ Downloaded singlestoredb-server 7.3.10
Installing SingleStoreDB on all hosts...
✓ Installed singlestoredb-server7.3.10-6c220bccfc on host 172.18.1.4 (1/2)
✓ Installed singlestoredb-server7.3.10-6c220bccfc on host 172.18.1.5 (2/2)
✓ Successfully installed on 2 hosts
✓ Created master node
✓ Successfully set license
✓ Bootstrapped master aggregator
✓ Enabled high availability mode
✓ Created aggregator nodes
✓ Added aggregators nodes to cluster
✓ Created leaf nodes
✓ Added leaf nodes to cluster
To view your cluster, run 'sdb-admin list-nodes'