setup-cluster

Description

Deploy a cluster across multiple hosts

This command conveniently runs various sdb-toolbox-config, sdb-deploy, and sdb-admin commands as a single command to produce a functional cluster that can be connected to and queried.

The --password flag is required and specifies the SingleStore root password on each node. Note that the MEMSQL_PASSWORD environment variable is a safer alternative option for setting the password.

If you would prefer to set up a local cluster for testing purposes, see sdb-deploy cluster-in-a-box.

Examples

Shell

sdb-deploy setup-cluster \
  --master-host user@host1:33 \
  --aggregator-hosts host2 \
  --leaf-hosts host3,host4 \
  --identity-file ~/my_key.pem \
  --license AAAAAAa/Aaa/AA== \
  --password testpass

This will install the latest released version of singlestoredb-server on each host and deploy a Master Aggregator on host1; a child aggregator on host2; and leaf nodes on host3 and host4. These hosts must be accessible via SSH using the current user and the key ~/my_key.pem. A valid license and password for the SingleStore root user are also required. If you set a MEMSQL_LICENSE environment variable, you can omit the --license flag.

"High Availability" mode is enabled by default. To disable it, set --high-availability to false:

Shell

sdb-deploy setup-cluster --master-host host1 ... --high-availability=false

The command will deploy SingleStore on port 3306 on each node by default. To configure it to a different port, set --memsql-port:

Shell

sdb-deploy setup-cluster --master-host host1 ... --memsql-port 3307

If you want to install a version other than the latest released version, you can specify a different released version, or the path to a singlestoredb-server package:

Shell

sdb-deploy setup-cluster --master-host host1 ... --version 6.7.0

Shell

sdb-deploy setup-cluster --master-host host1 ... --file-path path/to/singlestoredb-server.deb

Shell

sdb-deploy setup-cluster --master-host host1 ... --file-path path/to/singlestoredb-server.rpm

If you need to specify a host's SSH user or SSH port, you can add the information to the host descriptor:

Shell

sdb-deploy setup-cluster --master-host admin@host1:10022 --leaf-hosts admin@host2:10022,admin@host3:10022 ...

If you want to use a YAML file to define your cluster you can do so by using the --cluster-file flag which takes in the path to a YAML file as input. Note that you have to give all the configurations as part of the cluster file and cannot use any other flag along with --cluster-file. If you want to learn about creating a cluster file please refer to YAML File Configuration.

Usage of the command with --cluster-file flag:

Shell

sdb-deploy setup-cluster --cluster-file path/to/cluster-file.yml

YAML File Configuration

As of Toolbox 1.3.0, the sdb-deploy setup-cluster command now accepts YAML configuration files, and validates the configuration file format before attempting to set up the specified cluster.

The command is designed to be consistent, where re-running the sdb-deploy setup-cluster command with the same cluster file will always produce the same cluster. It is also resilient, allowing errors encountered at any stage of the cluster construction process to be corrected, and sdb-deploy setup-cluster re-run, in order to generate the desired cluster.

Steps for using a Configuration File

Install Toolbox onto the main deployment host.

Using the template below, create a YAML file with your desired cluster configuration.

YAML

license: <license-from-portal.singlestore.com>
memsql_server_version: <version>
package_type: <rpm | deb | tar>
hosts:
- hostname: <ip-address>
  localhost: true
  nodes:
  - register: false
    role: Master
    config:
      password: <secure-password>
      port: 3306
  - register: false
    role: Leaf
    config:
      password: <secure-password>
      port: 3307

Run the following with the path to the YAML file as input.
Shell
```
sdb-deploy setup-cluster --cluster-file <path-to-cluster-file>
```

Note: By default, the sdb-deploy setup-cluster command configures the host's operating system per SingleStore's recommendations. To disable this auto-configuration, add the --skip-auto-config=true option to the hosts field in the YAML configuration file.

Definitions

The following is a detailed explanation of each field that can be configured in the cluster file.

CLUSTER

Option	Description
`allow_duplicate_host_fingerprints`	When set to `true`, Toolbox allows hosts to have the same `ssh` fingerprints.
`assert_clean_state`	When set to `true`, Toolbox verifies that no nodes currently exist in the cluster.
`high_availability`	This enables high availability on your cluster. This will only work if there are an even number of leaves on your cluster. The default value is `false` if not specified. If set to `true`, each node may be assigned an availability group via the `availability_group` field. Refer to Managing High Availability for more information.
`license`	The license that will be used for the cluster. You can obtain a free or existing license from the Cloud Portal. Your license will be used to bootstrap a Master Aggregator (MA).
`optimize`	When set to `true`, optimizes memory usage for the cluster with the recommended memory percentage. Also optimizes for NUMA.
`optimize_config`	Includes two fields that can change how the cluster is optimized. The `memory_percentage` option dictates the percentage of memory that SingleStore uses on each host. If the `no_numa` option is set to `true`, the NUMA optimization on the cluster will not be performed.
`root_password`	The default password for all the nodes if a password is not specified in the node password field. Required if a node-level password is not specified.
`skip_install`	When set to `true`, Toolbox skips the installation of `singlestoredb-server`. This can be useful for cases such as migration, where each host requires a custom installation.
`skip_validate_env`	When set to `true`, Toolbox will perform cluster setup without validating the environment.
`sync_variables`	The list of sync variables that will be set on the Master Aggregator node.

The following two options are related to the version of SingleStore that will be installed for your cluster. Either option can be specified. If neither option is specified, the latest version of singlestoredb-server is installed.

Option	Description
`memsql_server_version`	The desired version of `singlestoredb-server` will be downloaded and installed cluster-wide.
`memsql_server_file_path`	Installs version of `singlestoredb-server` package specified by the path to the `singlestoredb-server` package.
`package_type`	The package type to use for the installation: `rpm`, `deb`, or `tar`. A package type of `tar` will only be used if explicitly specified. If the host has only one package manager for either `.rpm` or `.deb` packages, `sdb-deploy setup-cluster` will use the associated package manager. If multiple package managers are available on a host, this option is required.

HOST

Option	Description
`hostname`	The cluster-addressable hostname of this host. If only one, local host is available, use `127.0.0.1`.
`localhost`	Use this option to specify that the target host is `localhost`. The default value is `false` if not specified. If provided, `localhost` commands do not require `ssh` access.
`memsqlctl_path`	The path to the `memsqlctl` binary on the target host. Only required if a custom installation of `singlestoredb-server` was performed.
`memsqlctl_config_path`	The path to the `memsqlctl.hcl` file on the target host. Only required if a custom installation of `singlestoredb-server` was performed.
`skip_auto_config`	When set to `true`, Toolbox will not configure system settings on the hosts.
`tar_install_dir`	The directory to use for tarball-based installs on the target host. The default value is `$homedir/memsql` if not specified and a tarball-based install is performed.
`tar_install_state`	The path to the `packages.hcl` file on the target host. This option must only be used in conjunction with `tar_install_dir`.

The following ssh-related fields are required for remote hosts:

Option	Description
`host`	The `ssh` hostname.
`port`	The `ssh` port (default: `22`).
`private_key`	The identity file for the host (typically found in `~<user>/.ssh`
`user`	The `ssh` user for the host.

NODE

Option	Description
`availability_group`	The availability group to assign to the node when `high_availability` is set to `true`. Refer to Availability Groups for more information.
`force_registration`	When set to `true`, allows Toolbox to register offline nodes.
`no_start`	If set to `true`, the node will not be started after creation. This option is not valid if `register` is set to `true`.
`register`	Set the value of this field to `false` to create a new node. Set the value to `true` if the node is already present and you want to register it to Toolbox. The `configpath` field and value are also required when `register` is set to `true`. Do not set this value to `true` to create a new node.
`role`	The role of the current node, which is one of `Master`, `Leaf`, `Aggregator`, or `Unknown`. Do not use this option with the register option.
`variables`	The list of local variables that will be set on the node before the node starts.

CONFIG

Option	Description
`auditlogsdir`	The absolute path to the `auditlogs` directory. If `register` is set to `true`, this option must be specified in the configuration file.
`baseinstalldir`	The absolute path to the base install directory. This option is only available when creating a node.
`configpath`	The absolute path to the node’s configuration file. If `register` is set to `true`, this option must be specified in the configuration file.
`datadir`	The absolute path to the node’s `data` directory. If `register` is set to `true`, this option must be specified in the configuration file.
`default install dir`	When a node is created, its associated data and directories will be tied to a base install directory known as the default install dir. The directory’s name is a randomly-generated UUID. Run `memsqlctl env` to determine where the default install dir is on a host. Note that the `singlestoredb-server.rpm` and `.deb` packages configure the default install dir to be `/var/lib/memsql`. If you prefer to configure a node’s directories, update the following `auditlogsdir`, `baseinstalldir`, `configpath`, `datadir`, `plancachedir`, and `tracelogsdir` fields.
`disable_auto_restart`	If set to `true`, the node will not be restarted automatically if it crashes. This option is not valid if `register` is set to `true`.
`password`	The SingleStore password for the node.
`plancachedir`	The absolute path to the `plancache` directory. If `register` is set to `true`, this option must be specified in the configuration file.
`port`	The port that this node uses on its host. Ensure that all the nodes on a host are set to different ports. The default port is `3306`. A new port must be specified if port `3306` is already in use.
`ssl_fips_mode`	When set to `true`, enables SSL/TLS FIPS mode on the node. To enable for the entire cluster, this parameter must be set to `true` on each node.
`tracelogsdir`	The absolute path to the `tracelogs` directory. If `register` is set to `true`, this option must be specified in the configuration file.

Specify Node Directories

During cluster deployment, the directories in which nodes are created can be specified through the use of the following options:

Option	Deployment via	Method	Level
`--base-install-dir`	RPM, DEB, Tarball	Command line	Host
`baseinstalldir`	RPM, DEB, Tarball	Cluster file YAML	Node
`tar_install_dir`	Tarball	Cluster file YAML	Host or node

With regard to level:

"Host" level reflects that each node will all be created in the same directory on each host
"Node" level reflects that each node will be created in a unique subdirectory on each host

The base install directory for a leaf node can also be updated post-deployment. Refer to Changing the Default Data Directory Post Installation for a Leaf Node for more information.

Same Directory

In these deployment examples, each node is created in the same /opt/singlestore/nodes directory on each host.

Multi-Host, Command Line

Shell

sdb-deploy setup-cluster \
--license <license>\
--version 8.5 \
--master-host <host-1> \
--aggregator-hosts <host-2>
--leaf-hosts <host-3>,<host-4> \
--password <secure-password> \
--base-install-dir "/opt/singlestore/nodes"

Single Host, Cluster File

YAML

license: <license>
memsql_server_file_path: </singlestoredb-server/including/path>
package_type: tar
hosts:
- hostname: host-1
  tar_install_dir: /opt/singlestore/nodes/
  localhost: true
  nodes:
  - register: false
    role: Master
    config:
      password: <secure-password>
      port: 3306
  - register: false
    role: Leaf
    config:
      password: <secure-password>
      port: 3307

Unique Directory

In these deployment examples, each node is created in a unique subdirectory in the /opt/singlestore/nodes directory on each host.

Package-Based

YAML

license: <license>
memsql_server_version: 8.5
package_type: rpm
hosts:
- hostname: host-1
  localhost: false
  nodes:
  - register: false
    role: Master
    config:
      baseinstalldir: /opt/singlestore/nodes/ma-3306
      password: <secure-password>
      port: 3306
- hostname: host-2
  localhost: false
  nodes:
  - register: false
    role: Leaf
    config:
      baseinstalldir: /opt/singlestore/nodes/leaf-3307
      password: <secure-password>
      port: 3307

Tarball-Based

YAML

license: <license>
memsql_server_file_path: </singlestoredb-server/including/path>
package_type: tar
hosts:
- hostname: host-1
  localhost: true
  nodes:
  - register: false
    role: Master
    config:
      baseinstalldir: /opt/singlestore/nodes/ma-3306
      password: <secure-password>
      port: 3306
- hostname: host-2
  localhost: false
  - register: false
    role: Leaf 
    config:
      baseinstalldir: /opt/singlestore/nodes/leaf-3307
      password: <secure-password>
      port: 3307

For multi-host, tarball-based deployments with a YAML file, using tar_install_dir at the host level is equivalent to using baseinstalldir at the node level as each node is created in a unique subdirectory.

YAML

license: <license>
memsql_server_file_path: </singlestoredb-server/including/path>
package_type: tar
hosts:
- hostname: host-1
  tar_install_dir: /opt/singlestore/nodes/ma-3306
  localhost: false
  nodes:
  - register: false
    role: Master
    config:
      password: <secure-password>
      port: 3306
- hostname: host-2
  tar_install_dir: /opt/singlestore/nodes/leaf-3307
  localhost: false
  nodes:
  - register: false
    role: Leaf
    config:
      password: <secure-password>
      port: 3307

Complete Cluster File Template

YAML

license:                           <LICENSE | /path/to/LICENSE-file> [Required to bootstrap Master Aggregator]
high_availability:                 <true | false>
memsql_server_version:             <the version of memsql you want to install (6.7+)>
memsql_server_file_path:           <path to the downloaded memsql server file>
memsql_server_preinstalled_path:   <equivalent to using the '--preinstalled-path' option;
                                   the path to the unpacked singlestoredb-server file
                                   where the unpacked folder name must be of the form
                                   'singlestoredb-server-<version>*' or
                                   'memsql-server-<version>*'>
skip_install:                      <true | false> [ADVANCED]
skip_validate_env:                 <true | false> [ADVANCED]
allow_duplicate_host_fingerprints: <true | false> [ADVANCED]
assert_clean_state:                <true | false> [ADVANCED]
package_type:                      <rpm | deb | tar> [Required if multiple package managers are present]
root_password:                     <default password to be used for all nodes>
optimize:                          <true | false>
optimize_config:
  memory_percentage:               <percentage of memory you want memsql to use>
  no_numa:                         <true | false>
sync_variables:                    [ADVANCED]
  <variable's name>:               <variable's value>            
hosts:
- hostname:                        <host-name> [Required]
  localhost:                       <true | false> 
  skip_auto_config:                <true | false>
  memsqlctl_path:                  <path to memsqlctl> [ADVANCED]
  memsqlctl_config_path:           <path to memsqlctl config> [ADVANCED]
  tar_install_dir:                 <path to tar install dir> [ADVANCED]
  tar_install_state:               <path to tar install state> [ADVANCED]
  ssh:                             [Required for remote Hosts]
    host:                          <ssh host name>
    port:                          <ssh port>
    user:                          <ssh user>
    private_key:                   <path to your identity key>
  nodes:
  - register:                      <true | false>
    force_registration:            <true | false> [ADVANCED] 
    role:                          <Unknown | Master | Leaf | Aggregator> (case sensitive) [Required]
    aggregator_role:               <voting_member | follower (by default)> [Applicable only for role Aggregator]
    availability_group:            <availability group>
    no_start:                      <true | false>
    config: 
      auditlogsdir:                <path to auditlogs directory> [ADVANCED]
      baseinstalldir:              <path to base install directory> [ADVANCED]
      configpath:                  <path to configuration path> [ADVANCED] [Required if register is true]
      datadir:                     <path to data directory> [ADVANCED]
      disable_auto_restart:        <true | false>
      password:                    <password>
      plancachedir:                <path to plancache directory> [ADVANCED]
      port:                        <port number> [Required for node creation]
      tracelogsdir:                <path to tracelogs directory> [ADVANCED]
      bind_address:                <bind address> [ADVANCED]
      ssl_fips_mode:               <true | false > [ADVANCED]
    variables:
      <variable's name>:           <variable's value>

Example Cluster Files

The following cluster file template examples can be used to set up a cluster.

Copy an example into an example.yaml file and pass this file to the sdb-deploy setup-cluster command.

Shell

sdb-deploy setup-cluster --cluster-file example.yaml

Note: Depending on how you installed Toolbox (for example, via tarball-based install, you may need to include the full path to the cluster file.

Shell

./sdb-deploy setup-cluster --cluster-file /full-path/example.yaml

For Example 2, Example 3, and Example 4, you will need multiple hosts and the ability to ssh into each host from the main deployment host.