# Migrate Cluster The following semi-automated process can be used to migrate from MemSQL Ops to SingleStore Tools. Command output examples are provided. Note that, as they are representative samples, what is displayed on your screen may vary. > **📝 Note**: Unless explicitly stated otherwise, perform all of the following steps on/from the **Master Aggregator**.In the commands provided, replace all instances of content in angle brackets with the referenced substitute. Note that multiple instances may need to be replaced in a single command.Change to the `/opt/singlestore/singlestoredb-toolbox-` directory first before running all commands.In some cases, a script may require non-password access to all hosts in the cluster. ## Review Current Cluster Use MemSQL Ops to validate the SingleStore cluster health. Confirm that all nodes are listed, online, healthy, and are running MemSQL 6.7 or later. Make sure that the nodes meet these requirements before proceeding with the migration. ```shell /path/to/memsql-ops/memsql-ops memsql-list ``` ```output ID Agent Id Process State Cluster State Role Host Port Version 44CDE71 A0c8a89 RUNNING CONNECTED MASTER 10.0.0.98 3306 x.x 3FD3FCF Aa5eab9 RUNNING CONNECTED AGGREGATOR 10.0.0.133 3306 x.x 9C0841F A04f65f RUNNING CONNECTED LEAF 10.0.0.236 3306 x.x 9A551DA A5f0572 RUNNING CONNECTED LEAF 10.0.0.136 3306 x.x ``` ## Confirm Connectivity Confirm that `memsql-ops agent-ssh` can be used to SSH (open a secure shell) into each host in the cluster. Although this command is used to SSH into hosts, the initial output of the command can be used to check whether SSH has been configured on all hosts.  **Note**: After invoking the following interactive prompt, exit the command immediately by using Control-C without actually connecting to the hosts via SSH. ```shell /path/to/memsql-ops/memsql-ops agent-ssh ``` ```output Index ID Host Port Role State Version Has Credential 1 Aeea1ec 10.0.0.98 9000 PRIMARY ONLINE x.x No 2 A005a87 10.0.0.133 9000 FOLLOWER ONLINE x.x Yes 3 Aa9367f 10.0.0.236 9000 FOLLOWER ONLINE x.x Yes 4 Ad90aba 10.0.0.136 9000 FOLLOWER ONLINE x.x Yes ``` In the output displayed above, only the main host where the Ops primary agent runs should reflect`No` for `Has Credential`. For all other hosts, it is strongly recommended to set an SSH credential before proceeding with the migration. SingleStore Toolbox must SSH into each host to run commands. This means that if the SSH credentials are not set, you will be prompted for a username and a password for every host in the cluster, for every SingleStore Toolbox command that you run. To configure `ssh` keys for each host, run the `memsql-ops agent-set-credential` command.  Alternatively, use the following script to automate this process. While executing the script, make sure to provide the `ssh` keys where applicable. ```shell /path/to/memsql-ops/memsql-ops agent-list -q | xargs -n 1 /path/to/memsql-ops/memsql-ops agent-set-credential -i ``` ## Enable Manual Control Enable manual control of the SingleStore cluster to prevent MemSQL Ops from interfering with the SingleStore Toolbox install. ```shell /path/to/memsql-ops/memsql-ops cluster-manual-control --enable ``` ```output The cluster is now under manual control. We will not automatically restart stopped nodes, update license files, etc. ``` > **⚠️ Warning**: MemSQL Ops will not automatically restart nodes that fail during the migration unless manual control of the cluster is enabled. While SingleStore Toolbox has been installed, no hosts or nodes are known to it yet. Hence, nodes will not be auto-restarted if they fail while the migration is underway. ## Add a License If you are currently using the Odin license, which uses an earlier format, update it to the Freya license with the latest format obtained from the [Cloud Portal](https://portal.singlestore.com/). To verify the format of the current license, use the command below: ```shell /path/to/memsql-ops/memsql-ops license-list -j ``` The license marked as `best` is the one in current use, and the expected format of the license is `==`. If the current license does not match the expected format, request a new license in the Cloud Portal and add it using the command below: ```shell /path/to/memsql-ops/memsql-ops license-add --license-key ``` ```output Successfully added license. ``` ## Set root Password Set the `root` password for all nodes in the cluster. Substitute the Master Aggregator’s password for `` in the following command. ```shell /path/to/memsql-ops/memsql-ops memsql-list -q | xargs -n 1 \ /path/to/memsql-ops/memsql-ops memsql-update-root-password --no-confirmation -p ``` ## Generate Cluster Migration File Run the following command to generate a cluster migration YAML file that will pass the cluster information from MemSQL Ops to SingleStore Toolbox. The cluster migration YAML file will be used in the command captured in the next step. ```shell /path/to/memsql-ops/memsql-ops migration-setup ``` ```output Check succeeded: All nodes on MemSQL version x.x Check succeeded: All nodes online and connected Check succeeded: All nodes with supported license version All checks passed. Migration configuration file written to /path/to/memsql-ops/data/cache/migration_config.yml ``` ## Modify Migration File At the bottom of the YAML file: 1. Change `memsql_server_version` to `memsql_server_file_path: /home//s2db/singlestoredb-server-.tar.gz` 2. Change `package_type: rpm` to `package_type: tar` These two lines should then resemble: ``` memsql_server_file_path: /home//s2db/singlestoredb-server-.tar.gz package_type: tar ``` ## Add Root Password Run the following command to add the `root_password` to the end of the cluster migration YAML file. Substitute the Master Aggregator’s password for ``. ```shell echo 'root_password: ' >> ``` ## Deploy Migration File Use the cluster migration YAML generated by the previous command. ```shell ./sdb-deploy setup-cluster --cluster-file ``` * *View the Deploy Cluster Migration File Output* ``` ✓ Connection successful to 10.0.0.136 ✓ Connection successful to 10.0.0.133 ✓ Connection successful to 10.0.0.236 sdb-deploy will perform the following actions: · Register Hosts - Host: 10.0.0.98 + Localhost: true - Host: 10.0.0.136 + Localhost: false + Port: 22 + Username: - Host: 10.0.0.133 + Localhost: false + Port: 22 + Username: - Host: 10.0.0.236 + Localhost: false + Port: 22 + Username: · Install MemSQL Server - Install singlestoredb-server x.x on 10.0.0.98 - Install singlestoredb-server x.x on 10.0.0.136 - Install singlestoredb-server x.x on 10.0.0.133 - Install singlestoredb-server x.x on 10.0.0.236 · Register Node - On Host: 10.0.0.136 + Run 'memsqlctl register-node --memsql-config /var/lib/memsql/leaf-3306-MI2fc1915e/memsql.cnf --datadir /var/lib/memsql/leaf-3306-MI2fc1915e/data --plancachedir /var/lib/memsql/leaf-3306-MI2fc1915e/plancache --tracelogsdir /var/lib/memsql/leaf-3306-MI2fc1915e/tracelogs --port 3306' - On Host: 10.0.0.133 + Run 'memsqlctl register-node --memsql-config /var/lib/memsql/child-3306-MIb4668da6/memsql.cnf --datadir /var/lib/memsql/child-3306-MIb4668da6/data --plancachedir /var/lib/memsql/child-3306-MIb4668da6/plancache --tracelogsdir /var/lib/memsql/child-3306-MIb4668da6/tracelogs --port 3306' - On Host: 10.0.0.236 + Run 'memsqlctl register-node --memsql-config /var/lib/memsql/leaf-3306-MId88e2bc0/memsql.cnf --datadir /var/lib/memsql/leaf-3306-MId88e2bc0/data --plancachedir /var/lib/memsql/leaf-3306-MId88e2bc0/plancache --tracelogsdir /var/lib/memsql/leaf-3306-MId88e2bc0/tracelogs --port 3306' - On Host: 10.0.0.98 + Run 'memsqlctl register-node --memsql-config /var/lib/memsql/master-3306-MI436853b1/memsql.cnf --datadir /var/lib/memsql/master-3306-MI436853b1/data --plancachedir /var/lib/memsql/master-3306-MI436853b1/plancache --tracelogsdir /var/lib/memsql/master-3306-MI436853b1/tracelogs --port 3306' · After executing the above actions the cluster is going to be in the following state: - 4 Registered nodes Would you like to continue? [y/N]: y ✓ Registered hosts ✓ Installed singlestoredb-server on host 10.0.0.236 (1/4) ✓ Installed singlestoredb-server on host 10.0.0.98 (2/4) ✓ Installed singlestoredb-server on host 10.0.0.136 (3/4) ✓ Installed singlestoredb-server on host 10.0.0.133 (4/4) ✓ Successfully installed on 4 hosts ✓ Successfully registered nodes ✓ No Nodes to Create ✓ No Leaves to add ✓ No Aggregators to add The Final Cluster State Hosts +----------------+------------+---------------------------+---------------+ | Host | Local Host | SSH address | Identity File | +----------------+------------+---------------------------+---------------+ | 10.0.0.98 | Yes | | | | 10.0.0.133 | No | @10.0.0.133:22 | | | 10.0.0.136 | No | @10.0.0.136:22 | | | 10.0.0.236 | No | @10.0.0.236:22 | | +----------------+------------+---------------------------+---------------+ Nodes +------------+------------+----------------+------+---------------+--------------+---------+----------------+--------------------+--------------+ | MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address | +------------+------------+----------------+------+---------------+--------------+---------+----------------+--------------------+--------------+ | 6C1E8E88A6 | Master | 10.0.0.98 | 3306 | Running | True | x.x | Online | | 0.0.0.0 | | BE9E87CCE3 | Aggregator | 10.0.0.133 | 3306 | Running | True | x.x | Online | | 0.0.0.0 | | 8419072D05 | Leaf | 10.0.0.136 | 3306 | Running | True | x.x | Online | 2 | 0.0.0.0 | | 36B777507F | Leaf | 10.0.0.236 | 3306 | Running | True | x.x | Online | 1 | 0.0.0.0 | +------------+------------+----------------+------+---------------+--------------+---------+----------------+--------------------+--------------+ ``` ## Confirm Registered Hosts Use SingleStore Toolbox to confirm that all hosts have been registered. ```shell ./sdb-toolbox-config list-hosts ``` ```output +------------+------------+-------------+--------------------------+ | Host | Local Host | SSH address | Identity File | +------------+------------+-------------+--------------------------+ | 10.0.0.98 | No | 10.0.0.98 | /home//.ssh/id_rsa | | 10.0.0.133 | No | 10.0.0.133 | /home//.ssh/id_rsa | | 10.0.0.136 | No | 10.0.0.136 | /home//.ssh/id_rsa | | 10.0.0.236 | No | 10.0.0.236 | /home//.ssh/id_rsa | +------------+------------+-------------+--------------------------+ ``` ## Confirm Registered Nodes Use SingleStore Toolbox to confirm that all nodes have been registered. ```shell ./sdb-admin list-nodes ``` ```output +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ | MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ | 0EAE75B920 | Master | 10.0.0.98 | 3306 | Running | True | x.x | Online | | | 35B6AA0317 | Aggregator | 10.0.0.133 | 3306 | Running | True | x.x | Online | | | B964A18F18 | Leaf | 10.0.0.136 | 3306 | Running | True | x.x | Online | 1 | | 1A8EDD42BB | Leaf | 10.0.0.236 | 3306 | Running | True | x.x | Online | 2 | +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ ``` ## Restart Nodes Use SingleStore Toolbox to restart all nodes. This transfers cluster control to SingleStore Toolbox. After this step is completed, SingleStore Toolbox will manage the cluster, including automatically starting any offline nodes. ```shell ./sdb-admin restart-node --all ``` For high-availability clusters, you may include the `--online` option to perform a rolling restart of the cluster, which allows the workload to continue while the nodes are restarted. ```shell ./sdb-admin restart-node --all --online ``` > **⚠️ Warning**: While this step is underway, the node state may be registered incorrectly in SingleStore Toolbox (as displayed by `sdb-admin list-nodes`). You can check the actual cluster state with `SHOW LEAVES` or `SHOW AGGREGATORS` executed in a database connection to the Master Aggregator. When this step is completed, SingleStore Toolbox should show the node state accurately. * *View Restart Nodes Output* ``` Toolbox is about to perform the following actions: · Restart all nodes in the cluster Would you like to continue? [y/N]: y ✓ Stopped Master node on 10.0.0.98 (1/1) ✓ Successfully stopped Master node on 1 host ✓ Stopped Master node ✓ Stopped Aggregator nodes on 10.0.0.133 (1/1) ✓ Successfully stopped Aggregator nodes on 1 host ✓ Stopped Aggregator node ✓ Stopped Leaf nodes on 10.0.0.136 (1/2) ✓ Stopped Leaf nodes on 10.0.0.236 (2/2) ✓ Successfully stopped Leaf nodes on 2 hosts ✓ Stopped Leaf nodes ✓ Started Leaf nodes on 10.0.0.236 (1/2) ✓ Started Leaf nodes on 10.0.0.136 (2/2) ✓ Successfully started Leaf nodes on 2 hosts ✓ Successfully connected to Leaf nodes ✓ Started Aggregator nodes on 10.0.0.133 (1/1) ✓ Successfully started Aggregator nodes on 1 host ✓ Successfully connected to Aggregator node ✓ Started Master node on 10.0.0.98 (1/1) ✓ Successfully started Master node on 1 host ✓ Successfully connected to Master node Operation completed successfully ``` ## Confirm Node Restart Confirm that all nodes have restarted. ```shell ./sdb-admin list-nodes ``` ```output +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ | MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ | D785AA80D7 | Master | 10.0.0.98 | 3306 | Running | True | x.x | Online | | | 9098564B01 | Aggregator | 10.0.0.133 | 3306 | Running | True | x.x | Online | | | 2189856FC2 | Leaf | 10.0.0.136 | 3306 | Running | True | x.x | Online | 2 | | 3094B68557 | Leaf | 10.0.0.236 | 3306 | Running | True | x.x | Online | 1 | +------------+------------+------------+------+---------------+--------------+---------+----------------+--------------------+ ``` ## Unmonitor Nodes Once SingleStore Toolbox is managing the SingleStore cluster, the competing MemSQL Ops management system **must** be uninstalled. Use MemSQL Ops to list all running nodes. ```shell /path/to/memsql-ops/memsql-ops memsql-list ``` ```output ID Agent Id Process State Cluster State Role Host Port Version 44CDE71 A0c8a89 RUNNING CONNECTED MASTER 10.0.0.98 3306 x.x 3FD3FCF Aa5eab9 RUNNING CONNECTED AGGREGATOR 10.0.0.133 3306 x.x 9C0841F A04f65f RUNNING CONNECTED LEAF 10.0.0.236 3306 x.x 9A551DA A5f0572 RUNNING CONNECTED LEAF 10.0.0.136 3306 x.x ``` Use MemSQL Ops to stop monitoring each SingleStore node. Use the IDs from the `ID` column in the MemSQL Ops output. > **⚠️ Warning**: Do not remove the leaf nodes if prompted. ## Main Host Run the following command on the main host, which is usually the host of the Master Aggregator. ```shell /path/to/memsql-ops/memsql-ops memsql-unmonitor 44CDE71 ``` ```output 2019-07-02 16:30:23: J9fe705 [INFO] Stopping monitoring for MemSQL node ABECB9CFBB84C5369982BE10A85EC050D20486B0 2019-07-02 16:30:23: J9fe705 [INFO] No longer monitoring MemSQL node ABECB9CFBB84C5369982BE10A85EC050D20486B0 ``` ## Unmonitor Remaining Nodes ## Child Aggregator ```shell /path/to/memsql-ops/memsql-ops memsql-unmonitor 3FD3FCF ``` ```output 2019-07-02 16:30:54: J6fb0f3 [INFO] Stopping monitoring for MemSQL node 4DF1BC43819C21D33B6D6AAE02C8851F40709FDF 2019-07-02 16:30:54: J6fb0f3 [INFO] No longer monitoring MemSQL node 4DF1BC43819C21D33B6D6AAE02C8851F40709FDF ``` ## Leaf Node 1 ```shell /path/to/memsql-ops/memsql-ops memsql-unmonitor 9C0841F ``` ```output Would you additionally like to remove the leaf node from the SingleStore cluster? (You can specify this behavior with --remove-leaf.) [y/n] n 2019-07-02 16:31:22: J2eae2d [INFO] Stopping monitoring for MemSQL node AF1C3A169BFE4FD792163909BDB592C65D0E37AF 2019-07-02 16:31:22: J2eae2d [INFO] No longer monitoring MemSQL node AF1C3A169BFE4FD792163909BDB592C65D0E37AF ``` **Note**: If a leaf is removed unintentionally, connect to the database and run `ADD LEAF` to add it back to the cluster: For example, on the Master Aggregator: `memsql> ADD LEAF 'root'@'<>':3307` ## Leaf Node 2 ```shell memsql-ops memsql-unmonitor 9A551DA ``` ```output Would you additionally like to remove the leaf node from the SingleStore cluster? (You can specify this behavior with --remove-leaf.) [y/n] n 2019-07-02 16:32:01: J52843c [INFO] Stopping monitoring for MemSQL node 143B60B39072A9E9C16D2D8ACC16E6997286079E 2019-07-02 16:32:01: J52843c [INFO] No longer monitoring MemSQL node 143B60B39072A9E9C16D2D8ACC16E6997286079E ``` **Note**: If a leaf is removed unintentionally, connect to the database and run `ADD LEAF` to add it back to the cluster: For example, on the Master Aggregator: `memsql> ADD LEAF 'root'@'<>':3307` ## Confirm Unmonitored Nodes Use MemSQL Ops to confirm that all SingleStore nodes are no longer monitored. ```shell /path/to/memsql-ops/memsql-ops memsql-list ``` ```output No MemSQL nodes were found. ``` ## Remove MemSQL Ops ```shell /path/to/memsql-ops/memsql-ops agent-uninstall --uninstall-ops-only --all ``` The `--uninstall-ops-only` flag ensures that the MemSQL Ops agent will be uninstalled without deleting any node data. * *View Remove MemSQL Ops Output* ``` This will completely delete all MemSQL Ops agents in the cluster, including this one. This operation will delete all MemSQL Ops data in the cluster and cannot be undone. If you want to delete all MemSQL Ops agents, please type the word DELETE: DELETE 2019-09-06 13:58:23: J91ad35 [INFO] Deleting MemSQL Ops agent Ae9f037643ccf4b5fb3b0c03f2dd0da58 2019-09-06 13:58:23: J4cc070 [INFO] Deleting MemSQL Ops agent Ae2838b5fd2164d3a8691ad71227beee3 2019-09-06 13:58:23: Jfb011f [INFO] Deleting MemSQL Ops agent Afea25a7c3e184241acdf5296f429298d 2019-09-06 13:58:27: J4cc070 [INFO] Successfully deleted MemSQL Ops agent Ae2838b5fd2164d3a8691ad71227beee3 2019-09-06 13:58:27: Jfb011f [INFO] Successfully deleted MemSQL Ops agent Afea25a7c3e184241acdf5296f429298d 2019-09-06 13:58:27: J91ad35 [INFO] Successfully deleted MemSQL Ops agent Ae9f037643ccf4b5fb3b0c03f2dd0da58 WARNING: An unmonitored MemSQL node may still exist on this machine. It had host 172.16.212.165 and port 3306 the last time MemSQL Ops connected to it. Stopping MemSQL Ops Waiting up to 60 seconds for a clean exit. Done. Successfully uninstalled MemSQL Ops ``` *** Modified at: March 14, 2024 Source: [/db/v9.1/user-and-cluster-administration/migrate-from-ops-to-tools/non-sudo/migrate-cluster/](https://docs.singlestore.com/db/v9.1/user-and-cluster-administration/migrate-from-ops-to-tools/non-sudo/migrate-cluster/) (An index of the documentation is available at /llms.txt)