Ops: Leaf Node Failures

Replace a Dead Leaf in a Redundancy-1 Cluster

This guide shows how to replace a dead leaf in a redundancy-1 cluster. It is assumed that data can be recovered from the dead leaf and will be restored at Step 5.

1. Unmonitor the dead SingleStore Leaf and uninstall the dead MemSQL Ops Agent by executing the following commands:

   Shell

   Copy

   memsql-ops memsql-unmonitor <DEAD-LEAF-ID>

   Shell

   Copy

   memsql-ops agent-uninstall --force --agent-id <DEAD-AGENT-ID>

1. Remove the dead leaf from cluster by running REMOVE LEAF on the dead leaf to remove it from SHOW LEAVES and free up its pairing slot.

   SQL

   Copy

   REMOVE LEAF "<DEAD-LEAF>"[:<PORT>];

2. Deploy a new leaf by deploying a new MemSQL Ops Agent and leaf via the web UI, or via CLI. Once the agent is deployed, you can replace the settings.conf file from the dead agent:

   Shell

   Copy

   memsql-ops agent-deploy --host <HOST-IP> [--user <USER> --identity-file /path/to/id_rsa]

   Shell

   Copy

   memsql-ops agent-stop <NEW-AGENT-ID>

   Edit your settings.conf file at /var/lib/memsql-ops/settings.conf.

   Shell

   Copy

   memsql-ops agent-start <NEW-AGENT-ID>

   Shell

   Copy

   memsql-ops memsql-deploy --agent-id <NEW-AGENT-ID> --role leaf

3. Stop the new leaf to make sure the new SingleStore Leaf is NOT running before copying the recovered data:

   Shell

   Copy

   memsql-ops memsql-stop <NEW-LEAF-ID>

4. Copy the recovered data into the new leaf data directory. Make sure to save the memsql_id file.

   Shell

   Copy

   sudo mv /var/lib/memsql/leaf-3306/data/memsql_id /tmp

   Shell

   Copy

   sudo cp -r /path/to/recovered/leaf-3306/data/* /var/lib/memsql/leaf-3306/data/

   Shell

   Copy

   sudo mv /tmp/memsql_id /var/lib/memsql/leaf-3306/data/

   Shell

   Copy

   sudo chown -R memsql.memsql /var/lib/memsql/leaf-3306/data/

5. Restart the new leaf.

   Shell

   Copy

   memsql-ops memsql-start <NEW-LEAF-ID>

6. Partitions are currently present in the new leaf, but SingleStore distributed system is still unaware of them, so you must reattach the partitions by trigger partitions detection. Temporarily remove the new leaf from the cluster. MemSQL Ops will automatically re-attach the leaf, and this action will trigger detecting (and attaching) all partitions.

   On the Master Aggregator, run:

   SQL

   Copy

   REMOVE LEAF "<NEW-LEAF>";

   In the examples below, 10.0.0.101 is the new SingleStore Leaf, and 10.0.2.128 is an existing leaf.

   Before reattaching partitions:

   SQL

   Copy

   SHOW PARTITIONS ON `memsql_demo`;

   +---------+------------+------+--------+--------+
   | Ordinal | Host       | Port | Role   | Locked |
   +---------+------------+------+--------+--------+
   |       0 | 10.0.2.128 | 3306 | Master |      0 |
   |       1 | 10.0.2.128 | 3306 | Master |      0 |
   |       2 | NULL       | NULL | NULL   |      0 |
   |       3 | NULL       | NULL | NULL   |      0 |
   ...

   After reattaching partitions:

   SQL

   Copy

   SHOW PARTITIONS ON `memsql_demo`;

   +---------+------------+------+--------+--------+
   | Ordinal | Host       | Port | Role   | Locked |
   +---------+------------+------+--------+--------+
   |       0 | 10.0.2.128 | 3306 | Master |      0 |
   |       1 | 10.0.2.128 | 3306 | Master |      0 |
   |       2 | 10.0.0.101 | 3306 | Master |      0 |
   |       3 | 10.0.0.101 | 3306 | Master |      0 |
   ...

One Leaf Dies

Any partitions for which the dead leaf was the partition master will be promoted on the dead leaf’s pair.

You can reintroduce the dead leaf, or add a new leaf to replace it.

Reintroducing the leaf is the simplest solution – SingleStore will automatically reattach the leaf as soon as it is back online.

If you decide to add a replacement leaf on a different host, you can follow this guide on how to replace a dead leaf in a redundancy-2 Cluster.

1. Unmonitor the dead SingleStore Leaf and uninstall the dead MemSQL Ops Agent to remove the dead leaf from MemSQL Ops:

   Shell

   Copy

   memsql-ops memsql-unmonitor <DEAD-LEAF-ID>

   Shell

   Copy

   memsql-ops agent-uninstall --force --agent-id <DEAD-AGENT-ID>

   Note that leaf and agent ids can be retrieved with memsql-ops memsql-list and memsql-ops agent-list.

2. Figure out which availability group the failed leaf was in by running SHOW LEAVES on the master aggregator to identify the dead leaf and its availability group:

   SQL

   Copy

   show leaves;

   +----------------+------+--------------------+----------------+-----------+---------+--------------------+------------------------------+
   | Host           | Port | Availability_Group | Pair_Host      | Pair_Port | State   | Opened_Connections | Average_Roundtrip_Latency_ms |
   +----------------+------+--------------------+----------------+-----------+---------+--------------------+------------------------------+
   | 54.242.219.243 | 3306 |                  1 | 54.196.216.103 |      3306 | online  |                  1 |                        0.640 |
   | 54.160.224.3   | 3306 |                  1 | 54.234.29.206  |      3306 | online  |                  1 |                        0.623 |
   | 54.196.216.103 | 3306 |                  2 | 54.242.219.243 |      3306 | online  |                  1 |                        0.583 |
   | 54.234.29.206  | 3306 |                  2 | 54.160.224.3   |      3306 | offline |                  0 |                         NULL |
   +----------------+------+--------------------+----------------+-----------+---------+--------------------+------------------------------+

   In this example, the offline leaf was in availability group 2.

3. Remove the dead leaf from cluster by running REMOVE LEAF on the master aggregator to remove the dead leaf from SHOW LEAVES and free up its pairing slot.

   SQL

   Copy

   REMOVE LEAF "<DEAD-LEAF>"[:<PORT>] FORCE;

4. Deploy a new leaf by deploying a new MemSQL Ops Agent and leaf via the web UI, or via CLI. Once the agent is deployed, you can replace the settings.conf file from the dead agent:

   Shell

   Copy

   memsql-ops agent-deploy --host <HOST-IP> [--user <USER> --identity-file /path/to/id_rsa]

   Shell

   Copy

   memsql-ops agent-stop <NEW-AGENT-ID>

   Edit your settings.conf file at /var/lib/memsql-ops/settings.conf.

   Shell

   Copy

   memsql-ops agent-start <NEW-AGENT-ID>

   Shell

   Copy

   memsql-ops memsql-deploy --agent-id <NEW-AGENT-ID> --role leaf --availability-group <GROUP>

   After this has completed, SHOW LEAVES on the master aggregator should indicate that all leaves are online and paired:

   Copy

   +----------------+------+--------------------+----------------+-----------+--------+--------------------+------------------------------+
   | Host           | Port | Availability_Group | Pair_Host      | Pair_Port | State  | Opened_Connections | Average_Roundtrip_Latency_ms |
   +----------------+------+--------------------+----------------+-----------+--------+--------------------+------------------------------+
   | 54.242.219.243 | 3306 |                  1 | 54.196.216.103 |      3306 | online |                  2 |                        0.578 |
   | 54.160.224.3   | 3306 |                  1 | 54.145.52.142  |      3306 | online |                  2 |                        0.624 |
   | 54.196.216.103 | 3306 |                  2 | 54.242.219.243 |      3306 | online |                  2 |                        0.612 |
   | 54.145.52.142  | 3306 |                  2 | 54.160.224.3   |      3306 | online |                  1 |                        0.568 |
   +----------------+------+--------------------+----------------+-----------+--------+--------------------+------------------------------+

5. Finally, you need to rebalance the cluster, by running REBALANCE PARTITIONS for all your databases. On the Master Aggregator, you can run REBALANCE ALL DATABASES.

A Pair of Leaves Die

In this case, the partitions that were hosted on the dead leaves have no remaining instances. This means that these partitions are offline to reads and writes.

If either of the leaf machines (or at least its data) is recoverable, you can reintroduce it and reattach its partitions, as in the Leaf Failures In a Redundancy-1 Cluster section. At this point, the partitions are online for reads and writes. The failure scenario is now the same as the One Leaf Dies scenario in a redundancy-2 cluster.

If neither leaf machine is recoverable, then data loss has occurred. You can now add replacement leaves, after which you may run REBALANCE PARTITIONS ... FORCE to create new (empty) replacement partitions.

As long as two paired leaves have not both died, all partitions are still available for reads and writes.

As a special case of this scenario, all leaves in one availability group can be down. No data loss is incurred as long as redundancy is restored before an additional leaf from the remaining availability group dies.

Many Leaves Die, Some of Them Paired

Every partition for which both leaves hosting it died is now offline to reads and writes. Partitions for which only one leaf in the relevant pair died remain online.

Offline partitions should be handled as they are in the scenario A pair of leaves die. However, RESTORE REDUNDANCY or REBALANCE PARTITIONS should be run only after all partitions has either been either recovered or given up as lost data (because both leaves that were hosting its data are unrecoverable).