Tools: Leaf Node Failures

Replace a Failed Leaf Node in a Redundancy-1 Cluster

This section covers the replacement of a failed leaf node within a redundancy-1 cluster. In this example, it is assumed that data can be recovered from the failed leaf node, which will then be restored.

1. Determine if the leaf node in question is still running within the cluster. If it is, first stop it, and then remove the leaf role from the node.

   Shell

   Copy

   sdb-admin stop-node --memsql-id <node-ID>

   Shell

   Copy

   sdb-admin remove-leaf --force --memsql-id <node-ID>

2. Assuming that the data is recoverable from the failed leaf node, preserve it by changing to the node directory (on the corresponding host) and compressing it to another location, such as the /tmp directory.

   Shell

   Copy

   cd /path/to/memsql/<node-directory>

   Shell

   Copy

   sudo tar -zcvf /tmp/data.tgz ./data/

3. As the leaf node’s data has been preserved, delete the leaf node from the cluster.

   Shell

   Copy

   sdb-admin delete-node --memsql-id <node-ID> --skip-remove

4. Create a new node to store this data in. After the node’s been created, stop it so that you can manipulate the node’s data directory.

   Shell

   Copy

   sdb-admin create-node --host [IP or hostname OF NODE HOST] --port [PORT OF NODE HOST] --password [PASSWORD OF NODE]

   Shell

   Copy

   sdb-admin stop-node --memsql-id <node-ID>

5. Navigate to the newly created node’s directory and remove the data directory within. Then, extract the data from the /tmp directory to this node’s data directory.

   Shell

   Copy

   cd /var/lib/memsql/<new-node-directory>

   Shell

   Copy

   sudo rm -r ./data

   Shell

   Copy

   sudo tar -zxvf /tmp/data.tgz

6. Once the extraction is finished, update the ownership and permissions on the directory’s files.

   Shell

   Copy

   sudo chown memsql:memsql -R ./data

7. Start the new node. The MemSQL ID will now be that of the former node, which you can confirm with the following command.

   Shell

   Copy

   sdb-admin start-node --all

8. Run the following SQL command on the Master Aggregator host to add this leaf node to the cluster.

   SQL

   Copy

   ADD LEAF <user>:'<password>'@'<node's-host-IP or hostname>':<port>;

Replace a Failed Leaf Node in a Redundancy-2 Cluster

This section details how to recover a failed leaf node in a high availability cluster.

Clusters using Async or Sync replication could potentially present different errors, so knowing which replication option that is being used will be key to recovering a failed cluster.

By default, all MemSQL v7.0/SingleStoreDB 7.1 or later clusters will run with high availability using sync replication. Sync replication ensures that the shared data housed within the primary and secondary leaf nodes is always in sync.

Async replication treats data differently and, as its first priority, allows the cluster to continue to run. While data will eventually be synced between the leaf nodes, it is not a requirement that it happen at the time of the transaction. Refer to Replication and Durability for more information.

Sometimes data can become out of sync when one or more sizable queries are run. If failover occurs during one of these queries, SingleStoreDB will recognize that it’s failing over from a leaf node that contains newer data. As the asynchronous replication of data on the secondary leaf node was not up to date when the failover occurred, there will be an unavoidable amount of data loss.

As a result, the cluster will stop performing transactions and will throw an Asynchronous replication with a FailoverLeaf: Couldn’t failover from node. error. This is intentional, as it affords the cluster administrator to take manual control of the cluster to assess what has occurred.

To acknowledge the potential data loss and manually move the data to the secondary leaf node that is now out of sync, use a SQL editor to run the REBALANCE PARTITIONS command on each database. This will prevent any potential data loss from the leaf nodes being out of sync.

Alert: If REBALANCE PARTITIONS is run before attempting to retain or fix the leaf/partition, the data will be lost and will no longer be recoverable.

REBALANCE PARTITIONS ON db-name;

You may also use the bash script in the One Leaf Node Fails section below to rebalance the partitions on all nodes and restore redundancy on all databases.

One Leaf Node Fails

A Pair of Leaf Nodes Fail

When a pair of leaf nodes fail, their partitions will no longer have any remaining instances, which effectively takes these partitions offline for both reads and writes.

If either of the leaf nodes’ hosts, or a leaf node’s data on a host, is recoverable, a failed leaf node can be reintroduced and reattached to its partitions by following the steps in the Replace a Failed Leaf Node in a Redundancy-1 Cluster section. After following those steps, the partitions will be back online for both reads and writes.

If neither leaf node’s host is recoverable, then data loss has occurred. You must now add replacement leaf nodes and run REBALANCE PARTITIONS ... FORCE to create new (empty) replacement partitions. This can be done by following the steps in the Replace a Leaf Node in a Redundancy-2 Cluster section.

Many Unpaired Leaf Nodes Fail

So long as two paired leaf nodes have not failed, all partitions are still available for reads and writes.

In certain circumstances, all of the leaf nodes in one availability group can fail, but no data loss will be incurred so long as redundancy is restored before another leaf node fails in the remaining availability group.

Many Leaf Nodes Fail, Some of them Paired

When both leaf nodes in a pair fail, every partition that is hosted by these two leaf nodes will be offline to reads and writes.

When one leaf node of a pair fails, the partitions of its pair will remain online for reads and writes.

Offline partitions should be handled using the method detailed in the A Pair of Leaf Nodes Fail section. However, as both leaf nodes are unrecoverable, RESTORE REDUNDANCY or Cluster Downsizing Steps should only be run after all partitions have either been recovered or abandoned as lost data.