Overview of Leaf Node Failover

This is a general overview of what happens when a leaf node failover occurs:

1. Connections to the failed node are flushed.

2. Slave partitions on the failed node are detached.

3. The slave partitions on the paired node are promoted and repointed as master partitions.

4. The failed node is detached from the cluster.

5. The node is flagged as "offline" in information_schema.mv_events. Offline in this case means either the node was literally offline, or simply the master could not talk to that node and failover was initiated.

6. Once the master aggregator can establish heartbeat connections, complete recovery of the node(s) typically takes a few minutes, depending on partition count and data size, etc.

While there are leaves offline in the cluster the paired node will serve content and respond to connections for all of its partitions which are all now master partitions if failover succeeds. For failover to replica nodes to be successful, the slave partitions must be in sync with the master partitions.

During this entire process, the tracelog reflects the view of the node that's logging them. From a single node's perspective, they may see something that constitutes an error, but that does not necessarily reflect the state of the system as a whole, since this is a distributed database. Failures in communication to other nodes/systems are conservatively logged as errors, even though the system might be healthy and online, and the error came from a temporary network issue.

System health is extremely context-dependent, and in a distributed system there is a lot of self-healing going on. Hence, despite the logged error messages, there are very few cases where the system is unhealthy. Sometimes during coordinated operations, some errors may be logged because other nodes/threads not involved in that operation may see some temporary side effects, and throw out error messages.SingleStore’s general recommendation is to pay attention to errors only if they keep persisting, and it's fine to ignore errors that show up temporarily, especially during some maintenance operations.

For a listing of some of the typical error/warning messages, refer Failover Related Messages in Tracelog.