Leaf States

Unknown

In this state, a leaf is not part of the cluster.

This is the state before you run ADD LEAF to introduce the leaf into the system. A leaf that is unknown will not show up in SHOW LEAVES. The REMOVE LEAF command will transition a leaf into the unknown state.

Online

This is the default, healthy state of a leaf. In the online state, the leaf is an active member of the distributed system and is either currently serving or ready to serve data to the aggregators.

Offline

The master aggregator periodically sends a heartbeat (ping) to all the nodes in a cluster to determine if they are responsive and online. A leaf enters the offline state if the master aggregator cannot reach it. The heartbeat frequency is based on typical network latencies and node responsiveness. It is set to a default of 150ms which has been empirically determined.

If the cluster is in redundancy 2 any partitions on the offline node will be failed over to the leaves pair. The master aggregator continues to ping offline nodes to detect when they should be moved into the attaching state.

See for failover cases.

Recovering

Not online, and not available for read or write queries. A leaf in recovering state has been restarted and is replaying data back into memory.

Detached

In this state, the leaf is detached from the cluster.

A leaf transitions from online to detached when DETACH LEAF is run. You can reintroduce the leaf into the online state with the ATTACH LEAF command. This command will automatically try to reuse the data present on the leaf; this is especially important in redundancy-1. DETACH LEAF is the same as REMOVE LEAF except that after rebalancing away a leaf’s partitions, it transitions it to detached instead of unknown.

Attaching

A leaf transitions from offline to attaching when it is once again reachable by heartbeats.

The master aggregator will automatically run ATTACH LEAF to attach any partitions on the leaf back into the cluster. While the attach is running the leaf will stay in the attaching state. When the attach is finished the leaf will move to the online state.