Leaf States

Each leaf is in one of the following states:

State

Description

Unknown

In this state, a leaf is not part of the workspace.

Online

This is the default, healthy state of a leaf. In the online state, the leaf is an active member of the distributed system and is either currently serving or ready to serve data to the aggregators.

Offline

The master aggregator periodically sends a heartbeat (ping) to all the nodes in a workspace to determine if they are responsive and online. A leaf enters the offline state if the master aggregator cannot reach it. The heartbeat frequency is based on typical network latencies and node responsiveness. It is set to a default of 150ms which has been empirically determined.

If the workspace is in redundancy 2 any partitions on the offline node will be failed over to the leaves pair. The master aggregator continues to ping offline nodes to detect when they should be moved into the attaching state.

See How Failover is Triggered in HA for failover cases.How Failover is Triggered in HA

Recovering

Not online, and not available for read or write queries. A leaf in recovering state has been restarted and is replaying data back into memory.

Detached

In this state, the leaf is detached from the workspace.

Attaching

A leaf transitions from offline to attaching when it is once again reachable by heartbeats.

The following diagram summarizes the leaf states and the transitions between them.

16083b57986b1b.png