When taking leaves nodes offline without cluster downtime:
1. What happens if an ingestion is in progress?
It will fail on the leaf that went offline, and get retried on the leaf to which the faileover occurred.
2. How can you be sure that the data will be consistent when a leaf node fails over?
Data will be consistent in synchronous replication. No transactions get committed until they have been replicated.
3. Do you need to pause the replication for stopping a leaf node, child aggregator, or master aggregator?
No, it’s not even possible to do so.
4. Does synchronous/asynchronous replication make any difference in this process?
Yes, asynchronous replication is not recommended. Synchronous replication provides stronger data consistency.
5. When performing a rolling restart of multiple leaf nodes, do you need to rebalance after restarting each leaf node?
No, it is not required.
6. How does a long running query or a running backup impact this operation?
The failover will not be blocked. However the long running queries or backups will fail when the failover happens.