Recover from a Master Aggregator Failure

Prerequisites

Before proceeding, determine if your cluster is running with Master Aggregator High Availability (MA HA) enabled, as the recovery steps differ significantly. MA HA provides enhanced reliability and automatic failover capabilities for mission-critical workloads. Refer to Multi-Datacenter Failover for more information about MA HA.

Change Master Aggregator Disk with MA HA Enabled

If your cluster is running with Master Aggregator MA HA enabled, the recovery process is simplified. You only need to change the storage class:

1. Update the cluster custom resource with the new storage class in masterAggregatorSpec.storageClass.

2. Update the Master Aggregator StatefulSet's PersistentVolumeClaim (PVC) template with the desired storage class.

3. Delete the PersistentVolumeClaim (PVC) for the master-0 pod.

4. Delete the master-0 pod itself.

5. The disk will be recreated with the new storage class by the StatefulSet controller. The promoted child aggregator continues to serve as the active Master during the operation.

Recover from MA Failure without MA HA

If MA HA is not enabled, perform the following steps to recover from a Master Aggregator failure.

1. To prevent the risk of disruptive actions when fixing the cluster, turn off the Operator by setting the replicas field to 0 in the sdb-operator.yaml file and apply the change to the cluster.

2. Determine which of the child aggregators is furthest ahead in data replication using the following SQL command.

   SQL

   Copy

   SHOW DATABASE EXTENDED

   From the output, review the position column for the cluster database on every node and select the child aggregator where:

   • The position is highest

   • All reference databases are not in an unrecoverable state

   • The position for all databases does not equal 0:0

3. Set the identified child aggregator as the Master Aggregator using the following SQL command.

   SQL

   Copy

   AGGREGATOR SET AS MASTER;

4. Ensure that the node on master-0 is emptied out. Depending on what occurred when the disk became corrupted, this may require re-emptying the master-0 volume.

   • If you attempted to start master-0 with an empty volume when the Operator was on, the Operator bootstraps it as a Master Aggregator. This would have resulted in a new single-node cluster that must be emptied out.

   • If you did not start master-0 when the Operator was on, then only its storage must be emptied out. When master-0 is started as a new empty node, it will not be bootstrapped as the Master Aggregator as the Operator is not running.

5. Remove the former Master Aggregator and clear its metadata by running the following SQL command on the temporary Master Aggregator.

   SQL

   Copy

   REMOVE AGGREGATOR '...-master-0';

6. Optional: If you need to change the Master Aggregator storage class after promoting a child aggregator to master, refer to Change Master Aggregator Disk with MA HA Enabled.

7. Re-add this child aggregator. This will add the empty node to the cluster as a child aggregator.

   SQL

   Copy

   ADD AGGREGATOR '...-master-0';

8. Promote the child aggregator to turn this node in the master StatefulSet into the new Master Aggregator.

   SQL

   Copy

   PROMOTE AGGREGATOR '...-master-0' TO MASTER;

9. Turn on the Operator by setting the replicas field to 1 in the sdb-operator.yaml file and apply the change to the cluster.