Taking Leaves Offline with Cluster Downtime

Taking Leaves Offline on Kubernetes with Cluster Downtime

Use this procedure only for clusters managed by the SingleStore Kubernetes Operator. For maintenance that can be performed without taking the cluster offline, use the following steps:

Prerequisites

Confirm that the cluster is healthy and that all required backups and maintenance approvals are complete.
- Check the MemsqlCluster phase:
  Shell
```
kubectl get memsql <cluster-name> -n <namespace> -o jsonpath="{.status.phase}"
```
- The phase must be Running.
- Confirm that your most recent backup completed according to your backup policy. Query the backup history from the MA:
  SQL
```
SELECT database_name, status, start_timestamp, end_timestamp
FROM information_schema.mv_backup_history
WHERE status = 'Success'
ORDER BY start_timestamp DESC;
```
- Refer to MV_BACKUP_HISTORY for more information on columns.
Confirm that the availability zone has enough spare capacity to reschedule pods displaced from the node under maintenance, or that your cluster autoscaler can provision additional capacity on demand. The exact verification steps depend on your Kubernetes environment and autoscaler configuration.
Do not modify the MemsqlCluster custom resource or change the Operator version during the maintenance window. The Operator can defer and replay specification changes, replace StatefulSets, or restart pods while the cluster is recovering.

Scale the Cluster Down

This procedure takes the cluster fully offline. Once all SingleStore pods are scaled to zero, you can perform maintenance on the Kubernetes nodes that hosted them and bring the cluster back up on the same pool of hosts. No additional spare capacity is required.

Scale the Operator deployment to zero replicas so that it does not reconcile the temporary StatefulSet changes during maintenance.
Shell
```
kubectl scale deployment <operator-deployment> --replicas=0
```
Scale the node StatefulSets to zero replicas directly with kubectl; do not use the MemsqlCluster custom resource for this temporary maintenance operation.
Shell
```
kubectl scale statefulset <statefulset-name> --replicas=0
```
Repeat this for the Master Aggregator (MA), child aggregators, and leaf nodes in the order confirmed for your Operator version.
Perform the required host maintenance.

Bring the Cluster Back Online

Bring the MA StatefulSet online first using kubectl. The Operator startup probe attaches nodes by querying the MA.The MA must be available before dependent nodes can start normally.
Shell
```
kubectl scale statefulset <master-aggregator-statefulset> --replicas=<ma_count>
```
Set <ma_count> to the value of spec.masterAggregatorSpec.count in the MemsqlCluster resource: 1 for a single-MA cluster or 3 when MA HA (MA consensus) is enabled. For leaves and child aggregators, use the counts from spec.leafSpec.count and spec.aggregatorSpec.count, respectively.
Bring the StatefulSets back online in the following order:
- For redundancy level 2, scale both leaf StatefulSets (one per availability group) to spec.leafSpec.count. For redundancy level 1, scale the single leaf StatefulSet.
  Shell
```
kubectl scale statefulset <leaf-ag1-statefulset> --replicas=<leaf-count>

kubectl scale statefulset <leaf-ag2-statefulset> --replicas=<leaf-count>
```
- Scale the child aggregator StatefulSet to spec.aggregatorSpec.count.
  Shell
```
kubectl scale statefulset <child-aggregator-statefulset> --replicas=<ca-count>
```
- After the cluster is healthy, scale the Operator deployment back up:
  Shell
```
kubectl scale deployment <operator-deployment> --replicas=1
```

Do not change the MemsqlCluster specification or the Operator version until recovery is complete and the cluster reports healthy.

Online Maintenance Alternative

For Kubernetes clusters, prefer an online maintenance workflow when the maintenance task permits it. The NoSchedule taint evicts SingleStore pods from the tainted Kubernetes node, and the Operator reschedules them onto other Kubernetes nodes in the same AZ. Spare capacity in the AZ is required so the Operator has somewhere to place the evicted pods.

Confirm that spare capacity is available for pod rescheduling.
Apply a NoSchedule taint to the Kubernetes node being maintained. Use the taint key and value appropriate for your environment.
Shell
```
kubectl taint node <kubernetes-node-name> <key>=<value>:NoSchedule
```
Trigger the Operator-managed rolling restart by updating an annotation under the node specification's objectMetaOverrides.annotations field. Add this annotation to each spec section that contains nodes: masterAggregatorSpec, childAggregatorSpec, and leafSpec. To perform maintenance across all Kubernetes nodes in the cluster, add the annotation to all three sections so the Operator rolls each node type in turn.
YAML
```
spec:
  masterAggregatorSpec:
    objectMetaOverrides:
      annotations:
        <restart-annotation-key>: "<unique-value>"
  childAggregatorSpec:
    objectMetaOverrides:
      annotations:
        <restart-annotation-key>: "<unique-value>"
  leafSpec:
    objectMetaOverrides:
      annotations:
        <restart-annotation-key>: "<unique-value>"
```
Wait for the Operator-managed rollout to finish and verify that the cluster is healthy before removing the taint.

The Operator detects the spec change and performs a rolling restart of the affected pods, keeping the cluster online. Persistent volumes are preserved across the restart.The annotation key is arbitrary. The Operator triggers the restart when the annotation value changes, regardless of the key. Select a key such as maintenance-annotation-key and update its value for each maintenance run. For example:

YAML

spec:
  leafSpec:
    objectMetaOverrides:
      annotations:
        maintenance-annotation-key: "2026-10-23-run-1"

For the next maintenance run, change the value (for example, to "2026-10-24-run-1") to trigger another rolling restart.