# Taking Leaves Offline with Cluster Downtime

##### **Steps for Offline Maintenance**

**Step 1:** Prior to performing any offline maintenance, SingleStore recommends that you take a backup as a standard precautionary measure. Refer to [Back Up and Restore Data](https://docs.singlestore.com/db/v9.0/manage-data/back-up-and-restore-data.md).

**Step 2:** Run the following commands from the master aggregator :

[SHOW LEAVES](https://docs.singlestore.com/db/v9.0/reference/sql-reference/cluster-management-commands/show-leaves.md)

[SHOW AGGREGATORS](https://docs.singlestore.com/db/v9.0/reference/sql-reference/cluster-management-commands/show-aggregators.md)

[SHOW CLUSTER STATUS](https://docs.singlestore.com/db/v9.0/reference/sql-reference/cluster-management-commands/show-cluster-status.md)

[EXPLAIN RESTORE REDUNDANCY](https://docs.singlestore.com/db/v9.0/reference/sql-reference/cluster-management-commands/explain-restore-redundancy.md)

[EXPLAIN REBALANCE PARTITIONS](https://docs.singlestore.com/db/v9.0/reference/sql-reference/cluster-management-commands/explain-rebalance-partitions.md)

From the output of these commands, confirm that the following are true.

* All leaves are online.
* All aggregators are online.
* There are no partitions with the “orphan” role.

`REBALANCE` or `RESTORE REDUNDANCY` commands need not be run.

**Step 3:** From the master aggregator, run the [SNAPSHOT DATABASE](https://docs.singlestore.com/db/v9.0/reference/sql-reference/operational-commands/snapshot-database.md) command for each database.

**Step 4:** From the master aggregator, run the `_SYNC_SNAPSHOT <databasename>;` command for each database

This command makes sure that a snapshot is triggered on the replica and the replica also has a successful snapshot.

**Step 5:** Use [sdb-admin stop-node](https://docs.singlestore.com/db/v9.0/reference/singlestore-tools-reference/sdb-admin-commands/stop-node.md) to stop all the SingleStore nodes in the following sequence: first the master aggregator, then child aggregators, and finally the leaves.

If you are not able to stop the nodes in the specified order then you need to run `SET GLOBAL leaf_failure_detection=OFF;` from the master aggregator before stopping the nodes.

**Step 6:** Perform the maintenance tasks on the hosts as required.

**Step 7:** Use [sdb-admin start-node](https://docs.singlestore.com/db/v9.0/reference/singlestore-tools-reference/sdb-admin-commands/start-node.md) to start all the SingleStore nodes in the following sequence: first, the leaf nodes, then the child aggregators, and finally the master aggregator. Note that the leaf nodes must be fully recovered before the child aggregators are started.

If you had turned `leaf_failure_detection` off in step 5, then run `SET GLOBAL leaf_failure_detection=ON;` from the master aggregator.

This procedure applies to self-managed deployments managed with SingleStore Tools. For Kubernetes operator-managed clusters, refer to [Taking Leaves Offline on Kubernetes with Cluster Downtime](https://docs.singlestore.com/#section-id235861317546066.md) instead of the `sdb-admin` commands.

## Taking Leaves Offline on Kubernetes with Cluster Downtime

Use this procedure only for clusters managed by the SingleStore Kubernetes Operator. For maintenance that can be performed without taking the cluster offline, use the following steps:

## Prerequisites

* Confirm that the cluster is healthy and that all required backups and maintenance approvals are complete.

  * Check the `MemsqlCluster` phase:
    ```shell
    kubectl get memsql <cluster-name> -n <namespace> -o jsonpath="{.status.phase}"
    ```
  * The phase must be Running.
  * Confirm that your most recent backup completed according to your backup policy. Query the backup history from the MA:
    ```sql
    SELECT database_name, status, start_timestamp, end_timestamp
    FROM information_schema.mv_backup_history
    WHERE status = 'Success'
    ORDER BY start_timestamp DESC;
    ```
  * Refer to [MV\_BACKUP\_HISTORY](https://docs.singlestore.com/db/v9.0/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-history.md) for more information on columns.
* Confirm that the availability zone has enough spare capacity to reschedule pods displaced from the node under maintenance, or that your cluster autoscaler can provision additional capacity on demand. The exact verification steps depend on your Kubernetes environment and autoscaler configuration.
* Do not modify the `MemsqlCluster` custom resource or change the Operator version during the maintenance window. The Operator can defer and replay specification changes, replace `StatefulSets`, or restart pods while the cluster is recovering.

## Scale the Cluster Down

This procedure takes the cluster fully offline. Once all SingleStore pods are scaled to zero, you can perform maintenance on the Kubernetes nodes that hosted them and bring the cluster back up on the same pool of hosts. No additional spare capacity is required.

1. Scale the Operator deployment to zero replicas so that it does not reconcile the temporary `StatefulSet` changes during maintenance.
   ```shell
   kubectl scale deployment <operator-deployment> --replicas=0
   ```

2. Scale the node `StatefulSets` to zero replicas directly with `kubectl`; do not use the `MemsqlCluster` custom resource for this temporary maintenance operation.
   ```shell
   kubectl scale statefulset <statefulset-name> --replicas=0
   ```
   Repeat this for the Master Aggregator (MA), child aggregators, and leaf nodes in the order confirmed for your Operator version.

3. Perform the required host maintenance.

## Bring the Cluster Back Online

1. Bring the MA `StatefulSet` online first using `kubectl`. The Operator startup probe attaches nodes by querying the MA.The MA must be available before dependent nodes can start normally.
   ```shell
   kubectl scale statefulset <master-aggregator-statefulset> --replicas=<ma_count>
   ```
   Set `<ma_count>` to the value of `spec.masterAggregatorSpec.count` in the `MemsqlCluster` resource: 1 for a single-MA cluster or 3 when MA HA (MA consensus) is enabled. For leaves and child aggregators, use the counts from `spec.leafSpec.count` and `spec.aggregatorSpec.count`, respectively.

2. Bring the `StatefulSets` back online in the following order:

   * For redundancy level 2, scale both leaf `StatefulSets` (one per availability group) to `spec.leafSpec.count`. For redundancy level 1, scale the single leaf `StatefulSet`.
     ```shell
     kubectl scale statefulset <leaf-ag1-statefulset> --replicas=<leaf-count>

     kubectl scale statefulset <leaf-ag2-statefulset> --replicas=<leaf-count>
     ```
   * Scale the child aggregator `StatefulSet` to `spec.aggregatorSpec.count`.
     ```shell
     kubectl scale statefulset <child-aggregator-statefulset> --replicas=<ca-count>
     ```
   * After the cluster is healthy, scale the Operator deployment back up:
     ```shell
     kubectl scale deployment <operator-deployment> --replicas=1
     ```

Do not change the `MemsqlCluster` specification or the Operator version until recovery is complete and the cluster reports healthy.

## Online Maintenance Alternative

For Kubernetes clusters, prefer an online maintenance workflow when the maintenance task permits it. The `NoSchedule` taint evicts SingleStore pods from the tainted Kubernetes node, and the Operator reschedules them onto other Kubernetes nodes in the same AZ. Spare capacity in the AZ is required so the Operator has somewhere to place the evicted pods.

1. Confirm that spare capacity is available for pod rescheduling.

2. Apply a `NoSchedule` taint to the Kubernetes node being maintained. Use the taint key and value appropriate for your environment.
   ```shell
   kubectl taint node <kubernetes-node-name> <key>=<value>:NoSchedule
   ```

3. Trigger the Operator-managed rolling restart by updating an annotation under the node specification's `objectMetaOverrides.annotations` field. Add this annotation to each spec section that contains nodes: `masterAggregatorSpec`, `childAggregatorSpec`, and `leafSpec`. To perform maintenance across all Kubernetes nodes in the cluster, add the annotation to all three sections so the Operator rolls each node type in turn.
   ```yaml
   spec:
     masterAggregatorSpec:
       objectMetaOverrides:
         annotations:
           <restart-annotation-key>: "<unique-value>"
     childAggregatorSpec:
       objectMetaOverrides:
         annotations:
           <restart-annotation-key>: "<unique-value>"
     leafSpec:
       objectMetaOverrides:
         annotations:
           <restart-annotation-key>: "<unique-value>"
   ```

4. Wait for the Operator-managed rollout to finish and verify that the cluster is healthy before removing the taint.

The Operator detects the spec change and performs a rolling restart of the affected pods, keeping the cluster online. Persistent volumes are preserved across the restart.The annotation key is arbitrary. The Operator triggers the restart when the annotation value changes, regardless of the key. Select a key such as `maintenance-annotation-key` and update its value for each maintenance run. For example:

```yaml
spec:
  leafSpec:
    objectMetaOverrides:
      annotations:
        maintenance-annotation-key: "2026-10-23-run-1"
```

For the next maintenance run, change the value (for example, to `"2026-10-24-run-1"`) to trigger another rolling restart.

***

Modified at: October 27, 2026

Source: [/db/v9.0/user-and-cluster-administration/maintain-your-cluster/taking-leaves-offline-with-cluster-downtime/](https://docs.singlestore.com/db/v9.0/user-and-cluster-administration/maintain-your-cluster/taking-leaves-offline-with-cluster-downtime/)

(An index of the documentation is available at /llms.txt)