Sign inTry Free

Taking Leaves Offline with Cluster Downtime

Steps for Offline Maintenance

Step 1: Prior to performing any offline maintenance, SingleStore recommends that you take a backup as a standard precautionary measure. Refer to Back Up and Restore Data.

Step 2: Run the following commands from the master aggregator :

SHOW LEAVES

SHOW AGGREGATORS

SHOW CLUSTER STATUS

EXPLAIN RESTORE REDUNDANCY

EXPLAIN REBALANCE PARTITIONS

From the output of these commands, confirm that the following are true.

  • All leaves are online.

  • All aggregators are online.

  • There are no partitions with the “orphan” role.

REBALANCE or RESTORE REDUNDANCY commands need not be run.

Step 3: From the master aggregator, run the SNAPSHOT DATABASE command for each database.

Step 4: From the master aggregator, run the _SYNC_SNAPSHOT <databasename>; command for each database

This command makes sure that a snapshot is triggered on the replica and the replica also has a successful snapshot.

Step 5: Use sdb-admin stop-node to stop all the SingleStore nodes in the following sequence: first the master aggregator, then child aggregators, and finally the leaves.

If you are not able to stop the nodes in the specified order then you need to run SET GLOBAL leaf_failure_detection=OFF; from the master aggregator before stopping the nodes.

Step 6: Perform the maintenance tasks on the hosts as required.

Step 7: Use sdb-admin start-node to start all the SingleStore nodes in the following sequence: first, the leaf nodes, then the child aggregators, and finally the master aggregator. Note that the leaf nodes must be fully recovered before the child aggregators are started.

If you had turned leaf_failure_detection off in step 5, then run SET GLOBAL leaf_failure_detection=ON; from the master aggregator.

Last modified: July 30, 2024

Was this article helpful?

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK