Sign inTry Free

Important

The SingleStore 9.1 release candidate (RC) is available now. It will be promoted to general availability (GA) soon and at that time it will be renamed to SingleStore 10.

In the interim, SingleStore 9.1 RC can be used to preview, evaluate, and provide feedback on the new and upcoming features in SingleStore 10, while SingleStore 9.0 is recommended for production workloads.

Recover from a Leaf Node Failure

Important

To prevent potential issues, the other availability group must be healthy before performing this operation.

Use the following steps to reintroduce a leaf node to a cluster where redundancyLevel is set to 2 with redundant leaf nodes.

  1. Scale down the Operator.

    kubectl scale deployment [operator deployment name] --replicas=0

  2. Scale down the STS with the pod you need to replace. For a leaf node in either of the availability groups, scale down the STS to 0 and then scale it back up to the number of pods in the group.

    kubectl scale statefulsets [StatefulSet Name] --replicas=0

  3. Delete the PersistentVolumeClaim (PVC) of the problematic pod.

    kubectl delete pvc pv-storage-node-ccd487dc-3b15-4c3b-88a2-a984dc0245ca-leaf-ag1-0

  4. Delete the pod.

    kubectl delete pod node-ccd487dc-3b15-4c3b-88a2-a984dc0245ca-leaf-ag1-0

  5. Scale the STS and Operator back up.

    kubectl scale statefulsets [StatefulSet Name] --replicas=[num of pods in this StatefulSet]

Last modified:

Was this article helpful?

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In