Configuring SingleStore for NUMA

Typically, multi-processor hosts provide for non-uniform memory access, or NUMA. In a NUMA configuration, the hosts’s motherboard has many physical processors on the same physical board with multiple banks of memory, and memory is designated as being “local” to an assigned processor. From that processor’s standpoint, accessing this local memory is much faster than accessing memory that is considered local to a different processor (hence the "non-uniform"). For hosts that allow NUMA, SingleStore recommends running a node per physical processor in conjunction with CPU pinning. Refer to Configure CPU Pinning with NUMA for more information.

When running SingleStore on hosts with multiple NUMA nodes, you should configure SingleStore for NUMA with numactl for optimal performance. If you do not configure SingleStore this way, performance will be greatly degraded due to expensive cross-NUMA-node memory access. Configuring for NUMA should be done as part of the installation process; however, you can reconfigure your deployment later, if necessary.

To simplify the configuration process, the SingleStore management tools (sdb-admin, memsqlctl, etc.) can automatically detect if a host host has multiple NUMA nodes and then configure SingleStore with numactl to bind individual SingleStore nodes to NUMA nodes. To download and use the management tools, see the deployment guides.

Configuring a new cluster for NUMA

  1. On each host that will contain SingleStore leaf nodes, create one leaf per NUMA node by running sdb-admin create-node and then sdb-admin add-leaf until you have the same number of SingleStore leaf nodes as NUMA nodes. You will need to provide a different --port argument for each SingleStore node on the same host.

    For example, on a host with two NUMA nodes, you would deploy two leaf nodes.

    sdb-admin create-node --host <hostname|IP> --port 3307 --password <secure_pass>
    sdb-admin create-node --host <hostname|IP> --port 3308 --password <secure_pass>
    # Assign the nodes the role of "leaf" and add them to the cluster.
    sdb-admin list-nodes -q -r unknown | xargs -I % sdb-admin add-leaf --memsql-id % -y
  2. Run sdb-admin optimize and confirm the prompt.

    sdb-admin optimize

    This will configure numactl to bind SingleStore nodes to NUMA nodes, and configure SingleStore memory settings.

    See sdb-admin optimize for more information.

  3. Restart the cluster for the configuration changes to take effect.

    sdb-admin restart-node --all

Reconfiguring an existing cluster for NUMA

  1. Follow the steps in the previous section to deploy and optimize your leaf nodes.

  2. Rebalance all SingleStore databases by running REBALANCE ALL DATABASES.

REBALANCE ALL DATABASES;

Last modified: October 31, 2023

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK