Platform and Kernel

Our recommended platforms are:

• Red Hat Enterprise Linux (RHEL) / AlmaLinux 7 or later

• Debian 8 or 9 (version 9 is preferred) / Ubuntu 14.04 and later

CentOS 6, 7, and 8 are supported but are not recommended for new deployments. CentOS 8 has reached end of life and CentOS has discontinued numbered releases.

When provisioning your hosts, the minimum Linux kernel version required is 3.10 or later. For SingleStore 8.1 and later, glibc 2.17 or later is also required.

Configure File Descriptor and Maximum Process Limits

A SingleStore cluster uses a substantial number of client and server connections between aggregators and leaf nodes to run queries and cluster operations. SingleStore recommends setting the Linux file descriptor and maximum process limits to the values listed below to account for these connections. Failing to increase this limit can significantly degrade performance and even cause connection limit errors. The ulimit settings can be configured in the /etc/security/limits.conf file, or directly via shell commands.

Permanently increase the open files limit and the max user processes limit for the memsql user by editing the /etc/security/limits.conf file as the root user and adding the following lines:

memsql   soft   NOFILE   1024000
memsql   hard   NOFILE   1024000
memsql   soft   nproc    128000
memsql   hard   nproc    128000

The file-max setting configures the maximum number of file handles (file descriptor limit) for the entire system. On the contrary, ulimit settings are only enforced on a process level. Hence, the file-max value must be higher than the NOFILE setting. Increase the maximum number of file handles configured for the entire system in /proc/sys/fs/file-max. To make the change permanent, append or modify the fs.file-max line in the /etc/sysctl.conf file.

Configure the Linux nice Setting

Given how the Linux kernel calculates the maximum nice limit, SingleStore recommends that you modify the /etc/security/limits.conf file and set the maximum nice limit to -10 on each Linux host in the cluster. This will allow the SingleStore engine to run some threads at higher priority, such as the garbage collection threads.

To apply this new nice limit, restart each SingleStore node in the cluster.

Alternatively, you may set the default nice limit to -10 on each Linux host in the cluster prior to deploying SingleStore.

Configure Linux ulimit Settings

Most Linux operating systems provide ways to control the usage of system resources such as threads, files and network at an individual user or process level. The per-user limitations for resources are called ulimits, and they prevent single users from consuming too much system resources. For optimal performance, SingleStore recommends setting ulimits to higher values than the default Linux settings. The ulimit settings can be configured in the /etc/security/limits.conf file, in the /etc/security/limits.d file, or directly via shell commands.

Configure Linux vm Settings

SingleStore recommends letting first-party tools, such as sdb-admin and memsqlctl, configure your vm settings to minimize the likelihood of getting memory errors on your hosts. The default values used by the tools are the following:

• vm.max_map_count set to 1000000000

• vm.overcommit_memory set to 0WARNING: vm.overcommit_memory should be set to 0. Using values other than 0 is recommended only for systems with swap areas larger than their physical memory. Please consult your distribution documentation.

• vm.overcommit_ratio ignore, unless vm.overcommit_memory is set to 2, in which case, set this to 99. See the warning above for vm.overcommit_memory values other than 0.

• vm.min_free_kbytes set to either 1% of system RAM or 4 GB, whichever is smaller

• vm.swappiness set between 1 and 10

If the SingleStore Tools cannot set the values for you, you will get an error message stating what the value should be and how to set it. You can set the values manually using the /sbin/sysctl command, as shown below.

sudo sysctl -w vm.max_map_count=1000000000

sudo sysctl -w vm.min_free_kbytes=<either 1% of system RAM or 4 GB, whichever is smaller>

Configure Swap Space

SingleStore recommends that you create a swap partition (or swap file on a dedicated device) to serve as an emergency backing store for RAM. SingleStore makes extensive use of RAM (especially with rowstore tables), so it is important that the operating system does not immediately start killing processes if SingleStore runs out of memory. Because typical hosts running SingleStore have a large amount of RAM (> 32 GB/node), the swap space can not be small (>= 10% of physical RAM).

For more information setting up and configuring swap space, please refer to your distribution’s documentation.

After enabling these settings, your hosts will be configured for optimal performance when running one or more SingleStore nodes.

Configure Transparent Huge Pages

Linux organizes RAM into pages that are usually 4 KB in size. Using transparent huge pages (THP), Linux can instead use 2 MB pages or larger. As a background process, THP transparently re-organizes memory used by a process inside the kernel by either merging small pages to huge pages or splitting few huge pages to small pages. This may block memory usage on the memory manager, which may span for a duration of a few seconds, and prevent the process from accessing memory.

As SingleStore uses a lot of memory, SingleStore recommends that you disable THP at boot time on all nodes (master aggregator, child aggregators, and leaf nodes) in the cluster. THP lag may result in inconsistent query run times or high system CPU (also known as red CPU).

To disable THP, add the following lines to the end of /etc/rc.local before the exit line (if present), and reboot the host:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
echo no > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
echo no > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag

On Red Hat distributions, THP will be under redhat_transparent_hugepage, on other distributions of Linux it will just be transparent_hugepage. You can check which your system uses by running ls /sys/kernel/mm/. Keep only the four relevant settings.

The khugepaged/defrag option will be 1 or 0 on newer Linux versions (e.g. CentOS 7+), but yes or no on older versions (e.g. CentOS 6). You can check which your system uses by running cat /sys/kernel/mm/*transparent_hugepage/khugepaged/defrag. Keep only the matching setting. For example, if you see 1 or 0, keep the line with echo 0; if you see yes or no, keep the line with echo no.

You should end up with at least three lines of settings, for enabled, defrag, and khugepaged/defrag. For example, the correct configuration for CentOS 7 is:

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo 0 > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag

Typically, you may include all eight settings and THP will still be disabled as expected.

Configure Network Time Protocol Service

Install and run ntpd to ensure that system time is in sync across all nodes in the cluster.

For Debian distributions (like Ubuntu):

sudo apt-get install ntpd

For Red Hat / CentOS/ AlmaLinux:

sudo yum install ntp

Configure Cluster-on-Die Mode

If you are installing SingleStore natively and have access to the BIOS, you should enable Cluster-on-Die in the system BIOS for hosts with Haswell-EP and later x86_64 CPUs. When enabled, this will result in multiple NUMA regions being exposed per processor. SingleStore can take advantage of NUMA nodes by binding specific SingleStore nodes to those NUMA nodes, which in turn will result in higher SingleStore performance.

Configure NUMA

If the CPU(s) on your host supports Non-Uniform Memory Access (NUMA), SingleStore can take advantage of that and bind SingleStore nodes to NUMA nodes. Binding SingleStore nodes to NUMA nodes allows faster access to in-memory data since individual SingleStore nodes only access data that’s collocated with their corresponding CPU.

If you do not configure SingleStore this way, performance will be greatly degraded due to expensive cross-NUMA-node memory access. Configuring for NUMA should be done as part of the installation process; however, you can reconfigure your deployment later, if necessary.

SingleStore Tools can do the NUMA binding for you; however, you must have numactl installed first. Perform the following steps on each host:

1. Log into each host and install the numactl package. For example, for a Debian-based OS:

   Shell

   Copy

   sudo apt-get install numactl

2. For Red Hat / CentOS / AlmaLinux, run the following:

   Shell

   Copy

   sudo yum install numactl

3. Check the number of NUMA nodes your hosts by running numactl --hardware. For example:

   Shell

   Copy

   numactl --hardware

   available: 2 nodes (0-1)

   The output shows that there are 2 NUMA nodes on this host, numbered 0 and 1.

For additional information, see Configuring SingleStore for NUMA.

Toolbox CPU Checkers

The following Toolbox checkers are run prior to deploying SingleStore.

Toolbox Memory Checkers

The following Toolbox checkers are run prior to deploying SingleStore.

POSIX-Compliant

To maintain data durability and resiliency, SingleStore’s data directory (as defined by the datadir engine variable, and which holds database snapshots, transaction logs, and columnstore segments) must reside on a POSIX-compliant filesystem.

SingleStore officially supports ext4 and XFS file systems. However, any POSIX-compliant file system can be used with SingleStore. We have conducted extensive testing with ext4 and XFS file systems and can provide support for all non-infrastructure related issues when running on a POSIX-compatible host.

Most Linux filesystems, including ext3, ext4, XFS, and ZFS are POSIX-compliant when mounted on a POSIX-compliant host.

Storage Requirements

Storage guidelines include:

• The storage location must be on a single contiguous volume, and should never be more than 60% utilized

• 30 MB/s per physical core or vCPU

• 300 input/output operations per second (IOPS) per physical core or vCPU. For example, for a 16-core host, the associated disk should be capable of 500MB/s and 5000 IOPS.

• For rowstore, the amount of required disk space should be about 5x the amount of RAM. Rowstore storage capacity is limited by the amount of RAM on the host. Increasing RAM increases the amount of available data storage.

• For columnstore, the amount of required disk space should be about the size of the raw data you are planning to load into the database. For concurrent loads on columnstore tables, SSD storage will improve performance significantly compared to HDD storage.

• When using high availability (HA), the amount of disk space required will be 2x the size of the raw data..

Toolbox Disk Checkers

The following Toolbox checkers are run prior to deploying SingleStore.

Configure Network Settings

Note: Perform the following steps on each host in the cluster.

1. As root, display the current sysctl settings and review the values of rmem_max and wmem_max.

   Shell

   Copy

   sudo sysctl -a | grep mem_max

2. Confirm that the receive buffer size (rmem_max) is 8 MB for all connection types. If not, add the following line to the /etc/sysctl.conf file.

   Copy

   net.core.rmem_max = 8388608

3. Confirm that the send buffer size (wmem_max) is 8 MB for all connection types. If not, add the following line to the /etc/sysctl.conf file.

   Copy

   net.core.wmem_max = 8388608

4. Confirm that the maximum number of connections that can be queued for a socket (net.core.somaxconn) is at least 1024.

   SingleStore will attempt to update this value to 1024 on a node's host when the node starts. After the cluster is up and running, run the following on each host to confirm that this value has been set.

   Shell

   Copy

   sudo sysctl -a | grep net.core.somaxconn

   If this value could not be set by SingleStore, add the following line to the host's /etc/sysctl.conf file. Note that values lower than 1024 could allow connection requests to overwhelm the host.

   Copy

   net.core.somaxconn = 1024

5. Persist these updates across reboots.

   Shell

   Copy

   sudo sysctl -p /etc/sysctl.conf

6. At the next system boot, confirm that the above values have persisted.

Default Network Ports

Depending on the host and its function in deployment, some or all of the following port settings should be enabled on hosts in your cluster. These routing and firewall settings must be configured to:

• Allow database clients (such as your application) to connect to the SingleStore aggregators

• Allow all nodes in the cluster to talk to each other over the SingleStore protocol (3306)

• Allow you to connect to management and monitoring tools

The service port values are configurable if the default values cannot be used in your deployment environment. For more information on how to change them, see the SingleStore configuration file, the sdb-toolbox-config register-host command, and Studio Installation Guide.

We also highly recommend configuring your firewall to prevent other hosts on the Internet from connecting to SingleStore.

Toolbox Network Checkers

The following Toolbox checkers are run prior to deploying SingleStore.