optimize

Description

Recommends best practices and optimizes the cluster for maximum performance.

This command makes or recommends the following best practices:

By default, if --memory-percentage is not specified, sets maximum_memory to either 90% of the total RAM available to a leaf node, or to the total RAM available to the leaf node minus 10GB, whichever is greater.
If --memory-percentage is specified, sets maximum_memory to that percentage of the total RAM available to a leaf node.
If there are multiple leaf nodes on the same host, the memory allocation will be divided evenly among the leaf nodes.
Queries for the NUMA node count on the host. If numactl is installed on the host, optimize will set up NUMA binding based on the number of NUMA nodes on the host and the number of leaf nodes created.
Sets the engine variables default_partitions_per_leaf and expected_leaf_core_count on each host if the --no-partition-change flag is not specified and if each host has the same number of cores and the same number of leaf nodes. If these two conditions are met, default_partitions_per_leaf will be set to either 1 or the number of cores over the number of leaf nodes, whichever is greater.
Alerts if memory allocation per leaf node varies across the cluster outside of an acceptable margin. Memory allocation per leaf node is the host memory divided by the number of leaf nodes present on that host.
Recommends that you install new leaf nodes if the number of leaf nodes is less than what is recommended by the NUMA configuration.
If --bind-to-l3 is specified, and the host has only one NUMA node, sdb-admin optimize will set up the CPU binding based on L3 cache indices and the number of leaf nodes created on the host.

Blob Cache Optimization

The following logic optimizes blob cache settings for each host with leaf nodes. This ensures efficient blob cache allocation while maintaining system stability through adequate free disk space and by respecting existing percentage-based configurations.

Check the existing configuration.
1. The value of the maximum_blob_cache_size_percent engine variable is retrieved from each available leaf node.
2. If this variable is not set, or its value cannot be retrieved, a value of 0 is used.
Analyze disk storage.

Leaf nodes that share the same block storage for their respective data directories are grouped together.
Verify the existing percentage-based configuration.

If the value of the maximum_blob_cache_size_percent engine variable is already set (is greater than 0):
1. The storage capacity for each group of leaf nodes is verified, where:
  
  ((The number of leaf nodes in the group) × maximum_blob_cache_size_percent × (Storage size)) < ((Storage size) - 100 GB)
2. If the total blob cache allocation exceeds the available storage capacity, where less than 100 GB would be left free, the optimizer issues a warning.
3. When the value of the maximum_blob_cache_size_percent engine variable is set, the optimizer respects the configuration and makes no modifications.

Calculate optimal values.

If maximum_blob_cache_size_percent is not set or equals 0, the optimizer calculates optimal maximum_blob_cache_size_mb values using the following principles:

The reserved disk space that must remain free: 100 GB
The minimum blob cache size per leaf node: 40 GB
Optimal cache size is 75% of total disk space (75% rule)
The maximum usable space is 90% of total disk (90% rule)

The following algorithm is used for each leaf node:

IF (disk_size × 90% > 100GB) AND (disk_size × 75% < disk_size - 100GB):
    # Standard case: large disk with plenty of space
    cache_size = disk_size × 75% ÷ number_of_leaves_on_disk
    
    # Apply 90% constraint if needed
    IF cache_size < (disk_size × 90% - 100GB) ÷ number_of_leaves_on_disk:
        cache_size = (disk_size × 90% - 100GB) ÷ number_of_leaves_on_disk

ELSE:
    # Fallback for smaller disks or constrained scenarios
    IF disk_size × 75% > 40GB:
        cache_size = (disk_size × 75%) ÷ number_of_leaves_on_disk
    ELSE:
        # Minimum cache size with warning for low disk space
        cache_size = min(40GB, disk_size) ÷ number_of_leaves_on_disk
        WARN about low disk space

Apply required updates.
1. The calculated optimal values and the current maximum_blob_cache_size_mb settings are compared.
2. For those leaf nodes where the current configuration differs from the optimal value, the leaf nodes are updated.
3. For those leaf nodes that required an update, a map of leaf node IDs to new blob cache sizes is returned.

Refer to List of Engine Variables (maximum_blob_cache_size_percent and maximum_blob_cache_size_mb) for more information.

Usage

Usage:
  sdb-admin optimize [flags]

  For flags that can accept multiple values (indicated by VALUES after the name of the flag),
  separate each value with a comma.

Flags:
      --bind-to-l3              Bind leaf nodes to CPUs based on the L3 cache index
  -h, --help                    Help for optimize
      --memory-percentage INT   The percentage of memory on each host to use for SingleStore
      --no-numa                 Do not change NUMA settings on any nodes
      --no-partition-change     Do not change partitions on leaves

Global Flags:
      --backup-cache FILE_PATH                File path for the backup cache
      --cache-file FILE_PATH                  File path for the Toolbox node cache
  -c, --config FILE_PATH                      File path for the Toolbox configuration
      --disable-colors                        Disable color output in console, which some terminal sessions/environments may have difficulty with
      --disable-spinner                       Disable the progress spinner, which some terminal sessions/environments may have issues with
  -j, --json                                  Enable JSON output
      --parallelism POSITIVE_INTEGER          Maximum number of operations to run in parallel
      --runtime-dir DIRECTORY_PATH            Where to store Toolbox runtime data
      --ssh-control-persist SECONDS           Enable SSH ControlPersist and set it to the specified duration in seconds
      --ssh-max-sessions POSITIVE_INTEGER     Maximum number of SSH sessions to open per host, must be at least 3
      --ssh-strict-host-key-checking          Enable strict host key checking for SSH connections
      --ssh-user-known-hosts-file FILE_PATH   Path to the user known_hosts file for SSH connections. If not set, /dev/null will be used
      --state-file FILE_PATH                  Toolbox state file path
  -v, --verbosity count                       Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
  -y, --yes                                   Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default

Remarks

This command is interactive unless you use either --yes or --json flag to override interactive behavior.

On this page