Investigating Memory Usage Discrepancies

When there is a large discrepancy between the sum of memory usage from information_schema.TABLE_STATISTICS vs what is reported in alloc_table_memory, review the internal table statistics via information_schema.INTERNAL_TABLE_STATISTICS to see where the rest of memory is being held. For example:

SHOW STATUS EXTENDED LIKE '%Alloc_table_memory%';
+--------------------+-----------------------+
| Variable_name      | Value                 |
+--------------------+-----------------------+
| Alloc_table_memory | 39389.780 (+0.125) MB |
+--------------------+-----------------------+
1 row in set (0.00 sec)

Compare that information with the results of:

SELECT SUM(memory_use)/1024/1024/1024 MEM_GB FROM information_schema.table_statistics
WHERE 1=1 AND host = 'leaf-1.example.com';
+----------------+
| MEM_GB         |
+----------------+
| 2.392731554806 |
+----------------+
1 row in set (0.76 sec)

The alloc variable is showing 39 GB while table statistics shows just 2.39 GB.

The next step would be to examine the md_<table> information in information_schema.INTERNAL_TABLE_STATISTICS for outliers to see where the rest of the memory is being held. For example, extremely large unlimited storage databases can result in md_columnar_blobs storing large amounts of data for each node.

Last modified: July 25, 2023

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK