Troubleshoot Your Monitoring Setup

Dashboards Stop Displaying Data

For those clusters running SingleStore v8.0, v8.1, or v8.5, should the Grafana monitoring dashboards stop displaying data, which may happen:

  • After the cluster has been upgraded or restarted

  • After one or more hosts in the cluster have been rebooted

  • While the monitoring pipelines are still running

You may resolve this issue by using either of the following methods.

Method 1

  1. Upgrade the cluster to one of the following SingleStore versions.

    1. For clusters running SingleStore v8.0, upgrade to v8.0.34 or later

    2. For clusters running SingleStore v8.1, upgrade to v8.1.49 or later

    3. For clusters running SingleStore v8.5, upgrade to v8.5.28 or later

  2. Run the following SQL statements after upgrading the cluster.

    ALTER PIPELINE metrics.metrics SET OFFSETS EARLIEST;
    ALTER PIPELINE metrics.blobs SET OFFSETS EARLIEST;
    ALTER PIPELINE metrics.event-traces SET OFFSETS EARLIEST;

Method 2

  1. Set MAX_RETRIES_PER_BATCH_PARTITION for the monitoring pipelines to a large value, such as 1000000, via the following SQL statements.

    ALTER PIPELINE metrics.metrics SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
    ALTER PIPELINE metrics.blobs SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
    ALTER PIPELINE metrics.event-traces SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
  2. Run the following SQL statements.

    ALTER PIPELINE metrics.metrics SET OFFSETS EARLIEST;
    ALTER PIPELINE metrics.blobs SET OFFSETS EARLIEST;
    ALTER PIPELINE metrics.event-traces SET OFFSETS EARLIEST;

Pipelines

Check the Monitoring Tables for Data

  1. Connect to the database.

  2. Run the following SQL. The default database name is metrics. If your database name is different from the default name, replace metrics with your database name.

    USE metrics;
    SELECT * FROM metrics LIMIT 10;

    Optional, run SELECT * FROM on all of the monitoring tables.

    If these queries return an empty set, review the pipelines error tables using the next step.

  3. Review the monitoring pipelines.

    SHOW PIPELINES;
  4. If a monitoring pipeline (with a name resembling *_metrics and *_blobs) is in a state other than running, start the pipeline.

    START PIPELINE <pipeline-name>;
  5. Check the information_schema.pipelines_errors table for errors.

    SELECT * FROM information_schema.pipelines_errors;

Resolve Pipeline Errors

If you receive an Cannot extract data for the pipeline error in the pipelines_error table, perform the following steps.

  1. Confirm that port 9104 is accessible from all hosts in the cluster. This is the default port used for monitoring. To test this, run the following command at the Linux command line and review the output.

    curl http://<endpoint>:9104/cluster-metrics

    For example:

    curl http://192.168.1.100:9104/cluster-metrics
  2. If the hostname of the Master Aggregator is localhost, and a pipeline was created using localhost, use the following SQL commands to recreate the pipeline with the Master Aggregator host’s IP addresses. For example:

    metrics pipeline:

    create or replace pipeline `metrics` as load data prometheus_exporter
    "http://<host-ip-address>:9104/cluster-metrics"
    config '{"is_memsql_internal":true}'
    into procedure `load_metrics` format json;
    start pipeline if not running metrics;

    blobs pipeline:

    create or replace pipeline `blobs` as load data prometheus_exporter
    "http://<host-ip-address>:9104/samples"
    config '{"is_memsql_internal":true, "download_type":"samples"}'
    into procedure `load_blobs` format json;
    start pipeline if not running blobs;

Last modified: August 6, 2025

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK