# Troubleshoot Your Monitoring Setup

## Dashboards Stop Displaying Data

For those clusters running SingleStore v8.0, v8.1, or v8.5, should the Grafana monitoring dashboards stop displaying data, which may happen:

* After the cluster has been upgraded or restarted
* After one or more hosts in the cluster have been rebooted
* While the monitoring pipelines are still running

You may resolve this issue by using either of the following methods.

## Method 1

1. Upgrade the cluster to one of the following SingleStore versions.

   1. For clusters running SingleStore v8.0, upgrade to v8.0.34 or later

   2. For clusters running SingleStore v8.1, upgrade to v8.1.49 or later

   3. For clusters running SingleStore v8.5, upgrade to v8.5.28 or later

2. Run the following SQL statements after upgrading the cluster.
   ```sql
   ALTER PIPELINE metrics.metrics SET OFFSETS EARLIEST;
   ```
   ```sql
   ALTER PIPELINE metrics.blobs SET OFFSETS EARLIEST;
   ```
   ```sql
   ALTER PIPELINE metrics.event-traces SET OFFSETS EARLIEST;
   ```

## Method 2

1. Set `MAX_RETRIES_PER_BATCH_PARTITION` for the monitoring pipelines to a large value, such as `1000000`, via the following SQL statements.
   ```sql
   ALTER PIPELINE metrics.metrics SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
   ```
   ```sql
   ALTER PIPELINE metrics.blobs SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
   ```
   ```sql
   ALTER PIPELINE metrics.event-traces SET MAX_RETRIES_PER_BATCH_PARTITION 1000000;
   ```

2. Run the following SQL statements.
   ```sql
   ALTER PIPELINE metrics.metrics SET OFFSETS EARLIEST;
   ```
   ```sql
   ALTER PIPELINE metrics.blobs SET OFFSETS EARLIEST;
   ```
   ```sql
   ALTER PIPELINE metrics.event-traces SET OFFSETS EARLIEST;
   ```

## Pipelines

## Check the Monitoring Tables for Data

1. Connect to the database.

2. Run the following SQL. The default database name is `metrics`. If your database name is different from the default name, replace `metrics` with your database name.
   ```sql
   USE metrics;
   SELECT * FROM metrics LIMIT 10;

   ```
   Optional, run `SELECT * FROM` on all of the monitoring tables.

   If these queries return an empty set, review the pipelines error tables using the next step.

3. Review the monitoring pipelines.
   ```sql
   SHOW PIPELINES;

   ```

4. If a monitoring pipeline (with a name resembling `*_metrics` and `*_blobs`) is in a state other than `running`, start the pipeline.
   ```sql
   START PIPELINE <pipeline-name>;

   ```

5. Check the `information_schema.pipelines_errors` table for errors.
   ```sql
   SELECT * FROM information_schema.pipelines_errors;

   ```

## Resolve Pipeline Errors

If you receive an `Cannot extract data for the pipeline error` in the `pipelines_error table`, perform the following steps.

1. Confirm that port `9104` is accessible from all hosts in the cluster. This is the default port used for monitoring. To test this, run the following command at the Linux command line and review the output.
   ```shell
   curl http://<endpoint>:9104/cluster-metrics

   ```
   For example:
   ```shell
   curl http://192.168.1.100:9104/cluster-metrics

   ```

2. If the hostname of the Master Aggregator is `localhost`, and a pipeline was created using `localhost`, use the following SQL commands to recreate the pipeline with the Master Aggregator host’s IP addresses. For example:

   `metrics` pipeline:
   ```sql
   create or replace pipeline `metrics` as load data prometheus_exporter 
   "http://<host-ip-address>:9104/cluster-metrics" 
   config '{"is_memsql_internal":true}' 
   into procedure `load_metrics` format json;

   ```
   ```sql
   start pipeline if not running metrics;

   ```
   `blobs` pipeline:
   ```sql
   create or replace pipeline `blobs` as load data prometheus_exporter 
   "http://<host-ip-address>:9104/samples" 
   config '{"is_memsql_internal":true, "download_type":"samples"}' 
   into procedure `load_blobs` format json;

   ```
   ```sql
   start pipeline if not running blobs;

   ```

***

Modified at: August 6, 2025

Source: [/db/v9.1/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring/troubleshoot-your-monitoring-setup/](https://docs.singlestore.com/db/v9.1/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring/troubleshoot-your-monitoring-setup/)

(An index of the documentation is available at /llms.txt)