Important

The SingleStore 9.1 release candidate (RC) gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 9.0 is recommended for production workloads, which can later be upgraded to SingleStore 9.1.

Troubleshoot Your Monitoring Setup

Pipelines

Check the Monitoring Tables for Data

Connect to the database.
Run the following SQL. The default database name is metrics. If your database name is different from the default name, replace metrics with your database name.
SQL
```
USE metrics;
SELECT * FROM metrics LIMIT 10;
```
Optional, run SELECT * FROM on all of the monitoring tables.

If these queries return an empty set, review the pipelines error tables using the next step.
Review the monitoring pipelines.
SQL
```
SHOW PIPELINES;
```
If a monitoring pipeline (with a name resembling *_metrics and *_blobs) is in a state other than running, start the pipeline.
SQL
```
START PIPELINE <pipeline-name>;
```
Check the information_schema.pipelines_errors table for errors.
SQL
```
SELECT * FROM information_schema.pipelines_errors;
```

Resolve Pipeline Errors

If you receive an Cannot extract data for the pipeline error in the pipelines_error table, perform the following steps.

Confirm that port 9104 is accessible from all hosts in the cluster. This is the default port used for monitoring. To test this, run the following command at the Linux command line and review the output.
Shell
```
curl http://<endpoint>:9104/cluster-metrics
```
For example:
Shell
```
curl http://192.168.1.100:9104/cluster-metrics
```

If the hostname of the Master Aggregator is localhost, and a pipeline was created using localhost, use the following SQL commands to recreate the pipeline with the Master Aggregator host’s IP addresses. For example:

metrics pipeline:

SQL

CREATE OR REPLACE PIPELINE `metrics` AS LOAD DATA prometheus_exporter
"http://<host-ip-address>:9104/cluster-metrics" CONFIG '{"is_memsql_internal":true}'
INTO PROCEDURE `load_metrics` FORMAT JSON;

SQL

START PIPELINE IF NOT RUNNING metrics;

blobs pipeline:

SQL

CREATE OR REPLACE PIPELINE `blobs` AS LOAD DATA prometheus_exporter
"http://<host-ip-address>:9104/samples" CONFIG '{"is_memsql_internal":true, "download_type":"samples"}'
INTO PROCEDURE `load_blobs` FORMAT JSON;

SQL

START PIPELINE IF NOT RUNNING blobs;

Disable Collection of Specific Metrics

In some environments, specific metrics can be expensive to generate, produce copious amounts of data, or capture information that is not required. In cases like these, the corresponding collectors or samplers can be disabled at the exporter level.

How the Monitoring Exporter Collects Metrics

The monitoring exporter uses the following internal components to collect metrics:

Collectors: Generate Prometheus-format metrics. Disable a collector by adding the --no-cluster-collect.<collector_name> flag to the exporter configuration. List of collectors:

Collector name	Description
`events`	Collect from `information_schema.MV_EVENTS`.
`info_schema.processlist`	Collect current thread state counts from `information_schema.PROCESSLIST`.
`info_schema.mv_processlist`	Collect current thread state counts from `information_schema.MV_PROCESSLIST`.
`memory.counters`	Collect memory aggregates from `GLOBAL STATUS` variables.
`info_schema.tables`	Collect metrics from `information_schema.TABLES`.
`info_schema.tablestats`	Collect table statistics from `information_schema.TABLE_STATISTICS`.
`info_schema.mv_nodes`	Collect metrics from `information_schema.MV_NODES`.
`info_schema.system_info`	Collect metrics from `MV_NODES` and `MV_HOSTS_INFORMATION`.
`info_schema.mv_disk_usage`	Collect disk usage stats by SingleStore nodes from `information_schema.MV_DISK_USAGE`.
`info_schema.mv_data_disk_usage`	Collect breakdown of disk usage by each database from `information_schema.MV_DATA_DISK_USAGE`.
`info_schema.resource_pools`	Collect resource pool configurations from `information_schema.RESOURCE_POOLS`.
`info_schema.resource_pool_status`	Collect resource pool status from `information_schema.MV_RESOURCE_POOL_STATUS`.
`info_schema.aggregators`	Collect Aggregator node information from `information_schema.AGGREGATORS`.
`info_schema.leaves`	Collect leaf node information from `information_schema.LEAVES`.
`distributed.partitions`	Collect partition status from `information_schema.DISTRIBUTED_PARTITIONS` (offline/online/total per DB/role).
`distributed.unavailable.partitions`	For each cluster, returns `1` if there exists a fully unavailable partition in any user database in `information_schema.DISTRIBUTED_PARTITIONS` (both Master and Replica offline).
`heartbeat`	Collect from a heartbeat table (e.g. pt-heartbeat) to expose stored vs current timestamps.
`info_schema.mv_bottomless_status_extended`	Collect statistics about remote storage usage from `information_schema.MV_BOTTOMLESS_STATUS_EXTENDED`.
`info_schema.mv_cloud_duplication_status`	Collect statistics about storage duplication from `information_schema.MV_CLOUD_DUPLICATION_STATUS`.
`info_schema.mv_cloud_branches`	Collect statistics about duplication attach points from `information_schema.MV_CLOUD_BRANCHES`.
`info_schema.mv_trace_events_status`	Collect Event Tracing status from `information_schema.MV_TRACE_EVENTS_STATUS`.
`pipeline.metrics`	Collect pipeline state metrics from `information_schema.PIPELINES` (state, IDs, times) and pipeline counts (running/stopped/errored).
`pipeline.errorstate`	Mark pipelines in `state = 'ERROR'` (per database and pipeline name).
`pipeline.errors`	Collect pipeline errors since the last seen timestamp from `information_schema.PIPELINES_ERRORS`.
`info_schema.pipeline_batches`	Collect batch-level pipeline metrics from `information_schema.PIPELINES_BATCHES`.

Samplers: Generate non-Prometheus-format metrics that populate internal metric tables/views used by dashboards and other components. Disable a sampler by adding the --no-sample.<sampler_name> flag to the exporter configuration. List of samplers:

Sampler name	Description
`activity`	Sample the contents of `information_schema.MV_ACTIVITIES_CUMULATIVE` (and optionally `information_schema.MV_QUERIES`).
`activity_extended`	Sample the contents of `information_schema.MV_ACTIVITIES_EXTENDED_CUMULATIVE` (and optionally `information_schema.MV_QUERIES`).
`node`	Sample the contents of `information_schema.MV_NODES`.
`cluster_info`	Expose `information_schema.MV_NODES`, `information_schema.MV_HOSTS_INFORMATION`, `information_schema.MV_DISTRIBUTED_DATABASES_STATUS`, and `information_schema.MV_CLUSTER_STATUS`.
`pipeline_metrics`	Expose the `information_schema.PIPELINES` table.
`pipeline_errors`	Expose `information_schema.PIPELINES_ERRORS` since the last sampled error timestamp.
`events`	Expose the `information_schema.MV_EVENTS` table as a stream of event records.
`connection_attributes`	Expose `information_schema.MV_CONNECTION_ATTRIBUTES`.
`columnstore_merge_status`	Expose `information_schema.MV_COLUMNSTORE_MERGE_STATUS`.

Disable Metrics

Update the exporter flags in the operator configuration.

Open the sdb-operator.yaml file for the deployment.
Append the required flags to --master-exporter-parameters argument. For example:
```
--master-exporter-parameters=" ... --no-sample.activity_extended"
```
Apply the updated configuration.

Common Flags to Disable Metrics

Use the following flags to disable the respective metric collection:

--no-sample.columnstore_merge_status: Disables queries to the MV_COLUMNSTORE_MERGE_STATUS information schema view.
--no-sample.pipeline_errors: Disables queries to the PIPELINES_ERRORS information schema view.
--no-sample.activity_extended: Disables queries to the MV_ACTIVITIES_EXTENDED_CUMULATIVE information schema view and stops the collection of parametrized query texts that populate MV_QUERIES metrics.

Disable Activity Metrics Collection

Use the --no-sample.activity_extended flag to disable collection of activity metrics. This mitigation is typically used in the following scenarios:

Activity metrics queries are slow or consume excessive resources on the source cluster.
Extended activity metrics generate more data than downstream systems (pipelines or monitoring stack) can handle.
Minimize metric collection for privacy or data minimization.

To re-enable activity metrics collection, remove the --no-sample.activity_extended flag from the exporter configuration and restart the exporter.