# Troubleshoot Your Monitoring Setup ## Pipelines ## Check the Monitoring Tables for Data 1. Connect to the database. 2. Run the following SQL. The default database name is `metrics`. If your database name is different from the default name, replace `metrics` with your database name. ```sql USE metrics; SELECT * FROM metrics LIMIT 10; ``` Optional, run `SELECT * FROM` on all of the monitoring tables. If these queries return an empty set, review the pipelines error tables using the next step. 3. Review the monitoring pipelines. ```sql SHOW PIPELINES; ``` 4. If a monitoring pipeline (with a name resembling `*_metrics` and `*_blobs`) is in a state other than `running`, start the pipeline. ```sql START PIPELINE ; ``` 5. Check the `information_schema.pipelines_errors` table for errors. ```sql SELECT * FROM information_schema.pipelines_errors; ``` ## Resolve Pipeline Errors If you receive an `Cannot extract data for the pipeline error` in the `pipelines_error table`, perform the following steps. 1. Confirm that port `9104` is accessible from all hosts in the cluster. This is the default port used for monitoring. To test this, run the following command at the Linux command line and review the output. ```shell curl http://:9104/cluster-metrics ``` For example: ```shell curl http://192.168.1.100:9104/cluster-metrics ``` 2. If the hostname of the Master Aggregator is `localhost`, and a pipeline was created using `localhost`, use the following SQL commands to recreate the pipeline with the Master Aggregator host’s IP addresses. For example: `metrics` pipeline: ```sql CREATE OR REPLACE PIPELINE `metrics` AS LOAD DATA prometheus_exporter "http://:9104/cluster-metrics" CONFIG '{"is_memsql_internal":true}' INTO PROCEDURE `load_metrics` FORMAT JSON; ``` ```sql START PIPELINE IF NOT RUNNING metrics; ``` `blobs` pipeline: ```sql CREATE OR REPLACE PIPELINE `blobs` AS LOAD DATA prometheus_exporter "http://:9104/samples" CONFIG '{"is_memsql_internal":true, "download_type":"samples"}' INTO PROCEDURE `load_blobs` FORMAT JSON; ``` ```sql START PIPELINE IF NOT RUNNING blobs; ``` ## Disable Collection of Specific Metrics In some environments, specific metrics can be expensive to generate, produce copious amounts of data, or capture information that is not required. In cases like these, the corresponding collectors or samplers can be disabled at the exporter level. ## How the Monitoring Exporter Collects Metrics The monitoring exporter uses the following internal components to collect metrics: * **Collectors**: Generate Prometheus-format metrics. Disable a collector by adding the `--no-cluster-collect.` flag to the exporter configuration. List of collectors: | Collector name | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `events` | Collect from`information_schema.MV_EVENTS`. | | `info_schema.processlist` | Collect current thread state counts from`information_schema.PROCESSLIST`. | | `info_schema.mv_processlist` | Collect current thread state counts from`information_schema.MV_PROCESSLIST`. | | `memory.counters` | Collect memory aggregates from`GLOBAL STATUS`variables. | | `info_schema.tables` | Collect metrics from`information_schema.TABLES`. | | `info_schema.tablestats` | Collect table statistics from`information_schema.TABLE_STATISTICS`. | | `info_schema.mv_nodes` | Collect metrics from`information_schema.MV_NODES`. | | `info_schema.system_info` | Collect metrics from`MV_NODES`and`MV_HOSTS_INFORMATION`. | | `info_schema.mv_disk_usage` | Collect disk usage stats bySingleStorenodes from`information_schema.MV_DISK_USAGE`. | | `info_schema.mv_data_disk_usage` | Collect breakdown of disk usage by each database from`information_schema.MV_DATA_DISK_USAGE`. | | `info_schema.resource_pools` | Collect resource pool configurations from`information_schema.RESOURCE_POOLS`. | | `info_schema.resource_pool_status` | Collect resource pool status from`information_schema.MV_RESOURCE_POOL_STATUS`. | | `info_schema.aggregators` | Collect Aggregator node information from`information_schema.AGGREGATORS`. | | `info_schema.leaves` | Collect leaf node information from`information_schema.LEAVES`. | | `distributed.partitions` | Collect partition status from`information_schema.DISTRIBUTED_PARTITIONS`(offline/online/total per DB/role). | | `distributed.unavailable.partitions` | For eachcluster, returns`1`if there exists a fully unavailable partition in any user database in`information_schema.DISTRIBUTED_PARTITIONS`(both Master and Replica offline). | | `heartbeat` | Collect from a heartbeat table (e.g. pt-heartbeat) to expose stored vs current timestamps. | | `info_schema.mv_bottomless_status_extended` | Collect statistics about remote storage usage from`information_schema.MV_BOTTOMLESS_STATUS_EXTENDED`. | | `info_schema.mv_cloud_duplication_status` | Collect statistics about storage duplication from`information_schema.MV_CLOUD_DUPLICATION_STATUS`. | | `info_schema.mv_cloud_branches` | Collect statistics about duplication attach points from`information_schema.MV_CLOUD_BRANCHES`. | | `info_schema.mv_trace_events_status` | Collect Event Tracing status from`information_schema.MV_TRACE_EVENTS_STATUS`. | | `pipeline.metrics` | Collect pipeline state metrics from`information_schema.PIPELINES`(state, IDs, times) and pipeline counts (running/stopped/errored). | | `pipeline.errorstate` | Mark pipelines in`state = 'ERROR'`(per database and pipeline name). | | `pipeline.errors` | Collect pipeline errors since the last seen timestamp from`information_schema.PIPELINES_ERRORS`. | | `info_schema.pipeline_batches` | Collect batch-level pipeline metrics from`information_schema.PIPELINES_BATCHES`. | * **Samplers**: Generate non-Prometheus-format metrics that populate internal metric tables/views used by dashboards and other components. Disable a sampler by adding the `--no-sample.` flag to the exporter configuration. List of samplers: | Sampler name | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `activity` | Sample the contents of`information_schema.MV_ACTIVITIES_CUMULATIVE`(and optionally`information_schema.MV_QUERIES`). | | `activity_extended` | Sample the contents of`information_schema.MV_ACTIVITIES_EXTENDED_CUMULATIVE`(and optionally`information_schema.MV_QUERIES`). | | `node` | Sample the contents of`information_schema.MV_NODES`. | | `cluster_info` | Expose`information_schema.MV_NODES`,`information_schema.MV_HOSTS_INFORMATION`,`information_schema.MV_DISTRIBUTED_DATABASES_STATUS`, and`information_schema.MV_CLUSTER_STATUS`. | | `pipeline_metrics` | Expose the`information_schema.PIPELINES`table. | | `pipeline_errors` | Expose`information_schema.PIPELINES_ERRORS`since the last sampled error timestamp. | | `events` | Expose the`information_schema.MV_EVENTS`table as a stream of event records. | | `connection_attributes` | Expose`information_schema.MV_CONNECTION_ATTRIBUTES`. | | `columnstore_merge_status` | Expose`information_schema.MV_COLUMNSTORE_MERGE_STATUS`. | ## Disable Metrics Update the exporter flags in the operator configuration. 1. Open the `sdb-operator.yaml` file for the deployment. 2. Append the required flags to `--master-exporter-parameters` argument. For example: ``` --master-exporter-parameters=" ... --no-sample.activity_extended" ``` 3. Apply the updated configuration. ## Common Flags to Disable Metrics Use the following flags to disable the respective metric collection: * `--no-sample.columnstore_merge_status`: Disables queries to the `MV_COLUMNSTORE_MERGE_STATUS` information schema view. * `--no-sample.pipeline_errors`: Disables queries to the `PIPELINES_ERRORS` information schema view. * `--no-sample.activity_extended`: Disables queries to the `MV_ACTIVITIES_EXTENDED_CUMULATIVE` information schema view and stops the collection of parametrized query texts that populate `MV_QUERIES` metrics. ## Disable Activity Metrics Collection Use the `--no-sample.activity_extended` flag to disable collection of activity metrics. This mitigation is typically used in the following scenarios: * Activity metrics queries are slow or consume excessive resources on the source cluster. * Extended activity metrics generate more data than downstream systems (pipelines or monitoring stack) can handle. * Minimize metric collection for privacy or data minimization. To re-enable activity metrics collection, remove the `--no-sample.activity_extended` flag from the exporter configuration and restart the exporter. *** Modified at: April 6, 2026 Source: [/db/v9.1/reference/singlestore-operator-reference/monitor-your-kubernetes-cluster/troubleshoot-your-monitoring-setup/](https://docs.singlestore.com/db/v9.1/reference/singlestore-operator-reference/monitor-your-kubernetes-cluster/troubleshoot-your-monitoring-setup/) (An index of the documentation is available at /llms.txt)