Troubleshoot Your Monitoring Setup

Connect to the database.

Run the following SQL. The default database name is metrics. If your database name is different from the default name, replace metrics with your database name.

USE metrics;
SELECT * FROM metrics LIMIT 10;

Optional, run SELECT * FROM on all of the monitoring tables.

If these queries return an empty set, review the pipelines error tables using the next step.

Review the monitoring pipelines.

SHOW PIPELINES;

If a monitoring pipeline (with a name resembling *_metrics and *_blobs) is in a state other than running, start the pipeline.

START PIPELINE <pipeline-name>;

Check the information_schema.pipelines_errors table for errors.

SELECT * FROM information_schema.pipelines_errors;

Confirm that port 9104 is accessible from all hosts in the cluster. This is the default port used for monitoring. To test this, run the following command at the Linux command line and review the output.

curl http://<endpoint>:9104/cluster-metrics

For example:

curl http://192.168.1.100:9104/cluster-metrics

If the hostname of the Master Aggregator is localhost, and a pipeline was created using localhost, use the following SQL commands to recreate the pipeline with the Master Aggregator host’s IP addresses. For example:

metrics pipeline:

CREATE OR REPLACE PIPELINE `metrics` AS LOAD DATA prometheus_exporter
"http://<host-ip-address>:9104/cluster-metrics" CONFIG '{"is_memsql_internal":true}'
INTO PROCEDURE `load_metrics` FORMAT JSON;

START PIPELINE IF NOT RUNNING metrics;

blobs pipeline:

CREATE OR REPLACE PIPELINE `blobs` AS LOAD DATA prometheus_exporter
"http://<host-ip-address>:9104/samples" CONFIG '{"is_memsql_internal":true, "download_type":"samples"}'
INTO PROCEDURE `load_blobs` FORMAT JSON;

START PIPELINE IF NOT RUNNING blobs;