Skip to main content

Configure Monitoring

Overview

SingleStore’s native monitoring solution is designed to capture and reveal SingleStoreDBcluster events over time. By analyzing this event data, you can identify trends and, if necessary, take action to remediate issues.

Terminology

Throughout this guide, the cluster that is being monitored will be referred to as the Sourcecluster, and the cluster that stores the monitoring data will be referred to as the Metricscluster. The databases that store monitoring data will be referred to as the metrics database.

High-Level Architecture

cluster_monitoring_hl_architecture.png

In SingleStore’s native monitoring solution, the Metrics cluster utilizes a SingleStore pipeline to pull the data from the exporter process on the Source cluster and stores it in a database named metrics. Note that this metrics database can either reside within the same cluster as the Source cluster, or within a dedicated cluster.

When these event data is then analyzed through the provided Grafana dashboards, trends can be identified and, if necessary, actions taken to remediate issues.

The provided Grafana dashboards include:

Dashboard

Description

Active session history

Aggregated resource consumption by activity and activity type

Activity history

Resource consumption by a specific activity over time

Detailed cluster view

A birds-eye view of a single SingleStoreDBcluster

Information schema view

Provides a view into information_schema views PROCESSLIST, TABLES, and TABLE_STATISTICS

Memory usage

Granular breakdown of memory use for a host

SingleStoreDB status and variables view

Collected status variables from each host in the cluster

Node breakout

System metrics from each host in the cluster

Node drilldown

System metrics from each host in the cluster, with the ability to focus on a specific metric subsystem

Prerequisites

Notice

These instructions have been developed for SingleStoreDB clusters that have been installed and deployed via .rpm or .deb packages as a sudo user.

If your cluster was deployed via tarball as a non-sudo user, change to the directory (cd) in which singlestoredb-toolbox was untarred and run all sdb-admin commands as ./sdb-admin.

SingleStoreDB Toolbox is recommended for managing the clusters as automation during setup is provided through sdb-admin commands. While monitoring can be enabled through a series of SQL commands, the preferred method is to use SingleStoreDB Toolbox.

Note that the Grafana instructions will require a user with sudo access to install and configure the associated Grafana components.

For HTTP Connections

  • A SingleStoreDB 7.3 or later cluster to monitor (the Source cluster).

  • Optional: A separate SingleStoreDB 7.3 or later cluster used to collect monitoring data (the Metrics cluster). This cluster can be the same as, or separate from, the Source cluster. If a separate cluster, the Metrics cluster:

    • Must be open to local (internal) network traffic only, and not open to the Internet.

    • Must only contain monitoring data from one or more Source clusters.

    • Should have two aggregator nodes and two leaf nodes, each with 2TB disks and with high availability (HA) enabled (SingleStore recommended)

  • Clusters are managed with SingleStoreDB Toolbox 1.9.3 or later.

  • A Grafana instance (the latest version of Grafana) that can access the Metrics cluster.

For HTTPS Connections

Similar to the HTTP connection prerequisites, with additional requirements:

  • Each Source and Metrics cluster must be running SingleStoreDB 7.6.24 or later, or SingleStoreDB 7.8.19 or later.

  • Clusters are managed with SingleStoreDB Toolbox 1.14.2 or later.

  • A server SSL certificate and a key signed with a CA certificate. This guide assumes that:

    • The server SSL certificate file is named server-cert.pem.

    • The server key file is named server-key.pem. Note that the server key may be protected with a passphrase.

    • The CA certificate file is named ca-cert.pem.

    • Refer to Generating SSL Certificates for an example of generating these certificates.

Port Configuration

Default Port

Used by

Invoked by

3000

Grafana

User browser

3306

SingleStoreDB

memsql_exporter

9104

memsql_exporter

SingleStore pipelines