Configure Alerts

The sdb-report tool allows you to run a variety of recommended health checks on your cluster. As of Toolbox 1.9.0, sdb-report allows you to create and send email alerts through an SMTP server to immediately inform you of potential cluster health issues as they are identified.

These instructions will guide you through the setup and scheduling of the alert mechanism by configuring a local SMTP server (Postfix). This guide assumes you have an email client that you can use to receive emails from the SMTP server. You can integrate the alert mechanism with any SMTP email server and mail clients.

Refer to sdb-report send-alert for more information on the available command-line arguments.

We use the SMTP server Postfix for the purpose of this guide, but you can configure this with any SMTP server.

An alert configuration file must be created to specify where your SMTP server resides. This file also holds any configuration changes from the default for the thresholds. Here is the template of the config.yaml file. This file will be passed in using the --config-file argument demonstrated in the next step. In this case, the user is not updating any default thresholds for the alert criteria:

location:
  email:
    receivers:
      - receiver@example.com
    sender: sender@example.com
    server:
      host: smtp.example.com
      port: 25
      username:
      password:

Modify this file by:

The following example file shows a configuration of an SMTP server running on an EC2 instance, sending an email to a singlestore.user Gmail user. You may add multiple recipients to the receiver list. Additionally, the user is setting custom thresholds for the leavesNotOnline, orphanDatabases and memoryCommitted checkers. Refer to the tables on the Next Steps page for the list of configurable thresholds. You can apply this change to other checkers that are based on output as well.

location:
  email:
    receivers:
      - singlestore.user@gmail.com
    sender: ec2-user@ip-172-31-77-34.ec2.internal
    server:
      host: ip-172-30-77-34.ec2.internal
      port: 25
      username: 
      password:
thresholds:
  leavesNotOnline:
    fail: 2
  memoryCommitted:
    warn: 80
  orphanDatabases:
    output_level: fail

The following is the simplest implementation of the sdb-report command and will use the default thresholds. You may also configure your own thresholds for the checkers in this file using the configuration reference in Next Steps.

The following command will set up alerts to the SMTP server (which was installed and configured in the previous step) and will generate and send an alert exactly once to the location specified in the YAML file. This “one-liner” is a convenient method for testing your configuration before officially scheduling your alerts.

sdb-report send-alert --config-file config.yaml

Refer to sdb-report send-alert for more information on the available command-line arguments.

*/5 * * * * sdb-report send-alert --config-file config.yaml

smtp     inet  n       -       n       -       -       smtpd

1025     inet  n       -       n       -       -       smtpd

sudo postfix reload

There are two distinct emails sent per alert command run: one for failures and one for warnings. If all checkers pass, no email will be sent.

This email was generated from the address specified, and sent to the server specified, in the alert-location configuration.

In the following example email, a failure was generated for a node being offline, but no other cluster issues were identified.

Date: Tue, 27 Oct 2020 21:39:16 +0000
From: ec2-user@ip-172-31-77-34.ec2.internal
To: singlestore.user@gmail.com
Subject: SingleStoreDB Cluster Alert: Cluster check failures

SingleStoreDB cluster cluster_name has generated the following failures:
leavesNotOnline ........................ [FAIL]
FAIL leaf node on host 127.0.0.1 and port 3308 is offline