Configure Alerts

Email

Before continuing, ensure that your SMTP server is configured to send outbound messages to your desired recipients.

1. Add the following section to your cluster’s grafana.yaml file and update it with the following values:

   1. host = <SMTP server hostname>:<port>

   2. user = The user that the SMTP server uses to authenticate the client that is sending the email

   3. password = The above user’s password

   4. from_address = The email address from which the email is sent

   5. from_name = The name of the sender

   YAML

   Copy

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: grafana-config
   data:
     grafana.ini: |-
       [smtp]
       enabled=true
       host=<SMTP server host>:<port>
       user=<The user that the SMTP server uses to authenticate the client that is sending the email>
       password=<The above user's password>
       from_address=<The address from which the email is sent>
       from_name=<The name of the sender>

2. Modify the following two sections in the grafana.yaml file:

   From:

   YAML

   Copy

   volumeMounts:
     - mountPath: /var/lib/grafana
       name: grafana-pv

   To:

   YAML

   Copy

   volumeMounts:
     - mountPath: /var/lib/grafana
       name: grafana-pv
     - mountPath: /etc/grafana
       name: grafana-config

   From:

   YAML

   Copy

   volumes:
     - name: grafana-pv
       persistentVolumeClaim:
         claimName: grafana-pvc

   To:

   YAML

   Copy

   volumes:
     - name: grafana-pv
       persistentVolumeClaim:
         claimName: grafana-pvc
     - name: grafana-config
       configMap:
         name: grafana-config

3. Restart the Grafana server.

   Shell

   Copy

   sudo systemctl restart grafana-server

4. In Grafana, add a new contact point and specify:

   1. Integration: Email

   2. Addresses: A list of addresses to send to the alerts to

   3. (Optional) Configure additional settings under Optional Email settings.

PagerDuty

1. In PagerDuty, navigate to Services and either create a new service or modify an existing one. Add Events API v2 as an integration when creating or modifying the service.

2. View the Events API v2 integration and copy the Integration Key (for example, 1v970cee7a8c4c01d0e407b31e098a59).

3. In Grafana, add a new contact point and specify:

   1. Integration: PagerDuty

   2. Integration Key: Paste the integration key from the previous step

   3. (Optional) Configure additional settings under Optional PagerDuty settings, such as severity, class, etc.

Slack

To message a specific Slack workspace channel when an alert is triggered:

1. Go to Slack API: Applications and create a new app.

   1. Select From Scratch.

   2. Specify the workspace to message.

2. Select Incoming Webhooks and enable Activate Incoming Webhooks.

3. Click Add new Webhook to Workspace and select a channel to message. Create the Slack channel if it does not exist. Copy the URL of the created Webhook.

4. In Grafana, add a new contact point and specify:

   1. Integration: Slack

   2. Webhook URL: Paste the URL of the Webhook from the previous step

   3. (Optional) Configure additional settings under Optional Slack settings, such as username, mention users, etc.

5. Test the integration setup and click the Save contact point button.

Import Alert Rules from SingleStore

1. Download and unzip the k8s_alert_rules.zip file.

   Note: Install jq, a command-line JSON processor, before proceeding to the next step.

2. Run the following on the Linux command line to import the alert rules into Grafana.

   Shell

   Copy

   GRAFANA_HOST=<Hostname of the Grafana instance (default=localhost)> \
   GRAFANA_PORT=<Port of the Grafana instance (default=3000)> \
   GRAFANA_USER=<Grafana user (default=admin)> \
   GRAFANA_PASSWORD=<Grafana user's password (default=admin)> \
   ./provision_alert_rules.sh

3. View the alert rules in Grafana.

4. (Optional) To customize the imported alert rules, click the edit button and:

   1. Modify threshold value to change when the alert is triggered.

   2. Edit the evaluation period for field and update the length of time that the alert must be triggered before a notification is sent. An alert must be constantly triggered for the specified period before a notification is sent. For example, a notification will not be sent if the alert is in a "Firing" state for two minutes and this parameter is set to five minutes.

   3. Add custom labels and other metadata.

Note: A minimum version of SingleStore may be required on the Source cluster for some alerts to function. The required SingleStore version is listed in the details of the alert rule. If this requirement is not met, the alert may reflect a status of “No data."

Create Custom Alert Rules

You may create custom rules by following the Create Grafana-managed alert rules instructions and specifying the monitoring data source.

You may also duplicate one of the imported alerts and modify the query and/or any of the other parameters.