Configure Core Dumps

List all pods in the cluster.

kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance=<cluster-name>

Confirm that a core dump will be produced upon crash.

kubectl exec <pod-name> -c node -- memsqlctl query --sql "SELECT @@core_file"
****
+-------------+
| @@core_file |
+-------------+
| 1           |
+-------------+

The default value for @@core_file is 1, which indicates that creating a core dump on crash is enabled. If this value is 0, creating a core dump on crash is disabled. To enable it, run the following command.

kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file --value ON

Restart the node for this change to take effect.

kubectl exec <pod-name> -c node -- memsqlctl -y restart-node

Configure the core dump to be either partial or full. By default, a partial core dump is created. Refer to the core_file_modeengine variable for more information on partial and full core dumps.

Run the following command to check the current value.

kubectl exec node-<cluster-name>-master-0 -c node -- memsqlctl query --sql "SELECT @@core_file_mode"
****
+------------------+
| @@core_file_mode |
+------------------+
| PARTIAL          |
+------------------+

Run the following command to change this value.

For a partial core dump:

kubectl exec <pod-namne> -c node -- memsqlctl -y update-config --key core_file_mode --value PARTIAL

For a full core dump:

kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file_mode --value FULL

Restart the node for this change to take effect.

kubectl exec <pod-name> -c node -- memsqlctl -y restart-node

To make the above core dump changes across all pods, add the following subsection to the sdb-cluster.yaml file’s spec section. Refer to the core_file_modeengine variable for more information on partial and full core dumps.

globalVariables:
  core_file: "ON"
  core_file_mode: "FULL"    # or "PARTIAL", depending on your goal

Apply these values to enable core dumps on all pods.

kubectl apply -f sdb-cluster.yaml

For each node that contains the pods you want to generate a core dump for, SSH into the node and run the following command.

echo "core.%d" > /proc/sys/kernel/core_pattern

Create a sdb-coredump.yaml file using the following template.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: singlestore-sysreq
  labels:
    "app.kubernetes.io/name": "singlestore-sysreq"
spec:
  selector:
    matchLabels:
      "app.kubernetes.io/name": "singlestore-sysreq"
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        "app.kubernetes.io/name": "singlestore-sysreq"
    spec:
      hostPID: true
      volumes:
      - name: sys
        hostPath:
          path: /sys
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: CLUSTER_NAME
                app.kubernetes.io/name: memsql-cluster

                #Add this line if you only want to generate core for Master Aggregator
                #app.kubernetes.io/component: master
                #change master to aggregator or leaf if you only want to generate core
                #for child aggregator or for leaf

                #Add this line if you only want to generate core for one particular pod, 
                #substitute POD_NAME with the name of the pod
                #statefulset.kubernetes.io/pod-name: POD_NAME
            topologyKey: kubernetes.io/hostname

      initContainers:
      - name: sysreq
        image: busybox
        securityContext:
          privileged: true
        volumeMounts:
        - name: sys
          mountPath: /rootfs/sys
        command:
        - "/bin/sh"
        - "-c"
        - |
          echo “core.%d” > /proc/sys/kernel/core_pattern

      containers:
      - name: pause
        image: gcr.io/google_containers/pause

Edit the following placeholders in this file:

Enable the core dump(s) at the Kubernetes host level. Note that you must have cluster admin privileges to run this command.

kubectl apply -f sdb-coredump.yaml