Configure Core Dumps

Individual Pods

1. List all pods in the cluster.

   Shell

   Copy

   kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance=<cluster-name>

2. Confirm that a core dump will be produced upon crash.

   Shell

   Copy

   kubectl exec <pod-name> -c node -- memsqlctl query --sql "SELECT @@core_file"

   +-------------+
   | @@core_file |
   +-------------+
   | 1           |
   +-------------+

   The default value for @@core_file is 1, which indicates that creating a core dump on crash is enabled. If this value is 0, creating a core dump on crash is disabled. To enable it, run the following command.

   Shell

   Copy

   kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file --value ON

   Restart the node for this change to take effect.

   Shell

   Copy

   kubectl exec <pod-name> -c node -- memsqlctl -y restart-node

3. Configure the core dump to be either partial or full. By default, a partial core dump is created. Refer to the core_file_modeengine variable for more information on partial and full core dumps.

   Run the following command to check the current value.

   Shell

   Copy

   kubectl exec node-<cluster-name>-master-0 -c node -- memsqlctl query --sql "SELECT @@core_file_mode"

   +------------------+
   | @@core_file_mode |
   +------------------+
   | PARTIAL          |
   +------------------+

   Run the following command to change this value.

   For a partial core dump:

   Shell

   Copy

   kubectl exec <pod-namne> -c node -- memsqlctl -y update-config --key core_file_mode --value PARTIAL

   For a full core dump:

   Shell

   Copy

   kubectl exec <pod-name> -c node -- memsqlctl -y update-config --key core_file_mode --value FULL

   Restart the node for this change to take effect.

   Shell

   Copy

   kubectl exec <pod-name> -c node -- memsqlctl -y restart-node

Across all Pods

1. To make the above core dump changes across all pods, add the following subsection to the sdb-cluster.yaml file’s spec section. Refer to the core_file_modeengine variable for more information on partial and full core dumps.

   YAML

   Copy

   globalVariables:
     core_file: "ON"
     core_file_mode: "FULL"    # or "PARTIAL", depending on your goal

2. Apply these values to enable core dumps on all pods.

   Shell

   Copy

   kubectl apply -f sdb-cluster.yaml

Using an SSH Connection

Note: These steps affect all core dump generation for processes on the nodes, not just SingleStore containers on the nodes.

1. For each node that contains the pods you want to generate a core dump for, SSH into the node and run the following command.

   Shell

   Copy

   echo "core.%d" > /proc/sys/kernel/core_pattern

Using a DaemonSet

Note: These steps affect all core dump generation for processes on the nodes, not just SingleStore containers on the nodes.

1. Create a sdb-coredump.yaml file using the following template.

   YAML

   Copy

   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: singlestore-sysreq
     labels:
       "app.kubernetes.io/name": "singlestore-sysreq"
   spec:
     selector:
       matchLabels:
         "app.kubernetes.io/name": "singlestore-sysreq"
     updateStrategy:
       type: RollingUpdate
     template:
       metadata:
         labels:
           "app.kubernetes.io/name": "singlestore-sysreq"
       spec:
         hostPID: true
         volumes:
         - name: sys
           hostPath:
             path: /sys
         affinity:
           podAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                 matchLabels:
                   app.kubernetes.io/instance: CLUSTER_NAME
                   app.kubernetes.io/name: memsql-cluster
   
                   #Add this line if you only want to generate core for Master Aggregator
                   #app.kubernetes.io/component: master
                   #change master to aggregator or leaf if you only want to generate core
                   #for child aggregator or for leaf
   
                   #Add this line if you only want to generate core for one particular pod, 
                   #substitute POD_NAME with the name of the pod
                   #statefulset.kubernetes.io/pod-name: POD_NAME
               topologyKey: kubernetes.io/hostname
   
         initContainers:
         - name: sysreq
           image: busybox
           securityContext:
             privileged: true
           volumeMounts:
           - name: sys
             mountPath: /rootfs/sys
           command:
           - "/bin/sh"
           - "-c"
           - |
             echo “core.%d” > /proc/sys/kernel/core_pattern
   
         containers:
         - name: pause
           image: gcr.io/google_containers/pause

2. Edit the following placeholders in this file:

   • Change CLUSTER_NAME to the SingleStore cluster name.

   • To generate the core dump in a specific pod, change POD_NAME to the desired pod.

3. Enable the core dump(s) at the Kubernetes host level. Note that you must have cluster admin privileges to run this command.

   Shell

   Copy

   kubectl apply -f sdb-coredump.yaml