DB Software v7.8

Generate Core Dumps

  1. Display the pods and the nodes they are on. Note that the following output may vary.

    kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance=<cluster-name> -o wide
****
NAME                               READY   STATUS    RESTARTS   AGE     IP          NODE                                   NOMINATED NODE   READINESS GATES
node-memsql-cluster-aggregator-0   2/2     Running   0          4h13m   10.4.0.20   gke-foo-1-default-pool-ed7b330b-2q1r   <none>           <none>
node-memsql-cluster-leaf-ag1-0     2/2     Running   0          4h13m   10.4.3.10   gke-foo-1-default-pool-ed7b330b-w12w   <none>           <none>
node-memsql-cluster-leaf-ag2-0     2/2     Running   0          4h13m   10.4.1.8    gke-foo-1-default-pool-ed7b330b-64r5   <none>           <none>
node-memsql-cluster-master-0       2/2     Running   0          4h13m   10.4.4.24   gke-foo-1-default-pool-ed7b330b-klnz   <none>           <none>

Using an SSH Connection

Note: These steps affect all core dump generation for processes on the nodes, not just SingleStore DB containers on the nodes.

  1. For each node that contains the pods you want to generate a core dump for, SSH into the node and run the following command.

    echo "core.%d" > /proc/sys/kernel/core_pattern

Using a DaemonSet

  1. Create a memsql-coredump.yaml file using the following template.

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: memsql-sysreq
  labels:
    "app.kubernetes.io/name": "memsql-sysreq"
spec:
  selector:
    matchLabels:
      "app.kubernetes.io/name": "memsql-sysreq"
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        "app.kubernetes.io/name": "memsql-sysreq"
    spec:
      hostPID: true
      volumes:
      - name: sys
        hostPath:
          path: /sys
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/instance: CLUSTER_NAME
                app.kubernetes.io/name: memsql-cluster

                #Add this line if you only want to generate core for Master Aggregator
                #app.kubernetes.io/component: master
                #change master to aggregator or leaf if you only want to generate core
                #for child aggregator or for leaf

                #Add this line if you only want to generate core for one particular pod, 
                #substitute POD_NAME with the name of the pod
                #statefulset.kubernetes.io/pod-name: POD_NAME
            topologyKey: kubernetes.io/hostname

      initContainers:
      - name: sysreq
        image: busybox
        securityContext:
          privileged: true
        volumeMounts:
        - name: sys
          mountPath: /rootfs/sys
        command:
        - "/bin/sh"
        - "-c"
        - |
          echo “core.%d” > /proc/sys/kernel/core_pattern

      containers:
      - name: pause
        image: gcr.io/google_containers/pause

  2. Edit the following placeholders in this file:

    • Change CLUSTER_NAME to the SingleStore DB cluster name.

    • To generate the core dump in a specific pod, change POD_NAME to the desired pod.

  3. Enable the core dump(s) at the Kubernetes host level. Note that you must have cluster admin privileges to run this command.

    kubectl apply -f memsql-coredump.yaml
In this section: