# Configure Core Dumps A core dump can be generated for a single pod, or for all pods in the SingleStore cluster. ## Enable Core Dumps Core dumps can be enabled on individual pods, or across all pods. ## Individual Pods 1. List all pods in the cluster. ```shell kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance= ``` 2. Confirm that a core dump will be produced upon crash. ```shell kubectl exec -c node -- memsqlctl query --sql "SELECT @@core_file" ``` ```output +-------------+ | @@core_file | +-------------+ | 1 | +-------------+ ``` The default value for `@@core_file` is `1`, which indicates that creating a core dump on crash is enabled. If this value is `0`, creating a core dump on crash is disabled. To enable it, run the following command. ```shell kubectl exec -c node -- memsqlctl -y update-config --key core_file --value ON ``` Restart the node for this change to take effect. ```shell kubectl exec -c node -- memsqlctl -y restart-node ``` 3. Configure the core dump to be either partial or full. By default, a partial core dump is created. Refer to the `core_file_mode`[engine variable](https://docs.singlestore.com/db/v9.1/reference/configuration-reference/engine-variables/list-of-engine-variables/#non-sync-variables-list.md) for more information on partial and full core dumps. Run the following command to check the current value. ```shell kubectl exec node--master-0 -c node -- memsqlctl query --sql "SELECT @@core_file_mode" ``` ```output +------------------+ | @@core_file_mode | +------------------+ | PARTIAL | +------------------+ ``` Run the following command to change this value. For a partial core dump: ```shell kubectl exec -c node -- memsqlctl -y update-config --key core_file_mode --value PARTIAL ``` For a full core dump: ```shell kubectl exec -c node -- memsqlctl -y update-config --key core_file_mode --value FULL ``` Restart the node for this change to take effect. ```shell kubectl exec -c node -- memsqlctl -y restart-node ``` ## Across all Pods 1. To make the above core dump changes across all pods, add the following subsection to the [sdb-cluster.yaml](https://docs.singlestore.com/db/v9.1/deploy/kubernetes/create-the-object-definition-files/sdb-cluster-yaml.md) file’s spec section. Refer to the `core_file_mode`[engine variable](https://docs.singlestore.com/db/v9.1/reference/configuration-reference/engine-variables/list-of-engine-variables/#non-sync-variables-list-2.md) for more information on partial and full core dumps. ```yaml globalVariables: core_file: "ON" core_file_mode: "FULL" # or "PARTIAL", depending on your goal ``` 2. Apply these values to enable core dumps on all pods. ```shell kubectl apply -f sdb-cluster.yaml ``` ## Generate Core Dumps 1. Display the pods and the nodes they are on. Note that the following output may vary. ```shell kubectl get pods -l app.kubernetes.io/name=memsql-cluster,app.kubernetes.io/instance= -o wide ``` ```output NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES node-memsql-cluster-aggregator-0 2/2 Running 0 4h13m 10.4.0.20 gke-foo-1-default-pool-ed7b330b-2q1r node-memsql-cluster-leaf-ag1-0 2/2 Running 0 4h13m 10.4.3.10 gke-foo-1-default-pool-ed7b330b-w12w node-memsql-cluster-leaf-ag2-0 2/2 Running 0 4h13m 10.4.1.8 gke-foo-1-default-pool-ed7b330b-64r5 node-memsql-cluster-master-0 2/2 Running 0 4h13m 10.4.4.24 gke-foo-1-default-pool-ed7b330b-klnz ``` ## Using an SSH Connection **Note**: These steps affect all core dump generation for processes on the nodes, not just SingleStore containers on the nodes. 1. For each node that contains the pods you want to generate a core dump for, SSH into the node and run the following command. ```shell echo "core.%d" > /proc/sys/kernel/core_pattern ``` ## Using a DaemonSet **Note**: These steps affect all core dump generation for processes on the nodes, not just SingleStore containers on the nodes. 1. Create a `sdb-coredump.yaml` file using the following template. ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: singlestore-sysreq labels: "app.kubernetes.io/name": "singlestore-sysreq" spec: selector: matchLabels: "app.kubernetes.io/name": "singlestore-sysreq" updateStrategy: type: RollingUpdate template: metadata: labels: "app.kubernetes.io/name": "singlestore-sysreq" spec: hostPID: true volumes: - name: sys hostPath: path: /sys affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app.kubernetes.io/instance: CLUSTER_NAME app.kubernetes.io/name: memsql-cluster #Add this line if you only want to generate core for Master Aggregator #app.kubernetes.io/component: master #change master to aggregator or leaf if you only want to generate core #for child aggregator or for leaf #Add this line if you only want to generate core for one particular pod, #substitute POD_NAME with the name of the pod #statefulset.kubernetes.io/pod-name: POD_NAME topologyKey: kubernetes.io/hostname initContainers: - name: sysreq image: busybox securityContext: privileged: true volumeMounts: - name: sys mountPath: /rootfs/sys command: - "/bin/sh" - "-c" - | echo “core.%d” > /proc/sys/kernel/core_pattern containers: - name: pause image: gcr.io/google_containers/pause ``` 2. Edit the following placeholders in this file: * Change `CLUSTER_NAME` to the SingleStore cluster name. * To generate the core dump in a specific pod, change `POD_NAME` to the desired pod. 3. Enable the core dump(s) at the Kubernetes host level. Note that you must have cluster admin privileges to run this command. ```shell kubectl apply -f sdb-coredump.yaml ``` ## Core Dump Location Given the above configuration, and should the SingleStore server crash, a core dump will be written to `/var/lib/memsql/instance/data/core.memsqld` in the restarted pod. *** Modified at: November 21, 2022 Source: [/db/v9.1/reference/singlestore-operator-reference/configure-core-dumps/](https://docs.singlestore.com/db/v9.1/reference/singlestore-operator-reference/configure-core-dumps/) (An index of the documentation is available at /llms.txt)