SingleStore DB

collect-kube
Description

Build a diagnostics report for the kube cluster.

Usage
Build a diagnostics report for the kube cluster.

Available collectors (some collectors are excluded by default):

+---------------------------------------+----------+--------+----------------------------------------------------------------------------------+
|                  ID                   | EXCLUDED | GLOBAL |                                   DESCRIPTION                                    |
+---------------------------------------+----------+--------+----------------------------------------------------------------------------------+
| analyzeTables                         | EXCLUDED |        | Save column statistical information, and histograms recorded on each table, into |
|                                       |          |        | a JSON file                                                                      |
| cgroupInfo                            | EXCLUDED |        | Collects cgroup information for each SingleStore DB process                      |
| chronydStatus                         | EXCLUDED |        | Collects status of the `chronyd` service                                         |
| clusterTopology                       |          |        | Runs SHOW LEAVES, SHOW AGGREGATORS, and SHOW LEAF STATUS EXTENDED on the Master  |
|                                       |          |        | Aggregator                                                                       |
| cpuFreqInfo                           |          |        | Collects information about CPU frequency configuration                           |
| cpuMemoryBandwidth                    |          |        | Measures CPU-memory bandwidth                                                    |
| cpuThreadingInfo                      |          |        | Collects information about CPUs threading configuration                          |
| cpuUtilization                        |          |        | Records the output of `sar 1 10 -u`                                              |
| dataSkewness                          | EXCLUDED |        | Collects information about data skew in the cluster                              |
| df                                    |          |        | Records the output of `df -h                                                     |
|                                       |          |        | --output=source,size,used,avail,pcent,ipcent,target`                             |
| diskBandwidth                         |          |        | Measures disk read/write bandwidth                                               |
| diskLatency                           |          |        | Records the output of `iostat -xdm 1 10`                                         |
| dmesg                                 |          |        | Records the output of `dmesg -T`                                                 |
| ethtool                               |          |        | Collects information about each network interface on the host                    |
| explainOrphanDatabases                |          |        | Runs EXPLAIN CLEAR ORPHAN DATABASES on the Master Aggregator                     |
| explainRebalancePartitions            |          |        | Runs EXPLAIN REBALANCE PARTITIONS for each user database                         |
| explainRestoreRedundancy              |          |        | Runs EXPLAIN RESTORE REDUNDANCY for each user database                           |
| free                                  |          |        | Records the output of `free -m`                                                  |
| hosts                                 |          |        | Collect information about cluster's hosts                                        |
| ifconfig                              |          |        | Records the output of `ifconfig`                                                 |
| informationSchemaColumnarSegments     | EXCLUDED |        | Collects columnstore debugging information from information_schema on the Master |
|                                       |          |        | Aggregator                                                                       |
| informationSchemaDistributedDatabases |          |        | Collects distributed databases names on the Master Aggregator                    |
| informationSchemaIndexStatistics      |          |        | Runs SELECT * FROM information_schema.INDEX_STATISTICS on the Master Aggregator  |
| informationSchemaMvBackupHistory      |          |        | Runs SELECT * FROM information_schema.MV_BACKUP_HISTORY on the Master Aggregator |
| informationSchemaMvEvents             |          |        | Runs SELECT * FROM information_schema.MV_EVENTS on the Master Aggregator         |
| informationSchemaMvNodes              |          |        | Runs SELECT * FROM information_schema.MV_NODES on the Master Aggregator          |
| informationSchemaMvQueries            |          |        | Runs SELECT * FROM information_schema.MV_QUERIES on the Master Aggregator        |
| informationSchemaMvSysinfo            |          |        | Collects all the MV_SYSINFO tables from information_schema master on the Master  |
|                                       |          |        | Aggregator                                                                       |
| informationSchemaPipelines            |          |        | Collects pipelines debugging information from information_schema on the Master   |
|                                       |          |        | Aggregator                                                                       |
| informationSchemaProcesslist          |          |        | Records the PROCESSLIST for each node                                            |
| informationSchemaRoutines             |          |        | Collects a list of user-defined functions and stored procedures from             |
|                                       |          |        | information_schema on the Master Aggregator                                      |
| informationSchemaStatistics           |          |        | Collects information_schema.statistics on each node                              |
| informationSchemaTableStatistics      |          |        | Runs SELECT * FROM information_schema.TABLE_STATISTICS on the Master Aggregator  |
| informationSchemaTables               |          |        | Collects table metadata from each node                                           |
| informationSchemaUserPrivileges       |          |        | Collects information_schema.USER_PRIVILEGES from each node                       |
| installedPermissions                  |          |        | Collects information about file and directory permissions on a cluster           |
| ipAddr                                |          |        | Records the output of `ip addr`                                                  |
| ipLink                                |          |        | Records the output of `ip link`                                                  |
| ipRoute                               |          |        | Records the output of `ip route`                                                 |
| licenseMetadata                       |          |        | Collects the output of `memsqlctl show-license` for each node                    |
| lsblk                                 |          |        | Records the output of `lsblk`                                                    |
| lsblkRota                             |          |        | Records the output of `lsblk -d -o rota`                                         |
| majorPageFaults                       |          |        | Records the output of `sar 1 10 -B`                                              |
| memoryCommitted                       |          |        | Records the output of `sar 1 10 -r`                                              |
| memsqlAuditlogs                       | EXCLUDED |        | Collects the audit logs from each node                                           |
| memsqlBacktraces                      |          |        | Collects backtrace information from each node                                    |
| memsqlCommandLogs                     |          |        | Collects the command logs from each node                                         |
| memsqlConfig                          |          |        | Collects the `memsql.cnf` file from each node                                    |
| memsqlNumaConfig                      |          |        | Collects the NUMA configuration files from each node                             |
| memsqlPlancache                       | EXCLUDED |        | Copies the plancache directory from each node                                    |
| memsqlStacks                          |          |        | Collects stack files from each node                                              |
| memsqlTracelogs                       |          |        | Collects the tracelogs from each node                                            |
| memsqlctlInfo                         |          |        | Collects information about memsqlctl from each node                              |
| memsqldProcessLimits                  |          |        | Collects limits for all `memsqld` processes                                      |
| mount                                 |          |        | Records the output of `mount`                                                    |
| netstat                               |          |        | Records the output of `netstat -s`                                               |
| nodeDirectoriesDiskUsage              |          |        | Collects disk usage information for each node directory                          |
| nodes                                 |          |        | Collects the output of `memsqlctl describe-node` from each node                  |
| numactl                               |          |        | Records the output of `numactl --hardware`                                       |
| osRelease                             |          |        | Collects information from those files in /etc that describe the host operating   |
|                                       |          |        | system                                                                           |
| partitionsStart                       | EXCLUDED |        | Collects information about each partition's start sector number                  |
| podLogs                               |          |        | Collects logs of Kube operator Pods (see memsqlTracelogs for logs of other Pods) |
| proc                                  |          |        | Collects diagnostic files from /proc                                             |
| procFs                                |          |        | Collects diagnostic files from /proc/fs                                          |
| processFiles                          |          |        | Collects top-level of /proc/<pid> for each SingleStore DB process                |
| ps                                    |          |        | Records the output of `ps aux`                                                   |
| psutil                                |          |        | Collects system and process information about the host                           |
| rebootTime                            |          |        | Records the output of `last -x reboot`                                           |
| schema                                |          |        | Collects database schemas from each aggregator node                              |
| securityLimits                        |          |        | Collects /etc/security/limits.conf                                               |
| showClusterStatus                     |          |        | Collects cluster status by running `SHOW CLUSTER STATUS` on the Master           |
|                                       |          |        | Aggregator                                                                       |
| showCronJobs                          |          |        | Collects information about Kube CronJobs                                         |
| showDatabaseStatus                    |          |        | Collects the database status for each database on each node                      |
| showDatabasesExtended                 |          |        | Runs SHOW DATABASES EXTENDED on each node                                        |
| showDeployments                       |          |        | Collects information about Kube Deployments                                      |
| showMemsqlClusters                    |          |        | Collects information about Kube MemsqlClusters                                   |
| showPartitions                        |          |        | Collects database partition metadata from each aggregator node                   |
| showPersistentVolumeClaims            |          |        | Collects information about Kube PersistentVolumeClaims                           |
| showPersistentVolumes                 |          |        | Collects information about Kube PersistentVolumes                                |
| showPlancache                         |          |        | Runs SHOW PLANCACHE on each node                                                 |
| showPods                              |          |        | Collects information about Kube Pods                                             |
| showRebalanceStatus                   |          |        | Collects rebalance status for each database on the Master Aggregator             |
| showReplicaSets                       |          |        | Collects information about Kube ReplicaSets                                      |
| showReplicationStatus                 |          |        | Runs SHOW REPLICATION STATUS on each node                                        |
| showResourcePools                     |          |        | Runs SHOW RESOURCE POOLS on each aggregator node                                 |
| showServices                          |          |        | Collects information about Kube Services                                         |
| showStatefulSets                      |          |        | Collects information about Kube StatefulSets                                     |
| showStatusExtended                    |          |        | Runs SHOW STATUS EXTENDED on each node                                           |
| showSvlAPITokenGroups                 |          |        | Collects information about Kube SvlAPITokenGroups                                |
| showSvlClusters                       |          |        | Collects information about Kube SvlClusters                                      |
| showSvlDatabaseReplicas               |          |        | Collects information about Kube SvlDatabaseReplicas                              |
| showSvlDatabases                      |          |        | Collects information about Kube SvlDatabases                                     |
| showSvlSites                          |          |        | Collects information about Kube SvlSites                                         |
| showTableStatus                       |          |        | Collects table status for each database on each aggregator node                  |
| showVariables                         |          |        | Runs SHOW VARIABLES LIKE '%' on each node                                        |
| showWorkloadManagementStatus          |          |        | Runs SHOW WORKLOAD MANAGEMENT STATUS on each aggregator node                     |
| shutdownTime                          |          |        | Records the output of `last -x shutdown`                                         |
| studioVersion                         |          | GLOBAL | Records the output of `memsql-studio version --log /dev/null --state /dev/null`  |
| swapUsage                             |          |        | Records the output of `sar 1 10 -S`                                              |
| syncVariables                         |          |        | Runs query DUMP CLUSTER VARIABLES; on each node                                  |
| sysctl                                |          |        | Records the output of `sysctl -a`                                                |
| toolboxConfig                         |          |        | Collects Toolbox's configuration and version                                     |
| toolboxEnv                            |          | GLOBAL | Collects global toolbox config information                                       |
| top                                   |          |        | Records the output of `top -b -n 1`                                              |
| transparentHugepage                   |          |        | Collects files from /sys/kernel/mm/transparent_hugepage                          |
| ulimit                                |          |        | Records the output of `sh -c ulimit -a`                                          |
| uptime                                |          |        | Records the output of `uptime`                                                   |
+---------------------------------------+----------+--------+----------------------------------------------------------------------------------+


Examples:

# Collect information on all of the available clusters from the current namespace
sdb-report collect-kube

# Collect information on all of the available clusters from the namespace ns
sdb-report collect-kube --namespace ns

# Collect information on the cl cluster in the namespace ns
sdb-report collect-kube --cluster-name cl --namespace ns

# Collect information by specifying the kubeconfig
sdb-report collect-kube --config-file ~/dir/.kube/config

# Collect information using the namespace of the context c
sdb-report collect-kube --context c

# Collect information on SingleStore Managed Service v1 clusters only
sdb-report collect-kube --version v1

Usage:
  sdb-report collect-kube [flags]

Flags:
      --cluster-name VALUES         The specific cluster name(s) to discover, separated by commas. Leave blank for all clusters
      --config-file FILE_PATH       The kube config filepath. Leave blank if the path is stored in the KUBECONFIG env variable or in the ~/.kube/config file (ADVANCED)
      --context string              The specific kube context to switch to when a kubeconfig has multiple contexts (ADVANCED)
      --exclude VALUES              Exclude the specified collectors
      --exclude-global              Exclude global collectors from the report file collector
  -h, --help                        Help for collect-kube
      --host strings                Specify host(s) to collect from (collects from all hosts by default)
      --include VALUES              Include the specified collectors
      --include-performance         Include collectors that create load on cluster (not recommended for active clusters)
      --mask-ip                     Mask usernames, hostnames, IP and MAC addresses in the report file collector
      --merge-path string           The directory from which to merge reports (ADVANCED)
      --namespace VALUES            The kube namespace(s) from which to collect reports, separated by commas. If missing, the default namespace will be used
      --only VALUES                 Only run the specified collectors
      --opt key=value               Specify collector options (default memsqlAuditlogs.auditLogSize=10MB,memsqlAuditlogs.compressAuditLog=False,memsqlCommandLogs.commandLogSize=10MB,memsqlCommandLogs.compressCommandLog=False,memsqlTracelogs.compressTracelog=False,memsqlTracelogs.tracelogSize=10MB)
  -o, --output-path ABSOLUTE_PATH   Write the report tarball to this file path
      --temp-dir ABSOLUTE_PATH      The directory on the local and remote host(s) to store temporary files during collection (ADVANCED)
      --validate-env                Run collectors that do not require SingleStore DB installation (performance collectors included)
      --version {both, v1, v2}      Either v1 or v2; if missing, both v1 and v2 clusters will be discovered (ADVANCED) (default both)

Global Flags:
      --backup-cache FILE_PATH              File path for the backup cache
      --cache-file FILE_PATH                File path for the Toolbox node cache
  -c, --config FILE_PATH                    File path for the Toolbox configuration
      --disable-colors                      Disable colored output in console, which some terminal sessions/environments may have issues with
      --disable-spinner                     Disable the progress spinner, which some terminal sessions/environments may have issues with
  -j, --json                                Enable JSON output
      --parallelism POSITIVE_INTEGER        Maximum number of operations to run in parallel
      --runtime-dir DIRECTORY_PATH          Where to store Toolbox runtime data
      --ssh-max-sessions POSITIVE_INTEGER   Maximum number of SSH sessions to open per host, must be at least 3
      --state-file FILE_PATH                Toolbox state file path
  -v, --verbosity count                     Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
  -y, --yes                                 Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default
Remarks

This command is interactive unless you use either --yes or --json flag to override interactive behavior.

Categorization of Collectors

The collectors used in reporting gather cluster information related to alerting, performance, and pre-installation environment validation. The following table maps collectors with the category of the information reported by them.

Category

Collectors

Alerting

clusterTopology

cpuUtilization (also reports performance and environment validation details)

df (also reports environment validation details)

diskLatency (also reports performance and environment validation details)

explainOrphanDatabases

explainRebalancePartitions

informationSchemaMvNodes

majorPageFaults (also reports performance and environment validation details)

memoryCommitted (also reports performance and environment validation details)

memsqlctlInfo

netstat

showDatabasesExtended

showPartitions

showReplicationStatus

showVariables

swapUsage (also reports performance and environment validation details)

Performance

cpuMemoryBandwidth (also reports environment validation details)

cpuUtilization (also reports alerting and environment validation details)

diskBandwidth also reports environment validation details

diskLatency (also reports alerting and environment validation details)

majorPageFaults (also reports alerting and environment validation details)

memoryCommitted (also reports alerting and environment validation details)

swapUsage (also reports alerting and environment validation details)

Pre-installation Environment Validation

cpuFreqInfo

cpuMemoryBandwidth (also reports performance-related information)

cpuThreadingInfo

cpuUtilization (also reports alerting- and performance-related information)

df (also reports alerting-related information)

diskBandwidth (also reports performance-related information)

diskLatency (also reports alerting- and performance-related information)

dmesg

ethtool

free

ifconfig

ipAddr

ipLink

ipRoute

lsblk

lsblkRota

majorPageFaults (also reports alerting- and performance-related information)

memoryCommitted (also reports alerting- and performance-related information)

mount

nodes

numactl

osRelease

proc

procFs

ps

psutil

rebootTime

securityLimits

shutdownTime

studioVersion

swapUsage (also reports alerting- and performance-related information)

sysctl

toolboxConfig

top

transparentHugepage

ulimit

uptime

In Kube clusters, the following additional collectors are used:

podLogs

showCronJobs

showDeployments

showKubeMemsqlClusters

showKubeSvlClusters

showPods

showPVCs

showPVs

showReplicaSets

showServices

showStatefulSets

showSvlAPITokenGroups

showSvlDBReplicas

showSvlDBs

showSvlSites

Note that some collectors do not fall under any of the three categories and are used to collect more general information about the cluster.