collect-kube

Description

Build a diagnostics report for the Kubernetes (kube) cluster.

Available collectors

Note that some collectors are excluded by default.

+------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+
| ID | EXCLUDED | GLOBAL | DESCRIPTION |
+------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+
| analyzeTables | EXCLUDED | | Save column statistical information, and histograms recorded on each table, into |
| | | | a JSON file |
| cgroupInfo | EXCLUDED | | Collects cgroup information for each SingleStore process |
| chronydStatus | EXCLUDED | | Collects status of the `chronyd` service |
| clusterTopology | | | Runs SHOW LEAVES, SHOW AGGREGATORS, and SHOW LEAF STATUS EXTENDED on the Master |
| | | | Aggregator |
| cpuFreqInfo | | | Collects information about CPU frequency configuration |
| cpuMemoryBandwidth | | | Measures CPU-memory bandwidth |
| cpuThreadingInfo | | | Collects information about CPUs threading configuration |
| cpuUtilization | | | Records the output of `sar 1 10 -u` |
| dataSkewness | EXCLUDED | | Collects information about data skew in the cluster |
| databaseDiskUsage | | | Capture disk usage per database |
| df | | | Records the output of `df -h |
| | | | --output=source,size,used,avail,pcent,ipcent,target` |
| diskBandwidth | | | Measures disk read/write bandwidth |
| diskLatency | | | Records the output of `iostat -xdm 1 10` |
| dmesg | | | Records the output of `dmesg -T` |
| ethtool | | | Collects information about each network interface on the host |
| explainOrphanDatabases | | | Runs EXPLAIN CLEAR ORPHAN DATABASES on the Master Aggregator |
| explainRebalancePartitions | | | Runs EXPLAIN REBALANCE PARTITIONS for each user database |
| explainRestoreRedundancy | | | Runs EXPLAIN RESTORE REDUNDANCY for each user database |
| failedHosts | | GLOBAL | List of hosts where a report could not be collected |
| free | | | Records the output of `free -m` |
| hosts | | | Collect information about cluster's hosts |
| ifconfig | | | Records the output of `ifconfig` |
| informationSchemaAvailabilityGroups | | | Collects information about availability groups from the information_schema on an |
| | | | aggregator node |
| informationSchemaColumnarSegments | EXCLUDED | | Collects columnstore debugging information from the information_schema on an |
| | | | aggregator node |
| informationSchemaDistributedDatabases | | | Collects distributed databases names on an aggregator node |
| informationSchemaDistributedDatabasesOnSites | | | Collects information about database distribution across clusters |
| informationSchemaIndexStatistics | | | Runs SELECT * FROM information_schema.INDEX_STATISTICS on an aggregator node |
| informationSchemaInternalTableStatistics | EXCLUDED | | Runs SELECT * FROM information_schema.INTERNAL_TABLE_STATISTICS on an aggregator |
| | | | node |
| informationSchemaMvAggregatedReplicationStatus | EXCLUDED | | Collects replication status information on an aggregator node |
| informationSchemaMvBackupHistory | | | Runs SELECT * FROM information_schema.MV_BACKUP_HISTORY on an aggregator node |
| informationSchemaMvBlockedQueries | | | Runs SELECT * FROM information_schema.MV_BLOCKED_QUERIES on an aggregator node |
| informationSchemaMvBottomless | | | Collects all of the MV_BOTTOMLESS tables, except MV_BOTTOMLESS_REMOTE_STATS from |
| | | | the information_schema on an aggregator node |
| informationSchemaMvBottomlessRemoteStats | EXCLUDED | | Collects the MV_BOTTOMLESS_REMOTE_STATS table from the information_schema on an |
| | | | aggregator node |
| informationSchemaMvEvents | | | Runs SELECT * FROM information_schema.MV_EVENTS on an aggregator node |
| informationSchemaMvNodes | | | Runs SELECT * FROM information_schema.MV_NODES on an aggregator node |
| informationSchemaMvQueries | | | Runs SELECT * FROM information_schema.MV_QUERIES on an aggregator node |
| informationSchemaMvReplicationManagement | | | Collects all of the MV_REPLICATION_MANAGEMENT tables from the information_schema |
| | | | on an aggregator node |
| informationSchemaMvSysinfo | | | Collects all of the MV_SYSINFO tables from the information_schema on an |
| | | | aggregator node |
| informationSchemaPipelines | | | Collects pipeline debugging information from the information_schema on an |
| | | | aggregator node |
| informationSchemaProcesslist | | | Records the PROCESSLIST for each node |
| informationSchemaRoutines | | | Collects all of the user-defined functions and stored procedures from the |
| | | | information_schema on an aggregator node |
| informationSchemaStatistics | | | Collects information_schema.statistics on each node |
| informationSchemaTableStatistics | | | Runs SELECT * FROM information_schema.TABLE_STATISTICS on an aggregator node |
| informationSchemaTables | | | Collects table metadata from each node |
| informationSchemaUserPrivileges | | | Collects information_schema.USER_PRIVILEGES from each node |
| installedPermissions | | | Collects information about file and directory permissions on a cluster |
| ipAddr | | | Records the output of `ip addr` |
| ipLink | | | Records the output of `ip link` |
| ipRoute | | | Records the output of `ip route` |
| licenseMetadata | | | Collects the output of `memsqlctl show-license` for each node |
| lsblk | | | Records the output of `lsblk` |
| lsblkRota | | | Records the output of `lsblk -d -o rota` |
| majorPageFaults | | | Records the output of `sar 1 10 -B` |
| memoryCommitted | | | Records the output of `sar 1 10 -r` |
| memsqlAuditlogs | EXCLUDED | | Collects node audit logs from a host |
| memsqlBacktraces | | | Collects backtrace information from each node |
| memsqlCommandLogs | | | Collects the command logs from each node |
| memsqlConfig | | | Collects the `memsql.cnf` file from each node |
| memsqlNumaConfig | | | Collects the NUMA configuration files from each node |
| memsqlPlancache | EXCLUDED | | Copies the plancache directory from each node |
| memsqlStacks | | | Collects stack files from each node |
| memsqlTracelogs | | | Collects the tracelogs from each node |
| memsqlctlInfo | | | Collects information about memsqlctl from each node |
| memsqldProcessLimits | | | Collects limits for all `memsqld` processes |
| mount | | | Records the output of `mount` |
| netstat | | | Records the output of `netstat -s` |
| nodeDirectoriesDiskUsage | | | Collects disk usage information for each node directory |
| nodeFilesDiskUsage | EXCLUDED | | Collects file-level disk usage information for each node directory |
| nodes | | | Collects the output of `memsqlctl describe-node` from each node |
| numactl | | | Records the output of `numactl --hardware` |
| osRelease | | | Collects information from those files in /etc that describe the host operating |
| | | | system |
| partitionsStart | EXCLUDED | | Collects information about each partition's start sector number |
| podLogs | | | Collects logs of Kube operator Pods (see memsqlTracelogs for logs of other Pods) |
| proc | | | Collects diagnostic files from /proc |
| procFs | | | Collects diagnostic files from /proc/fs |
| processFiles | EXCLUDED | | Collects top-level of /proc/<pid> for each SingleStore process |
| ps | | | Records the output of `ps aux` |
| psutil | | | Collects system and process information about the host |
| rebootTime | | | Records the output of `last -x reboot` |
| schema | | | Collects database schemas from each aggregator node |
| securityLimits | | | Collects /etc/security/limits.conf |
| showClusterStatus | | | Collects cluster status by running `SHOW CLUSTER STATUS` on an aggregator node |
| showCronJobs | | | Collects information about Kube CronJobs |
| showDatabaseStatus | | | Collects the database status for each database on each node |
| showDatabasesExtended | | | Runs SHOW DATABASES EXTENDED on each node |
| showDeployments | | | Collects information about Kube Deployments |
| showMemsqlClusters | | | Collects information about Kube MemsqlClusters |
| showPartitions | | | Collects database partition metadata from each aggregator node |
| showPersistentVolumeClaims | | | Collects information about Kube PersistentVolumeClaims |
| showPersistentVolumes | | | Collects information about Kube PersistentVolumes |
| showPlancache | | | Runs SHOW PLANCACHE on each node |
| showPods | | | Collects information about Kube Pods |
| showRebalanceStatus | | | Collects rebalance status for each database on the Master Aggregator |
| showReplicaSets | | | Collects information about Kube ReplicaSets |
| showReplicationStatus | | | Runs SHOW REPLICATION STATUS on each node |
| showResourcePools | | | Runs SHOW RESOURCE POOLS on each aggregator node |
| showServices | | | Collects information about Kube Services |
| showStatefulSets | | | Collects information about Kube StatefulSets |
| showStatusExtended | | | Runs SHOW STATUS EXTENDED on each node |
| showSvlAPITokenGroups | | | Collects information about Kube SvlAPITokenGroups |
| showSvlClusters | | | Collects information about Kube SvlClusters |
| showSvlDatabaseReplicas | | | Collects information about Kube SvlDatabaseReplicas |
| showSvlDatabases | | | Collects information about Kube SvlDatabases |
| showSvlSites | | | Collects information about Kube SvlSites |
| showTableStatus | | | Collects table status for each database on each aggregator node |
| showVariables | | | Runs SHOW GLOBAL VARIABLES LIKE '%' on each node |
| showWorkloadManagementStatus | | | Runs SHOW WORKLOAD MANAGEMENT STATUS on each aggregator node |
| shutdownTime | | | Records the output of `last -x shutdown` |
| studioFiles | | GLOBAL | Collects Studio configuration and state files |
| studioVersion | | GLOBAL | Records the output of `memsql-studio version --log /dev/null --state /dev/null` |
| swapUsage | | | Records the output of `sar 1 10 -S` |
| syncVariables | | | Runs query DUMP CLUSTER VARIABLES; on each node |
| sysctl | | | Records the output of `sysctl -a` |
| toolboxConfig | | GLOBAL | Collects Toolbox's configuration and version |
| toolboxEnv | | GLOBAL | Collects global toolbox config information |
| top | | | Records the output of `top -b -n 1` |
| transparentHugepage | | | Collects files from /sys/kernel/mm/transparent_hugepage |
| ulimit | | | Records the output of `sh -c ulimit -a` |
| uptime | | | Records the output of `uptime` |
+------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+

Examples

Collect information on all of the available clusters from the current namespace.

sdb-report collect-kube

Collect information on all of the available clusters from the namespace ns.

sdb-report collect-kube --namespace ns

Collect information on the cl cluster in the namespace ns.

sdb-report collect-kube --cluster-name cl --namespace ns

Collect information by specifying the kubeconfig.

sdb-report collect-kube --config-file ~/dir/.kube/config

Collect information using the namespace of the context c.

sdb-report collect-kube --context c

Collect information on SingleStore Helios v1 clusters only.

sdb-report collect-kube --version v1

Specify the container in which to run the report.

sdb-report collect-kube --container node

Usage

Usage:
sdb-report collect-kube [flags]
For flags that can accept multiple values (indicated by VALUES after the name of the flag),
separate each value with a comma.
Flags:
--cluster-name VALUES The specific cluster name(s) to discover, separated by commas. Leave blank for all clusters
--config-file FILE_PATH The kube config filepath. Leave blank if the path is stored in the KUBECONFIG env variable or in the ~/.kube/config file (ADVANCED)
--container string The name of the container in which to run the command. If not specified, the default container will be used (ADVANCED)
--context string The specific kube context to switch to when a kubeconfig has multiple contexts (ADVANCED)
--exclude VALUES Exclude the specified collectors
--exclude-global Exclude global collectors from the report file collector
-h, --help Help for collect-kube
--host strings Specify host(s) to collect from (collects from all hosts by default)
--include VALUES Include the specified collectors
--include-performance Include collectors that create load on cluster (not recommended for active clusters)
--mask-ip Mask usernames, hostnames, IP and MAC addresses in the report file collector
--merge-path string The directory from which to merge reports (ADVANCED)
--namespace VALUES The kube namespace(s) from which to collect reports, separated by commas. If missing, the default namespace will be used
--only VALUES Only run the specified collectors
--opt key=value Specify collector options (default informationSchemaPipelines.limitRows=None,informationSchemaStatistics.collectInformationSchemaStatisticsAllNodes=False,informationSchemaTables.collectInformationSchemaTablesAllNodes=False,memsqlAuditlogs.auditLogSize=10MB,memsqlAuditlogs.auditLogStartTimestamp=Not set,memsqlAuditlogs.compressAuditLog=False,memsqlCommandLogs.commandLogSize=10MB,memsqlCommandLogs.commandLogStartTimestamp=Not set,memsqlCommandLogs.compressCommandLog=False,memsqlTracelogs.compressTracelog=False,memsqlTracelogs.tracelogSize=10MB,memsqlTracelogs.tracelogStartTimestamp=Not set)
-o, --output-path ABSOLUTE_PATH Write the report tarball to this file path
--temp-dir ABSOLUTE_PATH The directory on the local and remote host(s) to store temporary files during collection (ADVANCED)
--validate-env Run collectors that do not require SingleStore installation (performance collectors included)
--version {both, v1, v2} Either v1 or v2; if missing, both v1 and v2 clusters will be discovered (ADVANCED) (default both)
Global Flags:
--backup-cache FILE_PATH File path for the backup cache
--cache-file FILE_PATH File path for the Toolbox node cache
-c, --config FILE_PATH File path for the Toolbox configuration
--disable-colors Disable color output in console, which some terminal sessions/environments may have difficulty with
--disable-spinner Disable the progress spinner, which some terminal sessions/environments may have issues with
-j, --json Enable JSON output
--parallelism POSITIVE_INTEGER Maximum number of operations to run in parallel
--runtime-dir DIRECTORY_PATH Where to store Toolbox runtime data
--ssh-control-persist SECONDS Enable SSH ControlPersist and set it to the specified duration in seconds
--ssh-max-sessions POSITIVE_INTEGER Maximum number of SSH sessions to open per host, must be at least 3
--ssh-strict-host-key-checking Enable strict host key checking for SSH connections
--ssh-user-known-hosts-file FILE_PATH Path to the user known_hosts file for SSH connections. If not set, /dev/null will be used
--state-file FILE_PATH Toolbox state file path
-v, --verbosity count Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
-y, --yes Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default

Remarks

This command is interactive unless you use either --yes or --json flag to override interactive behavior.

Categorization of Collectors

The collectors used in reporting gather cluster information related to alerting, performance, and pre-installation environment validation. The following table maps collectors with the category of the information reported by them.

Category

Collectors

Alerting

clusterTopology

cpuUtilization (also reports performance and environment validation details)

df (also reports environment validation details)

diskLatency (also reports performance and environment validation details)

explainOrphanDatabases

explainRebalancePartitions

informationSchemaMvNodes

majorPageFaults (also reports performance and environment validation details)

memoryCommitted (also reports performance and environment validation details)

memsqlctlInfo

netstat

showDatabasesExtended

showPartitions

showReplicationStatus

showVariables

swapUsage (also reports performance and environment validation details)

Performance

cpuMemoryBandwidth (also reports environment validation details)

cpuUtilization (also reports alerting and environment validation details)

diskBandwidth also reports environment validation details

diskLatency (also reports alerting and environment validation details)

majorPageFaults (also reports alerting and environment validation details)

memoryCommitted (also reports alerting and environment validation details)

swapUsage (also reports alerting and environment validation details)

Pre-installation Environment Validation

cpuFreqInfo

cpuMemoryBandwidth (also reports performance-related information)

cpuThreadingInfo

cpuUtilization (also reports alerting- and performance-related information)

df (also reports alerting-related information)

diskBandwidth (also reports performance-related information)

diskLatency (also reports alerting- and performance-related information)

dmesg

ethtool

free

ifconfig

ipAddr

ipLink

ipRoute

lsblk

lsblkRota

majorPageFaults (also reports alerting- and performance-related information)

memoryCommitted (also reports alerting- and performance-related information)

mount

nodes

numactl

osRelease

proc

procFs

ps

psutil

rebootTime

securityLimits

shutdownTime

studioVersion

swapUsage (also reports alerting- and performance-related information)

sysctl

toolboxConfig

top

transparentHugepage

ulimit

uptime

In Kube clusters, the following additional collectors are used:

podLogs

showCronJobs

showDeployments

showKubeMemsqlClusters

showKubeSvlClusters

showPods

showPVCs

showPVs

showReplicaSets

showServices

showStatefulSets

showSvlAPITokenGroups

showSvlDBReplicas

showSvlDBs

showSvlSites

Note that some collectors do not fall under any of the three categories and are used to collect more general information about the cluster.

Last modified: February 29, 2024

Was this article helpful?