collect-and-check
Description
Build a diagnostics report for the cluster and check it for issues.
Collectors and checkers are grouped into categories such as alerting, performance, and pre-installation environment validation. For more information, refer to Categorization of Collectors and Categorization of Checkers.
Available collectors
Note that some collectors are excluded by default.
+------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+ | ID | EXCLUDED | GLOBAL | DESCRIPTION | +------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+ | analyzeTables | EXCLUDED | | Save column statistical information, and histograms recorded on each table, into | | | | | a JSON file | | cgroupInfo | EXCLUDED | | Collects cgroup information for each SingleStoreDB process | | chronydStatus | EXCLUDED | | Collects status of the `chronyd` service | | clusterTopology | | | Runs SHOW LEAVES, SHOW AGGREGATORS, and SHOW LEAF STATUS EXTENDED on the Master | | | | | Aggregator | | cpuFreqInfo | | | Collects information about CPU frequency configuration | | cpuMemoryBandwidth | | | Measures CPU-memory bandwidth | | cpuThreadingInfo | | | Collects information about CPUs threading configuration | | cpuUtilization | | | Records the output of `sar 1 10 -u` | | dataSkewness | EXCLUDED | | Collects information about data skew in the cluster | | df | | | Records the output of `df -h | | | | | --output=source,size,used,avail,pcent,ipcent,target` | | diskBandwidth | | | Measures disk read/write bandwidth | | diskLatency | | | Records the output of `iostat -xdm 1 10` | | dmesg | | | Records the output of `dmesg -T` | | ethtool | | | Collects information about each network interface on the host | | explainOrphanDatabases | | | Runs EXPLAIN CLEAR ORPHAN DATABASES on the Master Aggregator | | explainRebalancePartitions | | | Runs EXPLAIN REBALANCE PARTITIONS for each user database | | explainRestoreRedundancy | | | Runs EXPLAIN RESTORE REDUNDANCY for each user database | | failedHosts | | GLOBAL | List of hosts where a report could not be collected | | free | | | Records the output of `free -m` | | hosts | | | Collect information about cluster's hosts | | ifconfig | | | Records the output of `ifconfig` | | informationSchemaAvailabilityGroups | | | Collects information about availability groups from the information_schema on an | | | | | aggregator node | | informationSchemaColumnarSegments | EXCLUDED | | Collects columnstore debugging information from the information_schema on an | | | | | aggregator node | | informationSchemaDistributedDatabases | | | Collects distributed databases names on an aggregator node | | informationSchemaDistributedDatabasesOnSites | | | Collects information about database distribution across clusters | | informationSchemaIndexStatistics | | | Runs SELECT * FROM information_schema.INDEX_STATISTICS on an aggregator node | | informationSchemaInternalTableStatistics | EXCLUDED | | Runs SELECT * FROM information_schema.INTERNAL_TABLE_STATISTICS on an aggregator | | | | | node | | informationSchemaMvAggregatedReplicationStatus | EXCLUDED | | Collects replication status information on an aggregator node | | informationSchemaMvBackupHistory | | | Runs SELECT * FROM information_schema.MV_BACKUP_HISTORY on an aggregator node | | informationSchemaMvBlockedQueries | | | Runs SELECT * FROM information_schema.MV_BLOCKED_QUERIES on an aggregator node | | informationSchemaMvBottomless | | | Collects all of the MV_BOTTOMLESS tables from the information_schema on an | | | | | aggregator node | | informationSchemaMvEvents | | | Runs SELECT * FROM information_schema.MV_EVENTS on an aggregator node | | informationSchemaMvNodes | | | Runs SELECT * FROM information_schema.MV_NODES on an aggregator node | | informationSchemaMvQueries | | | Runs SELECT * FROM information_schema.MV_QUERIES on an aggregator node | | informationSchemaMvReplicationManagement | | | Collects all of the MV_REPLICATION_MANAGEMENT tables from the information_schema | | | | | on an aggregator node | | informationSchemaMvSysinfo | | | Collects all of the MV_SYSINFO tables from the information_schema on an | | | | | aggregator node | | informationSchemaPipelines | | | Collects pipeline debugging information from the information_schema on an | | | | | aggregator node | | informationSchemaProcesslist | | | Records the PROCESSLIST for each node | | informationSchemaRoutines | | | Collects all of the user-defined functions and stored procedures from the | | | | | information_schema on an aggregator node | | informationSchemaStatistics | | | Collects information_schema.statistics on each node | | informationSchemaTableStatistics | | | Runs SELECT * FROM information_schema.TABLE_STATISTICS on an aggregator node | | informationSchemaTables | | | Collects table metadata from each node | | informationSchemaUserPrivileges | | | Collects information_schema.USER_PRIVILEGES from each node | | installedPermissions | | | Collects information about file and directory permissions on a cluster | | ipAddr | | | Records the output of `ip addr` | | ipLink | | | Records the output of `ip link` | | ipRoute | | | Records the output of `ip route` | | licenseMetadata | | | Collects the output of `memsqlctl show-license` for each node | | lsblk | | | Records the output of `lsblk` | | lsblkRota | | | Records the output of `lsblk -d -o rota` | | majorPageFaults | | | Records the output of `sar 1 10 -B` | | memoryCommitted | | | Records the output of `sar 1 10 -r` | | memsqlAuditlogs | EXCLUDED | | Collects node audit logs from a host | | memsqlBacktraces | | | Collects backtrace information from each node | | memsqlCommandLogs | | | Collects the command logs from each node | | memsqlConfig | | | Collects the `memsql.cnf` file from each node | | memsqlNumaConfig | | | Collects the NUMA configuration files from each node | | memsqlPlancache | EXCLUDED | | Copies the plancache directory from each node | | memsqlStacks | | | Collects stack files from each node | | memsqlTracelogs | | | Collects the tracelogs from each node | | memsqlctlInfo | | | Collects information about memsqlctl from each node | | memsqldProcessLimits | | | Collects limits for all `memsqld` processes | | mount | | | Records the output of `mount` | | netstat | | | Records the output of `netstat -s` | | nodeDirectoriesDiskUsage | | | Collects disk usage information for each node directory | | nodeFilesDiskUsage | EXCLUDED | | Collects file-level disk usage information for each node directory | | nodes | | | Collects the output of `memsqlctl describe-node` from each node | | numactl | | | Records the output of `numactl --hardware` | | osRelease | | | Collects information from those files in /etc that describe the host operating | | | | | system | | partitionsStart | EXCLUDED | | Collects information about each partition's start sector number | | proc | | | Collects diagnostic files from /proc | | procFs | | | Collects diagnostic files from /proc/fs | | processFiles | EXCLUDED | | Collects top-level of /proc/<pid> for each SingleStoreDB process | | ps | | | Records the output of `ps aux` | | psutil | | | Collects system and process information about the host | | rebootTime | | | Records the output of `last -x reboot` | | schema | | | Collects database schemas from each aggregator node | | securityLimits | | | Collects /etc/security/limits.conf | | showClusterStatus | | | Collects cluster status by running `SHOW CLUSTER STATUS` on an aggregator node | | showDatabaseStatus | | | Collects the database status for each database on each node | | showDatabasesExtended | | | Runs SHOW DATABASES EXTENDED on each node | | showPartitions | | | Collects database partition metadata from each aggregator node | | showPlancache | | | Runs SHOW PLANCACHE on each node | | showRebalanceStatus | | | Collects rebalance status for each database on the Master Aggregator | | showReplicationStatus | | | Runs SHOW REPLICATION STATUS on each node | | showResourcePools | | | Runs SHOW RESOURCE POOLS on each aggregator node | | showStatusExtended | | | Runs SHOW STATUS EXTENDED on each node | | showTableStatus | | | Collects table status for each database on each aggregator node | | showVariables | | | Runs SHOW VARIABLES LIKE '%' on each node | | showWorkloadManagementStatus | | | Runs SHOW WORKLOAD MANAGEMENT STATUS on each aggregator node | | shutdownTime | | | Records the output of `last -x shutdown` | | studioVersion | | GLOBAL | Records the output of `memsql-studio version --log /dev/null --state /dev/null` | | swapUsage | | | Records the output of `sar 1 10 -S` | | syncVariables | | | Runs query DUMP CLUSTER VARIABLES; on each node | | sysctl | | | Records the output of `sysctl -a` | | toolboxConfig | | GLOBAL | Collects Toolbox's configuration and version | | toolboxEnv | | GLOBAL | Collects global toolbox config information | | top | | | Records the output of `top -b -n 1` | | transparentHugepage | | | Collects files from /sys/kernel/mm/transparent_hugepage | | ulimit | | | Records the output of `sh -c ulimit -a` | | uptime | | | Records the output of `uptime` | +------------------------------------------------+----------+--------+----------------------------------------------------------------------------------+
Examples
Run a single collector.
sdb-report collect-and-check --only procFs
Exclude specific collectors.
sdb-report collect-and-check --exclude procFs --exclude securityLimits
Include those collectors that are excluded by default.
sdb-report collect-and-check --include informationSchemaColumnarSegments
Runs all included collectors by default. The --all option has been deprecated.
sdb-report collect-and-check --all
Increase the amount of data collected from tracelogs to 15MB (default is 10MB).
sdb-report collect-and-check --opt memsqlTracelogs.tracelogSize=15mb
Usage
Usage:
sdb-report collect-and-check [flags]
Flags:
--exclude VALUES Exclude the specified collectors
--exclude-global Exclude global collectors from the report file collector
-h, --help Help for collect-and-check
--host strings Specify host(s) to collect from (collects from all hosts by default)
--include VALUES Include the specified collectors
--include-performance Include collectors that create load on cluster (not recommended for active clusters)
--mask-ip Mask usernames, hostnames, IP and MAC addresses in the report file collector
--only VALUES Only run the specified collectors
--opt key=value Specify collector options (default informationSchemaPipelines.limitRows=None,informationSchemaStatistics.collectInformationSchemaStatisticsAllNodes=False,informationSchemaTables.collectInformationSchemaTablesAllNodes=False,memsqlAuditlogs.auditLogSize=10MB,memsqlAuditlogs.auditLogStartTimestamp=Not set,memsqlAuditlogs.compressAuditLog=False,memsqlCommandLogs.commandLogSize=10MB,memsqlCommandLogs.commandLogStartTimestamp=Not set,memsqlCommandLogs.compressCommandLog=False,memsqlTracelogs.compressTracelog=False,memsqlTracelogs.tracelogSize=10MB,memsqlTracelogs.tracelogStartTimestamp=Not set)
--shard-queries Shard queries across aggregators. All queries are run on the Master Aggregator node by default (ADVANCED)
--show-skips Display more information about skipped checks
--temp-dir ABSOLUTE_PATH The directory on the local and remote host(s) to store temporary files during collection (ADVANCED)
--validate-env Run collectors that do not require SingleStoreDB installation (performance collectors included)
Global Flags:
--backup-cache FILE_PATH File path for the backup cache
--cache-file FILE_PATH File path for the Toolbox node cache
-c, --config FILE_PATH File path for the Toolbox configuration
--disable-colors Disable color output in console, which some terminal sessions/environments may have difficulty with
--disable-spinner Disable the progress spinner, which some terminal sessions/environments may have issues with
-j, --json Enable JSON output
--parallelism POSITIVE_INTEGER Maximum number of operations to run in parallel
--runtime-dir DIRECTORY_PATH Where to store Toolbox runtime data
--ssh-control-persist SECONDS Enable SSH ControlPersist and set it to the specified duration in seconds
--ssh-max-sessions POSITIVE_INTEGER Maximum number of SSH sessions to open per host, must be at least 3
--ssh-strict-host-key-checking Enable strict host key checking for SSH connections
--ssh-user-known-hosts-file FILE_PATH Path to the user known_hosts file for SSH connections. If not set, /dev/null will be used
--state-file FILE_PATH Toolbox state file path
-v, --verbosity count Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
-y, --yes Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default
Remarks
This command is interactive unless you use either --yes or --json flag to override interactive behavior.