SingleStore DB

calibrate
Description

Run ‘calibrate’ to measure cluster performance.

Usage
Run 'calibrate' to measure cluster performance.

A 'calibrate' database is created with rowstore and columnstore tables that are populated with data. A number of queries are run against this data to measure cluster performance. 

The 'calibrate' tool requires a dataset, which will be loaded into tables. By default, the dataset is downloaded from the web, unpacked, and deleted at the end of the run. However, if your cluster cannot access the internet or already has the dataset, use the '--data-path' flag, which is an absolute path to the dataset on the Master Aggregator node. 
You can download the dataset here: https://s3.amazonaws.com/calibrate.singlestore.com/calibrate-dataset.tar.gz

The dataset download uses 1.5 Gb of disk space, and the unpacking of data takes up another 12 Gb.

The '--partition-ratio' flag is used to test clusters under different concurrency conditions, such as, four CPU cores per partition. This flag specifies a proportion of CPU cores to Partitions and defaults to 2:1. 

Workload consists of the following steps:
 - Download the dataset
 - Create the 'calibrate' database and set up the required variables
 - Load data into tables
 - Run calibration queries
 - Retrieve results of the run and save them in a file on the Master Aggregator node

Workload duration divided into three main categories:
 - FAST: under 12 minutes
 - INTERMEDIATE: between 12 and 16 minutes
 - SLOW: over 16 minutes

In case the cluster gets 'SLOW' or 'INTERMEDIATE' result, the tool will print data about the run, in addition to the file version, into the terminal.

Examples:

# A simple example that runs 'calibrate' workload with the root user of the cluster and downloads dataset from the internet
sdb-report calibrate

# Runs 'calibrate' workload with the user 'user' and password 'pass' with a dataset on path '/home/admin/dataset.tar.gz' on the Master Aggregator node
sdb-report calibrate -u usr -p pass --data-path /home/admin/dataset.tar.gz

# Run calibrate to test non-default partition ratio and retain calibrate database after the run
sdb-report calibrate --partition-ratio 4:1 --retain-database

Execution time of the following queries will be measured:

Load data queries:

+--------------+------------------------------------------------------------------------------------------------------+
|     NAME     |                                                 SQL                                                  |
+--------------+------------------------------------------------------------------------------------------------------+
| Load Queries | LOAD /*! calibrate_load_rs_1 */ DATA INFILE '{{CALIBRATE_LOAD_RS_1}}' INTO TABLE disttable2 FIELDS   |
|              | TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                              |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_2 */ DATA INFILE '{{CALIBRATE_LOAD_CS_2}}' INTO TABLE disttable2_cs       |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_rs_3 */ DATA INFILE '{{CALIBRATE_LOAD_RS_3}}' INTO TABLE foreignstring440k   |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_4 */ DATA INFILE '{{CALIBRATE_LOAD_CS_4}}' INTO TABLE                     |
|              | foreignstring440k_cs FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';  |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_rs_5 */ DATA INFILE '{{CALIBRATE_LOAD_RS_5}}' INTO TABLE primarystring2      |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
|              |                                                                                                      |
|              | LOAD /*! calibrate_load_cs_6 */ DATA INFILE '{{CALIBRATE_LOAD_CS_6}}' INTO TABLE primarystring2_cs   |
|              | FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' LINES TERMINATED BY '\n';                       |
+--------------+------------------------------------------------------------------------------------------------------+


Calibration queries:

+--------------------------------+------------------------------------------------------------------------------------------------------+
|              NAME              |                                                 SQL                                                  |
+--------------------------------+------------------------------------------------------------------------------------------------------+
| Reshuffle Joins                | SELECT /*! calibrate_reshuff_rs_q1    */ COUNT(1) FROM ( SELECT b.* FROM  disttable2 a STRAIGHT_JOIN |
|                                | primarystring2 b ON a.a=b.a) A;                                                                      |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_rs_cs_q2 */ COUNT(1) FROM ( SELECT b.* FROM  disttable2 a STRAIGHT_JOIN |
|                                |  primarystring2_cs b ON a.a=b.a) A;                                                                  |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_rs_cs_q3 */ COUNT(1) FROM ( SELECT b.* FROM  disttable2_cs a            |
|                                | STRAIGHT_JOIN  primarystring2 b ON a.a=b.a) A;                                                       |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_cs_q4    */ COUNT(1) FROM ( SELECT b.* FROM  disttable2_cs a            |
|                                | STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a) A;                                                    |
|                                |                                                                                                      |
| Broadcast Joins                | SELECT /*! calibrate_bcast_rs_q1     */ COUNT(1) FROM ( SELECT b.* FROM disttable2 a  STRAIGHT_JOIN  |
|                                | WITH(broadcast_left=true) primarystring2 b ON a.a=b.a) A;                                            |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_rs_cs_q2  */ COUNT(1) FROM ( SELECT b.* FROM disttable2 a  STRAIGHT_JOIN  |
|                                | WITH(broadcast_left=true) primarystring2_cs b ON a.a=b.a) A;                                         |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_rs_cs_q3  */ COUNT(1) FROM ( SELECT b.* FROM disttable2_cs a              |
|                                | STRAIGHT_JOIN WITH(broadcast_left=true) primarystring2 b ON a.a=b.a) A;                              |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_bcast_cs_q4     */ COUNT(1) FROM ( SELECT b.* FROM disttable2_cs a              |
|                                | STRAIGHT_JOIN WITH(broadcast_left=true) primarystring2_cs b ON a.a=b.a) A;                           |
|                                |                                                                                                      |
| Local Group By                 | SELECT /*! calibrate_l_gby_rs_q1     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k.a),primarystring2.a FROM foreignstring440k STRAIGHT_JOIN primarystring2      |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a GROUP BY                  |
|                                | primarystring2.a) A;                                                                                 |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_rs_cs_q2  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k_cs.a),primarystring2.a FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a GROUP   |
|                                | BY primarystring2.a) A;                                                                              |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_rs_cs_q3  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k.a),primarystring2_cs.a FROM foreignstring440k STRAIGHT_JOIN                  |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a      |
|                                | GROUP BY primarystring2_cs.a) A;                                                                     |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_l_gby_cs_q4     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT               |
|                                | COUNT(foreignstring440k_cs.a),primarystring2_cs.a FROM foreignstring440k_cs STRAIGHT_JOIN            |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a   |
|                                | GROUP BY primarystring2_cs.a) A;                                                                     |
|                                |                                                                                                      |
| Local Joins                    | SELECT /*! calibrate_lj_rs_q1       */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN primarystring2  |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a;                          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_rs_cs_q2    */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN              |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a;        |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_rs_cs_q3    */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN                 |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a;     |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_cs_q4       */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN              |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a;  |
|                                |                                                                                                      |
| Single Part Join and Filter    | SELECT /*! calibrate_spj_rs_q1     */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN primarystring2   |
|                                | ON primarystring2.a=foreignstring440k.a AND primarystring2.a = 'A1000027';                           |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_rs_cs_q2  */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2  ON primarystring2.a=foreignstring440k_cs.a AND primarystring2.a = 'A1000027';        |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_rs_cs_q3  */ COUNT(1) FROM foreignstring440k STRAIGHT_JOIN                  |
|                                | primarystring2_cs  ON primarystring2_cs.a=foreignstring440k.a AND primarystring2_cs.a = 'A1000027';  |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_spj_cs_q4     */ COUNT(1) FROM foreignstring440k_cs STRAIGHT_JOIN               |
|                                | primarystring2_cs  ON primarystring2_cs.a=foreignstring440k_cs.a AND primarystring2_cs.a =           |
|                                | 'A1000027';                                                                                          |
|                                |                                                                                                      |
| Reshuffle and Distributed      | SELECT /*! calibrate_reshuff_dgby_rs_q1    */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
| Group By                       | COUNT(b.a) FROM disttable2 a STRAIGHT_JOIN  primarystring2 b ON a.a=b.a GROUP BY a.b) A;             |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_rs_cs_q2 */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2_cs a STRAIGHT_JOIN  primarystring2 b ON a.a=b.a GROUP BY a.b) A;          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_rs_cs_q3 */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2 a STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a GROUP BY a.b) A;          |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_reshuff_dgby_cs_q4    */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT         |
|                                | COUNT(b.a) FROM disttable2_cs a STRAIGHT_JOIN  primarystring2_cs b ON a.a=b.a GROUP BY a.b) A;       |
|                                |                                                                                                      |
| Local Join and Distributed     | SELECT /*! calibrate_lj_dgby_rs_q1     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
| Group By                       | COUNT(foreignstring440k.a),primarystring2.id FROM foreignstring440k STRAIGHT_JOIN primarystring2     |
|                                | WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k.a GROUP BY                  |
|                                | primarystring2.id) A;                                                                                |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_rs_cs_q2  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                |  COUNT(foreignstring440k_cs.a),primarystring2.id FROM foreignstring440k_cs STRAIGHT_JOIN             |
|                                | primarystring2 WITH(table_convert_subSELECT=true) ON primarystring2.a=foreignstring440k_cs.a GROUP   |
|                                | BY primarystring2.id) A;                                                                             |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_rs_cs_q3  */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                |  COUNT(foreignstring440k.a),primarystring2_cs.id FROM foreignstring440k STRAIGHT_JOIN                |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k.a      |
|                                | GROUP BY primarystring2_cs.id) A;                                                                    |
|                                |                                                                                                      |
|                                | SELECT /*! calibrate_lj_dgby_cs_q4     */ WITH(leaf_pushdown=true) COUNT(1) FROM (SELECT             |
|                                | COUNT(foreignstring440k_cs.a),primarystring2_cs.id FROM foreignstring440k_cs STRAIGHT_JOIN           |
|                                | primarystring2_cs WITH(table_convert_subSELECT=true) ON primarystring2_cs.a=foreignstring440k_cs.a   |
|                                | GROUP BY primarystring2_cs.id) A;                                                                    |
+--------------------------------+------------------------------------------------------------------------------------------------------+

Usage:
  sdb-report calibrate [flags]

Flags:
      --data-path string        Absolute path to the folder containing calibration datasets on the Master Aggregator
  -h, --help                    Help for calibrate
      --partition-ratio RATIO   CPU Core:DB Partition ratio, both values must be less than or equal to 16 (default 2:1)
  -p, --password STRING         SQL password for connecting to SingleStore DB
      --retain-database         Retain the calibrate database after calibration process ends
  -u, --user string             SQL user for connecting to SingleStore DB (default "root")

Global Flags:
      --backup-cache FILE_PATH              File path for the backup cache
      --cache-file FILE_PATH                File path for the Toolbox node cache
  -c, --config FILE_PATH                    File path for the Toolbox configuration
      --disable-colors                      Disable colored output in console, which some terminal sessions/environments may have issues with
      --disable-spinner                     Disable the progress spinner, which some terminal sessions/environments may have issues with
  -j, --json                                Enable JSON output
      --parallelism POSITIVE_INTEGER        Maximum number of operations to run in parallel
      --runtime-dir DIRECTORY_PATH          Where to store Toolbox runtime data
      --ssh-max-sessions POSITIVE_INTEGER   Maximum number of SSH sessions to open per host, must be at least 3
      --state-file FILE_PATH                Toolbox state file path
  -v, --verbosity count                     Increase logging verbosity: valid values are 1, 2, 3. Usage -v=count or --verbosity=count
  -y, --yes                                 Enable non-interactive mode and assume the user would like to move forward with the proposed actions by default
Remarks

This command is interactive unless you use either --yes or --json flag to override interactive behavior.