# Maintenance Release Changelog ## 2023-05-31 Version 7.5.25 * Fixed an issue where errors could occur during backup when large files are copied. * The `sql_mode` session variable now shows up in `SHOW VARIABLES` and the mv\_global\_variables view. * Fixed an issue that could cause an incorrect error message to be generated on an out of memory error. * Fixed an issue where `CLEAR ORPHAN DATABASE` could cause data loss if run when the master aggregator has a database in an unrecoverable state. * Updated timezone metadata to include Mexico's latest timezone change. ## 2023-03-21 Version 7.5.24 * Improved the parsing performance of queries that contain several tables. * Fixed an issue causing incorrect trace messages in master logs where clocks were incorrectly advancing from "0". * Fixed the case where the `REGEXP_REPLACE` and `REGEXP_SUBSTR` expressions can produce non-utf8 strings by introducing the new [engine variable](https://docs.singlestore.com/db/v9.1/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md)`regexp_output_validation_mode`. Regular expression built-ins can produce non-utf8 strings because they don't have full support for multi-byte characters. The engine variable controls this behavior if regular expression built-ins return strings that are invalid under its collation settings. * Improved the performance on columnstore scans that perform multiple JSON extraction operations on the same JSON column. * Fixed an issue with databases becoming unrecoverable when upgrading to version 8.0 if any tables contained shard keys using computed columns that call `JSON_EXTRACT_STRING`. * Fixed a MySQL protocol incompatibility where SingleStore was not properly sending back the SERVER\_STATUS\_IN\_TRANS flag. This was impacting some MySQL php drivers. * Fixed a crash when parsing certain Parquet data into a pipeline. * The `SHOW STATUS EXTENDED` command contains a new "`Gv_clock`" key whose value is the current logical clock of the server. * The `SHOW DATABASE STATUS` command contains a new "`gv_clock`" key whose value is the current logical clock of the server. ## 2023-01-06 Version 7.5.23 * Fixed an edge case issue which could cause the engine to hang during shutdown. * Fixed some error handling issues with unlimited storage download and upload processes. * Fixed a bug that may cause a query to hang when comparing an utf8 column with an utf8mb4 constant. This issue occurs when `collation_server` is set to one of the utf8mb4 collations. * Fixed an edge case issue where the engine could crash when performing multi-inserts. * Fixed an issue with a transaction leak on the master aggregator when running `CREATE TABLE AS SELECT` on a child aggregator using `autocommit=0` on the connection. * Fixed an issue with aggregate functions using incorrect enum/set types that may result in inaccurate output in the operator tree. * Fixed an issue where environments with several nodes having several partitions may not auto-attach when restarted simultaneously. * Fixed a performance issue when creating an aggregator result table on the master aggregator. * Fixed an issue when if auto-creation is turned off (`auto.create.topics.enable=false`), `SELECT INTO KAFKA` would return a success message even if the insert fails. * Improved performance of comparing utf8mb4 strings. * Added the ability to allow code-generation of queries against the information schema when the cluster is very low on disk/memory. * Fixed a potential issue using multi-statement batches inside of prepared statements without parameters that may result in a crash. * Fixed a potential issue that could cause a lost second prepare statement when using a binary protocol over SSL. * Fixed a potential issue that could cause a NFS backup error to trigger a crash. * Reduced the memory usage of querying information\_schema.pipeline\_errors when there are many past pipeline errors. * Fixed an issue where environments with several nodes having several partitions may not auto-attach when restarted simultaneously. * Fixed an issue which caused unlimited database S3 uploads to fail due to an ExpiredToken error. This issue impacted the refresh of EC2 instance meta credentials caused by the public open-source AWS C++ SDK after encountering a transient network issue. * Fixed a specific query shape which failed CodeGen for columnstore unique keys. * Allow `REVOKE` from `ROLE` permissions for tables that have been dropped. * Fixed an issue that may cause a restarted master aggregator to crash when the STOP PIPELINE command runs on a detached pipeline. * Fixed an issue where having too many rows on the same columnstore hash key could return "Read failed with error 14 (Bad address)". * Fixed a potential issue when a query contains a full outer join and a window function partitioning by the full outer join columns yields the wrong result. * Fixed an issue where an incorrect result could be returned when a query contains multiple joins that all match a multi-column hash index. * Fixed a performance issue with count distinct aggregation and streaming subqueries. ## 2022-09-14 Version 7.5.22 * Fixed an issue with the accuracy of memory and disk space availability confirmed in the `REPLICATE DATABASE` command. This fix will ensure any secondary cluster has enough memory and disk space to replicate the database. * Fixed an issue where commands forwarded to the master aggregator during an online upgrade could cause a crash. ## 2022-09-07 Version 7.5.21 * Fixed an issue where canceling a `RESTORE` would not completely drop the partially restored database. * Fixed a memory leak issue due to `alloc_warnings` when using reference table pipelines. * Removed ability to use `RESTORE DATABASE` on unlimited databases with sync durability enabled. * Fixed an issue where S3 can silently fail to upload a file for an unlimited database. * Removed ability to use `utf8_general_mysql500_ci collation`. This collation is not supported by SingleStore. * Fixed an issue with window functions which refer to the window definition at the end of a `SELECT` statement. * Improved the `Subselect To Join` rewrite to recognize more cases with subselects in filter conditions for left join. * Added support for `s3_force_path_style` to S3 backup. `s3_force_path_style` (optional boolean, defaults to true) specifies whether to use path style (region.amazonaws.com/bucket) or virtual address style (bucket.region.amazonaws.com) syntax when specifying the location of the bucket. For S3 compatible services that only support virtual address paths, specify `"s3_force_path_style":"false"` in the configuration options. Refer to the [BACKUP DATABASE](https://docs.singlestore.com/db/v9.1/reference/sql-reference/operational-commands/backup-database.md) section for more information. * Fixed an issue that may cause insert `into mv_queries` to become invalid when the cluster is in `ONLY_FULL_GROUP_BY sql_mode`. * Fixed an edge case issue with the blob cache eviction algorithm may cause an infinite loop. * Fixed an issue when specifying all columns in the `SELECT` command resulted in columns being out of order rather than matching the table definition. ## 2022-07-25 Version 7.5.20 * Improved the performance of downloading blob files for unlimited databases when SSL is enabled between SingleStore and the blobstore. * Improved the performance of hash joins over expressions. * Fixed an incorrect error behavior when dropping and adding a multicolumn columnstore hash index on the same set of columns in a single alter. * Reduced the memory used by the blob cache by 100-200 bytes per cached blob. * Fixed HDFS pipelines failures when loading from HA/BigIP clusters that advertise a different host name than the one used to connect. * Fixed an issue where the incorrect backup size showed in `mv_backup_history` for incremental backups. * Reduce the memory used to track cached blob files and run scans over columnstore tables. * Fixed a server crash caused by serializing big table or view definitions in snapshots. * Added support for `NOT NULL` when using persisted computed columns. * Fixed a potential issue with transaction rolled back-mid query errors that may occur when a database name has a number suffix. * Added support for Reloading SSL Certs without having to restart the node. * Added a new engine variable named `columnstore_small_blob_combination_threshold` that can be used to control the size of column files created by combining compressed columnstore segments. Its default value is 32 Mb (the previously hard-coded value in the engine). * Added for compatibility with other databases: * Global engine variable: `default_storage_engine`. * Information schema field: `information_schema.COLUMNS DATETIME_PRECISION`. * Fixed a server crash caused by a rewrite trying to inline nested expressions. * Fixed an issue where `DROP FUNCTION` would result in an unknown table error. * Added an engine variable `columnstore_validate_blob_before_merge`. If enabled, the [background merger](https://docs.singlestore.com/db/v9.1/create-a-database/columnstore/managing-columnstore-segments.md) would validate the blob checksums before merging blobs. * Fixed issues with the `information_schema.mv_activities` and `information_schema.mv_activities_extended` tables reporting incorrect activities on database names ending with an integer. Refer to the [CREATE DATABASE](https://docs.singlestore.com/db/v9.1/reference/sql-reference/data-definition-language-ddl/create-database.md) section for more information. ## 2022-05-02 Version 7.5.19 * Fixed a server crash caused by table-valued functions parameters that assume the parameter name is the same as the alias of the table-valued function operator. * Fixed an issue with out of memory errors being generated while compiling queries that use multiple `UNION` clauses. * Adjusted the maximum memory allocation function to proactively run even when the `total_server_memory` usage is well below the `maximum_memory` allocation. * Fixed an issue where an incorrect error was generated for an aggregator result table when it was used by a parallel read. `DROP RESULT TABLE` now returns the correct error. * Improved heuristic estimates for queries with complex join predicates. * Fixed an issue with incremental autostats where duplicate rows were included in the autostats resulting in invalid statistics in a disaster recovery cluster when executing an `ALTER` statement in the primary cluster. * Added a new engine variable named `columnstore_small_blob_combination_threshold` that can be used to control the size of column files created by combining compressed columnstore segments. Its default value is 32 Mb (the previously hard-coded value in the engine). * Added for compatibility with other databases: * Global engine variable: `default_storage_engine`. * Information schema field: i`nformation_schema.COLUMNS DATETIME_PRECISION`. * Fixed a server crash caused by a rewrite trying to inline nested expressions. * Fixed an issue where `DROP FUNCTION` would result in an unknown table error. ## 2022-03-14 Version 7.5.18 * Fixed an issue with calling nested stored procedures that have different definers. ## 2022-03-07 Version 7.5.17 * Allowed the execution of `UPDATE/UPSERT` command on sort key columns via pipelines. Prior to this update, an error was generated. * Fixed an issue where table-valued functions crashed when a parameter is used more than once while compiling the query. * Improved the performance of sampling queries executed by the query optimizer. * Fixed the `CLEAR ORPHAN DATABASES` command to ensure all databases are dropped when the `leaf_failover_fanout` global variable is set to `load_balanced` and there are partition databases with more than replica partition. * Fixed an issue with query forwarding incorrectly triggering aggregator sync causing the aggregators to run out of disk space. * Fixed an issue with incorrect `nullCount` and `rowCount` statistics for larger tables with autostats enabled. * Fixed an issue with `memsql_exporter` ceasing to produce monitoring results. * Fixed an issue with C# MySql client drivers getting "no database errors" on connection. * Improved rowstore garbage collection by eliminating the delay for unlimited storage uploads to occur after a snapshot. * Fixed an issue with partitions in unlimited storage databases going into an unrecoverable state while snapshots download during server restarts. * Fixed the `CREATED` column in the `information_schema.ROUTINES` table by now recording the creation date for table-valued functions. * Fixed a bug that incorrectly pushes a join before a window function when attempting to perform a distributed join in an aggregator. * Fixed a bug that a variety of conversion operators failed to check errors during the construction of the operator tree. ## 2022-01-24 Version 7.5.16 * Fixed timeout errors when performing large `BACKUP/RESTORE` operations against an Azure blob store. * Fixed an issue where utf8mb4 characters were mistakenly extracted to the resulting utf8 string for the `JSON_EXTRACT_STRING` command. * Fixed a cluster database log corruption that could occur if the root user was dropped when `sync_permissions` was enabled. * Improved the predicate transitivity rewrite during query execution to cover more cases in an `EXISTS/NOT EXISTS` clause. * Stopped row-level locking when an `UPDATE` or `DELETE` join condition uses a full-text index. ## 2022-01-03 Version 7.5.15 * Fixed an issue with garbage collection in unlimited storage databases that could cause a point in time restore to fail. * Fixed an issue with inconsistent snapshots caused by running `SNAPSHOT DATABASE` concurrently with an `ALTER TABLE` command against a columnstore table. * Clarified error messaging when an incorrect configuration is given while creating an unlimited storage database. * Reduced the probability of failed connections to clusters due to WebSockets proxy issues. * Added support for `multipart_chunk_size_mb` configuration if `compatibility_mode=true`. See [SELECT … INTO S3](https://docs.singlestore.com/db/v9.1/reference/sql-reference/data-manipulation-language-dml/select/#select-into-s-3.md) for more information on the `multipart_chunk_size_mb` configuration. * Fixed the calculation for `total_runtime_ms` for queries profiled from within stored procedures. Prior to this release, `total_runtime_ms` was not calculated accurately. * Improved how the engine starts the prometheus exporter. * Added an engine variable `columnstore_validate_blob_before_merge`. If enabled, the [background merger](https://docs.singlestore.com/db/v9.1/create-a-database/columnstore/managing-columnstore-segments.md) would validate the blob checksums before merging blobs. ## 2021-11-30 Version 7.5.14 * Fixed the default value for `max_async_compilation_concurrency` so that it is now set appropriately when [control groups](https://en.wikipedia.org/wiki/Cgroups) are in use limiting CPU usage. * Added [control group](https://en.wikipedia.org/wiki/Cgroups) awareness for `information_schema.mv_nodes` which now displays the correct number of CPUs a process has access to. * Fixed an issue with queries not assigned to resource group pools with hard CPU limits from using the CPU assigned to the pool. * Fixed an issue with a crash occurring when an incremental backup was performed on an empty database. * Clarified error messaging when invalid data is inserted into a utf8 charset data type column. * Fixed an issue with an aggregator results table checking `INSERT` privileges. * Fixed an issue where results data is incorrectly blank in particular columns when querying `information_schema.mv_plancache`. * Fixed a race condition involving concurrent `UPDATE/UPSERT` queries that update sort keys against columnstore tables. * Fixed an issue with a crash occurring when DDL is run inside a stored procedure that is executed within a resource pool on a child aggregator. * Block the `DROP TEMPORARY TABLE` command from running in a transaction that writes to the temporary table being dropped. The `DROP TEMPORARY TABLE` command result will display an error. * Removed the disk space check that occurred when `ALTER TABLE ADD INDEX` is run. The check occurred prior to altering a columnstore table in an unlimited storage database. * Fixed an intermittent "Failed to GetObject" error when restoring a non-unlimited storage database into a cluster where unlimited storage is configured. * Locked down `SET` clauses that attempt to update sparse columns in correlated subselects. * Added support for `MAX_ERRORS` clause use when using `LOAD DATA` into columnstore tables with unique keys. * Removed restrictions on nested `SELECT` queries that utilize the `PARTITION_ID()` function in their intermediate results. * Improved the performance of columnstore `UPDATE` queries that do not modify unique key columns. ## 2021-11-15 Version 7.5.13 * Improved the performance of `UPDATE` queries that do not modify unique key columns against columnstore tables. * Added support for `BACKUP DATABASE` into S3 buckets using the `sts_regional_endpoints` configuration option. * Introduced `mv_bottomless_summary``information_schema` table that shows various aggregated statistics about unlimited storage databases. The `mv_bottomless_summary` table is an at-a-glance rollup table that shows what is happening **right now** with the unlimited storage cluster. | Column Name | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `UPLOAD_QUEUE_WAIT_SECS` | Number of seconds the uploads to remote storage are delayed in the entirecluster | | `UPLOAD_MB_PER_SEC` | The rate at which files are being uploaded to remote storage in the entirecluster | | `LAST_UPLOAD_RESULT` | If all partitions in theclusterreport SUCCESS for upload, then this reports SUCCESS | | `BLOB_CACHE_DOWNLOAD_MB_PER_SEC` | Rate at which the blob cache is downloading files from remote storage in the entirecluster | | `BLOB_CACHE_EVICTION_MB_PER_SEC` | Rate at which the blob cache is evicting files in the entirecluster | | `OLDEST_DATA_AGE_DAYS` | Age of the oldest piece of data in the entirecluster | | `API_REQUESTS_PER_SEC` | Rate of API requests made to remote storage in the entirecluster | | `IS_BEING_RATE_LIMITED` | Indicates whether any node in the entireclusteris being rate limited on API requests to remote storage. On S3, rate limiting starts when we receive`503 SLOWDOWN`errors. | * Fixed an issue with possible connection crashes on disaster recovery clusters when DDL queries are forwarded to the master aggregator and a database has not been selected. * Fixed sync timeout errors occurring when `REBALANCE PARTITIONS` of large databases is being processed. The `REBALANCE PARTITIONS` operation now waits for the replication of large snapshot files. * Added `password_expiration` column to `information_schema.USERS` table. If the `password_expiration_seconds` engine variable is not enabled, the `password_expiration` column will be NULL. If the `password_expiration_seconds` engine variable is enabled, the `password_expiration` column will display the number of seconds remaining for the password to expire. * Updated time zone data for some countries that recently changed their time zone rules. * Fixed a crash caused by a check on the maximum number of partitions allowed on a leaf node. An error message is displayed when this threshold is exceeded. * Added support for allowing spilling to disk of hash join output row. * Added support for running `PROFILE` on queries that execute inside of transactions the include multiple statements. * Fixed an issue where a crash could occur while attempting to compile queries for `UPDATE` statements with multiple columns due to insufficient memory resources. The amount of memory necessary to compile these query shapes was reduced. * Fixed a race condition involving concurrent `UPDATE/UPSERT` queries that update sort keys against columnstore tables. ## 2021-10-12 Version 7.5.12 * Fixed an issue in the join optimizer that incorrectly eliminated candidate plans from the search. * Fixed a write amplification issue associated with row versioning. The behavior is disabled if the database is not unlimited storage. * Added support for `MySQLDump` 8.0. * Fixed a server startup failure when `LD_PRELOAD` is set. The behavior now reverts to a previous version which would overwrite any existing `LD_PRELOAD` setting. * Fixed an issue where the `memsql_exporter` would sometimes produce invalid JSON errors. The `memsql_exporter` is a satellite process that collects data about a running cluster. * Reduced the memory used to download blob files from remote storage. * Fixed a regression in recovery performance for databases using remote storage. * Fixed a crash that could occur when using `LINK` syntax with Avro pipelines. * Added Linux dependencies for SASL to SingleStore installation packages so they do not need to be installed separately. ## 2021-10-01 Version 7.5.11 * Masked blob store `CREDENTIALS` text in `CREATE DATABASE` command for unlimited storage in `SHOW PROCESSLIST`, `information_schema.PROCESSLIST`, and all error messages. ## 2021-09-21 Version 7.5.10 * Removed the 16384 size limit on the Avro array and map sizes in pipelines and `LOAD DATA` when pipelines aren't utilized. * Masked blob store `CREDENTIALS` text in `BACKUP`, `RESTORE`, and `LINK` commands in `SHOW PROCESSLIST`, `information_schema.PROCESSLIST`, and all error messages. * Increased incremental blob file streaming to disk to reduce memory overhead for unlimited storage. * Added support for incremental backups to utilize `LINK` syntax to specify `CONFIG` and `CREDENTIALS` clauses. * Added support for `READ ONLY` or `READ WRITE` syntax to the `SET TRANSACTION` statement for MySQL compatibility. * Added support for optimizing cases of `IN` joins in subselects in the project list. * Added session variable named `subquery_merge_with_outer_joins` that fine-tunes the behavior of the rewrite `SubQueryMerge`. If the variable is set to ON: always merge the sub-select with outer join(s). If the variable is set to HEURISTIC: check to see if merging the sub-select with outer join(s) will cause negative performance for the overall query. If the variable is set to OFF: don’t merge sub-selects with outer join(s). If the variable is set to AUTO (this is the default): For SingleStore 7.5 and previous versions, AUTO will behave like ON. For SingleStore versions after 7.5, AUTO will behave like HEURISTIC. * Fixed auto histograms to determine when to build new histograms based on both master and child aggregators. * Fixed an aggregator crash that was caused when a malformed `INIT_DB` command is sent by a client. * Added support for complex joins (joins with non-linear/nested join tree) involving correlated nested `SELECT` queries. * Added an new engine variable named `rewrite_verbose` to output tracings of the rewrite phase on the current query in the tracelogs. * Fixed warning indicating "partition metadata is out of sync on table", which could appear in the tracelog after a node restart. * Added support for `SKIP CONSTRAINT ERRORS` options for Avro and JSON pipelines. * Increased the timeout used when connecting to AWS to determine credentials if none are specified in a `BACKUP/RESTORE` command.  Also added a new engine variable `subprocess_ec2_metadata_timeout_ms` which allows users to set that timeout. ## 2021-08-30 Version 7.5.9 * **New Feature**: Added support for alias references in `WHERE` clauses and as column expressions. See [SELECT](https://docs.singlestore.com/db/v9.1/reference/sql-reference/data-manipulation-language-dml/select.md) for more information. * Fixed an issue where Disaster Recovery of a non-unlimited storage database was not blocked if a cluster was configured to use remote storage. * The `AGGREGATOR SYNC AUTO_INCREMENT` command now can be performed by anyone with the ability to update the target database(s) or table in any way. * Fixed an issue that could cause defunct processes for pipelines or code generation in a rare race condition. * Increased batch deletions from unlimited storage to improve the performance of GCing files from remote storage. * Improved JSON built-in functions to better handle out of memory errors. * Improved the cardinality estimates for columns with an equality predicate. ## 2021-08-16 Version 7.5.8 * Fixed per-predicate `PROFILE` statistics on columnstore filter operations. * Fixed a crash that could occur when running `CREATE PIPELINE` or `REPLACE PIPELINE` on a pipeline that already exists. * Removed truncation of the query text in `query.log` when `general_log=on`. * Fixed a crash that could occur when `ALTER TABLE COMPRESSION=SPARSE` is run in the same `ALTER` statement with an operation that doesn’t require data movement (ie., `AUTOSTATS_ENABLED = ON)`. ## 2021-08-09 Version 7.5.7 * **New Feature**: Added [anonymous code blocks](https://docs.singlestore.com/db/v9.1/reference/sql-reference/procedural-sql-reference/anonymous-code-blocks.md) which are unnamed blocks of SingleStore Procedural SQL (PSQL) code that are executed immediately. They allow you to create a code block without defining and calling a stored procedure and are useful in situations such as testing or debugging. * Added a new system variable called `parametrizer_query_max_params` which allows users to set the maximum number of scalar constants a single query can contain. Any query with more than `parametrizer_query_max_params` will be failed with an error. The default value is `1048576`. * Fixed an issue where the exporter process was not being cleaned up when `memsqld` is shut down. * Fixed an issue where modifying the case of a column via `ALTER TABLE` could cause a pipeline using that table to crash. * Improved usability of various JSON-builtins to support utf8mb4 charset arguments. * Increased frequency of merging during load for unique keys on columnstores. * Added `information_schema.mv_bt` which shows backtrace information for all nodes in the cluster. * Added `information_schema.mv_plancache` which shows plancache information for all nodes in the cluster (cluster-wide version of `information_schema.plancache`). * Added support for the `transaction_read_only` and `tx_read_only` session variables for MySQL compatible client drivers that query them. * Fixed a performance regression establishing SSL connections introduced in 7.5. * Fixed a deadlock which can occur if a failover happens during an `ALTER TABLE`. * Changed the `SET PASSWORD` statement to query forward from the child aggregator to the master aggregator. ## 2021-07-20 Version 7.5.6 * Initial GA release of version 7.5.5 ## 2021-07-07 Version 7.5.5 * Stability improvements to version 7.5.4 ## 2021-06-28 Version 7.5.4 * Fixed an "invalid LSN" error when restoring a local storage database into a remote storage database. ## 2021-06-24 Version 7.5.3 * Fixed an issue that could cause `DETACH DATABASE` to hang while a pipeline that is attempting to load batches is failing. * Fixed an issue that could generate incorrect out of memory errors when using a remote storage database. ## 2021-06-22 Version 7.5.2 * Beta release of SingleStore 7.5 *** Modified at: March 8, 2024 Source: [/db/v9.1/release-notes/singlestore-memsql/7-5-release-notes/maintenance-release-changelog/](https://docs.singlestore.com/db/v9.1/release-notes/singlestore-memsql/7-5-release-notes/maintenance-release-changelog/) (An index of the documentation is available at /llms.txt)