Maintenance Release Changelog

Fixed an issue where errors could occur during backup when large files are copied.

The sql_mode session variable now shows up in SHOW VARIABLES and the mv_global_variables view.

Fixed an issue that could cause an incorrect error message to be generated on an out of memory error.

Fixed an issue where CLEAR ORPHAN DATABASE could cause data loss if run when the master aggregator has a database in an unrecoverable state.

Updated timezone metadata to include Mexico's latest timezone change.

Improved the parsing performance of queries that contain several tables.

Fixed an issue causing incorrect trace messages in master logs where clocks were incorrectly advancing from "0".

Fixed the case where the REGEXP_REPLACE and REGEXP_SUBSTR expressions can produce non-utf8 strings by introducing the new engine variableregexp_output_validation_mode. Regular expression built-ins can produce non-utf8 strings because they don't have full support for multi-byte characters. The engine variable controls this behavior if regular expression built-ins return strings that are invalid under its collation settings.

Improved the performance on columnstore scans that perform multiple JSON extraction operations on the same JSON column.

Fixed an issue with databases becoming unrecoverable when upgrading to version 8.0 if any tables contained shard keys using computed columns that call JSON_EXTRACT_STRING.

Fixed a MySQL protocol incompatibility where SingleStore was not properly sending back the SERVER_STATUS_IN_TRANS flag. This was impacting some MySQL php drivers.

Fixed a crash when parsing certain Parquet data into a pipeline.

The SHOW STATUS EXTENDED command contains a new "Gv_clock" key whose value is the current logical clock of the server.

The SHOW DATABASE STATUS command contains a new "gv_clock" key whose value is the current logical clock of the server.

Fixed an edge case issue which could cause the engine to hang during shutdown.

Fixed some error handling issues with unlimited storage download and upload processes.

Fixed a bug that may cause a query to hang when comparing an utf8 column with an utf8mb4 constant. This issue occurs when collation_server is set to one of the utf8mb4 collations.

Fixed an edge case issue where the engine could crash when performing multi-inserts.

Fixed an issue with a transaction leak on the master aggregator when running CREATE TABLE AS SELECT on a child aggregator using autocommit=0 on the connection.

Fixed an issue with aggregate functions using incorrect enum/set types that may result in inaccurate output in the operator tree.

Fixed an issue where environments with several nodes having several partitions may not auto-attach when restarted simultaneously.

Fixed a performance issue when creating an aggregator result table on the master aggregator.

Fixed an issue when if auto-creation is turned off (auto.create.topics.enable=false), SELECT INTO KAFKA would return a success message even if the insert fails.

Improved performance of comparing utf8mb4 strings.

Added the ability to allow code-generation of queries against the information schema when the cluster is very low on disk/memory.

Fixed a potential issue using multi-statement batches inside of prepared statements without parameters that may result in a crash.

Fixed a potential issue that could cause a lost second prepare statement when using a binary protocol over SSL.

Fixed a potential issue that could cause a NFS backup error to trigger a crash.

Reduced the memory usage of querying information_schema.pipeline_errors when there are many past pipeline errors.

Fixed an issue where environments with several nodes having several partitions may not auto-attach when restarted simultaneously.

Fixed an issue which caused unlimited database S3 uploads to fail due to an ExpiredToken error. This issue impacted the refresh of EC2 instance meta credentials caused by the public open-source AWS C++ SDK after encountering a transient network issue.

Fixed a specific query shape which failed CodeGen for columnstore unique keys.

Allow REVOKE from ROLE permissions for tables that have been dropped.

Fixed an issue that may cause a restarted master aggregator to crash when the STOP PIPELINE command runs on a detached pipeline.

Fixed an issue where having too many rows on the same columnstore hash key could return "Read failed with error 14 (Bad address)".

Fixed a potential issue when a query contains a full outer join and a window function partitioning by the full outer join columns yields the wrong result.

Fixed an issue where an incorrect result could be returned when a query contains multiple joins that all match a multi-column hash index.

Fixed a performance issue with count distinct aggregation and streaming subqueries.

Fixed an issue with the accuracy of memory and disk space availability confirmed in the REPLICATE DATABASE command. This fix will ensure any secondary cluster has enough memory and disk space to replicate the database.

Fixed an issue where commands forwarded to the master aggregator during an online upgrade could cause a crash.

Fixed an issue where canceling a RESTORE would not completely drop the partially restored database.

Fixed a memory leak issue due to alloc_warnings when using reference table pipelines.

Removed ability to use RESTORE DATABASE on unlimited databases with sync durability enabled.

Fixed an issue where S3 can silently fail to upload a file for an unlimited database.

Removed ability to use utf8_general_mysql500_ci collation. This collation is not supported by SingleStore.

Fixed an issue with window functions which refer to the window definition at the end of a SELECT statement.

Improved the Subselect To Join rewrite to recognize more cases with subselects in filter conditions for left join.

Added support for s3_force_path_style to S3 backup. s3_force_path_style (optional boolean, defaults to true) specifies whether to use path style (region.amazonaws.com/bucket) or virtual address style (bucket.region.amazonaws.com) syntax when specifying the location of the bucket. For S3 compatible services that only support virtual address paths, specify "s3_force_path_style":"false" in the configuration options. Refer to the BACKUP DATABASE section for more information.

Fixed an issue that may cause insert into mv_queries to become invalid when the cluster is in ONLY_FULL_GROUP_BY sql_mode.

Fixed an edge case issue with the blob cache eviction algorithm may cause an infinite loop.

Fixed an issue when specifying all columns in the SELECT command resulted in columns being out of order rather than matching the table definition.

Improved the performance of downloading blob files for unlimited databases when SSL is enabled between SingleStore and the blobstore.

Improved the performance of hash joins over expressions.

Fixed an incorrect error behavior when dropping and adding a multicolumn columnstore hash index on the same set of columns in a single alter.

Reduced the memory used by the blob cache by 100-200 bytes per cached blob.

Fixed HDFS pipelines failures when loading from HA/BigIP clusters that advertise a different host name than the one used to connect.

Fixed an issue where the incorrect backup size showed in mv_backup_history for incremental backups.

Reduce the memory used to track cached blob files and run scans over columnstore tables.

Fixed a server crash caused by serializing big table or view definitions in snapshots.

Added support for NOT NULL when using persisted computed columns.

Fixed a potential issue with transaction rolled back-mid query errors that may occur when a database name has a number suffix.

Added support for Reloading SSL Certs without having to restart the node.

Added a new engine variable named columnstore_small_blob_combination_threshold that can be used to control the size of column files created by combining compressed columnstore segments. Its default value is 32 Mb (the previously hard-coded value in the engine).

Added for compatibility with other databases:

• Global engine variable: default_storage_engine.

• Information schema field: information_schema.COLUMNS DATETIME_PRECISION.

Fixed a server crash caused by a rewrite trying to inline nested expressions.

Fixed an issue where DROP FUNCTION would result in an unknown table error.

Added an engine variable columnstore_validate_blob_before_merge. If enabled, the background merger would validate the blob checksums before merging blobs.

Fixed issues with the information_schema.mv_activities and information_schema.mv_activities_extended tables reporting incorrect activities on database names ending with an integer. Refer to the CREATE DATABASE section for more information.

Fixed a server crash caused by table-valued functions parameters that assume the parameter name is the same as the alias of the table-valued function operator.

Fixed an issue with out of memory errors being generated while compiling queries that use multiple UNION clauses.

Adjusted the maximum memory allocation function to proactively run even when the total_server_memory usage is well below the maximum_memory allocation.

Fixed an issue where an incorrect error was generated for an aggregator result table when it was used by a parallel read. DROP RESULT TABLE now returns the correct error.

Improved heuristic estimates for queries with complex join predicates.

Fixed an issue with incremental autostats where duplicate rows were included in the autostats resulting in invalid statistics in a disaster recovery cluster when executing an ALTER statement in the primary cluster.

Added a new engine variable named columnstore_small_blob_combination_threshold that can be used to control the size of column files created by combining compressed columnstore segments. Its default value is 32 Mb (the previously hard-coded value in the engine).

Added for compatibility with other databases:

• Global engine variable: default_storage_engine.

• Information schema field: information_schema.COLUMNS DATETIME_PRECISION.

Fixed a server crash caused by a rewrite trying to inline nested expressions.

Fixed an issue where DROP FUNCTION would result in an unknown table error.

Fixed an issue with calling nested stored procedures that have different definers.

Allowed the execution of UPDATE/UPSERT command on sort key columns via pipelines. Prior to this update, an error was generated.

Fixed an issue where table-valued functions crashed when a parameter is used more than once while compiling the query.

Improved the performance of sampling queries executed by the query optimizer.

Fixed the CLEAR ORPHAN DATABASES command to ensure all databases are dropped when the leaf_failover_fanout global variable is set to load_balanced and there are partition databases with more than replica partition.

Fixed an issue with query forwarding incorrectly triggering aggregator sync causing the aggregators to run out of disk space.

Fixed an issue with incorrect nullCount and rowCount statistics for larger tables with autostats enabled.

Fixed an issue with memsql_exporter ceasing to produce monitoring results.

Fixed an issue with C# MySql client drivers getting "no database errors" on connection.

Improved rowstore garbage collection by eliminating the delay for unlimited storage uploads to occur after a snapshot.

Fixed an issue with partitions in unlimited storage databases going into an unrecoverable state while snapshots download during server restarts.

Fixed the CREATED column in the information_schema.ROUTINES table by now recording the creation date for table-valued functions.

Fixed a bug that incorrectly pushes a join before a window function when attempting to perform a distributed join in an aggregator.

Fixed a bug that a variety of conversion operators failed to check errors during the construction of the operator tree.

Fixed timeout errors when performing large BACKUP/RESTORE operations against an Azure blob store.

Fixed an issue where utf8mb4 characters were mistakenly extracted to the resulting utf8 string for the JSON_EXTRACT_STRING command.

Fixed a cluster database log corruption that could occur if the root user was dropped when sync_permissions was enabled.

Improved the predicate transitivity rewrite during query execution to cover more cases in an EXISTS/NOT EXISTS clause.

Stopped row-level locking when an UPDATE or DELETE join condition uses a full-text index.

Fixed an issue with garbage collection in unlimited storage databases that could cause a point in time restore to fail.

Fixed an issue with inconsistent snapshots caused by running SNAPSHOT DATABASE concurrently with an ALTER TABLE command against a columnstore table.

Clarified error messaging when an incorrect configuration is given while creating an unlimited storage database.

Reduced the probability of failed connections to clusters due to WebSockets proxy issues.

Added support for multipart_chunk_size_mb configuration if compatibility_mode=true. See SELECT … INTO S3 for more information on the multipart_chunk_size_mb configuration.

Fixed the calculation for total_runtime_ms for queries profiled from within stored procedures. Prior to this release, total_runtime_ms was not calculated accurately.

Improved how the engine starts the prometheus exporter.

Added an engine variable columnstore_validate_blob_before_merge. If enabled, the background merger would validate the blob checksums before merging blobs.

Fixed the default value for max_async_compilation_concurrency so that it is now set appropriately when control groups are in use limiting CPU usage.

Added control group awareness for information_schema.mv_nodes which now displays the correct number of CPUs a process has access to.

Fixed an issue with queries not assigned to resource group pools with hard CPU limits from using the CPU assigned to the pool.

Fixed an issue with a crash occurring when an incremental backup was performed on an empty database.

Clarified error messaging when invalid data is inserted into a utf8 charset data type column.

Fixed an issue with an aggregator results table checking INSERT privileges.

Fixed an issue where results data is incorrectly blank in particular columns when querying information_schema.mv_plancache.

Fixed a race condition involving concurrent UPDATE/UPSERT queries that update sort keys against columnstore tables.

Fixed an issue with a crash occurring when DDL is run inside a stored procedure that is executed within a resource pool on a child aggregator.

Block the DROP TEMPORARY TABLE command from running in a transaction that writes to the temporary table being dropped. The DROP TEMPORARY TABLE command result will display an error.

Removed the disk space check that occurred when ALTER TABLE ADD INDEX is run. The check occurred prior to altering a columnstore table in an unlimited storage database.

Fixed an intermittent "Failed to GetObject" error when restoring a non-unlimited storage database into a cluster where unlimited storage is configured.

Locked down SET clauses that attempt to update sparse columns in correlated subselects.

Added support for MAX_ERRORS clause use when using LOAD DATA into columnstore tables with unique keys.

Removed restrictions on nested SELECT queries that utilize the PARTITION_ID() function in their intermediate results.

Improved the performance of columnstore UPDATE queries that do not modify unique key columns.

Improved the performance of UPDATE queries that do not modify unique key columns against columnstore tables.

Added support for BACKUP DATABASE into S3 buckets using the sts_regional_endpoints configuration option.

Introduced mv_bottomless_summaryinformation_schema table that shows various aggregated statistics about unlimited storage databases. The mv_bottomless_summary table is an at-a-glance rollup table that shows what is happening right now with the unlimited storage cluster.

Fixed an issue with possible connection crashes on disaster recovery clusters when DDL queries are forwarded to the master aggregator and a database has not been selected.

Fixed sync timeout errors occurring when REBALANCE PARTITIONS of large databases is being processed. The REBALANCE PARTITIONS operation now waits for the replication of large snapshot files.

Added password_expiration column to information_schema.USERS table. If the password_expiration_seconds engine variable is not enabled, the password_expiration column will be NULL. If the password_expiration_seconds engine variable is enabled, the password_expiration column will display the number of seconds remaining for the password to expire.

Updated time zone data for some countries that recently changed their time zone rules.

Fixed a crash caused by a check on the maximum number of partitions allowed on a leaf node. An error message is displayed when this threshold is exceeded.

Added support for allowing spilling to disk of hash join output row.

Added support for running PROFILE on queries that execute inside of transactions the include multiple statements.

Fixed an issue where a crash could occur while attempting to compile queries for UPDATE statements with multiple columns due to insufficient memory resources. The amount of memory necessary to compile these query shapes was reduced.

Fixed a race condition involving concurrent UPDATE/UPSERT queries that update sort keys against columnstore tables.

Fixed an issue in the join optimizer that incorrectly eliminated candidate plans from the search.

Fixed a write amplification issue associated with row versioning. The behavior is disabled if the database is not unlimited storage.

Added support for MySQLDump 8.0.

Fixed a server startup failure when LD_PRELOAD is set. The behavior now reverts to a previous version which would overwrite any existing LD_PRELOAD setting.

Fixed an issue where the memsql_exporter would sometimes produce invalid JSON errors. The memsql_exporter is a satellite process that collects data about a running cluster.

Reduced the memory used to download blob files from remote storage.

Fixed a regression in recovery performance for databases using remote storage.

Fixed a crash that could occur when using LINK syntax with Avro pipelines.

Added Linux dependencies for SASL to SingleStore installation packages so they do not need to be installed separately.

Masked blob store CREDENTIALS text in CREATE DATABASE command for unlimited storage in SHOW PROCESSLIST, information_schema.PROCESSLIST, and all error messages.

Removed the 16384 size limit on the Avro array and map sizes in pipelines and LOAD DATA when pipelines aren't utilized.

Masked blob store CREDENTIALS text in BACKUP, RESTORE, and LINK commands in SHOW PROCESSLIST, information_schema.PROCESSLIST, and all error messages.

Increased incremental blob file streaming to disk to reduce memory overhead for unlimited storage.

Added support for incremental backups to utilize LINK syntax to specify CONFIG and CREDENTIALS clauses.

Added support for READ ONLY or READ WRITE syntax to the SET TRANSACTION statement for MySQL compatibility.

Added support for optimizing cases of IN joins in subselects in the project list.

Added session variable named subquery_merge_with_outer_joins that fine-tunes the behavior of the rewrite SubQueryMerge. If the variable is set to ON: always merge the sub-select with outer join(s). If the variable is set to HEURISTIC: check to see if merging the sub-select with outer join(s) will cause negative performance for the overall query. If the variable is set to OFF: don’t merge sub-selects with outer join(s). If the variable is set to AUTO (this is the default): For SingleStore 7.5 and previous versions, AUTO will behave like ON. For SingleStore versions after 7.5, AUTO will behave like HEURISTIC.

Fixed auto histograms to determine when to build new histograms based on both master and child aggregators.

Fixed an aggregator crash that was caused when a malformed INIT_DB command is sent by a client.

Added support for complex joins (joins with non-linear/nested join tree) involving correlated nested SELECT queries.

Added an new engine variable named rewrite_verbose to output tracings of the rewrite phase on the current query in the tracelogs.

Fixed warning indicating "partition metadata is out of sync on table", which could appear in the tracelog after a node restart.

Added support for SKIP CONSTRAINT ERRORS options for Avro and JSON pipelines.

Increased the timeout used when connecting to AWS to determine credentials if none are specified in a BACKUP/RESTORE command.  Also added a new engine variable subprocess_ec2_metadata_timeout_ms which allows users to set that timeout.

New Feature: Added support for alias references in WHERE clauses and as column expressions. See SELECT for more information.

Fixed an issue where Disaster Recovery of a non-unlimited storage database was not blocked if a cluster was configured to use remote storage.

The AGGREGATOR SYNC AUTO_INCREMENT command now can be performed by anyone with the ability to update the target database(s) or table in any way.

Fixed an issue that could cause defunct processes for pipelines or code generation in a rare race condition.

Increased batch deletions from unlimited storage to improve the performance of GCing files from remote storage.

Improved JSON built-in functions to better handle out of memory errors.

Improved the cardinality estimates for columns with an equality predicate.

Fixed per-predicate PROFILE statistics on columnstore filter operations.

Fixed a crash that could occur when running CREATE PIPELINE or REPLACE PIPELINE on a pipeline that already exists.

Removed truncation of the query text in query.log when general_log=on.

Fixed a crash that could occur when ALTER TABLE COMPRESSION=SPARSE is run in the same ALTER statement with an operation that doesn’t require data movement (ie., AUTOSTATS_ENABLED = ON).

New Feature: Added anonymous code blocks which are unnamed blocks of SingleStore Procedural SQL (PSQL) code that are executed immediately. They allow you to create a code block without defining and calling a stored procedure and are useful in situations such as testing or debugging.

Added a new system variable called parametrizer_query_max_params which allows users to set the maximum number of scalar constants a single query can contain. Any query with more than parametrizer_query_max_params will be failed with an error. The default value is 1048576.

Fixed an issue where the exporter process was not being cleaned up when memsqld is shut down.

Fixed an issue where modifying the case of a column via ALTER TABLE could cause a pipeline using that table to crash.

Improved usability of various JSON-builtins to support utf8mb4 charset arguments.

Increased frequency of merging during load for unique keys on columnstores.

Added information_schema.mv_bt which shows backtrace information for all nodes in the cluster.

Added information_schema.mv_plancache which shows plancache information for all nodes in the cluster (cluster-wide version of information_schema.plancache).

Added support for the transaction_read_only and tx_read_only session variables for MySQL compatible client drivers that query them.

Fixed a performance regression establishing SSL connections introduced in 7.5.

Fixed a deadlock which can occur if a failover happens during an ALTER TABLE.

Changed the SET PASSWORD statement to query forward from the child aggregator to the master aggregator.

Initial GA release of version 7.5.5

Stability improvements to version 7.5.4

Fixed an "invalid LSN" error when restoring a local storage database into a remote storage database.

Fixed an issue that could cause DETACH DATABASE to hang while a pipeline that is attempting to load batches is failing.

Fixed an issue that could generate incorrect out of memory errors when using a remote storage database.

Beta release of SingleStore 7.5