Maintenance Release Changelog

Enhancement: Added support for loading data from compressed Avro datafiles.

Bugfix: Fixed an issue that occurred when primary key is NULLABLE when copying a table using "Create table as select" statement.

Enhancement: Added a new engine variable, enable_extended_types_metadata. Used to enable the handling of extended data types that are not defined in MySQL enum_field_types. Default is disabled. When enabled, SingleStore may send extended metadata to the client. It provides the ability to distinguish between different data types, and handle them appropriately in client applications.

Bugfix: Addressed a crash issue occurring while executing queries with vector builtins on clusters running via Docker containers on Mac machines equipped with Apple silicon processors (M1).

Bugfix: No longer show information_schema pipeline metadata of databases that are not attached to the workspace.

Enhancement: Use SeekableString encoding by default for storing VECTOR datatype columns in columnstore tables.

Bugfix: Fixed an issue that caused rare "Arrangement already exists in namespace" errors in recursive CTE queries.

Bugfix: Improved the hashGroupBy spilling condition to trigger (spill) when appropriate.

Enhancement: Added a new engine variable costing_max_djo_tables which sets the maximum amount of tables we allow in full DQO estimate search when filling table row estimates for distributed join hints.

Enhancement: Made many more engine variables configurable on SingleStore Helios:

multi_insert_tuple_count, columnstore_row_value_table_lock_threshold, internal_columnstore_max_uncompressed_blob_size, internal_columnstore_max_uncompressed_rowindex_blob_size, columnstore_validate_blob_before_merge, default_max_histogram_buckets, enable_varbuffer_dictionary_compression, enable_alias_space_trim, skip_segelim_with_inlist_threshold, default_autostats_histogram_mode, default_autostats_columnstore_cardinality_mode, default_autostats_rowstore_cardinality_mode, default_autostats_columnstore_sampling, experimental_disable_multipartition_parallel_read, internal_enable_parallel_query_throttling, enable_spilling, spilling_node_memory_threshold_ratio, spilling_query_operator_memory_threshold, regexp_compile_mem_mb, selectivity_estimation_min_sample_size, repartition_minimum_buffer_size, use_seekable_json, json_document_max_children, json_document_max_leaves, json_document_sparse_children_check_threshold, json_document_sparse_children_check_ratio, json_document_absolute_sparse_key_check_ratio, json_document_page_data_soft_limit, ignore_insert_into_computed_column, max_subselect_aggregator_rowcount, leaf_pushdown_enable_rowcount, reshuffle_group_by_base_cost, max_broadcast_tree_rowcount, enable_broadcast_left_join, display_full_estimation_stats, force_bushy_joins, force_heuristic_rewrites, force_table_pushdown, force_bloom_filters, explain_joinplan_costs, statistics_warnings, optimizer_warnings, optimize_constants, optimize_huge_expressions, optimize_expressions_larger_than, optimize_stmt_threshold, quadratic_rewrite_size_limit, batch_external_functions, batch_external_functions_size, optimizer_enable_json_text_matching, optimizer_enable_orderby_limit_self_join, distributed_optimizer_broadcast_mult, distributed_optimizer_old_selectivity_table_threshold, distributed_optimizer_selectivity_fallback_threshold, distributed_optimizer_verbose, old_local_join_optimizer, optimizer_hash_join_cost, optimizer_merge_join_cost, optimizer_nested_join_cost, optimizer_disable_right_join, interpreter_mode_sampling_threshold, hash_groupby_segment_distinct_values_threshold, sampling_estimates_for_complex_filters, enable_histogram_to_unwrap_literals, estimate_zero_rows_when_sampling_data_is_missing, exclude_scalar_subselects_from_filters, leaf_pushdown_default, distributed_optimizer_nodes, optimizer_num_partitions, enable_binary_protocol, enable_multipartition_queries, enable_local_shuffle_group_by, enable_skiplist_sampling_for_selectivity, columnstore_sample_per_partition_limit, disable_sampling_estimation, disable_histogram_estimation, inlist_precision_limit, allow_materialize_cte_with_union, optimizer_cross_join_cost, distributed_optimizer_run_legacy_heuristic, distributed_optimizer_min_join_size_run_initial_heuristics, distributed_optimizer_unrestricted_search_threshold, singlebox_optimizer_cost_based_threshold, distributed_optimizer_estimated_restricted_search_cost_bound, disable_reference_table_leftjoin_where_pushdown, disable_sampling_estimation_with_histograms, disable_subquery_merge_with_straight_joins, default_columnstore_table_lock_threshold, default_spill_dependent_outputters, query_rewrite_loop_iterations, verify_fields_in_transitivity, optimizer_min_reference_tables_for_gathered_join, optimizer_min_reference_rows_for_gathered_join, max_expression_query_limit, max_expression_item_limit, optimizer_empty_tables_limit, optimizer_beam_width, optimizer_disable_subselect_to_join, disable_remove_redundant_gby_rewrite, subquery_merge_with_outer_joins, optimizer_disable_operator_tree_nullability_check, clamp_histogram_date_estimates, varchar_column_string_optimization_length, histogram_column_correlation, use_column_correlation, consider_secondary_projection, optimizer_disable_semi_join_reduction, optimizer_gb_placement_table_limit, optimizer_disable_transitive_predicates

Enhancement: Added more config information about pipelines to memsql_exporter.

Enhancement: Improved error handling for VECTOR type.

Enhancement: Added support for aggregate functions (FIRST, LAST) with VECTOR datatype.

Enhancement: Added support for Kafka key (producer) in the SELECT ... INTO KAFKA command.

Enhancement: Improved Kafka error messages to make them more specific.

Bugfix: Fixed a case where a killed query could incorrectly return a JSON error rather than query interrupted message.

Bugfix: MV_DISK_USAGE queries can no longer block garbage collection.

Bugfix: Fixed a crash-on-startup issue on Centos9.

Bugfix: Fixed potential deadlock between table module eviction and GC manager.

Bugfix: Fixed a crash that could occur when a scalar subselect is passed as an argument to an external TVF.

Bugfix: Fixed an issue with restoring from public backups.

Bugfix: Backups now throttle after snapshot.

New Feature: Added ability to create a Projection which is a copy of some or all columns in a table, and may be sharded and shorted differently than the primary table. Projections can be used to speed up several types of queries including range filters, ORDER BY/LIMIT, GROUP BY, count(DISTINCT...), DISTINCT, joins, and row lookups on secondary keys. Projections depend on the table and are updated in real time when you update the table. Related information schema table: PROJECTIONS.

Bugfix: Removed an unnecessary metadata copy to improve JSON performance.

Enhancement: Multi-insert queries now respect lower columnstore_segment_rows settings.

Enhancement: Subprocess and S3 API tracing are now also enabled via the enable_subprocess_tracing engine variable.

Enhancement: Added the optional IF NOT EXISTS clause to the CREATE TABLE ... AS INFER PIPELINE statement. Refer to Replicate MongoDB® Collections to SingleStore for more information.

Bugfix: LAST_INSERT_ID will now correctly retrieve the last inserted ID for a forwarded INSERT query.

Bugfix: The rule based optimization warning message will only be shown when cost based optimization is meaningful.

Bugfix: Fixed a minor performance issue where a table module could be evicted prematurely during database recovery.

Bugfix: Fixed a very rare deadlock between ALTER of a table and a DROP of a database.

Bugfix: Fixed a JSON insertion failure to old utf8 sample tables that contain utf8mb4 characters by encoding it with base64 format beforehand for backward compatibility.

Enhancement: Disk Spilling now takes QUERY_MEMORY_PERCENTAGE resource pool setting into consideration.

Bugfix: Fixed an issue with a blob file leak in a rare crash scenario.

Bugfix: Fixed a potential deadlock in a scenario involving DDL, a 2PC transaction, and a concurrent failover.

Bugfix: Fixed a deadlock involving BACKUP and ALTER in a rare case.

Enhancement: Added support for utf8mb4 symbols in column comments, table comments, and user comments.

Enhancement: Enabled additional query hint for non-distributed queries so that first run of those queries can be queued.

Enhancement: Updated timezone data, now using IANA's tzdata version 2024a; new timezones Kyiv, Ciudad_Juarez, and Kanton supported.

Bugfix: Fixed an issue in background merger that allowed concurrent deletes of empty segments.

Bugfix: Fixed an issue to ensure preserving a field of a table correctly.

Enhancement: Idle Table Eviction improvement. In the 8.5 GA version, an idle table's objects cannot be evicted from memory if the table has one or more UNIQUE key columns (both PRIMARY KEYs and any other keys with a UNIQUE constraint). In the 8.5.11 patch, idle tables' code objects can be evicted from memory if the table has one or more UNIQUE key columns.

Enhancement: Added numInferredPaths and schemaSize as new columns to JSON_COLUMN_SCHEMA table. numInferredPaths is the number of key paths inferred for the segment. schemaSize is the size of schema_json in bytes.

Bugfix: Added safety checks around dropping databases without exclusive access to storage.

Bugfix: Fixed an issue where explicitly defining a JSON type in the RETURNS of a Wasm TVF can cause an error when it is run.

Bugfix: Fixed a rare crash scenario.

Bugfix: Fixed an issue that could lead to the risk of undefined behavior when running DROP EXTENSION IF EXISTS.

Enhancement: IN-lists will now use hashmap optimization for matching parameters in *_MATCH_ANY statements.

Enhancement: Made default_distributed_ddl_timeout a sync variable on all nodes.

Enhancement: Modified the conversion logic for converting from VECTOR to BSON types. Now, when casting VECTOR(F32), it will generate a BSON array of doubles, rather than a combination of numeric types.

Bugfix: Addressed a family of issues relating to optimal execution of JSON extracts on schema'd data.

Bugfix: Fixed an issue with regression in replay performance for databases with many tables.

Enhancement: Added spilling metrics for TopSort (ORDER BY with LIMIT).

Bugfix: Fixed an issue where the existing websocket connection would close when variables are updated in global scope.

Bugfix: Removed the hard-coded 'collation_server' from constant.StrictModeParams.

Enhancement: Added new metrics to memsql_exporter based on the information_schema.mv_sysinfo_disk columns.

Enhancement: Added support for creating numeric histograms based on JSON values. (Feature flag gated, contact your SingleStore representative)

Enhancement: Added ability for information_schema.optimizer_statistics to display JSON keys.

Enhancement: Improved JSON histograms to support numeric histograms, analyze command, info_schema.optimizer_statistics, and displays JSON histograms.

Enhancement: Added support for warming blob cache with table's column data. Syntax is: OPTIMIZE TABLE <table_name> WARM BLOB CACHE FOR COLUMN <column_names>:

OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN c1, c2;

OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN *;

Enhancement: Added ability to skip eviction when log replaying hits the blob cache space limit, providing a greater chance to succeed.

Enhancement: Disabled the default semi join reduction rewrite.

Bugfix: Fixed an issue with a sensitive information leak inside of out-of-memory reports.

Bugfix: Fixed the result collation from VECTOR built-ins to be utf8mb4_bin instead of binary.

Enhancement: Creation of computed columns with VECTOR data type is now allowed.

Enhancement: Added support to the VECTOR built-ins for all VECTOR elements types (e.g., F32, F64, I8, I16, I32, and I64).

Enhancement: Added support for usage of vector built-ins with string/JSON arguments without requiring an explicit typecast to VECTOR (e.g., 'SELECT @vec<*> '[1,2,3]').

Enhancement: Now recognize more EXTRACT/MATCH functions as candidates for pushdown/computed column matching.

Enhancement: Replaced expression pushdown (EPD) approach with more complete version that does not over project rows.

Bugfix: Fixed an issue with nested extracts in the JSON_MATCH_ANY() statement.

Bugfix: Fixed an issue with backup restore when the backup has a corrupted GV timestamp in the about snapshot record.

Enhancement: The new engine variable parametrize_json_keys allows JSON keys to be parametrized and plans can be reused.

Bugfix: Fixed an issue that causes CREATE TABLES AS INFER PIPELINE to fail with an Out Of Memory error.

Bugfix: Fixed an issue with exporter not working with Public-Key Cryptography Standards #8(PKCS#8) pem format.

Bugfix: Fixed an issue with potential crashes when specifying a vector index in index hint.

Bugfix: Fixed a crash that occurred when trying to grant access to the table with no context database in per-privilege mode.

Bugfix: Fixed a memory issue, when calling a Table-Valued Function (TVF) that has arguments with default values.

Bugfix: Fixed an issue with the RG pool selector function return type check. The maximum value for n in VARCHAR(n) is now 256.

Enhancement: information_schema.mv_connection_attributes now shows tls_version and tls_cipher for connections where SSL is enabled.

Enhancement: Added ability for predicate pushdown with NOW() and user defined functions (UDF).

Enhancement.: Reduced amount of memory used by unlimited storage file listing, including point-in-time recovery (PITR).

Enhancement. Added an engine variable, innodb_lock_wait_timeout. This variable exists for backwards compatibility with MySQL and is non-operational in SingleStore Helios.

Enhancement. Optimized performance when using VECTOR built-ins with the VECTOR data type.

Bugfix: Fixed an issue where a backup would inadvertently carry metadata about the retention period of the backed-up unlimited storage database.

Bugfix: Fixed the change to aggregator activity queries in memsql_exporter not being applied properly.

Enhancement: Added ability to memsql_exporter to always return all sub-activities of an aggregator activity.

Enhancement: Improved ability to configure Kafka extract/egress by allowing additional options.

Enhancement: Improved performance for certain query shapes used by Kai.

Bugfix: Fixed the query-events endpoint link on the memsql_exporter landing page.

Bugfix: Fixed an issue where a segmentation fault (crash) appeared after terminating a query on the remote node.

Bugfix: Fixed an issue with crashes occurring when using certain JSON_EXTRACTs in the predicate expression of JSON_MATCH_ANY().

Bugfix: Fixed an issue where the number of rows affected with SELECT INTO object storage was erroneously reported as 0 rows.

Bugfix: Fixed an issue with START PIPELINE FOREGROUND skipping files in some circumstances.

Enhancement: The new engine variable max_table_memory_room_mb sets the maximum amount of memory required when creating or attaching a database. Configuring this engine variable allows more control over whether a detached database can be reattached.

Bugfix: Fixed an optimization out-of-memory issue cause by operators generated for wide user tables.

Bugfix: Fixed a crash that occurs in rare scenarios involving ALTER TABLE and failovers.

Bugfix: Fixed ineffective search options that change in subsequent vector search queries.

Bugfix: Resolved an issue related to FROM_BASE64 and TO_BASE64 builtins when processing large string inputs thereby preventing potential errors in reading communication packets.

Enhancement: The engine variable backup_multipart_upload_concurrency maximum value has been increased to 15.

Bugfix: Fixed the code involved in backups to improve download error messaging.

Bugfix: Fixed an bug that could cause scans using non-unique indexes which could return incorrect results or cause crashes.

Bugfix: Fixed a bug that could result in unrecoverable databases if the database had 1024 or more tables.

Enhancement: Improved performance for JSON_EXTRACT_<type> built-ins in ORDER BY clauses.

Enhancement: Added the blob_cache_miss_b and blob_cache_wait_time_ms columns to the following information schema views: information_schema.plancache, information_schema.MV_ACTIVITIES, information_schema.MV_ACTIVITIES_CUMULATIVE, information_schema.MV_TASKS, information_schema.MV_FINISHED_TASKS, and information_schema.MV_QUERY_ACTIVITIES_EXTENDED_CUMULATIVE.

Bugfix: Fixed an issue where INSERT...SELECT queries with a partition_id() filter generating an error.

Bugfix: Fixed an issue with memory crashing when using REGEXP_MATCH, JSON_INCLUDE_MASK, or JSON_EXCLUDE_MASK built-ins.

Enhancement: INFER PIPELINE for MongoDB CDC-in now generates tables with BSON column types.

Enhancement: Improved performance by optimizing joins on TABLE(JSON_TO_ARRAY()) queries.

Enhancement: Now suppressing a harmless traceSuspiciousClockUpdate trace message during recovery.

Enhancement: Added the ability for users to kill queries on a remote node without the user having PROCESS privileges.

Bugfix: Fixed an allocation issue that caused poor performance on high load insertion queries.

Enhancement: Enabled support for external UDFs used in INSERT statements with multiple VALUE clauses.

Enhancement: Added BSON fundamentals and column type support for SingleStore Kai.