Indexed Approximate-Nearest-Neighbor Search

A major feature introduced in this release is support for Vector Indexing. Vector indexing allows approximate-nearest-neighbor (ANN) search to improve performance of queries to find the top-K closest matches to a query vector. Supported index types include inverted file (IVF), hierarchical navigable small world (HNSW) and variants of both that use product quantization (PQ) to reduce memory usage. ANN search enables semantic text search, retrieval-augmented generation, and image search on larger-scale data sets more economically and with faster response time than was previously possible. It is an important tool to enable generative AI applications over larger data sets and with higher concurrency.

Query Plan Pinning

SingleStore now offers query plan pinning. Plan pinning ensures the optimizer will always pick a specific plan for a given query. Plan pinning allows users to select the most effective query plans for every query in their workload.

Query plan pinning can be used to prevent query plan regression after automatic upgrades. It can be used to avoid changes to query plans after statistics change or global settings change that affect query plans. It can also be used to avoid changes to query plans when a cluster is resized.

Extensions

Extensions in SingleStore allow you to combine user-defined objects, such as UDFs or UDAFs, into a packaged archive (the extension) and then create, manage, and deploy these objects and other resources using a single command. Extensions support both Wasm-based and PSQL functions. For more information, refer to Extensions.

Trace Events and Query History

Added the ability to trace query completions as events, which is the initial installment of the larger event tracing framework. The Query History feature relies on query event tracing, and can be used to display query trace events over time. The Query History feature can therefore be used to troubleshoot and optimize query performance, including, but not limited to, tracing and recording expensive queries, resolving unexpected slowdowns, and viewing and optimizing workloads in real time. Refer to Query History for more information.

Trace Log Rotation

SingleStore now has a built-in trace log rotation function. This is separate from and mutually exclusive of any manual log rotation implementation already in place, so if you want to use the new function and already have a solution in place using logrotate or similar, you will need to disable that before enabling this new rotation feature.

The built-in trace log rotation feature can be configured to rotate logs based on their size, based on time, or both. You can also, optionally, configure it to automatically delete logs after a specified number of days. The default, when the feature is enabled, is to never delete logs.

The new trace log rotation feature is disabled by default. See Rotating Trace Log to Manage its Size for information on how to enable and configure the new rotation feature.

Improved Memory Management for Resource Pools

Added the QUERY_MEMORY_PERCENTAGE option for resource pools, which restricts memory usage in the pool on a per-individual query basis. This in contrast to MEMORY_PERCENTAGE which restricts usage based on total memory used within the current pool.

For example, when creating or altering a resource pool, setting MEMORY_PERCENTAGE to 60% and QUERY_MEMORY_PERCENTAGE to 50% would configure the system so that all queries running within the specified resource pool should together use a maximum of 60% of system memory, and any single query running within the pool should use, at most, 50% of system memory. Example syntax:

CREATE RESOURCE POOL rpoolmain WITH
   MEMORY_PERCENTAGE = 60,
   QUERY_MEMORY_PERCENTAGE = 50, 
   SOFT_CPU_LIMIT_PERCENTAGE = 65,
   MAX_CONCURRENCY = 40;

Load Data Updates

SingleStore now supports loading data using the Change Data Capture (CDC) pipelines from the following data sources: MongoDB® and MySQL. Refer to Replicate Data from MongoDB® or Replicate Data from MySQL for information on loading data from the respective data source.

Projections

New Feature: Added ability to create a Projection which is a copy of some or all columns in a table, and may be sharded and sorted differently than the primary table. Projections can be used to speed up several types of queries including range filters, ORDER BY/LIMIT, GROUP BY, count(DISTINCT...), DISTINCT, joins, and row lookups on secondary keys. Projections depend on the table and are updated in real time when you update the table. Related information schema table: PROJECTIONS. (Added in 8.5.14)

Other Performance Enhancements

• SingleStore now supports creating shallow copies of tables. The WITH SHALLOW COPY feature copies an existing table and creates a new table that will have the same structure as the original table. The data is not physically copied to the new table, but referenced against the original table.

• SingleStore now supports sorted scan query plan operators for queries containing ORDER BY/LIMIT clauses when utilizing flexible parallelism. Before this enhancement, there could be performance regressions for this query shape using flexible parallelism.

• Improved performance when completing large sets of security operations (creating a lot of groups/users/roles, etc.).

• Added the ability to use named argument notation when calling a PSQL SP or function. Can reduce total lines of code and make code more readable.

• Added reduction of memory pre-allocation during columnstore JSON reads.

• Added ability to check if all leaf node partitions are available, before processing new batches.

• Addressed a table resolution issue for embedded recursive Common Table Expressions (CTEs).

Query Optimization Enhancements:

• Added support for Row Count and Selectivity hints in views.

• Added new join logic to recognize when a non-reference table is being joined exclusively to reference tables and then gather the non-reference table to avoid duplicating work across every partition.

• Remove redundant aggregation functions and GROUP BY statements.

• JSON expressions are properly pushed down.

• Allow columnstore optimization for JSON_MATCH_ANY with JSON_EXTRACT in predicate.

• Added support for hash joins on null-accepting expressions in the ON condition of outer joins.

• Automatically rewrite A=B OR (A IS NULL AND B IS NULL) to null safe equal (A<=>B) so that many important optimizations (e.g. shard key joins, hash joins) will work.

• Perform a subselect to join rewrite in an UPDATE statement when there are multiple columns in the subselect.

• Removed some query shape lockdowns.

• Added support for flipping join order for full outer join.

• Improved performance by not executing query optimization procedures for read queries during the process of persistent plan cache lookup. This optimization strategy has resulted in improved lookup performance, leading to faster data retrieval operations.

• Added support for statistics on correlations between columns in cases where highly correlated filters are used.

• The Data API now supports HTTPS for connections where ssl_key is encrypted with ssl_key_passphrase.

• Removed parametrization of LIMIT 0 and LIMIT 1 to unlock more rewrites, especially for subselects.

• Modified computed column matching to accurately evaluate JSON expressions containing equality and non-safe equality comparisons.

• Fixed an issue where filtering with a JSON_EXTRACT_<type> function performs inconsistently.

• The query optimizer now considers more LEFT JOIN elimination cases.

New Information Schema Views and Columns

• Enhancement: Added the TABLE_NAME column to the LOAD_DATA_ERRORS information schema view. TABLE_NAME is the name of the table associated with the error. (8.5.38)

• Enhancement: Fixed the KILLED_QUERIES counter in MV_RESOURCE_POOL_STATUS. (8.5.33)

• Enhancement: Add bottle service reliability metrics to MV_BOTTOMLESS_STATUS_EXTENDED. (8.5.32)

• New Feature: New Projections-related information schema table: PROJECTIONS. (8.5.14)

• Enhancement: Added the usage_telemetry_status information schema table, which shows the state of the usage telemetry feature and also an example of a telemetry report. (8.5.10)

• Enhancement: Added the blob_cache_miss_b and blob_cache_wait_time_ms columns to the following information schema views: information_schema.plancache, information_schema.MV_ACTIVITIES, information_schema.MV_ACTIVITIES_CUMULATIVE, information_schema.MV_TASKS, information_schema.MV_FINISHED_TASKS, and information_schema.MV_QUERY_ACTIVITIES_EXTENDED_CUMULATIVE. (8.5.2)

• Added a new view, correlated_column_statistics, to provide metadata on correlated columns.

• Added a new view, RESOURCE_POOL_PRIVILEGES, to provide information about resource pool grants and privileges.

• Added the following new columns to MV_BACKUP_HISTORY:

  • error_code: Error code for failed backups.

  • error_message: Error message for failed backups.

• Added the following new columns to MV_SYSINFO_DISK:

  • read_operations_cumulative_per_device: Number of read operations performed by the device since start up.

  • write_operations_cumulative_per_device: Number of write operations performed by the device since start up.

  • device_name: Name of the device to which the values in read_operations_cumulative_per_device and write_operations_cumulative_per_device are associated.

• Added the following to support trace events that are used by the Query History feature:

  • MV_TRACE_EVENTS: A snapshot of all trace events, the size of which is dictated by the trace_events_queue_size variable

  • MV_TRACE_EVENTS_STATUS: A view that reflects the status of current trace events

  • LMV_TRACE_EVENTS: A snapshot of each node's trace events

• Added the following new column to ADVANCED_HISTOGRAMS, L/MV_QUERY_PROSPECTIVE_HISTOGRAMS, and L/MV_PROSPECTIVE_HISTOGRAMS:

  • JSON_KEY: an entry for each (column, json_key) pair. For non-json columns JSON_KEY is NULL.

• Bugfix: Updated the information_schema.USERS view to reflect the account status for locked users.

New Commands and Functions

• Enhancement: Added the EXPORT TELEMETRY SQL command, used to write usage telemetry data to a local file. (8.5.10)

• Enhancement: INFER PIPELINE for MongoDC CDC-in now generates tables with BSON column types. (8.5.3)

• Added support for the REGEXP_MATCH() function. This function returns a JSON array of matching substring(s) within the first match of a regular expression pattern to a string.

• Added support for CUBE and ROLLUP grouping operations to Wasm-based user-defined aggregate functions (UDAFs) in SingleStore. For more information, refer to CREATE AGGREGATE.

• Added the following to support trace events that are used by the Query History feature:

  • CREATE EVENT TRACE to create a trace event

  • DROP EVENT to drop a trace event

• Added ability to use DELETE on identical keys with the ON DUPLICATE KEY clause. This is in addition to existing "upsert" support with ON DUPLICATE KEY UPDATE. This allows new scenarios such as the ability to manage streaming aggregation with INSERT … ON DUPLICATE KEY UPDATE … ELSE DELETE ….

New or Modified Engine Variables

• Enhancement: Added a new engine variable scheduler_slow_threshold_seconds that specifies the threshold for triggering the verbose logging of scheduler thread timing. (8.5.39)

• Enhancement: Added a new engine variable scheduler_slow_ready_queue_seconds that specifies the threshold for triggering logging of slow ready queue draining. (8.5.39)

• Enhancement: Added a new max_autostats_update_workers global variable that sets the maximum number of merger threads that the autostats updater can run on. (8.5.33)

• Enhancement: Added a new filter derivation rewrite controlled by the optimizer_not_null_filter_derivation engine variable. (8.5.33)

• Enhancement: Added a new engine variable costing_max_djo_tables which sets the maximum amount of tables we allow in full DQO estimate search when filling table row estimates for distributed join hints. (8.5.16)

• Enhancement: Made the following engine variable settable in SingleStore Helios (8.5.15):

  multi_insert_tuple_count, columnstore_row_value_table_lock_threshold, internal_columnstore_max_uncompressed_blob_size, internal_columnstore_max_uncompressed_rowindex_blob_size, columnstore_validate_blob_before_merge, default_max_histogram_buckets, enable_varbuffer_dictionary_compression, enable_alias_space_trim, skip_segelim_with_inlist_threshold, default_autostats_histogram_mode, default_autostats_columnstore_cardinality_mode, default_autostats_rowstore_cardinality_mode, default_autostats_columnstore_sampling, experimental_disable_multipartition_parallel_read, internal_enable_parallel_query_throttling, enable_spilling, spilling_node_memory_threshold_ratio, spilling_query_operator_memory_threshold, regexp_compile_mem_mb, selectivity_estimation_min_sample_size, repartition_minimum_buffer_size, use_seekable_json, json_document_max_children, json_document_max_leaves, json_document_sparse_children_check_threshold, json_document_sparse_children_check_ratio, json_document_absolute_sparse_key_check_ratio, json_document_page_data_soft_limit, ignore_insert_into_computed_column, max_subselect_aggregator_rowcount, leaf_pushdown_enable_rowcount, reshuffle_group_by_base_cost, max_broadcast_tree_rowcount, enable_broadcast_left_join, display_full_estimation_stats, force_bushy_joins, force_heuristic_rewrites, force_table_pushdown, force_bloom_filters, explain_joinplan_costs, statistics_warnings, optimizer_warnings, optimize_constants, optimize_huge_expressions, optimize_expressions_larger_than, optimize_stmt_threshold, quadratic_rewrite_size_limit, batch_external_functions, batch_external_functions_size, optimizer_enable_json_text_matching, optimizer_enable_orderby_limit_self_join, distributed_optimizer_broadcast_mult, distributed_optimizer_old_selectivity_table_threshold, distributed_optimizer_selectivity_fallback_threshold, distributed_optimizer_verbose, old_local_join_optimizer, optimizer_hash_join_cost, optimizer_merge_join_cost, optimizer_nested_join_cost, optimizer_disable_right_join, interpreter_mode_sampling_threshold, hash_groupby_segment_distinct_values_threshold, sampling_estimates_for_complex_filters, enable_histogram_to_unwrap_literals, estimate_zero_rows_when_sampling_data_is_missing, exclude_scalar_subselects_from_filters, leaf_pushdown_default, distributed_optimizer_nodes, optimizer_num_partitions, enable_binary_protocol, enable_multipartition_queries, enable_local_shuffle_group_by, enable_skiplist_sampling_for_selectivity, columnstore_sample_per_partition_limit, disable_sampling_estimation, disable_histogram_estimation, inlist_precision_limit, allow_materialize_cte_with_union, optimizer_cross_join_cost, distributed_optimizer_run_legacy_heuristic, distributed_optimizer_min_join_size_run_initial_heuristics, distributed_optimizer_unrestricted_search_threshold, singlebox_optimizer_cost_based_threshold, distributed_optimizer_estimated_restricted_search_cost_bound, disable_reference_table_leftjoin_where_pushdown, disable_sampling_estimation_with_histograms, disable_subquery_merge_with_straight_joins, default_columnstore_table_lock_threshold, default_spill_dependent_outputters, query_rewrite_loop_iterations, verify_fields_in_transitivity, optimizer_min_reference_tables_for_gathered_join, optimizer_min_reference_rows_for_gathered_join, max_expression_query_limit, max_expression_item_limit, optimizer_empty_tables_limit, optimizer_beam_width, optimizer_disable_subselect_to_join, disable_remove_redundant_gby_rewrite, subquery_merge_with_outer_joins, optimizer_disable_operator_tree_nullability_check, clamp_histogram_date_estimates, varchar_column_string_optimization_length, histogram_column_correlation, use_column_correlation, consider_secondary_projection, optimizer_disable_semi_join_reduction, optimizer_gb_placement_table_limit, optimizer_disable_transitive_predicates

• Enhancement: The Tracelog rotation threshold now applies to rotated logs. Added two new engine variables to control the total size of retained log files: auditlog_retention_size and tracelog_retention_size. (8.5.13)

• Enhancement: Dedicated Admin Connections via a TCP port. Added two new engine variables, enable_dac_ip_port and dac_ip_port - these can be used to open an extra mysql port on startup which will accept dedicated admin connections. (8.5.13)

• Enhancement: Added the send_usage_telemetry engine variable used to enable/disable the collection and sending of usage telemetry data. This defaults to OFF. (8.5.10)

• Enhancement: Added the verbose_time_heartbeat_thread engine variable used to enable/disable logging to heartbeat thread in order to better diagnose process stalls. The default value is off, setting this to on will enable the logging. (8.5.10)

• Enhancement: The new engine variable parametrize_json_keys allows JSON keys to be parametrized and plans can be reused. (8.5.7)

• Enhancement. Added an engine variable, innodb_lock_wait_timeout . This variable exists for backwards compatibility with MySQL and is non-operational in SingleStore. (8.5.6)

• Enhancement: The engine variable backup_multipart_upload_concurrency maximum value has been increased to 15. (8.5.4)

• Enhancement: The new engine variable max_table_memory_room_mb sets the maximum amount of memory required when creating or attaching a database. Configuring this engine variable allows more control over whether a detached database can be reattached. (8.5.4)

• Added enable_idle_table_optimizations and enable_idle_table_eviction, which are used to reduce table memory overhead for idle tables on a cluster. The feature is enabled by default on all new and existing clusters. The variable enable_idle_table_optimizations needs to be set at runtime and requires a restart for changes to take effect. It can be set to OFF or ON (default). The other related variable, enable_idle_table_eviction, can be set during a session (though a very small amount of overhead will remain until the server is restarted) and can be set to Full, SkipListsOnly, and Off. It defaults to SkipListsOnly, which means SingleStore will only evict skiplist indexes for idle tables on the cluster. Full means it will evict skiplists and table modules, and Off means no eviction.

• The engine variable lockfree_backup is now deprecated.

• The engine variable disable_update_delete_distributed_transactions is now deprecated.

• Added the bottomless_experimental_blobstore_mode engine variable, which when enabled, completes additional verification of persisted files immediately after upload. This mode is experimental and may reduce upload speed. Please use with caution.

• Added privilege_caches_update_mode, which can be used to address some performance issues that occur when performing large sets of security operations (creating a lot of groups/users/roles, etc.).

• The query_parallelism engine variable (which was deprecated in 8.1) now is non-functional. To modify Flexible Parallelism settings, use query_parallelism_per_leaf_core instead.

• Added the use_user_provided_index_types_in_show engine variable which controls what will be displayed via the DESCRIBE <table_name> or SHOW COLUMNS syntaxes for backward compatibility.

• Added the throttle_replay_behind_tail_mb engine variable which controls how far the system allows a child aggregator to lag behind replicating the transaction log of it's master aggregator before throttling is applied to the master aggregator.

• Added the ability to use the ANALYZE command with JSON keys to create histograms when the new engine variable enable_json_statistics is enabled. The engine variable enable_json_statistics is disabled by default.

• Added trace_events_queue_size to capture trace events, the first of which is query event tracing. This engine variable is disabled by default (set to a value of 0 MB, where the value must be provided in bytes). Refer to Query History for more information.

• Added the optimizer_disable_transitive_predicates engine variable which disables predicate transitivity on query rewrites if set to TRUE. This engine variable defaults to FALSE.

• Added the java_pipelines_java11_path engine variable to specify the path of the (JRE 11+) java binary.

• Added a new engine variable pipelines_cdc_java_heap_size to specify the JVM heap size limit for CDC-in pipelines.

Miscellaneous

• Enhancement: Improved optimization speed of parameterized IN-lists by limiting traversal depth. (8.5.43)

• Bugfix: Fixed an issue that caused a crash when a query referenced too many tables. (8.5.43)

• Enhancement: Added support for updated Standard and Premium licenses. (8.5.42)

• Enhancement: The deprecated Strict Mode feature is officially removed from SingleStore. (8.5.42)

• Bugfix: Fixed a bug in JSON_TO_ARRAY() join pushdown optimization that occured when the JSON value being extracted from table_col is a JSON array. (8.5.42)

• Bugfix: Fixed an issue to prevent LRU eviction from stalling when several table modules are stale. (8.5.42)

• Bugfix: Fixed ALTER RESOURCE POOL to respect queue depth constraints. (8.5.42)

• Bugfix: Fixed a bug that produced incorrect results when performing a hash join with a condition that compares mismatched types where one of the types is BSON. (8.5.42)

• Bugfix: Fixed an issue that impacted creating or altering a resource pool with a CPU percentage limitation after a MA failover. (8.5.42)

• Bugfix: Fixed a bug that occurred when a query used I0 as a column name. (8.5.42)

• Enhancement: Added an internal allocator Alloc_connection_context which tracks certain per-connection allocations that were previously tracked using the standard allocator. (8.5.40)

• Enhancement: Improved pipeline error clearing. Additional errors are cleared when CLEAR PIPELINES ERRORS is run or when the pipelines_errors_retention_minutes limit is reached. (8.5.39)

• Enhancement: Added logging for LRU compiled unit eviction. (8.5.39)

• New feature: Added support for the file_compression parameter in the CONFIG clause of CREATE PIPELINE AS ... LOAD DATA S3 statement. Decompresses files with the specified extensions. Refer to S3 Configurations for more information. (8.5.38)

• Bugfix: Fixed bottomless upload throttling criteria. (8.5.38)

• Bugfix: Blocked creation of temporary table as a shallow copy. (8.5.38)

• Bugfix: Fixed display of default BSON and string values with null-terminators in the information_schema. (8.5.38)

• Bugfix: Fixed IN-list index matching for columnstore tables when a query has multiple IN-list predicates. (8.5.38)

• Bugfix: Fixed a bug that led to a master aggregator crash when using dynamic resource pools.

• Bugfix: Fixed a crash that occurred during spilling when executing a query with a large number of GROUP BY columns. (8.5.37)

• Bugfix: Allowed cached table memory to be freed for empty tables in replica databases. (8.5.37)

• Bugfix: Fixed a deadlock in a corner case involving index eviction and replaying two-phase commit transactions. (8.5.37)

• Bugfix: Fixed the result of ORDER BY ... LIMIT queries on tables with a sort key column defined in DESC order. (8.5.37)

• Bugfix: Fixed an issue caused when multi-part names were used in common table expressions (CTEs) within UNION.

• Enhancement: Improved out of memory (OOM) error message to make the message more accurate. (8.5.36)

• Bugfix: Fixed a crash that could occur when columns were dropped with ALTER TABLE. (8.5.36)

• Bugfix: Fixed an issue where left join sometimes returned an incorrect result when spilling occurred. (8.5.36)

• Enhancement: Improved the performance of regular expression (regex) operations. (8.5.35)

• Bugfix: Fixed deadlock in db recovery due to evicting table module holding an open LOCK. (8.5.34)

• Bugfix: Fixed an issue where the garbage collection of a partition database's 2PC transaction table could unnecessarily delay the reprovisioning of the reference database. (8.5.34)

• Enhancement: When printing the table digest SHA value, the table DDL schema is also printed to memsql_METADATA.log. (8.5.33)

• Enhancement: Added metrics to track the availability of unlimited storage and the bottle service. (8.5.32)

• Bugfix: Fixed a certificate refresh issue for connections to remote storage in long-running clusters. (8.5.32)

• Bugfix: Fixed a rare issue where a membership-related clustering operation could delay the re-provision or deletion of a cluster database. (8.5.32)

• Bugfix: Fixed an issue where JSON_MATCH_ANY() queries with MATCH_ELEMENTS evaluated incorrectly on non-array values. (8.5.32)

• Bugfix: Fixed sporadic reading failures from MV_SYSINFO_CPU. (8.5.32)

• Bugfix: Fixed an issue in ALTER TABLE when adding a computed column containing JSON_BUILD_OBJECT. (8.5.21)

• Bugfix: Addressed a crash issue with REGEX_REPLACE. (8.5.21)

• Bugfix: Reduced stack usage during compilation for some CASE expressions with many WHEN-THEN statements. (8.5.20)

• Bugfix: Fixed a potential deadlock issue between background garbage collector threads and foreground query execution threads. (8.5.20)

• Bugfix: Fixed a collation issue in the JSON_COLUMN_SCHEMA information schema table. (8.5.20)

• Enhancement: SingleStore now supports module eviction of idle tables with an ALTER history. ALTER includes CREATE INDEX, ALTER TABLE ADD COLUMN, and some others. (8.5.20)

• Bugfix: Fixed a bug with Wasm UDFs where REPLACE FUNCTION did not correctly evict from the plancache, causing continued usage of the old UDF after replacement. (8.5.19)

• Bugfix: Fixed an upgrade issue present in version 8.5.17. (8.5.18)

• Bugfix: Fixed a BSON issue that could occur when large documents are read and mistakenly reported as corrupted. (8.5.18)

• Enhancement: Upgraded OpenSSL 1.0.2u to 1.0.2zj (8.5.18)

• Bugfix: Fixed a crash issue that could occur under a rare race condition in some distributed join queries. (8.5.18)

• Bugfix: Fixed statement atomicity in transactions writing to tables that have projections on them. (8.5.18)

• Enhancement: Allow Kerberos users to skip password validation for users who are authenticated with plugins. (8.5.18)

• Enhancement: Added support for loading data from compressed Avro datafiles. (8.5.17)

• Bugfix: Fixed an issue that occurred when primary key is NULLABLE when copying a table using "Create table as select" statement. (8.5.17)

• Bugfix: Addressed a crash issue occurring while executing queries with vector builtins on clusters running via Docker containers on Mac machines equipped with Apple silicon processors (M1). (8.5.17)

• Bugfix: No longer show information_schema pipeline metadata of databases that are not attached to the workspace. (8.5.17)

• Enhancement: Use SeekableString encoding by default for storing VECTOR datatype columns in columnstore tables. (8.5.17)

• Bugfix: Fixed an issue that caused rare "Arrangement already exists in namespace" errors in recursive CTE queries. (8.5.17)

• Bugfix: Improved the hashGroupBy spilling condition to trigger (spill) when appropriate. (8.5.17)

• Enhancement: Added more config information about pipelines to memsql_exporter. (8.5.15)

• Enhancement: Improved error handling for VECTOR type. (8.5.15)

• Enhancement: Added support for aggregate functions (FIRST, LAST) with VECTOR datatype. (8.5.15)

• Enhancement: Added support for Kafka key (producer) in the SELECT ... INTO KAFKA command. (8.5.15)

• Enhancement: Improved Kafka error messages to make them more specific. (8.5.15)

• Bugfix: Fixed a case where a killed query could incorrectly return a JSON error rather than query interrupted message. (8.5.15)

• Bugfix: MV_DISK_USAGE queries can no longer block garbage collection. (8.5.15)

• Bugfix: Fixed a crash-on-startup issue on RHEL-based operating systems (CentOS 9, etc). (8.5.15)

• Bugfix: Fixed potential deadlock between table module eviction and GC manager. (8.5.15)

• Bugfix: Fixed a crash that could occur when a scalar subselect is passed as an argument to an external TVF. (8.5.15)

• Bugfix: Fixed an issue with restoring from public backups. (8.5.15)

• Bugfix: Backups now throttle after snapshot. (8.5.15)

• Bugfix: Removed an unnecessary metadata copy to improve JSON performance. (8.5.14)

• Enhancement: Multi-insert queries now respect lower columnstore_segment_rows settings. (8.5.14)

• Enhancement: Subprocess and S3 API tracing are now also enabled via the enable_subprocess_tracing engine variable. (8.5.14)

• Enhancement: Added the optional IF NOT EXISTS clause to the CREATE TABLE ... AS INFER PIPELINE statement. Refer to Replicate Data from MongoDB® for more information. (8.5.14)

• Enhancement: Dedicated Admin Connections via a TCP port. Added two new engine variables, enable_dac_ip_port and dac_ip_port - these can be used to open an extra mysql port on startup which will accept dedicated admin connections. (8.5.13)

• Enhancement: The Tracelog rotation threshold now applies to rotated logs. Added two new engine variables to control the total size of retained log files: auditlog_retention_size and tracelog_retention_size. (8.5.13)

• Bugfix: LAST_INSERT_ID will now correctly retrieve the last inserted ID for a forwarded INSERT query. (8.5.13)

• Bugfix: The rule based optimization warning message will only be shown when cost based optimization is meaningful. (8.5.13)

• Bugfix: Fixed a minor performance issue where a table module could be evicted prematurely during database recovery. (8.5.13)

• Bugfix: Fixed a very rare deadlock between ALTER of a table and a DROP of a database. (8.5.13)

• Bugfix: Fixed a JSON insertion failure to old utf8 sample tables that contain utf8mb4 characters by encoding it with base64 format beforehand for backward compatibility. (8.5.13)

• Enhancement: Disk Spilling now takes QUERY_MEMORY_PERCENTAGE resource pool setting into consideration. (8.5.13)

• Bugfix: Fixed an issue with a blob file leak in a rare crash scenario. (8.5.12)

• Bugfix: Fixed a potential deadlock in a scenario involving DDL, a 2PC transaction, and a concurrent failover. (8.5.12)

• Bugfix: Fixed a deadlock involving BACKUP and ALTER in a rare case. (8.5.12)

• Enhancement: Added support for utf8mb4 symbols in column comments, table comments, and user comments. (8.5.12)

• Enhancement: Updated timezone data, now using IANA's tzdata version 2024a; new timezones Kyiv, Ciudad_Juarez, and Kanton supported. (8.5.12)

• Enhancement: Enabled additional query hint for non-distributed queries so that first run of those queries can be queued. (8.5.12)

• Enhancement: Idle Table Eviction improvement. In the 8.5 GA version, an idle table's objects cannot be evicted from memory if the table has one or more UNIQUE key columns (both PRIMARY KEYs and any other keys with a UNIQUE constraint). In the 8.5.11 patch, idle tables' code objects can be evicted from memory if the table has one or more UNIQUE key columns. (8.5.11)

• Bugfix: Fixed an issue in background merger that allowed concurrent deletes of empty segments. (8.5.11)

• Bugfix: Fixed an issue to ensure preserving a field of a table correctly. (8.5.11)

• Enhancement: Added numInferredPaths and schemaSize as new columns to JSON_COLUMN_SCHEMA table. numInferredPaths is the number of key paths inferred for the segment. schemaSize is the size of schema_json in bytes. (8.5.10)

• Enhancement: Added logging to heartbeat thread in order to better diagnose process stalls. This logging is enabled/disabled via the new verbose_time_heartbeat_thread engine variable. The default value is off, setting this to on will enable the logging. (8.5.10)

• Bugfix: Added safety checks around dropping databases without exclusive access to storage. (8.5.10)

• Enhancement: Added logging to heartbeat thread in order to better diagnose process stalls. (8.5.10)

• Enhancement: Added the ability to collect and send SingleStore Self-Managed usage telemetry data to SingleStore via the EXPORT TELEMETRY SQL command (for 1-time or ad hoc use) or the send_usage_telemetry engine variable. This defaults to OFF. Added an associated information schema table: usage_telemetry_status. (8.5.10)

• Bugfix: Fixed an issue where explicitly defining a JSON type in the RETURNS of a Wasm TVF can cause an error when it is run. (8.5.10)

• Bugfix: Fixed a rare crash scenario. (8.5.10)

• Bugfix: Fixed an issue that could lead to the risk of undefined behavior when running DROP EXTENSION IF EXISTS. (8.5.10)

• Enhancement: IN-lists will now use hashmap optimization for matching parameters in *_MATCH_ANY statements. (8.5.10)

• Enhancement: Made default_distributed_ddl_timeout a sync variable on all nodes. (8.5.10)

• Enhancement: Modified the conversion logic for converting from VECTOR to BSON types. Now, when casting VECTOR(F32), it will generate a BSON array of doubles, rather than a combination of numeric types. (8.5.10)

• Bugfix: Addressed a family of issues relating to optimal execution of JSON extracts on schema'd data.

• Bugfix: Fixed an issue with regression in replay performance for databases with many tables. (8.5.9)

• Bugfix: Fixed an issue where the existing websocket connection would close when variables are updated in global scope. (8.5.8)

• Enhancement: Added new metrics to memsql_exporter based on the information_schema.mv_sysinfo_disk columns. (8.5.8)

• Enhancement: Now recognize more EXTRACT/MATCH functions as candidates for pushdown/computed column matching. (8.5.8)

• Enhancement: Replaced expression pushdown (EPD) approach with more complete version that does not over project rows. (8.5.8)

• Enhancement: Added spilling metrics for TopSort (ORDER BY with LIMIT). (8.5.8)

• Enhancement: Added support for usage of vector built-ins with string/JSON arguments without requiring an explicit typecast to VECTOR (e.g., 'SELECT @vec<*> '[1,2,3]'). (8.5.8)

• Enhancement: Added support for creating numeric histograms based on JSON values. (Feature flag gated, contact your SingleStore representative). (8.5.8)

• Enhancement: Disabled the default semi join reduction rewrite. (8.5.8)

• Enhancement: Creation of computed columns with VECTOR data type is now allowed. (8.5.8)

• Bugfix: Removed the hard-coded 'collation_server' from constant.StrictModeParams. (8.5.8)

• Bugfix: Fixed the result collation from VECTOR built-ins to be utf8mb4_bin instead of binary. (8.5.8)

• Bugfix: Fixed an issue with a sensitive information leak inside of out-of-memory reports. (8.5.8)

• Enhancement: Improved JSON histograms to support numeric histograms, analyze command, info_schema.optimizer_statistics, and displays JSON histograms. (8.5.8)

• Enhancement: Added ability for information_schema.optimizer_statistics to display JSON keys. (8.5.8)

• Enhancement: Added ability to skip eviction when log replaying hits the blob cache space limit, providing a greater chance to succeed. (8.5.8)

• Enhancement: Added support for warming blob cache with table's column data. Syntax is: OPTIMIZE TABLE <table_name> WARM BLOB CACHE FOR COLUMN <column_names>: (8.5.8)

  SQL

  Copy

  OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN c1, c2;
  
  OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN *;

• Enhancement: Added support to the VECTOR built-ins for all VECTOR elements types (e.g., F32, F64, I8, I16, I32, and I64). (8.5.8)

• Bugfix: Fixed an issue with nested extracts in the JSON_MATCH_ANY() statement. (8.5.7)

• Bugfix: Fixed an issue with backup restore when the backup has a corrupted GV timestamp in the about snapshot record. (8.5.7)

• Bugfix: Fixed an issue that causes CREATE TABLES AS INFER PIPELINE to fail with an Out Of Memory error. (8.5.7)

• Bugfix: Fixed an issue with exporter not working with Public-Key Cryptography Standards #8(PKCS#8) pem format. (8.5.7)

• Bugfix: Fixed an issue with potential crashes when specifying a vector index in index hint. (8.5.7)

• Bugfix: Fixed a crash that occurred when trying to grant access to the table with no context database in per-privilege mode. (8.5.7)

• Bugfix: Fixed a memory issue, when calling a Table-Valued Function (TVF) that has arguments with default values. (8.5.7)

• Bugfix: Fixed an issue with the RG pool selector function return type check. (8.5.7)

• Enhancement: information_schema.mv_connection_attributes now shows tls_version and tls_cipher for connections where SSL is enabled. (8.5.7)

• Enhancement: Added ability for predicate pushdown with NOW() and user defined functions (UDF). (8.5.6)

• Enhancement.: Reduced amount of memory used by unlimited storage file listing, including point-in-time recovery (PITR). (8.5.6)

• Enhancement. Optimized performance when using VECTOR built-ins with the VECTOR data type. (8.5.6)

• Bugfix: Fixed an issue where a backup would inadvertently carry metadata about the retention period of the backed-up unlimited storage database. (8.5.6)

• Bugfix: Fixed the change to aggregator activity queries in memsql_exporter not being applied properly. (8.5.6)

• Enhancement: Added ability to memsql_exporter to always return all sub-activities of an aggregator activity. (8.5.5)

• Enhancement: Improved ability to configure Kafka extract/egress by allowing additional options. (8.5.5)

• Enhancement:Improved performance for certain query shapes used by SingleStore Kai. (8.5.5)

• Bugfix: Fixed the query-events endpoint link on the memsql_exporter landing page. (8.5.5)

• Bugfix: Fixed an issue where a segmentation fault (crash) appeared after terminating a query on the remote node. (8.5.5)

• Bugfix: Fixed an issue with crashes occurring when using certain JSON_EXTRACTs in the predicate expression of JSON_MATCH_ANY(). (8.5.5)

• Bugfix: Fixed an issue where the number of rows affected with SELECT INTO object storage was erroneously reported as 0 rows. (8.5.5)

• Bugfix: Fixed an issue with START PIPELINE FOREGROUND skipping files in some circumstances. (8.5.5)

• Bugfix: Fixed an bug that could cause scans using non-unique indexes which could return incorrect results or cause crashes. (8.5.4)

• Bugfix: Fixed ineffective search options that change in subsequent vector search queries. (8.5.4)

• Bugfix: Fixed the code involved in backups to improve download error messaging. (8.5.4)

• Bugfix: Fixed a crash that occurs in rare scenarios involving ALTER TABLE and failovers. (8.5.4)

• Bugfix: Fixed an optimization out-of-memory issue cause by operators generated for wide user tables. (8.5.4)

• Bugfix: Resolved an issue related to FROM_BASE64 and TO_BASE64 builtins when processing large string inputs thereby preventing potential errors in reading communication packets. (8.5.4)

• Bugfix: Fixed a bug that could result in unrecoverable databases if the database had 1024 or more tables. (8.5.3)

• Enhancement: Improved performance for JSON_EXTRACT_<type> built-ins in ORDER BY clauses. (8.5.2)

• Bugfix: Fixed an issue with memory crashing when using REGEXP_MATCH, JSON_INCLUDE_MASK, or JSON_EXCLUDE_MASK built-ins. (8.5.2)

• Bugfix: Fixed an issue where INSERT...SELECT queries with a partition_id() filter generating an error. (8.5.2)

• Enhancement: Now suppressing a harmless traceSuspiciousClockUpdate trace message during recovery. (8.5.2)

• Enhancement: Enabled support for external UDFs used in INSERT statements with multiple VALUE clauses. (8.5.2)

• Enhancement: Added the ability for users to kill queries on a remote node without the user having PROCESS privileges. (8.5.2)

• Enhancement: Added BSON fundamentals and column type support for SingleStore Kai. (8.5.2)

• Bugfix: Fixed an allocation issue that caused poor performance on high load insertion queries. (8.5.2)

• Enhancement: Improved performance by optimizing joins on TABLE(JSON_TO_ARRAY()) queries. (8.5.2)

• Enhancement: Introduced a new VECTOR data type that is recommended over the BLOB data type for vector operations and use with ANN indexes. The VECTOR type improves code readability, error checking, and reduces total lines of code.

• Enhancement: Added support for DDL forwarding for CLEAR BACKUP_HISTORY.

• Enhancement: Improved retry logic for connection reset: write errors.

• Enhancement: SingleStore now natively supports the BSON data type.

• Enhancement: Added support for collection.exclude.list, database.include.list, and database.exclude.list parameters to the CONFIG/CREDENTIAL clause of the CREATE AGGREGATOR PIPELINE ... AS LOAD DATA MONGODB statement. Refer to Replicate Data from MongoDB® for more information.

• Enhancement: Added support for Approximate Nearest Neighbor (ANN) vector search using inverted file (IVF) and hierarchical navigable small world (HNSW) indexes, and variants of them based on product quantization (PQ). Enables support of larger-scale semantic search and generative AI applications.

• Enhancement: Increased name length limit to 256 characters for  tables, views, table columns, view columns, procedures, functions, and aliases.

• Enhancement: Added the ability to truncate plancache file names if they exceed the operating system's specified limit (255 bytes).

• Bugfix: Both SHOW PIPELINES and SELECT * FROM information_schema.pipelines now show consistent pipeline state information across master and child aggregators.

• Bugfix: Specific pipeline built-ins like pipeline_source_file() and pipeline_batch_Id() should not be used in UPSERT clause when creating a pipeline.

• Bugfix: Improved the ability to terminate expressions containing JSON built-ins.

• Enhancement: Added error-handling details for pipelines, including state, error, and performance-related metrics through monitoring solutions.

• Enhancement: MemSQL Procedural SQL (MPSQL) has been renamed to simply Procedural SQL (PSQL). The name change will only show in some SHOW command output and information schema views. For example, SHOW FUNCTIONS output changed.

• Enhancement: Added support for %ROWTYPE and %TYPE for use in declaring scalar type variables and parameters. Employing these abbreviations in Procedural SQL (PSQL) can lead to a reduction in the required lines of code.

• Enhancement: Introduced blob cache profiling metrics for columnstore tables on unlimited storage databases. Refer to the PROFILE page for more information on what blob cache metrics are available.

• Enhancement: Added infix operators for dot_product (<*>) and euclidean_distance (<->).

• Enhancement: Added ability to delay retry attempts for pipeline retries.

• Bugfix: Updated the output for the Key_name and Index_type columns in the SHOW INDEX, SHOW INDEXES, and SHOW KEYS commands for primary keys on columnstore tables. Refer to the SHOW INDEX, SHOW INDEXES, SHOW KEYS page for more information.

• Bugfix: Improved the error message displayed when trying to create a primary key on an existing table.

• Bugfix: Improved the error message displayed when a GRANT command fails due to missing permissions. The error message will now show the missing permissions:

  SQL

  Copy

  GRANT SELECT, UPDATE, DELETE, EXECUTE ON *.* TO test2;

  ERROR 1045 (28000): Current user is missing UPDATE, DELETE permission(s) for this GRANT

• Bugfix: Fixed a case when UNIX_TIMESTAMP() was incorrectly returning 999999999.999999 for DATETIME data types with precision.

• Bugfix: Fixed an issue where a user could be erroneously marked as deleted.

• Bugfix: Fixed an issue where a child aggregator could crash if it ran out of memory during query forwarding.

• Bugfix: Fixed an issue with blob cache LRU2 eviction that could occur when a query fetches a blob, evict it, and fetches it again.

• Bugfix: Removed the word "FORCE" from the log message for DETACH LEAF.

• Bugfix: Fixed an issue that could cause information for a blob to be missing from an Information Schema table.

• Enhancement: It is now possible to set tracelog_rotation_size and tracelog_rotation_time to 0 (disable the Log Rotation feature) directly in the configuration file. For example, tracelog_rotation_size = 0.

• Bugfix: Disk Spilling now takes the resource pool settings into consideration.

• Enhancement: Auto user creation is deprecated and the NO_AUTO_CREATE_USER variable is enabled by default.

• Bugfix: Fixed an erroneous access denied issue to views selecting from shard tables with computed columns.

• Bugfix: Fixed a rare issue where the incorrect timezone could be used in logging.

• Bugfix: Fixed an issue where using user-defined variables inside ORDER/GROUP BY statements could cause a crash.

• Enhancement: Added column name to error messages when invalid date/time is inserted into a column.

• Enhancement: Added BSON columnstore functionality over Parquet storage.

• Enhancement: Added support for SELECT ... INTO KAFKA using OAUTH credentials.

• Bugfix: Prevent the ability to create a Kafka Pipeline using Parquet Format.

• Bugfix: Queued Time is now excluded from the cost estimate for workload management leaf memory.

• Bugfix: Specific error messages are now logged for GCS subprocess failures.

• Bugfix: JSON_KEY escape characters were not working as expected.

• Bugfix: ALTER PIPELINE setting max_partitions_per_batch to use a default 0 value is now allowed.

• Bugfix: Added check for query interruption in a parsing step during query compilation.

Changes in Patch Releases Since 8.1GA

Several features have been added since the GA release of 8.1 (8.1.2). Particularly notable ones include Tracelog Rotation, renaming MPSQL to PSQL, and the 8.1 data_conversion_compatibility_level; however, there are others. Please refer to the 8.1 Release Notes or maintenance release change log for additional details.