9.1 Release Notes

Full-text Search

• Observability

  • Added the SHOW FULLTEXT INDEX STATUS command and the MV_FULLTEXT_INDEX_STATUS information schema view that show partition-level information about the status of full-text indexes.

  • Added the ANALYZE FULLTEXT command that displays tokens that would be generated by a Lucene analyzer for a specified string.

• Performance

  • Added a full-text index merger which combines per-segment full-text indexes to create a cross-segment full-text index which improves the performance of full-text search queries by reducing the number of indexes that must be examined.

Iceberg

• Added Hive and REST catalog support for Iceberg external tables.

JSON

• Added support for JSON extract shorthand operators (::, ::%, and ::$) to the result of UDFs, which enables key-path access directly to UDF output, for example udf('a')::key.

• Performance improvements for certain queries that use JSON builtins for filtering.

• Added support for the JSON data type in external functions.

Observability

• Full-text performance and observability were enhanced by the addition of the the SHOW FULLTEXT INDEX STATUS command, the MV_FULLTEXT_INDEX_STATUS information schema view information schema view, and the ANALYZE FULLTEXT command.

• Improved Memory Observability: The new MV_MEMORY_USAGE information schema view provides users a simple way to obtain high level memory usage metrics and a practical way to identify the set of most significant memory consumers on the system. Refer to Identifying and Reducing Memory Usage for more information.

• Added support for tracing DDL activity, allowing users to audit recent DDL statements across the cluster via Query History and the MV_TRACE_EVENTS information schema view. Refer to Query History for more information.

• Enhanced logging for DPC operations.

• Added time based invalidation to cross-query statistics cache.

• Added statistics related to AWS S3 lookup latency.

• Added the TLS cipher and application name to the auditlog.

• Added collection of enhanced query stats from kill query Recursive Common Table Expressions (RCTE) writer statistics and sorted iterator error cases.

Pipelines

• Added the pipeline_source_metadata() built-in function to expose source file metadata when loading data through pipelines.

• Introduced support for tracking pipeline vCPU and memory usage using the activity_tracking_id column.

• Enhanced S3 pipeline glob pattern support to include ? (single-character match), [abc] (character set), [a-z] (character range), and [^abc] (negated set), in addition to the existing * wildcard.

• Added the CONFIG column to the output of SHOW LINKS and the LINKS information schema view to display connection link configurations in JSON format.

• Added CREATE OR REPLACE LINK syntax to update existing connection links without dropping and recreating pipelines.

• Added support for loading Parquet nested types (LIST, MAP, and GROUP) into JSON or BSON columns.

• Added support for the OPTIONS 'compact' parameter in INFER PIPELINE for Parquet files for loading Parquet groups and nested types (LIST and MAP).

• Improved error reporting for CDC pipelines with detailed messages on multiple parsing errors.

• Added support for skipping NULL records when using Kafka properties, skip all errors, or skip parser errors in the Kafka pipeline definition.

• Improved memory management for pipelines.

• Blocked non-CDC sources in CREATE {TABLE|TABLES} AS INFER PIPELINE queries.

• Added the filename to the error message returned by an INFER PIPELINE statement when an incorrect file format is detected.

• Locked down ALTER PIPELINE queries for KAFKACONNECT source.

• Long running batches no longer block CREATE OR REPLACE PIPELINE queries. If running, the current batch exits gracefully and the pipeline is replaced.

• Enabled support for ingestion of parquet repeated and grouped fields, such as lists, maps, or groups, as SingleStore JSON type.

• Added support for SHOW PROFILE PIPELINE.

• Added support for vended credentials in Iceberg pipelines with S3 storage.

• Added support for metadata garbage collection for Azure pipelines.

Query Execution

• The Distributed Plancache (DPC) preview feature has been enhanced with a Plancache Manager (PCM) and asynchronous lookup in hot path execution. These features improve the DPC's usefulness in Fast Scale and Suspend & Resume.

• Added support for DROP MATERIALIZED VIEW.

• Added support for correlated EXISTS subqueries with HAVING clause on single aggregate.

• Added an ALTER TABLE ... UNDO POPULATION command that enables cancellation of an ongoing columnstore alter operation.

• Added the ALTER LINK command that enables updates to the CONFIG and CREDENTIALS clauses of an existing link.

• Added support for subselects and joins in a writable view definition. Added support for writing to nested views.

• Updated the permissions required for querying writable views that contain joins.

• If a selectivity optimization source is normalization in an EXPLAIN or PROFILE query, the output now displays the source of normalization.

• Improved the query optimizer's handling of out-of-memory issues.

• Improved the performance of queries on MV_CLUSTER_STATUS information schema view.

• Added support for collecting target table DDL for INSERT queries in error reproduction.

• KILLED_QUERIES counter now includes queries killed due to exceeding QUERY_MEMORY_PERCENTAGE.

• Added support for BEGIN, COMMIT, and ROLLBACK statements in DML forwarding when performing writes to reference tables in multi-statement transactions.

• Added support for the IF EXISTS option in the DROP PROJECTION command.

• Extended the TABLE() function to have an ordinality column when used with WITH ORDINALITY clause.

• Added support for returning either all columns (RETURNING *) or a selected subset of columns in UPDATE ... RETURNING; stored procedures, CTEs, multi-table updates (joins/subselects), and column expressions are not supported in the RETURNING clause.

• Enhanced JSON handling to correctly process emoji and surrogate pair representations.

• Improved query performance by executing eligible operations directly on encoded columnstore data.

• SHOW PROFILE PIPELINE queries now support pipelines with a . in the pipeline name.

• Added the REQUIRE SUBJECT clause to the CREATE USER command.

• Added the ability to skip heuristic checks when using projection for JSON documents for ORDER BY self join rewrites.

• Added support for an alter node port.

• Enabled the use of column names from the base table for materialized view column names.

• Enabled synchronous Distributed Plancache lookup during query execution.

• Enabled using estimated table row count as default column cardinality for better join estimation.

• Distributed Plancache pool size adjustment fails gracefully.

• Removed the CAST operator if a redundant cast is performed.

• Added support for increasing the length of variable-length string type columns using ALTER TABLE MODIFY COLUMN on columnstore tables.

• Added support for COUNT() in aggregated views.

• Added display of original line number when re-raising exceptions with RAISE statement in stored procedures.

• Added support for PSQL ARRAY type arguments to the TO_JSON built-in function.

• Disabled prewritten ingest transformations by default.

• Added an optimizer rewrite to push LIMIT below external User Defined Functions (UDFs). This optimization can be turned off with the optimizer_disable_limit_pushdown_below_external_udf engine variable.

• Improved JSON_EXTRACT_BIGINT performance by performing encoding-level extraction instead of full JSON decoding.

• Added functionality to copy the rowstore segment by default when a shallow copy of a table is created.

• Updated character_set_client to display the correct character set value in SHOW CREATE queries.

Security

• Added support for mutual TLS (mTLS) authentication in Kafka pipelines that enables secure two-way authentication between SingleStore and Kafka brokers.

Vector Search

• Added support for SIMD implementations of DOT_PRODUCT and EUCLIDEAN_DISTANCE builtins for the VECTOR F16 type.

• Added compaction steps to FTS index merger.

• Added support for typecasts between VECTOR F32 and VECTOR F16 types.

• Added Lucene logs to the cluster report.

• Backfilled index type and index size of vector index metadata during information schema queries.

• Improved performance of full-text index.

• Added a vector index cache to limit the amount of memory used by vector indexes. Refer to Vector Indexing for details.

• Updated vector index and search to use internal memory allocator instead of malloc.

General Updates

• Added support for displaying the collation for JSON, ENUM, or SET data types in information schema views.

• Engine gRPC calls to the plancache manager now include more information.

• For invalid, NULL, or empty UDF names, DML queries are now routed to the default pool.

• Made Resource Governance aware of cgroup CPU limits in Kubernetes clusters. The minimum of host cores and cgroup cores is now used as the CPU limit for governing resource pools.

• The engine now waits for the plancache manager during uploads.

• Added support for considering cgroup cores if they are less than host core count in the resource governor.

• Improved the error message returned when removing a leaf fails during rebalance.

• Setting a resource pool inside a stored procedure does not take effect immediately, and the change is applied only after the stored procedure finishes executing.

• Increased the maximum number of user-defined resource group pools to 50.

• New users created without an explicit resource pool are now automatically assigned the default_pool resource pool.

• Added information on bucket parameters to gRPC request to the plancache manager.

• Removed password validation for ALTER USER queries that do not modify the password.

• CPU resource governance now works for clusters of all sizes.

• If a shard key is not specified, it is added by default on the group by columns.

• Allowed users that do not require X509 (mTLS) to connect regardless of the ssl_ca_for_client_cert value and presence of the file it points to (if set).

• Added the ability to ignore join selectivity estimates of zero.

• Added charset validation for JSON type values in external functions.

• Changed the default collation from utf8mb4_general_ci to utf8mb4_bin.

• Delimited batch sets now return and accept original rowIDs.

• Disabled critical_diagnostics by default.

• Engine now displays NULL for vector index size and type for vector indexes created in versions older than 9.0.

• Introduced background IO submission thread for sync durability and batching multiple IO requests.

• Result tables now explicitly set table-level collation.

• Upgraded the Almalinux version to Almalinux 10.

• Added option to merge credentials with external secret storage.

• Improved error handling for regular expressions; errors are now properly raised instead of returning SQL NULL.

Engine Variables

This update adds the following new engine variables:

• max_user_rg_pools: Allows a user to create a maximum number of 50 resource pools from the default of 10. Refer to Set Resource Limits.

• remove_leaf_force_remove_online_leaf: Controls the behavior of the REMOVE LEAF ... FORCE command when leaf_failure_detection is OFF. When enabled, the REMOVE LEAF ... FORCE command forcefully removes the target leaf node from the cluster even if it is online.

• suppress_dml_warnings: Suppresses all DML query warnings.

• ssl_ca_for_client_cert: Specifies the Certificate Authority (CA) to use for client certification.

• regexp_error_handling_level: Controls the behaviour of error handling for regular expression functions.

• Distributed Plan Cache: Added the following engine variables to control this feature: enable_distributed_plancache, distributed_plancache_worker_threads, distributed_plancache_max_download_plans, distributed_plancache_agg_sync_s, enable_periodic_distributed_plancache_agg_sync, enable_synchronous_dpc_lookup.

• enable_overloaded_functions: Enables or disables the ability to overload functions and stored procedures.

Updates to existing engine variables:

• Increased the maximum value of pipelines_max_concurrent to 4096.

• Updated the default value of cardinality_threshold_for_join_split_rewrite to 9223372036854775807 (LLONG_MAX).

Information Schema

This update adds the following information schema views and columns:

• PIPELINES_SOURCE_OFFSETS:Tracks source offset information for KafkaConnect pipelines in the new pipeline extensibility framework.

• MV_SEQUENCES: Displays status, statistics, and cache metrics for AUTO_INCREMENT AS SEQUENCE columns, including pre-fetch counts, cache misses, and owner leaf information for performance monitoring.

• Added the PLAN_SOURCE column to the PLANCACHE information schema view that displays the source of a query plan.

• Added the CONFIG column to the LINKS information schema view to display connection link configurations in JSON format.

• Added the ACTIVITY_TRACKING_ID column the specifies the ID that can be used to identify the activities associated with a session to the MV_ACTIVITIES, MV_ACTIVITIES_CUMULATIVE, MV_ACTIVITIES_EXTENDED, and MV_ACTIVITIES_EXTENDED_CUMLATIVE information schema views.

• MV_ACTIVE_METADATA_VERSIONS: Provides visibility into active metadata transaction read versions.

• MV_GC_EVENTS. Provides visibility into garbage collection passes. This view is gated behind the preview feature and engine variable enable_gc_events.

• Added ATTACH DATABASE, DETACH DATABASE, CREATE DATABASE, and DROP DATABASE to MV_EVENTS information schema view.

• Added the INDEX_NAME column to the VECTOR_INDEX information schema view.

• Added the FUNCTION_NAME column to the MV_PROCESSLIST information schema view. When a query is invoked from a stored procedure, this column will display the stored procedure name.

• Increased the MV_BT timeout from 60 seconds to 180 seconds.

• Added a new BLOB_COMPUTE_ID column for COLUMNAR_SEGMENTS and MV_COLUMSTORE_FILES information schema views.

• source_type is now included as part of the config_json in the PIPELINES information schema view.

Updates to existing information schema views:

• Added information on projections to the COLUMNAR_SEGMENTS, COLUMNS, MV_COLUMNAR_SEGMENT_INDEX, MV_COLUMNSTORE_FILES, TABLES, JSON_COLUMN_SCHEMA, MV_COLUMNSTORE_MERGE_STATUS, STATISTICS, TABLE_CONSTRAINTS, and OPTIMIZER_STATISTICS information schema views.