10 Release Notes

Full-text Search

• Observability

  • Added the SHOW FULLTEXT INDEX STATUS command and the MV_FULLTEXT_INDEX_STATUS information schema view that show partition-level information about the status of full-text indexes.

  • Added the ANALYZE FULLTEXT command that displays tokens that are generated by a Lucene analyzer for a specified string.

  • Added Lucene logs to the cluster report.

• Performance

  • Added a full-text index merger which combines per-segment full-text indexes to create a cross-segment full-text index which improves the performance of full-text search queries by reducing the number of indexes that must be examined.

  • Added compaction steps to the full-text index merger.

  • Improved full-text index performance.

Iceberg

• Batch history for Iceberg pipelines now records more complete file information for deletes performed by merge pipelines, including filenames for v1 deletes, removed delete files, and v2 delete files. This makes delete activity easier to trace and troubleshoot.

JSON

• Added support for JSON extract shorthand operators (::, ::%, and ::$) to the result of UDFs, which enables key-path access directly to UDF output, for example udf('a')::key.

• Improved performance of certain queries that use JSON builtins for filtering.

• Added support for the JSON data type in external functions.

Observability

• Added the SHOW FULLTEXT INDEX STATUS command and the MV_FULLTEXT_INDEX_STATUS information schema view that provide partition-level visibility into full-text index creation and identify rows associated with broken indexes.

• Added the ANALYZE FULLTEXT command command that displays tokens generated by a Lucene analyzer for a specified string.

• Improved Memory Observability: The new MV_MEMORY_USAGE information schema view enables users to obtain high-level memory usage metrics and identify the set of most significant memory consumers on the system. Refer to Identifying and Reducing Memory Usage for more information.

• Added support for tracing DDL activity, allowing users to audit recent DDL statements across the cluster via Query History and the MV_TRACE_EVENTS information schema view. Refer to Query History for more information.

• Added an in-memory cross query statistics and sampling cache on aggregator nodes that reuses runtime row count statistics and sampling results across related queries to improve cardinality estimates and reduce optimization overhead; time-based invalidation and memory limits are controlled by engine variables.

• Added the DROP ALL FROM cross_query_stats_cache, DROP SAMPLE FROM cross_query_stats_cache, and SHOW cross_query_stats_cache STATUS commands to manage and monitor the cross query statistics cache.

• Added statistics related to AWS S3 lookup latency.

• Added the TLS cipher and application name to the auditlog.

• Added enhanced query statistics for kill query, Recursive Common Table Expressions (RCTE) writer statistics, and sorted iterator error cases.

• Updated profile JSON to report agg_node using the node ID instead of the IP address.

• Added audit log completion records that include success or failure status, error code, OS user, and the number of affected rows.

• Added logout events to audit logging.

• Added logging for security operation duration and the time spent holding the exclusive security manager lock.

Pipelines

• Added the pipeline_source_metadata() built-in function to expose source file metadata when loading data through pipelines.

• Introduced support for tracking pipeline vCPU and memory usage using the activity_tracking_id column.

• Enhanced S3 pipeline glob pattern support to include ? (single-character match), [abc] (character set), [a-z] (character range), and [^abc] (negated set), in addition to the existing * wildcard.

• Added the CONFIG column to the output of the SHOW LINKS command and the LINKS information schema view to display connection link configurations in JSON format.

• Added CREATE OR REPLACE LINK syntax to update existing connection links without dropping and recreating pipelines.

• Added support for loading Parquet nested types (LIST, MAP, and GROUP) into JSON or BSON columns.

• Added support for the OPTIONS 'compact' parameter in INFER PIPELINE for Parquet files to load Parquet groups and nested types (LIST and MAP).

• Improved error reporting for CDC pipelines with detailed messages on multiple parsing errors.

• Added support for skipping NULL records when using SKIP ALL ERRORS or SKIP PARSER ERRORS in the Kafka pipeline definition.

• Improved memory management for pipelines.

• Blocked non-CDC sources in CREATE {TABLE|TABLES} AS INFER PIPELINE statements.

• Added the filename to the error message returned by an INFER PIPELINE statement when an incorrect file format is detected.

• Locked down ALTER PIPELINE queries for KAFKACONNECT source.

• Long-running batches no longer block CREATE OR REPLACE PIPELINE queries. If running, the current batch exits gracefully and the pipeline is replaced.

• Added support for SHOW PROFILE PIPELINE.

• Added support for vended credentials in Iceberg pipelines with S3 storage.

• Added support for metadata garbage collection for Azure pipelines.

• Updated the Kinesis connector.

• Added single-value output mode for Kafka Connect.

• Updated the default output mode for Kafka Connect FORMAT AVRO and FORMAT JSON.

• Stored procedures run within pipelines now inherit the pipeline's resource pool.

• Added support for CloudPubSubSourceConnector short name in Pub/Sub Connector pipelines.

• Added support for Google Cloud Pub/Sub connector in Connector pipelines.

• Added offsets metadata garbage collection for FS pipelines through ENABLE OFFSETS METADATA GC to prevent unbounded growth in pipelines_cursors.

• S3 pipelines can now discover new files through S3 event notifications routed to Kinesis Data Streams.

• Improved Kafka pipelines caching to use a per-partition, per-pipeline scope. This reduces resource contention across pipelines.

Query Execution

• Added support for correlated EXISTS subqueries with HAVING clause on single aggregate.

• Added the ALTER TABLE ... UNDO POPULATION command that enables cancellation of an ongoing columnstore alter operation.

• Added the ALTER LINK command that enables updates to the CONFIG and CREDENTIALS clauses of an existing link.

• Added support for subselects and joins in a writable view definition. Added support for writing to nested views.

• Updated the permissions required for querying writable views that contain joins.

• If normalization is a selectivity optimization source in an EXPLAIN or PROFILE query, the output now displays the source of normalization.

• Improved the query optimizer's handling of out-of-memory issues.

• Improved the performance of queries on MV_CLUSTER_STATUS information schema view.

• Added support for collecting target table DDL for INSERT queries in error reproduction.

• KILLED_QUERIES counter now includes queries killed due to exceeding QUERY_MEMORY_PERCENTAGE.

• Added support for BEGIN, COMMIT, and ROLLBACK statements in DML forwarding when performing writes to reference tables in multi-statement transactions.

• Added support for the IF EXISTS option in the DROP PROJECTION command.

• Extended the TABLE() function to have an ordinality column when used with the WITH ORDINALITY clause.

• Added support for returning either all columns (RETURNING *) or a selected subset of columns in UPDATE ... RETURNING. Stored procedures, CTEs, multi-table updates (joins/subselects), and column expressions are not supported in the RETURNING clause.

• Enhanced JSON handling to correctly process emoji and surrogate pair representations.

• Improved query performance by executing eligible operations directly on encoded columnstore data.

• SHOW PROFILE PIPELINE queries now support pipelines with a . in the pipeline name.

• Added the REQUIRE SUBJECT clause to the CREATE USER command.

• Added the ability to skip heuristic checks when using projection for JSON documents for ORDER BY self join rewrites.

• Added support for ALTER NODE ... SET HOST, PORT which enables updating the internal host and/or port of a node. Refer to Update Internal Host or Port for more information.

• Enabled using estimated table row count as default column cardinality for better join estimation.

• Removed the CAST operator if a redundant cast is performed.

• Added support for increasing the length of variable-length string type columns using ALTER TABLE ... MODIFY COLUMN on columnstore tables.

• Added support for COUNT() in aggregated views.

• Added display of the original line number when re-raising exceptions with the RAISE statement in stored procedures.

• Added support for PSQL ARRAY type arguments to the TO_JSON() built-in function.

• Disabled prewritten ingest transformations by default.

• Added an optimizer rewrite to push LIMIT below external User Defined Functions (UDFs). This optimization can be turned off with the optimizer_disable_limit_pushdown_below_external_udf engine variable.

• Improved JSON_EXTRACT_BIGINT performance by performing encoding-level extraction instead of full JSON decoding.

• Added functionality to copy the rowstore segment by default when a shallow copy of a table is created.

• Updated character_set_client to display the correct character set value in SHOW CREATE queries.

• Added a FILE WRITE privilege check for table-less SELECT ... INTO OUTFILE statements.

• Added guardrail preventing DROP DATABASE, DETACH DATABASE, and DROP FUNCTION operations from removing functions currently set as resource_pool_statement_selector_function.

• Reduced the time the security manager write lock is held on child aggregators during security operations.

• Enabled TABLE() ordinality with JOIN PUSHDOWN optimization.

• Added support for ALTER PIPELINE ... SET OFFSETS on Connector pipelines.

• Disabled calling overloaded functions with named notation.

• Prevented creation of overloaded functions that can lead to ambiguous function calls.

• Improved out-of-memory error handling for overloaded functions and stored procedures.

• DROP DATABASE now completes in bounded time even if a pipeline extractor subprocess fails to exit. In-progress pipeline batches are actively signaled to terminate instead of waiting on a leaked database metadata reference.

• Improved OR filter decomposition for columnstore tables.

• Enabled block elimination for some columnstore index filters.

• Added support for updating table comments using the ALTER TABLE ... COMMENT command.

• Added a new protocol version (PROTOCOL V2) to the external-function handshake, which includes the input parameter types for the function.

• Added support for overloading external functions.

• Enabled Flexible Parallelism in multi-statement transactions.

• Improved short-query response latency by optimizing idle connection sweeps in the connection scheduler.

• Improved error messages for invalid polygons inserted in a GEOGRAPHY type column by including the coordinates involved in ring-crossing cases to easily identify and correct invalid WKT geometries.

• Improved query compilation performance for certain tree query shapes.

• Removed password validation checks for users configured with REQUIRE SUBJECT.

• Improved IN-subselect rewrite handling by not adding ISNULL() checks for non-nullable columns.

• Improved query plan printing efficiency, lowering query compilation overhead.

• Added proper refresh handling for STS tokens during long-running operations.

• Added the ability to avoid group by placement rewrites when grouped tables have no join predicates, improving query plan stability for multi-table aggregates.

• Improved join selectivity estimation for predicates of the form col1 = IFNULL(col2, col3) by leveraging column cardinality statistics.

Security

• Added support for mutual TLS (mTLS) authentication in Kafka pipelines that enables secure two-way authentication between SingleStore and Kafka brokers.

• Added support for TLSv1.3.

• Disabled TLSv1 and TLSv1.1 at the OpenSSL level.

• Users can now revoke permissions from roles only if they currently hold the permission being revoked.

• Added support for granting and revoking role permissions on the target scope without requiring the GLOBAL GRANT OPTION.

Vector Search

• Added support for SIMD implementations of DOT_PRODUCT and EUCLIDEAN_DISTANCE built-in functions for the VECTOR F16 type.

• Added support for typecasts between VECTOR F32 and VECTOR F16 types.

• Backfilled index type and index size of vector index metadata during information schema queries.

• Added a vector index cache to limit the amount of memory used by vector indexes. Refer to Vector Indexing for details.

• Updated vector index and search to use internal memory allocator instead of malloc.

• Updated VECTOR_SUBVECTOR() to support direct VECTOR inputs, including F16, without requiring a BLOB cast.

General Updates

• Added support for displaying the collation for JSON, ENUM, or SET data types in information schema views.

• Engine gRPC calls to the plancache manager now include more information.

• For invalid, NULL, or empty UDF names, DML queries are now routed to the default pool.

• Made Resource Governance aware of cgroup CPU limits in Kubernetes clusters. The minimum of the host core count and cgroup core count is now used as the CPU limit for governing resource pools.

• Added support for considering cgroup cores if they are less than host core count in the resource governor.

• Improved the error message returned when removing a leaf fails during rebalance.

• Setting a resource pool inside a stored procedure does not take effect immediately, and the change is applied only after the stored procedure finishes executing.

• Increased the maximum number of user-defined resource group pools to 50.

• New users created without an explicit resource pool are now automatically assigned the default_pool resource pool.

• Added information on bucket parameters to the gRPC request to the plancache manager.

• Removed password validation for ALTER USER queries that do not modify the password.

• If a shard key is not specified, it is added by default on the GROUP BY columns.

• Allowed users that do not require X509 (mTLS) to connect regardless of the ssl_ca_for_client_cert value and presence of the file it points to (if set).

• Added the ability to ignore join selectivity estimates of zero.

• Added charset validation for JSON type values in external functions.

• Changed the default collation from utf8mb4_general_ci to utf8mb4_bin.

• Delimited batch sets now return and accept original rowIDs.

• Disabled critical_diagnostics by default.

• Engine now displays NULL for vector index size and type for vector indexes created in versions older than 9.0.

• Introduced background IO submission thread for sync durability and batching multiple IO requests.

• Result tables now explicitly set table-level collation.

• Upgraded the AlmaLinux version to AlmaLinux 10.

• Added option to merge credentials with external secret storage.

• Improved error handling for regular expressions; errors are now properly raised instead of returning SQL NULL.

• Password expiration warnings are no longer displayed on every query when a user's password is about to expire.

• Allowed out-of-memory (OOM) threads to exit early when waiting for the garbage collection mechanism to provide memory buffers.

• Upgraded librdkafka to version 2.13.2.

Engine Variables

New engine variables:

• max_user_rg_pools: Allows a user to create a maximum number of 50 resource pools from the default of 10. Refer to Set Resource Limits.

• remove_leaf_force_remove_online_leaf: Controls the behavior of the REMOVE LEAF ... FORCE command when leaf_failure_detection is OFF. When remove_leaf_force_remove_online_leaf is enabled, the REMOVE LEAF ... FORCE command forcefully removes the target leaf node from the cluster even if it is online.

• suppress_dml_warnings: Suppresses all DML query warnings.

• ssl_ca_for_client_cert: Specifies the Certificate Authority (CA) to use for client certification.

• regexp_error_handling_level: Controls the behavior of error handling for regular expression functions.

• enable_overloaded_functions: Enables or disables the ability to overload functions and stored procedures.

• Added the following engine variables to control cross query stats and sampling cache: cross_query_stats_collection_strategy, cross_query_stats_max_wait_ms, cross_query_stats_sampling_valid_time_min, cross_query_stats_expiration_minutes, cross_query_stats_max_mb, enable_cross_query_stats_store_query_text, enable_cross_query_stats_for_sampling, optimizer_use_cross_query_feedback_stats.

• parquet_emit_field_ids: When set to FALSE, suppresses field_id from every column in subsequent SELECT INTO ... FORMAT PARQUET output files.

• skip_sharding_warmup_query: Disables sending single-partition queries to all leaves for warm-up.

• auditlog_completion_records: Enables logging of completion records. Refer to Audit Log File Formats for more information.

• enable_avoid_udf_projection_duplication: When enabled, the optimizer avoids copying user-defined functions during rewrites, reducing repeated UDF invocations and improving workloads that use expensive UDFs.

• dr_cluster_provision_timeout_ms: Used to mitigate cases where REPLICATE DATABASE takes too long to provision the remote cluster's metadata.

• internal_csi_filter_minmax_block_elimination: Enables block elimination for some columnstore hash index filters.

• skip_early_snapshot_after_truncate: Prevents scheduling an early snapshot after a TRUNCATE command.

• bottomless_uploader_s3_part_size: Used to address the 10,000-part limit encountered when uploading files larger than 50GB to bottomless.

• rebalance_retry_any_error: Enables retrying rebalances regardless of the error.

• pipelines_extractor_debug_logging: Enables logging detailed, step-by-step timings for each operation within BATCH_PARTITION_TIME.

• restore_download_max_retries: Controls the number of retry attempts for file downloads from object storage during incremental restore operations.

• allow_nested_leaf_errors: Allows nested "Leaf Error" messages, and disables the new behavior that prevents multiple nested "Leaf Error".

Updates to existing engine variables:

• Increased the maximum value of pipelines_max_concurrent to 4096.

• Updated the default value of cardinality_threshold_for_join_split_rewrite to 9223372036854775807 (LLONG_MAX).

Information Schema

New and updated information schema views and columns:

• PIPELINES_SOURCE_OFFSETS: Tracks source offset information for Connector pipelines in the new pipeline extensibility framework.

• MV_SEQUENCES: Displays status, statistics, and cache metrics for AUTO_INCREMENT AS SEQUENCE columns, including pre-fetch counts, cache misses, and owner leaf information for performance monitoring.

• MV_ACTIVE_METADATA_VERSIONS: Provides visibility into active metadata transaction read versions.

• MV_GC_EVENTS. Provides visibility into garbage collection passes. This view is gated behind the preview feature and enable_gc_events engine variable.

• CROSS_QUERY_STATS_CACHE: Contains metadata about cross query statistics cache entries for the connected aggregator, including hash keys, rowcounts, hit counts, and timing information.

• MV_CROSS_QUERY_STATS_CACHE: Managed view that aggregates cross query statistics cache entries across all aggregators in the cluster, including node-level identifiers and cache metrics.

• MV_FTS_MERGED_INDEXES: Displays the merged full-text search (FTS) metadata.

• Added ATTACH DATABASE, DETACH DATABASE, CREATE DATABASE, and DROP DATABASE to MV_EVENTS information schema view.

• Added the INDEX_NAME column to the VECTOR_INDEX information schema view.

• Added the FUNCTION_NAME column to the MV_PROCESSLIST information schema view. When a query is invoked from a stored procedure, this column displays the stored procedure name.

• Added a new BLOB_COMPUTE_ID column for COLUMNAR_SEGMENTS and MV_COLUMNSTORE_FILES information schema views.

• source_type is now included as part of the config_json in the PIPELINES information schema view.

• Increased the MV_BT timeout from 60 seconds to 180 seconds.

• Added the PLAN_SOURCE column to the PLANCACHE information schema view that displays the source of a query plan.

• Added the CONFIG column to the LINKS information schema view to display connection link configurations in JSON format.

• Added the ACTIVITY_TRACKING_ID column that specifies the ID that can be used to identify the activities associated with a session to the MV_ACTIVITIES, MV_ACTIVITIES_CUMULATIVE, MV_ACTIVITIES_EXTENDED, and MV_ACTIVITIES_EXTENDED_CUMULATIVE information schema views.

• Added the SOURCE_OFFSETS_NEW column to the PIPELINES_BATCHES information schema view.

• Added the BATCH_SOURCE_OFFSETS_POLLED column to the PIPELINES_BATCHES_METADATA and PIPELINES_BATCHES_SUMMARY information schema views.

• Added the EXTERNAL_PROTOCOL_VERSION column to the ROUTINES information schema view to identify the handshake protocol version used by external functions.

• Added information on projections to the COLUMNAR_SEGMENTS, COLUMNS, MV_COLUMNAR_SEGMENT_INDEX, MV_COLUMNSTORE_FILES, TABLES, JSON_COLUMN_SCHEMA, MV_COLUMNSTORE_MERGE_STATUS, STATISTICS, TABLE_CONSTRAINTS, and OPTIMIZER_STATISTICS information schema views.