# SingleStore Helios Documentation --- # AI SingleStore’s AI Services offer a comprehensive set of features that bring built-in intelligence, seamless AI/ML model integration, and scalable AI-powered workflows directly to SingleStore databases. These features include: * **Database Integration**: AI Services connect directly to SingleStore, eliminating data movement while enabling models and functions to execute with low latency on real-time data. They provide easy integration with SingleStore pipelines and notebooks. * **In-Database Analytics**: AI Services run alongside SQL queries, enabling inference, real-time insights, multi-database integration, and LLM-powered workflows directly in SingleStore. * **Secure Isolation**: AI services provide role-based access control (RBAC) at the database level, securing both data and the associated AI applications. * **Scalability**: AI applications scale automatically based on their workload. * **Monitoring**: Built-in logging and metrics track workload performance, inference latency, and model accuracy for full observability. ## In this section * [Aura Analyst](https://docs.singlestore.com/cloud/ai/aura-analyst.md) * [AI & ML Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions.md) * [Models](https://docs.singlestore.com/cloud/ai/models.md) * [SingleStore MCP Server](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server.md) * [SingleStore AI Rules](https://docs.singlestore.com/cloud/ai/singlestore-ai-rules.md) *** Modified at: August 4, 2026 Source: [/cloud/ai/](https://docs.singlestore.com/cloud/ai/) (An index of the documentation is available at /llms.txt) --- # AI & ML Functions > **📝 Note**: This is a Preview feature. ## Overview AI & ML Functions enable running generative AI models, embedding models, and machine learning (ML) models directly on the data using simple SQL commands in SingleStore. This eliminates the need for data movement, reducing latency, cost, and complexity. AI & ML Functions are delivered as preconfigured, managed Python User-Defined Functions (UDFs) that run in optimized containers within [SingleStore Aura](https://docs.singlestore.com/cloud/container-services.md). This containerized design provides a secure, fully managed execution environment for building and running AI & ML workloads. AI & ML Functions enable SQL-based execution tasks such as sentiment analysis, summarization, and translation without requiring additional frameworks, external API integrations, or complex pipelines. AI & ML Functions run alongside SingleStore Helios, delivering results in milliseconds. Combined with SingleStore’s HTAP architecture, these functions run on real-time operational data as soon as it is created. ## In this section * [AI Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions.md) * [ML Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions.md) * [Shared Embedding Model Service](https://docs.singlestore.com/cloud/ai/ai-ml-functions/shared-embedding-model-service.md) *** Modified at: August 4, 2026 Source: [/cloud/ai/ai-ml-functions/](https://docs.singlestore.com/cloud/ai/ai-ml-functions/) (An index of the documentation is available at /llms.txt) --- # AI Functions ## Install AI Functions To install AI Functions, navigate to **AI > AI & ML Functions**, select the cluster on which to install AI Functions. In the **AI Functions** tab, select **Install**, review the **AI Functions Summary** and then select **Deploy**. Once the AI Functions are installed, query them in the SQL Editor or SingleStore notebooks. SingleStore provides the following AI Functions: | **Category** | **Function** | | --------------------------------------------------------------- | -------------------------- | | Text Processing Functions | `AI_COMPLETE(text, model)` | | `AI_SENTIMENT(text, model)` | | | `AI_TRANSLATE(text, source_languages, target_languages, model)` | | | `AI_SUMMARIZE(text, model, max_lengths)` | | | `AI_CLASSIFY(text, categories, model)` | | | `AI_EXTRACT(text, questions, model)` | | | Embedding Function | | | `EMBED_TEXT(text, model)` | | ## Edit AI Functions To edit AI Functions, navigate to **Settings** in the right navigation of the **AI Functions** tab and select **Edit**. Alternatively, select the ellipsis (vertical three dots) in the right and select **Edit AI Functions**. On the **Edit AI Functions** page, view or select the following: | **ClusterGroup** | Theclustergroup in which the AI Function is installed and running. | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Models** | Default models used by AI Functions for inference and embedding. | | **Settings** | | | **Cost Estimation** | View the cost estimation of LLM Model, Embedding Model, and Compute. In Compute, runtime cost scales with the replica count. | | **Region** | View the region of the following: | Select **Next**, review the changes, and select **Save** to save your settings. ## Uninstall AI Functions To uninstall AI Functions, select the ellipsis (vertical three dots) in the right and then select **Uninstall AI Functions**. Confirm the uninstallation and select **Uninstall**. ## Text Processing Functions ## AI\_COMPLETE Provides batched LLM powered completion of every input text. Used for general purpose text generation, completion, and complex reasoning. ## Syntax ```sql AI_COMPLETE(text, model) ``` ## Arguments * `text`: A prompt. * `model`: An LLM model. ## Return Type `string` ## Usage | Basic usage with the default model | `SELECT cluster.AI_COMPLETE('Life is like a box of') AS completion;` | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Basic usage with a specific model | `SELECT cluster.AI_COMPLETE('Life is like a box of', model => 'anthropic-claude-3-5-haiku') as completion;` | | Input example on database | `SELECT cluster.AI_COMPLETE(column_1) FROM table;` | ## AI\_SENTIMENT Provides sentiment classification and score for all user-defined inputs. ## Syntax ```sql AI_SENTIMENT(text, model) ``` ## Arguments * `text`: A prompt. * `model`: An LLM model. ## Return Type `string` ## Usage | Basic usage with default model | `SELECT cluster.AI_SENTIMENT('The migration tool saved us hours, but the error messages were completely cryptic and we had to open three support tickets just to get past the setup.') AS sentiment;` | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected model | `SELECT cluster.AI_SENTIMENT('The migration tool saved us hours, but the error messages were completely cryptic and we had to open three support tickets just to get past the setup.', 'anthropic-claude-3-5-haiku') as sentiment;` | | Input example on database | `SELECT cluster.AI_SENTIMENT(column_1) FROM table;` | ## AI\_TRANSLATE Provides translation of user-provided documents from source language to target language. Supports multi-language translation. ## Syntax ```sql AI_TRANSLATE(text, source_languages, target_languages, model) ``` ## Arguments * `text`: A prompt. * `source_languages`: The language in which the prompt is written. * `target_languages`: The language to which the prompt gets translated. * `model`: An LLM model. ## Return Type `string` ## Usage | Basic usage with default model | `SELECT cluster.AI_TRANSLATE( '平素より大変お世話になっております。株式会社テクノソリューションズの田中と申します。先日よりご利用いただいておりますデータ移行サービスについて、 重大な問題が発生しておりますため、緊急のご連絡を差し上げております。昨日の午後3時頃より、移行ジョブが途中で停止し、エラーログには「接続タイムアウト」 と記録されております。弊社のシステム管理者が調査いたしましたが、原因の特定には至っておりません。本件は弊社の本番環境に影響を及ぼしており、 業務に支障をきたしております。至急、担当エンジニアよりご連絡いただけますようお願い申し上げます。何卒よろしくお願いいたします。','Japanese','English') AS translated_escalation;` | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected model | `SELECT cluster.AI_TRANSLATE('平素より大変お世話になっております。株式会社テクノソリューションズの田中と申します。 先日よりご利用いただいておりますデータ移行サービスについて、重大な問題が発生しておりますため、緊急のご連絡を差し上げております。 昨日の午後3時頃より、移行ジョブが途中で停止し、エラーログには「接続タイムアウト」と記録されております。弊社のシステム管理者が調査いたしましたが、 原因の特定には至っておりません。本件は弊社の本番環境に影響を及ぼしており、業務に支障をきたしております。至急、 担当エンジニアよりご連絡いただけますようお願い申し上げます。何卒よろしくお願いいたします。','Japanese','English', 'anthropic-claude-3-5-haiku') as translation;` | | Input example on database | `SELECT cluster.AI_TRANSLATE(column_1, "source_language", "target_language") FROM table;` | ## AI\_SUMMARIZE Provides summary of user-provided documents within the specified length. ## Syntax ```sql AI_SUMMARIZE(text, model, max_lengths) ``` ## Arguments * `text`: A prompt. * `model`: An LLM model. * `max_lengths`: Maximum length of the summary. ## Return Type `string` ## Usage | Basic usage with default model and length | `SELECT cluster.AI_SUMMARIZE( 'At 14:32 UTC, our monitoring system detected elevated error rates on the primary ingestion pipeline. The on-call engineer was paged at 14:35. Initial investigation pointed to a misconfigured rate limit introduced during the 14:00 deployment. A rollback was initiated at 14:50 and completed at 15:02. Error rates returned to baseline by 15:05. Total customer-facing impact: approximately 33 minutes of degraded write throughput.') AS incident_one_liner;` | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected model | `SELECT cluster.AI_SUMMARIZE( 'At 14:32 UTC, our monitoring system detected elevated error rates on the primary ingestion pipeline. The on-call engineer was paged at 14:35. Initial investigation pointed to a misconfigured rate limit introduced during the 14:00 deployment. A rollback was initiated at 14:50 and completed at 15:02. Error rates returned to baseline by 15:05. Total customer-facing impact: approximately 33 minutes of degraded write throughput.','anthropic-claude-3-5-haiku',15) AS incident_one_liner;` | | Input example on database | `SELECT cluster.AI_SUMMARIZE(column_1) FROM table;` | ## AI\_CLASSIFY Provides classification of each input text into one of the given categories or labels. ## Syntax ```sql AI_CLASSIFY(text, categories, model) ``` ## Arguments * `text`: A prompt. * `categories`: Categories for classification. * `model`: An LLM model. ## Return Type `string` ## Usage | Basic usage with default model | `SELECT cluster.AI_CLASSIFY( 'Hi, I was charged twice for my subscription this month. I have already checked my payment history and both transactions are showing as completed.', '["billing", "technical_issue", "account_access", "feature_request", "abuse_report"]') AS assigned_queue;` | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected model | `SELECT cluster.AI_CLASSIFY( 'Hi, I was charged twice for my subscription this month. I have already checked my payment history and both transactions are showing as completed.', '["billing", "technical_issue", "account_access", "feature_request", "abuse_report"]','anthropic-claude-3-5-haiku') AS assigned_queue;` | | Input example on database | `SELECT cluster.AI_CLASSIFY(column_1, categories) FROM table;` | ## AI\_EXTRACT Extracts information from a block or text based on the specified natural language question. ## Syntax ```sql AI_EXTRACT(text, questions, model) ``` ## Arguments * `text`: A prompt. * `questions`: Input natural language question on which the LLM model extracts information. * `model`: An LLM model. ## Return Type `string` ## Usage | Basic usage with default model | `SELECT cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'What is the contract expiration date?' ) AS expiration_date, cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'What is the total contract value?' ) AS contract_value, cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'How many days notice is required for termination?' ) AS termination_notice;` | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected model | `SELECT cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'What is the contract expiration date?' ) AS expiration_date, cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'What is the total contract value?' ) AS contract_value, cluster.AI_EXTRACT( 'This Service Agreement is entered into on March 1, 2025 between Acme Corp and DataStream Inc. The total contract value is $240,000 USD, payable in quarterly installments. The agreement expires on February 28, 2027, with a 60-day written notice required for termination.', 'How many days notice is required for termination?', 'anthropic-claude-3-5-haiku' ) AS termination_notice;` | | Input example on database | `SELECT cluster.AI_EXTRACT(column_1, question) FROM table;` | ## Embedding Function ## EMBED\_TEXT Provides batched embeddings of all input text. Converts text into high-dimensional vector embeddings for semantic search and RAG applications. ## Syntax ```sql EMBED_TEXT(text, model) ``` ## Arguments * `text`: A prompt. * `model`: An embedding model. ## Return Type `bytes` ## Usage | Basic usage with default model | `SELECT cluster.EMBED_TEXT('Lightweight running shoes with breathable mesh fabric, cushioned soles, and flexible support designed for daily workouts and long-distance comfort.') AS embedding;` | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with selected embedding model | `SELECT cluster.EMBED_TEXT('Lightweight running shoes with breathable mesh fabric, cushioned soles, and flexible support designed for daily workouts and long-distance comfort.', 'shared-qwen3-embed-0-6b') as embedding;` | | Input example on database | `SELECT cluster.EMBED_TEXT(column_1) FROM table;` | ## Examples The following examples demonstrate how to use AI functions with the following `customer_reviews` table. ```sql CREATE DATABASE reviews; USE reviews; CREATE TABLE customer_reviews ( review_id INT PRIMARY KEY, product_id VARCHAR(50), product_name VARCHAR(255), customer_name VARCHAR(100), review_text TEXT, rating INT, review_date DATETIME, language VARCHAR(20) DEFAULT 'English', response_text TEXT, review_embedding BLOB ); -- Insert data INSERT INTO customer_reviews ( review_id, product_id, product_name, customer_name, review_text, rating, review_date, language ) VALUES (1, 'PROD-101', 'Wireless Headphones', 'John Smith', 'These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. Highly recommend for anyone looking for quality audio.', 5, '2026-05-01', 'English'), (2, 'PROD-101', 'Wireless Headphones', 'Maria Garcia', 'Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful.', 1, '2026-05-02', 'English'), (3, 'PROD-102', 'Smart Watch', 'David Lee', 'Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price.', 4, '2026-05-03', 'English'), (4, 'PROD-101', 'Wireless Headphones', 'Sophie Martin', 'Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs.', 5, '2026-05-04', 'French'), (5, 'PROD-103', 'Laptop Stand', 'Ahmed Hassan', 'Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny.', 5, '2026-05-05', 'English'); ``` ## Generate Follow-up Questions for Negative Reviews The following example uses the `AI_COMPLETE` function to generate follow-up questions for negative reviews: ```sql SELECT review_id, customer_name, product_name, review_text, cluster.AI_COMPLETE( CONCAT( 'Based on this negative review: "', review_text, '", generate three specific follow-up questions to better understand the issue.' ) ) AS follow_up_questions FROM customer_reviews WHERE rating <= 2; ``` ```output +-----------+----------------+----------------------+--------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | review_id | customer_name | product_name | review_text | follow_up_questions | +-----------+----------------+----------------------+--------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 2 | Maria Garcia | Wireless Headphones | Disappointed with the build quality. They broke after just | 1. Can you describe exactly how the product broke and in what specific way? | | | | | two weeks of normal use. Customer service was unhelpful. | 2. Did you contact customer service through phone, email, or another method, and what precisely did they say that made you feel they were unhelpful? | | | | | | 3. What type of normal use were you subjecting the product to when it broke? | +-----------+----------------+----------------------+--------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Compare Customer Ratings with AI Sentiment The following example uses the `AI_SENTIMENT` function to compare customer ratings with AI-generated sentiment analysis: ```sql SELECT review_id, customer_name, product_name, rating AS star_rating, review_text, cluster.AI_SENTIMENT(review_text) AS ai_sentiment FROM customer_reviews ORDER BY review_date DESC; ``` ```output +-----------+----------------+----------------------+-------------+---------------------------------------------------------------------------------------------------------------------+----------------------------------------------+ | review_id | customer_name | product_name | star_rating | review_text | ai_sentiment | +-----------+----------------+----------------------+-------------+---------------------------------------------------------------------------------------------------------------------+----------------------------------------------+ | 3 | David Lee | Smart Watch | 4 | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | {'sentiment': 'neutral', 'score': '0.5'} | | 1 | John Smith | Wireless Headphones | 5 | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. | {'sentiment': 'positive', 'score': '0.9'} | | 5 | Ahmed Hassan | Laptop Stand | 5 | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | {'sentiment': 'positive', 'score': '0.95'} | | 4 | Sophie Martin | Wireless Headphones | 5 | Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs. | {'sentiment': 'positive', 'score': '0.90'} | | 2 | Maria Garcia | Wireless Headphones | 1 | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful. | {'sentiment': 'negative', 'score': '0.85'} | +-----------+----------------+----------------------+-------------+---------------------------------------------------------------------------------------------------------------------+----------------------------------------------+ ``` ## Standardize Reviews in English The following example uses the `AI_TRANSLATE` function to standardize the reviews in English language: ```sql SELECT review_id, customer_name, product_name, language AS original_language, review_text AS original_review, CASE WHEN language = 'English' THEN review_text ELSE cluster.AI_TRANSLATE( review_text, language, 'English' ) END AS standardized_review_english FROM customer_reviews ORDER BY review_date DESC; ``` ```output +-----------+----------------+----------------------+-------------------+---------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+ | review_id | customer_name | product_name | original_language | original_review | standardized_review_english | +-----------+----------------+----------------------+-------------------+---------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+ | 3 | David Lee | Smart Watch | English | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | | 1 | John Smith | Wireless Headphones | English | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. | | 5 | Ahmed Hassan | Laptop Stand | English | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | | 4 | Sophie Martin | Wireless Headphones | French | Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs. | Excellent sound quality! I use them every day for work and leisure. | | 2 | Maria Garcia | Wireless Headphones | English | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful. | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful. | +-----------+----------------+----------------------+-------------------+---------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------+ ``` ## Generate Product-Level Summaries The following example uses the `AI_SUMMARIZE` function to generate product-level summaries from customer reviews: ```sql SELECT product_id, product_name, COUNT(*) AS review_count, AVG(rating) AS avg_rating, cluster.AI_SUMMARIZE( GROUP_CONCAT(review_text SEPARATOR '. ') ) AS product_summary FROM customer_reviews GROUP BY product_id, product_name; ``` ```output +------------+----------------------+--------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | product_id | product_name | review_count | avg_rating | product_summary | +------------+----------------------+--------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | PROD-103 | Laptop Stand | 1 | 5.0000 | Excellent ergonomic home office solution with sturdy construction and adjustable height, providing good value. | | PROD-101 | Wireless Headphones | 3 | 3.6667 | Mixed reviews for headphones: praised for excellent sound quality and long battery life, but criticized for poor durability and potential customer service issues.| | PROD-102 | Smart Watch | 1 | 4.0000 | Fitness tracker offers good value, though battery life is shorter than expected. | +------------+----------------------+--------------+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Classify Review Intent The following example uses the `AI_CLASSIFY` function to classify the review intent: ```sql SELECT review_id, customer_name, product_name, rating, review_text, cluster.AI_CLASSIFY( review_text, '[praise, complaint, suggestion, question, comparison]' ) AS review_intent FROM customer_reviews ORDER BY review_date DESC; ``` ```output +-----------+----------------+----------------------+--------+---------------------------------------------------------------------------------------------------------------------+---------------+ | review_id | customer_name | product_name | rating | review_text | review_intent | +-----------+----------------+----------------------+--------+---------------------------------------------------------------------------------------------------------------------+---------------+ | 3 | David Lee | Smart Watch | 4 | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | complaint | | 1 | John Smith | Wireless Headphones | 5 | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. | praise | | 5 | Ahmed Hassan | Laptop Stand | 5 | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | praise | | 4 | Sophie Martin | Wireless Headphones | 5 | Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs. | praise | | 2 | Maria Garcia | Wireless Headphones | 1 | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful. | complaint | +-----------+----------------+----------------------+--------+---------------------------------------------------------------------------------------------------------------------+---------------+ ``` ## Extract Multiple Insights The following example uses the `AI_EXTRACT` function to extract multiple insights from customer reviews: ```sql SELECT review_id, product_name, review_text, cluster.AI_EXTRACT( review_text, 'What product features are mentioned?' ) AS features_mentioned, cluster.AI_EXTRACT( review_text, 'How long has the customer used this product?' ) AS usage_duration FROM customer_reviews; ``` ```output +-----------+----------------------+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------+-----------------------------+ | review_id | product_name | review_text | features_mentioned | usage_duration | +-----------+----------------------+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------+-----------------------------+ | 5 | Laptop Stand | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | Sturdy construction, adjustable height | Not specified in the review | | 2 | Wireless Headphones | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service | Build quality issues, broke after two weeks was unhelpful. | Two weeks | | 4 | Wireless Headphones | Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs. | Sound quality | Daily (ongoing) | | 3 | Smart Watch | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | Battery life (shorter than advertised), good value for price. | Not specified in the review | | 1 | Wireless Headphones | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. | Sound quality (crystal clear), long battery life | Not specified in the review | +-----------+----------------------+--------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------+-----------------------------+ ``` ## Perform Semantic Search The following example uses the `EMBED_TEXT` function to perform semantic search on customer reviews. ```sql SET batch_external_functions = AUTO; SELECT review_id, product_name, review_text, cluster.EMBED_TEXT(review_text) AS review_embedding FROM customer_reviews; UPDATE customer_reviews SET review_embedding = cluster.EMBED_TEXT(review_text) WHERE review_embedding IS NULL; WITH search_query AS ( SELECT cluster.EMBED_TEXT( 'battery problems and short lifespan' ) AS query_embedding ) SELECT cr.review_id, cr.product_name, cr.review_text, cr.rating, DOT_PRODUCT( cr.review_embedding, sq.query_embedding ) AS relevance_score FROM customer_reviews cr, search_query sq WHERE cr.review_embedding IS NOT NULL ORDER BY relevance_score DESC; ``` ```output +-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-----------------+ | review_id | product_name | review_text | rating | relevance_score | +-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-----------------+ | 3 | Smart Watch | Great fitness tracker, but the battery life is shorter than advertised. Overall, good value for the price. | 4 | NULL | | 1 | Wireless Headphones | These headphones are amazing! The sound quality is crystal clear and the battery lasts for days. Highly recommend for anyone looking for quality audio.| 5 | NULL | | 2 | Wireless Headphones | Disappointed with the build quality. They broke after just two weeks of normal use. Customer service was unhelpful. | 1 | NULL | | 4 | Wireless Headphones | Excellente qualité sonore! Je les utilise tous les jours pour le travail et les loisirs. | 5 | NULL | | 5 | Laptop Stand | Perfect ergonomic solution for my home office. Sturdy construction and adjustable height. Worth every penny. | 5 | NULL | +-----------+---------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------+--------+-----------------+ ``` ## Generate Embeddings at Load Time Using a Pipeline The following example uses a pipeline and the `EMBED_TEXT` AI function to generate embeddings for new customer reviews loaded from the AWS S3 bucket. The pipeline loads rows into a staging table and an `UPDATE … JOIN` statement then populates `review_embedding` in batches instead of one row at a time. Create a staging table that mirrors `customer_reviews` without the embedding column: ```sql DROP TABLE IF EXISTS customer_reviews_staging; CREATE TABLE customer_reviews_staging ( review_id INT PRIMARY KEY, product_id VARCHAR(50), product_name VARCHAR(255), customer_name VARCHAR(100), review_text TEXT, rating INT, review_date DATETIME, language VARCHAR(20) DEFAULT 'English' ); ``` Create a pipeline named `customer_reviews_pipeline` and start the pipeline: ```sql CREATE PIPELINE customer_reviews_pipeline AS LOAD DATA S3 's3://singlestore-docs-example-datasets/reviews/customer_reviews.csv' SKIP DUPLICATE KEY ERRORS INTO TABLE customer_reviews_staging FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES; START PIPELINE customer_reviews_pipeline; ``` Insert the new rows into `customer_reviews`, then generate embeddings in a single batched operation: ```sql INSERT INTO customer_reviews (review_id, product_id, product_name, customer_name, review_text, rating, review_date, language) SELECT s.review_id, s.product_id, s.product_name, s.customer_name, s.review_text, s.rating, s.review_date, s.language FROM customer_reviews_staging s LEFT JOIN customer_reviews r ON r.review_id = s.review_id WHERE r.review_id IS NULL; CREATE TEMPORARY TABLE review_embeddings_scratch AS SELECT review_id, cluster.EMBED_TEXT(review_text) AS review_embedding FROM customer_reviews WHERE review_embedding IS NULL; UPDATE customer_reviews r JOIN review_embeddings_scratch s ON r.review_id = s.review_id SET r.review_embedding = s.review_embedding; DROP TABLE review_embeddings_scratch; ``` Calling `EMBED_TEXT` inside a `SELECT` allows the engine to send rows to the embedding model in batches (default `512` rows per request). The `UPDATE … JOIN` statement then writes the embeddings back to `customer_reviews` in a single set-oriented statement. Verify the results: ```sql SELECT review_id, product_name, LEFT(review_text, 40) AS review_preview, LENGTH(review_embedding) AS embedding_bytes FROM customer_reviews WHERE review_id BETWEEN 6 AND 10 ORDER BY review_id; ``` ```output +-----------+----------------------+--------------------------------------------------------------------+-----------------+ | review_id | product_name | review_preview | embedding_bytes | +-----------+----------------------+--------------------------------------------------------------------+-----------------+ | 6 | Smart Watch | Love the sleep tracking and heart rate a | 8192 | | 7 | Laptop Stand | Decent stand but the height adjustment i | 8192 | | 8 | Wireless Headphones | ノイズキャンセリングがとても効果的で、通勤中の電車でも音楽がクリアに聞こえます。 | 8192 | | 9 | Mechanical Keyboard | Die Tastatur fühlt sich hochwertig an, a | 8192 | | 10 | Smart Watch | Battery dies in under a day with GPS on. | 8192 | +-----------+----------------------+--------------------------------------------------------------------+-----------------+ ``` ## Generate Embeddings at Load Time Using a Pipeline and Stored Procedure The following example uses a pipeline that calls a stored procedure to generate embeddings on each batch as it arrives. The stored procedure inserts the batch into `customer_reviews` with embeddings generated inline using `EMBED_TEXT` in a singles-set oriented operation. Create the stored procedure named `load_customer_reviews_with_embeddings`: ```sql DELIMITER // CREATE OR REPLACE PROCEDURE load_customer_reviews_with_embeddings( batch QUERY( review_id INT, product_id VARCHAR(50), product_name VARCHAR(255), customer_name VARCHAR(100), review_text TEXT, rating INT, review_date DATETIME, language VARCHAR(20) ) ) AS BEGIN INSERT INTO customer_reviews (review_id, product_id, product_name, customer_name, review_text, rating, review_date, language, review_embedding) SELECT review_id, product_id, product_name, customer_name, review_text, rating, review_date, language, cluster.EMBED_TEST(review_text) FROM batch; END // DELIMITER ; ``` Create a pipeline named `customer_reviews_sp_pipeline` and start the pipeline: ```sql CREATE OR REPLACE PIPELINE customer_reviews_sp_pipeline AS LOAD DATA S3 's3://singlestore-docs-example-datasets/reviews/customer_reviews.csv' BATCH_INTERVAL 2500 INTO PROCEDURE load_customer_reviews_with_embeddings FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES START PIPELINE customer_reviews_sp_pipeline; ``` Verify that the pipeline loaded the rows and populated the embeddings: ```sql SELECT review_id, product_name, LEFT(review_text, 40) AS review_preview, LENGTH(review_embedding) AS embedding_bytes FROM customer_reviews WHERE review_id BETWEEN 6 AND 10 ORDER BY review_id; ``` ```output +-----------+----------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------+ | review_id | product_name | review_text | embedding_bytes | +-----------+----------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------+ | 6 | Smart Watch | Love the sleep tracking and heart rate accuracy. The strap is comfortable enough to wear overnight without irritation. | 8192 | | 7 | Laptop Stand | Decent stand but the height adjustment is wobbly under heavier laptops. Works fine for an ultrabook. | 8192 | | 8 | Wireless Headphones | ノイズキャンセリングがとても効果的で、通勤中の電車でも音楽がクリアに聞こえます。 | 8192 | | 9 | Mechanical Keyboard | Die Tastatur fühlt sich hochwertig an, aber die Beleuchtung ist zu hell und lässt sich nicht ausschalten. | 8192 | | 10 | Smart Watch | Battery dies in under a day with GPS on. Returning it. The app is also buggy on Android. | 8192 | +-----------+----------------------+--------------------------------------------------------------------------------------------------------------------------+-----------------+ ``` Each pipeline batch invokes the stored procedure with up to `512` rows. `EMBED_TEXT` is called inside a `SELECT` so the rows are sent to the embedding model as a single batched request, and the `UPDATE … JOIN` statement writes the results back in one set-oriented statement. ## Complete Customer Insights The following example uses the `AI_SUMMARIZE` and `AI_EXTRACT` functions to generate complete customer insights for each product: ```sql SELECT product_id, product_name, COUNT(*) AS total_reviews, AVG(rating) AS avg_rating, cluster.AI_SUMMARIZE( GROUP_CONCAT(review_text SEPARATOR '. ') ) AS overview, cluster.AI_EXTRACT( GROUP_CONCAT(review_text SEPARATOR '. '), 'What are the top three issues customers mention?' ) AS top_issues, cluster.AI_EXTRACT( GROUP_CONCAT(review_text SEPARATOR '. '), 'What do customers love most about this product?' ) AS top_strengths FROM customer_reviews GROUP BY product_id, product_name; ``` ```output +------------+----------------------+---------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+---------------------------------------------+ | product_id | product_name | total_reviews | avg_rating | overview | top_issues | top_strengths | +------------+----------------------+---------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+---------------------------------------------+ | PROD-103 | Laptop Stand | 1 | 5.0000 | Excellent ergonomic home office solution with sturdy construction and adjustable height, providing good value. | 1. Sturdy construction 2. Adjustable height 3. Good value for money | Sturdy construction and adjustable height | | PROD-101 | Wireless Headphones | 3 | 3.6667 | Mixed reviews for headphones: praised for excellent sound quality and long battery life, but criticized for poor durability and potential customer service issues. | 1. Sound quality 2. Battery life 3. Build quality (durability) | Sound quality and battery life | | | | PROD-102 | Smart Watch | 1 | 4.0000 | Fitness tracker offers good value, though battery life is shorter than expected. | 1. Battery life 2. Price 3. Functionality | Good value for the price | +------------+----------------------+---------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------+---------------------------------------------+ ``` ## Example Notebook The following notebook demonstrates AI Functions: ## Usage Recommendations for AI Functions To optimize performance and control costs when using AI Functions, SingleStore recommends the following: * Use Common Table Expressions (CTEs) to filter rows before making calls to large language models (LLMs). The query engine currently sends data to the LLM before applying LLM or `WHERE` filters. * LLM calls are expensive. Begin with a small dataset to evaluate response quality and verify the results meet the requirements before scaling up. * Enterprise plans support three model providers; Aura, Amazon Bedrock, and Azure AI Services. Data is processed according to each provider’s policies. If a row violates provider rules, the system fails the batch that includes the row and returns errors for these rows. * Strict usage quotas apply per model and per organization. These quotas are not configurable by end users. For higher usage limits, contact [SingleStore Support](https://support.singlestore.com/). Self-service quota configuration will be available in the future. *** Modified at: July 9, 2026 Source: [/cloud/ai/ai-ml-functions/ai-functions/](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions/) (An index of the documentation is available at /llms.txt) --- # ML Functions ## Overview Machine learning (ML) functions enable trained models to be run directly within SQL queries. They support real-time classification of new data and detection of anomalies without requiring custom code. These functions allow predictions to be embedded directly into workflows that operationalize insights at the data layer. ## Introduction to Machine Learning Machine learning is a field of study in artificial intelligence that develops and applies methods for learning patterns from historical data and using those patterns to make predictions or decisions on new data. Unlike rule-based systems, ML models adapt automatically based on the data. Common use cases include fraud detection, predictive maintenance, and customer behavior analysis. Once trained, models can be deployed and invoked using ML functions to generate predictions at scale. ## Classification Classification is a supervised learning technique that assigns each input to one of a predefined set of classes or labels. Models are trained on labeled datasets, where each input is paired with its correct output, and then used to classify new data. For example, a model may predict whether an incoming email is "spam" or "not spam" based on its content and metadata. Following are the types of classification: * **Binary Classification**: Predicts one of two possible outcomes (for example, fraud vs. non-fraud). * **Multiclass Classification**: Predicts one label from multiple possible categories (for example, product type A, B, or C). * **Multilabel Classification**: Assigns multiple labels to a single data point (for example, tagging an image with "beach" and "sunset"). Examples: * A credit-card transaction classified as “fraudulent” or “legitimate.” * Customer support tickets categorized as “billing,” “technical issue,” or “account upgrade.” The `ML_CLASSIFY` is a supervised machine learning function for classification tasks. It supports both binary (two classes) and multi-class (more than two classes) classification. It leverages algorithms such as logistic regression, random forest, and gradient boosting. Use SQL queries to call `ML_CLASSIFY` function and return predicted class labels. ## Anomaly Detection Anomaly detection identifies data points that deviate significantly from expected patterns. Anomalies signal critical issues such as fraud, equipment failure, or network intrusions. An anomaly is any value or pattern that does not match normal behavior. Anomalies indicate the following: * Performance issues (for example, server overload) * System faults (for example, failed jobs or memory leaks) * Opportunities (for example, traffic spikes caused by a marketing campaign) For example, if a cluster's CPU usage normally stays between 20–60% and suddenly rises to 95%, the spike is an anomaly. ## Time-Series Anomaly Detection Time-series anomaly detection analyzes data collected over time. For example, CPU or memory utilization per minute or hour. It considers not only individual values, but also the sequence and patterns in the data. It learns seasonal patterns (daily or weekly), long‑term trends, and normal variability ranges. The system flags values that deviate from these learned patterns. Following are the types of time-series anomaly detection: * **Supervised**: Supervised models use labeled anomalies to learn failure patterns. * **Unsupervised**: Unsupervised models learn normal behavior from historical data and flag deviations without labels. The `ML_ANOMALY_DETECT` function is an unsupervised time-series anomaly detection function currently. It supports statistical methods (e.g., z-score, interquartile range) and machine learning methods (e.g., Isolation Forest, One-Class SVM). This function returns a prediction for each row, identifying it as normal or anomalous, which can trigger alerts or be recorded for further analysis. ## Install ML Functions To install ML Functions, navigate to **AI > AI & ML Functions**, select the deployment on which to install ML Functions. In the **ML Functions** tab, select **Install**, review the **ML Functions Summary** and then select **Deploy**. Once the ML Functions are installed, query them in the SQL Editor or SingleStore Notebooks. SingleStore provides the following ML Functions: | **Category** | **Function** | | --------------------------------------------------------- | --------------------------------------------------- | | Statistical and Predictive Functions | `ML_CLASSIFY(model_name, TO_JSON(selected_data.*))` | | `ML_ANOMALY_DETECT(model_name, TO_JSON(selected_data.*))` | | ## Statistical and Predictive Functions ## ML\_CLASSIFY Performs binary and multi-class classification on a dataset using standard machine learning algorithms. Supports common algorithms including: * Logistic Regression * Random Forest * Gradient Boosting ## Syntax ```sql ML_CLASSIFY(model_name, TO_JSON(selected_data.*)) ``` ## Arguments * `model_name`: Name of the trained ML model to use. * `selected_data`: A row or set of rows selected for prediction. ## Return Type `string` ## Usage | Basic usage | `SELECT cluster.ML_CLASSIFY(model_name, TO_JSON(selected_data.*)) AS predictions FROM (SELECT * FROM table) AS selected_data;` | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with`LIMIT` | `SELECT cluster.ML_CLASSIFY(model_name, TO_JSON(selected_data.*)) AS predictions FROM (SELECT * FROM table WHERE column1 > 100000LIMIT 100) AS selected_data;` | | Insert predictions into a table | `INSERT INTO predictions_table (id, prediction); SELECT selected_data.id, cluster.ML_CLASSIFY(model_name, TO_JSON(selected_data.*)) AS prediction FROM (SELECT * FROM table LIMIT 100) AS selected_data;` | ## ML\_ANOMALY\_DETECT Detects outliers and anomalies in datasets using statistical or machine learning-based methods. Suitable for security, monitoring, and anomaly detection applications. Supports the following methods: * **Statistical**: z-score, interquartile range (IQR) * **ML-based**: Isolation Forest, One-Class SVM ## Syntax ```sql ML_ANOMALY_DETECT(model_name, TO_JSON(selected_data.*)) ``` ## Arguments * `model_name`: Name of the trained ML model to use. * `selected_data`: A row or set of rows selected for prediction. ## Return Type `string` ## Usage | Basic usage | `SELECT cluster.ML_ANOMALY_DETECT(model_name, TO_JSON(selected_data.*)) AS predictions FROM (SELECT * FROM table) AS selected_data;` | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Basic usage with`LIMIT` | `SELECT cluster.ML_ANOMALY_DETECT(model_name, TO_JSON(selected_data.*)) AS predictions FROM (SELECT * FROM tableWHERE column1 > 100000LIMIT 100) AS selected_data;` | | Insert predictions into a table | `INSERT INTO predictions_table (id, prediction); SELECT selected_data.id, cluster.ML_ANOMALY_DETECT(model_name, TO_JSON(selected_data.*)) AS prediction FROM (SELECT * FROM table LIMIT 100) AS selected_data;` | ## Train a New ML Model To train a new ML model, follow these steps: 1. Navigate to **AI > Models**. 2. Select **ML Models** tab and then select **Train New ML Model**. 3. In the **Select Function** dialog, select one of the following ML functions: * `ML_CLASSIFY` * `ML_ANOMALY_DETECT` Select **Next** to configure the model. **Configure Model** | **Model Name** | Enter the name of the ML model. | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Training Description** | Enter the training description. | | **Cluster** | Select theSingleStoredeployment (cluster) the notebook connects to.Specifying aclusterallows natively connecting theSingleStoredatabases referenced in the notebook. | | **Compute Size** | Select one of the following compute sizes: | | **Run as** | Run the notebook for training a model with or without personal credentials. Select one of the following: | Select **Next**. **Select Training Data** | **Database** | Select the database that contains the training data. | | -------------------------- | -------------------------------------------------------------------------------- | | **Table** | Select the table from the selected database to train the machine learning model. | | **Target Column** | Select the column that represents the prediction target for the model. | | **Feature Selection Mode** | Specify how feature columns are selected. | | **Feature Column** | Select one or more columns to be used as input features for training the model. | Preview the data and select **Next**. Review the **Summary** and generated Fusion SQL syntax in the **Generated SQL Script**. The generated script performs the following: * Creates and trains a ML model * Uses data from the selected table in the selected database * Predicts values of target column status * Runs on the selected compute instance * Uses all available features by default Following is the syntax of Fusion SQL script: ```sql %s2ml train --model --db --input_table --target_column --description --runtime --selected_features { \"mode\": , \"features\": } ``` Select **Start Training** to train the ML model. ## Example Notebooks The following notebooks demonstrates how to use ML Functions: ## ML Functions: Classification ## ML Functions: Anomaly Detection ## Manage an Existing ML Model Existing ML models can be managed by performing the following actions: * View details * Run prediction * Share * Delete ## View Details of an Existing ML Model To view details of an existing ML model, select the ellipsis under **Actions** column of the trained ML model, and select **View Details**. Alternatively, select the ML model in the **Name** column. Select the **Details** tab to view training status, training configuration, training logs, and details about how to use the ML model. ## Run Prediction on an Existing ML Model Run batch prediction on the existing ML model. ## Run a Batch Prediction To run a batch prediction on the existing ML model, select the ellipsis under **Actions** column of the trained ML model, and select **Run Prediction**. **Select Prediction Data** | **Database** | Select the database. | | -------------------- | ------------------------------------------------------------------------------ | | **Target Table** | Select the target table on which the prediction will be run. | | **Target Column** | Select the target column on which the prediction will focus on. | | **Timestamp Column** | Select the column having timestamp data. Available for`ML_ANOMALY_DETECT`only. | Preview the data and select **Next**. **Configure Destination** | **Prediction Interval Width** | Select the interval width of prediction. Available for`ML_ANOMALY_DETECT`only. | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Destination Table Name** | Select the destination table in which the prediction results will be stored. | | **Destination Column** | Select the destination column in which the prediction data will be saved. | | **Run as** | Run the notebook for training a model with or without personal credentials. Select one of the following:
  • Run as <username>: Runs the notebook using the permissions and access of the current user account.
  • Run as a Service Account: Runs the notebook independently of personal credentials, using a service account.

    📝 Note:

    Service accounts are only created by Admin.

| Review the **Summary** and generated Fusion SQL syntax in the **Generated SQL Script**. Select **Start Prediction** to run batch prediction on the trained ML model. ## View Predictions of an Existing ML Model To view the predictions of the trained ML model, select the ML model in the **Name** column. Select the **Predictions** tab to view prediction metadata and status. ## Share an Existing ML Model To share an existing ML model, select the ellipsis under the **Actions** column of the trained ML model, and select **Share**. ## Delete an Existing ML Model To delete an existing ML model, select the ellipsis under **Actions** column of the trained ML model, and select **Delete**. ## Status of ML Models | **Status** | **Description** | | -------------- | --------------------------------------------------------------------------------------------- | | Pre-processing | The system is preparing data for ML model training (e.g., data cleaning, feature extraction). | | Training | The ML model is currently being trained but results are not yet available. | | Done | The ML model has been successfully trained and is ready for use. | | Error | The ML model training or processing failed due to an error. | ## In this section * [ML Functions Release Notes](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions/ml-functions-release-notes.md) *** Modified at: July 9, 2026 Source: [/cloud/ai/ai-ml-functions/ml-functions/](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions/) (An index of the documentation is available at /llms.txt) --- # Shared Embedding Model Service The Shared Embedding Model Service is a multi-tenant embedding service that provides a shared embedding model for AI Functions. Unlike the dedicated embedding service provisioned per user, this shared service allows multiple users and organizations to use the same embedding infrastructure without provisioning individual resources. ## Overview When AI Functions are installed, a shared embedding model is automatically provisioned. This model runs on SingleStore Aura container service and is available to all organizations with AI Functions enabled. The shared embedding model is powered by Qwen3-Embedding-0.6B, a lightweight yet high-quality embedding model optimized for text vectorization, semantic search, and RAG applications. The shared service is currently available in US East (N. Virginia). | Property | Value | | ------------ | -------------------------- | | Model name | `shared-qwen3-embed-0-6b` | | Model family | Qwen3 Embedding | | Dimensions | 1024 | | Region | US East (N. Virginia) | | Hosting | Aura-hosted (multi-tenant) | ## How It Works The shared embedding model service differs from dedicated embedding models in the following ways: | | Shared Embedding Model | Dedicated Embedding Model | | -------------- | ----------------------------------------- | ----------------------------------- | | Infrastructure | Multi-tenant, shared across organizations | Single-tenant, provisioned per user | | Provisioning | Automatic during AI Functions install | Manual setup required | | Scaling | SingleStore-managed | User-configurable | | Model | Qwen3-Embedding-0.6B | User's choice | The shared embedding service runs on GPU-accelerated infrastructure and automatically scales based on demand. SingleStore manages the lifecycle, scaling, and availability of the service. ## Prerequisites * A SingleStore Helios cluster group with AI Functions installed. * The Shared Embedding Service feature must be enabled for your organization. ## Select the Embedding Model You can view and change your default embedding model in the Cloud Portal: 1. Navigate to **AI > AI & ML Functions**. 2. Select your cluster group. 3. On the **AI Functions** tab, in **Settings**, select **Edit**. 4. On the **Edit AI Functions** page, in **Models**, select the **Embedding Model** section. 5. The available embedding models include both the shared model (`shared-qwen3-embed-0-6b`) and the dedicated models provisioned for your organization. To update the default embedding model used by `EMBED_TEXT`, select the desired model and save the change. ## Usage Use the shared embedding model with the `EMBED_TEXT` function by specifying the model name `shared-qwen3-embed-0-6b`. The following examples use the `customer_reviews` table. ## Basic Usage ``` SELECT cluster.EMBED_TEXT( 'The headphones have poor sound quality, disconnect frequently, and the battery lasts only a few hours. I would not recommend this product.', 'shared-qwen3-embed-0-6b' ) AS embedding; ``` ## Using the Default Model If the shared embedding model is set as your default embedding model, you can remove the model parameter: ``` SELECT cluster.EMBED_TEXT( 'The headphones have poor sound quality, disconnect frequently, and the battery lasts only a few hours. I would not recommend this product.' ) AS embedding; ``` ## Generating Embeddings for a Table Column ``` SET batch_external_functions = AUTO; UPDATE customer_reviews SET review_embedding = cluster.EMBED_TEXT(review_text, 'shared-qwen3-embed-0-6b') WHERE review_embedding IS NULL; ``` ## Performance Considerations * **Batching**: Use `SET batch_external_functions = AUTO`, before bulk embedding operations to enable automatic batching, which significantly improves throughput. * **Token limits**: The Qwen3-Embedding-0.6B model processes text in token chunks. For large text inputs, the service automatically handles tokenization and batching. * **Region**: For lowest latency, use cluster groups in the same region. > **📝 Note**: The shared embedding model is available only in the US East (N. Virginia) region. As a shared multi-tenant service, throughput may vary based on overall system demand. The model selection is fixed to Qwen3-Embedding-0.6B; for custom model requirements, use a dedicated embedding model. ## Related Topics * [AI Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions.md) * [EMBED\_TEXT](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions/#section-id235175351219875.md) * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) *** Modified at: August 4, 2026 Source: [/cloud/ai/ai-ml-functions/shared-embedding-model-service/](https://docs.singlestore.com/cloud/ai/ai-ml-functions/shared-embedding-model-service/) (An index of the documentation is available at /llms.txt) --- # Aura Analyst > **📝 Note**: This is a Preview feature. ## Overview Aura Analyst is an AI-powered data analyst that makes it easy to converse with the data in plain English. It is a fully managed agentic AI service built natively for SingleStore that translates natural language questions into precise SQL queries and instantly delivers insights from the databases. By removing the requirement of manually writing SQL or building complex dashboards, Aura Analyst accelerates analytics and empowers users to uncover insights faster. Leveraging SingleStore’s Agentic AI platform, it ensures that queries are accurate, context-aware, and governed, helping teams make smarter, data-driven decisions with ease. ## How Aura Analyst Works Aura Analyst is a fully managed agentic, multi-component AI system designed to transform natural language into reliable, actionable insights. Instead of relying on a single, monolithic model, it orchestrates a series of specialized stages, including intent interpretation, query planning, SQL generation, validation, execution, and result summarization. This modular architecture enhances accuracy, ensures control, and adapts flexibly across diverse datasets and query types. By clarifying ambiguous inputs, exposing query plans, and automatically retrying on failures, Aura Analyst provides a more dependable experience while building user trust. Beyond running SQL, the system makes results easy to interpret through natural-language summaries and visualizations, enabling business and technical users alike to act on data with confidence. ## What is a Domain A **Domain** is the context service for Aura Analyst. It acts as a knowledge base, supplying the semantic and structural framework required to interpret questions accurately, generate the right SQL, and deliver governed insights. Users with the “Agent Domain Owner” permission can configure Domains by selecting relevant clusters, databases, and tables. They can enhance Domains with custom instructions, business-specific formulas, or logic that Aura Analyst applies when answering questions.Once configured, a Domain abstracts the complexity of raw schemas. Users interact with a curated, context-rich view of the data, making analytics more accurate, consistent, and accessible. Domains are dynamic. As data changes and new business requirement emerges, users can continuously refine them. Each update enhances the context service, ensuring that insights remain reliable, governed, and immediately actionable. ## Create a Domain Once the Aura Analyst is enabled, select **Create domain**. In the **Create Domain** dialog: | **Domain Details** | | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Enter the name of the domain. | | **Description** | Enter the domain description. | | **Connect Data** | | | **Connection** | Select theSingleStoredeployment (cluster) the domain connects to.Specifying aclusterallows connecting theSingleStoredatabases referenced in the domain.SingleStorerecommends creating a newclusterwith a read-only database attachment for Aura Analyst to maintain clarity and separation of workloads. | | **Select tables** | Select the tables from theSingleStoredatabases. | Select **Create** to create the domain for Aura Analyst. > **📝 Note**: Aura Analyst keeps the cluster active as hourly metadata collection prevents the cluster from remaining idle for more than one hour. ## Configure an Existing Domain To configure an existing domain, enable **Creator Mode**, and enter or select the following: | **Build** | | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Context** | Contexts are reference materials that the Analyst can draw on to answer questions more accurately. Use them to provide custom instructions, background knowledge, assumptions, or business logic that may not be immediately obvious from the raw data.
  • Instructions: Add general instructions to set response style, constraints, and domain-specific guidance. To add an instruction, enter your context in Instructions, and then select Save. The following are the context states for the instructions:
    • Draft: A working copy of domain instructions that can be edited and tested without affecting the live published version. It is created when a domain owner begins editing instructions in Creator Mode and changes remain isolated until explicitly published. This allows domain owners to test and refine instructions, make multiple edits without disrupting active users, and discard changes if needed.
    • Published: The current live version of domain instructions that the Analyst actively uses to answer questions. All users querying the domain see and use the published version, and the Analyst applies these instructions when generating SQL and interpreting questions. This is the production state visible to all domain users.
  • Learned Context: The system automatically learns contexts from user questions and interactions. The following are the context states for the learned contexts:
    • Pending: Learned contexts that have been automatically generated by the system but have not yet been reviewed by a domain owner. These insights are awaiting approval or rejection.
    • Approved: Learned contexts that have been reviewed and approved by a domain owner. Only approved contexts are actively used by the Analyst to answer questions and generate SQL queries.
    • Rejected: Learned contexts that have been reviewed and rejected by a domain owner.
    You can update and delete existing learned contexts.
| | **Data** | The Data section allows you to review and enhance schema metadata and includes relationship management capabilities.
  • Database: Navigate to the Database tab. Auto-generated descriptions for tables and columns help the Analyst understand the data model. Domain owners can edit and refine these descriptions to improve accuracy. Add more tables to the domain by selecting Add. Tables can be added from the deployment connected to the domain. To edit the table description, select the ellipsis (three dots), and then select Edit. To delete the table, select the ellipsis (three dots), and then select Delete. In the Table column, select the table and you can view and edit the description of each column of the table.
  • Entity-Relationship: Navigate to the Relationship tab. A relationship crawler automatically discovers foreign key relationships using schema analysis and LLM inference, assigning each relationship a confidence score between 0.0 and 1.0. Discovered relationships include source and target database, table, column, cardinality, and description. You can also manually create relationships by selecting source and target tables and columns, specifying cardinality (> for many-to-one, < for one-to-many, - for one-to-one, <> for many-to-many), and saving entries individually or in bulk. Manually created relationships are labeled accordingly. To add a relationship, select Add. In Add table relationship, enter of select the following:
    • Left Table: Select the table from the list.
    • Right Table: Select the table from the list.
    • Join Condition: Select the columns from each table to create a join condition.
    • Type: Select the cardinality type.
    • Description (Optional): Enter the description.
    Select Save Relationship to save the table relationship. To edit the relationship, select the ellipsis (three dots), and then select Edit. To delete the relationship, select the ellipsis (three dots), and then select Delete.
| | **Governance** | | | **Chat Review** | Review recorded analyst conversations and user feedback to monitor quality and identify areas for improvement.View the question, rating, reason, and comment of the response. Select the question to view the Analyst’s response. Monitor response quality, review user feedback, and identify areas for improvement. | | **API Keys** | Create and manage API keys used for authenticating requests to the[Aura Analyst API](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api.md).Copy the Aura Analyst API`/chat`endpoint URL. | | **Access Controls** | Access Control provides the ability to assign roles, set permissions, and control who can access the domain.View all the users or teams and their roles in this tab. To share access to the domain with a user or a team, select**Add**. In the**Share \**dialog, select**User**or**Team**, select the access role, and then select**Save**.To remove access for any**User**or**Team**, select**Remove****Access**in the access role of the selected user or team.

📝 Note:

Inherited roles cannot be modified.

| | **Settings** | Update the domain name and description in the**Name**and**Description**fields, respectively. Enable**Record conversations for review**. When enabled, the system saves all conversations for the selected domain to the**Chat Review**. If this setting is disabled, only questions with user feedback are captured in**Chat Review**. | ## Aura Analyst Access Controls SingleStore Aura defines three RBAC (role-based access control) roles for Aura Analyst: * **Organization Level Controls**: * **Aura Creator**: Provides the ability to install and uninstall Aura Analyst. * **Agent Domain Owner**: Provides the ability to create and manage domains. > **📝 Note**: Domain creation is limited to the tables accessible by the "Agent Domain Owner". * **Domain Level Controls**: * **Owner**: Provides the ability to manage a given domain. > **📝 Note**: Domain creation is limited to the tables accessible by the "Owner". * **User**: Provides the ability to use the domain and ask questions in natural language and receive responses. When an Agent Domain Owner creates a domain, the system performs the following actions: * Aura Analyst fetches the data accessible to the Agent Domain Owner. * Aura Analyst uses AI to infer tables and column definitions. * SingleStore recommends owners to review the description and modify as required. * When creating a new domain, a database user `_auraanalyst` is created with select permissions to the tables selected in the domain. * All queries run by Aura Analyst within a Domain are run as `_auraanalyst` user. A user can be granted Domain-level access even without having privileges on the underlying databases or tables referenced by the Domain. Aura Analyst runs queries inside the Domain as the `_auraanalyst` database user. This allows any user with access to the Domain to query the underlying databases. A user may be an Aura Domain **User** or **Domain** Owner regardless of other organization roles. A user can be a member without any additional roles (a "roleless" org member) and still receive Domain-level permissions. However, these permissions do not grant or imply elevated privileges on the underlying databases. > **📝 Note**: Domain owners must manage Domain roles and access settings to align with their organization’s governance policies. ## Interacting with Aura Analyst Once a domain is created and configured, you can ask any data related questions to Aura Analyst using natural language. To start a conversation, select the desired domain in the chatbox. Domains can be switched at any time during a conversation. > **📝 Note**: All queries and interactions require an active Domain context. Within a conversation, following features can be accessed: * **Thoughts block**: View how Aura Analyst approached to provide the insights as a response including the SQL queries executed. * **Left panel controls**: * View the chat history. * Start a new session. * Delete an existing session. * **Feedback**: Your feedback enables a human-in-the-loop system that continuously improves results. * Improve accuracy over time: Feedback helps refine how queries are interpreted and answered to deliver more reliable and complete responses for future queries. * Domain expert review: Domain owners can review query interactions to identify gaps and improve overall system performance. For a query response, use the “**Is this correct?**” option under the query result to provide feedback. * Select **Yes** (thumbs-up) if the response is correct. * Select **Needs Improvement** (thumb-down) if the response is incorrect or incomplete. When **Needs Improvement** is selected, submit feedback by selecting one of the following options: * Values look off * Missing data * Misunderstood question * Analyst made something up * Other Optionally, enter additional details in the field to describe what went wrong. > **📝 Note**: Observers can use [Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring/#section-idm33412590041948.md) and [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) to track queries run by Aura Analyst, including slow-performing queries. ## Share a Chat You can share an Aura Analyst chat with other users in your organization, enabling them to review the conversation and its associated insights. To share a chat: 1. Open a chat that includes at least one assistant response. 2. Select **Share** from the right top of the chat. 3. Copy the generated link and share it with the users who need access. > **📝 Note**: - The shared chat is a snapshot of the conversation up to the most recent assistant response at the time of sharing. It does not update if the chat continues. > - Shared chats are not public. A user can access a shared link only if the following conditions are met: > > * The user is signed in as a member of the same organization that owns the chat. > * The user has permission to use the Aura Analyst domain associated with the chat. > - Users who do not meet these conditions see an access denied message. > - Sharing a chat link is separate from other sharing features in Aura Analyst. It does not affect the following: > > * Saving or sharing charts to dashboards > * Domain access or domain sharing settings in **Domain settings** ## Continue a Shared Chat When a user opens a shared link, the conversation appears in **read-only** mode. To continue working from that conversation: 1. Select **Continue conversation** at the bottom of the page. 2. Aura Analyst creates a new chat session that copies the shared conversation. New messages in this session do not affect the original chat or the shared snapshot. ## Examples ## Example 1: SaaS Product Usage Analytics This example analyzes product adoption by creating a domain focused on it, using the relevant tables from the `saas_usage` database. ![Create Domain dialog showing 'SaaS Usage Domain' name, connection deployment set to workspace-1, and four tables selected from saas\_usage database: subscriptions, support\_tickets, usage\_logs, and users. Cancel and Create buttons appear at the bottom.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1346d433d5114b76/6a7ada8ec7a40c424430ea75/Screenshot_2025-09-30_at_7_54_13_AM-O3JPRu.png) Once the domain is created, questions can be asked about the data. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltdd3c4b4cf5f418ef/6a7adb31448e4efcf95b5451/saasusage-Nhec4j.gif) ## Example 2: E-commerce Marketing and Campaign Analytics This example analyzes marketing product adoption by creating a domain focused on it, using the relevant tables from the `marketing_analytics` database. ![Create Domain dialog showing 'Marketing Analytics Domain' name, connection deployment set to workspace-1, and five tables selected from marketing\_analytics database. Options to cancel or create are at the bottom.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc4c83f4cbfa6038c/6a7adad86c6eac620df08cf9/Screenshot_2025-09-30_at_7_55_08_AM-ZwlaGx.png) Once the domain is created, questions can be asked about the data. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltfd9f3cdb8a55097d/6a7adaf22f00b2a5a5ef3647/marketing_analytics-KwH8NB.gif) ## In this section * [Aura Analyst API](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api.md) * [Aura Analyst API Reference](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference.md) *** Modified at: July 13, 2026 Source: [/cloud/ai/aura-analyst/](https://docs.singlestore.com/cloud/ai/aura-analyst/) (An index of the documentation is available at /llms.txt) --- # Aura Analyst API Reference *** Modified at: July 14, 2026 Source: [/cloud/ai/aura-analyst/aura-analyst-api-reference/](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference/) (An index of the documentation is available at /llms.txt) --- # Aura Analyst API > **📝 Note**: This is a Preview feature. [Aura Analyst](https://docs.singlestore.com/cloud/ai/aura-analyst.md) is a managed, AI-powered data analyst built natively for SingleStore. It enables you to submit natural language queries, which are automatically translated into optimized SQL statements and executed against the selected data sources. Built on SingleStore's Agentic AI platform, Aura Analyst delivers context-aware, governed, and accurate insights. SingleStore Aura Analyst API ("the API") enables you to embed [Aura Analyst](https://docs.singlestore.com/cloud/ai/aura-analyst.md) capabilities in your applications, workflows, and user experiences via a REST interface to provide conversational access to data and analytics. Query the API using natural language queries and receive insights on your SingleStore data as tables, charts, or summaries. ## Why Use the Analyst API The Analyst API provides a streamlined model to integrate Aura Analyst in your applications, which allows you to: * Send natural language queries and receive generated SQL, results, and visualizations. * Build interactive analytics experiences using a streaming chat endpoint that delivers real-time events over Server-Sent Events (SSE). * Maintain multi-turn analytic sessions enabling follow-up questions to reuse prior context and results. ## Analyst API Endpoint Reference Analyst API currently supports the following endpoints: | Endpoint | HTTP Method | Request Body | Description | | ------------------------ | ----------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/query` | `POST` | Required | **Structured Query Endpoint**Runs a natural language analytics query and returns a single, structured JSON response containing generated SQL, optional result data, charts, and explanatory text. Use this endpoint for integrations, pipelines, scheduled jobs, and embedded use-cases that require deterministic, machine-readable output instead of a chat-based stream. | | `/chat` | `POST` | Required | **Streaming Chat Endpoint**Runs a natural language analytics query and streams back Server-Sent Events (SSE) for use in interactive chat-style UIs. Use this endpoint when real-time typing indicators or incremental responses are required for a chat-based stream. | | `/conversation-starters` | `GET` | | **Conversation Starter Endpoint**Returns a list of generated starter prompts tailored to the current domain and agent context. Use this endpoint to populate suggested questions in the UI before a user submits the first query or starts a chat session. | Use the `/query` endpoint for most integrations. It returns a single structured JSON response and is easier to parse in back-end services, pipelines, and embedded product workflows. Use the `/chat` endpoint for a streaming, interactive chat experience over Server-Sent Events (SSE), such as real-time typing indicators or incremental responses in a UI. Refer to [Aura Analyst API Reference](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference.md) for more information. ## Authentication Analyst API authenticates to the server using [Bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/) standard, which must be used in conjunction with SSL (HTTPS) for secure communication over the network. To access the API, provide the Analyst API key in the Authorization request header in each HTTP request (API call). The API key is scoped at the domain-level, i.e., each API key is unique to the domain selected while generating the API key. ## Generate an API Key To generate an API key for Analyst API: 1. On the [Cloud Portal](https://portal.singlestore.com), select **AI > Analyst** on the left navigation pane. 2. On the **Analyst** page, enable **Creator Mode**. 3. From the **Domain** list, select a domain. 4. Under **Governance**, select **API Keys** in the left pane. 5. On the **API Keys** page, select **Create API Key**. 6. In the **Create API key** dialog, enter a name for the API key and select an expiration window. 7. Select **Create API Key**. 8. Copy and securely store the API key. Alternatively, select **Save As Secret** to store the API key as a secret; enter a name for the secret in the **New Secret** dialog, and select **Create Secret**. > **📝 Note**: The API key is displayed only once. 9. Select **Close**. Use this key in the API call to authorize access to the API. ## Revoke an API Key To revoke an existing API key: 1. Select **AI > Analyst**, enable **Creator Mode**, and select a domain. 2. Navigate to the **API Keys** section in the left pane. 3. Under the **Actions** column for the API key, select the delete icon. 4. On the **Revoke API Key** dialog, acknowledge the warning, and select **Revoke**. The API key is now successfully revoked. ## Access the Analyst API Copy the endpoint URL from the [Cloud Portal](https://portal.singlestore.com): 1. Select **AI > Analyst**, enable **Creator Mode**, and select a domain. 2. Navigate to the **API Keys** section in the left pane. 3. Select **Copy Endpoint** to copy the URL of the `/chat` endpoint. Replace `chat` with the endpoint to access in the URL. The base URL of the Analyst API conforms to the following format: ```URL https://apps.{region}.cloud.singlestore.com/v1/organizations/{orgID}/projects/{projectID}/analyst/ ``` > **📝 Note**: The Analyst API endpoints use a domain-scoped API key for authentication. The API key's JWT claims encode the allowed domain, project, organization, and creator for a request. The `{orgID}` and `{projectID}` in the request path must match the key’s claims; otherwise, the request is rejected with `403 INVALID_API_KEY`. To access the Analyst API, specify the API key in the Authorization header of the API request. For example, while making API calls through an HTTP client (such as cURL), specify the endpoint URL and API key as follows: ```shell curl -X 'POST' \ '' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' ``` ## Getting Started with Aura Analyst API To get started with the API: 1. [Create a domain](https://docs.singlestore.com/cloud/ai/aura-analyst/#section-id235182638851576.md) on the [Cloud Portal](https://portal.singlestore.com), or select an existing domain. 2. [Generate an API key](https://docs.singlestore.com/#section-id235672341710795.md) for the Analyst API. 3. [Copy the endpoint URL](https://docs.singlestore.com/#section-id235672343574668.md). 4. Use the API endpoint to access Aura Analyst via the API. ## Start a New Chat Session Use the `POST /chat` endpoint to start an interactive Aura Analyst conversation over Server-Sent Events (SSE). To create a new chat session, omit `session_id` from the request body. You can also send `null` or an empty string and the gateway will create a new session automatically. The returned session can then be reused in later requests to continue the conversation. For example (update the endpoint, the API key, and message before running the following command): ```shell curl -X POST "https://apps.${REGION}.cloud.singlestore.com/v1/organizations/${ORG_ID}/projects/${PROJECT_ID}/analyst/chat" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ -d '{ "message": "Show me sales data for Q4." }' ``` The first stream includes a `response.created` event containing the generated `session_id`. Capture this value and include it in the next request to continue the same chat context. ## Continue an Existing Chat Session Use the same `POST /chat` endpoint (used to start a new chat) to continue a prior Aura Analyst conversation. Include the previously returned `session_id` in the request body so the Analyst can reuse the earlier context and interpret follow-up questions correctly. For example (update the endpoint, the API key, message, and session ID before running the following command): ```shell curl -X POST "https://apps.${REGION}.cloud.singlestore.com/v1/organizations/${ORG_ID}/projects/${PROJECT_ID}/analyst/chat" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ -d '{ "message": "Break that down by region", "session_id": "${SESSION_ID}" }' ``` The `session_id` must be a valid UUID. ## Get the Generated SQL from a Chat ## Streaming Chat Endpoint To retrieve generated SQL from the streaming chat endpoint, send the request to `POST /chat` endpoint and set `included_events` to `["sql"]`, which is the event type used for generated SQL queries. The `included_events` field controls which event types are returned for chat responses. If you omit `included_events` or pass an empty array, the API returns all chat events. For example (update the endpoint, the API key, and message before running the following command): ```shell curl -X POST "https://apps.${REGION}.cloud.singlestore.com/v1/organizations/${ORG_ID}/projects/${PROJECT_ID}/analyst/chat" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ -d '{ "message": "What were the top 10 products by revenue last month?", "included_events": ["sql"] }' ``` To query the results in addition to the SQL, use `["sql", "data"]` instead. SingleStore recommends this combination for developer and debugging workflows because it returns both the generated SQL and the executed results. Refer to [Aura Analyst API Reference](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference.md) for more information. ## Structured Query Endpoint To retrieve the generated SQL for a back-end integration, use the `POST /query` endpoint instead of the streaming chat endpoint. Set `output_modes` to `["sql"]` to return only the generated SQL or `["sql", "data"]` to return both the generated SQL and the executed results. For example (update the endpoint, the API key, and message before running the following command): ```shell curl -X POST "https://apps.${REGION}.cloud.singlestore.com/v1/organizations/${ORG_ID}/projects/${PROJECT_ID}/analyst/query" \ -H "Authorization: Bearer ${API_KEY}" \ -H "Content-Type: application/json" \ -d '{ "message": "What were the top 10 products by revenue last month?", "output_modes": ["sql"] }' ``` The generated SQL is returned in `results[].sql.command` in the response. Refer to [Aura Analyst API Reference](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference.md) for more information. ## Related Topics * [Aura Analyst](https://docs.singlestore.com/cloud/ai/aura-analyst.md) * [Aura Analyst API Reference](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api-reference.md) *** Modified at: July 16, 2026 Source: [/cloud/ai/aura-analyst/aura-analyst-api/](https://docs.singlestore.com/cloud/ai/aura-analyst/aura-analyst-api/) (An index of the documentation is available at /llms.txt) --- # Models > **📝 Note**: This is a Preview feature. Models integrate directly with existing SingleStore workflows and provide enterprise-grade performance, security, and observability to support building GenAI-ready applications. This Helios feature supports: 1. **LLM models**: Deploy and run large language models (LLMs) for text generation and conversational tasks. 2. **Embedding models**: Generate embeddings for semantic search, recommendations, and similarity tasks. 3. **ML models**: Build, train, and run custom ML models for specific applications. SingleStore Models supports the following deployment options: * Externally hosted: * Azure AI Services * Amazon Bedrock * Aura hosted: This option hosts the model on SingleStore Aura container service that provides lowest latency (model co-located with compute and data) and ensures that data never leaves the SingleStore VPC. It supports custom and open-source models. ## LLM Models SingleStore supports the following LLM models: | Provider | Publisher | Model | | ---------------------- | ----------------------------------------------------------- | -------------------------------------------------------------- | | Amazon Bedrock | Anthropic |
  • Claude Opus 4.6
  • Claude Sonnet 4.6
| | Azure AI Services | OpenAI |
  • gpt-5.4
  • gpt-4.1-mini
| | Aura Hosted LLM Models | Open source models (Refer to License for individual models) |
  • Llama 3.2
  • Qwen
| ## Embedding Models SingleStore supports the following embedding models: | Provider | Publisher | Model | | ---------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------- | | Amazon Bedrock | Amazon |
  • Titan Embeddings G1 - Text
  • Titan Text Embeddings V2
| | Amazon Bedrock | Anthropic |
  • Amazon Nova Multimodal Embeddings
| | Azure AI Services | OpenAI |
  • text-embedding-3-small
  • text-embedding-3-large
| | Aura Hosted embedding models | Open source models (Refer to License for individual models) |
  • qwen 0.6B
  • qwen 4B
| ## ML Models Refer to [ML Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions.md) for more information. ## Create a Model Inference ## LLM Model Inference Create an LLM model inference by following these steps: 1. Navigate to the [Cloud Portal](http://portal.singlestore.com). 2. In the left navigation, select **AI > Models**. 3. Select the **LLM Models** tab, and then select **New**. 4. In **Select Model**, select the **Provider** and the model of your choice, and then select **Next**. 5. In **Model Settings**, enter and review the following information: | **Name** | Enter a name for the selected model. | | ------------------ | --------------------------------------------------------------------- | | **Description** | Enter the selected model description. | | **Region** | Select the region for the selected model. | | **Estimated Cost** | Displays the maximum number of tokens allowed for the selected model. | 6. Accept the terms of service agreement and select **Publish** for the selected LLM model inference deployment. Once the LLM model inference is deployed, use the LLM model inference via the following options: * API: Follow the sample code to start integrating with the model in either Python or Node.js. * AI Functions: Refer to [Text Processing Functions](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions/#section-id235175307180532.md) for more information. ## Embedding Model Inference Create an embedding model inference by following these steps: 1. Navigate to the [Cloud Portal](http://portal.singlestore.com). 2. In the left navigation, select **AI > Models**. 3. Select the **Embedding Models** tab, and then select **New**. 4. In **Select Model**, select the **Provider** and the model of your choice, and then select **Next**. 5. In **Model Settings**, enter and review the following information: | **Name** | Enter a name for the selected model. | | ------------------ | --------------------------------------------------------------------- | | **Description** | Enter the selected model description. | | **Region** | Select the region for the selected model. | | **Estimated Cost** | Displays the maximum number of tokens allowed for the selected model. | 6. Accept the terms of service agreement and select **Publish** for the selected embedding model inference deployment. Once the embedding model inference is deployed, use the embedding model inference via the following options: * API: Follow the sample code to start integrating with the model in either Python or Node.js. * AI Functions: Refer to [Embedding Function](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ai-functions/#section-id235175344319444.md) for more information. ## Train a New ML Model Refer to [Train a New ML Model](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions/#section-id23517557347527.md) for more information. ## Manage LLMs and Embedding Model Inference Existing LLMs and embedding model inferences can be managed by performing following actions: * Explore the existing model inferences in Playground * Update * Share * Delete A notebook can be generated for the embedding model inference only. ## Use an Existing Model Inference To use the existing model inference, perform the following actions: * Select the model in the **Name** column and then select **Details** to view the sample code for the integration. * Create an Aura App API key to use the existing model inference. Refer to [Aura App API Keys](https://docs.singlestore.com/cloud/container-services/container-app-api-keys.md) for related information. ## Generate an Embeddings Notebook Embedding notebooks can be generated for an embedding model inference. Select **Details > Generate Notebook**. The embedding notebook uses an existing embedding model inference to generate embeddings for a column in a specified table. It stores the resulting vectors in a new column within the same table that enables it to manage and query the embeddings alongside the original data. In the **Embeddings Notebook** dialog, enter or select the following: | **Cluster** | Select acluster. | | ---------------------- | ---------------------------------------------------------------------- | | **Database** | Select a database. | | **Table** | Select a table. The selected table must have embeddable columns. | | **Source Column** | Select a source column. | | **Destination Column** | Enter the name of the destination column. | | **API Key Secret** | Select the Aura App API key created for the embedding model inference. | Select **Generate** to generate the embedding notebook. A **Vector Embedding Pipeline Notebook** is created. View the generated embedding notebook in **Editor > Shared**. ## Explore an Existing Model Inference in Playground To explore an existing model inference in the playground, select the model inference in the **Name** column and then select **Playground**. ## Embedding Model In **Text Input**, enter the text to generate embeddings and view the results in **Embedding Results**. ## Chat Completion Model In **Text Input**, enter the text to generate text or chat with the model. Set model parameters as required. **Parameters** | **System Prompt** | Enter the system prompt for the chat completion model. | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Temperature** | Set the temperature of the model. Lower values are less random and near zero is deterministic. The default value is 0.7. It ranges between 0 and 1. | | **Max. Tokens** | Set the maximum number of tokens generated by the model. The default value is 512. It ranges between 1 and 1024. | | **Frequency Penalty** | Set the frequency penalty of the model. It penalizes new tokens based on existing frequency which reduces repetition. The default value is 0. It ranges between -2 and 2. | | **Top P** | Set the top P parameter of the model. It controls diversity via nucleus sampling (for example, 0.5 considers half of options). The default value is 1. It ranges between 0 and 1. | ## Update an Existing Model Inference To update an existing model inference, select the model in the **Name** column and then select **Update** in the upper right. Enter the updates required for the existing model inference, and select **Update**. ## Share an Existing Model Inference To share an existing model inference, navigate to **Models > LLM Models/Embedding Models**, select the ellipsis in the **Actions** column of the model, and select **Share**. Select the **User** or **Team** with which to share the model inference and select **Save**. > **📝 Note**: Custom models must be shared before use. Hence, to use a newly created model instance in an AI function, you have to explicitly share the model with the service account created during AI function installation. ## Delete an Existing Model Inference To delete an existing model inference, select the ellipsis in the **Actions** column of the model, and select **Delete**. ## Manage an Existing ML Model Refer to [Manage an Existing ML Model](https://docs.singlestore.com/cloud/ai/ai-ml-functions/ml-functions/#section-id235175587330847.md) for more information. *** Modified at: June 1, 2026 Source: [/cloud/ai/models/](https://docs.singlestore.com/cloud/ai/models/) (An index of the documentation is available at /llms.txt) --- # SingleStore AI Rules SingleStore AI rules are context rules that help AI-assisted development tools or MCP (Model Context Protocol) clients, such as Cursor, in understanding SingleStore’s features. These rules improve the accuracy of code suggestions when working with SingleStore. AI rules are defined in `.mdc` files. Each `.mdc` file specifies the file types it applies to. When you open a matching file, your AI tool automatically applies the rules to provide more relevant suggestions. ## Add AI Rules To enable SingleStore AI rules in your project: 1. Add the SingleStore rules files to the `.cursor/rules` directory in your project. 2. The AI tool automatically applies the rules when working with SingleStore related code. 3. Mention these rules in your prompts to receive more specific suggestions. ## AI Context Rules for Drizzle ORM AI Context rules simplify working with Drizzle ORM (Object-Relational Mapping) in combination with SingleStore when using AI tools. To use these rules with Drizzle ORM: * **In Cursor**: Save the [rules](https://docs.cursor.com/context/rules-for-ai#project-rules-recommended) file as `.cursor/rules/singlestore-drizzle.mdc`. The rules are applied automatically when working with files that match supported patterns. * **In other AI tools**: Include the rules as context when interacting with your AI assistant. Refer to your tool’s documentation for details on how to provide external context (for example, Include file or context commands). * **Clone or download the rules**: You can clone or download the [SingleStore AI rules](https://github.com/singlestore-labs/mcp-server-singlestore/blob/main/src%2Fapi%2Fresources%2Fdocs%2Fsinglestore-drizzle.mdc). Once the rules are part of your project, AI tools automatically apply them when generating Drizzle ORM code for SingleStore. You can mention these rules in prompts to get more consistent results. Refer to [Create a Drizzle ORM Project Using the SingleStore MCP Server](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/create-a-drizzle-orm-project-using-the-singlestore-mcp-server.md) to know more about how to create a Drizzle ORM project using SingleStore MCP Server. *** Modified at: September 22, 2025 Source: [/cloud/ai/singlestore-ai-rules/](https://docs.singlestore.com/cloud/ai/singlestore-ai-rules/) (An index of the documentation is available at /llms.txt) --- # SingleStore MCP Server ## Overview The Model Context Protocol (MCP) is an open protocol that standardizes how external tools and interfaces communicate context to large language models (LLMs). With MCP, you can describe an environment in which a model operates, including available tools, user intent, system state, and related contextual information. ## Why Use MCP? MCP enables you to: * Define context programmatically which enables the model to accurately interpret your goals and available tools. * Eliminate the need for writing custom prompts for each integration scenario. * Expose your system’s capabilities and usage conditions for scalable and collaborative AI agent design. ## Capabilities and Benefits The [SingleStore MCP server](https://github.com/singlestore-labs/mcp-server-singlestore) implements Model Context Protocol (MCP) to enable seamless integration between large language models (LLMs) and SingleStore database services. The server provides a set of tools, resources and prompts templates that allows MCP compatible clients, such as Claude Desktop, VS Code, and Cursor, to interact with SingleStore using natural language prompts. These tools expose essential functionality for database management, notebook workflows, job scheduling, and usage tracking. The SingleStore MCP server has the following key capabilities: * **Authentication and Configuration** * Authenticates with SingleStore’s management API using secure API keys. * Maintains user-specific session state, including preferences and input history, to support contextual interactions. * **Cluster and Database Management** * Retrieves metadata about accessible cluster groups, including cluster group IDs and names. * Lists clusters within a selected cluster group and provides endpoint and status details. * Returns details about the user’s current organization, including metadata and settings. * Lists all supported deployment regions with relevant identifiers and region-specific metadata. * Runs SQL queries on connected clusters and returns structured results with rows, columns, and error handling. * Manages Shared clusters within SingleStore: * Lists available Shared clusters. * Creates new shared clusters with specific configurations. * Organizes user access controls and permissions. * Runs SQL on virtual clusters. * **Notebook and File Management** * Manages notebooks: * Lists available notebook samples. * Creates new notebooks in the user’s personal cluster. * Supports custom notebook content and configurations. * Manages files: * Lists files in both personal and shared spaces. * Creates and manages files. * Organizes file access controls and permissions. * **Scheduling Jobs** * Creates scheduled jobs: * Creates jobs that run notebooks on a defined schedule (one-time or recurring). * Configures runtime environments and notebook parameters. * Manages notebook snapshots. * Monitors jobs: * Retrieves detailed information about scheduled jobs. * Lists historical executions. * Monitors job status and execution results. ## Set Up SingleStore MCP Server > **📝 Note**: To use the SingleStore MCP Server, you need a SingleStore Helios account and API access credentials. ## Prerequisites Ensure the following tools are installed: * Python 3.11+ * [uvx](https://docs.astral.sh/uv/guides/tools/) * [uv](https://docs.astral.sh/uv/getting-started/installation/) * Preferred MCP client ## Supported MCP Clients The SingleStore MCP server supports the following clients: * Claude Desktop * Claude Code * Cursor * Visual Studio Code * Windsurf * Gemini CLI * LM Studio * Goose * Qodo Gen Following MCP clients are available through Docker Desktop: * Continue.dev * Gordon ## MCP Client Setup You can set up a supported MCP client using any of the following approaches: 1. **Using the initialization command**: The initialization command automatically locates the appropriate configuration file for your client platform and adds or updates it to include the SingleStore MCP server. Additionally, it sets up browser-based OAuth authentication and displays instructions for starting the server. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc612c85907c50158/6a7adb2114d87c42933c76d6/Screenshot_2025-07-31_at_3_24_03_PM-LlgRoZ.png) Use the following command to initiate the server: ```shell uvx singlestore-mcp-server init ``` Use the following command to specify a client: ```shell uvx singlestore-mcp-server init --client= ``` Replace `` with a supported MCP client. Following are the example syntax for the supported MCP client. * Claude Desktop ```shell uvx singlestore-mcp-server init --client=claude-desktop ``` * Claude Code ```shell uvx singlestore-mcp-server init --client=claude-code ``` * Cursor ```shell uvx singlestore-mcp-server init --client=cursor ``` * Visual Studio Code ```shell uvx singlestore-mcp-server init --client=vscode ``` * Windsurf ```shell uvx singlestore-mcp-server init --client=windsurf ``` * Gemini CLI ```shell uvx singlestore-mcp-server init --client=gemini ``` * LM Studio ```shell uvx singlestore-mcp-server init --client=lm-studio ``` 2. **Installing via Smithery**: Use the following command to install `singlestore-mcp-server` via Smithery: ```shell npx -y @smithery/cli install @singlestore-labs/mcp-server-singlestore --client= ``` Replace `` with a supported MCP client. 3. **Manual Configuration**: You can configure the following supported MCP clients manually. Add the following standard configuration to your MCP client’s configuration file: ```shell { "mcpServers": { "singlestore-mcp-server": { "command": "uvx", "args": [ "singlestore-mcp-server", "start" ] } } } ``` Use the configuration file of the MCP client as mentioned in the following client's documentation: 1. **Claude Desktop**: Refer to the MCP install [guide](https://modelcontextprotocol.io/quickstart/user) for the related information. 2. **Claude Code**: Run the following command to manually setup the MCP client: ```shell claude mcp add singlestore-mcp-server uvx singlestore-mcp-server start ``` 3. **Cursor**: Navigate to **Cursor Settings > MCP > Add new MCP Server**. Enter the name of the MCP server. Select `command` type, and enter the command `uvx singlestore-mcp-server start`. Select **Edit** to verify configuration or add command line arguments. 4. **Visual Studio Code**: Refer to the MCP install [guide](https://code.visualstudio.com/docs/copilot/chat/mcp-servers#_add-an-mcp-server) for the related information. You can also install using the VS Code CLI: ```shell code --add-mcp '{"name":"singlestore-mcp-server","command":"uvx","args":["singlestore-mcp-server","start"]}' ``` After installation, the SingleStore MCP server is available for use with your GitHub Copilot agent in VS Code. 5. **Windsurf**: Refer to the Windsurf [documentation](https://docs.windsurf.com/windsurf/cascade/mcp) for the related information. 6. **Gemini CLI:** Refer to the MCP install [guide](https://github.com/google-gemini/gemini-cli/blob/main/docs/tools/mcp-server.md#configure-the-mcp-server-in-settingsjson) for the related information. 7. **LM Studio**: Navigate to **Program** in the right sidebar, then select **Install > Edit `mcp.json`**. 8. **Goose**: Navigate to **Advanced settings > Extensions > Add custom extension**. Enter the name of your custom extension. Select type `STDIO`, and enter the command `uvx singlestore-mcp-server start`. Select **Add Extension** to add your custom extension. 9. Qodo Gen: Open [Qodo Gen](https://docs.qodo.ai/qodo-documentation/qodo-gen) chat panel in VSCode or IntelliJ. Select **Connect more tools > + Add new MCP**, and paste the standard configuration. > **📝 Note**: This approach does not require any API keys, tokens, or environment variables. When the server starts, authentication is automatically handled via browser-based OAuth. 4. **Using Docker**: 1. Run the following command to build the docker image of the SingleStore MCP server: ```shell docker build -t singlestore/mcp-server-singlestore . ``` 2. Use the following configuration: ```shell { "mcpServers": { "singlestore-mcp-server": { "command": "docker", "args": [ "run", "-i", "--rm", "--init", "--pull=always", "-e", "MCP_API_KEY=your_api_key_here", "singlestore/mcp-server-singlestore" ] } } } ``` 3. Update the configuration of your MCP configuration file, and then restart your MCP client. > **📝 Note**: When running the server in a Docker container, an API key is required. Browser-based OAuth is not supported for locally hosted containers. For improved security, SingleStore recommends using Docker Desktop to configure the SingleStore MCP server. Refer to The Docker MCP Catalog: the Secure way to Discover and Run MCP Servers for more information. ## Tools in SingleStore MCP Server The SingleStore MCP server implements the following tools: | **Tool** | **Description** | **Arguments** | **Returns** | | ----------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `get_user_info` | Retrieves details about the user. | None | User details | | `organization_info` | Retrieves details about the current organization. | None | Organization details | | `choose_organization` | Retrieves list of organizations. | None | List of organizations | | `set_organization` | Sets the selected organization. |
  • organization_id
| Sets the organization | | `workspace_groups_info` | Retrieves details about accessibleclustergroups. | None | Clustergroup details | | `workspaces_info` | Retrievesclusters in a specificclustergroup. |
  • workspaceGroupID
| Clusterdetails | | `resume_workspace` | Resumes a pausedclusterwithin theclustergroup. |
  • workspace_id
| Resumescluster | | `list_starter_workspaces` | Lists all Sharedclusters accessible to the user. | None | Sharedclusterdetails | | `create_starter_workspace` | Creates a Sharedclusterand user. |
  • name
  • database_name
  • username
  • password
  • workspace_group (optional)
| Details of the createdclusterand user | | `terminate_starter_workspace` | Terminates Sharedcluster. This action cannot be undone. |
  • workspace_id
| Terminates Sharedcluster | | `list_regions` | Lists all regions that supportclusters. | None | List of regions | | `list_sharedtier_regions` | Lists the available regions before creating a Sharedcluster. | None | Lists region name and cloud provider. | | `run_sql` | Runs SQL on a connectedcluster. |
  • workspace_group_identifier
  • workspace_identifier
  • username
  • password
  • database
  • sql_query
| SQL results in structured format | | `create_notebook_file` | Creates a new notebook in the specified space (personal or shared). |
  • notebook_name
  • content (optional)
| Created notebook details | | `upload_notebook_file` | Uploads created notebook file in theCloud Portal. |
  • local_path
  • upload_name (optional)
  • upload_location (optional)
| Uploads created notebook | | `create_job_from_notebook` | Creates a scheduled job to run a notebook. |
  • name
  • notebook_path
  • schedule_mode
  • execution_interval_minutes (optional)
  • start_at (optional)
  • description (optional)
  • create_snapshot (optional)
  • runtime_name
  • parameters (optional)
  • target_config (optional)
| Created job details | | `get_job` | Retrieves details of an existing job |
  • job_id
| Existing job details | | `delete_job` | Permanently deletes a scheduled job. |
  • job_id
| Deletes the scheduled job permanently | ## In this section * [Examples](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/examples.md) * [Create a Drizzle ORM Project Using the SingleStore MCP Server](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/create-a-drizzle-orm-project-using-the-singlestore-mcp-server.md) *** Modified at: July 10, 2026 Source: [/cloud/ai/singlestore-mcp-server/](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/) (An index of the documentation is available at /llms.txt) --- # Create a Drizzle ORM Project Using the SingleStore MCP Server You can easily create a Drizzle ORM project using the SingleStore MCP server and the VS Code MCP Client to connect to your SingleStore deployment and run queries. ## Prerequisites * Python 3.11+ * [uv](https://docs.astral.sh/uv/getting-started/installation/) * [uvx](https://docs.astral.sh/uv/guides/tools/) * GitHub Copilot for VS Code ## Connect to SingleStore using Drizzle ORM The following example creates a Drizzle ORM project using the SingleStore MCP Server and the GitHub Copilot in VS Code MCP client, and performs the following tasks in response to a specified prompt: 1. Initializes a `npm` project and installs the required dependencies. 2. Creates the required configuration files (Typescript config, Drizzle config, etc.). 3. Connects to a SingleStore deployment and runs the specified migration. 1. Creates a table named `stock` in the SingleStore database. 2. Inserts a sample data set into the `stock` table. 3. Queries the table. 4. Provides an overview of the steps performed. The Copilot automates each of these steps, prompting the user only for a confirmation to proceed. ## Step 1: Install the SingleStore MCP Server Open an empty folder in VS Code and perform the following tasks: 1. Run the following command in the terminal to initialize the SingleStore MCP Server in VS Code: ```shell uvx singlestore-mcp-server init ``` 2. Authenticate via browser OAuth (log in to the [Cloud Portal](https://portal.singlestore.com)). 3. Restart the MCP client (VS Code) to apply the changes. ## Step 2: Add the SingleStore Drizzle ORM Context After the changes are applied, 1. Select **Add context** in the Copilot Chat view. 2. From the list of resources, select **MCP Resources**. * (If required) Authenticate via browser OAuth. And, then select **Add context > MCP Resources**. * (If required) Start the SingleStore MCP server, select the icon in the chat view, and then select **Start Server** from the list. 3. Select **get\_singlestore\_drizzle\_guide**. ## Step 3: Initialize the Project and Run the Application using Copilot Upon adding the `get_singlestore_drizzle_guide` MCP context, enter the following prompt (or similar) in the Chat view: > **📝 Note**: Update the connection configuration (connection string) of your SingleStore Helios deployment before submitting the prompt. ```Prompt Create a sample Drizzle project using SingleStore. The endpoint URL of the SingleStore deployment is: singlestore://:@:/. Create a table named stock, with the ID, Type, and quantity columns. Insert sample data into the table and then query the data. Also calculate the total stock of each Type. ``` After submitting the prompt, follow the instructions in the Chat view. The following example demonstrates the steps performed and the output generated by the Chat view in response to the specified prompt. > **📝 Note**: The Copilot automates each of these steps, prompting the user only for a confirmation to proceed. (You may be prompted to select **Continue** after completing each step.) 1. Initialize a new `npm` project. ```Response I'll help you create a Drizzle project with SingleStore integration. Let's break this down into steps: First, let's create a new workspace for the project: - Generating plan to create a new workspace Let me help you create a Drizzle project with SingleStore integration. I'll create the necessary files and set up the project structure. First, let's initialize a new npm project and install the required dependencies: Run command in terminal npm init -y Initializing a new npm project ``` 2. Install the required dependencies. ```Response Now, let's install the required dependencies: Run command in terrminal: npm install drizzle-orm mysql2 dotenv && npm install -D drizzle-kit typescript ts-node @types/node Installing required dependencies ``` 3. Generate the following necessary files: * `tsconfig.json` (TypeScript configuration) * `schema.ts` (database schema) * `drizzle.config.ts` (Drizzle configuration) * `index.ts` (database connection and queries) * `.env` (environment variables) * `package.json` (package scripts) ```Response Let's create the necessary TypeScript configuration: Created tsconfig.json Now, let's create our database schema: Created schema.ts Let's create the Drizzle configuration file: Created drizzle.config.ts Let's create our database connection and queries: Created index.ts Let's create the environment file with the SingleStore connection URL: Created .env Let's update the package.json scripts: Created package.json ``` 4. Run the migrations and start the application. ```Response Now, let's run the migrations and start the application. First, we'll generate and push the migrations: Run command in terminal npm run generate && npm run push Generating and pushing the database migrations Now let's run the application. npm run start ``` ```output Connecting to database... Creating stock table... Successfully connected to the database Inserting sample data... Querying all stocks: [ { id: 4503599627370498, type: 'Furniture', quantity: 50 }, { id: 4503599627370499, type: 'Electronics', quantity: 75 }, { id: 4503599627370500, type: 'Clothing', quantity: 200 }, { id: 4503599627370497, type: 'Electronics', quantity: 100 }, { id: 4503599627370501, type: 'Furniture', quantity: 25 } ] Calculating total stock by type: [ { type: 'Electronics', totalQuantity: '175' }, { type: 'Clothing', totalQuantity: '200' }, { type: 'Furniture', totalQuantity: '75' } ] ``` 5. Overview of the steps performed. ```Response Let's break down what we've accomplished: * Successfully connected to the SingleStore database * Created the stock table with the required columns: * id (BIGINT, auto-incrementing primary key) * type (VARCHAR) * quantity (INT) * Successfully inserted sample data with different types of stock: * Electronics (total: 175 units) * Furniture (total: 75 units) * Clothing (total: 200 units) * Successfully queried the data: * Retrieved all stock entries * Calculated total quantity by type using GROUP BY aggregation The application is now fully functional! ``` As instructed in the prompt, the Copilot successfully completes the specified tasks. ## Related Topics * [Connect with Drizzle ORM](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/connect-with-drizzle-orm.md) *** Modified at: September 22, 2025 Source: [/cloud/ai/singlestore-mcp-server/create-a-drizzle-orm-project-using-the-singlestore-mcp-server/](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/create-a-drizzle-orm-project-using-the-singlestore-mcp-server/) (An index of the documentation is available at /llms.txt) --- # Examples The following examples demonstrate how to use the SingleStore MCP Server with the Claude Desktop. This requires an initial setup. Perform the following tasks to initiate the MCP client: 1. Run the following command in the terminal: ```shell uvx singlestore-mcp-server init ``` 2. Restart Claude Desktop. > **📝 Note**: When using third-party tools such as Claude with the SingleStore MCP server, a permission prompt appears requesting access to run actions like `run_sql`. Review each action carefully before granting access. These tools operate independently, and Claude cannot guarantee the security or privacy practices of third-party integrations.You can choose one of the following options:- **Allow once**: Approves the action for a single use. > - **Allow always**: Grants persistent access for future requests. > - **Decline**: Denies the request.For improved security, SingleStore recommends selecting **Allow once** during initial setup or testing.![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcec6d63237882a10/6a7ada949f5251194265e8e0/Screenshot_2025-07-31_at_7_59_54_PM-qriRvN.png) ## Create a Notebook in Personal Space This example creates a notebook in personal space. Open Claude Desktop, and enter the following prompt: ```shell Create a notebook named myrag in personal space. In this notebook, write a python code to create a RAG application. The RAG application will use arxiv papers as data source, SingleStore as vector database, Gemma model as the LLM, and use LangChain orchestration framework. ``` Once the prompt is submitted, the following tools will run to perform the relative task: * `choose_organization`: Selects the organization. * `set_organization`: Sets the organization. * `create_notebook_file`: Creates a notebook with a comprehensive RAG application that uses ArXiv papers, SingleStore as the vector database, Gemma as the LLM, and LangChain orchestration framework. * `upload_notebook_file`: Uploads the notebook to your personal space with the name **myrag**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7fb48f9fe8745de0/6a7ada97227b1c5eb358ce17/Screenshot_2025-07-31_at_5_55_25_PM-4zibTx.png) The notebook is created with the MCP integration. You can view the notebook in your personal space in the [Cloud Portal](http://portal.singlestore.com). ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltf8cc1dcdfe4d7596/6a7ada96227b1c075458ce15/Screenshot_2025-07-31_at_5_57_52_PM-f2iblS.png) ## Query a Database This example creates a table, ingests data into the table, and queries the inserted data.Before proceeding with this example, create a database named `sales` and an associated table. Open Claude Desktop, and update your cluster, cluster group, and organization details in the following prompt: ```shell Create a new database named sales in under of , and then create a table called SalesData with the following columns: - Date: Text - Store_ID: Big integer (20), nullable - ProductID: Text - Product_Name: Text, - Product_Category: Text - Quantity_Sold: Big integer (20), nullable - Price: Float, nullable - Total_Sales: Float, nullable ``` Once the prompt is submitted, the following tools will run to perform the relative task: * `choose_organization`: Checks the organization information. * `set_organization`: Sets the organization to the specified organization. * `cluster_groups_info`: Checks and finds information on the specified cluster group. * `clusters_info`: Checks and finds information on the specified cluster. * `run_sql`: Creates the specified database. * `run_sql`: Creates the specified tables. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltf69ddddf4bed3514/6a7ada9c80ee3807ba6078de/Screenshot_2025-07-31_at_4_44_57_PM-4dl8Wf.png) To ingest data using a pipeline, enter the following prompt: ```shell Create a pipeline SalesData_pipeline using 's3://singlestoreloaddata/SalesData/*.csv' ``` Once the prompt is submitted, the following tool will run to perform the relative task: * `run_sql`: Creates the pipeline in the `sales` database. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt69de16f786efe8a8/6a7ada9a3666876afb27e36d/Screenshot_2025-07-31_at_4_50_10_PM-VvE68M.png) To start the pipeline, enter the following prompt: ```shell Start the pipeline ``` Once the prompt is submitted, the following tool will run to perform the relative task: * `run_sql`: Starts the pipeline. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltec561dd86a81c966/6a7ada9977b034bba23f3f84/Screenshot_2025-07-31_at_4_52_57_PM-RT12v0.png) To query the data in the table, enter the following prompt: ```shell Identify the top three stores with the highest sales in the Electronics category. ``` Once the prompt is submitted, the following tool will run to perform the relative task: * `run_sql`: Queries the `SalesData` table to find the top three stores with the highest sales in the Electronics category. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltd200fe1f89674f17/6a7adadc14d87c07ee3c76c8/Screenshot_2025-07-31_at_5_01_25_PM-AiTGqD.png) ## Create a Scheduled Job This example creates a notebook in shared space and schedules the job. Open Claude Desktop, and update your cluster, cluster group, and organization details in the following prompt: ``` Create a notebook named SalesData in the shared space. In the notebook, use the SalesData table from the sales database in under of , to generate appropriate visualizations based on the data. ``` Once the prompt is submitted, the following tools will run to perform the relative task: * `choose_organization`: Checks the organization information. * `set_organization`: Sets the organization to the specified organization. * `cluster_groups_info`: Checks and finds information on the specified cluster group. * `clusters_info`: Checks and finds information on the specified cluster. * `run_sql`: Explores SalesData table structure. * `run_sql`: Understands the sample data of the table. * `run_sql`: Understands the aggregate information of the table. * `run_sql`: Explores the distinct product categories. * `create_notebook_file`: Creates the **SalesData** notebook file. * `upload_notebook_file`: Uploads the **SalesData** notebook file in shared space. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt28f396ae4f103140/6a7adade14d87c74ec3c76ca/Screenshot_2025-08-08_at_5_37_31_PM-a3s7N6.png) The notebook is created in the shared space in the Cloud Portal. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9e6c064a19aec494/6a7ada909f52511a5965e8de/Screenshot_2025-08-08_at_5_43_13_PM-bJHGss.png) To schedule the notebook, enter the following prompt: ```shell Create a schedule job using the SalesData notebook, with the name of job as SalesJob, start executions "Now", with "Weekly" "Recurring" schedule mode. Save all the execution results. ``` Once the prompt is submitted, the following tool will run to perform the relative task: * `create_job_from_notebook`: Creates a scheduled job using SalesData notebook in weekly recurring schedule mode. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt20f8bb8287d70ac5/6a7ada922f00b23f10ef361c/Screenshot_2025-08-08_at_5_38_03_PM-600s2e.png) *** Modified at: May 13, 2026 Source: [/cloud/ai/singlestore-mcp-server/examples/](https://docs.singlestore.com/cloud/ai/singlestore-mcp-server/examples/) (An index of the documentation is available at /llms.txt) --- # Connect to SingleStore SingleStore is MySQL wire-compatible and offers a wide range of tools and integrations to connect to your database. In most cases, you can connect to your SingleStore Helios deployment from MySQL-compatible tools without any customization using the [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md). ## Develop with SingleStore SingleStore simplifies the process of connecting to your database and starting application development using various libraries: | [C/C++](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-c-c.md) | [C#/.NET](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core.md) | [LangChain](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-langchain.md) | | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Java/JDBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc.md) | [Laravel](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-laravel.md) | [Node.js](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js.md) | | [Perl](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl.md) | [PHP](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-php.md) | [Python](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python.md) | | [Ruby](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby.md) | [Rust](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-rust.md) | [Go](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-go.md) | Refer to [Connect with Application Development Tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md) for more information. ## SQL Client Drivers SingleStore offers native drivers which may provide enhancements and performance improvements over standard MySQL drivers. ## Native Client Drivers SingleStore provides the following native client drivers: * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) * [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) ## MySQL Client Drivers Refer to [Connect with MySQL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql.md) for supported MySQL Clients and MySQL-compatible tools. ## Data Access APIs You can connect to your SingleStore Helios deployments via a data access API using the following: * [Connect with Data API](https://docs.singlestore.com/cloud/reference/data-api.md) * [Connect with Hasura](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-hasura.md) ## Integrated Clients * [Connect with Integrated Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) * [Connect with SQL Editor](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-sql-editor.md) ## Supported SQL Clients SingleStore supports the following following SQL clients: * [Connect with DBeaver](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbeaver.md) * [Connect with DbVisualizer](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbvisualizer.md) * [Connect with Sequel Pro](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sequel-pro.md) * [Connect with SQL Developer](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-developer.md) * [Connect with SQL Workbench](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-workbench.md) ## Supported Analytics and BI Tools | [Alteryx Designer](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer/) | [IBM Cognos Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics/) | [Denodo](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo/) | | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | [Dremio](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio/) | [Oracle GoldenGate](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate/) | [GoodData](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata/) | | [Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker/) | [MicroStrategy](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy/) | [Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/) | | [Sisense](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense/) | [Superset](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset/) | [Tableau Desktop](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau/) | | [ThoughtSpot](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot/) | [Tibco Spotfire](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire/) | [Zoho Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics/) | Refer to [Connect with Analytics and BI Tools](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools.md) for more supported tools. ## Supported Data Integration and ETL Tools | [AWS Glue](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-aws-glue.md) | [Apache Beam](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-beam.md) | [Apache Flink](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-flink.md) | | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | [Apache NiFi](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-apache-nifi.md) | [Azure Data Factory (ADF)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-data-factory-adf.md) | [Azure Event Hubs](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-event-hubs.md) | | [Data Virtuality Pipes](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-data-virtuality-pipes.md) | [dbt](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/transform-data-with-dbt.md) | [Estuary Flow](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-estuary-flow.md) | | [Fivetran](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-fivetran.md) | [Flyway](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway.md) | [HVR](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-hvr.md) | | [Informatica PowerCenter](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-informatica-powercenter.md) | [Liquibase](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-liquibase.md) | [Qlik Replicate](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate.md) | | [Kafka Sink](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector.md) | [Spark](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark.md) | [StreamSets](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets.md) | | [Tableau Prep](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-prep.md) | | | Refer to [Integrate with SingleStore Helios](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) for more supported tools. ## Supported IDEs SingleStore supports the following IDEs: * [Connect with JetBrains DataGrip](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-jetbrains-datagrip.md) * [Connect with Visual Studio Code](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-visual-studio-code.md) ## Other Connection Methods You can also connect to your SingleStore Helios deployments via the following: * [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md) ## Private Service and Networks To configure private connections using AWS PrivateLink, Azure Private Link, and Google Private Service Connect, refer to [Connect with Private Services and Networks](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections.md). > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. ## In this section * [Client and Driver Downloads](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads.md) * [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) * [Connect with DBeaver](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbeaver.md) * [Connect with DbVisualizer](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbvisualizer.md) * [Connect with Hasura](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-hasura.md) * [Connect with JetBrains DataGrip](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-jetbrains-datagrip.md) * [Connect with MySQL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql.md) * [Connect with SQL Developer](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-developer.md) * [Connect with the SingleStore Client](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-singlestore-client.md) * [Connect with the SQL Editor](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-sql-editor.md) * [Connect with Sequel Pro](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sequel-pro.md) * [Connect with SQL Workbench](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-workbench.md) * [Connect with Visual Studio Code](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-visual-studio-code.md) * [Connect with Data API](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-data-api.md) * [Connect with Private Services and Networks](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections.md) * [Connect to Your Cluster - Related Topics](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-to-your-cluster-related-topics.md) *** Modified at: June 15, 2026 Source: [/cloud/connect-to-singlestore/](https://docs.singlestore.com/cloud/connect-to-singlestore/) (An index of the documentation is available at /llms.txt) --- # Client and Driver Downloads SingleStore recommends using the following drivers, connectors, and clients. ## Language Database Clients/Drivers | Driver Connector | Reference | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Go | [GoSingleStoreDriver](https://github.com/singlestore-labs/go-singlestore-driver) | | Java/JDBC | [SingleStoreJDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) | | ODBC | [SingleStoreODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) | | .NET and .NET Core | [SingleStoreConnector for .NET and .NET Core](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core.md) | | Node.js | [SingleStoreNode.js Driver](https://github.com/singlestore-labs/singlestore-nodejs) | | Python | [SingleStorePython Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) | | R2DBC | [SingleStoreR2DBC Connector](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-r-2-dbc.md) | Refer to [Connect with Application Development Tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md) for more supported drivers/languages, including third-party drivers and connectors. ## ORMs and Application Frameworks | Framework | Reference | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Entity Framework Core | [SingleStoreEntity Framework Core Provider](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core/connect-using-entity-framework-core.md) | | Django | [Django Connector forSingleStore](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-django.md) | | Hasura | [HasuraSingleStoreData Connector](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-hasura.md) | | Hibernate | [HibernateSingleStoreDialect](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/hibernate-singlestore-dialect.md) | Refer to [Connect with Application Development Tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md) for more supported ORMs and application frameworks. ## In this section * [Client Version Support Policy](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads/client-version-support-policy.md) *** Modified at: April 20, 2026 Source: [/cloud/connect-to-singlestore/client-and-driver-downloads/](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads/) (An index of the documentation is available at /llms.txt) --- # Client Version Support Policy SingleStore is MySQL compatible and provides native software (drivers, connectors, etc.) which may provide specific enhancements to SingleStore's performance or features. The native software must be installed on the system to leverage these optimizations. SingleStore regularly releases new versions to address bugs and introduce new features. SingleStore recommends using the latest version of SingleStore drivers and connectors. SingleStore maintains a driver and connector version support policy with the following guidelines: * SingleStore generally supports each driver and connector release for at least two years, except when more recent releases provide critical fixes for security or performance issues. SingleStore may not support connectivity with versions older than this. * SingleStore recommends using versions released within the scope of SingleStore's support policy. SingleStore does not distribute versions that are no longer supported. * Latest driver and connector releases provide performance enhancements, new features, and bug fixes. They are not retroactively made available in the older versions. * For critical fixes, SingleStore may require updating the driver or connector to the latest version or a version higher than the oldest supported version. This policy does not apply to drivers and connectors provided by third-party companies outside of SingleStore's control. Contact the respective third-party provider for their latest support policies. *** Modified at: July 29, 2024 Source: [/cloud/connect-to-singlestore/client-and-driver-downloads/client-version-support-policy/](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads/client-version-support-policy/) (An index of the documentation is available at /llms.txt) --- # Connect to Your Cluster - Related Topics * [Create a Database](https://docs.singlestore.com/cloud/create-a-database.md) * [Load Data](https://docs.singlestore.com/cloud/load-data.md) * [Query Data](https://docs.singlestore.com/cloud/query-data.md) *** Modified at: February 6, 2025 Source: [/cloud/connect-to-singlestore/connect-to-your-cluster-related-topics/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-to-your-cluster-related-topics/) (An index of the documentation is available at /llms.txt) --- # Connect with Data API SingleStore provides the Data API to execute SQL statements against your database over an HTTP connection. The Data API can be used to integrate SingleStore with serverless architecture, develop custom applications, and build seamless integrations with applications. SingleStore's Data API is authenticated to the server using the HTTP Basic and Bearer Authentication standards. To authenticate using JWTs, use the Bearer authentication standard. The Data API uses standard HTTP methods and response codes and supports URI versioning. It also uses JSON-encoded responses with support for conventional data type conversions. The Data API is accessible via a built-in HTTP server. The Data API supports all SQL statements that can be run in a direct database connection with SingleStore Helios. To learn more about SingleStore's Data API, refer to the following topics. * [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) * [Data API Authentication](https://docs.singlestore.com/cloud/reference/data-api/data-api-authentication.md) * [Access the Data API](https://docs.singlestore.com/cloud/reference/data-api/access-the-data-api.md) * [Enable the Data API](https://docs.singlestore.com/cloud/reference/data-api/enable-the-data-api.md) * [Data API Data Type Conversion](https://docs.singlestore.com/cloud/reference/data-api/data-api-data-type-conversion.md) * [Data API Error Handling](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling.md) * [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md) *** Modified at: October 18, 2024 Source: [/cloud/connect-to-singlestore/connect-with-data-api/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-data-api/) (An index of the documentation is available at /llms.txt) --- # Connect with DBeaver > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. You can manage your SingleStore Helios databases with [DBeaver](https://dbeaver.io/) using the SingleStore JDBC driver. ## Prerequisites * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md): Download the latest version of the driver from [GitHub](https://github.com/memsql/S2-JDBC-Connector/releases/latest). ## Connect with SingleStore Helios from DBeaver To connect to your cluster from DBeaver, 1. In DBeaver, select **Database > Driver Manager**. 2. On the **Driver Manager** dialog, select **MariaDB > Copy**. The **Create New Driver** dialog opens. 3. In the **Create new driver** dialog, on the **Settings** tab, enter the following details: * **Driver Name**: Enter **SingleStore**. * **Driver Type**: From the list, select **MySQL**. * **Class Name**: Specify the Class Name for the SingleStore JDBC driver: `com.singlestore.jdbc.Driver`. * **URL Template**: Enter the following template: `jdbc:singlestore://{host}[:{port}]/[{database}]`. * **Default Port**: The default is `3306`. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blteb6a0a4422459c3f/6a7ada889f52512ba665e8da/dbeaver-2Yy0Js.png) 4. On the **Libraries** tab, 1. Remove the existing MariaDB dependencies. 2. Select **Add file**. Select the SingleStore JDBC driver `.jar` file downloaded earlier, and select **Open**. Alternatively, you can select **Add Artifact**, and then add the following dependency for the SingleStore JDBC driver under **Dependency Declaration**. ```xml com.singlestore singlestore-jdbc-client 1.1.9 ``` Update the version in the dependency declaration with the latest version. 5. Select **OK**. The SingleStore driver is now configured in DBeaver. 6. In DBeaver, select **New Database Connection > SingleStore**. 7. On the **Connect to a database** dialog, enter the connection configuration of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. 8. Select **Test Connection** to test your connection. The connection is now configured, and you can connect to your SingleStore databases from DBeaver. *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-dbeaver/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbeaver/) (An index of the documentation is available at /llms.txt) --- # Connect with DbVisualizer You can manage your SingleStore database using [DbVisualizer](https://www.dbvis.com/database/singlestore/). ## Prerequisites * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) (You may also install the SingleStore JDBC driver from DbVisualizer, select **Tools > Driver Manager**. Find the **SingleStore** driver from the list, and install it.) ## Connect with DbVisualizer You need to setup a **Database Connection** in DbVisualizer to access your SingleStore database. 1. In DbVisualiser, select **Database > Create Database Connection**. 2. From the **Driver Name** list, select **SingleStore**. 3. In the **SingleStore** tab, enter the following details in the corresponding fields: * `Database Server`: the endpoint of the SingleStore cluster Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `Database Port`: the port used by the SingleStore Helios cluster (default: `3306`) * `Database`: the name of the SingleStore database * `Database Userid`: ID or name of the SingleStore user * `Database Password`: password for the SingleStore user ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2dec8ef1fadc49dc/6a7adac16c6eacf593f08cf7/dbvisualizer2-uJumia.png) 4. Select the **Connect** button. You may also select **Ping Server** to verify the connectivity between DbVisualizer and SingleStore. You can now explore your SingleStore databases using DbVisualizer. *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-dbvisualizer/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-dbvisualizer/) (An index of the documentation is available at /llms.txt) --- # Connect with Hasura The Hasura SingleStore Data Connector ("the connector") connects your SingleStore database to Hasura Data Delivery Network (DDN) and enables you to query and update your SingleStore data via a GraphQL API. Once the connector is initialized, it automatically deploys a GraphQL API on top of your SingleStore data. The Hasura SingleStore Data Connector is built using the [Native Data Connector SDK for TypeScript](https://github.com/hasura/ndc-sdk-typescript), and it implements Hasura's [Native Data Connectors](https://github.com/hasura/ndc-spec) specification. ## Prerequisites * A [Hasura Cloud](https://console.hasura.io/) account. * Install the [DDN CLI](https://hasura.io/docs/3.0/cli/installation) and [Docker](https://docs.docker.com/engine/install/). * Create a [supergraph](https://hasura.io/docs/3.0/getting-started/init-supergraph). * Create a [subgraph](https://hasura.io/docs/3.0/getting-started/init-subgraph). * An active SingleStore Helios deployment. ## Connect to SingleStore To connect to your SingleStore deployment using the connector: 1. Set the context for an existing subgraph. 2. Initialize the connector. ```shell ddn connector init -i ``` 3. Specify the connection configuration for your SingleStore deployment using the following environment variables: | Variable Name | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SINGLESTORE_HOST` | Hostname or IP address of yourSingleStore Heliosdeployment.Refer to[SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md)to determine the endpoint (`host:port`) of your deployment. | | `SINGLESTORE_PORT` | Port of yourSingleStore Heliosdeployment.Default is`3306`. | | `SINGLESTORE_USER` | Name of theSingleStoredatabase user with which to access the database. | | `SINGLESTORE_PASSWORD` | Password for theSingleStoredatabase user | | `SINGLESTORE_DATABASE` | Name of theSingleStoredatabase to connect with. | Refer to the [Hasura SingleStore Data Connector](https://github.com/singlestore-labs/singlestore-hasura-connector?tab=readme-ov-file#using-the-singlestore-connector) GitHub repository for information on additional configuration options and connection pool options. 4. Once the connector is initialized, [introspect the source database](https://hasura.io/docs/3.0/cli/commands/ddn_connector_introspect/) to update the configuration. 5. Add your [models](https://hasura.io/docs/3.0/cli/commands/ddn_model_add/), [commands](https://hasura.io/docs/3.0/cli/commands/ddn_command_add/), and [relationships](https://hasura.io/docs/3.0/cli/commands/ddn_relationship_add/). 6. Create a new [build](https://hasura.io/docs/3.0/cli/commands/ddn_supergraph_build_local/), and [run](https://hasura.io/docs/3.0/cli/commands/ddn_run/) the scripts using the connector. The connector is now configured and ready. ## Supported Features | Feature | Supported | | --------------------------------- | --------- | | Native queries and logical models | | | Simple object query | ✔ | | Filter or search | ✔ | | Simple aggregation | ✔ | | Sort | ✔ | | Paginate | ✔ | | Table relationships | ✔ | | Views | ✔ | | Distinct | ✔ | | Remote relationships | ✔ | | Mutations | | ## Limitations SingleStore does not support foreign keys. Therefore, relationships between tables must be manually defined by appending the relationship information to the `.hml` files. Refer to [Relationships](https://hasura.io/docs/3.0/supergraph-modeling/relationships/) for more information. ## References * [SingleStore Data Connector](https://hasura.io/connectors/singlestore) * [Hasura DDN Documentation](https://hasura.io/docs/) *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-hasura/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-hasura/) (An index of the documentation is available at /llms.txt) --- # Connect with JetBrains DataGrip > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. Use [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) to connect to your SingleStore databases from the JetBrains DataGrip IDE. DataGrip natively supports the SingleStore JDBC driver; select SingleStore from the list of supported drivers and then configure SingleStore as a data source. ## Prerequisites * [Install](https://www.jetbrains.com/help/datagrip/installation-guide.html) JetBrains DataGrip. * [Download](https://github.com/memsql/S2-JDBC-Connector/releases/latest) the latest version of the SingleStore JDBC Driver (`singlestore-jdbc-client-.jar`). * An active SingleStore Helios deployment. ## Configure the Connection to SingleStore To connect to SingleStore from JetBrains DataGrip: 1. Open JetBrains DataGrip, and then select **+** (icon) **> Driver**. Alternatively, select **File > Data Sources > Drivers** (tab). 2. On the **Data Sources and Drivers** dialog, select **SingleStore**. 3. Under **Driver Files**, select **+** (icon) **> Custom JARs**. 4. Select and upload the downloaded SingleStore JDBC driver `.jar` file. 5. Select **Create Data Source**. 6. On the SingleStore data source page, enter the connection configuration of your SingleStore deployment: * **Host**: Hostname or IP address of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Default is `3306`. * **User**: Username of the SingleStore database user. * **Password**: Password for the SingleStore database user. * **Database**: Enter the name of the SingleStore database to connect with. 7. Select **Test Connection** to test connectivity to the SingleStore database. 8. Upon successfully connecting to SingleStore, select **OK** to save the configuration. The SingleStore connection is configured and ready. *** Modified at: November 7, 2025 Source: [/cloud/connect-to-singlestore/connect-with-jetbrains-datagrip/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-jetbrains-datagrip/) (An index of the documentation is available at /llms.txt) --- # Connect with MySQL SingleStore is wire-compatible with MySQL so you can connect using most other MySQL compatible tools/languages. You’ll need to specify the `host`, `port`, `user`, `password`, and possibly the `database`. ## In this section * [Connect with MySQL Client](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client.md) * [Connect with Any MySQL Compatible Tool to SingleStore Helios](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-any-mysql-compatible-tool-to-singlestore-helios.md) * [Connect with MySQL Workbench](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-workbench.md) *** Modified at: July 27, 2021 Source: [/cloud/connect-to-singlestore/connect-with-mysql/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/) (An index of the documentation is available at /llms.txt) --- # Connect with Any MySQL Compatible Tool to SingleStore Helios In general, if the client you are using to connect supports the MySQL protocol, you can use it to connect to SingleStore Helios. For example, if you connect Wordpress to MySQL, you can probably use most of the same configuration to connect to SingleStore Helios. Ensure that the connection details are correct. Refer to [Common Issues](https://docs.singlestore.com/#section-idm4564738306016032548222614039.md) to troubleshoot some common connectivity issues. > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. ## Connection Details You’ll need to specify the `host`, `port`, `user`, `password`, and possibly the `database`. * `host`: IP address or hostname of your cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `port`: Port of the SingleStore deployment. Default is `3306`. * `user`: Username of the SingleStore database user. For example, admin. ## Common Issues Some of the issues you may encounter, and how to solve them: * Wrong Authentication Adapter: MySQL v8 authentication has some issues in SingleStore Helios and SingleStore recommends that you use a newer version. Sometimes you can work around this by specifying `--default-auth=mysql_native_password` in the connection details. * Binary Protocol: Use the text protocol when connecting to SingleStore Helios. You may need to disable server-side prepared statements (`useServerPrepStmts=false`). * Connection Failed: Check the firewall configuration and perform other general connectivity troubleshooting. *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-mysql/connect-with-any-mysql-compatible-tool-to-singlestore-helios/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-any-mysql-compatible-tool-to-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Connect with MySQL Client > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. SingleStore is wire-compliant with MySQL meaning connecting to SingleStore is as easy as connecting to a MySQL database. You may need the `-u ,` `-h `, and `-P ` flags to ensure you are connecting to SingleStore properly. For example: ```shell mysql -u admin -h svc-xxxx-ddl.aws-oregon-1.svc.singlestore.com -P 3306 --default-auth=mysql_native_password -p ``` where, * `-u`: Specifies the name of the SingleStore database user. * `-h`: Specifies the IP address or hostname of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `-P`: Specifies the port to connect to. Default: `3306`. * `--default-auth=mysql_native_password`: (Optional) For proper authentication. You may also use the `--prompt` flag to change the prompt in the interactive session, simply to help distinguish SingleStore from MySQL: ```shell mysql -u admin -h svc-[...].svc.singlestore.com -P 3306 --default-auth=mysql_native_password -p --prompt="singlestore> " ``` The output when you connect should look similar to this: ```sql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 6 Server version: 5.5.8 MemSQL source distribution (compatible; MySQL Enterprise & MySQL Commercial) Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. singlestore> ``` These commands show more information about the connection and the server: Note that `SHOW STATUS LIKE '%License%';` provides additional license information. ```sql status ``` ```output -------------- mysql Ver 8.0.23 for osx10.15 on x86_64 (Homebrew) Connection id: 15507 Current database: Current user: admin@ SSL: Cipher in use is AES256-SHA Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.5.58 MemSQL source distribution (compatible; MySQL Enterprise & MySQL Commercial) Protocol version: 10 Connection: svc-9be5e8a1-8a5f-4aa7-b75d-a395570828f2-ddl.aws-oregon-1.svc.singlestore.com via TCP/IP Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 TCP port: 3306 Binary data as: Hexadecimal -------------- ``` ```sql SHOW DATABASES; ``` ```output +--------------------+ | Database | +--------------------+ | cluster | | information_schema | | memsql | | test | +--------------------+ 4 rows in set (0.03 sec) ``` > **❗ Important**: The `@@version` and `@@version_comment` global variables are for compatibility with MySQL clients, which expect certain minimum version numbers and strings to be present. To check the real version of SingleStore Helios you are running, use the `@@memsql_version` variables instead. To check the license you are running, see the `license_key` and `license_type` status codes. ## In this section * [MySQL Command-Line Considerations](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/mysql-command-line-considerations.md) * [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md) * [MariaDB Command-line Client from MariaDB Server Version 10.3.12 (GPLv2)](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/mariadb-command-line-client-from-mariadb-server-version-10-3-12-gplv-2.md) *** Modified at: July 23, 2026 Source: [/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/) (An index of the documentation is available at /llms.txt) --- # Connect with MySQL Workbench > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. To connect to SingleStore Helios from MySQL Workbench: 1. Open MySQL Workbench and create a new MySQL connection. 2. In the **Setup New Connection** dialog, enter the following details: * **Connection Name**: A name for your connection, for example **SingleStoreDB**. * **Hostname**: The IP address or hostname of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Username**: The SingleStore database username. Enter `admin`. * **Password**: The password for the SingleStore database user. > **📝 Note**: For proper authentication, you may need to specify `defaultAuth=mysql_native_password` in the **Others** field under the **Advanced** tab. * **Port**: Default is `3306`. 3. Select **Test Connection** to test the connection. Select **OK** to save the connection. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt39d7f5fe94c47ace/6a7adba926618c7053843019/mysql_workbench1-vhDpFA.png) *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-workbench/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-workbench/) (An index of the documentation is available at /llms.txt) --- # Connect with Sequel Pro > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. To connect to SingleStore Helios from Sequel Pro: 1. Open Sequel Pro and configure a new connection using **Quick Connect**. 2. On the **Quick Connect** window, enter the following details: * **Name**: A name for your connection, for example **SingleStoreDB**. * **Host **: The IP address or hostname of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Username**: The SingleStore database username. Enter `admin`. * **Password**: The password for the SingleStore database user. * **Database**: The name of the SingleStore database. * **Port**: Default is `3306`. 3. Select **Test Connection** to test the connection. Select **Connect** to connect to your SingleStore Helios database. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc3d89854967050e1/6a7adbd22f00b24c14ef3681/sequel_pro-qzarGw.png) *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-sequel-pro/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sequel-pro/) (An index of the documentation is available at /llms.txt) --- # Connect with SQL Developer > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. To connect to SingleStore Helios from SQL Developer: 1. Download a MySQL connector for Java, for example, [MySQL Connector Java 8.0.24](https://packages.atlassian.com/maven-external/mysql/mysql-connector-java/8.0.24/mysql-connector-java-8.0.24.jar). 2. Open SQL Developer, and add a new third-party connector. Select **Preferences > Database > Third Party Drivers**. 3. Select **Add Entry**, and then add the **.jar** file downloaded earlier. 4. Create a new connection, select **Connections > New Connection**. 5. In the **New / Select Database Connection** dialog, select or enter the following information: * **Name**: A name for your connection, for example **SingleStoreDB**. * **Database Type**: From the list, select **MySQL**. * **Hostname**: The IP address or hostname of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Username**: The SingleStore database username. Enter `admin`. * **Password**: The password for the SingleStore database user. * **Port**: Default is `3306`. 6. Select **Connect**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcf55bf645d77ea14/6a7ada643ce8e20132cea0f1/16083b57631f91-kxQ6wa.png) *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-sql-developer/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-developer/) (An index of the documentation is available at /llms.txt) --- # Connect with SQL Workbench > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. To connect to SingleStore Helios from SQL Workbench: 1. Open SQL Workbench, and then open the **Connectors** window. **Note**: Some users have reported issues connecting with SQL Workbench v8 and have recommended using v6 instead. 2. In the **Select Connection Profile** window, select or enter the following information: * A name for your connection, for example **test-connection**. * **Driver**: From the list, select **MySQL (com.mysql.jdbc.Driver)**. This Driver requires the [MySQL Java connector](http://dev.mysql.com/downloads/connector/j/). * **URL**: Enter the connection string in the `jdbc:mysql://:` format. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Username**: The SingleStore database username. Enter `admin`. * **Password**: The password for the SingleStore database user. 3. Complete the other fields as required. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt84942213f9fdc71b/6a7ada6cde23153e3dfcc494/16083b576527d1-jNLaA2.png) 4. Select **OK**. The SQL Workbench window opens. You can issue queries in the top pane and view results in the bottom pane. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt242075ba9ddb477e/6a7ada5a26618c1e1e842fba/16083b5765a052-zNvPE2.png) ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7ed7e3a0174cd257/6a7ada66c7a40c22a830ea5e/16083b57660cb0-539IVz.png) *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-sql-workbench/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-workbench/) (An index of the documentation is available at /llms.txt) --- # Connect with the SingleStore Client > **📝 Note**: When using the SingleStore client, or a third-party SQL client, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. The SingleStore client, similar to the MySQL client, is a lightweight Linux application that allows you to connect to SingleStore Helios and run SQL queries directly from a terminal session. Use `singlestore --help` to for help and to view additional connection options. ## Install SingleStore Client * *Red Hat Distribution* ## Online InstallationFor online installations where the target host can access the SingleStore YUM repository, perform the following steps.1) Add the SingleStore repository to your repository list. ```shell sudo yum-config-manager --add-repo https://release.memsql.com/production/rpm/x86_64/repodata/memsql.repo ``` 2) Verify that the SingleStore repo information is listed under `repolist`. ```shell sudo yum repolist ``` 3) Verify that the `which` package installed. This is used during the install process to identify the correct package type for your installation. ```shell rpm -q which ``` If `which` is not installed, it must be installed before proceeding. ```shell sudo yum install -y which ``` 4) Install the SingleStore client. ```shell sudo yum install -y singlestore-client ```## Offline InstallationDownload the SingleStore client package onto a host that can access your SingleStore Helios cluster.Install the SingleStore client.```shell sudo rpm -ivh /tmp/singlestore-client--.x86_64.rpm ``` * *Debian Distribution* ## Online InstallationFor online installations where the target host can access the SingleStore APT repository, perform the following steps.1) SingleStore packages are signed to ensure integrity, so the GPG key needs to be added to this host. When done, verify that the SingleStore signing key has been added using `apt-key list`. ```shell wget -O - 'https://release.memsql.com/release-aug2018.gpg' 2>/dev/null | sudo apt-key add - && apt-key list ``` **Without using `apt-key`**: ```shell wget -q -O - 'https://release.memsql.com/release-aug2018.gpg' | sudo tee /etc/apt/trusted.gpg.d/memsql.asc 1>/dev/null ``` 2) Verify that `apt-transport-https` is installed. ```shell apt-cache policy apt-transport-https ``` If `apt-transport-https` is not installed, it must be installed before proceeding. ```shell user-shell sudo apt -y install apt-transport-https ``` 3) Add the SingleStore repository to retrieve its packages. ```shell echo "deb [arch=amd64] https://release.memsql.com/production/debian memsql main" | sudo tee /etc/apt/sources.list.d/memsql.list ``` 4) Install the SingleStore client. ```shell sudo apt update && sudo apt -y install singlestore-client ```## Offline InstallationDownload the SingleStore client package onto a host that can access your SingleStore Helios cluster.Install the SingleStore client.```shell sudo dpkg -i /tmp/singlestore-client___amd64.deb ``` * *Tarball* ## Download the SingleStore ClientDownload the SingleStore client tarball file onto a host that can access your SingleStore Helios cluster.## Transfer the SingleStore ClientTransfer the SingleStore client tarball file into a dedicated `singlestore` directory on the target host (typically the main deployment host when deploying SingleStore) that has been configured so that non-`sudo` users can access it, such as `/opt/singlestore`.## Unpack the SingleStore ClientUnpack the SingleStore client into a directory of your choice, such as or `/opt/singlestore`.```shell tar xzvf singlestore-client--.x86_64.tar.gz ```Using symbolic links (or "symlinks") can make using and upgrading the SingleStore client easier. Performing the following steps will allow the `singlestore` command to be run from anywhere on the filesystem, even after an upgrade by updating the symlink.1) Navigate to the directory that contains the unpacked SingleStore client tarball file and create a `singlestore-client` symlink that points to the SingleStore client directory. ```shell ln -s singlestore-client-- singlestore-client ``` 2) Verify that the symlink has been created. ```shell ls -l ``` ```output singlestore-client -> singlestore-client-1.0.6-c3803db03b ``` 3) Update the `PATH` environment variable with the path to the new `singlestore-client` directory. This path can also be added to your shell startup file so you won't have to run this command each time you log in. ```shell export PATH=/opt/singlestore/singlestore-client:$PATH ``` 4) You may now run the `singlestore` command from anywhere on the filesystem. ```shell singlestore ``` ## Connect to Your SingleStore Helios Cluster The `singlestore_bundle.pem` file is required to configure an SSL connection. Once downloaded, this file can be used to connect to all of your SingleStore Helios clusters via SSL. 1. Download the `singlestore_bundle.pem` file. 1. Log into the [Cloud Portal](https://portal.singlestore.com). 2. Navigate to **Clusters**. 3. Select **Connect > CLI Client** in the **Actions** column for your cluster. 4. Under **TLS/SSL Certificate**, select **Download**. This will download the `singlestore_bundle.pem` file to your computer. 2. To confirm that the connection is using SSL, run the following command to check the connection status. ``` singlestore -h -P -u -p --ssl-ca=singlestore_bundle.pem \ -e status ``` where: * `client-endpoint` is the SingleStore Helios endpoint for your cluster * `port` is the port number to connect on (typically `3306`) * `database-user` is either the [database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to connect as, or the default `admin` user (which was created with the cluster) * `database-user-password` is the database user password. You may receive the following message when including the database user password on the command line: ``` singlestore-client: [Warning] Using a password on the command line interface can be insecure. ``` To prevent this message, leave the database user password blank and you will be prompted for the password. From the output, review the `SSL` line for the cipher in use. ``` -------------- /usr/lib/singlestore-client/singlestore-client Ver 14.14 Distrib 5.7.17, for Linux (x86_64) using EditLine wrapper Connection id: 17323 Current database: Current user: admin@ SSL: Cipher in use is AES128-GCM-SHA256 Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.7.32 MemSQL source distribution (compatible; MySQL Enterprise & MySQL Commercial) Protocol version: 10 Connection: via TCP/IP Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 TCP port: 3306 -------------- ``` 3. Run the following command to connect to your SingleStore Helios cluster. Note that this is the same command and parameters from Step 2, only without the `-e status` flag. ```shell singlestore -h -P -u -p --ssl-ca=singlestore_bundle.pem ``` When connected, the following (or similar) output will be displayed. ``` Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 10141 Server version: 5.7.32 SingleStoreDB source distribution (compatible; MySQL Enterprise & MySQL Commercial) Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. singlestore> ``` ## Upgrade SingleStore Client * *Red Hat Distribution* To upgrade the SingleStore client, use the same steps for upgrading any RPM package.**Online**```shell sudo yum update -y singlestore-client ```**Offline**1) Download the SingleStore client package. 2) Upgrade the SingleStore client. ```shell sudo rpm -Uvh /tmp/singlestore-client--.x86_64.rpm ``` * *Debian Distribution* To upgrade the SingleStore client, use the sane steps for upgrading any Debian package.**Online**1) SingleStore packages are signed to ensure integrity, so the GPG key needs to be added to this host. When done, verify that the SingleStore signing key has been added using `apt-key list`. ```shell wget -O - 'https://release.memsql.com/release-aug2018.gpg' 2>/dev/null | sudo apt-key add - && apt-key list ``` **Without using `apt-key`**: ```shell wget -q -O - 'https://release.memsql.com/release-aug2018.gpg' | sudo tee /etc/apt/trusted.gpg.d/memsql.asc 1>/dev/null ``` 2) Upgrade the SingleStore client. ```shell sudo apt update && sudo apt install -y singlestore-client ```**Offline**1) Download the SingleStore client package. 2) Upgrade the SingleStore client. ```shell sudo dpkg -i /tmp/singlestore-client___amd64.deb ``` * *Tarball* To upgrade the SingleStore client, you must manually replace the existing SingleStore client with a new one.1) Obtain the latest version of the SingleStore client. ```shell curl https://release.memsql.com/production/index/memsqlclient/latest.json ``` The JSON returned contains a relative path to the `singlestore-client-tar` file in the following format. ```shell "Path": "production/tar/x86_64/singlestore-client--.x86_64.tar.gz" ``` 2) Use `wget` to download the file by copying, pasting, and appending the path to . For example: ```shell wget https://release.memsql.com/production/tar/x86_64/singlestore-client--.x86_64.tar.gz ``` As an alternative, you may download the latest version of the SingleStore client tarball file using the following link. 3) Place this SingleStore client tarball file on your device where the existing SingleStore client has been installed. 4) Unpack the SingleStore client tarball file. This directory will now contain two versions of the SingleStore client: the existing version, and the new version. ```shell tar xzvf singlestore-client--.x86_64.tar.gz ```Using symbolic links (or "symlinks") can make using and upgrading the SingleStore client easier. Performing the following steps will allow the `singlestore` command to be run from anywhere on the filesystem, even after an upgrade.1) Check if a `singlestore-client` symlink already exists within the current directory. ```shell ls -l ``` If so, remove the existing `singlestore-client` symlink. ```shell rm singlestore-client ``` 2) Create a `singlestore-client` symlink by pointing it to the new version of the SingleStore client directory. ```shell ln -s singlestore-client-- singlestore-client ``` 3) Verify that the symlink has been created. ```shell ls -l ``` ```output singlestore-client -> singlestore-client-1.0.6-c3803db03b ``` 4) Update the `PATH` environment variable with the path to the new `singlestore-client` directory. This path can also be added to your shell startup file so you won't have to run this command each time you log in. ```shell export PATH=/opt/singlestore/singlestore-client:$PATH ``` 5) You may now run the `singlestore` command from anywhere on the filesystem. ```shell singlestore ```The upgrade is now complete.Should you encounter any issues with the new SingleStore client, you can simply point the `singlestore-client` symlink back to the previous version.After validating that the new version of the SingleStore client is working as expected, you may remove the previous version.```shell rm -rf singlestore-client-- ``` ## Uninstall SingleStore Client * *Red Hat Distribution* You may either uninstall the SingleStore client individually or remove it while uninstalling SingleStore Helios. If you want to completely uninstall SingleStore Helios and its related Tools packages, first [uninstall SingleStore Helios](https://docs.singlestore.com/cloud/user-and-cluster-administration/#uninstall-singlestore-helios.md) from your nodes by using the `sdb-deploy uninstall` command. This will remove the `singlestoredb-server` package for that version of SingleStore Helios. To uninstall all versions of SingleStore Helios, include the `--all-versions` flag.> **❗ Important**: The `remove` command and `-e` flag will only remove the binaries in the package and not remove any configuration files.Run the following command to uninstall the SingleStore client.**Online**```shell sudo yum remove singlestore-client ```**Offline**```shell sudo rpm -e singlestore-client ``` * *Debian Distribution* You may either uninstall the SingleStore client individually or remove it while uninstalling SingleStore Helios. If you want to completely uninstall SingleStore Helios and its related Tools packages, first [uninstall SingleStore Helios](https://docs.singlestore.com/cloud/user-and-cluster-administration/#uninstall-singlestore-helios-2.md) from your nodes by using the `sdb-deploy uninstall` command. This will remove the `singlestoredb-server` package for that version of SingleStore Helios. To uninstall all versions of SingleStore Helios, include the `--all-versions` flag).Run the following command to uninstall the SingleStore client.> **❗ Important**: The `purge` command and `-P` flag will remove the binaries in the package as well as any configuration files.To keep the configuration files, use the `remove` command or `-r` flag instead.**Online**```shell sudo apt purge singlestore-client ```**Offline**```shell sudo dpkg -P /tmp/singlestore-client___amd64.deb ``` * *Tarball* Run the following commands to uninstall the SingleStore client.1) Change to the directory where the SingleStore client has been installed. ```shell cd /opt/singlestore ``` 2) Remove the SingleStore client directory (or directories, if more than one version has been installed). ```shell rm -rf singlestore-client-- ``` *** Modified at: May 12, 2026 Source: [/cloud/connect-to-singlestore/connect-with-the-singlestore-client/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-singlestore-client/) (An index of the documentation is available at /llms.txt) --- # Connect with the SQL Editor The SQL Editor in the [Cloud Portal](https://portal.singlestore.com) allows you to easily run queries against your cluster. ## Multi-tab SQL Editor The multi-tab SQL Editor allows you to open and work with multiple SQL files in parallel within a single browser session. You can load, edit, and run queries against different clusters and databases, and save individual SQL files for future use. ## Create and Open a New SQL File **Note**: Ensure that the correct cluster and database are selected at the top of the file, and all queries are run against the intended database. 1. On the [Cloud Portal](https://portal.singlestore.com/), select **Editor** > **Open SQL Editor**. 2. Navigate to the **+ New **button on the right, and select **New SQL file**. A new tab opens with an untitled SQL Editor (for example, Untitled-1.sql). Each additional untitled tab is automatically numbered incrementally (such as Untitled-2.sql, Untitled-3.sql, etc.). ## Save a SQL File To save an untitled SQL file, select the **Save As** button on the top right. In the dialog, enter the filename and confirm its storage location. **Note**: SQL files can only be saved in the **Personal** folder. For further modifications, select **Save**, or you can rely on autosave. The editor autosaves content every 5 seconds, as indicated by the **Save** button changing to **Saved**.  If you try to close a file with unsaved changes, you will be prompted to save, discard, or cancel. Unsaved or untitled files are lost permanently when closed. ## Set a Row Limit for Results To limit the number of rows in the query result: 1. Select the three dots icon in the top right corner of the editor.From the list, select **Settings**. 2. The Settings panel shows a **Limit results to** option. 3. Select a row limit: 300,1000, or 3000. The default is 300 rows. Alternatively, you can set the row limit by running the following command in the SQL Editor: ```sql SET SESSION SQL_SELECT_LIMIT = 5000; ``` This example sets the value to 5000. ## Troubleshooting * If you experience connection errors when running queries, verify that your IP address is allowed in the cluster group's firewall settings. * For additional assistance, contact [SingleStore Support](https://support.singlestore.com/). *** Modified at: May 22, 2026 Source: [/cloud/connect-to-singlestore/connect-with-the-sql-editor/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-sql-editor/) (An index of the documentation is available at /llms.txt) --- # Connect with Visual Studio Code You can connect to your SingleStore Helios cluster from Visual Studio Code using the SQLTools SingleStore Driver (aka "SingleStore Driver for SQLTools") extension. This driver allows you to perform the following tasks: * Connect to your SingleStore Helios cluster. * Manage connections in the Database Explorer. * Run SQL queries. You can also export query results in JSON and CSV formats. * View tables, views, columns, functions, procedures, etc. * Generate `INSERT` queries. * Autocomplete SQL keywords, table and view names, column names, SingleStore built-in functions, etc. Refer to [SingleStore Driver for SQLTools](https://marketplace.visualstudio.com/items?itemName=singlestore.sqltools-singlestore-driver) for more information. ## Install the SQLTools SingleStore Driver Extension To install the SQLTools SingleStore Driver in Visual Studio Code, run the following command in the Visual Studio Code Quick Open (Cmd+p/Ctrl+p): ```vs code ext install singlestore.sqltools-singlestore-driver ``` You can also install this extension in Visual Studio Code IDE from the **Extensions** tab. ## Connect with SingleStore Helios from Visual Studio Code To connect to your cluster from Visual Studio Code, perform the following tasks: 1. In Visual Studio Code IDE, on the navigation pane, select **SQLTools**. 2. Under **Connections**, select **Add New Connection**. 3. On the **SQLTools Settings** tab, select the **SingleStore** database driver. 4. Under **Connection Settings**, enter the following details: * **Connection name** (Required): Name for this connection. * **Server Address** (Required): IP address or hostname of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port** (Required): Default is `3306`. * **Username** (Required): Username for your SingleStore Helios cluster. To test connectivity, select **TEST CONNECTION**. 5. Select **SAVE CONNECTION**. 6. Under **Review connection details**, select **CONNECT NOW**. You can now explore your SingleStore databases in Visual Studio Code. *** Modified at: September 26, 2025 Source: [/cloud/connect-to-singlestore/connect-with-visual-studio-code/](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-visual-studio-code/) (An index of the documentation is available at /llms.txt) --- # Connect with Private Services and Networks You can connect from a SingleStore Helios cluster to private services and networks. Private services/networks simplify networking and connectivity by allowing users to connect directly from a virtual/private network to SingleStore through VPC/private endpoints. Use the [SingleStore Private Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections.md) feature to configure a private connection to AWS PrivateLink. Setting up a connection to Azure Private Link and Google Private Service Connect involves actions from both your and SingleStore's end. You may need to share the details related to your cluster and Azure/Google account with SingleStore support. To make the setup process seamless, acquire the required information before reaching out to SingleStore support. The list of information required is specified in the configuration process for the respective private service/network. To successfully set up a private connection, you need to configure both the outbound and inbound connections. Additionally, SingleStore can only process the connection request when your cluster is in the `Active` state. If you need separate reader and writer endpoints for each cluster, specify this in the support ticket while configuring Azure/Google private service/networks. SingleStore recommends using a single endpoint as SingleStore internally manages (reader/writer) query forwarding to this endpoint. SingleStore recommends having hands-on experience on using private services and networks. For more information, refer to the relevant documentation: * [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/what-is-privatelink.html) * [Azure Private Link](https://learn.microsoft.com/en-us/azure/private-link/private-link-overview) * [Google Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. ## SingleStore Private Connections SingleStore Private Connections (SPC) allows you set up and configure connections to private services and networks through the [Cloud Portal](https://portal.singlestore.com/). This feature is only supported for SingleStore Helios clusters deployed using AWS, and it only supports connections to AWS PrivateLink. For more information, refer to [SingleStore Private Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections.md). ## Configure Private Connections Configure both the outbound and inbound connections to connect to any of the following private services/networks: ## AWS PrivateLink * [Connect to SingleStore Helios using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink.md) ## Azure Private Link You can connect multiple private links to a single cluster. Refer to [ExpressRoute](https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/hybrid-networking/expressroute) for more information. * [Configure Outbound Connection](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-azure-private-link.md) * [Configure Inbound Connection](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-azure-private-link.md) ## Google Private Service Connect If you have more than one subnet/zone you want to connect via Private Service Connect to your cluster, you should select each of them while configuring the load balancer backend. For more information, refer to [Configure the load balancer](https://cloud.google.com/vpc/docs/configure-private-service-connect-controls#configure-the-load-balancer). * [Configure Outbound Connection](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-google-private-service-connect.md) * [Configure Inbound Connection](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-google-private-service-connect.md) ## FAQs * **Once a private connection is set up, for example AWS PrivateLink, does the data to be ingested have to go through AWS PrivateLink?** No, this is simply a separate endpoint. * **While using Kubernetes with Google Private Service Connect, suppose that each node pool creates 3 instance groups, 1 per zone. Should you select every group available in the backend step of the create a load balancer step?** Yes, if you have more than a single subnet/zone and you want to connect via Google Private Service Connect to your cluster, you should select every available group. * **Can I connect directly to a cluster via a private link (service/network)?** No, you need a private service endpoint to connect to a cluster via a private link. You cannot directly access the endpoint of a cluster from private services and networks. ## In this section * [SingleStore Private Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections.md) * [Connect to SingleStore Helios using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink.md) * [Connect to SingleStore Helios Clusters from Private Networks/Services via Azure Private Link](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-azure-private-link.md) * [Connect out from SingleStore Helios Clusters to Private Networks/Services via Azure Private Link](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-azure-private-link.md) * [Connect to MongoDB® using Azure Private Link](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-mongodb-using-azure-private-link.md) * [Connect to SingleStore Helios Clusters from Private Networks/Services via Google Private Service Connect](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-google-private-service-connect.md) * [Connect out from SingleStore Helios Clusters to Private Networks/Services via Google Private Service Connect](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-google-private-service-connect.md) *** Modified at: June 23, 2026 Source: [/cloud/connect-to-singlestore/private-connections/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/) (An index of the documentation is available at /llms.txt) --- # Connect out from SingleStore Helios Clusters to Private Networks/Services via Azure Private Link For outbound connections, create a private link service and send the **alias** to SingleStore. To create outbound connections from SingleStore Helios to private services (like Kafka) via Azure Private Link, for tasks like ingesting data through pipelines, perform the following tasks: 1. Request the subscription ID from SingleStore. 2. Create a private link service. 3. Send the alias to SingleStore. ## Request the Subscription ID from SingleStore Contact [SingleStore Support](https://support.singlestore.com) and request the subscription ID from SingleStore. You'll need to whitelist this subscription ID while creating your private link service (as explained below). ## Create a Private Link Service 1. On the [Azure portal](https://portal.azure.com/), select **Create a resource > Private Link > Create > Create private link service**. 2. On the **Basics** tab, enter or select the following information: * **Subscription**: Your Azure subscription. * **Resource group**: The resource group corresponding to your Azure subscription. It should have the same name as your Azure subscription name. * **Name**: Name of your private endpoint service. * **Region**: Select a region in which you want to create the private link service. > **📝 Note**: Your cluster and private link service must be in the same region. 3. On the **Outbound settings** tab, select or enter the following information: * **Load balancer** * **Load balancer frontend IP address** * **Source NAT subnet** * Use defaults for other configuration settings 4. On the **Access security** tab, whitelist the subscription ID received from SingleStore. 5. On the **Review + create** tab, verify the configuration setting, and select **Create**. Your private link service is created, and you are redirected to a new page. 6. Select **Go to resource group**, and then select your private link service. 7. Copy the **alias**. ## Send the Alias to SingleStore Contact [SingleStore Support](https://support.singlestore.com) and send the alias of your private endpoint service. Provide the following information in the support ticket: * Cluster ID. SingleStore can only process the connection request when your cluster is in the `Active` state. * Region details * The **alias** of your private endpoint service * In the support ticket, specify that the request is for outbound connection ## References * [Create a private link service](https://learn.microsoft.com/en-us/azure/private-link/create-private-link-service-portal) *** Modified at: April 1, 2025 Source: [/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-azure-private-link/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-azure-private-link/) (An index of the documentation is available at /llms.txt) --- # Connect out from SingleStore Helios Clusters to Private Networks/Services via Google Private Service Connect For outbound connections, you'll send the service attachment to SingleStore. You'll also receive a project name from SingleStore so that you can whitelist the project while creating the service. To create outbound connections from SingleStore Helios to private networks/services (not accessible from the Internet, like Kafka) for tasks like ingesting data via pipelines perform the following tasks: 1. Obtain a project name from SingleStore. 2. Create a Network Load Balancer (NLB). 3. Publish the Service. 4. Send the Service Attachment to SingleStore. ## Obtain a Project Name from SingleStore Contact SingleStore Customer Support (see [Support FAQ](https://docs.singlestore.com/cloud/support/support-faq.md)) and obtain a project name for the service you are going to create. Provide the following information in the support ticket: * Cluster ID. SingleStore can only process the connection request when your cluster is in the `Active` state. * Region details * Request a project name from SingleStore * In the support ticket, specify that the request is for outbound connection You'll need to whitelist this project name while creating the service later. ## Create a Network Load Balancer 1. In the GCP console, select **Networking > Networking Services > Load Balancing > Create load balancer**. 2. Under **TCP Load Balancing**, select **Start configuration**. 3. Under **Internet facing or internal only**, select **Only between my VMs** and **Single region only**. 4. Under **Backend type**, select **Backend Service**. Select **Continue**. 5. On the **New TCP load balancer** screen, enter a **Name** for the Network Load Balancer. 6. Select **Backend configuration**, enter the **Region** and **Network** information, and then add your service's **Instance group**. > **📝 Note**: If you do not have an Instance group yet, but you do have a running service, select **GCP Console > Compute Engine > Instance groups > Create Instance Groups** to create an instance group. If you already have an instance, but do not have a group, you may want to create an *unmanaged* group to which you can add existing instances. See [Creating groups](https://cloud.google.com/compute/docs/instance-groups/creating-groups-of-unmanaged-instances#create_unmanaged_group) to create an unmanaged instance group. 7. Fill out the Backend and Frontend configurations and then click **Create**. For more NLB configuration related information, see [Configuring Load Balancer](https://cloud.google.com/load-balancing/docs/internal/setting-up-internal#configure_the_load_balancer). Once you have an NLB, you can test it.  An easy test would be to use `curl` with the Network Load Balancer's IP and Port from within the VPC to verify that the connection is established. ```shell $ curl : ``` ## Publish the Service When you publish a service, you create a service attachment. Send the Service attachment information to SingleStore. You will also need to whitelist the project that SingleStore sent you earlier. 1. In the GCP Console, select **Networking > Network Services > Private Service Connect > Published Services > Publish Service**. 2. Under **Load balancer** type, select **Internal TCP/UDP Load Balancer**. 3. Add the project name you received from SingleStore to the whitelist for your Service. 4. Enter the necessary details and create a subnet if needed. See [Publish a Service](https://cloud.google.com/vpc/docs/configure-private-service-connect-producer#publish-service) for more information. Do NOT enable **Use Proxy Protocol**. 5. Select **Add service**. For more information, see [Publish Services using Private Service Connect](https://cloud.google.com/vpc/docs/configure-private-service-connect-producer). ## Send the Service Attachment to SingleStore Contact [SingleStore Support](https://support.singlestore.com) and provide the Service Attachment. Follow these steps: 1. On the Google Cloud console, go to the **Private Service Connect** page. 2. On the **Published Service** tab, select the service you just created. Open the **Private Service Connect service details** screen. 3. Send the **Service attachment** information to [SingleStore Support](https://support.singlestore.com). Service attachment names usually have the following format: `projects//regions//serviceAttachments/`. *** Modified at: April 1, 2025 Source: [/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-google-private-service-connect/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-out-from-singlestore-helios-clusters-to-private-networks-services-via-google-private-service-connect/) (An index of the documentation is available at /llms.txt) --- # Connect to MongoDB® using Azure Private Link SingleStore supports connecting to a MongoDB® instance via Azure Private Link. ## Prerequisites * Create a SingleStore Kai-enabled cluster. Refer to [Getting Started with SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai.md) for related information. * The cluster must be deployed in an Azure region. ## Configure the Connection To connect to the MongoDB® instance using Azure Private Link: 1. Copy the **VPC Endpoint Service Name** from the SingleStore Helios Cloud Portal. 1. On the [Cloud Portal](https://portal.singlestore.com/), select **Clusters** in the left navigation pane. 2. Select the three dots under the **Actions** column for your cluster, and select **Access & Security** from the list. 3. On the **Access** tab, under **Private Links**, select **Create Connection**. 4. On the **Create Connection** dialog, copy and store the **Private Link Service Alias**. The Endpoint and Connection Type fields are automatically set to MongoDB® and Inbound, respectively. 5. (Optional) Keep the Create Connection dialog open to enter the resource ID of your Azure private endpoint created in the next step. 2. Create a private endpoint on the Azure portal. While configuring the connection, perform the following tasks: 1. Under **Resource**, select **Connect to an Azure resource by resource ID or alias**. Enter the Private Link Service alias copied in the previous step in the **Resource ID or alias** field. 2. Allow incoming connections to `TCP` port `27017` of your private endpoint. 3. Approve the private endpoint connection request. 4. Copy the Private Endpoint Resource ID of your Azure private endpoint. The resource ID has the following format: `/subscriptions//resourceGroups//providers/Microsoft.Network/privateEndpoints/` 3. Create an inbound private connection on the Cloud Portal. On the **Create Connection** dialog from the first step, enter the **Private Endpoint Resource ID**. The Endpoint and Connection Type fields are automatically set to MongoDB® and Inbound, respectively. 4. Select **Create Connection**. The connection is ready to use once it is in `ACTIVE` state. Connect to the MongoDB® endpoint using the following connection string (update the credentials and private endpoint address): ``` mongodb://:@:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true&tlsAllowInvalidHostnames=true&tlsAllowInvalidCertificates=true ``` Refer to [Connect to SingleStore Helios Clusters from Private Networks/Services via Azure Private Link](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-azure-private-link.md) for related information. *** Modified at: July 6, 2026 Source: [/cloud/connect-to-singlestore/private-connections/connect-to-mongodb-using-azure-private-link/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-mongodb-using-azure-private-link/) (An index of the documentation is available at /llms.txt) --- # Connect to SingleStore Helios Clusters from Private Networks/Services via Azure Private Link For inbound connections, you need to provide your cluster ID and Azure subscription details so that SingleStore can accept the connection request. To set up inbound connections to SingleStore Helios using private networks/services via Azure Private Link, perform the following tasks: 1. Request an alias from SingleStore. 2. Create a private endpoint. ## Request an Alias from SingleStore Contact [SingleStore Support](https://support.singlestore.com), and provide the following information: * Cluster ID and name. SingleStore can only process the connection request when your cluster is in the `Active` state. * Your Azure subscription ID and name * Region details * In the support ticket, specify that the request is for an inbound connection SingleStore will provide an alias. Use this alias to create a private endpoint (as explained below). ## Create a Private Endpoint > **📝 Note**: Your cluster and private endpoint must be in the same region. 1. On the [Azure portal](https://portal.azure.com/), select **Create a resource > Networking > Private Link > Create**. 2. Under **Private Link Center**, select **Private endpoints > Add**. 3. Under **Create a private endpoint**, on the **Basics** tab, enter or select the following information: * **Subscription**: Your Azure subscription. * **Resource group**: The resource group corresponding to your Azure subscription. It should have the same name as your Azure subscription name. Select **Create new** to create a new resource group. * **Name**: Name of your private endpoint service. * **Region**: Select a region in which you want to create the private link service. 4. On the **Resource** tab, under **Connection method**, select **Connect to an Azure resource by resource ID or alias**. 5. Under **Resource ID or alias**, enter the alias received from SingleStore support. 6. On the **Configuration** (or **Virtual Network)** tab, select the **Virtual Network** and **Subnet** where the private endpoint is created. 7. On the **Review + create** tab, review the private endpoint configuration, and select **Create**. The private endpoint is ready to use. > **📝 Note**: SingleStore Helios does not support Certificate Authority (CA) verification for inbound connections. For information on connecting to SingleStore Helios using SSL, refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md). ## References * [What is a private endpoint?](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview) * [Create a private endpoint](https://learn.microsoft.com/en-us/azure/private-link/create-private-link-service-portal#create-private-endpoint-1) *** Modified at: April 1, 2025 Source: [/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-azure-private-link/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-azure-private-link/) (An index of the documentation is available at /llms.txt) --- # Connect to SingleStore Helios Clusters from Private Networks/Services via Google Private Service Connect For inbound connections, you will receive a Service Attachment from SingleStore. You'll also need to provide your Google PSC project name to SingleStore, so that SingleStore can whitelist the project name from where you are trying to connect to the cluster. To set up inbound connections to SingleStore Helios using private networks/services via Google Private Service Connect, perform the following tasks: 1. Contact SingleStore Support (refer to [Support FAQ](https://docs.singlestore.com/cloud/support/support-faq.md)), and provide the following information: * Cluster ID. SingleStore can only process the connection request when your cluster is in the `Active` state. * Your Google Private Service Connect project name * Region details * In the support ticket, specify that the request is for inbound connection SingleStore will send you the Service Attachment. 2. On the **Google Cloud** console, under **Private Service Connect**, select a project in the region you want to connect to the Service Attachment sent by SingleStore. 3. [Create a Private Service Connect endpoint](https://cloud.google.com/vpc/docs/configure-private-service-connect-services#create-endpoint) and use the Service Attachment from step 1. SingleStore will then accept the request into the service to complete the connection. > **📝 Note**: Your cluster and Private Service Connect endpoint must be in the same region. > **📝 Note**: SingleStore Helios does not support Certificate Authority (CA) verification for inbound connections. For information on connecting to SingleStore Helios using SSL, refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md). *** Modified at: April 1, 2025 Source: [/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-google-private-service-connect/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-clusters-from-private-networks-services-via-google-private-service-connect/) (An index of the documentation is available at /llms.txt) --- # Connect to SingleStore Helios using AWS PrivateLink Configure both outbound and inbound connections to connect your cluster to AWS PrivateLink. For information on managing private connections, refer to [SingleStore Private Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections.md). To connect via AWS PrivateLink using Flow, specify the VPC endpoint of your private link in the destination database configuration. Refer to [Configure Flow](https://docs.singlestore.com/#section-id235511456889768.md). Contact [SingleStore Support](https://support.singlestore.com) for assistance with setting up or configuring private connections. > **📝 Note**: [This tutorial](https://aws.amazon.com/blogs/big-data/how-goldman-sachs-builds-cross-account-connectivity-to-their-amazon-msk-clusters-with-aws-privatelink/) builds cross-account connectivity to Amazon MSK clusters with AWS PrivateLink by fronting all brokers in the cluster with a single NLB that has cross-zone load balancing enabled. Refer to **Pattern 2: Front all MSK brokers with a single shared interface endpoint** in the tutorial for more information. ## Configure Inbound Connections To successfully set up an inbound connection to SingleStore Helios using AWS PrivateLink, perform the following tasks: 1. [Create an Inbound Connection on the Cloud Portal](https://docs.singlestore.com/#section-idm4577804916545633710638230996.md) 2. [Create a Private Endpoint on the Amazon VPC Console](https://docs.singlestore.com/#section-idm4587784974278433710639955862.md) ## Create an Inbound Connection on the Cloud Portal On the [Cloud Portal](https://portal.singlestore.com), 1. Select **Clusters**. 2. Select the three dots under **Actions** for your cluster and select **Access & Security** from the list. 3. Under **Private Links**, select **Create Connection**. 4. On the **Create Connection** dialog, enter or select the following information: 1. **Endpoint**: Select **SingleStore Endpoint**. 2. **Connection Type**: Select the **Inbound** connection type from the list. 3. **AWS Account ID** (Inbound connections only): Enter the AWS Account ID associated with your VPC/private endpoint. 5. Select **Create Connection**. Once the connection is ready to use, which may take a few minutes, its status changes to `ACTIVE`. If an error occurs while creating the private connection, the connection is deleted automatically. Hover over the `DELETED` status indicator to view the error message. Copy the **VPC Endpoint Service Name** of your connection, and enter it in the **Service name** field while creating a private endpoint on the Amazon VPC Console. Refer to [Manage Private Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections/#section-idm4493093347076833710473823166.md) for information on how to view the private connection details. ## Create a Private Endpoint on the Amazon VPC Console > **📝 Note**: Your cluster and endpoint must be in the same region. Create a private endpoint using the **Service name** copied earlier: 1. On the [Amazon VPC console](https://console.aws.amazon.com/vpc/), select **Endpoints > Create endpoint**. 2. Under **Service Category**, select **Other endpoint services**. 3. Enter the **Service name** copied from the [Cloud Portal](https://portal.singlestore.com) in the **Service name** box. 4. Select **Verify service** to verify the Service name. 5. Under VPC, select the VPC from which you'll connect with the AWS service. 6. Under **Subnets**, select one subnet per Availability Zone from which you'll connect to the AWS service. 7. Select **Create endpoint**. You can use the endpoint after it enters the `Available` state. Refer to [Endpoint states](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html#concepts-service-consumers) for more information. Create a security group to control access to the endpoint, and then attach the security group to the endpoint. Refer to [Control traffic to resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) for more information. > **📝 Note**: SingleStore Helios does not support Certificate Authority (CA) verification for inbound connections. For information on connecting to SingleStore Helios using SSL, refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md). ## Configure Outbound Connections To successfully set up an outbound connection to SingleStore Helios using AWS PrivateLink, perform the following tasks: 1. [Copy the AWS account ID from the Cloud Portal](https://docs.singlestore.com/#section-idm4587785065080033710540738094.md) 2. [Create an Endpoint Service on the AWS Console](https://docs.singlestore.com/#section-idm4573444058233633710541583593.md) 3. [Create an Outbound Connection on the Cloud Portal](https://docs.singlestore.com/#section-idm4602413659161633710543006469.md) 4. [Accept the Connection Request in your AWS Console](https://docs.singlestore.com/#section-id23551147332954.md) If you are using Kafka brokers with AWS MSK, you must specify the IP address of the broker endpoints while creating the target groups of the load balancer. Run the `nslookup` command with the DNS names of the MSK brokers to get their IP addresses. Note that the IP address of the endpoint does not change since it is attached to the VPC ENI (elastic network interfaces). Hence, resolve the broker endpoint IP address before initiating the connection. When using Kafka brokers, use the broker name with the port instead of the endpoint name in the `CREATE PIPELINE` command. ## Copy the AWS Account ID from the Cloud Portal On the [Cloud Portal](https://portal.singlestore.com), 1. Select **Clusters**. 2. Select the three dots under **Actions** for your cluster and select **Access & Security** from the list. 3. Under **Private Links**, select **Create Connection**. 4. On the **Create Connection** dialog, from the **Connection Type** list, select **Outbound**. Copy the AWS account ID displayed. You'll need to whitelist this ID while creating your endpoint service. ## Create an Endpoint Service on the AWS Console On the AWS Console, 1. Create a target group for each of the AWS services that you want to access using AWS PrivateLink, select **EC2 > Target groups > Create Target group**. Refer to [Target Groups](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html) for more information. 2. Create a network load balancer, select **EC2 > Load Balancers > Create Load Balancer**. 3. Under **Network Load Balancer**, select **Create**. Your cluster and the load balancer must be in the same region. Ensure that **Cross-zone load balancing** is enabled. Refer to [Create a Network Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html) for related information. 4. In the AWS Console, select **VPC > Endpoint Services > Create Endpoint Service**. > **📝 Note**: Your cluster and endpoint service must be in the same region. * Associate the endpoint service with the Network Load Balancer created in the previous step. * Enable **Require acceptance for endpoint** for additional security. 5. For this service, under **Allow principals**, add the AWS account ID copied from the Cloud Portal in the `"arn:aws:iam:::root"` format. This enables SingleStore to find and access the private endpoint service. 6. Verify that the security group rules in your VPC allow inbound traffic from the endpoint service, including traffic from internal private IP ranges (for example, RFC1918 ranges such as 10.0.0.0/8), depending on your network configuration. Refer to [Control traffic to resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) for more information. 7. Copy the **Service Name** of this AWS endpoint service. ## Create an Outbound Connection on the Cloud Portal On the [Cloud Portal](https://portal.singlestore.com), 1. Select **Clusters**. 2. Select the three dots under **Actions** for your cluster and select **Access & Security** from the list. 3. Under **Private Links**, select **Create Connection**. 4. On the **Create Connection** dialog, enter or select the following information: 1. **Endpoint**: Select **SingleStore Endpoint**. 2. **Connection Type**: Select the **Outbound** connection type from the list. 3. **Service name** (Outbound connections only): Enter the **Service Name** associated with your AWS endpoint service. 5. Select **Create Connection**. 6. (Optional) [Accept the connection request in your AWS Console](https://docs.singlestore.com/#section-id23551147332954.md). The connection is ready to use once the endpoint status changes to `ACTIVE`. If an error occurs while creating the private connection, the connection is deleted automatically. Hover over the `DELETED` status indicator to view the error message. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt550f79e363bf3f6d/6a7adbdc80ee3875a960792d/spc_deleted_status-aY9GVm.png) ## Accept the Connection Request in your AWS Console If **Require acceptance for endpoint** is enabled while creating the endpoint service, you must accept the connection request from SingleStore in your AWS account. On the AWS Console, 1. Select **VPC > Endpoint Services**, and then select your endpoint service. 2. On the **Endpoints Connections** tab, find the request from the SingleStore AWS account with the **Pending Acceptance** status. 3. From the **Actions** menu, select **Accept Endpoint Connection Request**. The connection status changes to **Available**, indicating that the connection is successfully established and is ready to use. ## Configure Flow To configure Flow to connect using AWS PrivateLink: 1. Log in to the [Cloud Portal](https://portal.singlestore.com). 2. Copy the **VPC Endpoint** of your outbound private link. 1. Select **Clusters**. 2. Select the three dots under **Actions** for your cluster and select **Access & Security** from the list. 3. Under **Private Links**, select the three dots under **Actions** for your private link, and then select **View Connection**. 4. Copy the **VPC Endpoint** for your private link. 3. Select **Ingestion > Load Data**, and then select a source supported by Flow. 4. Configure the destination database, connection name, Flow instance size, and then select **Create Flow Instance**. 5. Select **Open Flow** under the **Actions** column of the Flow instance created in the previous step. 6. On the **Setup** tab, configure the source database and then select **Next**. 7. Under **Destination Database**, enter the **VPC Endpoint** copied earlier in the **Host Name** field. 8. Enter the username and password of the SingleStore database user with which to connect. 9. Select **Test** to test the connection. Once the connection is verified, [configure the Flow instance](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/use-flow-on-helios.md) as required and proceed with data ingestion. Refer to [Load Data with SingleStore Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios.md) for more information. ## References * [AWS PrivateLink Concepts](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html#concepts-service-consumers) * [Create a private endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/create-endpoint-service.html) ## In this section * [Connect SingleStore Helios to AWS MSK using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/connect-singlestore-helios-to-aws-msk-using-aws-privatelink.md) * [Connect to MongoDB® using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/connect-to-mongodb-using-aws-privatelink.md) *** Modified at: May 15, 2026 Source: [/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/) (An index of the documentation is available at /llms.txt) --- # SingleStore Private Connections The SingleStore Private Connections (SPC) feature allows you to connect from a SingleStore Helios cluster to private services and networks. Private services/networks simplify networking and connectivity by allowing users to connect directly from a virtual private network to SingleStore Helios. To successfully set up a private connection, you need to configure both the outbound and inbound connections. Each cluster can only have one active inbound and outbound connection. The SPC feature is only supported for SingleStore Helios clusters deployed using AWS, and it only supports connections to AWS PrivateLink. > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. ## Prerequisites The SPC feature must be enabled for your organization. If this feature is not enabled by default, contact [SingleStore Support](https://support.singlestore.com). ## Configure Private Connections To create a private connection to AWS PrivateLink, refer to [Connect to SingleStore Helios using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink.md). ## Manage Private Connections To manage a private connection on the [Cloud Portal](https://portal.singlestore.com/), navigate to **Clusters**, select the three dots under the **Actions** menu for your cluster and select **Access & Security** from the list. Under **Private Links**, select the three dots in the **Actions** column of your private connection. Once the private connection is in `ACTIVE` state, you can perform the following actions: * **Edit Connection**: Displays the private connection details, and allows you to edit the AWS Account ID of your (inbound) private connection. * **Delete Connection**: Deletes a private connection. ## Private Connection Status Any private connection can have one of the following statuses: | **Status** | **Description** | | ---------- | ------------------------------------------- | | `ACTIVE` | The private connection is ready for use. | | `DELETED` | The private connection has been deleted. | | `PENDING` | The private connection is being configured. | *** Modified at: May 12, 2026 Source: [/cloud/connect-to-singlestore/private-connections/singlestore-private-connections/](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/singlestore-private-connections/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Endpoints An endpoint is a URL used to connect with SingleStore Helios clusters. The SingleStore Helios endpoints are available on the [Cloud Portal](https://portal.singlestore.com). Once logged in, 1. Navigate to **Clusters**. 2. Select the **Connect** under the **Actions** column of your cluster, and then select **CLI Client** from the list. 3. On the **CLI Client** tab, select **SingleStore Endpoints** from the list. The endpoint of your SingleStore deployment is displayed in the **Endpoint** section. Access SingleStore Helios deployment using the available endpoint with any of the supported [SQL clients](https://docs.singlestore.com/cloud/connect-to-singlestore.md) and/or [development tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md). > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. ## Connect to SingleStore Helios via TLS/SSL To establish an encrypted connection from a stand-alone SQL client, you need a TLS/SSL public certificate. This certificate ensures that clients without a valid certificate are not allowed to connect to your cluster. This CA certificate can also be used to verify the identity of each cluster host. Download the TLS/SSL certificate from the Cloud Portal. Install the certificate via the instructions provided at [How Do I Install a Certificate?](https://www.ssl.com/faqs/faq-how-do-i-install-a-certificate/). For example, using the SingleStore command line (CLI) client, you can connect to the SingleStore Helios endpoints using the `singlestore_bundle.pem` [certificate file](https://portal.singlestore.com/static/ca/singlestore_bundle.pem). The `.pem` file enables support for encrypted connections. Include the `--ssl-mode=VERIFY_CA` option to verify the certificate. **Note**: Include the `--ssl-mode=REQUIRED` option to establish a secure connection in older versions of the MySQL client, even when the `--ssl-ca` option is specified. ```shell singlestore -u admin -p -h -P 3306 --default-auth=mysql_native_password --ssl-ca=./singlestore_bundle.pem --ssl-mode=VERIFY_CA ``` > **❗ Important**: If your SQL client connects using the `singlestore_bundle.pem` file with the `--ssl-mode=VERIFY_CA` flag and is unable to establish a connection to SingleStore Helios, download and use the latest `singlestore_bundle.pem` file. See [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md) topic for more information. ## Secure the SingleStore Helios Connection ## Server Configuration to Require Secure Client Connections To make the server restrict access to clients over SSL only, add the `REQUIRE SSL` clause to the user’s `GRANT` statement, for example: ```sql CREATE USER 'user'@'%' IDENTIFIED BY 'password'; ALTER USER 'user'@'%' REQUIRE SSL; ``` For example, if `REQUIRE SSL` is specified for the user `user`: ```shell ## This connection attempt is rejected with an "Access denied" error: singlestore -u user -h 1.2.3.4 ``` ```shell ## This one works: singlestore -u user -h 1.2.3.4 --ssl-ca=ca-cert.pem ``` Unless the client is configured properly, the client may or may not use SSL to connect to SingleStore Helios even if SSL is enabled on the SingleStore Helios cluster. Adding `REQUIRE SSL` helps protect against misconfigured clients by preventing them from connecting over an insecure plaintext connection. However, proper client configuration is still necessary for security against active network attacks, regardless of server configuration. Note that the server currently uses a hardcoded version of the TLS protocol. *** Modified at: May 14, 2026 Source: [/cloud/connect-to-singlestore/singlestore-helios-endpoints/](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints/) (An index of the documentation is available at /llms.txt) --- # Container Services > **📝 Note**: Containers and the medium/GPU pool are currently in private preview. Please contact [Support](https://support.singlestore.com) for more information SingleStore’s Aura Container Service (SingleStore Aura service, also referred to as "Managed Container Service") is a serverless platform designed to provide instant access to pre-warmed containers for running custom applications with minimal latency. While initially developed to support Jupyter Notebooks, the platform is designed to execute any containerized application across supported hardware configurations. ## Key Features * **Instant Container Availability**: The platform maintains a pool of pre-warmed containers, enabling sub-second container acquisition times for your applications. This eliminates the traditional wait times associated with container startup and initialization. * **Enhanced Security**: The platform implements the [gVisor](https://gvisor.dev/) container runtime for robust security isolation, allowing you to safely execute untrusted code, LLM-generated content, or third-party applications without compromising the host system. * **GPU Support**: Pre-configured container images come with NVIDIA drivers pre-installed, making it seamless to run GPU-accelerated workloads without additional setup. * **Simplified Developer Experience**: The platform abstracts away the complexity of Docker and Kubernetes management and handles all container orchestration automatically, allowing developers to focus on their code rather than infrastructure concerns. ## How It Works ## Container Pool Management The platform automatically maintains and manages a pool of warm containers, ensuring immediate availability when requested. ## Streamlined Workflow Traditional container deployment typically involves: 1. Spinning up a new container 2. Configuring the environment 3. Installing required dependencies 4. Starting application services With the Aura Container Service, this process is simplified to: 1. Request a container 2. Connect to a pre-configured environment 3. Begin working immediately ## Container Sizing and Node Type | **Node** | **Container** | | ------------------------------------ | --------------------------------------------- | | CPU Pool | Small(1750m vCPU, 14 GB RAM, 40 GB) | | Medium(3750m vCPU, 27 GB RAM, 40 GB) | | | GPU Pool | GPU - T4(1 GPU, 6500m vCPU, 27 GB RAM, 40 GB) | ## Rate Limits To prevent abuse, either accidental or intentional, SingleStore enforces limits on the number of containers that can be created in an organization, which can be increased upon request. By default, an organization can have up to 50 active sessions and can create up to 20 containers per minute. ## Session Limits The session limits for different container service workloads are: | **Workload Type** | **Idle Timeout (Minimum)** | **Idle Timeout (Maximum)** | **Long Code Execution Limit** | | --------------------- | -------------------------- | -------------------------- | ----------------------------- | | Interactive Notebooks | 15 minutes (Default) | 4 hours | 1 hour | | Dashboards | 15 minutes (Default) | No Timeout | 1 hour | | Cloud Functions | 15 minutes (Default) | No Timeout | 1 hour | | Python UDFs | 15 minutes (Default) | No Timeout | 1 hour | Idle Timeout can be configured while creating or updating the workload instance. Once the timeout limit is reached, the system automatically terminates the workload session and deletes any unsaved data (state/variables/files). Interactive notebooks allow a single active session per user with each session having a maximum lifetime of 8 hours. Organization-level workloads such as Dashboards, Cloud Functions, and Python UDFs allow only one active session at a time, and that session is shared across all users. ## Session Persistence and State Management Workload sessions can run indefinitely when no idle timeout is configured or when activity remains within the configured limits. However, workloads must not rely on in-memory state or stateful connections. The system may move a session across multiple containers during its lifetime. As a result, in-memory state can be lost at any time and stateful protocols or long-lived connections are not guaranteed to persist. ## In this section * [Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) * [Scheduled Jobs](https://docs.singlestore.com/cloud/container-services/scheduled-jobs.md) * [Dashboard Apps](https://docs.singlestore.com/cloud/container-services/dashboard-apps.md) * [Cloud Functions](https://docs.singlestore.com/cloud/container-services/cloud-functions.md) * [Container App API Keys](https://docs.singlestore.com/cloud/container-services/container-app-api-keys.md) * [Container Services RBAC](https://docs.singlestore.com/cloud/container-services/container-services-rbac.md) * [Python UDFs](https://docs.singlestore.com/cloud/container-services/python-udfs.md) *** Modified at: December 19, 2025 Source: [/cloud/container-services/](https://docs.singlestore.com/cloud/container-services/) (An index of the documentation is available at /llms.txt) --- # Cloud Functions > **📝 Note**: This is a Preview feature. SingleStore Cloud Functions is a serverless, cloud-based service that allows users to run code without managing compute resources. It enables the execution of code such as SQL queries, Python scripts, and API calls in response to trigger actions. As a fully managed service, it eliminates the need to provision or manage servers. It allows users to run complex queries and perform ETL (Extract, Transform and Load) tasks without the need to manage dedicated compute resources. Create and manage cloud functions using the Cloud Portal or via the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md). ## Publish Cloud Functions using the Cloud Portal ## Create a Cloud Function Cloud Functions can be created for shared notebooks only. Create a cloud function using one of the following methods: 1. Navigate to **Cloud Functions** in the left navigation and select **Get Started**. 2. Navigate to **Editor > Shared**. 1. Select a shared notebook. 2. Select **Publish** (on the top right). ## New Cloud Function After selecting **Publish**, enter or select the following in **Publish Settings**: | **Publish Settings** | | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Publish as** | Select**Cloud Function**. | | **Name** | Enter a name for the cloud function. | | **Description** | Enter the cloud function description. | | **Notebook** | Select a shared notebook to publish as a cloud function. The shared notebook is pre-selected when the cloud function is published through notebooks. | | **Cluster** | Select theSingleStoreclusterthe notebook will connect to.Selecting aclusterallows connecting to theSingleStoredatabases referenced in the notebook natively. A cloud function can be published without aclusterattached to it. | | **Runtime** | Select a runtime from the following:
  • Small
  • Medium
  • GPU-T4

📝 Note:

This field is in preview.

| | **Region** | Select a region. | | **Idle Timeout** | Select an idle timeout.

📝 Note:

This field is in preview.

| Select **Next** , review the **Summary**, and then select **Publish** to publish the notebook as a cloud function. ## Example Notebook The following notebook shows how to publish your first cloud function: ## Manage an Existing Cloud Function To view an existing cloud function, select **Cloud Functions** in the left navigation. Existing cloud functions can be managed by performing the following actions: * View * Update * Share * Delete ## View an Existing Cloud Function To view an existing cloud function, select the cloud function name under the **Name** column and view the details for each cloud function. View the details of a cloud function on the right navigation pane. Following actions can be performed for a cloud function from this page: * Copy URL * View API Keys * View Live Logs * Share * Update * Delete ## View API Keys Refer to [Container App API Keys](https://docs.singlestore.com/cloud/container-services/container-app-api-keys.md) for related information. ## View Live Logs To view live logs of the selected cloud function, select **View Live Logs** from the ellipsis on the right side. A new window appears, where the **Timestamp** and the message in the **Body** column can be viewed. View the **Log JSON** by selecting the eye icon. ## Update an Existing Cloud Function To update an existing cloud function, select the ellipsis in the **Actions** column of the cloud function, and select **Update**. A cloud function can also be updated from a shared notebook. Select **Update Cloud Function** and select **Update**. ## Share an Existing Cloud Function To share an existing cloud function, select the ellipsis in the **Actions** column of the cloud function, and select **Share**. ## Delete an Existing Cloud Function To delete an existing cloud function, select the ellipsis in the **Actions** column of the cloud function, and select **Delete**. ## Publish Cloud Functions using the Management API Use the `CloudFunctions` path (`/v1/cloudfunctions` endpoint) to create and manage jobs using the Management API. Refer to [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. For example, the following API call lists all the cloud functions in the current organization: ```shell curl -X 'GET' \ 'https://api.singlestore.com/v1/cloudfunctions' \ -H 'accept: application/json' ``` ## Status of Cloud Functions | **Status** | **Description** | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Initializing | The notebook is creating or updating the cloud function. | | Active | The notebook is successfully published as a cloud function. | | Failed | The cloud function is not configured correctly. View the notebook snapshot to debug the failure. | | Error | An error unrelated to the notebook code prevented the cloud function from initializing. Refer to[Troubleshoot Cloud Functions](https://docs.singlestore.com/#section-idm234665756837262.md). | ## Troubleshoot Cloud Functions SingleStore automatically saves a snapshot of the notebook for each execution that fails. Navigate to **Cloud Functions** in the left navigation, select the cloud function, and download the snapshot associated with the failed cloud function to diagnose the error. The following table lists some common errors and their resolutions: | **Error** | **Solution** | | ---------------------------- | ----------------------------------------------------------------------------------------------------------- | | ClusterDeleted | Update the cloud function and select acluster. | | ClusterSuspended | Resume thecluster, or create a new cloud function with a differentcluster. | | Database Detached | Reattach the database with the right permissions or create a new cloud function with a different database. | | Notebook Deleted/Not Present | Create a new cloud function with a different notebook. | | Internal Errors/Misc | Reach out to[SingleStoreSupport](http://support.singlestore.com)or use the chat feature in theCloud Portal. | *** Modified at: July 7, 2026 Source: [/cloud/container-services/cloud-functions/](https://docs.singlestore.com/cloud/container-services/cloud-functions/) (An index of the documentation is available at /llms.txt) --- # Container App API Keys > **📝 Note**: This is a Preview feature. Container App API keys allow you to programmatically interact with SingleStore Aura environment. These keys are used to authenticate requests to Container App APIs, which enables automation, integration with CI/CD pipelines, and secure service-to-service communication. A Container App API key can be created for and used to access any Container App, such as **Cloud Functions** and **Dashboard Apps**. The API key is unique and only enables access to the Container App for which it is created. ## Create a Container App API Key To create an API key, perform the following steps: 1. In the Cloud Portal, under the **Container Services** section, navigate to a Container App. 2. Select an App instance. An example with a Cloud Function App is demonstrated. 3. Select **View API Keys** in the upper right corner. 4. Select **Create API Key**. 5. In the **Create API key for \ - \ **dialog, 1. Enter the name in the **API Key Name** field. 2. In **Expiration**, select the expiration date. 3. Select **Create API Key**. 6. Once the API key is generated, select **Copy API Key** to copy it or **Save As Secret** to store it securely in **[Secrets](https://docs.singlestore.com/cloud/developer-resources/secrets.md)**. The API key is displayed only **once**. 7. If **Save As Secret** is selected: 1. In the **New Secret** dialog, enter the name for the secret in the **Name** field. 2. Select **Create Secret**. The API key is stored in a secret. 8. Select **Close**. ## Types of Container App API Keys There are two types of Container App API keys: 1. **Active API key**: An API key that is currently valid, associated with an existing App instance or user, and has not been revoked. It can successfully authenticate requests to the Container App APIs. 2. **Orphaned API key**: An API key that is not associated with an existing App instance, as the App or user that created it, was deleted. Orphaned keys are inactive and cannot be used to authenticate API requests. > **📝 Note**: Orphaned API keys pose a potential security risk and should be identified and revoked promptly. ## Manage a Container App API Key You can manage a Container App API key in two ways: 1. Using the App instance 2. Using the Container Service ## Using the App Instance 1. Navigate to the Container Service. 2. Under the **Name** column, select the App instance for which you want to manage the Container App API key. 3. Select **View API Keys** in the upper right corner. You can view the total number of API keys for the App instance on the **View API Keys** button. 4. To revoke an active or orphaned Container App API key, perform the following tasks: 1. Revoke an Active Container App API key: 1. Navigate to the **Active** tab. 2. Select the delete icon under the **Actions** column next to your API key. 3. In the **Revoke API key** dialog, enable the agreement and select **Revoke**. 2. Revoke an Orphaned Container App API key: 1. Navigate to the **Orphaned** tab. 2. Select the delete icon under the **Actions** column next to your API key. 3. In the **Revoke API key** dialog, enable the agreement and select **Revoke**. > **📝 Note**: Orphaned keys can only be revoked by the **Aura Owner** or **Owner** of the organization. ## Using the Container Service 1. Navigate to the Container Service. 2. Select **View API Keys** in the upper right corner. 3. To revoke an active or orphaned Aura App API key, perform the following tasks: 1. Revoke an Active Container App API key: 1. Navigate to the **Active** tab. 2. Select the delete icon under the **Actions** column next to your API key. 3. In the **Revoke API key** dialog, enable the agreement and select **Revoke**. 2. Revoke an Orphaned Container App API key: 1. Navigate to the **Orphaned** tab. 2. Select the delete icon under the **Actions** column next to your API key. 3. In the **Revoke API key** dialog, check the agreement and select **Revoke**. > **📝 Note**: Orphaned keys can only be revoked by **Aura Owner** or **Owner** of the organization. *** Modified at: July 7, 2026 Source: [/cloud/container-services/container-app-api-keys/](https://docs.singlestore.com/cloud/container-services/container-app-api-keys/) (An index of the documentation is available at /llms.txt) --- # Container Services RBAC > **📝 Note**: This is a Preview feature. SingleStore Aura supports Role-Based Access Control (RBAC), which allows administrators to assign granular permissions at multiple levels. This ensures that users and teams have only the access they need to develop, deploy, and manage Container services such as Cloud Functions, Dashboard Apps, and Scheduled Jobs. Refer to [Role-Based Access Control (RBAC) for SingleStore Helios](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios.md) for more information on RBAC in SingleStore Helios. ## Benefits of RBAC in SingleStore Aura * **Operational Efficiency**: You can onboard or offboard users efficiently and modify their access based on role changes. * **Secure Collaboration**: RBAC ensures that users access only the resources they need. For example, data scientists can build applications, while analysts can view them without making changes. * **Compliance-Ready**: Role-to-resource mappings and access logs simplify audit processes and help meet regulatory requirements. * **Orphaned Resource Prevention**: When users leave, RBAC helps prevent lost or unmanaged assets by enabling easy transfer of ownership. ## Predefined Roles for Container App Resource | **Role** | **Description** | **Permission** | | -------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | Owner | Owners are granted full access including the ability to manage access, operate, delete, monitor and use the Container App. |
  • Control Access
  • Create API Keys
  • Delete
  • Revoke API Keys
  • Update
  • Use
| | User | Users are granted access to only use Container App and create Container App API keys. |
  • Create API Keys
  • Use
| ## How to Use RBAC in SingleStore Aura Aura App users can enforce RBAC in SingleStore Aura using both the Cloud Portal and Management API. ## Using Cloud Portal To use RBAC in the Cloud Portal, perform the following steps: 1. Under the **Container Services** section, navigate to an Aura App. 2. Select the three dots under the **Actions** column next to your Container App instance, and then select **Share**. 3. From the list, select **User** or **Team** to share the Container App with. 4. In the **Access** list, select the desired role next to the selected user or team. 5. To remove access for a user or team, select **Remove Access** in the **Access** list. ## Using Management API Use the `Users` (`/v1/users` endpoint) and `Teams` path (`/v1/teams` endpoint) in the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) to use RBAC in SingleStore Aura. Refer to [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/#section-idm4608309617228834306792661294.md) for more information. *** Modified at: July 7, 2026 Source: [/cloud/container-services/container-services-rbac/](https://docs.singlestore.com/cloud/container-services/container-services-rbac/) (An index of the documentation is available at /llms.txt) --- # Dashboard Apps > **📝 Note**: This is a Preview feature. SingleStore Dashboard Apps enables users to create, publish, and manage real-time interactive dashboards through the Cloud Portal. It integrates with SingleStore Notebooks and uses Plotly's Dash framework. It helps in monitoring key metrics, exploring trends, and making data-driven decisions in real time. Create and manage dashboard apps using the Cloud Portal. ## Publish Dashboard Apps using the Cloud Portal ## Create a Dashboard App Dashboard Apps can be created for shared notebooks only. Create a dashboard app using one of the following methods: 1. Navigate to **Dashboards** in the left navigation and select **Get Started**. 2. Navigate to **Editor > Shared**. 1. Select a shared notebook. 2. Select **Publish** (on the top right). ## New Dashboard App After selecting **Publish**, enter or select the following in **Publish Settings**. | **Publish Settings** | | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Publish as** | Select**Dashboard**. | | **Name** | Enter a name of the dashboard app. | | **Description** | Enter the dashboard app description. | | **Notebook** | Select a shared notebook to publish as a dashboard app. The shared notebook is pre-selected when the dashboard app is published through notebooks. | | **Cluster** | Select theSingleStoreclusterthe notebook connects to.Selecting aclusterallows connecting to theSingleStoredatabases referenced in the notebook natively. A dashboard app can be published without a attached to it. | | **Runtime** | Select a runtime from the following:
  • Small
  • Medium
  • GPU-T4

📝 Note:

This field is in preview.

| | **Region** | Select a region. | | **Idle Timeout** | Select an idle timeout.

📝 Note:

This field is in preview.

| Select **Next**, review the **Summary**, and select **Publish** to publish the notebook as a dashboard app. ## Example Notebook The following notebook shows how to publish your first dashboard app. ## Manage an Existing Dashboard App To view an existing dashboard app, select **Dashboards** in the left navigation. Existing dashboard App can be managed by performing the following actions: * View * Update * Share * Delete ## View an Existing Dashboard App To view an existing dashboard app, select the dashboard app name under the **Name** column and view the details for each dashboard app. View the details of a dashboard app on the right navigation. Following actions can be performed for a dashboard app from this page: * View Dashboard in New Tab * View API Keys * View Live Logs * Share * Update * Delete ## View API Keys Refer to [Container App API Keys](https://docs.singlestore.com/cloud/container-services/container-app-api-keys.md) for related information. ## View Live Logs To view live logs of the selected dashboard app, select **View Live Logs** from the ellipsis on the right side. A new window appears, where the **Timestamp** and the message in the **Body** column can be viewed. View the **Log JSON** by selecting the eye icon. ## Update an Existing Dashboard App To update an existing dashboard app, select the ellipsis in the **Actions** column of the dashboard app, and select **Update**. A dashboard app can also be updated using the shared notebook. Select **Update Dashboard** and select **Update**. ## Share an Existing Dashboard App To share an existing dashboard app, select the ellipsis in the **Actions** column of the dashboard app, and select **Share** ## Delete an Existing Dashboard App To delete an existing dashboard app, select the ellipsis in the **Actions** column of the dashboard app, and select **Delete**. ## Status of Dashboard Apps | **Status** | **Description** | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Initializing | The notebook is creating or updating the dashboard. | | Active | The dashboard is configured correctly. The notebook is successfully published as a dashboard. | | Failed | The dashboard is not configured correctly. View the notebook snapshot to debug the failure. | | Error | An error unrelated to the notebook code prevented the dashboard from initializing. Refer to[Troubleshoot Dashboard Apps](https://docs.singlestore.com/#section-idm234663965101069.md). | ## Troubleshoot Dashboard Apps SingleStore automatically saves a snapshot of the notebook for each execution that fails. Navigate to **Apps** in the left navigation, select the dashboard, and download the snapshot associated with the failed dashboard to diagnose the error. | **Error** | **Solution** | | ---------------------------- | ----------------------------------------------------------------------------------------------------------- | | ClusterDeleted | Update the dashboard and select acluster. | | ClusterSuspended | Resume theclusteror create a new dashboard with a differentcluster. | | Database Detached | Reattach the database with the right permissions or create a new dashboard with a different database. | | Notebook Deleted/Not Present | Create a new dashboard with a different notebook. | | Internal Errors/Misc | Reach out to[SingleStoreSupport](http://support.singlestore.com)or use the chat feature in theCloud Portal. | *** Modified at: July 7, 2026 Source: [/cloud/container-services/dashboard-apps/](https://docs.singlestore.com/cloud/container-services/dashboard-apps/) (An index of the documentation is available at /llms.txt) --- # Notebooks Notebooks are a common tool for developing code and presenting results. In SingleStore, notebooks enable collaboration between users with a native integration with SingleStore runtime using SQL and Python. We have implemented notebooks to be useful tools for data engineers and scientists, and app developers who want to quickly prototype and demo application code. Notebooks are implemented with security in mind. Administrators have control over which endpoint they allow notebooks to connect. Notebooks can be private or shared with all other users with access to the same workgroup. Notebooks are run in separate containers. In general, notebooks in SingleStore behave as standard Jupyter notebooks. Notebooks can also be run on a schedule with SingleStore [Job Service](https://docs.singlestore.com/cloud/container-services/scheduled-jobs.md). This helps you to easily make your notebooks production-ready for various scenarios like Data Prep and ML Flows, Python-based transformations, and building dashboards on real-time data. You can power these scenarios without any data having to leave your SingleStore environment and through the familiar and collaborative notebook environment. The topics in this section provide a high-level overview of how to work with notebooks, as well as more detail on using notebooks with SQL and SingleStore Helios specific features. ## In this section * [Using Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks/using-notebooks.md) * [Notebook Version History](https://docs.singlestore.com/cloud/container-services/notebooks/notebook-version-history.md) * [Github Integration for Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks/github-integration-for-notebooks.md) * [Develop with Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks/develop-with-notebooks.md) * [Load Dataframes From Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks/load-dataframes-from-notebooks.md) * [Running a Notebook from Another Notebook using Fusion SQL](https://docs.singlestore.com/cloud/container-services/notebooks/running-a-notebook-from-another-notebook-using-fusion-sql.md) *** Modified at: November 17, 2025 Source: [/cloud/container-services/notebooks/](https://docs.singlestore.com/cloud/container-services/notebooks/) (An index of the documentation is available at /llms.txt) --- # Develop with Notebooks Prototyping applications or analyzing via notebooks in SingleStore Helios follows the same general principles as developing with notebooks in general. To get started, connect to a data source. ## Connect to Data Sources SingleStore Helios supports internal and external data sources. Internal data sources are databases that exist within your cluster. An external data source could be an AWS S3 bucket, for example. ## Connect to a SingleStore Data Source Once you select a cluster, you can access all of the databases attached to that cluster. You cannot connect to databases that are not attached to the cluster you are using. You can specify a default database for your notebook, eliminating the need to specify the database context every time you make a query on that default database. To set a default database, select a database from the drop-down menu. No default database specified: ![A workspace with no specified database.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt87ebd5c0607eb207/6a7adb7c6c6eaccfaff08d26/notebook_defaultdbunset-bcInqP.png) Default database specified: ![A workspace with a specified database.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltfe783545afc22389/6a7adacfc2cc097a3723db87/notebook_defaultdbset-e6ebNS.png) ## Connecting via SQL To connect to the default database, you do not need to pass any database context to query that database: ```sql %%sql SELECT * FROM mytable; ``` To connect to a non-default database, you will need to specify the database in your query: ```sql %%sql USE mydatabase; SELECT * FROM mytable; ``` You can also use "dot" notation: ```sql %%sql SELECT * FROM mydatabase.mytable; ``` ## Connecting via Python When connecting via Python to the default database, use the predefined connection string variable, `connection_url`: ```python from sqlalchemy import * db_connection = create_engine(connection_url) ``` You can then use that connection string (`db_connection` above) to connect to SingleStore. Here's an example of creating a table using that connection string: ```python query1 = 'create table people (filename varchar(255), vector blob, shard(filename))' with db_connection.connect() as conn: conn.execute(text(query1)) ``` When connecting to other databases, use the following method where user (`connection_user`), password (`connection_password`), host (`connection_host`), and port (`connection_port`) are already defined based on the cluster you selected. ```python from sqlalchemy import * database_name = 'mydatabase' db_connection_str = "singlestoredb://"+connection_user+":"+connection_password+"@"+connection_host+":"+connection_port+"/"+database_name db_connection = create_engine(db_connection_str) ``` You can then use that connection string (`db_connection` above) to connect to SingleStore: ```python query1 = 'create table people (filename varchar(255), vector blob, shard(filename))' with db_connection.connect() as conn: conn.execute(text(query1)) ``` ## Connect to an External Data Source SingleStore Helios lets you control which endpoint to access from notebooks to provide secure outbound access to trusted resources.  By default, connections are limited to SingleStore databases; however, you can enable and disable connections to other external endpoints via the allowlist. Only users with the **Organization Owners** role can add external connections. When trying to access an external endpoint from a notebook that is not in the allowlist, you should see the following warning: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt54dada4501343dbe/6a7ada9fc2cc09586623db6f/notebook_externalendpoint-Sx47gr.png) To fix this issue, select **Add to Firewall** and then select **Save**. The external connection is automatically added into the allowlist. The notebook cell will be executed. ![Adding external endpoint in FQDN allowlist.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1776188a6bcc031f/6a7adaaf96b5a60a6286fdb8/notebook_addexternalendpoint-aJ91oq.png) If you do not have access due to your role, your organization owner can manually add or remove the external endpoint using the following steps: ![Add external connection to firewall.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9de5e560f2d268a7/6a7adb20437e0f1a34dcfc5a/Screenshot_2025-11-18_at_12_15_42_PM-o3DZx9.png) 1. In the left navigation, select **Editor**. 2. Select the **Firewall** tab in the main window. 3. Select **Edit** to add new endpoints: ![The Edit FQDN Allowlist dialog with Add FQDN button and Add Suggested FQDN button, with Cancel and Save buttons.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt64ae0bdac5a4457b/6a7adb804c4bfbaf6acc1f43/notebook_edit_fqdn-D4cK4t.png) 4. In the Edit Allowlist dialog, you can add a Fully Qualified Domain Name (FQDN) or select from a list of suggested FQDNs (for example pypi.org, github.com, or \*.s3.\*.amazonaws.com). You can provide wildcard access to an endpoint by using the `*` character. The wildcard character represents 0 or more valid DNS characters, except for '.'. `*.singlestore.com` matches `docs.singlestore.com` but not `singlestore.com` because it only matches up to the '.' character. `ex*le.com` matches example.com but not `examp-site.com`. For example, to access any AWS S3 endpoints, you can use the following syntax:  \*.s3.\*.amazonaws.com. 5. Select **Save**. To remove a connection select the Trash icon next to the connection in the Edit FQDN Allowlist dialog. ## Manage Cells Use the same commands and techniques you use with most other Python based cells in notebooks. ## Cell Types A notebook cell can be one of these types: * **Markdown**: lightweight markup language used to add formatting to plain text. More information [here](https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html). * **Python**: language supported is Python 3. * **SQL**: language supported is SQL (for SingleStore). Specify the cell type by selecting a cell and choosing a type from the drop-down menu: ![Specify the cell type as Python from the drop-down list.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte4abd6708a175fda/6a7adab8bdcff091fcc377e8/notebook_cell-BAot37.png) ## Run a cell Execute/run a cell via keyboard shortcut (**Shift+Return/Enter**) or by selecting the run (play button) icon at the left of the notebook cell, or by selecting **Run Selected Cell** option from the **Run** menu in the toolbar. ![Select play button to run the cell.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt240bdb057219dde7/6a7adb8442a117661f95095f/notebook_cell_run-zmGQpK.png) ## Run Multiple Cells Execute/run multiple selected cells using the following options from the **Run** menu in the toolbar: * Run Selected Cell and All Below * Run All Above and Selected Cell * Run All Cells * Restart and Run All Cells ## Manipulating Cells When you select a cell, some icons for basic cell manipulation appear: | Action | Icon | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | Add a cell | ![Icons for adding a cell above or below the current cell.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt449ebf8b1af3cec4/6a7ada9e26618c2c9d842fce/notebook_add_up_down-UuXJ7l.png) | Add a new cell above or below the selected cell. | | Move | ![Arrow up and down icons.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt116eaaebea916f46/6a7adb7f2f00b2332fef3667/notebook_up_down-Ef2g3G.png) | Move the selected cell up or down. | | Copy | ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt580bc028856b8c88/6a7ada7fc5d193df095db8c4/notebook_copy-SVw0ll.png) | Copy the selected cell. | | Duplicate | ![Duplicate Icon](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltefded23f506c0dc9/6a7adb896c6eac4f2af08d2a/notebook_dup-AkKb6M.png) | Duplicate the current cell and place it immediately below the current cell. | | Delete | ![Trash Can icon for Delete.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0d83795fff381e20/6a7adb343ce8e2af98cea12e/notebook_delete-YkzwJe.png) | Delete the selected cell. | | Ask SQrL | ![Ask SQrL icon](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5d37890ebb249f60/6a7ada7bc7a40c975130ea70/notebook_ask_sqrl-P9Zq2h.png) | An AI-powered co-pilot (SQrL) answers questions. | For more options, right-click/control-click on a cell to open the cell context menu. From the context menu, you can perform additional cell functions, such as split and merge. Keyboard shortcuts exist for most common tasks as well. ![Right-click/control-click on a cell to open the cell context menu. Keyboard shortcuts for manipulating cells](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6966c660ef0b8c22/6a7adb81fc63ab433b64185a/notebook_context2-Zg8vm0.png) ## Manage Cell Inputs and Output Right-click/control-click on a cell to open the cell context menu. ![Right-click/control-click on a cell to open the cell context menu to manage cells.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9bf24333455020d1/6a7adb769f525167dc65e920/cell_context_menu-my1AoQ.png) An additional option **Format SQL Cell** is available when the selected cell is using the SQL language (via the `%%sql` magic command or the SQL cell). You can also show/hide cell output by selecting the vertical bar (purple) next to the output. Before: ![Example of a vertical "bar" alongside a cell's output, expanded below the cell.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5927580954ee4421/6a7ada80227b1c2d4758ce13/notebook-hide-results-b-joJ2pk.png) After selecting the bar: ![An example of the truncated vertical "bar" after the cell output is collapsed.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltbbd89d83ccf55a67/6a7adace2f00b2148def363d/notebook-hide-results-a-tRZ3P5.png) ## Use Multiple Languages You can set the default language for notebooks to either SQL or Python3. You can select the language for a cell via the toggle located above the cell: ![Cell type from the drop-down list above each cell.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7fe89d531fe92035/6a7adad02f00b2a1d4ef363f/notebook_cell_types-0aiTPS.png) ## Python Cells Python 3 is the default language for Python cells. See the [Python documentation](https://docs.python.org/3/index.html) for more information about using Python. ## SQL Cells There are two options to write SQL in notebooks. The first involves selecting a cell type to be SQL. The following example shows how to specify a default database (mydatabase) and the syntax used to connect to another database (mydatabase2) for use in joins with an SQL cell: ```sql USE mydatabase; -- the database to use by default SELECT * from mytable mt1 INNER JOIN mydatabase2.mytable2 mt2 ON mt1.mycolumnid1 = mt2.mycolumnid2; ``` The next option is to use the SQL magic from within a Python cell. This gives you the ability to write SQL and Python code in a single block: ```sql %%sql - switch the cell to SQL USE mydatabase; -- the database to use by default SELECT * FROM mytable mt1 INNER JOIN mydatabase2.mytable2 mt2 ON mt1.mycolumnid1 = mt2.mycolumnid2; ``` The results are displayed as a table. You can also save and use the output from the calculation in an SQL cell (called `output_reviews` in this example) in another Python cell with the following options: Using cell output - **Save result to**: ![Execute SQL commands using SQL cell type.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8d321418205d0269/6a7adb77bdcff04114c37822/notebooks_usecelloutput-FmzYy5.png) Or, using SQL magic: ```sql %%sql output_reviews << -- switch the cell to SQL and specify the output as output_reviews SELECT * FROM reviews ``` This output can then be used in other cells, etc. For example, in this example, `output_reviews` is used with Python as a dataframe: ```python df = pd.DataFrame(output_reviews) ``` ## SQL Line Use `%sql` to enable a single line of SQL in a Python cell: ```python result = %sql USE s2_dataset_martech; SELECT * FROM offers GROUP BY customer LIMIT 10 ``` Combine SQL and Python in the same cell: ```python result = %sql USE s2_dataset_martech; SELECT * FROM offers GROUP BY customer df = pd.DataFrame(result) ``` ## Manage Libraries SingleStore notebooks come with pre-installed libraries, and you can install additional libraries as needed. ## Pre-installed Libraries Run the following command in a Python cell to see the list of pre-installed libraries: ```python !pip list ``` ## Install and Import Libraries SingleStore supports libraries available from . This example shows how to install a library from the [Kaggle open dataset](https://pypi.org/project/opendatasets/). ```python !pip3 install opendatasets ``` To update the version of a pre-installed library: ```python !pip3 install plotly --upgrade ``` Once a library is installed, you can import library components and add an alias: ```python import opendatasets as od ``` > **💡 Tip**: For better clarity, have one cell at the beginning of the notebook with all additional libraries to install and have a second cell listing the libraries to import. You can then collapse the two cells to remove clutter. ## Magic Commands For a specific cell, to see all the supported magic commands: ```python %lsmagic ``` For information about the full list of available magic commands: ```python %quickref ``` Some helpful magic commands: | Magic Command | Description | | ------------------ | ---------------------------------------------------------------- | | %history | View the log of the session activity for the notebook. | | %pinfo \ | Details of the object stored in the specified variable. | | %time | Show the time of execution of a Python or SQL statement. | | %who | List all of the currently defined variables within the notebook. | ## Useful Shortcuts There are two modes in a notebook: command and edit. Command mode is activated by pressing ESC. Edit mode is activated by pressing Enter. ## Command and Edit Mode Shortcuts | Task | Mac | Windows | | ---------------------------------- | --------------- | ------------- | | Run the current cell, select below | Shift + Enter | Shift + Enter | | Run selected cells | Command + Enter | Ctrl + Enter | | Run the current cell, insert below | Option + Enter | Alt + Enter | | Save and checkpoint | Command + S | Ctrl + S | ## Command Mode Shortcuts | Task | Mac | Windows | | -------------------------------- | ------------------------ | ------------------------ | | Enter edit mode | Enter | Enter | | Select cell above | Up | Home | | Select cell below | Down | End | | Extend selected cells above | Shift + Up | Shift + Home | | Extend selected cells below | Shift + Down | Shift + End | | Insert cell above | A | A | | Insert cell below | B | B | | Cut selected cells | X | X | | Copy selected cells | C | C | | Paste cells below | V | V | | Paste cells above | Shift + V | Shift + V | | Move cell up | Ctrl + Shift + Up | Ctrl + Shift + Home | | Move cell down | Ctrl + Shift + Down | Ctrl + Shift + End | | Split cell | Ctrl + Shift + Minus (-) | Ctrl + Shift + Minus (-) | | Delete selected cells | D, D (press twice) | D, D (press twice) | | Undo cell operation | Z | Z | | Redo cell operation | Shift + Z | Shift + Z | | Merge selected cells | Shift + M | Shift + M | | Merge cell above | Ctrl + Delete | Ctrl + Backspace | | Merge cell below | Ctrl + Shift + M | Ctrl + Shift + M | | Save and checkpoint | S | S | | Change the cell type to Code | Y | Y | | Change the cell type to Markdown | M | M | | Scroll notebook up | Shift + Space | Shift + Space | | Scroll notebook down | Space | Space | | Find | Command + F | Ctrl + F | | Find next | Command + G | Ctrl + G | | Find previous | Shift + Command + G | Shift + Ctrl + G | | Show line numbers | Shift + L | Shift + L | | Render side-by-side | Shift + R | Shift + R | ## Edit Mode Shortcuts | Task | Mac | Windows | | ------------------ | ------------------- | ---------------- | | Go to command mode | Esc | Esc | | Select all cells | Command + A | Ctrl + A | | Undo | Command + Z | Ctrl + Z | | Redo | Shift + Command + Z | Shift + Ctrl + Z | | Go to cell start | Command + Up | Ctrl + Home | | Go to cell end | Command + Down | Ctrl + End | | Go one word left | Command + Left | Ctrl + Left | | Go one word right | Command + Right | Ctrl + Right | *** Modified at: May 11, 2026 Source: [/cloud/container-services/notebooks/develop-with-notebooks/](https://docs.singlestore.com/cloud/container-services/notebooks/develop-with-notebooks/) (An index of the documentation is available at /llms.txt) --- # Github Integration for Notebooks > **📝 Note**: This is a Preview feature. GitHub integration for notebooks supports the use of Git workflows directly within the notebook environment. This integration allows you to commit notebook changes, track history, and push updates to connected GitHub repositories without switching tools. Key features include: * Create pull requests, add comments, view code diffs, and conduct code reviews. * Reproduce notebook results by running and sharing exact versions of the notebook files. * Collaborate with team members by sharing notebooks and integrating contributions. * Connect notebook development to existing CI/CD pipelines. ## Connect a GitHub Account 1. In the [Cloud Portal](http://portal.singlestore.com), navigate to the bottom-left corner and select the organization name to open the menu. Select **User Integrations**. 2. In the **User Integrations** page, enable **GitHub**. 3. The page redirects to **GitHub**. Sign in using the GitHub credentials. If two-factor authentication (2FA) is enabled, complete the verification process to proceed. 4. Select the organization or account required to connect. > **📝 Note**: SingleStore recommends selecting a personal GitHub account to complete the OAuth flow and enable the integration. A GitHub organization can be connected only by an Organization Admin. If the connecting account does not have admin privileges, a connection request is sent to the organization’s GitHub admin for approval. If a GitHub organization appears as **Configured**, it indicates that the integration has already been set up by another member of the organization. In this case, the organization is automatically added to the connected GitHub accounts once the OAuth flow is completed with a personal account. 5. Select the repositories required to grant access under the selected organization or account: * To allow access to all repositories, select **All Repositories**. * To limit access, select **Only select repositories**, and then select the desired repositories. 6. Select **Install & Authorize**. 7. After authorization completes, view the connected GitHub account under **Connected User**. > **📝 Note**: Each user can connect only one GitHub account at a time. To connect a different account, the existing connection must be disabled first, and then a new connection can be established. 8. Navigate to the bottom-left corner and select the organization name to open the menu. Select **Organization Details > Integrations** in the Cloud Portal to view the GitHub integration and number of users. This page is accessible only to SingleStore organization owners. It can also be accessed from the info alert displayed on the **User Integrations** page visible to SingleStore organization owners only. ## Configure GitHub Integration 1. In the **User Integrations** page, under **GitHub**, select **Configure**. 2. In the **Configure GitHub Integration** dialog, under the **Actions** column of the selected GitHub account, select **Configure**. The dialog also displays any pending connection requests for GitHub organizations that require approval from the respective GitHub organization admins. 3. To install and authorize the GitHub integration for another GitHub account, select **Installation**. ## Load a Notebook from GitHub 1. Navigate to the **Editor**, and then select **New > Import** from GitHub. 2. Select the repository and branch that contains the notebook to import. 3. Select the notebook file. 4. Select the location of the notebook and then select **Import** to complete the process. ## GitHub Version Control Once the notebook is imported, the synced GitHub connection can be viewed in the right-hand navigation. The following actions can be managed in GitHub version control: * Fetch * Commit & Push ## Fetch GitHub Content To fetch the contents from Github repository, select **Fetch**. Compare the contents and select **Fetch**. Select **Confirm & Overwrite** to overwrite the current notebook version with the notebook content in GitHub repository. ## Commit and Push Notebook Changes To commit and push the changes in the notebook to the GitHub repository, select **Commit & Push**. Compare the contents and select **Compare & Push**. In the **Push to GitHub** dialog, enter the following details: | **Commit message** | Enter the commit message. | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | **New branch name** | Enter the branch name if the changes are not being pushed to the default branch (main) used when connecting the notebook to the repository. | Enable **Include Notebook Cell Outputs** if required and select **Commit & Push**. ## Enable the GitHub Connection GitHub version control can be enabled for individual notebooks that were not initially imported via GitHub integration. Perform the following steps to enable the GitHub connection: 1. Open the desired notebook. 2. Select the GitHub icon on the right upper side. Select **Enable GitHub Version Control**. 3. In the **Upload & Push to GitHub** dialog, enter the following details: | **Repository** | Select the repository in which the notebook will be uploaded. | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | **New branch name** | Enter the branch name if the changes are not being pushed to the default branch (main) used when connecting the notebook to the repository. | | **Commit message** | Enter the commit message. | 4. Enable **Include Notebook Cell Outputs** if required. 5. Enable **Overwrite existing file if it already exists at selected GitHub path** if required. 6. Select **Upload & Push** to upload the notebook in the selected GitHub repository. To perform GitHub version control actions after uploading the notebook to the selected GitHub repository, refer to [GitHub Version Control](https://docs.singlestore.com/#section-id235248638096036.md). ## Disconnect the GitHub Connection To disconnect the GitHub connection, select the ellipsis in the **GitHub Connection** and select **Disconnect**. The existing GitHub connection and the selected notebook is disconnected from the selected GitHub repository. This action does not delete or modify the notebook file in GitHub. ## Disable GitHub Integration To disable GitHub integration, select **Organization Details > Integrations** in the Cloud Portal and disable GitHub in the **Availability** column. Select **Confirm and Disable** to proceed. Only an Organization Admin can disable the GitHub integration. Even if users have the integration installed in their GitHub accounts, they cannot import notebooks into SingleStore Helios once the Organization Admin has disabled the integration. *** Modified at: November 17, 2025 Source: [/cloud/container-services/notebooks/github-integration-for-notebooks/](https://docs.singlestore.com/cloud/container-services/notebooks/github-integration-for-notebooks/) (An index of the documentation is available at /llms.txt) --- # Load Dataframes From Notebooks The simplest and fastest way to load data is using a pandas dataframe. However, the schema inference is not always optimal. Define the table and schema in advance to avoid having to make future schema alterations. ## Create a Dataset with Geospatial and Embeddings Create a dataset with complex data types, such as, integer, date time, geospatial, and vectors. Run the following command to install Shapely: ```python !pip3 install shapely ``` Create the dataset: ```python from shapely import wkt import pandas as pd import numpy as np data = { 'Integer': [1, 2, 3, 4], 'Datetime': [pd.Timestamp(x) for x in ['2023-06-01', '2023-06-02', '2023-06-03', '2023-06-04']], 'Geospatial Data': [wkt.loads(x) for x in ['POLYGON((1 1,2 1,2 2, 1 2, 1 1))', 'POLYGON((5 1,6 1,6 2,5 2,5 1))', 'POLYGON((5 5,6 5,6 6,5 6,5 5))', 'POLYGON((1 5,2 5,2 6,1 6,1 5))']], 'Vectors': [np.array(x) for x in [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]] } df = pd.DataFrame(data) display(df) ``` ## Connecting to the Database Select the Connection and the Database to use from the dropdown boxes at the top of the notebook. ```python from sqlalchemy import * db_connection = create_engine(connection_url) ``` ## Check Table Schema Check the schema for the table; it is not optimal since geospatial and embeddings data were interpreted as text by the engine. ```sql %%sql SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'table_schema_inference' AND TABLE_SCHEMA = 'MyDatabase'; ``` | COLUMN\_NAME | COLUMN\_TYPE | | --------------- | ------------ | | Integer | bigint(20) | | Datetime | datetime | | Geospatial Data | text | | Vectors | text | ## Create a Table with the Correct Schema Geospatial indexes are only supported with a rowstore table. Ensure the schema is correct and rowstore in the creation process: ```sql %%sql CREATE ROWSTORE TABLE `table_schema_defined` ( `Integer` bigint(20) DEFAULT NULL, `Datetime` datetime DEFAULT NULL, `Geospatial Data` GEOGRAPHY NOT NULL, `Vectors` blob, index (`Geospatial Data`) ) ``` With the changes made to the SingleStore Python client library, no extra steps are required to convert the data before inserting it into a database. All numpy arrays, shapely and pygeos geometry objects in a dataframe will be converted into the correct format automatically. ## Load the Dataframe Into the Predefined Table Schema This method is fast but will not provide the best schema inference. ```sql df.to_sql('table_schema_defined', con = db_connection, if_exists = 'append', index = False, chunksize = 1000) ``` ## Verify the Data Check that the geospatial and vector data have been correctly loaded. ```sql %%sql select * from table_schema_defined ``` *** Modified at: November 17, 2025 Source: [/cloud/container-services/notebooks/load-dataframes-from-notebooks/](https://docs.singlestore.com/cloud/container-services/notebooks/load-dataframes-from-notebooks/) (An index of the documentation is available at /llms.txt) --- # Notebook Version History The Version History feature allows users to view and compare visual differences between two notebook versions. It provides a Git-like experience directly within the notebook interface. This feature displays a chronological list of all saved versions with timestamps, which makes it easy to track a notebook's evolution. By visually highlighting changes, Version History reduces the risk of data loss and eliminates the need for manual version management, such as duplicating notebooks at various stages. It streamlines version tracking and simplifies understanding the notebook's development over time. ## View the Notebook Version History To view the version history in the notebook, select the clock icon to select **Version History**. ![An image that shows the "Version History" icon within a Notebook.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltacb155ebce5d89f0/6a7adb113ce8e21fdecea127/version_history_show_version_history-Hp0y8S.png) ## Compare Versions By default, the **Compare Version** toggle is on. To compare two versions, toggle on **Compare Version** and select the versions you want to compare. When you toggle off **Compare Version**, you can view the selected version on the right side of the panel in its complete n otebook format, including the outputs. ![An image that shows the "Compare Versions" toggle selected, and the Notebook version dropdown being used to compare two Notebook versions.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8e69d437ba6cae35/6a7ada442f00b2eb5bef35f7/version_history_compare_versions_1-qSVFLo.png) The side-by-side visual differences are displayed between the two versions. You can also view the versions with timestamps in the Version History (right navigation) pane. In the version selection list, the **Current** option shows the last saved notebook contents, while the **Version History** pane shows the most recently created version from the version history. These versions are not the same, as new versions are created every 2 minutes. ![An image that shows the Version History pane, which lists saved Notebook versions.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0dcf2dc5a38426ff/6a7ada47366687227927e34b/version_history_compare_versions_2-A3x2Vb.png) ## Version Retention Policy * A new version is created when changes are made and two minutes have passed since the previous version was created. * All versions created on the current day are retained. * For versions created on previous days, only the oldest version for each day is retained, up to a maximum of 30 days. * Versions older than 30 days are not retained. * To preserve a specific version, SingleStore recommends adding it to your favorites list on the day you create the version. > **📝 Note**: New versions are auto-created only when changes occur. ## Manage Versions You can view details, edit details, add to favorites, remove from favorites, duplicate, restore, and delete a version of the notebook in the **Version History** pane. ## View Version Details To view the details of a version of the notebook, select the ellipsis (three dots) in the selected version, and select **View Details**. You can view the version name, the name of the user who created it, the timestamp of its creation, and the description. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt637b65eec173e65c/6a7ada4d59471178974c2647/version_history_view_details-FecTJe.png) ## Edit Version Details To edit the details of a notebook version, select a version in the **Version History** pane. Select the ellipsis (three dots) in the selected version, and select **Edit Details**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt03699ea8a5e0fe70/6a7ada49c33f4fc7a1d4eeac/version_history_edit_version_details_1-1f6S24.png) In the **Edit Version Details** dialog, enter the **Version Name** and **Description**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc1b8b2167869bccc/6a7adafdc7a40c2b9630ea8d/version_history_edit_version_details_2-Wh8OmQ.png) ## Add a Version to Favorites To bookmark a notebook version, select a version in the **Version History** pane and click on the star icon next to the selected version to add it to **Favorites**. After adding a version to favorites, a list of bookmarks (favorite versions) is displayed under **Favorites**, above **Version History**, in the **Version History ** pane. To permanently keep a version, transfer it to the **Favorites** list, which has a limit of 10 versions. However, versions in **Favorites** are always retained. To add an 11th version to **Favorites**, you must delete or remove an existing version. ![An image that show how to add a Notebook version to your favorites.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt481b9101734c9cc9/6a7ada42889c379ca5057c52/version_history_add_version_to_favorites-pSdoRa.png) ## Remove a Version from Favorites Select a version in the **Version History** pane and tap on the star icon to remove it from **Favorites**. ## Duplicate a Version To duplicate a version of the notebook, select a version in the **Version History** pane. Select the ellipsis (three dots) in the selected version, and select **Duplicate Version**. In the **Duplicate from \** dialog, enter the name in **Notebook Name** and the location, either Shared or Personal, and select **Duplicate Notebook**. ![An image that shows how to duplicate a Notebook using the "Duplicate Notebook" button.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta51b26b55f21e24f/6a7ada48bdcff002d9c377c6/version_history_duplicate_version-jD63n5.png) ## Restore a Version To restore a version of the notebook, select a version in the **Version History** pane. Select the ellipsis (three dots) next to the selected version, and select **Restore Version**. Alternatively, you can restore a version by selecting the **Restore Version** button at the bottom. > **📝 Note**: Restoring a version replaces the current contents of the notebook with the version you selected. ![An image that shows how to restore a Notebook version by using the highlighted "Restore Version" button.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6491990d3f4c7b55/6a7ada4cd402d934e9eadc94/version_history_restore_version-LXfK8D.png) ## Delete a Version To delete a version of the notebook, select a version in the **Version History** pane. Select the ellipsis (three dots) in the selected version, and select **Delete version**. *** Modified at: November 17, 2025 Source: [/cloud/container-services/notebooks/notebook-version-history/](https://docs.singlestore.com/cloud/container-services/notebooks/notebook-version-history/) (An index of the documentation is available at /llms.txt) --- # Running a Notebook from Another Notebook using Fusion SQL SingleStore Notebooks support running one notebook from another using [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md), either within the same session or in a new session. This is useful when you want to: * Avoid code duplication such as reusable functions or environment setup by running the shared logic in a separate notebook. * Run code in parallel by executing notebooks in separate sessions. ## Create a Sample Notebook You can create a sample notebook in one of two ways: * On the [Cloud Portal](https://portal.singlestore.com/) select **New** > **New Notebook **> **Shared**.  * Run the following Python code in a notebook: ```python import nbformat as nbf nb = nbf.v4.new_notebook() cell = nbf.v4.new_code_cell("""# This is a code cell if 'sample_var' not in globals(): sample_var = 'sample value' print('Sample Notebook has been executed!')""") cell.metadata = { "language": "python" } nb.cells.append(cell) # Save the notebook to a file with open('sample_notebook.ipynb', 'w') as f: nbf.write(nb, f) print("Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.") ``` ## Example Notebook The following notebook shows how to run a notebook from another notebook with Fusion SQL: ## Upload the Notebook to Shared Notebooks in the Editor This step generates a unique notebook name by appending a timestamp to the notebook filename to avoid naming conflicts. Skip this step if you already created a Shared notebook via the UI in the first step. ```python import time sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000)) %sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb'; print("Notebook '{}' has been created in the Editor shared files.".format(sample_notebook_name)) ``` ## Run the Notebook in the Current Session Use `%run_shared` to execute the sample notebook in the current session. Confirm that the `sample_var` variable set in the sample notebook is accessible in the current session. ```python if 'sample_var' in globals(): del sample_var %run_shared {{ sample_notebook_name }} print("The value of 'sample_var' is '{}'.\n".format(sample_var)) ``` **Note**: If you created the shared notebook via the UI, replace `sample_notebook_name` in the code above with the actual notebook name in single quotes: ```python % run_shared {{ 'Sample Notebook.ipynb' }} ``` ## Run the Sample Notebook in a New Session You can also run the sample notebook **in a new session** using jobs. This allows you to execute multiple notebooks in parallel using `RUN JOB USING NOTEBOOK`. Refer to [RUN JOB USING NOTEBOOK](https://docs.singlestore.com/cloud/reference/fusion-sql/run-job-using-notebook.md) for syntax and more information. ```python job_ids = [] for x in range(2): print("Running job for {}...".format(x)) job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {"sample_var": "{{x}}"} job_ids.append(job_res[0].JobID) print(f'Waiting for jobs to complete... {job_ids}') success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES print(f'All jobs completed with success: {bool(success[0].Success)}') ``` **Note**: If you created the shared notebook via UI, replace `sample_notebook_name` in the code with your notebook name in single quotes (no brackets): ```sql %sql RUN JOB USING NOTEBOOK 'Sample Notebook.ipynb' WITH PARAMETERS {"sample_var": "{{x}}"} ``` ## View Job Executions Use `SHOW JOB EXECUTIONS` to inspect job runs. ```python for job_id in job_ids: execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1 print(execs) ``` ## Delete the Jobs Use `DROP JOBS` to delete the jobs. ```python for id in job_ids: print(f"Dropping job '{id}'...") %sql DROP JOBS {{id}} ``` **Note**: You can also view, inspect, and delete jobs from the **Jobs** section in the left navigation on the Cloud Portal. Refer to [Scheduling Notebooks with SingleStore Job Service](https://docs.singlestore.com/cloud/container-services/scheduled-jobs.md) for more information. ## Delete the Sample Notebook Delete the sample notebook using `DROP SHARED FILE`, or via the Cloud Portal by selecting **Delete** from the **Actions** column for your notebook. ```sql %%sql DROP SHARED FILE '{{ sample_notebook_name }}' ``` *** Modified at: May 19, 2026 Source: [/cloud/container-services/notebooks/running-a-notebook-from-another-notebook-using-fusion-sql/](https://docs.singlestore.com/cloud/container-services/notebooks/running-a-notebook-from-another-notebook-using-fusion-sql/) (An index of the documentation is available at /llms.txt) --- # Using Notebooks SingleStore Helios supports Jupyter Notebooks for development using SQL and Python. ## Import and Run Notebooks > **💡 Tip**: **Get Started Quickly**: Notebooks in SingleStore Helios are Jupyter notebooks and if you are familiar with how to use them, you can jump right in and get started. You can access a collection of notebooks in the Gallery, where you can find how-to guides, tutorials, and getting-started information. ## Import Notebooks To access Notebooks on the [Cloud Portal](https://portal.singlestore.com) , select **Editor**. There are multiple ways to get started using notebooks: * **New Notebook**: Start with a new notebook (select **New > New Notebook**) or use a beginner notebook template such as "SingleStoreDB Notebook Basics" (recommended). * **Import Notebook**: Select **New > Import From File** to import a notebook from your local machine using the **Import file** dialog. Notebook files must be in Jupyter format (`.ipynb`) and the file must not exceed 5GB in size. * **Gallery**: Select the **Gallery** tab to find how-to guides, tutorials, and getting-started notebooks. If you are new to notebooks, SingleStore recommends using the `beginner` flag and taking a look at the "SingleStoreDB Notebook Basics" or "Getting Started with Notebooks" notebooks. Use labels to filter the notebooks in Gallery. To use or explore a notebook, select the notebook, and then select **Try It Now**. Once opened, this notebook is saved and accessible from the **Recent** and **Personal** tabs on the **Editor** page. ## Run Notebooks You can run a notebook on the Cloud Portal using any of the following methods. ## Using the Notebooks UI To run a notebook using the Notebooks UI, 1. Open the notebook on the Cloud Portal. 2. Select a SingleStore deployment, database, and available runtime to run the notebook. 3. Use the available options in the **Run** menu to run the selected cells or all the cells in the notebook. ## Using Fusion SQL Commands in a Notebook Cell Use the `RUN JOB USING NOTEBOOK` Fusion SQL command in a notebook cell to run another notebook. For example: ```sql RUN JOB USING NOTEBOOK 'example_notebook.ipynb' WITH RUNTIME 'notebooks-cpu-small'; ``` Refer to [RUN JOB USING NOTEBOOK](https://docs.singlestore.com/cloud/reference/fusion-sql/run-job-using-notebook.md) for more information on this Fusion SQL command. Optionally, explore the [Running Notebooks from Another Notebook with Fusion SQL](https://www.singlestore.com/spaces/running-notebooks-from-another-notebook-with-fusion-sql/) notebook for an end-to-end example of running a notebook from a different notebook. ## Manage Notebooks You can perform the basic, non-development notebook tasks such as find, share, create, open/edit, delete, and rename. ## Shared and Personal Tabs Notebooks are stored in either the **Shared** or **Personal** tabs in SingleStore Helios. Notebooks stored in the Shared tab are accessible by any member of the organization. Notebooks stored in the Personal tab are only visible to the notebook creator and cannot be shared. ## Create or Open/Edit a Notebook To create a notebook: 1. From **New**, select **New Notebook**. ![Navigate to Editor and create a new notebook.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt519b2348cc165197/6a7adb19fc63ab3f64641839/Screenshot_2025-11-18_at_11_51_24_AM-0TgP4z.png) 2. Enter a name for the notebook. 3. Select a location ([Personal or Shared](https://docs.singlestore.com/#section-idm4531404441417633540186021063.md)) for the notebook. 4. Select a template from the template list. To create an empty notebook, select **(Blank Notebook)**. 5. Select **Create**. ![Create a blank notebook.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt62b067b88da6e0a2/6a7adb1796b5a6e41f86fdca/Screenshot_2025-11-18_at_12_06_30_PM-5tzXDz.png) To open (or edit) an existing notebook, select the notebook in the list of notebooks. ## Save a Notebook Notebooks are automatically saved (autosave) every 5 seconds. ## Share a Notebook All users within the same organization have access to the notebooks that are located in the **Shared** tab. To share a notebook, navigate to the **Shared** tab, select **Copy Link** from the (three-dot) under the **Actions** column for your notebook. Share the link with other users in the same organization. To share a notebook , navigate to the **Personal** tab, [duplicate](https://docs.singlestore.com/#section-idm4498060002691233569399455212.md) the notebook and change the location to a **Shared** tab. ## Rename a Notebook To rename a notebook, select **Rename** from the (three-dot) under the **Actions** column for your notebook ## Duplicate a Notebook To duplicate a notebook, select **Duplicate** from the (three-dot) under the **Actions** column for your notebook. Enter a name for the notebook and select a location. If you duplicate a notebook from the Personal tab to a Shared tab, the new notebook will not be private. ## Delete a Notebook To delete a notebook, select **Delete** from the (three-dot) under the **Actions** column for your notebook *** Modified at: May 25, 2026 Source: [/cloud/container-services/notebooks/using-notebooks/](https://docs.singlestore.com/cloud/container-services/notebooks/using-notebooks/) (An index of the documentation is available at /llms.txt) --- # Python UDFs > **📝 Note**: This is a Preview feature. A Python User-Defined Function (UDF) is an external function that allows you to execute Python code outside of the SingleStore engine's process. It enables you to extend SingleStore with custom Python logic in a SingleStore Notebook. Python UDFs are especially useful when you need to integrate with AI applications, machine learning (ML) models and perform vector operations with libraries like NumPy, Pandas, Polars, or call external APIs. ## Prerequisites To enable Python UDFs in the SingleStore deployment, ensure the following: * **SingleStore Version**: SingleStore version 8.9 or later. * **Environment**: The SingleStore deployment must run in an AWS EKS IRSA-supported environment. After ensuring, contact [SingleStore Support](http://support.singlestore.com) to enable this feature for your organization. ## Publish a Python UDF ## Create a Python UDF Python UDFs can be created in Shared notebook only. Python UDFs are not supported for Shared Edition clusters. Python UDFs can only be installed per individual database, which requires a Standard or Enterprise cluster. To create a new Python UDF, perform the following steps: 1. In the left navigation, select **Editor > Shared**. 2. Create or open a shared notebook. 3. Select **Publish** (on the top right). ## New Python UDF After selecting **Publish**, a new dialog box appears. **Publish Settings** | **Publish as** | Select**Python UDF**. | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Enter a name of the Python UDF. | | **Description** | Enter the Python UDF description. | | **Notebook** | Select a shared notebook to publish as a Python UDF. The shared notebook is pre-selected when the Python UDF is published through notebooks. | | **Deployment** | Select theSingleStoredeployment (cluster) the notebook connects to.Selecting aclusterallows connecting to theSingleStoredatabases referenced in the notebook natively. | | **Runtime** | Select a runtime from the following:
  • Small
  • Medium
  • GPU-T4

📝 Note:

This field is in preview.

| | **Region** | Select a region. | | **Idle Timeout** | Select an idle timeout.

📝 Note:

This field is in preview.

| Select **Next**. Select **Publish** to publish the notebook as Python UDF. Once the Python UDF is published, call the function using SQL Editor. > **📝 Note**: Python UDF names must be unique. If you attempt to create a Python UDF with a name that already exists, the system returns a "Duplicate UDF Function Name" error. To update an existing Python UDF, delete the existing UDF from Container Services and then create the new Python UDF with the same name. ## Example Notebook The following notebook shows how to publish your first Python UDF: ## Manage an Existing Python UDF To view an existing Python UDF, select **Python UDFs** in the left navigation. Existing Python UDFs can be managed by performing the following actions: * View * Update * Delete ## View an Existing Python UDF To view an existing Python UDF, select the Python UDF from the **Name** column. Following actions can be performed for a dashboard app from this page: * View Live Logs * Update * Delete ## View Live Logs To view live logs of the selected Python UDF, select **View Live Logs** from the ellipsis on the right side. A new window appears, where the **Timestamp** and the message in the **Body** column can be viewed. View the **Log JSON** by selecting the eye icon. ## Update an Existing Python UDF To update an existing Python UDF, select the ellipsis in the Actions column of the Python UDF, and select **Update**. ## Delete an Existing Python UDF To delete an existing Python UDF, select the ellipsis in the **Actions** column of the Python UDF, and select **Delete**. ## Status of Python UDF | Status | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Initializing | The notebook is creating or updating the Python UDF. | | Active | The notebook is successfully published as a Python UDF. | | Failed | The Python UDF is not configured correctly. View the notebook snapshot to debug the failure. | | Error | An error unrelated to the notebook code prevented the Python UDF from initializing. Refer to[Troubleshoot Python UDFs](https://docs.singlestore.com/#section-id23547459240214.md). | ## Troubleshoot Python UDFs SingleStore automatically saves a snapshot of the notebook for each failed execution. Navigate to **Container Services** in the left navigation, select the Python UDF, and download the snapshot associated with the failed Python UDF to diagnose the error. | Error | Solution | | ---------------------------- | --------------------------------------------------------------------------------------------------------- | | ClusterDeleted | Update the Python UDF and select a different deployment. | | ClusterSuspended | Resume theclusteror create a new Python UDF with a different deployment. | | Database Detached | Reattach the database with the right permissions or create a new Python UDF with a different database. | | Notebook Deleted/Not Present | Create a new Python UDF with a different notebook. | | Internal Errors/Misc | Contact[SingleStore Support](http://support.singlestore.com/)or use the chat feature in the Cloud Portal. | ## Defining Python UDFs Each Python UDF must meet the following requirements: 1. The function's parameters and return types must be annotated. 2. The function must be wrapped with the `@udf` decorator, which is located in `singlestoredb.functions`. The `@udf` decorator is a critical component, as it automatically analyzes the type annotations to map Python data types to SingleStore data types. The mapping is subsequently used to generate the necessary `CREATE EXTERNAL FUNCTION` statement in the SingleStore database, ensuring a reliable connection between the Python code and the SQL queries. Refer to [Equivalent Data Types](https://docs.singlestore.com/cloud/container-services/python-udfs/working-with-python-udfs/#section-id235163203492027.md) for related information. There are two main types of Python UDFs, defined by the type annotations: * Scalar * Vectorized ## Scalar Python UDFs Scalar Python UDFs are defined with standard Python type annotations, such as `int`, `float`, or `str`. When called from the database, the Python UDF server receives a batch of rows, but the Python UDF itself is invoked once for each individual row of data. This is useful in complex logic with individual records. The following example demonstrates a scalar Python UDF: ```python from singlestoredb.functions import udf import singlestoredb.apps as apps @udf async def multiply(x: float, y: float) -> float: return x * y # Start Python UDF server connection_info = await apps.run_udf_app() print("UDF server running. Connection info:", connection_info) ``` This creates the following external function: ```sql CREATE EXTERNAL FUNCTION multiply(x DOUBLE NOT NULL, y DOUBLE NOT NULL) RETURNS DOUBLE NOT NULL AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` Use `async def` for improved cancellation handling. Invoke the scalar Python UDF using the following command: ```sql SELECT multiply(5.0, 10.0) AS result; ``` ```output +-----------------------+ | result | +-----------------------+ | 50.0 | +-----------------------+ ``` ## Vectorized Python UDFs Vectorized Python UDFs are defined with vector type annotations, such as `numpy.ndarray`, `pandas.Series`, `polars.Series`, or `pyarrow.Array`. The Python UDF is called only once for each batch of rows received from the SingleStore database. The entire batch is converted into vectorized inputs, where each column of data corresponds to a single vector object passed as a function parameter. This is useful in high-performance numerical processing. The following example demonstrates a vectorized Python UDF. ```python import numpy as np import numpy.typing as npt from singlestoredb.functions import udf @udf async def vec_multiply( x: npt.NDArray[np.float64], y: npt.NDArray[np.float64] ) -> npt.NDArray[np.float64]: return x * y # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This creates the following external function: ```sql CREATE EXTERNAL FUNCTION vec_multiply(x DOUBLE NOT NULL, y DOUBLE NOT NULL) RETURNS DOUBLE NOT NULL AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` Invoke the vectorized Python UDF using the following command: ```sql SELECT vec_multiply(vec_col1, vec_col2)AS result; ``` ```output +-------------------+ | result | +-------------------+ | [4.0, 10.0, 18.0] | +-------------------+ ``` Where `vec_col1 = [1,2,3]` and `vec_col2 = [4,5,6]`. ## Scalar Python TVFs Scalar Python TVFs are defined in the same way as scalar Python UDFs, except that a scalar TVF uses a `Table` annotation to indicate that the function returns a table. The function must also return the final result wrapped in a `Table` object. The following example demonstrates a scalar Python TVF: ```python import numpy as np import numpy.typing as npt from singlestoredb.functions import udf, Table @udf async def number_stats( n: npt.NDArray[np.int_], ) -> Table[npt.NDArray[np.int_], npt.NDArray[np.int_], npt.NDArray[np.float64]]: numbers = np.arange(1, n[0] + 1, dtype=np.int_) squares = numbers ** 2 roots = np.sqrt(numbers).round(2) return Table(numbers, squares, roots) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This creates the following external function: ```sql CREATE EXTERNAL FUNCTION `number_stats`(`n` BIGINT NOT NULL) RETURNS TABLE( `numbers` BIGINT NOT NULL, `squares` BIGINT NOT NULL, `roots` DOUBLE) AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` Invoke the scalar Python TVF using the following command: ```sql SELECT * FROM number_stats([5]); ``` ```output +---------+---------+-------+ | numbers | squares | roots | +---------+---------+-------+ | 1 | 1 | 1.00 | | 2 | 4 | 1.41 | | 3 | 9 | 1.73 | | 4 | 16 | 2.00 | | 5 | 25 | 2.24 | +---------+---------+-------+ ``` SingleStore automatically generates generic column names if no names are associated with the return fields (`numbers`, `squares`, and `roots` in this example). Explicitly name result columns using overrides or schema classes. Refer to [Overriding Parameters and Return Value Types](https://docs.singlestore.com/cloud/container-services/python-udfs/working-with-python-udfs/#section-id235163211595581.md) for related information. ## Vectorized Python TVFs Vectorized Python TVFs return a table of results. Unlike Python UDFs, which are invoked for each row in a query, a TVF is invoked once with a set of parameters and returns multiple rows and columns. In a vectorized Python TVF, each returned vector represents a column in the output table. The length of each vector determines the number of rows returned. The following example demonstrates a vectorized Python TVF: ```python import numpy as np import numpy.typing as npt from singlestoredb.functions import udf, Table @udf async def vec_table_function( n: npt.NDArray[np.int_], ) -> Table[npt.NDArray[np.int_], npt.NDArray[np.float64], npt.NDArray[np.str_]]: x = np.array([10] * n[0], dtype=np.int_) y = np.array([10.0] * n[0], dtype=np.float64) z = np.array(['ten'] * n[0], dtype=np.str_) # Returns a tuple of vectors (each column of the output) return Table(x, y, z) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This creates the following external function: ```sql CREATE EXTERNAL FUNCTION `vec_table_function`(`n` BIGINT NOT NULL) RETURNS TABLE( `x` BIGINT NOT NULL, `y` DOUBLE NOT NULL, `z` TEXT NOT NULL ) AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` Invoke the vectorized Python TVF using the following command: ```sql SELECT * FROM vec_table_function([1]); ``` ```output +----+------+-----+ | x | y | z | +----+------+-----+ | 10 | 10.0 | ten | +----+------+-----+ ``` SingleStore automatically generates generic column names if no names are associated with the return fields (`x`, `y`, `z`). Explicitly name result columns using overrides or schema classes. Refer to [Overriding Parameters and Return Value Types](https://docs.singlestore.com/cloud/container-services/python-udfs/working-with-python-udfs/#section-id235163211595581.md) for related information. ## In this section * [Working with Python UDFs](https://docs.singlestore.com/cloud/container-services/python-udfs/working-with-python-udfs.md) *** Modified at: August 4, 2026 Source: [/cloud/container-services/python-udfs/](https://docs.singlestore.com/cloud/container-services/python-udfs/) (An index of the documentation is available at /llms.txt) --- # Working with Python UDFs ## Equivalent Data Types When defining Python UDFs, input parameters and return values must be mapped between Python and SingleStore data types. The framework automatically infers these mappings from type annotations, but they can be overridden if necessary. Following are the default Python to SingleStore mappings: | **Python Type** | **SingleStoreType** | | --------------- | ------------------- | | `int` | `BIGINT` | | `float` | `DOUBLE` | | `str` | `TEXT` | | `bytes` | `BLOB` | | `bool` | `BOOL` | For finer control, use the `singlestoredb.functions.dtypes` module to specify types such as `SMALLINT`, `VARCHAR`, `DECIMAL`, or `JSON`. ## Handling NULL Values By default, Python UDFs and TVFs do not allow `NULL` values. The method for enabling `NULL` support depends on whether the Python function is scalar or vectorized. ## Scalar Python Functions (UDFs and TVFs) * Use `Optional[...] `(or `type` | `None` in Python 3.10+) to allow parameters and return values to accept `NULL`. * This makes the entire parameter or return value nullable. * If any argument is `NULL`, the function can return `NULL` as needed. ## Vectorized Python Functions (UDFs and TVFs) * `Optional[...]` cannot be used for element-level NULLs inside vectors. * Instead, use the `Masked` annotation for parameters and return values that must support `NULL`. * A `Masked` value consists of: * A data vector containing the non-NULL values. * A boolean mask vector, where `True` indicates a `NULL` element. * This ensures that `NULL` values are preserved and propagated correctly across inputs and outputs. * In Python TVFs, each output column can independently use `Masked` to indicate nullability. ## Overriding Parameter and Return Value Types You can specify an output schema using the `returns=` parameter in the `@udf` decorator. This schema can be defined as: * A list of SQL types with `name=` specified, or * A `NamedTuple`, `TypedDict`, or `pydantic.BaseModel`. The schema class is automatically inferred and it does not need to be returned explicitly, it is only used to define the output schema. ## Cancelling Running Python UDFs When a query that uses a Python UDF is cancelled or when the connection between the database engine and the Python UDF server is broken, the Python UDF execution continues until it can be safely interrupted. * **Synchronous Python UDFs** * **Scalar Python UDFs**: Cancellation can only occur between row function calls. * **Vectorized Python UDFs**: Cancellation occurs only after all rows in the current batch are processed. * **Asynchronous Python UDFs** * Cancellation is detected more quickly. * When a disconnect is detected, an `asyncio.CancelledError` is raised the next time the Python UDF becomes active. * If synchronous operations are used inside an `async` Python UDF, they must complete before cancellation occurs. * Nested async calls are cancelled as soon as they activate again. SingleStore recommends using `async` before the function definition for better cancellation handling. > **📝 Note**: Use asynchronous Python UDFs and `async` libraries wherever possible to ensure that Python UDFs can be cancelled promptly. ## Timeouts A timeout can be applied directly in the `@udf`decorator using the `timeout=` parameter. * The value must be specified in seconds. * If the timeout period is exceeded, the Python UDF is cancelled automatically. ## User Permissions To allow users other than the organization owner to execute Python UDFs or TVFs, the organization owner must grant the appropriate permissions. These permissions are applied within the customer’s cluster using the SQL Editor. ## Required Permissions 1. `EXECUTE` (Object-Level Permission) * Required to execute a function in a specific database. * Can be granted on all functions in a database or a specific function. * Following example demonstrates the syntax: ```sql GRANT EXECUTE ON . TO 'user'@'%'; ``` 2. `OUTBOUND` (Global Permission) * Required to allow functions to make external network requests. * Must be granted globally. * Following example demonstrates the syntax: ```sql GRANT OUTBOUND ON *.* TO 'user'@'%'; ``` > **📝 Note**: Both permissions must be granted to ensure the user can execute Python UDFs or TVFs successfully. Without these permissions, execution may fail due to insufficient privileges. ## Examples ## Example 1: Calculate Sales Metrics with Vectorized Python TVFs This example demonstrates a vectorized Python TVF that computes total and average sales per row and returns a table with named columns. ```python import typing import numpy as np import pandas as pd from singlestoredb.functions import udf, Table import numpy.typing as npt # Define output schema using NamedTuple class SalesOutput(typing.NamedTuple): total_sales: float average_sales: float category: str @udf(returns=SalesOutput) async def vector_sales( units_sold: npt.NDArray[np.float64], # Vector of units sold unit_price: npt.NDArray[np.float64], # Vector of unit prices category: npt.NDArray[np.str_] # Vector of categories ) -> Table[pd.DataFrame]: """ Calculate total and average sales per row with category. Parameters ---------- units_sold : np.ndarray[np.float64] Number of units sold unit_price : np.ndarray[np.float64] Price per unit category : np.ndarray[str] Category name for each row Returns ------- pd.DataFrame Table with columns: - total_sales - average_sales - category """ # Compute vectorized total and average total_sales = units_sold * unit_price average_sales = total_sales / np.maximum(units_sold, 1) # Avoid divide by zero # Return as a DataFrame with named columns df = pd.DataFrame({ 'total_sales': total_sales, 'average_sales': average_sales, 'category': category }) return Table(df) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This results in the following external function: ```sql CREATE EXTERNAL FUNCTION `vector_sales`( `units_sold` DOUBLE NOT NULL, `unit_price` DOUBLE NOT NULL, `category` TEXT NOT NULL ) RETURNS TABLE( `total_sales` DOUBLE NOT NULL, `average_sales` DOUBLE NOT NULL, `category` TEXT NOT NULL ) AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` ## Example 2: Vector Embeddings This example demonstrates a Python UDF that can connect to any deployed embedding service in SingleStore Aura. ```python import base64 import sys import numpy as np import requests import numpy.typing as npt from singlestoredb.functions import udf # Configuration for the embedding service MODEL_NAME = 'name-of-embedding-service' MODEL_URL = 'url-of-embedding-server' TOKEN = 'api-token-of-embedding-service' HEADERS = { 'accept': 'application/json', 'Content-Type': 'application/json', 'Authorization': f'Bearer {TOKEN}', } @udf async def mixedbread_embeddings( text: npt.NDArray[np.str_], # Input: array of strings ) -> npt.NDArray[np.bytes_]: # Output: array of bytes """ Generate vector embeddings for an array of text inputs. Parameters ---------- text : numpy.ndarray Array of input strings Returns ------- numpy.ndarray Array of byte embeddings """ # Send request to embedding service res = requests.post( MODEL_URL, headers=HEADERS, json=dict( model=MODEL_NAME, input=text.tolist(), encoding_format='base64', ), ) # Raise an error if the request failed if res.status_code >= 400: print(res.content.decode('utf8'), file=sys.stderr) raise RuntimeError(res.content.decode('utf8')) # Decode embeddings and return as a NumPy array of bytes return np.array( [base64.b64decode(x['embedding']) for x in res.json()['data']], dtype=object, # Preserve byte arrays correctly ) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This results in the following external function: ```sql CREATE EXTERNAL FUNCTION `mixedbread_embeddings`(`text` TEXT NOT NULL) RETURNS BLOB NOT NULL AS REMOTE SERVICE "base-url-of-udf-server-including-port" FORMAT ROWDAT_1; ``` ## Example 3: OpenAI Integration with Structured Output This example demonstrates a Python table-valued function (TVF) that returns structured output using a `pydantic.BaseModel`. The function takes a word as input and returns 10 synonyms along with a score indicating how close each synonym is to the original word. ```python import os from typing import List from pydantic import BaseModel, Field from singlestoredb.functions import udf, Table import instructor from openai import OpenAI # Create OpenAI client via instructor wrapper client = instructor.from_openai(OpenAI(api_key=os.getenv("OPENAI_API_KEY"))) # Define output schema class Synonym(BaseModel): word: str = Field(description='Synonym of the given word') score: float = Field( description='Closeness score from 0.0 to 1.0' ) # Define the TVF @udf async def get_synonyms(word: str) -> Table[List[Synonym]]: """Return a list of synonyms of the given word and a score.""" return Table(client.create( model='gpt-4o-mini', messages=[ dict(role='system', content='You are a helpful assistant'), dict( role='user', content=f''' * Get a list of synonyms of the word "{word}" * Limit the number of results to 10 ''' ) ], response_model=List[Synonym], max_retries=0, )) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` This results in the following external function: ```sql CREATE EXTERNAL FUNCTION `get_synonyms`(`word` TEXT NOT NULL) RETURNS TABLE( `word` TEXT NOT NULL, `score` DOUBLE NOT NULL ) AS REMOTE SERVICE "http:////invoke" FORMAT ROWDAT_1; ``` Run the following command to retrieve the synonyms of the word “danger”. ```sql SELECT * FROM get_synonyms('danger'); ``` The command results the following: ```sql word score 0 hazard 0.90 1 risk 0.85 2 threat 0.88 3 peril 0.90 4 jeopardy 0.87 5 threatening 0.80 6 threatening situation 0.70 7 threatened 0.75 8 insecurities 0.60 9 unsafe 0.65 ``` ## Example 4: Machine Learning Model Scoring This example demonstrates how to use a pre-trained Keras model stored in [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) to score a collection of input parameters. ```python import numpy as np import pandas as pd import numpy.typing as npt import singlestoredb.notebook as nb from tensorflow.keras.models import load_model from singlestoredb.functions import udf # Download model from Stage nb.stage.download_file('my_model.keras', local_path='my_model.keras') # Load the model model = load_model('my_model.keras') @udf async def keras_score( param_1: npt.NDArray[np.float32], param_2: npt.NDArray[np.float32], param_3: npt.NDArray[np.float32], ) -> npt.NDArray[np.float32]: """ Score rows using a Keras model. Parameters ---------- param_1 : np.ndarray[np.float32] First input parameter. param_2 : np.ndarray[np.float32] Second input parameter. param_3 : np.ndarray[np.float32] Third input parameter. Returns ------- np.ndarray[np.float32] Predicted scores from the model. """ X_test = pd.DataFrame({ 'param_1': param_1, 'param_2': param_2, 'param_3': param_3 }) return model.predict(X_test).reshape((-1,)) # Start Python UDF server import singlestoredb.apps as apps connection_info = await apps.run_udf_app() ``` ## Example 5: Overriding Types with dtypes By default, Python types map to general SQL types (for example, `int` maps to `BIGINT`). Refer to [Equivalent Data Types](https://docs.singlestore.com/#section-id235163203492027.md) for related information. Use types from `singlestoredb.functions.dtypes` to override these defaults and define precise SQL signatures. Pass these types to the `@udf(args=[...], returns=...)` decorator. The following examples show how to override default type mappings: * Use `SMALLINT` for integer inputs and outputs This example overrides the default `BIGINT` mapping for Python `int` by explicitly using `SMALLINT`. Use this approach when it is required to match an existing schema that uses smaller integer types. ```python from singlestoredb.functions import udf from singlestoredb.functions.dtypes import SMALLINT @udf( args=[SMALLINT(nullable=False)], returns=SMALLINT(nullable=False), ) def double_small(x: int) -> int: return x * 2 ``` * Control string length and return structured JSON data This example defines a bounded string input using `VARCHAR(n)` and returns structured data as JSON. Use `VARCHAR(n)` to enforce input size limits and return a JSON-encoded string when using the `JSON` type. ```python from singlestoredb.functions import udf from singlestoredb.functions.dtypes import JSON, VARCHAR @udf( args=[VARCHAR(128, nullable=False)], returns=JSON(nullable=False), ) def wrap_as_json(label: str) -> str: import json return json.dumps({"label": label}) ``` > **📝 Note**: - Ensure `VARCHAR(n)` length matches the expected input size. Values longer than this limit may be truncated or rejected. > - When using `JSON` as the return type, return a JSON-encoded string. > - Set `nullable` to match whether your Python types allow `None`. *** Modified at: May 22, 2026 Source: [/cloud/container-services/python-udfs/working-with-python-udfs/](https://docs.singlestore.com/cloud/container-services/python-udfs/working-with-python-udfs/) (An index of the documentation is available at /llms.txt) --- # Scheduled Jobs A scheduled execution of a notebook is referred to as a “job.” The SingleStore job service provides a flexible and convenient way to schedule jobs using cron expressions that can be tailored to meet your requirements. Cron-based scheduling uses the [standard cron syntax](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/), providing precise control over when workloads are run. It supports the format - minute hour day month weekday, and allows for complex scheduling patterns (e.g., running jobs at 8:00 AM every Monday). Create, edit, and delete cron schedules, and validate cron expressions in real time. After scheduling, the job service creates a snapshot. This snapshot represents a read-only notebook captured at the time the job was scheduled. It can be used for subsequent executions at the scheduled time until the user makes modifications. To ensure that the execution reflects the new changes, be sure to update the snapshot explicitly after modifying the notebook. Create and manage scheduled jobs using the Cloud Portal or via the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md). ## Scheduled Jobs using the Cloud Portal ## Create a Scheduled Job Jobs can be scheduled for shared notebooks only. Create a scheduled job using the one of the following methods: 1. Navigate to **Jobs** in the left navigation and select **Schedule** (on the top right). 2. Navigate to **Editor > Shared**. 1. Select a shared notebook. 2. Select **Schedule** (on the top right). ## New Scheduled Job After selecting **Schedule**, enter or select the following in **Job Settings**. | **Job Settings** | | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Name** | Enter a name for the job. | | **Description** | Enter the job description. | | **Notebook** | Select a shared notebook to run the job. The shared notebook is pre-selected when Job is scheduled through notebooks. | | **Cluster** | Select theSingleStoreclusterthe notebook connects to.Selecting aclusterallows connecting theSingleStoredatabases referenced in the notebook natively. A scheduled job can also be run without aclusterattached to it. | | **Runtime** | Select a runtime from the following:
  • Small
  • Medium
  • GPU-T4

📝 Note:

This field is in preview.

| | **Region** | Select a region. | Select **Next**. Enter the **Schedule Options**. | **Scheduled Options** | | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Start Executions After** | Specify the date and time for the first run of the scheduled job. Select one of the following available options:
  1. Now: The job runs as soon as it is created.
  2. Datetime: Specify a Starting Date and Starting Time.
| | **Schedule Mode** | Specify the frequency to run a scheduled job. Select one of the following available options:
  1. One Time: The job runs a single time.
  2. Recurring: Specify when the job needs to be run. Select from the following:
    • Hourly
    • Weekly
    • Monthly
    • Weekdays at 9 PM
    • Every 6 Hours
    • Enter a custom valid CRON expression.
| | **Parameters** | Specify the job parameters for the scheduled job.
  1. Type: Select the type of parameters from the following:
    • String
    • Integer
    • Float
    • Boolean
  2. Name: Enter the name of the parameter.
  3. Value: Enter the value of the parameter.

📝 Note:

The system injects parameters as Python variables into the notebook while the scheduled job runs, and access them using the variables where required.

| | **Additional Settings** |
  1. Save all execution results: Enable this setting to save the execution results. An execution result is a saved version of the notebook along with the cells containing outputs after the completion of a job run. This action enables to validate the execution result and the version of the notebook used for the run. By default, all snapshots for error and failed runs are saved.
  2. Auto resume the cluster on job execution: Enable this setting to resume the cluster when the job runs. This action resumes any suspended clusters. The previous auto-suspend settings of the clusters remain unchanged.
| Select **Next**, review the **Summary**, and then select **Create Job** to schedule the job. ## Manage an Existing Job To view an existing scheduled jobs and job runs, select **Jobs** in the left navigation pane. Existing jobs can be managed by performing the following actions: * View * Edit * Pause * Resume * Delete ## View an Existing Job To view a job, select the job name under the **Name** column and view the job execution details for each job. View the details of a job on the **Job Details** pane in the right navigation. This page also supports refresh, edit, pause, resume, and delete actions for a job. ## Edit an Existing Job To edit an existing job, navigate to the selected job page, select the ellipsis, and select **Edit Job**. ## Pause an Existing Job To pause an existing job, navigate to the **Jobs** page, select the ellipses in the **Actions** column of the job, and select **Pause**. Alternatively, navigate to the selected job page, and select **Pause Job**. ## Resume an Existing Job To resume an existing job, navigate to the **Jobs** page, select the ellipses in the **Actions** column of the job, and select **Resume**. ## Delete an Existing Job To delete an existing job, navigate to the **Jobs** page, select the ellipses in the **Actions** column of the job, and select **Delete**. > **⚠️ Warning**: Deleting a job removes all associated job runs, including snapshots. ## Schedule Jobs using the Management API Use the `Jobs` path (`/v1/jobs` endpoint) to create and manage jobs using the Management API. Refer to [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. For example, the following API call returns information about the job with the specified `jobID`: ```shell curl -X 'GET' \ 'https://api.singlestore.com/v1/jobs/7b68b78d-a000-4000-9000-0864fd331524' \ -H 'accept: application/json' ``` ## Status of Job Executions | Status | Description | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Scheduled | The next scheduled run for the job. The latest version of the notebook runs at the Start Time associated with the job. | | Completed | The notebook in this job run ran to completion. | | Failed | The execution failed due to user error. View execution result to debug the failure. | | Error | An error unrelated to the notebook code prevented this job from completing. Refer to[Troubleshoot Job Runs](https://docs.singlestore.com/#section-idm4502415224011234112089885605.md). | ## Troubleshoot Job Runs SingleStore automatically saves an execution result notebook of an execution that fails. Navigate to **Jobs** in the left navigation, find the job, and download the snapshot associated with the failed job run to diagnose the error. | Error | Solution | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ClusterDeleted | Edit the Job and select acluster. | | ClusterSuspended | Resume thecluster, or create a new scheduled job with a differentcluster. | | Database Detached | Reattach the database with the right permissions or create a new scheduled job with a different database. | | Internal Errors / Misc | Reach out to[SingleStoreSupport](http://support.singlestore.com)or use the chat feature in theCloud Portal. | | Notebook Timed Out | View the snapshot to identify the cells where the notebook is timed out.Refer to[Configurability and Key Considerations](https://docs.singlestore.com/#section-idm4561569057196834112075552002.md)for the Execution Time limits set on a job run. | | Notebook Deleted/Not present | Create a new scheduled job with a different notebook. | ## Configurability and Key Considerations Scheduled notebook jobs run using a snapshot of the notebook captured at the time the job is created or last refreshed. Updates made to the notebook after the snapshot is captured do not affect scheduled job runs. To apply updates to a scheduled job, explicitly refresh the notebook snapshot. In the **Jobs** page, select the job and then select **Refresh** in the top right. Alternatively, select the **ellipsis** (three dots), select **Edit Job**, and enable **Update snapshot** to reflect the updates from the associated notebook in the job. An execution time limit is set for each job: * **Maximum allowed execution duration**: Specifies the maximum total time a notebook job can run, including all retries. The job is terminated when this limit is reached, even if execution is still in progress. The maximum allowed execution duration limit is 8 hours. * **Minimum execution interval**: Specifies the minimum required time interval between consecutive notebook job runs. Schedules with intervals shorter than this limit are not allowed. The minimum execution interval limit is 60 minutes. * **Maximum execution interval**: Specifies the maximum allowed time between consecutive notebook job runs. Schedules with intervals longer than this limit are not allowed. The maximum execution interval limit is 31 days. ## Examples The following examples demonstrate how **maximum allowed execution duration**, **minimum execution interval**, and **maximum execution interval** are used while creating a scheduled job and how those patterns apply to scheduling notebooks within the SingleStore job service. ## Example 1: Hourly Data Refresh Job Run a notebook every hour to refresh business analytics tables. ## Configuration Cron Expression: `0 * * * *` **Execution Time Limits** | **Execution Time Parameter** | **Value** | | ---------------------------------- | ---------- | | Minimum execution interval | 60 minutes | | Maximum execution interval | 31 days | | Maximum allowed execution duration | 8 hours | ## Behavior This configuration allows the notebook to run once every hour, which satisfies the minimum execution interval of 60 minutes and remains within the maximum execution interval of 31 days. Each job run can execute for up to 8 hours, including retries, after which the system terminates the running job automatically. Cron expressions that schedule executions more frequently than once per hour, such as `*/30 * * * *` are not allowed. ## Example 2: Monthly Compliance Report Run a notebook once per month to generate a compliance report. ## Configuration Cron Expression: `0 0 1 * *` **Execution Time Limits** | **Execution Time Parameter** | **Value** | | ---------------------------------- | ---------- | | Minimum execution interval | 60 minutes | | Maximum execution interval | 31 days | | Maximum allowed execution duration | 8 hours | ## Behavior This configuration allows the notebook to run approximately once every 31 days, which satisfies both the minimum execution interval of 60 minutes and the maximum execution interval of 31 days. Each execution is limited to 8 hours, including retries, and the system terminates the running job if it exceeds this duration. Cron expressions that schedule executions less frequently than once every 31 days, such as `0 0 1 1 *` are not allowed. *** Modified at: July 7, 2026 Source: [/cloud/container-services/scheduled-jobs/](https://docs.singlestore.com/cloud/container-services/scheduled-jobs/) (An index of the documentation is available at /llms.txt) --- # Create a Database > **📝 Note**: As a database is automatically assigned to you in a SingleStore Helios [Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition.md) cluster, additional databases cannot be created. To create additional databases, and for even greater functionality, SingleStore recommends creating a [Standard](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md) cluster. After deploying your cluster, you are ready to create a database. This guide explains how to create a database using the [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) command. After you have set up your database, navigate to the [Choosing a Table Storage Type](https://docs.singlestore.com/cloud/create-a-database/choosing-a-table-storage-type.md) topic for more information on what table storage type will best suit your needs. ## In this section * [Choosing a Table Storage Type](https://docs.singlestore.com/cloud/create-a-database/choosing-a-table-storage-type.md) * [Columnstore](https://docs.singlestore.com/cloud/create-a-database/columnstore.md) * [Creating a Columnstore Table](https://docs.singlestore.com/cloud/create-a-database/creating-a-columnstore-table.md) * [Configuring the Columnstore to Work Effectively](https://docs.singlestore.com/cloud/create-a-database/configuring-the-columnstore-to-work-effectively.md) * [Rowstore](https://docs.singlestore.com/cloud/create-a-database/rowstore.md) * [Understanding Keys and Indexes in SingleStore](https://docs.singlestore.com/cloud/create-a-database/understanding-keys-and-indexes-in-singlestore.md) * [Understanding Shard Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection.md) * [Understanding Sort Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-sort-key-selection.md) * [Understanding How Datatype Can Affect Performance](https://docs.singlestore.com/cloud/create-a-database/understanding-how-datatype-can-affect-performance.md) * [Using JSON](https://docs.singlestore.com/cloud/create-a-database/using-json.md) * [Multi-Value Hash Index (JSON)](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json.md) * [Using Persistent Computed Columns](https://docs.singlestore.com/cloud/create-a-database/using-persistent-computed-columns.md) * [Other Schema Concepts](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md) * [Detecting and Resolving Data Skew](https://docs.singlestore.com/cloud/create-a-database/detecting-and-resolving-data-skew.md) * [Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md) * [Specifying Unenforced Unique Constraints](https://docs.singlestore.com/cloud/create-a-database/specifying-unenforced-unique-constraints.md) *** Modified at: May 11, 2026 Source: [/cloud/create-a-database/](https://docs.singlestore.com/cloud/create-a-database/) (An index of the documentation is available at /llms.txt) --- # Choosing a Table Storage Type The most important consideration when creating a table is its data layout. SingleStore Helios supports two types of tables: on-disk columnstores (which is the default table type for SingleStore Helios) and in-memory rowstore. Columnstore is also known as **Universal Storage**. The following table enumerates the strengths and intended uses of the columnstore and rowstore. | Flash, SSD, or Disk-based Columnstore/Universal Storage | In-Memory Rowstore | | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | Analytical workloads, but can handle operational/transactional workloads with relatively frequent updates and deletes | Demanding operational/transactional workloads with high update and delete rates | | Fast inserts over a small or large number of rows | Fast inserts and updates over a small or large number of rows | | Fast aggregations and table scans via sub-segment access | Random seek performance | | Compression | `SPARSE`compression option | ## Conceptual Differences Between Columnstore and Rowstore Column-oriented stores or “columnstores” (columnstore is also referred to as Universal Storage) treat each column as a unit. Columnstore tables sequentially store data segments for each column on the same physical location on disk. This enables two important capabilities. One capability is being able to scan only the columns needed for the query with good cache locality during the scan. The other capability is that columnstores lend themselves well to compression. For example, repeating and similar values can easily be compressed together. In addition, columnstore tables utilize a feature called sub-segment access with allows quick retrieval of a row once its position is known via hash indexing. Columnstore tables can have multiple multi-column hash indexes. Finally, row-level locking allows many concurrent updates of multiple rows in a columnstore to proceed, without requiring transactions to wait. Columnstores are usually batch-loaded for optimal performance, although small-batch and streaming insert to columnstores can also perform well. The combination of efficient batch loading and processing of analytic queries make columnstore highly suitable for data warehousing scenarios. Columnstores can handle operational and transactional workloads with relatively frequent updates and deletes, but rowstore can perform better for a demanding operational and transactional workload with high update and delete rates. [Column group indexes](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) can be used on columnstore tables to speed up full-row retrievals and updates, particularly for wide tables. In general, columnstore tables are effective when your workload matches the following descriptions as much as possible: * A large number of rows are scanned sequentially (i.e. millions of rows or >5% of the table) * Aggregation happens over only a few columns (e.g. <10 columns) A simplified example of how columnstore data is physically stored is shown here: **Columnstore** | ProductId | | --------- | | 1 | | 2 | | 3 | | 4 | | Color | | ---------- | | Red**x 2** | | Black | | White | *“x 2” indicates that the value is repeated twice.* | Price | | --------- | | 10 | | 20**x 3** | *“x 3” indicates that the value is repeated three times.* Row-oriented stores or “rowstores” are the most common type of data stores used by relational databases. As the name suggests, a rowstore treats each row as a unit and stores all fields for a given row together in the same physical location. This results in very fast performance when running queries seek specific rows. This makes rowstores great for transactional workloads, where the database frequently selects, inserts, updates, and deletes individual rows, often referencing either most or all columns. With rowstore, data is stored in lock-free indexes for great performance at high concurrency. Rowstore tables can use multiple indexes, allowing them to flexibly support many types of queries. Finally, when rowstore tables are created with [SPARSE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#UUID-003272e3-b877-41c1-7250-e56d395f7c4d.md) compression, the tables are allowed to store more data in the same amount of RAM. Nullable, structured columns can use sparse data compression. The column data types include `BIGINT`, `INT`, `MEDIUMINT`, `SMALLINT`, `TINYINT`, `DATE`, `DATETIME`, `TIME`, `TIMESTAMP`, and `VARCHAR`. **Logical table and rowstore representation** | ProductId | Color | Price | | --------- | ----- | ----- | | 1 | Red | 10 | | 2 | Red | 20 | | 3 | Black | 20 | | 4 | White | 20 | **Related Topics** * Training: [SingleStore Helios Storage](https://training.singlestore.com/learn/course/internal/view/elearning/679/storage) ## Check Your Understanding **Q**: Do columnstore tables use memory? **A**: Absolutely – SingleStore Helios uses the operating system disk buffer cache to cache segment files in memory. Good performance in columnstore can only be achieved when there is enough memory to cache the working set. In addition, columnstore tables use a rowstore buffer table as a special segment to batch writes to the disk. **Q**: Do rowstore tables ever write to disk? **A**: Yes – data in a rowstore is also written to the transaction log on disk so that it can be recovered when SingleStore Helios is restarted. **Q**: Are rowstore tables always faster than columnstore tables? **A**: No, columnstore tables are faster on some workloads – if the workload is batch inserts and sequential reads (e.g. an analytical workload with lots of scans) a columnstore can be significantly faster. **Q**: What if I need small high-concurrency updates and my data doesn’t fit in memory? **A**: You can also use columnstore tables while setting hash index(es) on columns you regularly filter on. Rowstore tables could also be effective if you create them with `SPARSE` compression. **Related Topics** * Training: [SingleStore Helios Storage](https://training.singlestore.com/learn/course/internal/view/elearning/679/storage) *** Modified at: February 8, 2024 Source: [/cloud/create-a-database/choosing-a-table-storage-type/](https://docs.singlestore.com/cloud/create-a-database/choosing-a-table-storage-type/) (An index of the documentation is available at /llms.txt) --- # Columnstore The columnstore is the default table type in SingleStore Helios. Columnstore is also known as **Universal Storage**. The following sections explain the basic concepts of columnstore/Universal Storage. ## In this section * [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) * [Writing Columnstore Data](https://docs.singlestore.com/cloud/create-a-database/columnstore/writing-columnstore-data.md) * [Locking in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/locking-in-columnstores.md) * [Managing Columnstore Segments](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments.md) * [Encoded Data in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/encoded-data-in-columnstores.md) * [Highly Selective Joins](https://docs.singlestore.com/cloud/create-a-database/columnstore/highly-selective-joins.md) * [Columnstore Seekability using JSON](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-seekability-using-json.md) * [Columnstore Sizing Estimations](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-sizing-estimations.md) * [Universal Storage](https://docs.singlestore.com/cloud/create-a-database/columnstore/universal-storage.md) *** Modified at: April 15, 2025 Source: [/cloud/create-a-database/columnstore/](https://docs.singlestore.com/cloud/create-a-database/columnstore/) (An index of the documentation is available at /llms.txt) --- # Columnstore Seekability using JSON SingleStore exposes a Javascript Object Notation (JSON) (An open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays.) column type that implements the [JSON standard](http://http://www.ietf.org/rfc/rfc4627.txt). The primary encoding for `JSON` columns in columnstore tables is `SeekableJson`. With this encoding, JSON data is automatically columnarized. Analytics on columns with the `SeekableJson` encoding are very efficient. `JSON` columns may also be encoded with the [seekable string encodings](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/encoding-string-data-in-columnstore-tables.md) `SeekableLZ4` and `SeekableStringRunLength`. These seekable string encodings compress data and save space, but do not support efficient search or extraction on JSON values. SingleStore recommends using `SeekableLZ4` and `SeekableStringRunLength` only for columns on which JSON values will be extracted as a whole. The non-seekable, columnarized format, `JSONParquet`, is provided for backwards compatibility. The following table summarizes the properties of the encodings. | Encoding | Seekable | Columnarized | Supports Efficient Search and Extraction | Default | | -------------------------------------- | -------- | ------------ | ------------------------------------------- | ------- | | `SeekableJson` | Yes | Yes | Yes | Yes | | `SeekableLZ4``SeekableStringRunLength` | Yes | No | No | No | | `JSONParquet` | No | Yes | Yes, but less efficient than`SeekableJson`. | No | ## Choose an Encoding SingleStore recommends using `SeekableJson` for any `JSON` columns on which analytics, search, or extraction will be done. That is, if you use functions like `JSON_EXTRACT_`, the `::` operators, or `JSON_MATCH_ANY` on the `JSON` column, use the `SeekableJson` encoding for good performance. `SeekableJson` is the default encoding. SingleStore recommends using `SeekableLZ4` (or `SeekableStringRunLength`) if you are concerned about memory usage and will not be searching on or extracting from the `JSON` column. Using the `SeekableLZ4` (or `SeekableStringRunLength`) encoding may reduce memory usage but will hurt performance if you are extracting keys and values from the JSON. ## Check which Encoding is Used The `information_schema.COLUMNAR_SEGMENTS` view stores information about each segment of a columnstore table. Query the `encoding` column of this view to find out which encoding is used. The following query returns the encoding and sizes of the segments for column ``; in table ``. ```sql SELECT column_name, segment_id, encoding, uncompressed_size, compressed_size FROM information_schema.COLUMNAR_SEGMENTS WHERE table_name = ‘’ and column_name = ''; ``` ## SeekableJson In the `SeekableJson` encoding, a schema is inferred from JSON keys and the data is split into columns by keypath (columnarized) and stored in an encoded Parquet-like format. The JSON data is stored as if you had created a schema with separate columns for every field. As a result, queries on `JSON` columns read only the parts of a JSON object that are relevant to a query and therefore have excellent performance. The `SeekableJson` encoding format is inspired by Apache Parquet but does not produce a Parquet file. `SeekableJson` is a seekable encoding. That is, during queries, the engine does not search entire segments, instead only a subset of rows in each segment is searched. This practice, called sub-segment seeking or row-level decoding, significantly improves performance. When using the `SeekableJson` encoding for JSON columns in columnstore tables, the data is stored as follows: * Top-level JSON properties are stored columnwise and use the same encodings as other data types (i.e., numbers, strings, and dates). * Nested properties (inside sub-documents) are stored in sequential column format, using SingleStore's standard encoding. * Arrays are pivoted and stored in column format. For example, a numeric array that is a property of an enclosing JSON document would be stored as a logical column, with multiple values in the column belonging to the same row. The array length information is kept to enable SingleStore to identify what values belong to what row. The engine variable `use_seekable_json` controls if the default encoding for `JSON` columns is `SeekableJson` or `JSONParquet`. When `use_seekable_json` is set to `ON`, the default encoding for `JSON` columns is `SeekableJson`. Check if `use_seekable_json` is enabled: ```sql SELECT @@use_seekable_json; ``` ```output +---------------------+ | @@use_seekable_json | +---------------------+ | 1 | +---------------------+ ``` ## Limit Number of Inferred Keypaths The engine variable `json_document_max_leaves` limits the number of JSON key paths inferred within a segment and can be used to tune schema inference. Refer to [Columnstore JSON Internals](https://docs.singlestore.com/#section-idm413515222379322.md) for more details. ## SeekableLZ4 and SeekableStringRunLength `JSON` columns may also be encoded with the seekable string encodings `SeekableLZ4` and `SeekableStringRunLength`. These encodings store the JSON as a string. Search or extraction on columns stored with these encoding decodes and parses the entire JSON value. Using these encodings may reduce memory usage and improve ingestion time, but they may significantly reduce performance for search and extraction on the JSON column, for example, when extracting keys and values from JSON data. SingleStore recommends using `SeekableLZ4` and `SeekableStringRunLength` only for columns on which JSON values are extracted as a whole. [Encoding String Data in Columnstore Tables](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/encoding-string-data-in-columnstore-tables.md) provides a general description of encoding schemes and encoding scheme selection for string (including JSON) columns. String encoding schemes are per-segment and are automatically selected with the goal of maximizing storage savings and lowering query latency. To specify a string encoding for a `JSON` column, use a collation specification as follows: ```sql CREATE TABLE json_table (json_lz4_col JSON COLLATE utf8_bin OPTION 'SeekableLZ4'); CREATE TABLE json_table (json_rle_col JSON COLLATE utf8_bin OPTION 'SeekableStringRunLength'); ``` ## JSON Parquet Like `SeekableJson`, the `JSONParquet` format is columnarized and inspired by the Apache Parquet format but does not produce a Parquet file. SingleStore recommends that you use `SeekableJson` instead of `JSONParquet`, as `SeekableJson` is more performant. `JSONParquet` is a precursor to `SeekableJson` and is not a seekable format, meaning that entire segments must be extracted and decoded for them to be processed. ## Apply SeekableJson Encoding to Existing Data To change the encoding of a `JSON` column from `ParquetJSON` to `SeekableJson` encoding, use the following options. ## Online Approach To keep the table online when `use_seekable_json` is not set to `ON`, you can upgrade existing `JSON` columns to use `SeekableJson` by setting `use_seekable_json = ON`, and then running `OPTIMIZE TABLE FULL`. This approach provides a best effort upgrade, when new segments are written, the new segments have the `SeekableJson` encoding. Segments that do not need to be re-written continue to use `JSONParquet` encoding. ## Offline Approach If there is not a consistent improvement in seek performance with the Online Approach, query the `information_schema.COLUMNAR_SEGMENTS` view using the following query to see if the `JSONParquet` encoding is still used for many segments. ```sql SELECT column_name, segment_id, encoding, uncompressed_size, compressed_size FROM information_schema.COLUMNAR_SEGMENTS WHERE table_name = '' and column_name = ''; ``` If many segments are still using `JSONParquet` and it is acceptable to take the table offline; use the following procedure to rewrite all the segments in the new seekable format. 1. Enable or check if the `use_seekable_json` engine variable is enabled for your database: `SET GLOBAL use_seekable_json = ON;` 2. Create a new table to copy the existing data to: `CREATE TABLE LIKE ;` 3. Insert the existing data from the old table to the new table: `INSERT INTO SELECT * FROM ;` 4. Run the optimize table statement with the full argument: `OPTIMIZE TABLE FULL;` 5. Rename the old table with a temporary name: `ALTER TABLE RENAME TO ;` 6. Rename the new table to the original (i.e., old table name): `ALTER TABLE TO `; 7. Drop the temporary table: `DROP TABLE ;` If the table needs to remain online but you need to ensure every segment is reorganized, update every row in the table gradually. When the merger is run after these updates, the merger reorganizes the segments and applies the seekable encodings. To avoid modifying data in the table, add a new dummy column to the table and update it. The updates must be true updates; setting a value to itself is considered an update. ## Columnstore JSON Internals This section provides details on the `SeekableJson` encoding. A Parquet schema has a JSON-like syntax and a nested structure. The schema consists of a group of fields called a message. Each field has three attributes: repetition, type, and name. The type attribute is either a group or a primitive (e.g., int, float, boolean, or string). The repetition attribute can only be one of the following: * Required: exactly one occurrence * Optional: 0 or 1 occurrence * Repeated: 0 or more occurrences The infer schema process works as follows: 1. The schema loops through the list of JSON objects. 1. The present keypaths are merged into a schema tree object. 2. If there is a type conflict, the node in the schema tree is marked as un-inferable. 3. If any node in the schema tree contains more children than indicated in the `json_document_max_children` engine variable, the node is marked as un-inferable. 4. If a node has a greater number of children than indicated in the engine variable `json_document_sparse_children_check_threshold` and the average occurrence of all children related to the parent is less than 1/ as indicated in the engine variable `json_document_sparse_children_check_ratio`, the node will be marked as un-inferable. 2. Once the schema tree object is constructed, analyze the schema tree and prune the tree until the number of key paths (distinct root to leaf paths) is less than the setting for the engine variable `json_document_max_leaves`. Also, prune any node where the average occurrence of all the children in relation to the number of rows is less than 1/ as indicated in the engine variable `json_document_absolute_sparse_key_check_ratio`. The examples will use the engine variables settings and the JSON object table shown below: * `json_document_max_children = 4` * `json_document_sparse_children_threshold = 3` * `json_document_sparse_children_ratio = 2` | JSON Objects | | ------------------------------------------------------------ | | {“a1”: 1, “a2”: {“d1”: 1}, “a3”: {“c1”: 1}, “a4”: {“b1”: 1}} | | {“a1”: 2, “a2”: 1, “a4”: {“b2”: 1}} | | {“a1”: 3, “a2”: 1, “a4”: {“b3”: 1}} | | {“a1”: 4, “a2”: 1, ”a3”: {“c2”: 1}, “a4”: {“b4”: 1}} | | {“a1”: 5,, “a3”: {“c3”: 1} “a4”: {“b5”: 1}} | The first pass at merging the keypaths will yield: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte8aa18724a22bced/6a7adb35366687fecc27e394/diagram_first-pass-W4ET0T.png) The second pass at merging the keypaths contains a type mismatch on a2. The results would be: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7ae24845148dc71f/6a7adaade88c6532f7fffddf/diagram_second-pass-oz6O26.png) The results of the third and fourth passes at merging yield: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7eb1655cd208468d/6a7adb7896b5a65eb886fdde/diagram_last-pass-wIyQM3.png) In the final merge, the number of children for a4 exceeds the limit (4) set for `json_document_max_children` so it is marked as un-inferable. The number of children for `a3` exceeds the limit (3) set for `json_document_sparse_children_threshold`. The average occurrences of children of a3 (1) relative to the number of occurrences of a3 (3) is calculated as **⅓ < ½**, so a3 would also be un-inferable. After the merging process, the schema tree with leaves are the inferred columns (also may be referred to as leaves, leaf columns, leaf paths, or key paths). The keypaths will be a1, a2, a3, a4. ## Encoding Nested Columns Parquet uses the `Dremel encoding` for nested columns with definition and repetition levels. * Definition levels specify how many optional fields in the path for the column are defined. * Repetition levels specify the repeated value of the field in the path. * SingleStore stores the leaf columns for the JSON schema along with their respective repetition and definition levels. Each of these internal columns will correspond to the value, definition level and repetition level columns which are encoded in SingleStore encodings (SeekableString, etc). The example tables will have these values for the internal columns: | a1 | | | | ----- | ---------------- | ---------------- | | Value | Definition Level | Repetition Level | | 1 | 1 | 0 | | 1 | 1 | 0 | | 1 | 1 | 0 | | 1 | 1 | 0 | | 1 | 1 | 0 | | a2 | | | | ----------- | ---------------- | ---------------- | | Value | Definition Level | Repetition Level | | ‘{“d1”: 1}’ | 1 | 0 | | 1 | 1 | 0 | | 1 | 1 | 0 | | 1 | 1 | 0 | | - | 0 | 0 | | a3 | | | | ----------- | ---------------- | ---------------- | | Value | Definition Level | Repetition Level | | ‘{“c1”: 1}’ | 1 | 0 | | - | 0 | 0 | | - | 0 | 0 | | ‘{“c2”: 1}’ | 1 | 0 | | ‘{“c3”: 1}’ | 1 | 0 | | a4 | | | | ----------- | ---------------- | ---------------- | | Value | Definition Level | Repetition Level | | ‘{“b1”: 1}’ | 1 | 0 | | ‘{“b2”: 1}’ | 1 | 0 | | ‘{“b3”: 1}’ | 1 | 0 | | ‘{“b4”: 1}’ | 1 | 0 | | ‘{“b5”: 1}’ | 1 | 0 | ## Related Topics * [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table/#UUID-95582480-6981-8db3-0ad8-c559a8cfe64e.md) * [Using JSON](https://docs.singlestore.com/cloud/create-a-database/using-json/#section-idm4634652054806433297983139685.md) *** Modified at: July 10, 2026 Source: [/cloud/create-a-database/columnstore/columnstore-seekability-using-json/](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-seekability-using-json/) (An index of the documentation is available at /llms.txt) --- # Columnstore Sizing Estimations There are two possible methods for estimating the total memory used by a columnstore table. The first method is brief, while the second is more precise. For more information about how columnstore works, see [Columnstore](https://docs.singlestore.com/cloud/create-a-database/columnstore.md), and the other topics nested under it. ## Simple Method For the simple method, start with calculating the average row size for your table by adding together the size of each column's data type from [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md).  Multiply the average row size by the expected number of rows in your table to get the estimated uncompressed size. Next, estimate the compression ratio for your dataset. Columnstore compression typically shrinks the data size by 3-7x. Based on these estimations, it is possible to get a very rough estimate of storage requirements for columnstore tables. ## Complex Method In the same way as the simple method, start with calculating the average row size for your table by adding together the size of each column's data type from [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md).  Multiply the average row size by the expected number of rows in your table to get the estimated uncompressed size. Then, to get an estimate that is more accurate than the base calculation demonstrated above, a more precise compression ratio is needed. Use the steps below to accomplish this by loading a large amount of representative data, then measuring the actual compression ratios achieved (by table, and by column). 1. Load a representative sample of data into a columnstore table. More data leads to more accurate measurements. 1. For information about creating a columnstore table, see [Creating a Columnstore Table](https://docs.singlestore.com/cloud/create-a-database/creating-a-columnstore-table.md). 2. To see options and procedures for loading data, see [Load Data](https://docs.singlestore.com/cloud/load-data.md) and the topics nested below it. 2. Run [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) on the table in order to ensure the data is optimized and on disk. Use caution when running this command, as it can impact the performance of all other workloads running in the cluster. For more information, see the linked topic. 3. Measure the compression ratios using SQL queries as outlined below. 1. By querying [COLUMNAR\_SEGMENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/columnar-segments.md). 1. Compression ratio per table (compressed / uncompressed): ```sql SELECT database_name, table_name, (1-SUM(compressed_size)/SUM(uncompressed_size)) AS ratio FROM information_schema.columnar_segments GROUP BY database_name, table_name ORDER BY ratio DESC; ``` 2. Compression ratio per column (compressed / uncompressed): ```sql SELECT database_name, table_name, column_name, (1-SUM(compressed_size)/SUM(uncompressed_size)) AS ratio FROM information_schema.columnar_segments GROUP BY database_name, table_name, column_name ORDER BY ratio DESC; ``` ## Additional Methods to Obtain Columnstore Table Statistics The following query can be used to obtain the total disk space used by columnstore secondary indexes: ```sql SELECT database_name, table_name, SUM(size) AS total_index_size FROM information_schema.MV_COLUMNSTORE_FILES WHERE file_type = "INVERTED_INDEX" OR file_type = "CROSS_SEGMENT_HASH_INDEX" GROUP BY 1,2; ``` For more information about the column descriptions, see [MV\_COLUMNSTORE\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-files.md). ## Additional Sizing Estimations for Columnstore Table Overhead Beyond the variable size estimates explained above, columnstore tables also require space for a constant metadata overhead of: * \~100 bytes of memory per segment per column. * \~100 KB for each segment that has at least one deleted row. * 7KB per column per partition for incremental cardinality statistics. *** Modified at: February 7, 2025 Source: [/cloud/create-a-database/columnstore/columnstore-sizing-estimations/](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-sizing-estimations/) (An index of the documentation is available at /llms.txt) --- # Encoded Data in Columnstores SingleStore Helios performs some query processing operations directly on encoded data in columnstore tables. Columnstore data is stored encoded in a number of different forms, including dictionary encoding, run-length encoding, and value encoding. For these encodings, certain operations, including filtering and aggregation, can be run very efficiently, through the use of special coding techniques, as well as Single-Instruction, Multiple-Data (SIMD) instructions on processors that support the Intel AVX2 instruction set. SIMD support on your hardware is not required to benefit from operations on encoded data. Performance may speed up for operations on encoded data anywhere from a few times to 30 times without SIMD support. Use of SIMD may give an additional increase in performance. A gain of another 2 to 3 times due to SIMD is not unusual. Your results will depend on your data and queries. Portions of larger queries may be done without operating on encoded data, so the end-to-end speedup you experience may vary. Operations on encoded data are performed automatically, by default. You don’t need to change any settings to benefit from it. The decision to use operations on encoded data is made by the query execution system at runtime, not by the query optimizer. ## What Is Encoded Data and What Does It Mean to Operate on It? SingleStore Helios supports several different kinds of data encodings to help compress data. All columnstore data is said to be encoded in one way or another. Common encodings are run-length, dictionary, and LZ4. Only certain encodings can be processed directly, i.e. “operated on” directly. These include dictionary encoding, run-length encoding, and integer value encoding. For example, with dictionary encoding, for a segment of a string column with only 3 distinct values, a 2-bit ID number is stored for each string. The ID numbers are used as references into the dictionary. These IDs are packed tightly together when stored in the columnstore. For example, consider this dictionary: | ID | value | | -- | ----- | | 0 | red | | 1 | green | | 2 | blue | This can be represented as a bit vector of packed string IDs when it is stored into the column segment: * Strings: “red”, “blue”, “green”, “green”, “red” * String IDs: 0, 2, 1, 1, 0 * Bit vector: 00 10 01 01 00 (2 bits per string ID) As an example of operating directly on encoded data, SingleStore Helios can perform a filter operation, say “t.a = “xyz””, on a string dictionary-encoded column segment by first finding the result of the filter for every entry in the dictionary. Then, while scanning the segment, the query execution system simply takes the encoded ID number for each value in t.a and uses it to look up the result of the string comparison for that ID that was computed in the initial scan of the dictionary. That tends to be much quicker than doing an actual string comparison. Other kinds of operations can also be done directly on encoded data ID values, including the comparison of ID values needed to do group-by operations. The details of these are beyond the scope of this documentation. The encoding for a blob depends on its type and data. It may hinge both on the ordering and cardinality for numeric delta encoding and dictionary encoding, respectively. It impacts the execution speed, as well as enables SIMD operations on encoded data. Most query processing in SingleStore Helios outside of columnstore scan is done row-at-a-time. Columnstore processing on encoded data is done in a vectorized fashion, where large batches of data from a column are processed in one or more relatively simple loops. These loops tend to be more friendly to modern CPUs than row-at-a-time processing, resulting in lower instruction count, better cache usage, and improved efficiency of the processor’s instruction pipeline. ## Queries That Can Benefit from Operations on Encoded Data Queries that can benefit from operations on encoded data are those that run over columnstore tables, and have one or more of these components: * Filters * Group-by * Aggregates * Expressions in aggregates * Expressions in group-by * Star joins on integer columns Operations on encoded data can be done regardless of the number of distinct values of the encoded columns being processed. Performance tends to be better when the number of distinct values is smaller. This is because when dictionaries are smaller the columnstore data is compressed to a smaller size, and intermediate working lookup tables fit more easily into the cache memory of the processor. Here’s an example of a query that can benefit from operations on encoded data in several ways. ```sql SELECT DAYOFWEEK(shipdate), location, SUM(price * (1 - discount)), AVG(quantity) FROM lineitem WHERE status = 'S' GROUP BY 1, 2; ``` Here’s another example of a query that can benefit from join operations on encoded data, if the table `f` is a columnstore. ```sql SELECT d1.x, d2.y, SUM(f.m) FROM f, d1, d2 WHERE f.d1_key = d1.key AND f.d2_key = d2.key AND d1.v > 100 AND d2.v in (1, 2, 3) GROUP BY d1.x, d2.y ``` This query is a simple example of a *star join*. If you are using a *star schema* (with a fact table linked to multiple dimension tables), and the joins are on integer columns, you can benefit from operations on encoded data. Use of a `group by` clause and aggregate functions is required to benefit from join operations on encoded data. ## Examples The following example shows encoded group-by in action: ```sql CREATE DATABASE db; USE db; DROP TABLE IF EXISTS r; DROP TABLE IF EXISTS is_tables; /* Extract rows for 50 different tables into a scratch table, is_tables. */ CREATE TABLE is_tables AS SELECT * FROM information_schema.tables LIMIT 50; /* Create a table with a million rows, with every table name appearing in most or all segments. */ CREATE TABLE r (INDEX USING CLUSTERED COLUMNSTORE(n)) AS SELECT s.*, (row_number () OVER (ORDER BY s.table_name) % 1000) AS n FROM (SELECT t1.* FROM is_tables t1, is_tables t2, is_tables t3, is_tables t4 LIMIT 1000000) AS s; ``` The table r created above is a columnstore with one million rows. You can see the encoding of the TABLE\_NAME column using this query: ```sql SELECT COUNT(*), encoding, column_name FROM information_schema.columnar_segments WHERE table_name = 'r' AND database_name = 'db' AND column_name = 'TABLE_NAME' GROUP BY column_name, encoding; ``` The result indicates that StringDictionary encoding is used for the column. Now, run the following group-by/aggregate query that groups on the table\_name column, in profile mode: ```sql PROFILE SELECT table_name, COUNT(*) FROM r GROUP BY table_name; ``` Finally, output the JSON profile for the query: ```sql SHOW PROFILE json; ``` The result includes the following text: ```json inputs":[ { "executor":"ColumnStoreScan", "db":"db", "table":"r", ... "segments_scanned":{ "value":16, "avg":2.000000, "stddev":0.000000, "max":2, "maxPartition":0 }, "segments_skipped":{ "value":0 }, "segments_fully_contained":{ "value":0 }, "segments_encoded_group_by":{ "value":16, "avg":2.000000, "stddev":0.000000, "max":2, "maxPartition":0 }, "inputs":[] } ``` Notice the portion labeled “segments\_encoded\_group\_by”. This part shows that 16 segments were processed in total and all of them were processed with encoded group-by operations. These operations are taking place directly on encoded data. Now, here is another query (run in profile mode) that is similar to the first, but also has a filter on column table\_name: ```sql PROFILE SELECT table_name, COUNT(*) FROM r WHERE table_name LIKE '%COLUMN%' GROUP BY table_name; SHOW PROFILE json; ``` The JSON profile for this query contains the text below: ```json "inputs":[ { "executor":"ColumnStoreScan", "db":"db", "table":"r", ... "segments_scanned":{ "value":16, "avg":2.000000, "stddev":0.000000, "max":2, "maxPartition":0 }, "segments_skipped":{ "value":0 }, "segments_fully_contained":{ "value":0 }, "segments_ops_compressed_data":{ "value":16, "avg":2.000000, "stddev":0.000000, "max":2, "maxPartition":0 }, "segments_encoded_group_by":{ "value":16, "avg":2.000000, "stddev":0.000000, "max":2, "maxPartition":0 }, "inputs":[] } ``` Notice that it includes sections for both “segments\_ops\_compressed\_data” and “segments\_encoded\_group\_by”. The “segments\_ops\_compressed\_data” section indicates that 16 segments had filters applied on encoded data. The final decision about use of operations on encoded data is made at runtime and depends on column encodings for each segment and (for group-by) the number of distinct values in the segment. So the segments\_ops\_compressed\_data and segments\_encoded\_group\_by values may be less than the number of segments scanned minus the number skipped. In this case, the “encoded\_group\_by\_bailout” section will indicate the reasons that encoded group by was disabled as well as the number of segments disabled for each reason. ## Encoded joins This example shows encoded joins in action. ```sql /* Create a simulated dimension table with 1000 rows with unique values for n, and around 100 categories. */ CREATE TABLE d AS SELECT n, MAX(CONCAT('cat', n % 100)) AS category FROM r GROUP BY n; /* Join dimension table d to "fact" table r on an integer column and group by category. This is a very simple star join. */ SELECT d.category, COUNT(*) FROM r, d WHERE r.n = d.n AND d.category LIKE 'cat1%' GROUP BY d.category; ``` The above join query takes only 0.02 seconds on a two-core laptop. Yet it is fully scanning the million-row table r and joining it with 110 rows from d. You can see that encoded joins are being used in the profile query execution plan for the above statement by using the graphical plan display or `show profile json` output. On the HashJoin operator you will see the property `"encoded_join_enabled":"yes"`. In addition, you can see that operations on encoded data are pushed to the ColumnStoreScan operator since it has the property `segments_encoded_group_by` with `value` set to 8. The HashJoin, ColumnStoreScan, and various GroupBy operators can work together for a star join with a sequence of one or more hash joins over the scan. ## Example Performance Results Operations on encoded data can yield some astonishingly short query execution times. Below are example queries and runtimes, on tables with 25 to 50 million rows of data, running on a single core. | query | time | | -------------------------------------------------------------------------- | ------ | | `SELECT COUNT(*) FROM y WHERE b IN (thomas1,thomas3,thomas5)` | 0.424s | | `SELECT COUNT(*) FROM col320 GROUP BY b {320 groups}` | 0.154s | | `SELECT SUM(j) FROM col10 GROUP BY b {10 groups}` | 0.053s | | `SELECT COUNT(*) FROM col10 WHERE c < 900000 GROUP BY b {90% selectivity}` | 0.114s | | `SELECT COUNT(*) FROM col10 WHERE c < 100000 GROUP BY b {10% selectivity}` | 0.097s | Best case performance in other internal tests have shown a processing rate of over one billion rows per second per core for queries like the following, when column `a` has eight or fewer distinct values, using a recent model high-performance Intel processor. ```sql SELECT a, COUNT(*) FROM t GROUP BY a; ``` These results can be extrapolated to much larger data sets on a large SingleStore Helios cluster, although memory bandwidth limits may limit total throughput in rows per second per core across the cluster, below the best-case numbers shown here. Your results will of course depend on your data, hardware, and queries. ## Performance Considerations Because operations on encoded data can process data so fast, their performance can become limited by the bandwidth of main memory (RAM). And the bandwidth of RAM is normally far higher than the bandwidth of your I/O system. For example, RAM bandwidth could be 50 GB/sec and the bandwidth of an SSD could be only 600 MB/sec. So, SingleStore recommends that you configure your system so that little or no I/O is being done while processing columnstore data, under a steady query workload. You can do this by ensuring that the operating system’s file system buffer cache has enough main memory to hold the working set of your columnstore data, i.e. the segments of columns that are frequently accessed. A good rule of thumb is to have enough RAM in the file system buffer cache to hold at least 20% of your compressed columnstore data. If your workload does a lot of full table scans that touch all the columns of the table, you may want to consider adding more than that. The Linux operating system will allocate available system memory to the file system buffer cache in an on-demand way, without the need to set any configuration knobs. Simply ensure that there is sufficient memory remaining after all the memory dedicated to SingleStore Helios, the operating system, and any other applications running on the hardware. ## Relevant Hints SingleStore Helios can perform group-by on encoded data using a HashGroupBy operation, and can also do a group-by on columnstore data using a StreamingGroupBy operation on the sort key of the columnstore. Due to the improvement of HashGroupBy with the introduction of operations on encoded data, it may be the case that HashGroupBy is faster than StreamingGroupBy, yet the query optimizer may choose StreamingGroupBy. If this occurs and it is important for your workload to tune the query to get better performance, you can use the **disable\_ordered\_scan** query hint. For example: ```sql SELECT SUM(a) FROM t WITH (disable_ordered_scan=true) GROUP BY b; ``` You can use EXPLAIN to see the plan for your query to check the type of group-by operation chosen. ## Data Encodings Supported Operations on encoded data are supported only for the following: * String columns (including varchar, char, text, and blob) with string dictionary and string run-length encodings * Numeric columns (including all integer types (tinyint, bigint, etc.) plus float, decimal, and double) with value and run-length encodings Group-by/aggregate operations on encoded data are supported only for integers. Filter acceleration on encoded data is supported for both strings and integers. SingleStore recommends that you allow SingleStore Helios to choose automatically how to encode data for columnstores. However, in the rare event that it chooses an encoding that does not support operations on encoded data, and it is important to you for your application that operations on encoded data be performed, you may wish to force the encoding chosen. You can do this with the `option ''` notation. For example: ```sql CREATE TABLE t (a int, b varchar(50) option 'StringDictionary', key(a) USING CLUSTERED COLUMNSTORE); ``` ## Operations Supported, and Limitations To summarize the operations supported on encoded data, as well as limitations to operations on encoded data, they include: * Scan * Faster decoding for integer encoding * Filter * Filters on the following: * Strings for string dictionary and string run-length encoding * Integers for run-length encoding * Or-of-ands of filter expressions; filter expressions must involve a single string column for faster processing * Support for Bloom filters on string columns (elimination of rows with no matches for joins on a single string column) * Aggregates * Aggregates supported: sum, min, max, count, any * Aggregate input data types supported: all numeric * Aggregate expressions supported: multiple table columns allowed in a single expression * Aggregates encodings supported: integer, integer run-length * Group-by: * Operations on encoded data are not done for aggregates without a group-by (a.k.a. scalar aggregates) * Group-by count star is very fast * Group-by: * Multiple group-by columns are allowed * A mix of columns and expressions is allowed * Group-by columns must only use these encodings: integer, integer run-length * There is a limit on number of distinct values per column of a few thousand rows, beyond which, the system reverts to row-at-a-time processing and the local aggregation of data for a row segment will output rows to the parent global aggregate operator * Similarly, there is a limit on total number of groups, of a few thousand groups, beyond which the optimized group-by processing is not used * Group-by on the sort key may not be as efficient as group-by on other columns because ordered group-by may be performed while hash groupby might be better * Group-by on expressions: * Only one input column is allowed in a single expression * The expression result must be integer * Each table column can be used only once in the set of group-by columns and expressions * Joins: * Joins must be on some type of integer column, or another column type represented internally as an integer, such as datetime * The join or joins selected by the query optimizer must be of type HashJoin * The join must be a many-to-one relationship * The HashJoin (or a sequence of HashJoins) must appear over a ColumnStoreScan * A Group By operation and aggregate must be present in the query over the result of the join ## Encoded group by bailout reasons The final decision on whether to use encoded group by is made at runtime. The possible bailout reasons are: * Join condition is not a many-to-one relationship * Unsupported column encoding * Aggregate not guaranteed to fit in aggregate type (the result may overflow the internal integer or decimal type) * Reached dynamic dictionary size limit (too many unique group by values) * Group by column cannot be placeholder blob (column does not yet exist in columnstore) ## Additional Tuning Considerations If you have upgraded from a MemSQL release below 6.0, you can benefit immediately from operations on encoded data. There is no need to rebuild your columnstore tables. However, columnstore encoding choices have changed slightly in the 6.0 release, with dictionary encoding being more preferred. So queries may run faster on newly loaded data. Encodings are chosen separately for each segment, so even if you don’t use OPTIMIZE on your columnstore tables, as new data is inserted and older data is removed, the encodings chosen will naturally evolve to be those preferred by the latest version of SingleStore Helios. If you find that operations on encoded data are not occurring as much as you expect them to you can rebuild the columnstore table involved. Before doing this, you can query information\_schema.columnar\_segments as described earlier to see if dictionary, run-length, or integer encodings are not used. If not, then consider rebuilding the columnstore table. One way to rebuild it is to use ```sql CREATE TABLE newTable (INDEX USING CLUSTERED COLUMNSTORE()) AS SELECT * FROM oldTable; ``` Then drop oldTable and rename newTable to oldTable. Alternatively you can use OPTIMIZE TABLE… FULL. However, be aware that this can take significantly longer than loading the data in the first place, because it causes a single sorted run to be created. ## Scalar count(\*) Because encoded group-by is so fast, and scalar aggregates are not done on encoded data, you may find the following unexpected behavior. A query like this: ```sql SELECT a, COUNT(*) FROM t GROUP BY a; ``` may be faster than this: ```sql SELECT COUNT(*) FROM t; ``` Although it is usually not necessary because count(\*) scalar aggregates are so fast anyway, if you want the fastest possible count(\*), consider this workaround: ```sql SELECT SUM(t2.c) FROM (SELECT a, COUNT(*) AS c FROM t GROUP BY a) AS t2; ``` The subquery will be done with encoded group-by, so the containing query may run faster that a scalar count(\*); ## Aggregation of Decimal Types When creating decimal type columns that are aggregated in queries, for best performance, SingleStore recommends to use precision of 18 digits or less if your application can accommodate this (i.e. the extra precision is not needed to represent meaningful information). Decimal values of 18 digits or less can be processed more efficiently than others because they can be handled internally as 64-bit integers during aggregation. Processing these 64-bit values takes much less time than interpreting decimal values of more than 18 digits, which must be handled in a more general way. *** Modified at: March 2, 2026 Source: [/cloud/create-a-database/columnstore/encoded-data-in-columnstores/](https://docs.singlestore.com/cloud/create-a-database/columnstore/encoded-data-in-columnstores/) (An index of the documentation is available at /llms.txt) --- # Highly Selective Joins ## Working with Highly Selective Joins Starting in version 7.0, SingleStore introduced support for columnstore hash indexes to broaden the support for OLTP-type queries. However, a common join pattern in OLTP is to have a very selective filter on one table, which produces a few rows from the source table, and then join those rows with another table. Databases for OLTP normally use a nested loop join for this. For each row from the outer table, an index seek is done on the inner table. SingleStore Helios supports these highly selective joins using an adaptive hash join algorithm. As a result, there’s no need to run a full columnstore scan when there’s a matching hash index. First, it builds a hash table for the table with the highly-selective filter. Then, depending on the number of rows in the hash table, the adaptive algorithm will switch strategies internally. * If there are only a few rows in the hash table, it switches to use a nested loop join strategy, seeking into the larger table (on the probe side) via the index on the join column of the table on the probe side. * If the hash build side produces a lot of rows, then it performs a normal hash join. The following example takes advantage of this strategy for selective joins. ```sql CREATE TABLE orders( oid INT, d DATETIME, SORT KEY(d), SHARD(oid), KEY(oid) USING hash); CREATE TABLE lineitems( id INT, oid INT, item INT, SORT KEY(oid), SHARD(oid) KEY(oid) USING HASH); ``` Now, add some sample data to `orders`: ```sql INSERT INTO orders VALUES(1, NOW()); ``` Execute the following command repeatedly, until the table has around 33.5 million rows in it. ```sql INSERT INTO orders SELECT oid+(SELECT MAX(oid) FROM orders), NOW() FROM orders; ``` Add 67.1 million rows of data to the `lineitems` table, such that each line item belongs to an order, and each order has exactly two line items. ```sql INSERT INTO lineitems SELECT oid, oid, 1 FROM orders; INSERT INTO lineitems SELECT oid + 1000*1000*1000, oid, 2 FROM orders; ``` Find a selective `DATETIME` value for `d` to search on: ```sql SELECT d, COUNT(*) FROM orders GROUP BY d; ``` The result shows that a number of `DATETIME` values only appear in one row in `orders`. For this example, let’s say that one such value is `2020-03-30 16:47:05`. The following query uses this date to produce a join result with exactly two rows: ```sql SELECT * FROM orders o JOIN lineitems l ON o.oid = l.oid WHERE o.d = "2020-03-30 16:47:05"; ``` This query filters rows on `o.d` to find a single row of orders, then joins to `lineitems` via the hash index on `lineitems.oid`, using the new selective join algorithm. In the JSON profile plan, you can see if a plan may be able to perform a join via a hash index. If this optimization strategy is available in your plan, you will see `join index` in descriptions of columnstore filter operators. For example: ``` "executor":"ColumnStoreFilter", "keyId":4294968023, "condition":[ "o.oid = l.oid bloom AND l.oid = o.oid join index" ], ``` You may see a condition that mentions `join index`. ``` o.oid = l.oid bloom AND l.oid = o.oid join index ``` The `join index` filter can also be viewed in the `EXPLAIN` output. ## Performance of Highly Selective Joins A higher number of join columns having a matching columnstore hash index and fewer number of hash values result in better join performance. For example, if `inventory` and `product` are two columnstore tables with hash indexes on the columns `id` and `code`, then the following query has better join performance than query with single join column: ```sql SELECT * FROM inventory JOIN product ON inventory.id = product.id AND inventory.code = product.code; ``` ## Related Content See [SingleStore Universal Storage – And Then There Was One](https://www.singlestore.com/blog/memsql-singlestore-then-there-was-one/) for additional information about (1) using columnstores with hash indexes, sub-segment access, and fine-grain locking to enable OLTP operations on data bigger than will fit in RAM, and (2) using SPARSE rowstore compression to reduce TCO for rowstore tables with many NULL values. *** Modified at: March 27, 2025 Source: [/cloud/create-a-database/columnstore/highly-selective-joins/](https://docs.singlestore.com/cloud/create-a-database/columnstore/highly-selective-joins/) (An index of the documentation is available at /llms.txt) --- # How the Columnstore Works In SingleStore Helios there are a few concepts that are important to understand in order to make the best possible use of the columnstore. * **Clustered columnstore index** - A clustered columnstore index represents the full table structure and is its primary storage. * **Sort key column(s)** - When creating a columnstore index, one or more columns need to be defined as the key column(s) for the columnstore index. The data in the columnstore is stored in key column order. Selecting a good columnstore key can significantly improve performance as covered later in this section. * **Row segment** - A row segment is a set of rows within a columnstore index that are stored together, each of which is made up of column segments. SingleStore Helios stores the metadata for each row, which includes the total row count for a given segment, as well as a bitmask tracking which rows have been deleted. * **Column group** - A [ column group](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#UUID-8f75a032-47ca-2ae7-1bb5-e398d5a05141.md) is an optional data structure that creates a materialized copy of a row as a separate index, called a column group index. This data structure can be used to speed up full-row retrievals and table updates. Column group indexes use less RAM than rowstore tables which can reduce operation costs. Using column group indexes on columnstores can allow you to get both fast lookups and fast analytics on the same table. The column group index improves the performance of lookups, and the standard columnar representation is available to give fast analytics. Using a column group index on a columnstore table is easier to manage than having to move data between rowstore and columnstore tables. * **Column segment** - Each row segment contains a column segment for every column in a table. The column segment is the unit of storage for a columnstore table and contains all values for a specific column within the row segment. Values in column segments are always stored in the same logical order across column segments within the same row segment. SingleStore stores in memory the metadata for each column segment, which includes the minimum and maximum values contained within the segment. This metadata is used at query execution time to determine whether a segment can possibly match a filter, a process known as `segment elimination`. * **Sorted row segment group** - A sorted row segment group represents a set of row segments that are sorted together on the columnstore key column(s). This means that within a sorted row segment group there will be no row segments with overlapping value ranges for the column(s) that make up the key for the columnstore index. New segment groups are formed when more segments are created after running `INSERT`, `LOAD`, or `UPDATE` queries on the table. Information on how this affects query performance and how to minimize the number of sorted row segment groups is covered later in this section. Below is an example of a SingleStore Helios columnstore index on the `Products` table using the `Price` column as its key. See [Choosing a Columnstore Key](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key.md) for an example of how to create the table. > **❗ Important**: Column segments typically contain on the order of hundreds of thousands of rows. In this example, the segment size is 5 rows for readability.Each row segment contains one column segment per column in the table. For simplicity, this example contains the initial load of the table and has a single sorted row segment group. **Logical table and rowstore representation** | ProductId | Color | Price | Qty | | --------- | ----- | ----- | --- | | 1 | Red | 10 | 2 | | 2 | Red | 20 | 2 | | 3 | Black | 20 | 2 | | 4 | White | 30 | 2 | | 5 | Red | 20 | 2 | | 6 | Black | 10 | 2 | | 7 | White | 25 | 2 | | 8 | Red | 30 | 2 | | 9 | Black | 50 | 2 | | 10 | White | 15 | 2 | | 11 | Red | 5 | 2 | | 12 | Red | 20 | 2 | | 13 | Black | 35 | 2 | | 14 | White | 30 | 2 | | 15 | Red | 4 | 2 | **Sorted row segment group #1 of 1** **Row segment #1 of 3** ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb79b720a0e904782/6a7ada7705b7b5905417ffe9/16083b57668734-6hJi6f.png) *“x N” indicates that the value is repeated N times.* **Row segment #2 of 3** ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4f9296aa9abfa359/6a7ada785967e50d945d1df1/16083b57670142-Xbbaup.png) *“x N” indicates that the value is repeated N times.* **Row segment #3 of 3**image ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt56c30f95a1f4440c/6a7ada6dfc63ab9962641807/16083b576782da-65fe2w.png) *“x N” indicates that the value is repeated N times.* ## Creating Efficient Columnstore Queries Queries against tables with a columnstore index in SingleStore can take advantage of five characteristics of a columnstore index: 1. All queries will be able to take advantage of the fact that only the column segments containing columns referenced in the query will need to be scanned, as well as the fact that the columnstore’s compression causes less data to need to be scanned. Taking the above table as an example. The query `SELECT SUM(Qty) FROM Products;` will only need to scan the three `Qty` column segments, each of which only contain a single value due to the compression. 2. Some queries can be performed simply by reading the in-memory metadata for column segments referenced in the query. As an example, the query `SELECT COUNT(*) FROM Products;` will only need to read the row count and delete bitmask for all row segments to produce its result, completely eliminating the need to read column segments from disk. Queries that use the `MIN` or `MAX` aggregates can eliminate the need to read column segments from disk if no delete has been performed against the column segment. 3. Some queries can reduce the number of segments that need to be read from disk by eliminating these segments based on the segment metadata (min and max values for the segment). The efficiency of this method depends on what percentage of segments can actually be eliminated. * For queries that filter on the key columns of the columnstore index, segment elimination is typically very efficient as segments within each row segment will not cover overlapping value ranges. For example, in the above table the query `SELECT AVG(Price), AVG(Qty) FROM Products WHERE Price BETWEEN 1 AND 10;` will eliminate all segments except `row segment: #1 column segment: Price 4-15` and `row segment: #1 column segment: Qty 2-2` which will be scanned. * For queries that filter on the non-key columns that don’t correlate with the key columns, segment elimination can be much less valuable as segments within each row segment can cover overlapping value ranges. For example, in the above table, the query `SELECT AVG(Price) FROM Products WHERE Color = 'Red';` will be forced to scan all segments for the `Price` and `Color` columns as no segment can be eliminated when the value `Red` is contained in all segments of the `Color` column. 4. Queries that join tables on columns that are the index columns of a columnstore index can be performed very efficiently through the use of a merge join algorithm allowing the join to be performed by simply scanning two segments that need to be joined in lockstep. 5. Certain types of data allow filters and group-by operations to be performed without decompressing data from its serialized-for-disk format. This greatly improves performance by reducing the amount of data that need to be processed, especially when the cardinalities of the involved columns are low. This optimization is only performed in cases when execution run time would be improved. See [Encoded Data in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/encoded-data-in-columnstores.md) for more information. 6. Queries with selective filters use subsegment access. These queries seek into column segments to read the needed values, rather than scanning whole segments. For a columnstore table `MyTable`, the query `SELECT Field1, Field2 FROM MyTable WHERE Field1 > 50` uses subsegment access. Selective filters with multiple conditions, such as `SELECT Field1, Field2 FROM MyTable WHERE Field1 > 50 AND Field2 > 100`, also use subsegment access. 7. Queries using equality filters can take advantage of hash indexes. See an [example](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md). The previous note applies to these queries, since equality filters are often selective filters. ## Create and Drop a Column Group The following examples show how to create, drop, and use a column group. Define an `emp` table with a column group named `cg_full`. ```sql CREATE TABLE emp ( id INT PRIMARY KEY, name VARCHAR(256), dept VARCHAR(16), salary FLOAT, manager INT, ... COLUMN GROUP cg_full (*) ); ``` The `*` indicates that the column group covers all the columns in the table. The `...` indicates there are more columns than shown. This table is intended to be a wide table. > **📝 Note**: Only column groups covering all columns in a table are supported. Add a column group to an existing table with an `ALTER TABLE` statement. ```sql ALTER TABLE emp ADD COLUMN GROUP cg_full (*); ``` Drop a column group with an `ALTER TABLE` statement: ```sql ALTER TABLE emp DROP COLUMN GROUP cg_full; ``` ## Use a Column Group Index The following query uses a column group index. ```sql SELECT * FROM emp WHERE id = 56231; ``` This query is a single-point lookup using the primary key `id`. To run this query, the engine locates the row using the primary key index. Since the result contains one row, and all columns are projected, the engine retrieves the entire row from the column group index using a single IO. The following query also uses a column group index. ```sql SELECT * FROM emp WHERE dept = 'IT'; ``` This query uses a filter to list all employees in the `IT` department. During query execution, the engine first uses the columnstore to perform the filter. The filter uses one column (`dept`), hence using the columnstore is the most efficient way to evaluate the filter. If `IT` is a small department, a small number of rows pass the filter. The engine then uses the column group index to read the full rows from the `emp` table. This execution plan is IO efficient. The following example query demonstrates use of the column group index in an update. ```sql UPDATE emp SET dept = 'IT' WHERE id = 56231; ``` This query updates the department name to `IT` for an employee with `id = 56231`. Although the query only affects the `dept` column, the engine still needs to read the entire row to implement row level locking (refer to [Locking in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/locking-in-columnstores.md)). In this example, the column group index is used to efficiently retrieve the row and load it into the in-memory segment of the `emp` table. Similarly, a column group index can improve delete query performance, since delete also requires reading and locking the entire row. ## Common Workflow A typical workflow using column group indexes is illustrated in the diagram below. 1. The user creates a columnstore table with a column group. 2. When the in-memory segment of the columnstore table is flushed to the disk, the engine creates a column group index blob (marked green in the diagram) in addition to creating the columnstore blobs themselves. 3. During query execution, columnstore hash indexes and columnstore blobs are used for efficient filtering. 4. The column group index blob is used to materialize the rows post-filtering for highly-selective queries that retrieve a small fraction of rows in the table. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltdfaab20aaeea4a2e/6a7adb3805b7b55e83180018/Columnstore_Segment_Diagram-sxhqMk.png) The query engine dynamically chooses different storage structures, combining hash indexes, columnstore blobs, and column group indexes to deliver optimal query execution. This efficiency is possible because the column group index and columnstore blobs live in the same columnstore segment — thus they can share the same row locators and indexes. ## Finding the Size of a Column Group Index The size (in bytes) of a column group index can be found in the [MV\_COLUMNSTORE\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-files.md) view as follows: ```sql CREATE TABLE colgrpsz_1 (a INT, b INT PRIMARY KEY, COLUMN GROUP(*)); ``` ```sql INSERT INTO colgrpsz_1 VALUES (1,2),(6,4),(36,8); ``` Run the `OPTIMIZE TABLE FULL` command on any table you want to know the column group index size on. This will create a more accurate reflection of the physical storage of the table's data. Please note, that this operation can be resource-intensive and time-consuming, particularly for large tables. Therefore, it should be scheduled during periods of low system usage to minimize the impact on system performance. ```sql OPTIMIZE TABLE colgrpsz_1 FULL; ``` Filter the view for the `COLUMN_GROUP` and only information about the column group index will be returned. If there are column group indexes on other tables, they will be displayed in the following query as well. ```sql SELECT * FROM information_schema.MV_COLUMNSTORE_FILES WHERE FILE_TYPE = 'COLUMN_GROUP; ``` ```output +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ | DATABASE_NAME | TABLE_NAME | FILE_TYPE | FILE | HOST | PORT | NODE_ID | PARTITION | SIZE | CREATION_TIME | LSN | TERM | CHECKSUM | +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ | test1 | colgrpsz_1 | COLUMN_GROUP | blobs/5006/0/0/173_51 | 127.0.0.1 | 3307 | 2 | 2 | 33 | 2024-05-06 11:43:05 | 51 | 173 | 1230141913 | | test1 | colgrpsz_2 | COLUMN_GROUP | blobs/5006/0/0/173_66 | 127.0.0.1 | 3307 | 2 | 2 | 33 | 2024-05-06 13:39:52 | 66 | 173 | 1230141913 | | test1 | colgrpsz_1 | COLUMN_GROUP | blobs/5002/0/0/174_52 | 127.0.0.1 | 3307 | 2 | 3 | 47 | 2024-05-06 11:43:05 | 52 | 174 | 2716425258 | | test1 | colgrpsz_2 | COLUMN_GROUP | blobs/5002/0/0/174_67 | 127.0.0.1 | 3307 | 2 | 3 | 47 | 2024-05-06 13:39:52 | 67 | 174 | 2716425258 | +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ ``` The `MV_COLUMNSTORE_FILES` view can be filtered to search for a specific table, the size summed to see the total column group index size per table, etc. ```sql SELECT * FROM information_schema.MV_COLUMNSTORE_FILES WHERE FILE_TYPE = 'COLUMN_GROUP' AND TABLE_NAME = 'colgrpsz_1'; ``` ```output +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ | DATABASE_NAME | TABLE_NAME | FILE_TYPE | FILE | HOST | PORT | NODE_ID | PARTITION | SIZE | CREATION_TIME | LSN | TERM | CHECKSUM | +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ | test1 | colgrpsz_1 | COLUMN_GROUP | blobs/5006/0/0/173_51 | 127.0.0.1 | 3307 | 2 | 2 | 33 | 2024-05-06 11:43:05 | 51 | 173 | 1230141913 | | test1 | colgrpsz_1 | COLUMN_GROUP | blobs/5002/0/0/174_52 | 127.0.0.1 | 3307 | 2 | 3 | 47 | 2024-05-06 11:43:05 | 52 | 174 | 2716425258 | +---------------+------------+--------------+-----------------------+-----------+------+---------+-----------+------+---------------------+-----+------+------------+ ``` ```sql SELECT DATABASE_NAME, TABLE_NAME, FILE_TYPE, SUM(SIZE) FROM information_schema.MV_COLUMNSTORE_FILES WHERE FILE_TYPE = 'COLUMN_GROUP' AND TABLE_NAME = 'colgrpsz_1' GROUP BY DATABASE_NAME, TABLE_NAME, FILE_TYPE; ``` ```output +---------------+------------+--------------+-----------+ | DATABASE_NAME | TABLE_NAME | FILE_TYPE | SUM(SIZE) | +---------------+------------+--------------+-----------+ | test1 | colgrpsz_1 | COLUMN_GROUP | 80 | +---------------+------------+--------------+-----------+ ``` *** Modified at: August 10, 2026 Source: [/cloud/create-a-database/columnstore/how-the-columnstore-works/](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works/) (An index of the documentation is available at /llms.txt) --- # Locking in Columnstores By default, `UPDATE` and `DELETE` queries lock columnstore tables at the row level. The following example demonstrates how multi-table filters may lock rows that do not match the filters. Consider the following query, ```sql UPDATE stock JOIN product ON stock.qty = 10 AND stock.id = product.id SET ... ``` This query locks all the rows of the table `stock` where `stock.qty = 10`, including the rows where `stock.id` is not equal to `product.id`. Alternatively, use a single table filter to trim the number of rows locked. Suppose that a database `errors_db` containing eight partitions has the `app_errors` table that is defined as follows. ```sql CREATE TABLE app_errors ( error_id INT, app_name TEXT, error_code TEXT, error_date DATE, SORT KEY (error_date), SHARD KEY (error_id) ); ``` The following `UPDATE` example demonstrates row-level locking. Assume the `app_errors` table on partition one contains 4500 records having an `app_name` of `App1`. ```sql UPDATE app_errors SET error_code = 'ERR-2000' WHERE app_name = 'App1'; ``` While this query is running: * In the `app_errors` table on partition one, other queries can `UPDATE` and `DELETE` the rows not having an `app_name` value of `App1`. * In the `app_errors` table on partition one, other queries may not `UPDATE` and `DELETE` the rows having an `app_name` value of `App1`. When a table is locked during a transaction, an error message similar to the following is generated: "Lock wait timeout exceeded; try restarting transaction. Table lock owned by connection id \, query \" ## Overriding Default Locking By default, `UPDATE` and `DELETE` queries on a columnstore table use row-level locking when they operate on fewer than `default_columnstore_table_lock_threshold` rows in a partition and use partition-level locking when they operate on `default_columnstore_table_lock_threshold` rows or more rows in a partition. In the latter case, all of the table’s rows in the partition are locked. The default for `default_columnstore_table_lock_threshold` is 0, meaning that a value of 5000 is used. For an `UPDATE` or a `DELETE` query, you can override the default threshold of `default_columnstore_table_lock_threshold` by specifying the `OPTION (columnstore_table_lock_threshold = )` hint. The `` indicates the row count threshold for which partition level locking takes effect. The following `UPDATE` example specifies that `columnstore_table_lock_threshold` is 4000. ```sql UPDATE app_errors SET error_code = 'ERR-2000' WHERE app_name = 'App1' OPTION (columnstore_table_lock_threshold = 4000);` ``` When you specify a higher `columnstore_table_lock_threshold` value, you can get higher concurrency, but more memory may be used for locking. In the following scenario, the `columnstore_table_lock_threshold` value is set to a higher value in order to avoid locking: If a database has 455000 rows and 48 partitions (for this example, we'll assume there is no Shard key and the data is evenly divided among the partitions), each partition would have about 9480 rows, which is higher than the default threshold of 5000. In this case, a set of `UPDATE` or `DELETE` queries that touch every partition would lock the entire table. Setting the threshold value to a value higher than 9480 (say, 12000) would cause the database to use row locking until the number of rows per partition hit 12000. *** Modified at: December 1, 2023 Source: [/cloud/create-a-database/columnstore/locking-in-columnstores/](https://docs.singlestore.com/cloud/create-a-database/columnstore/locking-in-columnstores/) (An index of the documentation is available at /llms.txt) --- # Managing Columnstore Segments A columnstore table will have the best performance if the rows in the table are in global sorted order across all the row segments. In reality, maintaining such an order is not feasible in the presence of continuous writes. ## Background Merger SingleStore uses an advanced algorithm that allows it to maintain the order of columnstore tables as close to sorted as possible, while data is being ingested or updated. This process is called a `background merger` and runs constantly in the background. The `background merger` runs in an optimistic manner. If at any point the background merger  tries to move data which is being changed by a concurrent [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) or [DELETE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/delete.md) query, it discards all the work it has done so far and starts over. The `background merger`  works on a small chunk of data at a time, so if a conflict with an `UPDATE` or `DELETE` occurs, a relatively small amount of work is discarded. However, in the presence of a very heavy update workload, the slowdown  can be significant compared to a pessimistic merger, which locks the row segments it is currently processing. A user can manually trigger a pessimistic merger by running the `OPTIMIZE TABLE` command. The following sections explain how to decide whether running an `OPTIMIZE TABLE` command is necessary, and how to run it. ## Sorted Row Segment Group SingleStore uses the concept of a `sorted row segment group` to describe a set of row segments that are sorted together on the columnstore key columns. Row segments form a sorted row segment group if and only if there is an order on the row segments such that for each row segment the smallest row in it is no smaller than the largest row in any row segment before it. Here and below when we say that one row is smaller than another row, we mean that the values of columns of the `SORT KEY` of that row are smaller than those of the other row. Within a sorted row segment group there are no row segments with overlapping value ranges for the sort key column(s). Refer to [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) for more information. A table may have multiple sorted row segment groups. The goal of the background merger is to reorganize the rows among row segments in such a way that the number of sorted row segment groups is as small as possible. > **📝 Note**: `KEY() USING CLUSTERED COLUMNSTORE` is a legacy syntax that is equivalent to `SORT KEY()`. SingleStore recommends using `SORT KEY()`. To inspect the current state of the sorted row segment groups of a table, named `sales`, run the `SHOW COLUMNAR MERGE STATUS FOR ` command: ```sql SHOW COLUMNAR MERGE STATUS FOR sales; ``` ```output +----------------------------------------------------------------------------+ | Merger | State | Plan | Progress | Partition | +----------------------------------------------------------------------------+ | (Current groups) | NULL | 741,16,1 | NULL | 0 | | (Current groups) | NULL | 782,20 | NULL | 1 | | (Current groups) | NULL | 701,40,5 | NULL | 2 | | (Current groups) | NULL | 326,207,123,37,21,19,17 | NULL | 3 | +----------------------------------------------------------------------------+ ``` Observe the first row of the result. According to that row, the slice of the table that is stored on partition 0 has three sorted row segment groups, one consists of 741 row segments, one consists of 16 row segments, and one consists of a single row segment, for a total of 758 row segments. Consider the impact of that number of sorted row segment groups on a simple query like: ```sql SELECT * FROM sales WHERE store_id = 15; ``` This query filters on the `SORT KEY` which is `store_id` in this example. The very first sorted row segment group will have at most one row segment that contains rows with `store_id` equal to `15`, unless rows with `store_id = 15` are on the boundary of two row segments, or if there is a large data skew and several row segments consist only of rows with `store_id = 15`. Similarly, at most one row segment in the second sorted row segment group contains relevant rows, and the only segment of the third sorted row segment group might also contain relevant rows. This way, only three row segments out of the total of 758 will be opened and materialized. While the query in this example is very simple, similar reasoning works for significantly more complex queries. ## Optimize Sorted Row Segment Groups Now take a look at the sorted row segment groups on partition 3, which is significantly less optimized than the other partitions. A `SELECT` query similar to the one above will materialize eight row segments. If the background merger is enabled, and no workload is running concurrently, the background merger would optimize this partition quickly, likely within several seconds. However, in the presence of a heavy workload, the optimistic background merger might fall behind. In this case, it may be reasonable to manually trigger a pessimistic merger by running the following command: ```sql OPTIMIZE TABLE sales; ``` By running `SHOW COLUMNAR MERGE STATUS` while `OPTIMIZE TABLE` is running, you may see the manual merger in action: ```sql SHOW COLUMNAR MERGE STATUS FOR sales; ``` ```output +--------------------------------------------------------------------------------+ | Merger | State | Plan | Progress | Partition | +--------------------------------------------------------------------------------+ | (Current groups) | NULL | 741,16,1 | NULL | 0 | | (Current groups) | NULL | 782,20 | NULL | 1 | | (Current groups) | NULL | 701,40,5 | NULL | 2 | | (Current groups) | NULL | 326,207,123,37,21,19,17 | NULL | 3 | | Manual Merger | Working | 326+207+123+37+21+19+17 | 53.12% | 3 | +--------------------------------------------------------------------------------+ ``` The last row indicates that there is a manual merger running on partition 3 and that at this time `53.12%` of the work is complete. When the merger is done, the table has fewer sorted row segment groups in partition 3: ```sql SHOW COLUMNAR MERGE STATUS FOR sales; ``` ```output +------------------------------------------------------------+ | Merger | State | Plan | Progress | Partition | +------------------------------------------------------------+ | (Current groups) | NULL | 741,16,1 | NULL | 0 | | (Current groups) | NULL | 782,20 | NULL | 1 | | (Current groups) | NULL | 701,40,5 | NULL | 2 | | (Current groups) | NULL | 730,20 | NULL | 3 | +------------------------------------------------------------+ ``` During the optimization process, no partitions were merged into a single sorted row segment group as can be seen in the output shown above. Both optimistic and pessimistic mergers use an advanced algorithm which is optimized to do small chunks of work in the presence of concurrent writes and maintain data in a few sorted row segment groups, instead of attempting to merge all the data into a single sorted row segment group. In cases when it is acceptable to sacrifice some time on data ingestion to achieve even higher `SELECT` performance, it is possible to run a manual command that merges data on each partition into a single sorted row segment group. The command `OPTIMIZE TABLE FULL` will merge the data on each partition into a single sorted row segment group, as shown below. ```sql OPTIMIZE TABLE sales FULL; SHOW COLUMNAR MERGE STATUS FOR groups; ``` ```output +---------------------------------------------------------+ | Merger | State | Plan | Progress | Partition | +---------------------------------------------------------+ | (Current groups) | NULL | 758 | NULL | 0 | | (Current groups) | NULL | 802 | NULL | 1 | | (Current groups) | NULL | 746 | NULL | 2 | | (Current groups) | NULL | 750 | NULL | 3 | +---------------------------------------------------------+ ``` When the table is in this state, any highly selective `SELECT` query will materialize one row segment per partition. > **❗ Important**: Unlike `OPTIMIZE TABLE `, which takes the amount of time proportional to the size of recently loaded data, `OPTIMIZE TABLE FULL` takes an amount of time proportional to the order of magnitude of the size of the entire table, unless data in that table is already sorted. ## Inserts, Deletes, and Garbage Collection When inserting a small number of rows into the columnstore table, an in-memory rowstore-backed segment is used to store the rows. As this rowstore-backed segment fills, the `background flusher` periodically flushes these rows to disk. A rowstore-backed segment can be flushed to disk manually by running `OPTIMIZE TABLE FLUSH`. ```sql OPTIMIZE TABLE t FLUSH; ``` Deletes in columnstores are marked as deleted but otherwise left in place (refer to [Writing Columnstore Data](https://docs.singlestore.com/cloud/create-a-database/columnstore/writing-columnstore-data.md)). However, when a significant number of rows are deleted from a segment, the background merger rewrites those segments if the number of deleted rows is at least 1/8th of the desired segment size, and the sorted run isn't going to be merged otherwise. This rewrite happens per sorted run so it does not break the order of the run. `MV_COLUMNSTORE_MERGE_STATUS` shows the details about the merger processes. Unneeded columnstore segments stored in memory are eliminated in a process called garbage collection. A buffer manager maintains unused memory blocks for future use until no query or operation is using the memory anymore. The garbage collection process automatically runs in the background. *** Modified at: November 18, 2025 Source: [/cloud/create-a-database/columnstore/managing-columnstore-segments/](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments/) (An index of the documentation is available at /llms.txt) --- # Universal Storage Universal storage allows you to support large-scale Online Transaction Processing (OLTP) and Hybrid Transactional and Analytical Processing (HTAP) at a lower total cost of ownership (TCO). Universal storage is a continuing evolution of the [columnstore](https://docs.singlestore.com/cloud/create-a-database/columnstore.md), supporting transactional workloads that would have traditionally used the [rowstore](https://docs.singlestore.com/cloud/create-a-database/rowstore.md). Although rowstores are well-suited for transaction processing, it can be costly to store large datasets in rowstores, as they store all data in RAM. Because columnstores are disk-based, it is more economical to store data in columnstores as opposed to rowstores. ## Current Features in Universal Storage Currently, universal storage has five features that allow columnstores to process transactional workloads more efficiently: * [Hash indexes on columnstores](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) which may optionally require uniqueness. * Subsegment access, which speeds up access to a row in a columnstore. For more details, see item 6 in [Creating Efficient Columnstore Queries](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works/#UUID-a6ee7177-8029-15e5-a603-ff36c4e4ff41.md). * [Row-level locking on columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/locking-in-columnstores.md). * Joins on columnstores, where [one of the tables in the join uses a highly selective filter](https://docs.singlestore.com/cloud/create-a-database/columnstore/highly-selective-joins.md). * [Upserts](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/performing-upserts.md) on columnstores. Additionally, [sparse compression for rowstores](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) enables wide tables with a large percentage of `NULL` values to be stored in as little as half of the RAM. ## Related Content See [SingleStore's Patented Universal Storage - Part 4](https://www.singlestore.com/blog/singlestore-universal-storage-episode-4/) for a general discussion of all the capabilities of Universal Storage. *** Modified at: October 18, 2024 Source: [/cloud/create-a-database/columnstore/universal-storage/](https://docs.singlestore.com/cloud/create-a-database/columnstore/universal-storage/) (An index of the documentation is available at /llms.txt) --- # Writing Columnstore Data Unlike other columnstore implementations, SingleStore supports very fast, small-batch writes (such as single row inserts and updates) directly into columnstore tables. In some cases, writes (inserts and updates) are implemented by storing newly written rows in a rowstore-oriented skiplist before flushing them to the column-oriented format. Rows are visible to reads as soon as they are committed to the rowstore-oriented skiplist. In other cases, the rowstore-skiplist is bypassed and writes are made directly to the column-oriented format on disk. The following explains the behavior of different operations that can be performed against a columnstore. 1. **Insert** - Inserts into a columnstore will either go into the rowstore-backed segment or a new columnstore-backed row segment. In the latter case, the inserts are written directly to disk. If the engine variable `columnstore_disk_insert_threshold` has a low value, `INSERT...SELECT` `, INSERT ... ON DUPLICATE KEY UPDATE``, INSERT ... IGNORE` and `INSERT` write directly to disk, instead of writing first to rowstore-backed memory. Otherwise, `INSERT` first writes to rowstore-backed memory. 2. **Delete** - Deleting a row in a columnstore causes the row to be marked as deleted in the segment metadata, leaving the data in place within the row segment but not included in the query results. Disk space will not immediately free up because the data remains on the segment. The deleted data is still stored on the segment unless it is merged with another segment. Then the file is rewritten to update the segment metadata. Segments that contain only deleted rows are removed, and the optimization process covered below will compact segments that require optimization. 3. **Update** - An update in a columnstore is internally performed as a delete followed by an insert within a transaction. If the `columnstore_disk_insert_threshold` engine variable has a low value, `UPDATE` writes directly to disk, instead of writing first to rowstore-backed memory. Otherwise, `UPDATE` first writes to rowstore-backed memory. 4. **Replace** If the `columnstore_disk_insert_threshold` engine variable has a low value,`REPLACE` writes directly to disk, instead of writing first to rowstore-backed memory. Otherwise, `REPLACE` first writes to rowstore-backed memory. 5. **Optimization** - SingleStore has optimization routines that run automatically and can also be started manually. These routines attempt to automatically merge row segments together in order to improve query efficiency. DML statements can be performed while columnstore optimizations take place. For more information, see the [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) topic. SingleStore recommends using the default setting for the `columnstore_disk_insert_threshold` variable, which does straight to disk automatically when the batch is large enough. Lowering the value will increase the creation of small files. This causes the inodes in the file system to fill up rapidly and will require more work for the merger process when merging all these small files. SingleStore does not recommend setting the `columnstore_disk_insert_threshold` variable to zero. *** Modified at: March 8, 2024 Source: [/cloud/create-a-database/columnstore/writing-columnstore-data/](https://docs.singlestore.com/cloud/create-a-database/columnstore/writing-columnstore-data/) (An index of the documentation is available at /llms.txt) --- # Configuring the Columnstore to Work Effectively > **⚠️ Warning**: Configuration options described in this section may have significant performance impact on the workload. Make sure to test on a staging environment first before deploying the configuration changes. ## Configuring Segment Size in Columnstore Tables In SingleStore Helios, data in a columnstore table are organized into multiple row segments. For certain workloads, the size of row segments significantly affects performance. Considerations include: * In general, larger segments compress better than smaller ones. This leads to less disk usage and faster scan across the table. * On the other hand, smaller segments could benefit more from segment elimination. As a result, queries with highly selective filters on the columnstore index run faster with smaller segments. In SingleStore Helios, the default size of the row segments is controlled by the global variable `columnstore_segment_rows`. By default, `columnstore_segment_rows` has a value of `1024000`, meaning that each segment contains `1024000` rows by default. In addition to the global variable, it is possible to overwrite the global setting and set the segment size for a specific columnstore table. This can be done during the table creation, or by altering an existing table. For example: ```sql CREATE TABLE t (id INT, SORT KEY (id) WITH (columnstore_segment_rows=100000)); SHOW CREATE TABLE t; ``` ```output +-------+-------------------------------------------------------------------------+ | Table | Create Table | +-------+-------------------------------------------------------------------------+ | t | CREATE TABLE `t` ( | | | `id` int(11) DEFAULT NULL, | | | SORT KEY `id` (`id`) | | | /*!90619 */ /*!90621 WITH(COLUMNSTORE_SEGMENT_ROWS=100000) */ | | | ) | +-------+-------------------------------------------------------------------------+ ``` Now, modify the value of `columnstore_segment_rows` for this columnstore. The `MODIFY KEY keyName` clause identifies the index whose settings are to be changed (in this case, the columnstore index). ```sql ALTER TABLE t MODIFY KEY id SET (columnstore_segment_rows=20000); SHOW CREATE TABLE t; ``` ```output +-------+-----------------------------------------------------------------------------------+ | Table | Create Table | +-------+-----------------------------------------------------------------------------------+ | t | CREATE TABLE `t` ( | | | `id` int(11) DEFAULT NULL, | | | SORT KEY `id` (`id`) | | | /*!90619 */ /*!90621 WITH(COLUMNSTORE_SEGMENT_ROWS=20000) */ | | | ) | +-------+-----------------------------------------------------------------------------------+ ``` ## Configuring the Rowstore-backed Segment Size in Columnstore Tables In addition to the on-disk segments, each SingleStore Helios columnstore also has an in-memory rowstore-backed segment. In some cases, inserts and updates will write to the rowstore-backed segment first. In these cases, the `background flusher` process periodically compresses those recently inserted rows and creates on-disk segments. In other cases, inserts and updates will bypass the rowstore-backed segment, and the writes are made to the column-oriented format on disk. See [Writing Columnstore Data](https://docs.singlestore.com/cloud/create-a-database/columnstore/writing-columnstore-data.md) for more information. The size of the rowstore-backed segment is controlled by the global variable `columnstore_flush_bytes`. The `background flusher` process starts to create on-disk segments when the amount of data in the rowstore-backed segment exceeds `columnstore_flush_bytes`, with a default of `32 MB`. Additionally, insert and load operations are considered small-batch if they write to each partition less than `columnstore_flush_bytes * columnstore_disk_insert_threshold`, which is by default `16 MB` at a time. The amount of time (in seconds) the background flusher waits before trying to flush a table is set via `internal_columnstore_idle_flush_wait_seconds`. If the table row count has not changed in that period of time then the rows are flushed regardless of the row count. This flushes idle tables that have not reached the threshold. The minimum size of the disk-backed row segment created by insert and load operations is controlled by the engine variable `columnstore_disk_insert_threshold` . It is a fractional value with a default of `0.5`. Note that if `OPTIMIZE TABLE FLUSH` is manually run, the minimum segment size can be much smaller. Considerations about tuning the rowstore-backed segment size include: * The rowstore-backed segment is stored in-memory. Therefore, the table consumes less memory when the rowstore-backed segment is smaller. * The `background flusher` process can write more rows to disk at once if the rowstore-backed segment is larger, reducing the number of disk write operations during data ingestion. Similar to `columnstore_segment_rows`, the `columnstore_flush_bytes` value can also be configured per-table with the following syntax: ```sql CREATE TABLE t2 (id INT, SORT KEY (id) WITH (columnstore_flush_bytes=4194304, columnstore_segment_rows=100000)); ALTER TABLE t2 MODIFY KEY id SET (columnstore_flush_bytes=8388608); ``` ## Advanced Columnstore Configuration If your database utilizes unlimited storage, it might be beneficial to set the `columnstore_small_blob_combination_threshold` engine variable which is the maximum size of a columnstore blob. SingleStore combines multiple columns from a segment into a single blob. This single blob size must be at most the `columnstore_small_blob_combination_threshold` value. The `columnstore_small_blob_combination_threshold` engine variable refers to the maximum file size of two or more column blobs. It doesn't apply to the maximum size of individual column blobs. For example, consider the `columnstore_small_blob_combination_threshold` is set to 64mb and there are 3 column blobs sized 31mb, 31mb, and 92mb. The first column blob is 31mb and goes into a file. Then the second column blob is also 31mb. Since `columnstore_small_blob_combination_threshold` is greater than 62mb (the sum of the two column blobs), the second column blob is added to the file with the first column blob. When the third column blob (92mb) is processed, it can't fit into the existing file because the combined size would exceed the threshold. So the third column blob gets its own file, even though it's greater than the threshold value. > **📝 Note**: SingleStore suggests only advanced users set this engine variable unguided unless told to do so under the direction of our support team. The following is a use case where setting the maximum value of the `columnstore_small_blob_combination_threshold` variable is beneficial: * A columnstore table contains a large number of columns, but only a few columns are queried. * The local blob cache is cold or the data must be read from the remote object store, e.g. the table being read is larger than the cache. * All blob files have been combined to total nearly 32Mb in size. If a column read by a query is small in comparison, then all 32Mb from the remote object store must be read to access the data for the small column. Setting the `columnstore_small_blob_combination_threshold` engine variable to a smaller size can make this query run faster by reducing the amount of data that must be read from remote object storage. *** Modified at: March 18, 2026 Source: [/cloud/create-a-database/configuring-the-columnstore-to-work-effectively/](https://docs.singlestore.com/cloud/create-a-database/configuring-the-columnstore-to-work-effectively/) (An index of the documentation is available at /llms.txt) --- # Creating a Columnstore Table The default table type in SingleStore is columnstore. The default can be changed to rowstore by updating the `default_table_type` engine variable to rowstore. See [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for more details. Currently, SingleStore Helios supports one sort key per table; however, if you do not want to define a key (e.g. if records are already inserted in the right order or you do not care about the sort order), `SORT KEY()`. > **📝 Note**: `KEY() USING CLUSTERED COLUMNSTORE` is a legacy syntax that is equivalent to `SORT KEY()`. SingleStore recommends using `SORT KEY()`. Here is an example of a query that creates a columnstore table: ```sql CREATE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, Qty INT, SORT KEY (Price), SHARD KEY (ProductId) ); ``` As with any other SingleStore Helios table, we define a `SHARD KEY` to explicitly control the data distribution. We define it on `ProductId` since sharding on a high cardinality identifier column generally allows for a more even distribution and prevents skew. It is also possible to randomly distribute data by either omitting the shard key, or defining an empty shard key `SHARD KEY()`. You can have at most one shard key that is defined on zero or more columns. Your columnstore table definition can contain metadata-only unenforced unique keys, single-column and multi-column hash keys (which may be UNIQUE or PRIMARY), and a FULLTEXT key. For more information on creating columnstore tables that use hash keys, see [USING HASH behavior](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#UUID-2d098a7e-1811-6223-8748-d20d3c8b53f6.md). Besides the keys discussed so far in this section, you cannot add other keys to a columnstore table. If a sort key is defined on a table, the rows of that table will be stored in a highly compressed columnar format on disk. Storage is optimized for efficient scan in the key order. Use of an empty key in the form of `SORT KEY()` is permitted and results in less expensive background maintenance of the columnstore table, without guaranteeing any particular order. We describe the details of the format below. While SingleStore Helios can execute any query on a columnstore table that it can execute on a rowstore, some queries are more suitable for columnstore tables than others. Some queries that can benefit from using columnstore include: * Queries that scan several columns out of a table with many columns. Columnstore table will only materialize columns that it actually needs to perform the query. * Queries that scan a lot of data in sort key order. Scans in the columnstore key order are more CPU and memory efficient. * Joins between two columnstore tables on the sort key. Such a join can be executed as a merge join, resulting in a good performance and low memory overhead. * Queries with high selectivity filters on the sort key. By leveraging a technique called `segment elimination`, SingleStore Helios can open only those row segments that actually contain relevant rows, significantly improving the performance of queries with high selectivity. Columnstore tables are also not constrained by the amount of available memory, unlike rowstore tables. ## Creating a Columnstore Table Using Standard CREATE TABLE Syntax The columnstore as default feature allows you to create a columnstore table using standard `CREATE TABLE` syntax. ## The Benefit of Using the Columnstore for Transaction Processing Although rowstores are well-suited for transaction processing, it can be costly to store large datasets in rowstores, as they store all data in RAM. Because columnstores are disk-based, it is more economical to store data in columnstores as opposed to rowstores. The columnstore [continues to evolve to support transactional workloads](https://docs.singlestore.com/cloud/create-a-database/columnstore.md) that would have traditionally used the rowstore. At the same time, the columnstore provides the fastest query processing performance for analytical workloads, making it a good choice for a mix of both transactional and analytical requirements. ## Enabling Columnstore as Default The `default_table_type` engine variable specifies whether `CREATE TABLE` creates a rowstore or columnstore table, by default. > **📝 Note**: `default_table_type` is set to `columnstore` for clusters created with engine version 7.3 or newer. For clusters created with an engine version prior to 7.3, `default_table_type` remains at `rowstore`, even after the upgrade. You cannot change the value of `default_table_type`. You can create a rowstore table with `CREATE ROWSTORE TABLE ...`, regardless of the setting of `default_table_type`. ## Conversion of Standard `CREATE TABLE` Syntax to Columnstore `CREATE TABLE` Syntax When the `default_table_type` is `columnstore`, the query engine internally converts standard `CREATE TABLE` syntax to equivalent columnstore-specific `CREATE TABLE` syntax, as follows: ## `SORT KEY()` The `SORT KEY()` clause is added, if you have not specified the clause explicitly. For example, ```sql CREATE TABLE t(a INT, b INT); ``` is equivalent to: ```sql CREATE TABLE t(a INT, b INT, SORT KEY()); ``` ## `KEY(...)` ## Single-Column Keys `KEY()` is equivalent to `KEY() USING HASH`. For example, ```sql CREATE TABLE t(a INT, b INT, KEY(a)); ``` is equivalent to: ```sql CREATE TABLE t(a INT, b INT, SORT KEY(), KEY(a) USING HASH); ``` ## Multi-Column Keys `KEY(,, ... )` is equivalent to `KEY(,,... ) USING HASH`. For example, ```sql CREATE TABLE t(a INT, b INT, c INT, KEY(a,b)); ``` is equivalent to: ```sql CREATE TABLE t(a INT, b INT, SORT KEY(), KEY(a,b) USING HASH); ``` Using multi-column unique keys in a columnstore table is supported starting in engine version 7.5. A query against `t` with an equality filter on `a`, an equality filter on `b`, or equality filters on both `a` and `b` could benefit from `KEY(a,b) USING HASH`. A query that uses both equality filters would be the most efficient. Unlike other database products, SingleStore utilizes per-segment inverted indexes and cross-segment indexes to support multi-column keys while minimizing storage costs. Each secondary index contains a collection of multiple cross-segment indexes, each of which is a hash table covering one or more segments. The background merger (An algorithm used by SingleStore that allows columnstore segments to maintain sort order (as close as possible), while data is being ingested or updated.) works to merge the hash tables to reduce the number of lookups necessary for a seek operation. These data structures work together to accomplish these fast seeks as follows: * The inverted index is built to map values of an index column to a posting list that stores row offsets within the segment with the value searched. The posting list logs which segment a specific value is located and uses row offsets to determine the value is located. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8394be32540f2af9/6a7adae29f52514b8065e8fd/diagram_per-segment-index-jkuNnw.png) * The cross-segment indexes map values of each indexed column with the ID of the segment where the searched value is located. The cross-segment indexes also map to the starting location of the corresponding postings list in the inverted index for each segment. Lastly, for a multi-column index like the example above (`KEY(a,b) USING HASH)`), another collection of cross-segment indexes are built on the row of indexed columns to map from the hash of each tuple (`value_a` and `value_b`) to the starting locations of the corresponding per-segment postings lists for `value_a` and `value_b`. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt997d6481b1d362d4/6a7adbb373d9bd723a2924e8/diagram_cross-segment-hash-tables-eEzbQs.png) ## `UNIQUE KEY` `UNIQUE KEY()` is created as `UNIQUE KEY() USING HASH`. For example, ```sql CREATE TABLE t(a INT, b INT, SHARD(a), UNIQUE KEY(a)); ``` is equivalent to: ```sql CREATE TABLE t(a INT, b INT, SORT KEY(), SHARD(a), UNIQUE KEY(a) USING HASH); ``` > **📝 Note**: Columnstore tables do not support multiple unique keys. A unique key must contain all the columns of the shard key. The shard key cannot be empty in this case and can contain multiple columns.A columnstore table can either have a primary key or a unique key, but not both. ## `PRIMARY KEY` A primary key can be defined for a columnstore table using the `PRIMARY KEY(column_name)` clause or the `PRIMARY KEY` keyword in the `CREATE TABLE` statement. For example: ```sql CREATE TABLE t(a INT PRIMARY KEY, b INT); ``` ```sql CREATE TABLE t(a INT, b INT, PRIMARY KEY(a)); ``` The index behavior is equivalent to defining a key as ``UNIQUE KEY `PRIMARY` (a) USING HASH``. However, semantically these definitions are not the same. > **📝 Note**: A primary key must contain all the column(s) of the shard key.A columnstore table can either have a primary key or a unique key, but not both. ## Considerations When Using Hash Keys and Sort Keys > **⚠️ Warning**: The previous section explained that in columnstore as default mode, a standard `CREATE TABLE` statement using `KEY()` stores `` as a hash key in the resulting columnstore table. Hash keys support fast equality lookups, but not fast range filters. If you want to use a range filter on a columnstore, you need to use a sort key `` by explicitly specifying `SORT KEY()` in your `CREATE TABLE` statement. ## Viewing the Storage Structure of a Table > **📝 Note**: After creating a columnstore table using standard `CREATE TABLE` syntax, run [SHOW CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-table.md) if you wish to see the table’s full, explicit definition of keys and sharding. ## Converting an Existing Rowstore Table to a Columnstore Table You can convert an existing rowstore table to a columnstore table by creating a new columnstore table, copying the data from the rowstore table to the columnstore table, dropping the rowstore table, and renaming the columnstore table to the rowstore table. For example, suppose you have an existing rowstore table `t` with the columns `a INT` and `b INT`. You can convert this table to a columnstore table as follows, assuming `default_table_type` is set to `columnstore`: ```sql CREATE TABLE t2(a INT, b INT) AS SELECT * FROM t; DROP TABLE t; ALTER TABLE t2 RENAME TO t; ``` ## Example Database Schema that is Created with Columnstore as Default Suppose you want to create a database schema `track_visits` that tracks page visits on a website. For each page (URL) that is visited, you store the host name and path of the URL. For example, for the URL `myexample.com/a/b/`, the host name is `myexample.com` and the path is `a/b`. For each page that is visited, you store the date/time of the visit and the user who visited the page. You also store the account that each user belongs to. The schema contains the tables `accounts`, `users`, `webpages` and `webpage_visits`. The definitions for these tables are provided in the [Standard Syntax for Creating the Tables in the Schema](https://docs.singlestore.com/cloud/create-a-database/creating-a-columnstore-table/#UUID-bd806520-e67c-53d2-f56f-f0392e5dafde.md) section. ## Use Case for Using Default as Columnstore to Create the `track_visits` Schema ## Analytical Processing Columnstores (as compared to rowstores) are optimized for analytical processing. With this schema, you can more quickly answer analytical questions such as: * How many visits occurred within a particular time period? * On average, how many visits are there per user? * On average, how many users are there per account? * How many unique webpages were clicked? ## Transactional Processing You can utilize the [Universal Storage](https://docs.singlestore.com/cloud/create-a-database/columnstore/universal-storage.md) capabilities to perform transaction processing operations on the tables in the schema. For example: * To find all of the webpage visits containing a particular hostname, you can run a query such as `SELECT * FROM webpages WHERE url_host = 'myexample.com';`. This query allows you take advantage of the hash index on `url_host`, which allows the query to run faster than if the hash index was not used. * To find all of the webpages visited on a particular date and time, you can run a query such as `SELECT * FROM webpage_visits v JOIN webpages w ON v.page_id = w.id WHERE v.when_clicked = '2020-01-01 16:35:00';` This query does a highly selective join, as there are only a few records in `webpage_visits` that match the equality filter on `when_clicked`. ## Standard `CREATE TABLE` Syntax for Creating the Tables in the `track_visits` Schema Following are the `CREATE TABLE` statements for the tables in the `track_visits` schema. The statements use standard `CREATE TABLE` syntax. The statements are run in columnstore as default mode. ```sql CREATE DATABASE track_visits; USE track_visits; CREATE REFERENCE TABLE accounts( id INT AUTO_INCREMENT, name TEXT, PRIMARY KEY(id), UNIQUE KEY(name) ); CREATE TABLE users ( id BIGINT AUTO_INCREMENT, user_name TEXT, account_id BIGINT, PRIMARY KEY (id) ); CREATE TABLE webpages( id BIGINT AUTO_INCREMENT, url_host TEXT, url_path TEXT, PRIMARY KEY (id), KEY(url_host) ); CREATE TABLE webpage_visits( id BIGINT AUTO_INCREMENT, user_id BIGINT, page_id INT, when_clicked DATETIME, PRIMARY KEY (id), KEY (user_id, page_id), KEY (when_clicked) ); ``` ## Storage Structure of Tables in the `track_visits` Schema The following definitions show the structure of the tables in the `track_visits` schema after they are stored as a result of running the [standard CREATE TABLE statements](https://docs.singlestore.com/cloud/create-a-database/creating-a-columnstore-table.md). > **📝 Note**: `INT`, and `BIGINT`, which are used in the standard `CREATE TABLE` statements, are stored as `INT(11)` and `BIGINT(20)`, respectively.In some cases, `TEXT` is stored as `BLOB` (they are equivalent).This behavior is independent of the columnstore as default feature. ## `accounts` ```sql CREATE REFERENCE TABLE accounts( id INT NOT NULL AUTO_INCREMENT, name TEXT, UNIQUE KEY `PRIMARY` (id) USING HASH, UNIQUE KEY(name) USING HASH, SORT KEY() ); ``` ## `users` ```sql CREATE TABLE users ( id BIGINT(20) NOT NULL AUTO_INCREMENT, user_name TEXT, account_id BIGINT, SHARD(id), UNIQUE KEY `PRIMARY` (id) USING HASH, SORT KEY() ); ``` ## `webpages` ```sql CREATE TABLE webpages( id BIGINT NOT NULL AUTO_INCREMENT, url_host BLOB, url_path BLOB, SHARD(id), UNIQUE KEY `PRIMARY` (id) USING HASH, KEY(url_host) USING HASH, SORT KEY() ); ``` ## `webpage_visits` ```sql CREATE TABLE webpage_visits( id BIGINT(20) NOT NULL AUTO_INCREMENT, user_id BIGINT(20), page_id INT(11), when_clicked DATETIME, SHARD(id), UNIQUE KEY `PRIMARY` (id) USING HASH, KEY(user_id, page_id) USING HASH, KEY (when_clicked) USING HASH, SORT KEY() ); ``` ## Querying `webpage_visits` Using a Range Filter If you intend to query `webpage_visits` using a range filter on `when_clicked`, you could have made `when_clicked` the sort key. You could have done this using the following standard `CREATE TABLE` syntax. ```sql CREATE TABLE webpage_visits( id BIGINT AUTO_INCREMENT, user_id BIGINT, page_id INT, when_clicked DATETIME, PRIMARY KEY (id), KEY (user_id, page_id), SORT KEY (when_clicked) ); ``` The query engine would internally convert the standard `CREATE TABLE` syntax to equivalent columnstore-specific `CREATE TABLE` syntax, as follows: ```sql CREATE TABLE webpage_visits( id BIGINT AUTO_INCREMENT, user_id BIGINT, page_id INT, when_clicked DATETIME, UNIQUE KEY `PRIMARY` (id) USING HASH, KEY (user_id) USING HASH, KEY (page_id) USING HASH, SORT KEY (when_clicked) ); ``` *** Modified at: September 17, 2025 Source: [/cloud/create-a-database/creating-a-columnstore-table/](https://docs.singlestore.com/cloud/create-a-database/creating-a-columnstore-table/) (An index of the documentation is available at /llms.txt) --- # Detecting and Resolving Data Skew An important property of a distributed database is that the data gets distributed more or less evenly. In rare cases the data may be “skewed” out of balance. This topic discusses how skew can happen, how to detect it, and how to resolve it. “Skew” is a condition in which a table’s data is unevenly balanced among partitions in the cluster. Probability being what it is, small amounts of skew are inevitable and harmless. By default, SingleStore distributes data based on a hash of the primary key. Since the hash values are unique and evenly distributed, this keeps skew to a minimum. You can tell SingleStore to shard a table by a set of columns instead of the primary key. For example, consider a table of URLs from web domains. It might be defined as follows: ```sql CREATE TABLE urls ( id BIGINT, domain_id BIGINT, path VARCHAR(8192), first_seen INT UNSIGNED NOT NULL, crawl count INT UNSIGNED NOT NULL, SHARD KEY (domain_id) ); ``` As you can see, we are sharding on `domain_id`. Queries made against this table will compile and run just fine. However, it’s probable that some domains will have many more times the number of URLs. For example, the partition that contains links for “youtube.com” will almost certainly have more rows than average, throwing off the balance of the cluster. That unlucky partition containing “youtube.com” will not only have to store more data, it will probably be asked to do more work during `SELECT`, `UPDATE`, and `DELETE` queries. For best performance, you should choose a shard key that minimizes data skew. ## Measuring and Detecting Skew > **📝 Note**: While bulk-loading data, recovering from a restart, or after deleting lots of data, it’s possible that one partition may temporarily show more memory use and memory skew than others, as memory allocation and garbage collection operations may not take effect immediately across partitions. This is normal, and corrects itself over time as the system reaches a new steady state. You can use the following methods to measure and detect data skew. ## Method 1: Inspect memory settings With any version of SingleStore, you can manually compare `maximum_table_memory` and `Alloc_table_memory` (memory used by tables) on each node: ```sql SHOW VARIABLES LIKE "maximum_%"; ``` ```output +----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | maximum_memory | 48291 | | maximum_table_memory | 43462 | +----------------------+-------+ ``` ```sql SHOW STATUS EXTENDED LIKE "Alloc_table_memory"; ``` ```output +--------------------+----------+ | Variable_name | Value | +--------------------+----------+ | Alloc_table_memory | 43462 MB | +--------------------+----------+ 1 row in set (0.01 sec) ``` If memory consumption is significantly higher on specific nodes, skew is present. If memory consumption is fairly uniform across the leaf nodes, no skew is present. ## Method 2: Find the number of rows and memory use per partition The built-in `INFORMATION_SCHEMA` database contains metadata about tables, columns, indexes, and partitions. In particular, the `TABLE_STATISTICS` table contains one row per table partition, and can be used to see if a given partition has a lot more rows than average: ```sql SELECT DATABASE_NAME, TABLE_NAME, ORDINAL AS PARTITION_ID, ROWS, MEMORY_USE FROM INFORMATION_SCHEMA.TABLE_STATISTICS WHERE TABLE_NAME = 'domains'; ``` ```output +---------------+------------+--------------+-------+------------+ | DATABASE_NAME | TABLE_NAME | PARTITION_ID | ROWS | MEMORY_USE | +---------------+------------+--------------+-------+------------+ | web | domains | 0 | 85916 | 11527936 | | web | domains | 1 | 85925 | 11528512 | | web | domains | 2 | 85427 | 11627712 | | web | domains | 3 | 86185 | 11545152 | | web | domains | 4 | 85592 | 11376128 | | web | domains | 5 | 85549 | 11504448 | ... ``` ## Method 3: Calculate the memory skew for all tables, per database Use the [relative standard deviation (RSD)](http://en.wikipedia.org/wiki/Relative_standard_deviation) to calculate the memory skew across all partitions and tables in your database. The RSD is a decent way to measure “skewness”. ```sql SELECT DATABASE_NAME, TABLE_NAME, FLOOR(AVG(ROWS)) AS avg_rows, ROUND(STDDEV(ROWS)/AVG(ROWS),3) * 100 AS row_skew, FLOOR(AVG(MEMORY_USE)) AS avg_memory, ROUND(STDDEV(MEMORY_USE)/AVG(MEMORY_USE),3) * 100 AS memory_skew FROM INFORMATION_SCHEMA.TABLE_STATISTICS GROUP BY 1, 2 HAVING SUM(ROWS) > 10000 ORDER BY row_skew DESC; ``` ```output +---------------+----------------------+----------+----------+--------------+-------------+ | DATABASE_NAME | TABLE_NAME | avg_rows | row_skew | avg_memory | memory_skew | +---------------+----------------------+----------+----------+--------------+-------------+ | web | urls | 28881401 | 9.520 | 25733328291 | 3.200 | | web | domains | 902817 | 0.100 | 111046491 | 0.200 | ... ``` A skew score of N means that there is about N% variation in the number of rows (or memory allocated) in a partition. As a general rule of thumb, skew less than 10% should not greatly affect system stability or performance and is not a cause for worry. If you do see persistent skew and observe other issues as well, such as a server running particularly “hot”, then you should further investigate the distribution of the values in your chosen shard keys. ## Method 4: Calculate the skew per partition for the columns in a table Again, use the [relative standard deviation (RSD)](http://en.wikipedia.org/wiki/Relative_standard_deviation) to calculate the memory skew across all partitions in a specific tables in your database. If the skew factor is high, that indicates that some partitions have very high cardinality and some have very low cardinality and can lead to long single-threaded operations. Considering the `urls` table above, you can calculate the data skew per partition in the following example: ```sql SELECT ROUND(STDDEV(id)/AVG(id),3)*100 AS group_skew, PARTITION_ID() FROM (SELECT id, domain_id, count(*) FROM urls GROUP BY 1, 2) sub GROUP BY PARTITION_ID(); ``` ## Method 5: Show the data distribution for a table if you were to reshard on another column or set of columns Run this query to check for data skew prior to making shard key changes: ```sql SELECT with(leaf_pushdown=true) SUM(c) rows, PARTITION_ID() FROM (SELECT count(*) c FROM GROUP BY ) reshuffle GROUP BY PARTITION_ID(); ``` Considering the `urls` table above, you can show the potential distribution of data if the table is resharded on a different column or set of columns. See the following example: ```sql SELECT with(leaf_pushdown=true) SUM(c) rows, PARTITION_ID() FROM (SELECT count(*) c FROM urls GROUP BY id) reshuffle GROUP BY PARTITION_ID(); ``` ```output +---------+----------------+ | rows | PARTITION_ID() | +---------+----------------+ | 2005 | 1 | | 1999 | 3 | | 2001 | 4 | +---------+----------------+ ``` This example shows if the `urls` table is resharded on the `id` column, there would be a more even distribution of data on each partition. ## Resolving Data Skew SingleStore distributes data based on the SHARD KEY specified in a table’s schema. If no SHARD KEY is explicitly specified, the PRIMARY KEY is used as the default SHARD KEY. If the SHARD KEY shards on a low cardinality column, data will accumulate on certain nodes. To resolve this, you should export your data, modify your schema, and then reload your data. To export your data, see [Export Data](https://docs.singlestore.com/cloud/manage-data/export-data.md). After you backup your data, you can `DROP TABLE low_cardinality_shard_key_table`, and then `CREATE TABLE` with a higher cardinality SHARD KEY. When you reload the data, SingleStore will automatically shard your data based on the new SHARD KEY. Example: ```shell $ mysqldump -h 127.0.0.1 -u root my_db low_cardinality_shard_key_table --no-create-info > databackup.sql ``` ```sql DROP TABLE low_cardinality_shard_key_table; source updated_shard_key_schema.sql; ``` *** Modified at: July 10, 2026 Source: [/cloud/create-a-database/detecting-and-resolving-data-skew/](https://docs.singlestore.com/cloud/create-a-database/detecting-and-resolving-data-skew/) (An index of the documentation is available at /llms.txt) --- # Multi-Value Hash Index (JSON) Multi-Value Hash index optimizes lookups within JSON objects and arrays. It can efficiently find rows where a JSON array contains a specific value or where a specific key-path holds an exact value match. For example, if an **orders** table has a JSON column **details** with the following data: ```JSON {"product_id":[{"id":1001,"type":"flour"},{"id":1002,"type":"yeast"}]} ``` An index created on the **details** column at the path **\["product\_id"**, **"id"]** enables exceptionally fast lookups for **orders** by **product\_id.id**. To index values at the top level in an array, specify an empty path array, such as, `"PATH":[]`. For example, if you have a column containing a top-level JSON array like `[1,2,3]`, you can accelerate searches for values within the array. The multi-value hash index stores pairs of values along with the list of row IDs where that value occurs. Since each value is stored only once, the index remains compact even when values are repeated across many rows. This index is fundamentally a hash index, leveraging the same high-performance technology as the regular columnar hash indexes. ## Syntax ```sql -- Adding the multi-value index to a new table -- CREATE TABLE ( JSON NULL, MULTI VALUE INDEX(col) INDEX_OPTIONS=''); -- Adding the multi-value index an to existing table column -- ALTER TABLE ADD MULTI VALUE INDEX (col) INDEX_OPTIONS=''; ``` where `` is a JSON object to specify the tokenizer and other metadata related to it. ```sql '{"TOKENIZER":"MATCH_ANY", "PATH":[]}' ``` where `` is a comma-separated list of values that specify the path to the field that needs to be indexed. If an empty path array, such as, `"PATH":[]`, is specified, then the top-level elements in the array are indexed (such as \[1,2,3]). ## Using the Multi-Value Hash Index The multi-value hash index can optimize queries that use the [JSON\_MATCH\_ANY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) function. This function checks for the existence of values inside JSON data and can be used to find values that are valid JSON types (object, array, integer, float, string, boolean, or JSON `null`) > **📝 Note**: To use the multi-value hash index, the [filter predicate](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any/#section-idm234523191190095.md) in the `JSON_MATCH_ANY` function must use the `MATCH_PARAM_JSON` expression.When matching string values with `JSON_MATCH_ANY` and `MATCH_PARAM_JSON`, string values must be JSON strings with quotes.Use either of the following to create JSON strings:- Use [TO\_JSON](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/to-json.md) to convert a string to a JSON string. > - Manually specify a quoted constant string literal, for example `'"val"'`. **Note**: Typecasting to JSON using `:>JSON` does not automatically wrap an unquoted string into a JSON string. (However, casting works to convert numeric types to their respective JSON type. For example, `2.5:>JSON` converts `2.5` into a JSON double.) **Supported Query Shapes** ```sql JSON_MATCH_ANY(MATCH_PARAM_JSON() = , ) JSON_MATCH_ANY(MATCH_PARAM_JSON() <=> , ) JSON_MATCH_ANY(MATCH_PARAM_JSON() IN (, ..), '') ``` **Shorthand Syntax** You can also use JSON shorthand syntax (`::` operators) in `JSON_MATCH_ANY` predicates. ```sql JSON_MATCH_ANY(::?, MATCH_PARAM_JSON() = ) JSON_MATCH_ANY(::?, MATCH_PARAM_JSON() <=> ) JSON_MATCH_ANY(::?, MATCH_PARAM_JSON() IN (, ..)) ``` `` can be a constant literal (such as a string, integer, float), a typecast operator (`:>`, `!:>`), a user-defined variable (UDV), a deterministic built-in expression (most built-in functions, except for non-deterministic ones like `UUID()` or `RAND()`), or a combination of these. In general, the lookup value must remain constant during query execution. If the value changes per row, such as a column reference, then the index cannot be effectively used and the query falls back to a regular table scan. The index is only used for queries that match the exact `` defined during index creation. For example, if the index is created on **addresses::city**, the query benefits from the index only when filtering values specifically under this path. It is not applied to deeper paths like **addresses::city::district** or to sibling keys like **addresses::zip**. To apply the index to any other path, you must create a separate index. To check whether the index was used for your query filter, run an `EXPLAIN` query - the phrase **multi-value hash index** in the predicate under **ColumnStoreFilter** indicates that the index was used.. ```sql EXPLAIN SELECT col FROM t WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON() = TO_JSON('type'), col, 'key1'); ``` ```output +--------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------+ | Project [t.col] | | ColumnStoreFilter [JSON_MATCH_ANY(MATCH_PARAM_JSON() = TO_JSON('type'), t.col, 'key1') multi-value hash index] | | ColumnStoreScan db.t, SORT KEY __UNORDERED () table_type:sharded_columnstore | +--------------------------------------------------------------------------------------------------------------+ ``` ## Remarks * Multi-value hash index is case-sensitive for JSON types. * Multi-value hash index for JSON activates when matched using the following operators: `=`, `<=>`, and `IN()`. * The following table lists the expressions that support Multi-Value Hash Index for each of these operators: | Operator | Supported Expression | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `=`,`<=>` |
  • Constant literals, for example strings, NULL, DOUBLE, INT, etc.
  • Typecast operators :> and !:>, for example, 100001:>JSON.
  • User-defined variables (UDVs), for example: 
    SET @obj = '{"city": "New York"}':>JSON;  SELECT a:>JSON FROM t  WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON()=@obj, a, 'city');
  • Procedural SQL variables (stored procedure arguments and local variables).
  • Nested constant (deterministic) built-in expressions, for example: 
    JSON_BUILD_ARRAY(JSON_SET_JSON('{}', 'a', @json), '{"a":4}':>JSON)
  • A combination of all of the above.
| | `IN()` |
  • Parameterized IN lists with:
    • Literals of the same types
    • Single argument built-in expressions of the same shape and literal types in the list, for example, IN(HEX('a'), HEX('b')).
| ## Limitations * The [filter predicate](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any/#section-idm234523191190095.md) in `JSON_MATCH_ANY` must use the `MATCH_PARAM_JSON` expression. * Only runtime constant expressions are supported for matching (i.e. UDV, builtins, typecasts, PSQL variables are supported; UDFs and column fields are not supported). * Using `OR` inside a `JSON_MATCH_ANY` predicate (for example, `JSON_MATCH_ANY(MATCH_PARAM_JSON() = 12 OR MATCH_PARAM_JSON() = 1, a, 'a'))` disables index usage. Instead, use either an `IN` list or multiple `JSON_MATCH_ANY` filters. * Only parameterized `IN` list is supported. * Indexing multiple key paths for the same JSON field requires separate indexes, for example, to index both **a::k1** and **a::k2**, you must create two indexes. * This index is supported only for columnstore tables. * Multi-value index for JSON does not support the following: * References to other unindexed fields on the right-hand side of an expression. * Non-deterministic built-in functions, for example `RAND()`. * `JSON_MATCH_ANY` predicate with `MATCH_ELEMENTS` option. * `!=` and other comparison operators (excluding `=` and `<=>`). * `NOT IN` lists. ## Examples ## Example 1- Searching `users` by `city` Consider the following table definition: ```sql CREATE TABLE users ( id BIGINT PRIMARY KEY AUTO_INCREMENT, created DATETIME(6) DEFAULT NOW(6), data JSON ); ``` User records are stored in the `data` column as JSON documents with the following structure: ```sql INSERT INTO users (data) VALUES ('{"id":1,"name":"Alice","addresses":[{"street":"123 Main St","city":"New York","zip":"10001"},{"street":"456 Oak Ave","city":"Los Angeles","zip":"90001"}]}'), ('{"id":2,"name":"Bob","addresses":[{"street":"789 Pine Ln","city":"Chicago","zip":"60601"}]}'), ('{"id":3,"name":"Charlie","addresses":[{"street":"101 Elm St","city":"New York","zip":"10002"},{"street":"222 Maple Dr","city":"Houston","zip":"77001"}]}'); ``` Adding a multi-value Hash Index to the path "**addresses, city**" can significantly improve lookup performance. ```sql ALTER TABLE users ADD MULTI VALUE INDEX (data) INDEX_OPTIONS='{"TOKENIZER":"MATCH ANY","PATH":["addresses", "city"]}'; ``` Find all `users` who live in Houston using the `JSON_MATCH_ANY` built-in function. ```sql SELECT id, data::name FROM users WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON()='"Houston"', data, 'addresses', 'city'); ``` ```output +----+------------+ | id | data::name | +----+------------+ | 3 | "Charlie" | +----+------------+ ``` The following example rewrites the previous query using shorthand syntax: ```sql SELECT id, data::name FROM users WHERE JSON_MATCH_ANY(data::?addresses.city, MATCH_PARAM_JSON() = '"Houston"'); ``` ```output +----+------------+ | id | data::name | +----+------------+ | 3 | "Charlie" | +----+------------+ ``` To verify that the multi-value hash index was used, run the `EXPLAIN` query as: ```sql EXPLAIN SELECT id, data::name FROM users WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON()='"Houston"', data, 'addresses', 'city'); ``` ```output +-----------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:segment | | Project [users.id, JSON_EXTRACT_JSON(users.data,'name') AS `data::name`] | | ColumnStoreFilter [JSON_MATCH_ANY(MATCH_PARAM_JSON() = '\"Houston\"',users.data,'addresses','city') multi-value hash index] | | ColumnStoreScan db1.users, SORT KEY __UNORDERED () table_type:sharded_columnstore | +-----------------------------------------------------------------------------------------------------------------------------+ ``` ## Example 2 - Multiple indexes In this example, two separate multi-value hash indexes are used to optimize common lookup patterns on a JSON column for the following two fields: * An integer field **eventId** * A string array **tags** These indexes help speed up queries that filter by **eventID** and check for specific tags without scanning the entire table. 1. Create a table. ```sql CREATE TABLE events ( id BIGINT PRIMARY KEY, created DATETIME(6), payload JSON ); ``` 2. Add multi-value hash indexes via `ALTER TABLE`. ```sql ALTER TABLE events ADD MULTI VALUE INDEX idx_event_id (payload) INDEX_OPTIONS = '{"TOKENIZER": "MATCH ANY", "PATH": ["eventId"]}'; ALTER TABLE events ADD MULTI VALUE INDEX idx_tags (payload) INDEX_OPTIONS = '{"TOKENIZER": "MATCH ANY", "PATH": ["tags"]}'; ``` 3. Insert data into the table. ```sql INSERT INTO events VALUES (1, NOW(6), JSON_BUILD_OBJECT('eventId', 101, 'tags', JSON_BUILD_ARRAY('sports', 'outdoor', 'fun'))), (2, NOW(6), JSON_BUILD_OBJECT('eventId', 202, 'tags', JSON_BUILD_ARRAY('music', 'concert'))), (3, NOW(6), JSON_BUILD_OBJECT('eventId', 101, 'tags', JSON_BUILD_ARRAY('sports', 'indoor'))); ``` 4. Query using both the indexed fields: ```sql SELECT * FROM events WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON() = 101:>JSON, payload, 'eventId') AND JSON_MATCH_ANY(MATCH_PARAM_JSON() = TO_JSON('sports'), payload, 'tags'); ``` ```output +----+----------------------------+---------------------------------------------------+ |id | created | payload | +----+----------------------------+---------------------------------------------------+ | 1 | 2025-04-29 15:36:52.797432 | {"eventId":101,"tags":["sports","outdoor","fun"]} | | 3 | 2025-04-29 15:36:52.797432 | {"eventId":101,"tags":["sports","indoor"]} | +----+----------------------------+---------------------------------------------------+ ``` The following example rewrites the previous query using shorthand syntax: ```sql SELECT * FROM events WHERE JSON_MATCH_ANY(payload::?eventId, MATCH_PARAM_JSON() = 101:>JSON) AND JSON_MATCH_ANY(payload::?tags, MATCH_PARAM_JSON() = TO_JSON('sports')); ``` ```output +----+----------------------------+---------------------------------------------------+ | id | created | payload | +----+----------------------------+---------------------------------------------------+ | 3 | 2026-04-02 17:36:43.595620 | {"eventId":101,"tags":["sports","indoor"]} | | 1 | 2026-04-02 17:36:43.595620 | {"eventId":101,"tags":["sports","outdoor","fun"]} | +----+----------------------------+---------------------------------------------------+ ``` 5. Verify index usage from the query `EXPLAIN`: ```sql EXPLAIN SELECT * FROM events WHERE JSON_MATCH_ANY(MATCH_PARAM_JSON() = 101:>JSON, payload, 'eventId') AND JSON_MATCH_ANY(MATCH_PARAM_JSON() = TO_JSON('sports'), payload, 'tags'); ``` ```output +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:segment | | Project [events.id, events.created, events.payload] | | ColumnStoreFilter [JSON_MATCH_ANY(MATCH_PARAM_JSON() = (101:>JSON COLLATE utf8mb4_bin NULL), events.payload, 'eventId') multi-value hash index AND JSON_MATCH_ANY(MATCH_PARAM_JSON() = TO_JSON('sports'), events.payload, 'tags') multi-value hash index] | | ColumnStoreScan db1.events, SORT KEY __UNORDERED () table_type:sharded_columnstore | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` *** Modified at: July 14, 2026 Source: [/cloud/create-a-database/multi-value-hash-index-json/](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json/) (An index of the documentation is available at /llms.txt) --- # Optimizing Table Data Structures This guide assumes the reader is familiar with the following concepts: * How to [connect to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md) * Creating tables and [running queries](https://docs.singlestore.com/cloud/query-data.md) in a SQL database * [Why you would add an index to a table](https://docs.singlestore.com/cloud/query-data/query-tuning.md) ## In this section * [Key Considerations for Understanding Your Workload](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/key-considerations-for-understanding-your-workload.md) * [Shard Keys](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/shard-keys.md) * [Choosing Rowstore Keys](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-rowstore-keys.md) * [Choosing a Columnstore Key](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key.md) *** Modified at: March 15, 2023 Source: [/cloud/create-a-database/optimizing-table-data-structures/](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/) (An index of the documentation is available at /llms.txt) --- # Choosing a Columnstore Key Columnstore tables have exactly one index, the sort key. Using this sort key, columnstores group the rows into logical segments, where each segment contains data for many rows. The sort key controls the ordering of data in the blobs that compose the segment’s data. Data within a segment is stored on disk into segment files containing the same field for many rows. This enables two important capabilities. One is to scan each column individually; in essence, being able to scan only the columns that are needed to execute a query with a high degree of locality. The other capability is that columnstores lend themselves well to compression; for example, repeating and similar values can be easily compressed together. In addition, SingleStore Helios stores metadata for each row segment in memory, which includes the minimum and maximum values for each column contained within the segment. This metadata is used at query execution time to determine whether a segment can possibly match a filter, a process known as segment elimination. For example, we will use this columnstore table: ```sql CREATE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, Quantity INT, SORT KEY (Price) ); ``` The following table represents a logical arrangement of the data in a single partition of the database: | ProductId | Color | Price | Quantity | | --------- | ----- | ----- | -------- | | 1 | Red | 10 | 2 | | 2 | Red | 20 | 2 | | 3 | Black | 20 | 2 | | 4 | White | 30 | 2 | | 5 | Red | 20 | 2 | | 6 | Black | 10 | 2 | | 7 | White | 25 | 2 | | 8 | Red | 30 | 2 | | 9 | Black | 50 | 2 | | 10 | White | 15 | 2 | | 11 | Red | 5 | 2 | | 12 | Red | 20 | 2 | | 13 | Black | 35 | 2 | | 14 | White | 30 | 2 | | 15 | Red | 4 | 2 | Since this is a columnstore table, the data is split into segments. Column segments typically contain on the order of tens of thousands of rows; in this example, the segment size is 5 rows for readability. In addition, we will use the convention “x N” to indicate that a value is repeated N times. *Row segment #1 of 3*: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7577298a98942efb/6a7ada5cfc63ab36b66417fb/16083b5768fd90-6I5D76.png) *Row segment #2 of 3*: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte0c6226d24af53a0/6a7ada6873d9bd61ac292472/16083b57697c3e-MW9Gsc.png) *Row segment #3 of 3*: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt561fc3b34abd0f60/6a7ada5d448e4e37b05b5419/16083b5769fea7-IJHyTg.png) In this example, we can see there are 3 segments containing different ranges of the `Price` column (4-15, 20-25, and 30-50). Each segment has the same number of rows and is organized data by column. For each column, we store the minimum and maximum values in the segment as metadata. ## Segment Elimination The single most important consideration for columnstore tables is setting a sort key. Each segment file contains metadata that holds the minimum and maximum values for the rows in the segment. The minimum/maximum value metadata is used at query execution time to determine whether a segment can possibly match a filter; if not, the segment is skipped entirely and no data is examined. This functionality is called segment elimination because the segment is eliminated from the scan. For queries that filter on the sort key column of the columnstore index, segment elimination is typically very efficient as row segments within each row segment group will not have overlapping segments. For example, in the above table the query `SELECT AVG(Price), AVG(Qty) FROM Products WHERE Price BETWEEN 1 AND 10;` will eliminate all segments except row segment #1 Price 4-15. The segment size (number of rows per segment) also impacts query execution with respect to the sort key. A smaller segment size means that a smaller number of rows are read from the segments that pass segment elimination. Column segments typically contain on the order of tens of thousands of rows. Indexing further cuts down on the number of rows scanned when a select query with a `WHERE` clause is executed. If the table has an index for the columns in the `WHERE` clause, the engine can quickly determine the location of the matching rows position to seek without having to look at all the data. This is much faster than reading every row sequentially. Sub-segment elimination is also supported. Refer to [Understanding Sort Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-sort-key-selection.md) for more information. ## Ordered Scans Ordered scan over a sort key is faster than sorting a table. A query that sorts with a sort key can benefit from running an ordered columnstore scan instead of performing a normal columnstore scan. ## Questions to Ask When Choosing a Columnstore Key: * Is the data always filtered by some column (e.g. insert timestamp or event type)? Ensure that the common columns for all queries are in the sort key to improve segment elimination.  * Is the data generally inserted in order by some column (e.g. insert timestamp)? It’s best to put that column first in the sort key to minimize the amount of work required by the background columnstore segment merger. * Does one column in your key have a higher cardinality than the other? It’s best to put the lowest cardinality columns first to increase the likelihood that segment elimination will be able to affect later columns. ## Check Your Understanding **Q**: Can the columnstore key and the shard key be different? **A**: Absolutely – they are not related at all. It is very common to pick a shard key that matches other rowstore tables (e.g. `event_id`) to improve join performance and to pick an unrelated columnstore key that matches the common filters on the table (e.g. `event_timestamp` or `event_type`). **Q**: Is `(insert_datetime6, region_id)` a good columnstore key? **A**: Not necessarily, because the number of distinct microsecond precision timestamps is likely very high. The partitioning of data into segments will likely be entirely controlled by `insert_datetime6` and not use `region_id`. This means that queries scanning microseconds of data will get good segment elimination, because they’ll match fewer segments. However, queries that scan days or months of data would not benefit from segment elimination on `region_id`. If the data is frequently accessed days or months at a time, it would be better to truncate the precision of the timestamp. For example, use a datetime type or a computed column of `DATE_TRUNC('hour', insert_datetime6)`. In general, your columnstore key should not be more precise than your query filters. For more information, see [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md). *** Modified at: July 24, 2025 Source: [/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key/](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key/) (An index of the documentation is available at /llms.txt) --- # Choosing Rowstore Keys You may define indexes, also called keys, on SingleStore Helios rowstore tables. SingleStore Helios uses these keys to efficiently find specific rows. There are two storage types for rowstore indexes: a lockfree skiplist and a lockfree hash table. In both cases, we use lockfree data structures to optimize the performance of concurrent updates to the table. * By default, indexes are stored as skiplists, which have similar functional and performance characteristics as B-trees in other databases. A skiplist is a data structure optimized for ordered data that stores rows in collections of increasingly smaller ordered lists. Queries can quickly seek data by binary searching using the different sized lists and can quickly scan over ranges of data by iterating over the largest list. For multi-column indexes, query filters must match a prefix of the index column list to be able to take advantage of the index. * A hash table is a data structure optimized for fast lookups, which stores rows in a sparse array of buckets indexed by a hash function on the relevant columns. Queries can quickly find exact match data by examining only the bucket identified by the hash function, but cannot easily scan over a subset of the table. For multi-column indexes, query filters must match all of the index columns to be able to take advantage of the index. Due to this inflexibility, we discourage the use of hash indexes. They should only be used when there is a demonstrated need and measurable benefit on your particular dataset and workload. Another consideration when choosing an index is the overhead of adding another index. Each added index uses extra memory for the additional data structures – on average about 40 bytes per row – and slightly slows inserts due to the additional data structures that need to be updated. Each rowstore table may have at most one primary key and optionally many secondary keys. Scans on the primary key are generally somewhat faster than on secondary keys. For example, if the data was inserted in primary key order, the rows would be in memory order for the primary key and have better cache locality for the primary key than for a secondary key. For more information, refer to the following two resources: * [The Story Behind SingleStore’s Skiplist Indexes](https://www.singlestore.com/blog/what-is-skiplist-why-skiplist-index-for-memsql/) * [Indexes for filters](https://docs.singlestore.com/cloud/query-data/query-tuning/#UUID-751c4d8c-49b4-edcb-9638-9e0af5b1d807.md) ## Check Your Understanding **Q**: For the table, `CREATE ROWSTORE TABLE t(a INT, b INT, KEY (a, b))`, will the query `SELECT SUM(a) FROM t WHERE b = 3` benefit from the index? **A**: No, since the only column in the filter list, `b`, is not a prefix of the key `(a, b)`, the query cannot benefit from the index. The query `SELECT SUM(a) FROM t WHERE a = 3` would be able to benefit from the index since `a` is a prefix of the key `(a, b)`. *** Modified at: February 22, 2023 Source: [/cloud/create-a-database/optimizing-table-data-structures/choosing-rowstore-keys/](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-rowstore-keys/) (An index of the documentation is available at /llms.txt) --- # Key Considerations for Understanding Your Workload Before attempting any schema design, it is critical to understand the nature of the workload. Ask yourself the following questions before getting started: * Is data loaded in trickles, large batches, or concurrent inserts? Is data frequently updated? See the [Load Data](https://docs.singlestore.com/cloud/load-data.md) and [Concurrent Multi-Insert Examples](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples.md) pages to learn more. * Is data ingest speed more important than query performance? See the [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) page for more information. * Are the queries mostly working with a small subset of rows related to the entire dataset (probably 0.1% or less)? Are the queries dealing with the entire dataset or a big subset of the data? See the [Use the Workload Manager](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md) page to learn more. * Which tables do you tend to join and what columns do you use to join them? See the [Query Performance Tools](https://docs.singlestore.com/cloud/query-data/query-tuning/query-performance-tools.md) page for more information on how to improve join performance. * What columns do you tend to use in filters? See the [Query Performance Tools](https://docs.singlestore.com/cloud/query-data/query-tuning/query-performance-tools.md) page for more information about how to improve query filtering performance. *** Modified at: February 6, 2025 Source: [/cloud/create-a-database/optimizing-table-data-structures/key-considerations-for-understanding-your-workload/](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/key-considerations-for-understanding-your-workload/) (An index of the documentation is available at /llms.txt) --- # Shard Keys The second consideration when putting data into SingleStore Helios is choosing the shard key for a table. Data is distributed across the SingleStore Helios cluster into a number of partitions on the leaf nodes. The shard key is a collection of the columns in a table that are used to control how the rows of that table are distributed. To determine the partition responsible for a given row, SingleStore Helios computes a hash from all the columns in the shard key to the partition ID. Therefore, rows with the same shard key will reside on the same partition. For example, the table below has the shard key that contains only the `first` column. All people with the same first name will be stored on the same partition. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4890eb99becb9d69/6a7ada675967e554445d1ded/16083b57680b3f-Sk7ZZm.png) ## Check Your Understanding **Q**: Why are there restrictions on `ALTER`ing or `UPDATE`ing the shard key of a table? **A**: Unlike adding a new index to a rowstore table (which only requires adding information to existing data without moving it), `ALTER`ing the shard key requires moving the table to different partitions. Similarly, `UPDATE`ing a column in the shard key could require a row to be moved to a different partition. **Q**: Why is the shard key required to be a subset of the primary key for a rowstore table? **A**: The primary key of a rowstore table requires that there are no duplicate rows. We can only enforce uniqueness by ensuring that all potentially duplicate data is on the same partition. ## Choosing a Shard Key Shard keys are specified at the time of `CREATE TABLE`. Once created, the shard key for a table cannot be modified. There are two competing considerations when sharding data: * Distributing data evenly across partitions * Partitioning data on columns you frequently filter or join First, distributing data evenly makes capacity planning much easier since the system will use its resources uniformly. Query performance can also suffer when the data is skewed or unevenly distributed, because there can be slow partitions. Multi-partition queries cannot run faster than the slowest partition involved. Second, queries perform significantly better and use fewer resources when the optimizer can precisely understand what partitions are used. SingleStore Helios only has to involve a single partition if the query filter matches the shard key, which greatly reduces cluster resource usage for high concurrency workloads. Similarly, joins of multiple tables that have the same shard key will be performed locally on the leaf partition rather than redistributing the data around the cluster, resulting in much faster joins that can be done at significantly higher concurrency. Joins of tables with different shard keys, or distributed joins, are substantially slower and more resource-intensive. Here, the order of the shard keys between one or more tables being joined must be the same in order to see this increase in performance. The only other affect of shard order is that rowstore [skiplist indexes](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md) are created on the same columns chosen for the shard key. However, this order can always be redefined. For example, the following query is considered “single partition” because the filter clause (`where first = ‘john'`) includes the columns of the shard key. The aggregator only needs to talk to one partition to get the data. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9448e5a19cdd5a34/6a7ada7973d9bda862292478/16083b57688791-ZJ01CX.png) For more about identifying and understanding the performance implications of single partition queries (and more), see the [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) guide. ## Skewed Partitions A shard key should ensure that the rows are (almost) evenly distributed across partitions. The two concerns specified above, typically, run into conflict when the data is skewed. For example, `first` is probably a poor shard key in the table above, because some first names are much more common than others. In cases like these, it is generally more important that the data be evenly distributed so that the storage and query processing requirements are balanced across the cluster. If the partitions are unbalanced, some partitions may face significantly higher workload than others for distributed queries. These partitions may become the limiting factor on parallelized operations. Hence, you should choose shard keys carefully to avoid problematic skewed partitions. In the example above, `user` is probably a better choice of shard key. It will be more evenly distributed, and it is likely to be a filter for queries against the table. If you want a table to get truly uniform sharding, you may also shard on an auto-increment column. ## Cardinality A shard key reduces the cardinality across the set of columns it contains. `GROUP BY`s execute more efficiently when they match the shard keys, because it reduces the amount of data transfer that is required, which subsequently allows more efficient computations. ## Joins If the join conditions match the shard keys, colocated data reduces the computational needs for join operations. Hence, ensure that the join conditions match shard keys as much as possible. ## Single-Partition Queries Queries which perform an equality filter on a shard key value can be executed against a single partition, instead of fanning out to all the partitions. ## More Ways to Distribute Data In addition to specifying a key the system will use to shard the table, you may also distribute data as follows: * Keyless sharding is the default for tables with no primary key and no explicit shard key. You can explicitly declare a table as keyless sharded by specifying a shard key with an empty list of columns: `shard key () ...` With keyless sharding, data is distributed uniformly across partitions in most cases. Cases when the data becomes non uniform typically occur because `INSERT … SELECT` statements are optimized to insert locally into the same partition. This makes these operations substantially faster, but it does mean that they can cause skew where there wasn’t before or amplify existing skew. To remove skew in these cases, you can force the `INSERT … SELECT` to redistribute the data with the `force_random_reshuffle` query hint. See more about this in the [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) reference topic. Keyless sharding also makes it impossible to do single partition queries or local (collocated) joins, because rows are not associated with specific partitions via an explicit key value. The following are examples of `CREATE TABLE` statements which will cause keyless sharding to be used. ```sql CREATE TABLE t1(a INT, b INT); CREATE TABLE t1(a INT, b INT, SHARD KEY()); ``` * Data in reference tables is duplicated on every node in the cluster, including aggregators. This makes it possible to do local joins between reference tables and any other tables; however, the table consumes a large amount of cluster storage because a copy of the entire table is stored on every node. Furthermore, reference tables replicate synchronously to the aggregators and asynchronously to the leaves, dramatically limiting the performance of writes. For star schema workloads, reference tables are ideal for small, slowly changing dimension tables. ## Questions to Ask When Choosing a Shard Key * Does this table have a primary key? *Make sure there is a shard key and that it is a subset of the primary key.* * Do you frequently join on a specific set of columns (e.g. `WHERE users.id = action.user_id and users.country = action.country`)? *Try to make the shard key a subset of the joined columns. Shard order should be the same between tables to accomplish this.* * Do you frequently filter on a specific set of columns (e.g. `WHERE user_id = 17 and date = '2007-06-14'`)? Try to make the shard key a subset of the filtered columns. > **📝 Note**: Filtering on a range of the shard key (e.g. `WHERE user_id BETWEEN 2 and 12`) can negatively affect the performance of your query. Multiple partitions may need to be scanned, thereby slowing the retrieval of your result set. * Do you have high concurrency queries? *Choose a shard key that allows these queries to be single partition.* * Is your data skewed for your current choice of shard key? *Try to add additional columns to the shard key to ensure even distribution.* * Do you need to `UPDATE` or `ALTER` any fields in the shard key? *Remove those fields from the shard key.* * Is this a small, infrequently changing table that needs to be on every node in the cluster to guarantee local joins? *Use a reference table instead of sharding the table.* In addition to these questions, see the [Detecting and Resolving Data Skew](https://docs.singlestore.com/cloud/create-a-database/detecting-and-resolving-data-skew.md) topic for more information. ## Check Your Understanding **Q**: Why can the SingleStore Spark Connector load data directly into the leaf partitions for keyless sharding only? **A**: If there is keyless sharding, data can be placed anywhere in the cluster. Since the leaf partitions are fully functional databases, the spark connector can connect directly to load the data in parallel. If a shard key is used, the connector must use an aggregator to ensure that data is inserted into the correct partition. **Q**: Why does SingleStore Helios use the primary key as a shard key if there is no explicit shard key? **A**: It is very common to query tables with a filter on the primary key; in addition, the uniqueness constraint of primary keys guarantees that data will not be skewed. **Q**: Why is it generally better to use a shard key with the fewest number of columns, as long as it’s not skewed? **A**: We can only optimize queries to run against a single partition when they filter or join on the entire shard key. A smaller shard key is likely to match more queries. **Q**: Why is it faster to run queries that group by the shard key? **A**: When the grouping matches the shard key, queries are faster because the `GROUP BY` can be executed locally on each partition. This effect is much more pronounced when the groups are high-cardinality. ## Related Topics * [Sharding](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md) * [Understanding Shard Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection.md) *** Modified at: September 16, 2025 Source: [/cloud/create-a-database/optimizing-table-data-structures/shard-keys/](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/shard-keys/) (An index of the documentation is available at /llms.txt) --- # Other Schema Concepts ## Reference Database When a user creates a database, two transaction logs are created for each partition of that database, in addition one special transaction log is created called the **reference database**. This database holds metadata about the whole database (for example, schemas of tables and stored procedures) as well as the data for special tables called reference tables. The reference database is replicated to each node in a cluster because every node needs this metadata. Further, reference tables take advantage of this database to implement joins that do not need to go over the network. The reference database has its master on the master aggregator, and only the master aggregator can write to it (this is why only the MA can execute DDL queries). ## Reference Tables **Reference tables** are relatively small tables that do not need to be distributed and are present on every node in the cluster. Reference tables are implemented via primary-secondary replication to every node in the cluster from the master aggregator to the leaves asynchronously. Replication enables reference tables to be dynamic: updates that you perform to a reference table on the master aggregator are quickly reflected on every machine in the cluster. Since reference tables are replicated to every node in the cluster, it eliminates the need to transfer the table’s data across the network during query execution. Reference tables should mostly be used for tables that change rarely, because the write operations on reference tables consume a lot more resources. SingleStore Helios aggregators can take advantage of reference tables’ ubiquity by pushing joins between reference tables and a distributed table onto the leaves. Imagine you have a distributed `clicks` table storing billions of records and a smaller `customers` table with just a few million records. Since the `customers` table is small, it can be replicated on every node in the cluster. If you run a join between the `clicks` table and the `customers` table, then the bulk of the work for the join will occur on the leaves. > **❗ Important**: Reference tables are a convenient way to implement **dimension tables**. **Best Practices for using Reference Tables** * Do not use reference tables to store a large number of rows, e.g. more than a few million rows. * If your workload requires intensive searches for individual rows (say, for OLTP-type operations), use a rowstore reference table. * Use columnstore reference tables to store dimension tables for a data warehouse type schema, with a typical star-join query and non-intensive updates. * Reference tables must have explicit primary keys. ## DDL Forwarding for Reference Tables Data Definition Language (DDL) commands for reference tables can be run from the master aggregator or a child aggregator node. DDL commands run on a child aggregator are forwarded to the master aggregator and propagated from there. (See [Cluster Management Commands](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands.md) for more information.) ## DML Forwarding for Reference Tables Data Manipulation Language (DML) commands for reference tables can be run from the master aggregator or a child aggregator node, however, with some exceptions. The following DML commands must be run directly on the master aggregator; they will fail with an error when run on a child aggregator. * The `LOAD DATA` command * Query profiling commands such as `PROFILE` and `SHOW PROFILE` if the `sync_permissions`[engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) is set to `ON`. * DML statements used in stored procedures * DML queries that reference temporary tables * DML queries in multi-statement transactions that are enclosed within BEGIN and COMMIT transaction statements. DDL commands run on a child aggregator are forwarded to the master aggregator and propagated from there. (See [Cluster Management Commands](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands.md) for more information.) ## Temporary Tables Temporary tables can be created as Columnstore or Rowstore tables. For more information, see [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md). Rowstore temporary tables are created in memory. Temporary tables exist for the duration of a client session. This means they are scoped to the connection that opened them, cannot be queried by other users, and are dropped once the connection has ended. They can also be dropped manually without removing the connection. SingleStore Helios does not write logs or take snapshots of temporary tables. Temporary tables are designed for temporary, intermediate computations. Since temporary tables are neither persisted nor replicated in SingleStore Helios, they have high availability disabled. This means that if a node in a cluster goes down and a failover occurs, all the temporary tables on the cluster lose data. Whenever a query references a temporary table after a node failover, it returns an error. For example, ``` "Temporary table is missing on leaf due to failover. The table will need to be dropped and recreated." ``` > **❗ Important**: To prevent loss of data on node failover, use SingleStore Helios tables that have high availability enabled. Views cannot reference temporary tables because temporary tables only exist for the duration of a client session. Although SingleStore Helios does not materialize views, views are available as a service for all clients, and so cannot depend on client session-specific temporary tables. `CREATE TEMPORARY TABLE` can be run on any aggregator, not just the master aggregator. Temporary tables are sharded tables, and can be modified and queried like any “permanent” table, including distributed joins. ## Global Temporary Tables Another type of temporary table is the global temporary table. Like temporary tables, they are not persisted. > **📝 Note**: Global temporary tables are not supported for columnstore. Unlike temporary tables, global temporary tables are shared across sessions, i.e. they exist beyond the duration of a session. They are never automatically dropped and must always be dropped manually. They can also be queried by other users since they are not session dependent. Global temporary tables can be used as a temporary space for processing data that can be accessed from multiple connections, for example, performing ETL (Extract, Transform, and Load) operations on modern dynamic databases. If failover occurs, global temporary tables lose data and enter an errored state; they need to be dropped and recreated. However, dropping a global or non-global temporary table does not drop its plancache from the disk and retains the cache if the table is recreated with the same schema. DDL such as `CREATE ROWSTORE GLOBAL TEMPORARY TABLE` and DML for global temporary tables can be run on a master, or a child aggregator node. **Note:** Both global temporary tables and non-global temporary tables cannot be altered. ## Types of Indexes SingleStore Helios, like all standard SQL databases, allows the creation of indexes on tables, which speed up certain access patterns. As with MySQL, SingleStore Helios supports two index type keywords. The BTREE keyword is for compatibility with MySQL syntax and will create the default SingleStore Helios skiplist index instead. The sort key index type is for [columnstore tables](https://docs.singlestore.com/cloud/create-a-database/columnstore.md). > **📝 Note**: `KEY() USING CLUSTERED COLUMNSTORE` is a legacy syntax that is equivalent to `SORT KEY()`. SingleStore recommends using `SORT KEY()`. ## Skiplist Indexes The default index type in SingleStore Helios is a skiplist. Skiplists in SingleStore Helios are meant to replace the B-Tree indexes used by most other databases, including MySQL. Skiplists are optimized to run in memory as they can be implemented lock free and offer extremely fast insert performance. Like B-Trees, they offer an expected O(log(n)) lookup performance and can be traversed in sorted order. Unlike B-Trees in MySQL, skiplists in SingleStore Helios are uni-directional (singly linked). Each column in a compound skiplist index can be specified as ascending (`ASC`) or descending (`DESC`). The default is `ASC`. Which one you pick will not impact lookup performance, but it does impact scan performance depending on the direction the index is scanned. Scanning a skiplist in reverse order is approximately twice as costly as scanning in forward order. So, if you have an `ASC` index and you run a query that would traverse the index in descending order (`ORDER BY DESC` for example), then the query will require a more expensive iteration than if the index were `DESC`. SingleStore Helios supports skiplists only on rowstore tables. For more information on what a skiplist is and why it is used in SingleStore Helios see: [The Story Behind SingleStore’s Skiplist Indexes](https://www.singlestore.com/blog/what-is-skiplist-why-skiplist-index-for-memsql/) ## Columnstore Indexes Columnstore indexes leverage columnstore technology to efficiently store and retrieve large numbers of values from disk (using flash or SSD is recommended). Because columnstore indexes are backed by disk and don’t have the requirement that all data must fit in memory, they are typically very useful for analytical workloads. SingleStore Helios currently supports sort key indexes which, when added to a table, will make the entire table structure backed by the columnstore. Currently columnstore indexes cannot be combined with in-memory row store indexes on the same table. For more information about using columnstore indexes in SingleStore Helios, see [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md). ## Hash Indexes Due to the restrictive case detailed above, `HASH` indexes should only be used when there is a demonstrated need and measurable benefit on your particular dataset and workload. In these specific cases, `HASH` indexes provide fast exact-match access to unique values. This is because the hash index is stored in a sparse array of buckets indexed by a hash function on the relevant columns eg: `hash(column_a, column_b)`. Queries can quickly find exact match data by examining only the bucket identified by the hash function. However they cannot easily scan over a subset of the index. For multi-column indexes, query filters must match all of the index columns to be able to take advantage of the index. SingleStore Helios supports `HASH` indexes on both columnstore and rowstore tables. For more information, see [USING HASH behavior](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#UUID-2d098a7e-1811-6223-8748-d20d3c8b53f6.md) and [Highly Selective Joins](https://docs.singlestore.com/cloud/create-a-database/columnstore/highly-selective-joins.md). > **⚠️ Warning**: For rowstore tables, a `HASH` index will only be utilized if the query filters on all the columns in a multi-column `HASH` index.For columnstore tables, a `HASH` index will be utilized on all or any of the columns in a multi-column `HASH` index. Consider an example table: ```sql CREATE TABLE t(a int, b int, INDEX(a, b) USING HASH); ``` Suppose we are running queries like: ```sql SELECT * FROM t WHERE a < 3; ``` `EXPLAIN` shows us that since we are performing a range scan and not filtering on all the columns in our hash index, a full Table Scan is performed. ```sql EXPLAIN SELECT * FROM t WHERE a < 3; ``` ```output +------------------------------------------------+ | EXPLAIN | +------------------------------------------------+ | Gather partitions:all | | Project [t.a, t.b] | | Filter [t.a < 3] | | TableScan db.t | +------------------------------------------------+ ``` The hash index is only utilized if the query uses only equality predicates, and filters on all columns in the hash index. ```sql EXPLAIN SELECT * FROM t WHERE a=3 AND b=7; ``` ```output +--------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------+ | Gather partitions:all | | Project [t.a, t.b] | | IndexRangeScan db.t, KEY a (a, b) USING HASH storage:lf_hashtable scan:[a = 3 AND b = 7] | +--------------------------------------------------------------------------------------------------------------------------+ ``` > **❗ Important**: A query on a columnstore table that employs an equality filter on a column that has a hash index defined on that column will always use the hash index. For this reason, if the filter is very unselective (returns many records), the hash index should not be used. ## Runtime Plan Choosing SingleStore Helios can dynamically select which index to use for a query at runtime. Instead of collecting statistics and building histograms, SingleStore Helios can compute these statistics for a given query on-demand by inspecting its indexes. If a query can match more than one index, SingleStore Helios compiles an execution plan for each choice, along with the necessary expression logic to cheaply analyze and evaluate which plan to choose at runtime. This process eliminates the need to manually recompute statistics on indexes. ## Index Hints SingleStore Helios supports the following index hint syntax: ```sql tbl_name [index_hint] index_hint: USE {INDEX | KEY} (index_list) | IGNORE {INDEX | KEY} (index_list) | FORCE {INDEX | KEY} (index_list) index_list: index_name [, index_name] ... ``` For example: ``` SELECT * FROM t1 USE INDEX(idx_name); --OR SELECT * FROM t1 FORCE INDEX(idx_name); ``` * `USE` and `FORCE` hints force the use of one of the specified indexes to run the query. In SingleStore Helios, there is no difference between a `USE` and `FORCE` hint. * `IGNORE` hints disallow the specified indexes from being used to run the query. The `EXPLAIN ` statement can be used to show which indexes the query considers and which one it will actually use. ## Index Commands * [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) * [DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-index.md) * [SHOW INDEX, SHOW INDEXES, SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) * [SHOW INDEXES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) * [SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) * [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) ## Related Commands An index may also be specified when creating or altering tables. * [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) * [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) ## Related Topics * Training: [SingleStore Helios Indexes](https://training.singlestore.com/learn/course/internal/view/elearning/616/memsql-indexes) *** Modified at: September 13, 2024 Source: [/cloud/create-a-database/other-schema-concepts/](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts/) (An index of the documentation is available at /llms.txt) --- # Rowstore A rowstore in SingleStore refers to a table storage format that stores data in rows, with all the fields of a row stored together in the same physical location. Rowstores are entirely in-memory, meaning they store data in the main memory (RAM) rather than on disk. This avoids disk I/O and significantly speeds up data access and manipulation. Rowstores are ideal for Online Transaction Processing (OLTP) and hybrid transactional/analytical (HTAP) applications that involve frequent point lookups, small inserts, updates, and deletes. Rowstore indexes in SingleStore are of two types - lockfree skiplists and hash tables. Skiplists maintain rows in sorted order to provide fast key lookups and range filters. Hash tables use a hash of the indexed columns to enable efficient exact-match queries. Both structures are optimized for in-memory access and enable high concurrency without traditional locking. Key advantages of rowstore tables include: * Rapid inserts and updates, even for large datasets * Fast retrieval of specific rows, leading to an improved random seek performance * Support for multiple indexes providing flexible query patterns The total combined size of all rowstore tables is limited by the total available RAM (A computer's short-term memory. It is where the data that the processor is currently using is stored temporarily. RAM can be accessed much faster than data on a hard disk, solid-state disk, or another long-term storage device, which is why RAM capacity is so important for system performance.) on the leaf nodes in the cluster. It is important to reserve a reasonable amount of RAM, say 20%, for query execution. So the effective total rowstore table capacity will be reduced by this amount. > **📝 Note**: Rowstore is no longer the default table storage format. As of version 7.3 of SingleStore, Columnstore is the default table storage format. This section provides a conceptual overview of SingleStore Helios’s rowstore and includes considerations for optimizing your database performance using the rowstore. ## Creating a Rowstore Table Typically you will specify a shard key and one or more indexes for a rowstore, although a shard key and indexes are optional. A primary key, which is enforced to be unique, is also supported. On rowstore tables, when the shard key is not the same column as the primary key, a skiplist index (A data structure optimized for ordered data that allows for queries to quickly seek data by binary searching.) will also implicitly be created on the shard key columns. > **❗ Important**: SingleStore supports both ordered (skiplist) and hash indices (A data structure optimized for fast equality lookups by a key.) on rowstore tables. Here is an example of a statement that creates a rowstore table: ```sql CREATE ROWSTORE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, dt DATETIME, KEY (Price), SHARD KEY (ProductId) ); ``` We define a `SHARD KEY` to explicitly control the data distribution. We define it on `ProductId` since sharding on a high cardinality column or columns generally allows for a more even distribution and prevents skew. The KEY specified on `Price` causes an index to be created on the `Price` column. It is also possible to randomly distribute data by either omitting the shard key, or defining an empty shard key `SHARD KEY()`, as long as no primary key is defined. ## Rowstore keys Rowstores with multiple indexes support very fast seeking to find one or a small number of rows via several different keys, or “access paths.” SingleStore Helios rowstores are able to provide extremely fast lookup along multiple different access paths with low variance in response time from query to query. The following statement creates a `products` table with keys on both `Price` and `Color`, and a unique (primary) key on `ProductId`. ```sql CREATE ROWSTORE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, dt DATETIME, KEY (Price), KEY (Color), PRIMARY KEY(ProductId), SHARD KEY (ProductId) ); ``` Primary keys must contain all the columns in the shard key so they can be enforced efficiently by looking at data in only a single shard (partition). Creating a primary key without specifying a shard key automatically shards on the primary key. You can also create a key by creating an index on a rowstore using the [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) statement. ## Rowstore persistence Rowstore data is fully persistent and updates on rowstore tables are done in transactions. Persistence is implemented for rowstores using periodic snapshots of the in-memory data, and a write-ahead log, which are both stored on a file system to make them permanent. If a SingleStore Helios node restarts, all its rowstore data will be recovered from the snapshots and log and the in-memory state for the rowstore will be rebuilt. ## Garbage collection Unneeded columnstore segments stored in memory are eliminated in a process called garbage collection. A buffer manager maintains unused memory blocks for future use until no query or operation is using the unused memory anymore. The garbage collection process automatically runs in the background. ## Porting applications to SingleStore Helios rowstores If you are porting an application from another database that uses a disk-based rowstore structure based on B-tree, ISAM or a similar access method, you should start by using a SingleStore Helios rowstore with a key for each index on the original system. If the original system had a “heap” access method (unordered rowstore), you can start with a SingleStore Helios rowstore with no key. ## OLTP operations on large data sets For OLTP operations on large tables, the total cost of provisioning enough RAM, and possibly enough servers, when using rowstores, can become a significant concern for some users. If this is the case, and the application doesn’t need the very fastest row lookup time, consider using columnstore tables with hash indexes on the most frequently-used lookup key columns instead. Or, for wide tables with many null values, the total cost of ownership (TCO) can be reduced by using rowstore tables with [SPARSE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) data compression. ## Related content See [The Story Behind SingleStore’s Skiplist Indexes](https://www.singlestore.com/blog/what-is-skiplist-why-skiplist-index-for-memsql) for details about how standard rowstore table indexes are implemented in SingleStore Helios. ## Rowstore Sizing Estimations In order to estimate the total memory used for a rowstore table, we simply need to estimate the amount of memory used for each row, and then multiply that by the number of rows in the table. ## Memory Use Per Row * **data**: memory use of each column in the table, which can be found by utilizing the size per each data type listed in [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md). * Be sure to consider changes in size that result from a column being nullable. * **metadata overhead**: 24 bytes * **index(es)**: * **40 bytes** per skiplist index * Skiplist indexes are created on rowstore tables by default in the following situations: if no primary key is specified, with the primary key unless otherwise specified, and when added explicitly after the table has already been been created. * **32 bytes** per hash index * Hash indexes are created on Rowstore tables in the following situations: with the primary key when specified, and when added explicitly after the table has already been created. * For more information about keys and indexes see [Understanding Keys and Indexes in SingleStore](https://docs.singlestore.com/cloud/create-a-database/understanding-keys-and-indexes-in-singlestore.md). ## Memory Use Per Table Based on the memory use per row, memory use for Rowstore tables has the following base formula: `(data + 64 bytes + (index size * number of secondary indexes)) * number of rows` `data` is the data size based on number of columns and their respective [Data Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md), `64 bytes` is the metadata overhead + the primary key index (or the index created in its place where one isn't specified), `index size` is the size of the index type of any additional indexes on the table, `number of secondary indexes` is the number of additional indexes, and `number of rows` is the total number of rows in a given table. ## Additional Sizing Estimates for Rowstore Table Overhead * Additional indexes and columns cause overhead to increase. The amount of increase depends on index and data type. * Every partition in a database will consume the same amount of table overhead. For example, if a leaf node has 8 partitions, and a table has a constant overhead of 32 MB, that lead will consume at least 256 MB of memory for that table. * Per row, a table with `SPARSE` compression enabled consumes 4 bytes per every 8 `SPARSE` columns, plus an additional 8 bytes.  * Deleted rows consume an additional 40 bytes, plus 8 bytes per index until they are physically deleted by the engine. * `DELETE` queries mark rows as deleted; they don't physically delete them right away. The row is removed at a later time when garbage collection runs, and the version can be safely deleted. Row garbage collection runs in the background, however [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) can be used to force it to run immediately. Note that `OPTIMIZE TABLE` can result in a performance impact and should be run with caution. *** Modified at: November 18, 2025 Source: [/cloud/create-a-database/rowstore/](https://docs.singlestore.com/cloud/create-a-database/rowstore/) (An index of the documentation is available at /llms.txt) --- # Specifying Unenforced Unique Constraints You can specify the `UNENFORCED` option on a `UNIQUE` constraint to disable enforcement of the constraint. This means that SingleStore does not prevent you from inserting duplicate rows, and cannot guarantee that the constraint is true. An unenforced unique constraint is informational: the query planner may use the unenforced unique constraint as a hint to choose better query plans. Unenforced unique constraints are useful on columnstore tables since on a columnstore table, an unenforced unique constraint is a logical-only index - there is no physical storage for the index. SingleStore supports unique constraints on both single-column and multiple-column keys in a columnstore reference table. Adding or altering a unique index is not supported on existing columnstore tables. On a rowstore table, an unenforced unique constraint is stored physically as a regular non-unique index. The `RELY` and `NORELY` options specify how the query planner may use the unenforced unique constraint. These options determine whether or not we depend on the index-provided constraint for ensuring correctness in execution. The `NORELY` option, which is the default, specifies that query semantics should not rely on the unenforced uniqueness constraint - queries will always return correct results regardless of whether the unenforced unique constraint is true or not. SingleStore may use a `NORELY` unenforced unique constraint to inform statistical estimates. The query planner assumes the number of unique values in a table is equal to the row count of a table. The `RELY` option specifies that SingleStore may choose query plans which assume that the unenforced unique constraint is true - queries are only guaranteed to return correct results if the constraint is true, and queries may return incorrect results if there are duplicate rows. You should only use the `RELY` option if you know the constraint is valid, e.g. if some process in your application enforces the constraint. SingleStore may use a `RELY` unenforced unique constraint to inform statistical estimates as well as eliminate redundant joins, for example. ## Syntax To specify a unique constraint as unenforced, add the `UNENFORCED` option to the index declaration, for example like: ```sql UNIQUE KEY (col1, col2) UNENFORCED ``` The syntax for the `UNENFORCED` clause is ```sql UNENFORCED [RELY | NORELY] ``` If neither `RELY` nor `NORELY` is specified, the default is `NORELY`. The `UNENFORCED` clause is an optional index\_option clause in the unique index declaration syntax found in [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md), [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md), or [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md). For example, as part of a `CREATE TABLE` statement: ```sql CREATE TABLE t (id INT, SORT KEY (id), UNIQUE KEY (id) UNENFORCED RELY); ``` ## Unenforced Foreign Key Constraints Foreign keys in SingleStore are column(s) in a table that establish a link or relationship with a `UNIQUE KEY` of another table. It defines the referential integrity between related tables, ensuring the values in the `FOREIGN KEY` column(s) have corresponding values in the `PRIMARY KEY` column of the referenced table. Foreign key enforcement is not supported, but `FOREIGN KEY` syntax can be supported in SingleStore by setting the `ignore_foreign_key` [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) to `ON`. The default value for `ignore_foreign_keys` is `OFF`. ## Syntax ```sql FOREIGN KEY [foreign_key_name](col1,..., coln) REFERENCES table_referenced (col_referenced)); ``` ## Example ```sql -- Referenced Table -- CREATE TABLE parent (pid INT PRIMARY KEY, col1 INT); -- Foreign Key Table -- CREATE TABLE child (cid INT PRIMARY KEY, pid INT, FOREIGN KEY fkey(pid) REFERENCES parent(pid)); ``` If you are migrating an existing application that already has foreign key syntax onto SingleStore, it can make the process easier to set `ignore_foreign_keys` to `ON`. ## Checking Foreign Keys in an Application or Stored Procedure Since SingleStore doesn't enforce foreign keys, if your application requires foreign key relationships to be maintained, you may wish to check foreign key relationships in application software. The most common use of a foreign key relationship is to ensure that a one-to-many relationship is maintained between two tables. For example, you may have a rule that any row in the `child` table must have a corresponding row in the `parent` table. You may be able to enforce this entirely on the client side (with no database queries needed) by making sure that all transactions update the database in such a way that every child has a parent. Or, you may wish to check the constraint by running database queries in your application before doing database operations that might violate it. For example, **before** inserting into the `child` table like: ```sql INSERT INTO child VALUES (100, 1); ``` your application can run a `SELECT` statement on the `parent` table: ```sql SELECT count(*) FROM parent WHERE pid = 1; ``` If the result is not `1`, your application can signal an error and not run the `INSERT` statement. Detecting a foreign key violation like this generally indicates an issue with the application logic that needs to be corrected. *** Modified at: February 28, 2024 Source: [/cloud/create-a-database/specifying-unenforced-unique-constraints/](https://docs.singlestore.com/cloud/create-a-database/specifying-unenforced-unique-constraints/) (An index of the documentation is available at /llms.txt) --- # Understanding How Datatype Can Affect Performance ## Comparing Mismatched Datatypes Using comparisons between mismatched data types may degrade query performance and may use unsafe type conversions which can yield undesirable query results. SingleStore will display a warning for queries with potentially problematic comparisons between mismatched data types in the `EXPLAIN` and `information_schema` records for the query. These warnings do not necessarily indicate a problem, and you may have valid reasons for comparing different data types. But these data type mismatches are flagged to help you identify potential problems that you may not be aware of. When you see these warnings, you should check whether the data types are expected to be different. You may wish to consider changing the data types of the fields or values involved. You may also wish to consider adding explicit type conversion operations, such as the [cast operators](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/cast-or-convert.md) or functions like [STR\_TO\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/str-to-date.md) which convert between different types. These warnings can be seen in `EXPLAIN `, `information_schema.plancache.plan_warnings`, `information_schema.mv_queries.plan_warnings`, and the `PlanWarnings` column of `show plancache` in textual form, as well as in `EXPLAIN JSON `, `information_schema.plancache.plan_info`, and `information_schema.mv_queries.plan_info` in JSON form. ## Example An example of these warnings is the following: ```sql CREATE TABLE t (id VARCHAR(50), PRIMARY KEY (id)); EXPLAIN SELECT * FROM t WHERE id = 123; ``` ```output +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | WARNING: Comparisons between mismatched datatypes which may involve unsafe datatype conversions and/or degrade performance. Consider changing the datatypes, or adding explicit typecasts. See https://docs.singlestore.com/docs/mismatched-datatypes for more information. | | | | WARNING: Comparison between mismatched datatypes: (`t`.`id` = 123). Types 'varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL' vs 'bigint(20) NOT NULL'. | | | | Gather partitions:all alias:remote_0 | | Project [t.id] | | Filter [t.id = 123] | | TableScan db.t, PRIMARY KEY (id) | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` In this example, the query involves the expression `t.i = 123`, where `t.i` is a string field. When a numeric value is compared to a string, the string is converted to a numeric value, e.g. the string `'123'` is converted to the number `123`. This comparison may be problematic for multiple reasons: * The mismatched data types may indicate a mistake in how the query is written or how the table is defined. The query may behave differently than intended and yield undesired results. For example, all of the strings `'123'`, `'123.0'`, `'0123'`, and `'123abc'` compare equal to the integer `123`, so while the query may be intended to retrieve a single row with the specified `id`, this equality comparison may yield multiple rows whose `id` values all compare equal to `123`. * The mismatched data types negatively impact the performance of the query. If the field and constant were either both strings or both integers, the query plan would be able to use the index to efficiently seek to lookup the matching `id`. But because there are many possible string `id`s that match the number `123`, which do not come in any particular order in terms of string lexicographic order, the query cannot seek into the index and instead must scan all the rows of the table. ## Data Type Conversion The global sync variable `data_conversion_compatibility_level` controls the way certain data conversions are performed. This variable can have the following possible values: '6.0', '6.5', '7.0','7.5', and '8.0'. Higher values introduce stricter checking of values and will error for conversions that worked at lower levels. For example, the `'7.0'` level will fail the conversion of `'abc'` to an integer value, while the `'6.0'` level will silently convert it to a NULL or 0. The following table lists the data type conversion checks and behavior changes that are introduced with each `data_conversion_compatibility_level` value. Note that the data type conversion checks introduced in a compatibility level are preserved in subsequent levels. | data\_conversion\_compatibility\_level | Data Type Conversion Checks | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `6.0` |
  • The lowest data type conversion compatibility level.
  • Default value for SingleStore engine versions lower than 8.0.
| | `6.5` |
  • Error on integer overflows and underflows in INSERT statements. This check applies to all INTEGER data types, including BIGINT, INT, MEDIUMINT, SMALLINT, TINYINT, BIT, and BOOL.
  • Error on string inputs to VARCHAR columns that are too short to store the data. This check is run against INSERT statements.
  • Error on string inputs to LONGBLOB, MEDIUMBLOB, BLOB, TINYBLOB, LONGTEXT, MEDIUMTEXT, TEXT, and TINYTEXT columns that are too short to store the data. This check is run against INSERT statements.
  • Error on invalid string to int conversion in INSERT statements. Only strings with valid characters will be converted to integers.
| | `7.0` |
  • Error on string inputs to CHAR columns that are too short to store the data. This check is run against INSERT statements.
  • Error on inputs to BINARY columns that are too short to store the data. This check is run against INSERT statements.
  • Error on decimal inputs that do not match the size specified in the DECIMAL column definition. This check is run against INSERT statements.
  • Error on invalid or out-of-range date inputs to DATE, DATETIME, DATETIME(6), and YEAR columns in INSERT statements.
  • Errors on zero value date inputs into DATE, DATETIME, and DATETIME(6) columns in INSERT statements.
  • Arithmetic operations involving DATE and INT values are performed by converting the INT data into intervals (days) instead of converting both data types to DOUBLE.
| | `7.5` |
  • Error on invalid string to DECIMAL conversion in INSERT statements. Only strings with valid characters undergo numeric conversion.
| | `8.0` |
  • Errors on conversions from STRING to DATE like data types instead of generating NULL or 0.
  • Default value beginning in version 8.0.
| | `8.1` |
  • Errors on the invalid or out-of-range timestamp inputs to TIMESTAMP and TIMESTAMP(6) columns in INSERT statements.
| ***It’s recommended to set the `data_conversion_compatibility_level` variable to the highest available value for new application development.*** SingleStore supports in-place change (An in-place change in a database refers to modifying the data or schema of the database without requiring a full data migration or recreation of the database. It allows you to make changes to the database while minimizing downtime and preserving existing data.) of `data_conversion_compatibility_level` for versions 8.0.24 and newer. This allows you to change your `data_conversion_compatibility_level` even if one of your shard keys is a persisted computed column. Prior to this, shard keys that were persisted computed columns could become logically corrupted when changing the `data_conversion_compatibility_level`. For existing applications, it’s also recommended to use the highest available level, but it’s recommended that you test your application before deploying this change. In particular, changing the value of `data_conversion_compatibility_level` can change the behavior of expressions in computed columns. If a computed column value changes due to a change in `data_conversion_compatibility_level`; columnstore sorting, indexes, and sharding can become logically corrupted. SingleStore does not recompile an existing plan when `data_conversion_compatibility_level` or `sql_mode` changes. > **📝 Note**: `sql_mode` is persisted to the `CREATE TABLE` statement. Therefore, the computed column of a table always uses the `sql_mode` that the table is created with, which may be different from the current `sql_mode`. If a persisted computed column is also a shard key and the `data_conversion_compatibility_level` has been changed, rows can move to a different partition. Updating the row will move it to the correct partition automatically, but some queries won’t be able to find the correct partition to fetch a row when they otherwise would. For example, see how expression evaluation may behave differently under the two levels: ```sql SET GLOBAL data_conversion_compatibility_level = '6.0'; SELECT CAST('2023-01-31' AS DATE) + 1; ``` ```output +--------------------------------+ | CAST('2023-01-31' AS DATE) + 1 | +--------------------------------+ | 20230132 | +--------------------------------+ ``` ```sql SET GLOBAL data_conversion_compatibility_level = '8.0'; SELECT CAST('2023-01-31' AS DATE) + 1; ``` ```output +--------------------------------+ | CAST('2023-01-31' AS DATE) + 1 | +--------------------------------+ | 20230201 | +--------------------------------+ ``` ## Finding Incorrect Computed Shard Key Values You can find all of the incorrect computed shard key values with a SQL query that searches for records where the computed columns are not equal to the value of their computed column expression. First, set the `allow_modifying_sysvars_with_computed_shard_key` engine variable to `ON`. This will allow you to change the `data_conversion_compatibility_level` on databases that contain tables with a persisted computed column as the shard key. Otherwise, an error would be generated. ```sql SET GLOBAL allow_modifying_sysvars_with_computed_shard_key = ON; ``` Consider a `dates` table where dates are inserted into the table as strings. Then those dates must be converted to the `DATE` format and 1 day is added to the inserted date. The persisted computed column is the shard key and the current `data_conversion_compatibility_level` is `6.0`. ```sql CREATE TABLE dates (date_str TEXT ,date_int AS CONVERT(CONVERT(date_str, DATE) + 1, SIGNED INT) PERSISTED INT ,SHARD KEY(date_int)); ``` ```sql INSERT dates VALUES('2023-01-31'), ('2023-01-30'); ``` ```sql SELECT * FROM dates; ``` ```output +------------+----------+ | date_str | date_int | +------------+----------+ | 2023-01-30 | 20230131 | | 2023-01-31 | 20230132 | +------------+----------+ ``` Next, follow these steps: 1. Set the `data_conversion_compatibility_level` to your desired version: 2. Compare the computed column with a `SELECT` statement with a `WHERE` filter that finds rows where the computed column is not equal to the value of its computed column expression. For example: ```sql SET GLOBAL data_conversion_compatibility_level = '8.0'; ``` > **📝 Note**: When creating a query to find values where the computed column changed, make sure that it is not using a shard key lookup, which might not find the desired row due to the change in shard key value.For example, the query used to search for changed computed columns uses a full columnstore scan, which you can verify using `EXPLAIN`:```sql > EXPLAIN SELECT * FROM dates > WHERE date_int <> CONVERT(CONVERT(date_str, DATE) + 1, > SIGNED INT); > > ``````output > > +----------------------------------------------------------------------------------------------+ > | EXPLAIN | > +----------------------------------------------------------------------------------------------+ > | Gather partitions:all alias:remote_0 parallelism_level:segment | > | Project [r.date_str, r.date_int] | > | Top limit:[?] | > | ColumnStoreFilter [r.date_int <> CAST(DATE_ADD(DATE(r.date_str), INTERVAL 1 DAY) AS SIGNED)] | > | ColumnStoreScan test1.r, SORT KEY __UNORDERED () table_type:sharded_columnstore | > +----------------------------------------------------------------------------------------------+ > 4 rows in set (0.00 sec) > > ``` 3. Once the affected rows have been identified, delete the row and re-insert it. * Create a temporary table where the data will be backed up: ```sql CREATE TEMPORARY TABLE tmp_dates LIKE dates; ``` * Insert the record(s) with the incorrect shard key value into the temporary table to back up the data. As previously explained, the value for `date_str` = ‘2023-01-31’ has been recomputed to a valid value: ```sql INSERT INTO tmp_dates (date_str) SELECT * FROM dates WHERE date_int <> CONVERT(CONVERT(date_str, DATE) + 1, SIGNED INT); ``` ```sql SELECT * FROM tmp_dates; ``` ```output +-------------+-----------+ | date_str | date_int | +-------------+-----------+ | 2023-01-31 | 20230201 | +-------------+-----------+ ``` * Using the query that finds rows where the computed column is not equal to the value of its computed column expression (#3), delete those rows from the temporary table: ```sql DELETE FROM dates WHERE date_int <> CONVERT(CONVERT(date_str, DATE) + 1, SIGNED INT); Query OK, 1 row affected (0.06 sec) Records: 1 Duplicates: 0 Warnings: 0 ``` ```sql SELECT * FROM dates; ``` ```output +------------+----------+ | date_str | date_int | +------------+----------+ | 2023-01-30 | 20230131 | +------------+----------+ ``` * Insert the data from the temporary table back into `dates` and confirm the `date_int` column is not valid under the new `data_conversion_compatibility_level`. ```sql INSERT INTO dates (date_str) SELECT date_str from tmp_dates; ``` ```sql SELECT * FROM dates; ``` ```output +------------+----------+ | date_str | date_int | +------------+----------+ | 2023-01-30 | 20230131 | | 2023-01-31 | 20230201 | +------------+----------+ ``` * Drop the temporary table: ```sql DROP TABLE tmp_dates; ``` 4. Lastly, set the `allow_modifying_sysvars_with_computed_shard_key` engine variable back to its default setting of `OFF`. The procedure above is one possible way to repair persisted computed column shard keys that may have changed after you change `data_conversion_compatibility_level`. Other ways are also possible–any update to a row will recompute a persisted computed column. However, setting a field value to itself may not update the row due to logic created to optimize updates by doing nothing when an update is known to have no effect. For example, if `t.A` is `5` in a row in table `t`, and you try to update `t.A` to `5`, the operation may not have any effect and not trigger recalculation of any persisted computed columns in the row. ## Handling invalid source database DATE and DATETIME values during Flow migrations When using SingleStore Flow to migrate data from supported source databases into SingleStore, the source data may contain non-standard or invalid DATE or DATETIME values such as `0001-01-01 07:07:24` or `0000-12-01`. If the `data_conversion_compatibility_level` is set to a stricter setting such as `8.1`, loading this data can fail with an error such as: ```sql Error(PGT54): Table core_data.`mage_order` stage 3 error java.sql.SQLException: (conn=142441) Leaf Error (node-83201ac7-c278-4902-8df7-ab7902e5adb3-leaf-ag2-0.svc-83201ac7-c278-4902-a1f2-600f50d9b580:3306): Invalid DATE/TIME in type conversion for column 'col_datetime' ``` To work around this issue without modifying the source data, the compatibility level can be temporarily set to an earlier version such as `6.5`: ```sql SET GLOBAL data_conversion_compatibility_level = '6.5'; ``` For example, the following executes successfully: ```sql SELECT @@data_conversion_compatibility_level; DROP TABLE IF EXISTS date_tbl; CREATE TABLE date_tbl ( id INT PRIMARY KEY NOT NULL, d DATETIME ); INSERT date_tbl VALUES (1, '0001-01-01 07:07:24.000000'); SELECT * FROM date_tbl; ``` > **❗ Important**: By default, SingleStore recommends using the highest available `data_conversion_compatibility_level` to enforce strict validation. Lowering it can allow non‑standard or invalid data to be inserted across the entire cluster. Use this workaround only in exceptional cases (for example, short-term Flow migrations that must preserve these values), and restore the highest supported compatibility level as soon as possible. ## Best Practices to Avoid Data Corruption * Review the computed column expressions when changing `data_conversion_compatibility_level`. * Perform application upgrade tests. For example, if a computed column contains corrupted data and you have to switch to a higher `data_conversion_compatibility_level`, you may need to drop the computed column before changing the level. Once the level is increased, add the computed column to the table. Alternatively, if a persisted computed column is used in a columnstore key, you may have to create a new table and select data into this new table. After copying the data, drop the old table and rename the new table. The following examples demonstrate how the behavior of expressions may change if `data_conversion_compatibility_level` is changed. **Example 1** ```sql SET GLOBAL data_conversion_compatibility_level = '6.0'; ``` ```sql SELECT DATE('2019-04-20') + 2; ``` ```output +------------------------+ | DATE('2019-04-20') + 2 | +------------------------+ | 20190422 | +------------------------+ ``` **Example 2** ```sql SET GLOBAL data_conversion_compatibility_level = '7.0'; ``` ```sql SELECT DATE('2019-04-20') + 2; ``` ```output +------------------------+ | DATE('2019-04-20') + 2 | +------------------------+ | 2019-04-22 | +------------------------+ ``` The following example shows the error message generated if a zero value is attempted to be inserted into a `DATE`, `DATETIME`, or `DATETIME(6)` column if the `data_conversion_compatibility_level` is set to `7.0` or higher. ```sql SET GLOBAL data_conversion_compatibility_level = '7.0'; ``` ```sql CREATE TABLE dates (dt DATETIME); INSERT INTO dates VALUES ('0000-00-00'); ``` ```output ERROR 2363 (HY000): Leaf Error: Invalid DATE/TIME in type conversion ``` See the table on the [Time and Date](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/time-and-date.md) page for examples of zero values for `DATE`, `DATETIME`, and `DATETIME(6)` data types. This page also discusses how date (and time) values are implicitly converted from string literals (e.g., '2023-02-03') and how to use date and time functions to perform explicit conversions from string literals. *** Modified at: October 18, 2021 Source: [/cloud/create-a-database/understanding-how-datatype-can-affect-performance/](https://docs.singlestore.com/cloud/create-a-database/understanding-how-datatype-can-affect-performance/) (An index of the documentation is available at /llms.txt) --- # Understanding Keys and Indexes in SingleStore This page seeks to explain behaviors of and interaction between keys and indexes in SingleStore. A key is a logical concept in relational databases, whereas an index is a physical artifact that assists with processing operations faster or enforcing uniqueness. For basic keys, there is a one to one relationship with indexes. If the syntax `KEY(...)` (alone), or `PRIMARY KEY(...)` is used in `CREATE TABLE` when defining a given key, an index is created. For all other types of keys that can be defined, an index is not automatically created. For more information about the implications of defining each type of key and/or index, see the below comparison table. In general, this documentation follows MySQL terminology when discussing keys and indexes. | **Key or Index** | **Function** | **Table Compatibility** | **Notes and Limitations** | **Column Compatibility** | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Shard key** | Responsible for distribution of data across partitions. Shard key columns should be as unique as possible. | A shard key (unique or non-unique) can be defined for either columnstore or rowstore tables. See[Sharding](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md)and[Choosing a Shard Key](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding/#section-idm4501210417348832654841869645.md)for more information. | If a shard and/or primary key is not defined for a given table, that table becomes keylessly sharded.If a primary key is defined, but a shard key is not, the primary key becomes the shard key.Only one primary key per table can be created.The shard key must be a subset of the columns in the primary key, if one is defined.Sharded tables do not support unique keys (unless the unique key contains all columns of the shard key).Sharded tables do not support`UPDATE`queries that modify the shard key.Sharded tables do not support`UPDATE...LIMIT`.To prevent an index on from being created on a shard key, use the`METADATA_ONLY`option when creating the shard key. This will decrease memory usage, but it can cause queries to run slower. | N/A | | **Primary key** | A group of 1 or more columns from a table that uniquely identifies a row in the table, and serves to prevent data skew. | A primary key can be defined for either columnstore or rowstore tables. See[Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md)for guidance on choosing a primary key. | The primary key must contain all columns in the shard key.`KEY(...)`is the same as`PRIMARY KEY(...),`except that the latter enforces uniqueness. | N/A | | **Columnstore sort key** | Stores data on disk in a columnstore format. | Columnstore sort keys may only be defined on columnstore tables. See[Choosing a Columnstore Key](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md)for more information. | A columnstore sort key can be created using the`SORT KEY()`syntax in a`CREATE TABLE`statement. When a key is specified, a sort key index is created, which defines the order in which rows are sorted in the columnstore table. The sort key can be specified as ascending or descending. When a key is not specified, no index is created and the columnstore data is stored unsorted. If you want to create an unsorted columnstore table, you can specify an empty key using`SORT KEY()`.`KEY() USING CLUSTERED COLUMNSTORE`is a legacy syntax that is equivalent to`SORT KEY()`.SingleStorerecommends using`SORT KEY()`. | N/A | | **Rowstore key** | Creates an index on a column or group of columns to support fast lookups, or uniqueness enforcement. | A rowstore key may only be defined on rowstore tables. See[Rowstore keys](https://docs.singlestore.com/cloud/create-a-database/rowstore.md)and[Choosing Rowstore Keys](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md)for more information. | The index created with this key can be defined as a skiplist or hash index. | N/A | | **Skiplist index** | A skiplist is a data structure optimized for ordered data that allows for queries to quickly seek data by binary searching. Skiplists inSingleStore Heliosare meant to replace the B-Tree indexes used by most other databases, including MySQL. | Skiplist indexes may only be defined on rowstore tables. See[Skip List Indexes](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md)for more information. | This is the default index type for rowstore tables, unless otherwise specified. | N/A | | **Hash index** | A hash table is a data structure optimized for fast equality lookups by a key. | A hash index can be defined for either columnstore or rowstore tables. See[Hash Table Indexes](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md)for more information and the syntax to define this type of index. | Only one unique hash index per table can be created, and one or multiple columns are allowed per hash index. | Cannot be created on columns with the following dataypes:`FLOAT`,`REAL`, or`DOUBLE` | | **Full-text index** | Full-text search allows searching for words or phrases in a large body of text through an inverted index. | Full-text indexes may only be defined on columnstore tables. See[Create Table with FULLTEXT Columns](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#UUID-5a7c66e9-736e-ce79-7a7e-1106691d9e5a.md)for the syntax to define this type of index. | N/A | Can only be created on columns with the following data types:`CHAR`,`VARCHAR`,`TEXT`, and`LONGTEXT` | | **Vector Index** | The vector index provides Approximate Nearest Neighbor (ANN) search which finds a set of k nearest neighbors very efficiently. | Vector indexes may only be defined on columnstore tables.Refer to[Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md)for the syntax to define this type of index | N/A | A vector index must be built on a single column of type`VECTOR (, [F32])`where``is the number of dimensions.The element type is restricted to`F32`. | | **Geospatial index** | A geospatial index can be defined on columns containing spatial data and serves to speed up queries on them. | Geospatial indexes may only be defined on rowstore tables. See[Working with Geospatial Features](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-geospatial-features.md)for the syntax to define this type of index. | N/A | Can only be created on columns with the following data types:`GEOGRAPHY`, and`GEOGRAPHYPOINT` | | **Multi-Value Hash Index** | Maps values to the row IDs containing the value. | Multi-value hash index is useful in searching for elements within composite data structures like arrays, BSON documents, etc.Refer to[Multi-Value Hash Index (BSON)](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/multi-value-hash-index-bson.md)or[Multi-Value Hash Index (JSON)](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json.md)for creating multi-key indexes on the respective column types. | N/A | Multi-value hash index is supported only on JSON and BSON type columns. | *** Modified at: October 1, 2025 Source: [/cloud/create-a-database/understanding-keys-and-indexes-in-singlestore/](https://docs.singlestore.com/cloud/create-a-database/understanding-keys-and-indexes-in-singlestore/) (An index of the documentation is available at /llms.txt) --- # Understanding Shard Key Selection ## Shard Key The shard key is a table column or multiple columns used to control how the rows of that table are distributed. Shard keys are vital in a distributed database like SingleStore. They are responsible for distribution of data across partitions. Shard key columns should be as unique as possible. To determine the partition responsible for a given row, SingleStore computes a hash from all the columns in the shard key and uses this hash to assign the partition ID. Therefore, rows with the same value for the shard key column will reside on the same partition. ## Data Distribution into Partitions Data is distributed amongst the partitions in the leaf nodes based on the shard key selected when a table is created. Rows with the same shard key value are stored in the same partition, even if they are in a different table. For example, the table below has the shard key that contains only the `first` column. All people with the same first name will be stored on the same partition. ```sql DROP TABLE IF EXISTS people; CREATE ROWSTORE TABLE people ( user VARCHAR(24), first VARCHAR(24), last VARCHAR(24), SHARD KEY(first) ); ``` Inside the node, values within the `first` column attempt to distribute evenly across all partitions. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt3045a4c089fd2d72/6a7adae126618c44d4842fdd/data_skew_illustration-a0qHct.jpeg) But using a first name column as the shard key can create data skew because some first names are more common than others. Data skew occurs when data is unevenly distributed across partitions. If `first` is used as the shard key, the data could be distributed amongst the partitions as follows. SingleStore computes a hash from all the columns in the shard key and uses this hash to assign the partition. Rows with the same value for the shard key column will reside on the same partition. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0a2f5d204db0e2b7/6a7adbaf96b5a6bacc86fdf2/Data_skew_diagram_only-AgP3ew.png) In the first partition, three of the rows have the same shard key value ("alex") so the rows would be assigned to the same partition. For the row with the shard key value of "tom", the hash computation just happened to assign this row to the first partition. This is the case with the other three partitions. The data in the `people` table is distributed unevenly across partitions resulting in data skew. If this table had millions of rows, the data skew could be even more pronounced. When data is skewed, some partitions will store more data than others, throwing off the balance of the cluster. Those partitions with more data than the average will probably be asked to do more work during `SELECT`, `UPDATE`, and `DELETE` queries. Since a query executes as fast as its slowest partition, data skew affects overall performance. Choosing an appropriate shard key for a table is important for minimizing data skew. In the `CREATE TABLE` example, the `user` column would be a better choice for the shard key, in terms of even data distribution. It will be more evenly distributed, and it is likely to be a filter for queries against the table. However, it is best **not** to use a varchar column as the shard key at all because of the inherent potential for data skew. Also, varchar data types take longer to process during query execution. Since `JOIN` operations are generally performed on the shard key, using a varchar data type as the shard key can impact performance. In our example, you would get optimal data distribution by creating an auto-increment `id` column as the shard key. ```sql DROP TABLE IF EXISTS people; CREATE ROWSTORE TABLE people ( id INT AUTO_INCREMENT PRIMARY KEY, user VARCHAR(24), first VARCHAR(24), last VARCHAR(24), SHARD KEY(id) ); ``` On a rowstore table, if a primary key is specified without a shard key, then the primary key will be used as the shard key. If the table doesn't have a primary key or a shard key, the engine will perform keyless sharding on that table. This means the engine generates a random number using a hash function, then maps the resulting random number to a partition. The `METADATA_ONLY` option on the `SHARD KEY` syntax prevents an index being created on the shard key. It will decrease overall memory usage. It can cause queries to run slower. It can only be used when creating your table. ## Resolving Data Skew SingleStore does not support the use of `ALTER TABLE` to add or change shard keys on existing tables. If your data becomes skewed, there are a couple of workarounds to reduce the skew in a table. One method to reduce skew in a table is by using the `INSERT INTO… SELECT` to redistribute the data with the `force_random_reshuffle` query hint. A common use case for this hint is on keyless sharded tables, as it allows forced redistribution of data among partitions, rather than inserting it locally into the same partition. Create a table without a specified shard key: ```sql DROP TABLE IF EXISTS people; CREATE ROWSTORE TABLE people ( id INT, user VARCHAR(24), first VARCHAR(24), last VARCHAR(24) ); ``` Create another table with one shard key: ```sql DROP TABLE IF EXISTS people_1; CREATE ROWSTORE TABLE people_1 ( id INT AUTO_INCREMENT, user VARCHAR(24), first VARCHAR(24), last VARCHAR(24), SHARD KEY(id) ); ``` Create another table with multiple shard keys: ```sql DROP TABLE IF EXISTS people; CREATE ROWSTORE TABLE people ( id INT AUTO_INCREMENT, user VARCHAR(24), first VARCHAR(24), last VARCHAR(24), SHARD KEY(id,user) ); ``` INSERT INTO...SELECT using force\_random\_reshuffle: ```sql INSERT INTO people_1 (id, user, first, last) SELECT WITH(force_random_reshuffle=1) * FROM people; ``` Run the following query to confirm the data is no longer skewed: ```sql SELECT DATABASE_NAME, TABLE_NAME, ORDINAL AS PARTITION_ID, ROWS, MEMORY_USE FROM INFORMATION_SCHEMA.TABLE_STATISTICS WHERE TABLE_NAME = 'people_1'; ``` ## Questions to Ask When Choosing Shard Keys * Does this table have a primary key? *Make sure there is a shard key and that it is a subset of the primary key.* * Do you frequently join on a specific set of columns? *Try to make the shard key a subset of the joined columns. Shard order should be the same between tables to accomplish this. For example, if you frequently join an `id` and `user_id` column, one of those columns should be the shard key.* * Do you frequently filter on a specific set of columns? *Try to make the shard key a subset of the filtered columns.* > **📝 Note**: Filtering on a range of the shard key can negatively affect the performance of your query. Multiple partitions may need to be scanned, thereby slowing the retrieval of your result set. * Do you have high concurrency queries? *Choose a shard key that allows these queries to be single partition.* * Is your data skewed for your current choice of shard key? *Try to add additional columns to the shard key to ensure even distribution.* * Do you need to `UPDATE` or `ALTER` any fields in the shard key? *Remove those fields from the shard key.* * Is this a small, infrequently changing table that needs to be on every node in the cluster to guarantee local joins? *Use a reference table instead of sharding the table.* ## Local Joins and Distributed Joins A local join is a join that can be executed in the partition where the data is currently stored. The tables that are joined must have a shard key that matches the join condition. The best way to accomplish this is to set the shard key on the column that you will be frequently joining on. In our first example of the `people` table with `id` as the shard key, say we are joining to an `address` table within the same database. To ensure your joins execute locally, make sure `id` is a column and a shard key in the `address` table. A distributed join is a join where the shard key that matches the join condition is stored on different partitions. Distributed joins occur in 2 ways: 1\) Repartition: * Each partition opens connections to every node. * Because the join requires the table to be sharded on the join condition, the query optimizer will create an internal, temporary results table in memory. This table will contain the results set at that stage of the query with a shard key that matches the join condition. 2\) Broadcast: * Occurs when one of the joining tables is small. Therefore, no cost is incurred by sending a copy of the table to each node. * After the broadcast of the table, the join is performed on the partition where the larger table is stored. A join between tables residing in different databases which reside in different partitions will be resolved through a distributed operation (broadcast or repartition) even if the shard keys of the tables match. ## Performance differences between repartition and broadcast operations Queries that require distributed joins, i.e. queries which involve Broadcasts or Repartitions, generally require much larger numbers of connections and threads than other queries. In most scenarios, the default settings for workload management will schedule your workload appropriately to utilize cluster resources without exhausting connection and thread limits. ## Related Topics * [Sharding](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md) * [Shard Keys](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/shard-keys.md) *** Modified at: June 18, 2026 Source: [/cloud/create-a-database/understanding-shard-key-selection/](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection/) (An index of the documentation is available at /llms.txt) --- # Understanding Sort Key Selection ## Sort Key The sort key is an index that groups rows of columnstore tables into logical segments, where each segment contains data for many rows. The sort key can be defined on a single column or multiple columns of a columnstore table. To specify a sort key, use `SORT KEY`. The sort key order can be specified as ascending (`SORT KEY()`) or descending (`SORT KEY( DESC)`). A columnstore sort key cannot be altered once the table has been created. To create an unsorted columnstore table, specify an empty sort key using `SORT KEY()`. A table scan using a sort key scans the table in the order of the sort key. A sort key cannot be used to scan a table in the reverse order of the sort key order. The following example creates a table, `t1`, with a sort key on `col1` in descending order. In the first query in the example below, the sort order in the query (`DESC`) matches the sort order of the `SORT KEY`, and the table is scanned using the `SORT KEY`. In the second query, the sort order in the query (`ASC`) does not match the sort order of the `SORT KEY`, and the `SORT KEY` is not used in the table scan. ```sql CREATE TABLE t1(col1 int, SORT KEY(col1 DESC)); EXPLAIN SELECT * FROM t1 ORDER BY col1 DESC; ``` ```output +------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------------------+ | GatherMerge [remote_0.col1 DESC] partitions:all alias:remote_0 | | Project [t1.col1] | | OrderedColumnStoreScan test1.t1, KEY col1 (col1 DESC) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +------------------------------------------------------------------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM t1 ORDER BY col1; ``` ```output +-----------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------------------------------+ | GatherMerge [remote_0.col1] partitions:all alias:remote_0 | | Project [t1.col1] | | Sort [t1.col1] | | ColumnStoreScan test1.t1, KEY col1 (col1 DESC) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +-----------------------------------------------------------------------------------------------------------+ ``` The following example creates a table with an empty sort key. ```sql CREATE TABLE t1(col1 int, SORT KEY( )); EXPLAIN SELECT * FROM t1; ``` ```output +---------------------------------------------------------------------------------------------------------+ | EXPLAIN | +---------------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 | | Project [t1.col1] | | ColumnStoreScan test8.t1, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +---------------------------------------------------------------------------------------------------------+ ``` > **📝 Note**: `KEY() USING CLUSTERED COLUMNSTORE` is a legacy syntax that is equivalent to `SORT KEY()`. SingleStore recommends using `SORT KEY()`. SingleStore recommends defining a shard key to explicitly control the data distribution. The following example defines a shard key on `id`. Sharding on a high cardinality identifier column generally allows for more even distribution and prevents skew. Refer to [Understanding Shard Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection.md) for more information. ```sql CREATE TABLE people ( id INT AUTO_INCREMENT, user VARCHAR(24), first VARCHAR(24), last VARCHAR(24), SORT KEY (user), SHARD KEY (id) ); ``` Setting the shard key and the sort key on the same column improves data compression. Refer to [Managing Columnstore Segments](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments.md) for additional information about columnstore segments. ## Segment Elimination The single most important consideration for columnstore tables is setting a sort key. Each segment file contains metadata that holds the minimum and maximum values for the rows in the segment. The minimum/maximum value metadata is used at query execution time to determine whether a segment can possibly match a filter; if not, the segment is skipped entirely and no data is examined. This functionality is called segment elimination because the segment is eliminated from the scan. For queries that filter on the sort key column of the columnstore index, segment elimination is typically very efficient as row segments within each row segment group will not have overlapping segments. The segment size (number of rows per segment) also impacts query execution with respect to the sort key. A smaller segment size means that a smaller number of rows are read from the segments that pass segment elimination. Column segments typically contain on the order of tens of thousands of rows. Indexing further cuts down on the number of rows scanned when a select query with a `WHERE` clause is executed. If the table has an index for the columns in the `WHERE` clause, the engine can quickly determine the location of the matching rows position to seek without having to look at all the data. This is much faster than reading every row sequentially. The figures below show three segments of the `people` table. The segment size is 3 rows for readability. Segment #1 of 3 - `user` values aa - jl ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1c672820402f2d00/6a7adb7e366687108627e3af/img_docs_understanding-sort-key-selection-image1_dark-ld7vn5.png) Segment #2 or 3 - `user` values jm - rl ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4d859a96274951fb/6a7adaac6c6eac61d3f08ce9/img_docs_understanding-sort-key-selection-image2_dark-TtuGJQ.png) Segment #3 of 3 - `user` values rm - zz ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte97ee0961ce962c1/6a7adab0d402d991efeadcf0/img_docs_understanding-sort-key-selection-image3_dark-rmgX3R.png) If a query is searching for a person whose first name starts with an "a", then only the first segment is scanned. The other two segments are eliminated from the scan since those `user` values are "jm - zz". ## Ordered Scans Ordered scan over a sort key is faster than sorting a table. A query that sorts with a sort key can benefit from running an ordered columnstore scan instead of performing a normal columnstore scan. ## Sub-segment Elimination Sub-segment elimination improves performance by eliminating data at the sub-segment level during query processing and thus extends the performance improvements provided by segment elimination. Segments consist of blocks of 4096 rows each. With sub-segment elimination, statistics are collected for blocks and are used during query processing to eliminate blocks. Sub-segment elimination is restricted to sort key columns. A segment with the default size (`columnstore_segment_rows = 1,024,000`) would have 1,024,000 / 4,096 = 250 sub-segments. Queries with highly selective predicates using flexible parallelism are expected to see performance improvements due to sub-segment elimination. Consider a table and a query as follows: ```sql CREATE TABLE sales(id INT, dt DATETIME, itemnum INT, qty FLOAT, price numeric(18,4), SORT KEY(dt)); SELECT itemnum, SUM(qty*price) FROM sales WHERE dt BETWEEN "2024-09-20 17:00:00" AND "2024-09-20 17:01:00" GROUP BY all; ``` This query has a very narrow time range (a highly selective predicate) on the `dt` column, and the `dt` column is the sort key. Sub-segment elimination will be applied to this query and is expected to improve the performance of this query. The following figure shows how sub-segment elimination can greatly reduce the number of rows read and significantly improve performance. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2a05193b4e0c210e/6a7adab573d9bd97fe292498/img_docs_sub-segment-elimination-image2-dark-jBe06m.png) Segments that were created before sub-segment elimination was introduced, or which were created when the `enable_block_stats_use_in_query` engine variable was set to `OFF`, will not have block-level statistics and will not benefit from sub-segment elimination. Such segments will need to be rebuilt to benefit from sub-segment elimination. The engine variable `enable_block_stats_use_in_query` is set to `ON` by default. ## Multi-Column Sort Keys Sort keys can be defined on multiple columns, which can improve performance in certain cases. ## Storage The following examples show how a table is stored with a single-column sort key followed by how the table is stored with a two-column sort key. The data in a partition is broken into segments based on the sort key order and the segment size. (Refer to [Configuring the Columnstore to Work Effectively](https://docs.singlestore.com/cloud/create-a-database/configuring-the-columnstore-to-work-effectively.md) for more information.) Create a table named `INTEGERS` with a single-column sort key. ```sql CREATE TABLE INTEGERS ( A int, B int, SORT KEY (A) ); ``` The data in the `INTEGERS` table will be sorted on column `A`, and then broken into segments, so that segments contain contiguous values of column `A`. The figure below shows how the data is stored with the single-column sort key. If the integers 1 - 10 are inserted into columns A and B, and the segment size is 8, the first three columnstore segments would look as follows. The segmentation is defined by the sort key order. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt15f773d8104b7b8b/6a7adad1c5d19326a45db8d7/img_docs_understanding-sort-key-selection-image4b_light-EU3nUX.png) For a multi-column sort key, the data in a partition is also broken into segments based on the sort key order and the segment size. The command below creates a table named `INTEGERS` with a multi-column sort key on columns `A` and `B`. ```sql CREATE TABLE INTEGERS ( A int, B int, SORT KEY (A,B) ); ``` The data in this table will be sorted on column `A`, and secondarily on column `B`, and then broken into segments so each segment contains a contiguous segment of data in that (multi-column) sort order. The figure below shows how the data is stored with the multi-column sort key. If the integers 1 - 10 are inserted into columns `A` and `B`, and the segment size is 10, the first three columnstore segments would look as follows. The segmentation is defined by the sort key order - sorting the data first on the values in `A`, and then secondarily on the values in column `B`. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6302090d4a982981/6a7adb984c4bfbf679cc1f4d/img_docs_understanding-sort-key-selection-image5b_light-URVuWt.png) ## Example Creating multi-column sort keys on a table can improve query performance. For example, queries on a table with date and region columns would be expected to benefit from a sort key defined on both columns (`SORT KEY (date, region)`). This sort key definition clusters the table on date and region columns, so that segments contain values with similar date and region values. This in turn makes encoding more effective so the data is better compressed. For queries on this table that filter on date and region columns, the engine eliminates segments and sub-segments that do not meet the filter criteria, reducing the amount of scanning needed for the query. Efficient encoding also helps reduce I/O during the scan. ## Questions to Ask When Choosing Sort Keys * Is the data always filtered by some column (e.g. insert timestamp or event type)? Ensure that the common columns for all queries are in the sort key to improve segment elimination.  * Is the data generally inserted in order by some column (e.g. insert timestamp)? It’s best to put that column first in the sort key to minimize the amount of work required by the background columnstore segment merger. * Does one column in your key have a higher cardinality than the other? It’s best to put the lowest cardinality columns first to increase the likelihood that segment elimination will be able to affect later columns. ## Related Topics * [Understanding Shard Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection.md) * [Managing Columnstore Segments](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments.md) * [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) *** Modified at: August 6, 2026 Source: [/cloud/create-a-database/understanding-sort-key-selection/](https://docs.singlestore.com/cloud/create-a-database/understanding-sort-key-selection/) (An index of the documentation is available at /llms.txt) --- # Using JSON ## About JSON in SingleStore SingleStore exposes a Javascript Object Notation (JSON) (An open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays.) column type that implements the [JSON standard](http://http://www.ietf.org/rfc/rfc4627.txt). You can define columns in SingleStore tables using the [JSON Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/json-type.md). Analytics on these JSON columns is very efficient as SingleStore automatically columnarizes JSON data. A schema is inferred from JSON keys and the data is split into columns by key path and stored in an encoded [Parquet-like format](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-seekability-using-json/#section-idm413515222379322.md). The JSON data is stored as if you had created a schema with separate columns for every field. As a result, queries on JSON columns read only the parts of a JSON object that are relevant to a query and therefore have excellent performance. SingleStore provides a set of JSON functions for extracting, searching, analyzing, and modifying JSON data, including: * [JSON\_EXTRACT\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md), to extract values out of JSON documents at specified keypaths. * Shorthand syntax, `::` operators, to extract values out of JSON documents. The `::` operators are convenient aliases for the `JSON_EXTRACT_` functions and follow the same rules. * JSON shorthand syntax is a path that uses `::` as a separator. * [JSON\_MATCH\_ANY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md), to check for the existence of values in a JSON document or array based on a path and a filter. * [JSON\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-to-array.md), to convert a JSON array to a SingleStore array. * `JSON_TO_ARRAY` can be used in combination with [TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/table.md) to operate on values of a JSON array as SQL rows. This functionality is similar to `UNNEST` in other database systems. JSON columns can be searched using SingleStore's [Java Lucene](https://lucene.apache.org/core/)-compatible [full-text search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) and indexed with the [Multi-Value Hash Index (JSON)](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json.md). In addition, JSON columns can be indexed using computed columns; refer to [Indexing Data in JSON Columns](https://docs.singlestore.com/#UUID-2ccafa6b-ead7-016d-adca-79fb8d0bdade.md) for more information. SingleStore has a native BSON data type, plus [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md), a MongoDB®-compatible API. The BSON data is stored in columns just like JSON data, and can be indexed with the [Multi-Value Hash Index (BSON)](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/multi-value-hash-index-bson.md) and computed columns. An alternative to using the `JSON` type is to map JSON fields to individual columns and use SQL queries to access the JSON data. Refer to [Load JSON Files with LOAD DATA](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/load-json-files-with-load-data.md) for information about how to map `JSON` fields to SingleStore columns during the loading process. ## JSON Data Encodings The default encoding for `JSON` columns is `SeekableJson`. As described above, this encoding is very efficient as it columnarizes the JSON. A schema is inferred from JSON keys and the data is split into columns by keypath and stored in an encoded Parquet-like format. Refer to [Columnstore Seekability using JSON](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-seekability-using-json.md) for more information. `JSON` columns may also be encoded with the [string encodings](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/encoding-string-data-in-columnstore-tables.md) `SeekableLZ4` and `SeekableStringRunLength`. These encodings compress data, but do not support efficient searches or extraction. SingleStore recommends using `SeekableLZ4` and `SeekableStringRunLength` only for columns on which JSON values will be extracted as a whole. **Note**: The information on this page is intended for use with `JSON` columns created with `SeekableJson` encoding or its precursor `JSONParquet`. Using functions described on this page on `JSON` columns with a string encoding will result in poor performance. ## Examples The following table is used in the examples. Note the `printings` array is intended to indicate the number of copies of the book printed in each printing; `printings` data is in millions of books and is not accurate. ```sql CREATE TABLE books_json (id INT, books JSON); INSERT INTO books_json VALUES (1, '{ "title": "Onyx Storm", "author": "Rebecca Yarros", "details": { "publisher": "Entangled:Red Tower Books", "numpages": 544, "publication date": "January 21, 2025", "printings": [2,1.3], "series": "The Empyrean" } }' ), (2, '{ "title": "The Maid", "author": "Nita Prose", "details": { "publisher": "Ballantine Books", "numpages": 385, "publication date": "January 4, 2022", "printings": [0.5,0.75,1.2] } }' ), (3, '{ "title": "The Last Letter", "author": "Rebecca Yarros", "details": { "publisher": "Entangled:Amara", "numpages": 432, "publication date": "February 26, 2019", "printings": [0.25,0.5,0.5] } }' ); ``` ## Example 1: Extract Values from JSON Using the `::` Operators The `::`, `::$`, and `::%` operators can be used to extract fields, strings, and SQL doubles from JSON documents. In the example below: * `books::title` extracts the `title` field, * `books::$title` extracts the `title` field as a SQL string, * `books::details::%numpages` extracts the `numpages` field as a double, and * `` books::details::printings::%`0` `` extracts the 0th element of the `printings` array. Backticks (\`) are required around numeric keys, as shown with `` `0` `` above Refer to [Using the ::$ and ::% Operators](https://docs.singlestore.com/#UUID-36dc6fd4-161c-cbc2-4a87-556e9d33d4dd.md) for details. ```sql SELECT id, books::title AS title, books::$title AS title_string, books::details::%numpages AS numpages, books::details::printings::%`0` AS first_printing FROM books_json ORDER BY id; ``` ```output +------+-------------------+-----------------+----------+----------------+ | id | title | title_string | numpages | first_printing | +------+-------------------+-----------------+----------+----------------+ | 1 | "Onyx Storm" | Onyx Storm | 544 | 2 | | 2 | "The Maid" | The Maid | 385 | 0.5 | | 3 | "The Last Letter" | The Last Letter | 432 | 0.25 | +------+-------------------+-----------------+----------+----------------+ ``` ## Example 2: Extract Values from JSON using `JSON_EXTRACT_` The [JSON\_EXTRACT\_](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md) functions can be used to extract values from a JSON document in addition to the path expression syntax shown in Example 1. The `JSON_EXTRACT_` functions can be used when you want to use variables or expressions in the keypath. The keypath in `JSON_EXTRACT_` functions is a comma-separated list of object keys or zero-indexed array positions. Below is a query similar to the query in Example 1, expressed using `JSON_EXTRACT_` functions. ```sql SELECT id, JSON_EXTRACT_JSON(books,'title') AS title, JSON_EXTRACT_STRING(books,'title') AS title_string, JSON_EXTRACT_BIGINT(books,'details','numpages') AS numpages, JSON_EXTRACT_DOUBLE(books,'details','printings',1-1) AS first_printing FROM books_json ORDER BY id; ``` ```output +------+-------------------+-----------------+----------+----------------+ | id | title | title_string | numpages | first_printing | +------+-------------------+-----------------+----------+----------------+ | 1 | "Onyx Storm" | Onyx Storm | 544 | 2 | | 2 | "The Maid" | The Maid | 385 | 0.5 | | 3 | "The Last Letter" | The Last Letter | 432 | 0.25 | +------+-------------------+-----------------+----------+----------------+ ``` Note that the expression `1-1` is used to extract the value in position 0 in the `printings` array. Such expressions are supported in `JSON_EXTRACT_` functions, but not when using the `::` operators. A `JSON_EXTRACT_` on a nested key will only scan the column for that key, not the entire JSON document. For example, `JSON_EXTRACT_JSON(jsondata,'details','title') AS title`, will only scan the column for the `title` key. ## Example 3: Find Existence of Values and Paths using JSON\_MATCH\_ANY The [JSON\_MATCH\_ANY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) and [JSON\_MATCH\_ANY\_EXISTS](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) functions can be used to find the existence of values matching predicates and paths in JSON documents. The `JSON_MATCH_ANY` function returns true if a value exists in the JSON at the filter path for which the filter predicate evaluates to true. The `JSON_MATCH_ANY_EXISTS` function returns true if there is a value (possibly null) in the JSON at the filter path. The following is an example of using `JSON_MATCH_ANY` to find books that are part of `The Empyrean` series. ```sql SELECT id, books::$title AS title, books::details::$series AS series FROM books_json WHERE JSON_MATCH_ANY(books::?details.series, MATCH_PARAM_STRING_STRICT() = "The Empyrean"); ``` ```output +------+------------+--------------+ | id | title | series | +------+------------+--------------+ | 1 | Onyx Storm | The Empyrean | +------+------------+--------------+ ``` ## Example 4: Aggregate Elements in a JSON Array Value using REDUCE The [REDUCE](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/reduce.md) function can be used to aggregate elements in a JSON array value. SingleStore recommends using the `REDUCE` function when aggregating elements within a JSON array in a row. Refer to [Example 5](https://docs.singlestore.com/#section-idm234948461588865.md) for an example of aggregating JSON array values across multiple rows which takes advantage of [JSON Array Performance Enhancements](https://docs.singlestore.com/#section-idm323419580718356.md). The `REDUCE` function applies an expression to each element of an array and returns a single value. In the following example, `REDUCE` is used to sum the elements of the printings array for each row. The result is the total number of copies of each book that have been printed. ```sql SELECT books::$title, REDUCE( 0 :> double, JSON_TO_ARRAY(books_json.books::details::printings), REDUCE_ACC() + REDUCE_VALUE() ) AS total_printed_by_book FROM books_json; ``` ```output +-----------------+-----------------------+ | books::$title | total_printed_by_book | +-----------------+-----------------------+ | The Maid | 2.45 | | The Last Letter | 1.25 | | Onyx Storm | 3.3 | +-----------------+-----------------------+ ``` In this example: 1. The `JSON_TO_ARRAY` function converts the array in the JSON field `jsondata::books::details::printings` to a SQL array. 2. The `REDUCE` function takes as input 1. An initial value: `0:>double`. 2. A SQL array: the result of `JSON_TO_ARRAY`. 3. An accumulator expression: `(REDUCE_ACC() + REDUCE_VALUE())`, which specifies that the array elements should be summed. ## Example 5: Use the `::` Operators in UPDATE Statements You can use the `::` operators for JSON keypaths in [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) queries. The `::` operators are simply convenient aliases for the [JSON\_SET\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-set-type.md) functions and follow the same rules. The following query updates the number of pages in the book, `Onyx Storm`. ```sql UPDATE books_json SET books::details::%numpages = 545 WHERE books::$title = "Onyx Storm"; ``` To add a printing for the book `Onyx Storm`, use [JSON\_ARRAY\_PUSH\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-array-push-type.md) as follows. ```sql UPDATE books_json SET books::details::printings = JSON_ARRAY_PUSH_DOUBLE(books::details::printings, 1.5) WHERE books::$title = "Onyx Storm"; ``` To add a series to the book `The Maid`. ```sql UPDATE books_json SET books::details::$series = 'Molly the Maid' WHERE books::$title = "The Maid"; ``` View the resulting updates. ```sql SELECT JSON_PRETTY(books) FROM books_json WHERE books::$title = "The Maid"; ``` ```output +---------------------------------------------+ | { "author": "Nita Prose", "details": { "numpages": 385, "printings": [ 0.5, 0.75, 1.2 ], "publication date": "January 4, 2022", "publisher": "Ballantine Books", "series": "Molly the Maid" }, "title": "The Maid" } | +---------------------------------------------+ ``` ## Example 6: Convert an Array (List) of JSON Objects to a Table Use the `JSON_TO_ARRAY` function and the `TABLE` built-in function to convert a list of JSON objects to a table. Create a table with a column to hold a JSON value and insert data into that table. ```sql CREATE TABLE json_list_example (json_list JSON); INSERT INTO json_list_example values( '[ { "title": "Onyx Storm", "author": "Rebecca Yarros", "numpages": 544 }, { "title": "The Last Letter", "author": "Rebecca Yarros", "numpages": 432 } ]'); ``` In the following query, the syntax `JOIN TABLE(JSON_TO_ARRAY(json_list))` converts the objects in the `json_list` column to a table, that is, each object in the JSON list is turned into a row in the table. * The `JSON_TO_ARRAY` function converts the JSON array to a SingleStore `ARRAY`. * The `TABLE` function converts a SingleStore `ARRAY` to a column named `table_col` that contains one row for each array entry. * The `JOIN` clause is required when using the `TABLE` function on an existing SingleStore table. ```sql SELECT json_list_as_table.table_col AS books_col FROM json_list_example JOIN TABLE(JSON_TO_ARRAY(json_list)) AS json_list_as_table; ``` ```output +----------------------------------------------------------------------+ | books_col | +----------------------------------------------------------------------+ | {"author":"Rebecca Yarros","numpages":544,"title":"Onyx Storm"} | | {"author":"Rebecca Yarros","numpages":432,"title":"The Last Letter"} | +----------------------------------------------------------------------+ ``` The `JSON_AGG` function can be used to combine the rows in the table into a single row and re-create the original JSON array. ```sql WITH books_table AS ( SELECT json_list_as_table.table_col AS books_col FROM json_list_example JOIN TABLE(JSON_TO_ARRAY(json_list)) AS json_list_as_table ) SELECT JSON_AGG(books_col) FROM books_table; ``` ```output +----------------------------------------------------------------------------------------------------------------------------------------+ | JSON_AGG(books_col) | +----------------------------------------------------------------------------------------------------------------------------------------+ | [{"author":"Rebecca Yarros","numpages":544,"title":"Onyx Storm"},{"author":"Rebecca Yarros","numpages":432,"title":"The Last Letter"}] | +----------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Example 7: Aggregate Elements in JSON Arrays Across Rows using TABLE and JSON\_TO\_ARRAY Use the [TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/table.md) and [JSON\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-to-array.md) functions to aggregate values from JSON arrays across rows. Aggregations over results of the `JOIN TABLE(JSON_TO_ARRAY())` operation are optimized to reduce the amount of data read. Queries will retrieve only the parts of the JSON column relevant to the query instead of fetching the entire JSON object from disk. Refer to [JSON Array Performance](https://docs.singlestore.com/#section-idm323419580718356.md) for details. The following example uses the `TABLE` and `JSON_TO_ARRAY` functions to sum the elements in the printings array across all rows for each author. ```sql SELECT books::$author AS author, SUM(JSON_EXTRACT_DOUBLE(i.table_col)) AS total_printed_by_author FROM books_json JOIN TABLE( JSON_TO_ARRAY(books_json.books::details::printings)) AS i GROUP BY author; ``` ```output +----------------+-------------------------+ | author | total_printed_by_author | +----------------+-------------------------+ | Rebecca Yarros | 4.55 | | Nita Prose | 2.45 | +----------------+-------------------------+ ``` In this example: 1. The `JSON_TO_ARRAY` function converts the array in the JSON field `books_json.books::details::printings` to a SQL array. 2. The `TABLE` function converts this array into a table. 3. The `JOIN` clause `FROM books_json JOIN TABLE(JSON_TO_ARRAY(books_json.books::details::printings)) AS i` creates a table that contains: 1. For each row in the `books_json` table: 1. One row for each element in the `printings` array in that row. 2. This row contains the values from the columns in the `books_json` table and a column, `table_col`, which contains the array value. 4. Finally, the `SELECT` and `GROUP BY` aggregate over the table created in step 3. In this example, which aggregates over multiple JSON documents, SingleStore only reads the `author` and `printings` values to compute the result. Additional JSON values, such as `title`, `publisher`, and `numpages`, will not be read making the query execution very efficient. **Note**: The `JSON_EXTRACT_DOUBLE()` function in the `SELECT` clause is necessary to use the `JSON_TO_ARRAY` optimization. If that function is removed, the query will execute, but will not use the optimization. ## Managing Collections of Metadata JSON is useful for managing a collection of diverse data, represented as name-value pairs, that might otherwise be cumbersome to refactor into a formalized key-value table, or that might be stored in a table that is sparsely populated. For example, suppose an organization had an asset management application using SingleStore Helios to track all the information about its physical assets. The asset data is diverse—what’s relevant for a desk differs from what’s important for a server machine or a company car. All assets might have common attributes, such as asset tag ID, asset type, asset name, and description. Each type of asset might have unique attributes, such as size and weight dimensions, hostname and IP address, or gas mileage. Instead of creating a highly granular table to manage all data as key-value pairs, this organization could simply create a SingleStore Helios table using a JSON column to efficiently manage the unique attribute data. With this design: * Each asset gets a row in the table. * Attributes that are common to all assets have their own column in the table. These columns allow you to query on common features and quickly narrow down the final result set as much as possible (for example, filtering by asset type). * The various, remaining attributes associated with each asset are stored in a JSON column (which might be named something like `property_bag`). For example, the JSON column for an office desk asset could include JSON data such as size, weight, and number of drawers. The JSON data for a server machine could include rack location, number of cores, and MAC address. ## DDL: Defining Tables with JSON Columns Any SingleStore Helios table can contain one or more columns of data type `JSON`. A JSON column can optionally be suffixed with `NOT NULL`. ## Comparing JSON and LONGTEXT Columns A JSON column is analogous to a `LONGTEXT` column in the following ways: * JSON columns can store arbitrarily large JSON values in a normalized text representation. * JSON columns have the same storage requirement, as if the JSON value were stored in a text column. The primary difference is that JSON data is stored in a normalized format, which makes many operations faster than if the data were stored manually in a text column. The following is an example of non-normalized data, which is valid JSON but is relatively difficult to parse: ```json '{ "b\u000a": 1,"a": 2 ,"a":3 } ' ``` Normalized data, on the other hand, is easier to parse because duplicate keys are merged, the data is sorted by keys, and extraneous whitespace is removed. ```json '{"a":3,"b\n":1}' ``` SingleStore recommends storing JSON data in `JSON` columns and not in `LONGTEXT` columns. `JSON` columns validate the JSON values and provide [Unicode Support](https://docs.singlestore.com/#UUID-e72c5528-d96b-6095-ab75-bf514f7f247a.md). If storage space and memory use is a concern, and search and extraction are not required on the column, use a string encoding on the column. Refer to [Columnstore Seekability using JSON](https://docs.singlestore.com/cloud/create-a-database/columnstore/columnstore-seekability-using-json.md) for more information. ## Defining JSON Columns Defining a JSON column in a SingleStore Helios table is as simple as specifying the JSON data type in the `CREATE TABLE` command: ```sql CREATE TABLE assets ( tag_id BIGINT PRIMARY KEY, name TEXT NOT NULL, description TEXT, properties JSON NOT NULL); ``` ## JSON Collation The collation for a JSON column can be set explicitly using a `COLLATE` clause. When the collation is not set explicitly, the collation is controlled by the `json_collation` engine variable. Refer to [Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md), [Character Set and Collation Override](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/character-set-and-collation-override.md), [Unicode Support](https://docs.singlestore.com/#UUID-e72c5528-d96b-6095-ab75-bf514f7f247a.md), and [Character Encodings and Collation](https://docs.singlestore.com/#UUID-00e2ca08-8dc7-bc24-f268-b6bfde552e6a.md) for more information. ## Indexing Data in JSON Columns JSON columns are not indexed directly - they are indexed using computed columns. For the fastest performance, you should not use JSON built-ins or `::` notation in your filters. Instead, create a computed column that includes the JSON column in the computation, and then use the computed column for the index. In this way, the index gets updated only when the relevant JSON data is updated in a row. ```sql CREATE TABLE assets ( tag_id BIGINT PRIMARY KEY, name TEXT NOT NULL, description TEXT, properties JSON NOT NULL, weight AS properties::%weight PERSISTED DOUBLE, license_plate AS properties::$license_plate PERSISTED LONGTEXT, KEY(license_plate), KEY(weight)); ``` JSON computed columns that are indexed will be utilized by the optimizer more efficiently. Queries that use indexed computed columns as filters or sorts, will perform faster by avoiding expression evaluation, and by seeking into or searching the indexes rather than scanning tables. In the following examples, we will use the `::` notation in the filtering and sorting clauses to illustrate how the optimizer matches the computed columns. ```sql EXPLAIN SELECT * FROM assets WHERE properties::$license_plate = "VGB116"; ``` ```output +------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 | | Project [assets.tag_id, assets.name, assets.description, assets.properties, assets.weight, assets.license_plate] | | ColumnStoreFilter [assets.license_plate = 'VGB116' index] | | ColumnStoreScan test1.assets, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +------------------------------------------------------------------------------------------------------------------+ 4 rows in set (0.00 sec) ``` ```sql EXPLAIN SELECT * FROM assets ORDER BY properties::%weight; ``` ```output +------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------------------+ | GatherMerge [remote_0.weight] partitions:all alias:remote_0 | | Project [assets.tag_id, assets.name, assets.description, assets.properties, assets.weight, assets.license_plate] | | Sort [assets.weight] | | ColumnStoreScan test1.assets, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +------------------------------------------------------------------------------------------------------------------+ 4 rows in set (0.00 sec) ``` ## DML: Accessing Data in JSON Columns This section describes how to insert and update data in a SingleStore Helios table with one or more JSON columns. ## Inserting Data into a JSON Column When inserting a row in a table, specifying JSON data is straightforward. For example: Given a table `t` defined as `CREATE TABLE test_table(col_a TEXT, col_b JSON);`, you can insert a row into `test_table` as follows: ```sql INSERT INTO test_table(col_a,col_b) VALUES ('hello','{"x":"goodbye","y":"goodnight"}'); ``` ## Columnstore Tables Having JSON Columns with Null Values or Empty Arrays By default, SingleStore Helios preserves columnstore JSON `NULL` values and empty arrays. To change this behavior and disable this setting, set the `preserve_original_colstore_json` global variable to `OFF`. This flag preserves the original columnstore JSON for any new data that is loaded; it does not update the existing data. By default, this variable is set to `AUTO` (same as `ON`). ## Behavior When `preserve_original_colstore_json` is Set to `OFF` When you store a JSON column in a columnstore table (and `preserve_original_colstore_json` is set to `OFF`), then null values and empty arrays in the JSON object are handled as follows when the object is written to the columnstore: * Name/value pairs with the value `NULL` are normally removed from the JSON object. * Name/value pairs containing empty arrays are normally removed from the JSON object. * If the JSON object has only the value `NULL` or `[]`, the value is replaced with `NULL`. Example commands you can use to store data are `INSERT`, `UPDATE`, and `LOAD DATA`. An example `INSERT` scenario follows. Consider a table that is defined as: ```sql CREATE TABLE json_empty_values_table(a INT, b JSON, SORT KEY (a)); ``` Insert five rows into the table: ```sql INSERT INTO json_empty_values_table VALUES (1, '{"v":null}'); INSERT INTO json_empty_values_table VALUES (2, '{"w":[]}'); INSERT INTO json_empty_values_table VALUES (3, '{"x":"foo","y":null,"z":[]}'); INSERT INTO json_empty_values_table VALUES (4, 'null'); INSERT INTO json_empty_values_table VALUES (5, '[]'); ``` Manually flush the inserted data to the columnstore: ```sql OPTIMIZE TABLE json_empty_values_table FLUSH; ``` Query the table: ```sql SELECT * FROM json_empty_values_table ORDER BY a; ``` ```output +------+-----------------------------+ | a | b | +------+-----------------------------+ | 1 | {"v":null} | | 2 | {"w":[]} | | 3 | {"x":"foo","y":null,"z":[]} | | 4 | null | | 5 | [] | +------+-----------------------------+ ``` ## Accessing Fields in a JSON Object To access a field of a JSON object stored in a column, use the name of the column suffixed with `::keyName`. For example, if column data contains `{"bits":[true,false]}`, then the expression `data::bits` evaluates to the JSON value `[true,false]`. ## Converting a JSON String Value into a SQL String The `::` operator, when applied to a string column of a JSON object, returns the value of the string enclosed in quotes. Use the `::$keyname` operator to return the value of the string without the enclosing quotes. For example, consider a table `TestJSON `that contains a JSON column `data`. ```sql INSERT INTO TestJSON VALUES ('{"first":"hello"}'); ``` Retrieve the value of the `data `column using `::` and `::$` operators: ```sql SELECT data::first, data::$first FROM TestJSON; ``` ```output +-------------+--------------+ | data::first | data::$first | +-------------+--------------+ | "hello" | hello | +-------------+--------------+ ``` In this example, the `data `JSON column contains two string values. ```sql INSERT INTO TestJSON VALUES ('{"first":"hello", "second":"world"}'); ``` To retrieve the concatenated SQL string, use the `::$` operator: ```sql SELECT CONCAT(data::$first, ' ', data::$second) FROM TestJSON; ``` ```output +------------------------------------------+ | CONCAT(data::$first, ' ', data::$second) | +------------------------------------------+ | hello world | +------------------------------------------+ ``` ## Converting a JSON Number or Boolean Value into a SQL DOUBLE To transparently convert a JSON number or Boolean value into a SQL `DOUBLE`, use the name of the column suffixed with `::%keyname`. For example, if column data contains `{"valid":true,"value":3.14}`, then `data::%valid` is 1 and `data::%value` is 3.14. ## Using the ::$ and ::% Operators If the `::$` or `::%` access operator is used on a JSON field that is not of the appropriate type, SingleStore converts the JSON result to a string or a double according to the rules defined for `JSON_EXTRACT_`. The `::` access operators are convenient aliases for the `JSON_EXTRACT_` built-in functions and follow the same rules. The `::` operator does not work on the output of user-defined functions (UDFs), such as `SELECT udf_name(1)::key`, or on expressions. It also does not work with non-scalar types (`RECORD`, `ARRAY`) because it is not a struct field accessor. Note that `ROW` is an expression that returns a `RECORD` type, not a type itself. Applying the `::` operator to a `RECORD` value, such as `table.row_col::field_name`, returns an error. For UDF outputs, use the `JSON_EXTRACT_` functions. For `RECORD` types, use the individual named columns. SingleStore uses the `::` operator in the following contexts: | Context | Syntax | Description | Example | | --------------------------- | ----------------------------- | -------------------------------------------------------- | ------------------------------------ | | JSON field extraction | `column::key` | Extracts a value from a JSON column by key | `metadata::name` | | Nested JSON extraction | `column::key1::key2` | Navigates nested JSON objects | `details::address::city` | | `JSON`type casting | `column::$key`,`column::%key` | Extracts a value as a string (`$`) or numeric (`%`) type | `data::$name`,`data::%age` | | Subvalue path in`LOAD DATA` | `col <- path::key` | Maps nested JSON fields during data loading | `publisher_s2 <- details::publisher` | | Remote table reference | `link::database.table` | References a table on a remote linked server | `mylink::remote_db.orders` | In addition, The `::` operators require that numeric keys be specified with backticks (\`). The following `SELECT` statement returns the 2nd (zero-indexed) array element of the array `{"a":[1,2,3,4]}`. A syntax error will be returned if the backticks (\`) are not included. ```sql SELECT json, json::a::`2` FROM (SELECT '{"a":[1,2,3,4]}' AS json) sub; ``` ```output +-----------------+--------------+ | json | json::a::`2` | +-----------------+--------------+ | {"a":[1,2,3,4]} | 3 | +-----------------+--------------+ ``` In addition, backticks can be used for non-numeric key names and array indexes, which is useful if the key name or index contains a space. The following query is valid and will return the same output as the query above. ```sql SELECT json, json::`a`::`2` FROM (SELECT '{"a":[1,2,3,4]}' AS json) sub; ``` ```output +-----------------+----------------+ | json | json::`a`::`2` | +-----------------+----------------+ | {"a":[1,2,3,4]} | 3 | +-----------------+----------------+ ``` The following is the same logical query except the array index 2 has been replaced by the expression `1+1` and the `JSON_EXTRACT_JSON` function is used. While this example uses a simple expression `(1+1)`, more functions such as `DAYOFWEEK` and more complex expressions can be used. Refer to [JSON\_EXTRACT\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md) for details. ```sql SELECT json, JSON_EXTRACT_JSON(json, "a", 1+1) FROM (SELECT '{"a":[1,2,3,4]}' AS json) sub; ``` ```output +-----------------+--------------+ | json | json::a::`2` | +-----------------+--------------+ | {"a":[1,2,3,4]} | 3 | +-----------------+--------------+ ``` ## Accessing Nested JSON Objects To access nested JSON objects, chain the colon-colon operator to form a `keypath`. For example, `data::address::street` means the `street` field of the `address` field of the `data` column. > **📝 Note**: If one of the keys in the keypath is not present in the nested object, then the `entire` colon-colon expression yields SQL `NULL`. ## Working with Nested Arrays in a JSON Column Consider a table defined as: ```sql CREATE TABLE json_tab (`id` INT(11) DEFAULT NULL,`jsondata` JSON COLLATE utf8_bin); ``` Insert values as follows: ```sql INSERT INTO json_tab VALUES ( 8765 ,' {"city":"SFO","sports_teams":[{"sport_name":"football","teams": [{"club_name":"Raiders"},{"club_name":"49ers"}]}, {"sport_name":"baseball","teams" : [{"club_name":"As"},{"club_name":"SF Giants"}]}]}') ; INSERT INTO json_tab VALUES ( 9876,'{"city":"NY","sports_teams" : [{ "sport_name":"football","teams" : [{ "club_name":"Jets"},{"club_name":"Giants"}]}, {"sport_name":"baseball","teams" : [ {"club_name":"Mets"},{"club_name":"Yankees"}]}, {"sport_name":"basketball","teams" : [{"club_name":"Nets"},{"club_name":"Knicks"}]}]}'); ``` Query the table: ```sql WITH t AS( SELECT id, jsondata::city city , table_col AS sports_clubs FROM json_tab JOIN TABLE(JSON_TO_ARRAY(jsondata::sports_teams))), t1 AS( SELECT t.id, t.city, t.sports_clubs::sport_name sport, table_col AS clubs FROM t JOIN TABLE(JSON_TO_ARRAY(t.sports_clubs::teams))) SELECT t1.id, t1.city,t1.sport,t1.clubs::club_name club_name FROM t1; ``` ```output +------+-------+--------------+-------------+ | id | city | sport | club_name | +------+-------+--------------+-------------+ | 9876 | "NY" | "football" | "Jets" | | 9876 | "NY" | "football" | "Giants" | | 9876 | "NY" | "baseball" | "Mets" | | 9876 | "NY" | "baseball" | "Yankees" | | 9876 | "NY" | "basketball" | "Nets" | | 9876 | "NY" | "basketball" | "Knicks" | | 8765 | "SFO" | "football" | "Raiders" | | 8765 | "SFO" | "football" | "49ers" | | 8765 | "SFO" | "baseball" | "As" | | 8765 | "SFO" | "baseball" | "SF Giants" | +------+-------+--------------+-------------+ ``` You can also further filter the results by applying conditions. For example, to find the city with the "Yankees" club, run the following query: ```sql WITH t AS (SELECT id, jsondata::city city , table_col AS sports_clubs FROM json_tab JOIN TABLE(JSON_TO_ARRAY(jsondata::sports_teams))), t1 AS (SELECT t.id, t.city, t.sports_clubs::sport_name sport, table_col AS clubs FROM t JOIN TABLE(JSON_TO_ARRAY(t.sports_clubs::teams))) SELECT t1.id, t1.city,t1.sport,t1.clubs::club_name club_name FROM t1 WHERE t1.clubs::$club_name = 'Yankees'; ``` ```output +------+------+------------+-----------+ | id | city | sport | club_name | +------+------+------------+-----------+ | 9876 | "NY" | "baseball" | "Yankees" | +------+------+------------+-----------+ ``` ## Nested JSON Ingest Working with a nested JSON ingest requires an additional escape as the value being inserted is interpreted as a SQL string first. Therefore, the backslash (\\) symbol needs an escape in addition to the JSON escape (‘\’): ```sql CREATE TABLE test_json(col_a json); INSERT INTO test_json VALUES ('{"addParams": "{\\"Emp_Id\\":\\"1487\\", \\"Emp_LastName\\":\\"Stephens\\",\\"Emp_FirstName\\":\\"Mark\\",\\"Dept\\":\\"Support\\"}"}'); ``` ```sql SELECT * FROM test_json; ``` ```output +------------------------------------------------------------------------------------+ | col_a | +------------------------------------------------------------------------------------+ | {"addParams":"{\"Emp_Id\":\"1487\", | | \"Emp_LastName\":\"Stephens\",\"Emp_FirstName\":\"Mark\",\"Dept\":\"Support\"}"} | +------------------------------------------------------------------------------------+ ``` ## Using Colon-Colon Notation in UPDATE Queries You can use the colon-colon notation for JSON keypaths in [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) queries. For example, the following two `UPDATE` queries perform the same operation: ```sql UPDATE users SET userdata::name::$first = 'Alex'; UPDATE users SET userdata = JSON_SET_STRING(userdata, 'name', 'first', 'Alex'); ``` In fact, these access operators are simply convenient aliases for the `JSON_SET_` built-in function (see [JSON\_SET\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-set-type.md)), and they follow all the same rules. ## Field Name Syntax for JSON Access and UPDATE Queries In both the JSON access and update contexts, each field name must either: * follow the syntax for a valid SQL identifier, or * be escaped with backticks in the same manner as a SQL identifier For example, the following two [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) queries perform the same operation: ```sql SELECT ticker_symbol FROM stocks WHERE statistics::%`P/E` > 1.5; SELECT ticker_symbol FROM stocks WHERE JSON_EXTRACT_DOUBLE(statistics, 'P/E') > 1.5; ``` When using the `JSON_EXTRACT_` form of the query (see [JSON\_EXTRACT\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md) ), there is no constraint on the contents of the keystring. A JSON object can contain any string as a key, even `""`, or a string with `"embedded\u0000nulls"`. For more information on extracting JSON data, see [JSON LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data/#UUID-390757f0-40b4-4d47-acec-d7c9a4c84ceb.md). ## JSON Performance The following example will demonstrate how to improve JSON seekability by utilizing sub-segments. Begin by turning off or disabling the variable: ```sql SET GLOBAL use_seekable_json = OFF; ``` Create a database and change to the newly created database.. ```sql CREATE DATABASE orders; USE orders; ``` Create a table with id as an integer and set as the primary key ```sql CREATE TABLE orders(id INT PRIMARY KEY not null, order_doc json, sort KEY(id)); ``` Create a stored procedure to generate documents until it has reached a specific number nearest to the power of two > = n. ```sql DELIMITER // CREATE OR REPLACE PROCEDURE gen_orders(n INT) AS DECLARE i INT; c INT; m INT; BEGIN DELETE FROM orders; INSERT INTO orders VALUES ( 1, '{ "id" : 1, "time" : "2022-11-17 23:03:54", "lineitems" : [ { "line_num" : 1, "SKU" : 688968, "description" : "Apple iPhone 14 Plus", "qty" : 1 }, { "line_num" : 2, "SKU" : 6514052, "description" : "Energizer - MAX AA Batteries (8 Pack), Double A Alkaline Batteries", "qty" : 3 }, { "line_num" : 3, "SKU" : 6457697, "description" : "Star Wars - Galactic Snackin Grogu", "qty" : 1 } ] }' ); i = 1; WHILE i < n LOOP SELECT MAX(id) INTO m FROM orders; INSERT INTO orders SELECT id + m, new_order_doc(order_doc, id + m) FROM orders; SELECT COUNT(*) INTO c FROM orders; i += c; END LOOP; END // DELIMITER ; ``` Create a function that will change the input order\_doc into a new one with random data, and a new id. ```sql DELIMITER // CREATE OR REPLACE FUNCTION new_order_doc(order_doc JSON, new_id INT) RETURNS JSON AS DECLARE j JSON; lineitems JSON; l1 JSON; l2 JSON; l3 JSON; BEGIN j = json_set_double(order_doc, "id", new_id); j = json_set_string(j, "time", NOW()); lineitems = json_extract_json(j, "lineitems"); l1 = json_extract_json(lineitems, 0); l2 = json_extract_json(lineitems, 1); l3 = json_extract_json(lineitems, 2); lineitems = json_array_push_json('[]', mutate_lineitem(l1)); lineitems = json_array_push_json(lineitems, mutate_lineitem(l2)); lineitems = json_array_push_json(lineitems, mutate_lineitem(l3)); j = json_set_json(j, "lineitems", lineitems); RETURN j; END // DELIMITER ; ``` Create a function to use the mutate\_lineitem. ```sql DELIMITER // CREATE OR REPLACE FUNCTION mutate_lineitem(item JSON) RETURNS JSON AS DECLARE j JSON; BEGIN j = json_set_double(item, "SKU", FLOOR(RAND() * 10 * 1000 * 1000)); j = json_set_string(j, "description", UUID()); j = json_set_double(j, "qty", CEIL(3 * RAND())); RETURN j; END // DELIMITER ; ``` Generate eight million rows of test data. ```sql CALL gen_orders(8*1024*1024); ``` Create a stored procedure that will measure the average lookup time. ```sql DELIMITER // DELIMITER // CREATE OR REPLACE PROCEDURE get_avg_time(n INT) AS DECLARE _id INT; _od JSON; _oid INT; m INT; st DATETIME(6); et DATETIME(6); BEGIN SELECT MAX(id) INTO m FROM orders; st = NOW(6); FOR i IN 1..n LOOP _oid = CEILING(m * RAND()); SELECT id, order_doc INTO _id, _od FROM orders WHERE id = _oid; END LOOP; et = NOW(6); ECHO SELECT (TIMESTAMPDIFF(MICROSECOND, st, et) / 1000000.0) / n AS avg_time; END // DELIMITER ; ``` Before obtaining the average query time make sure the table has been optimized and all the data is in columnstore format. ```sql OPTIMIZE TABLE orders FULL; ``` Obtain average query time. ```sql CALL get_avg_time(100); ``` ```output +------------+ | avg_time | +------------+ | 1.73598285 | +------------+ 1 row in set (2 min 53.61 sec) ``` To measure the new query time, enable to variable. ```sql SET GLOBAL use_seekable_json = ON ``` Create a second table using the same format as the first table. ```sql CREATE TABLE orders2 LIKE orders; ``` Use the same data from the first table and make sure to optimize the new table. ```sql INSERT INTO orders2 SELECT * FROM orders; ``` ```sql OPTIMIZE TABLE orders2 FULL; ``` Update the stored procedure `get_avg_time` by changing *orders* to *orders2* in both places. ```sql DELIMITER // CREATE OR REPLACE PROCEDURE get_avg_time(n INT) AS DECLARE _id INT; _od JSON; _oid INT; m INT; st DATETIME(6); et DATETIME(6); BEGIN SELECT MAX(id) INTO m FROM orders2; st = NOW(6); FOR i IN 1..n LOOP _oid = CEILING(m * RAND()); SELECT id, order_doc INTO _id, _od FROM orders2 WHERE id = _oid; END LOOP; et = NOW(6); ECHO SELECT (TIMESTAMPDIFF(MICROSECOND, st, et) / 1000000.0) / n AS avg_time; END // DELIMITER ; ``` Run the `get_avg_time` stored procedure again with the changed table name. ```sql CALL get_avg_time(100); ``` ```output +------------+ | avg_time | +------------+ | 0.03216373 | +------------+ 1 row in set (3.47 sec) ``` ## JSON Array Performance Aggregations over array objects using the `JOIN TABLE(JSON_TO_ARRAY())` operation are optimized in SingleStore to run efficiently on large batches of rows in parallel. The optimization also reduces the amount of data read from the JSON column by retrieving only the necessary data instead of fetching the entire JSON object. The `JSON_TO_ARRAY` optimization requires the column created by the `TABLE` function (`table_col`) be wrapped in a `JSON_EXTRACT_()` function. A `JSON_EXTRACT_()` function can be used to directly extract a value as shown in Example 7 in the Introduction, or the shorthand (`::`) syntax can be used when there are keys to extract, as is shown in the following examples. The following example uses the [TPC-H](https://www.tpc.org/tpch/) data. [Load this TPC-H data](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-tpc-h-data-into-singlestore.md) into your cluster, and run the following queries to modify the data for the current use case: ```sql ALTER TABLE orders ADD COLUMN lineitems_json JSON; CREATE TABLE new_table ( l_orderkey bigint(11) NOT NULL, lineitems JSON ); INSERT INTO new_table SELECT l_orderkey, JSON_AGG( JSON_BUILD_OBJECT( 'l_partkey', l_partkey, 'l_suppkey', l_suppkey, 'l_linenumber', l_linenumber, 'l_quantity', l_quantity, 'l_extendedprice', l_extendedprice, 'l_discount', l_discount, 'l_tax', l_tax, 'l_returnflag', l_returnflag, 'l_linestatus', l_linestatus, 'l_shipdate', l_shipdate, 'l_commitdate', l_commitdate, 'l_receiptdate', l_receiptdate, 'l_shipinstruct', l_shipinstruct, 'l_shipmode', l_shipmode, 'l_comment', l_comment ) ) as lineitems FROM lineitem GROUP BY l_orderkey; UPDATE orders o JOIN new_table t ON o.o_orderkey = t.l_orderkey SET o.lineitems_json = t.lineitems; ``` Here are a few sample query structures that benefit from the optimization. * The following query calculates the total number of line items for each shipping mode by aggregating and grouping on the JSON array columns: ```sql SELECT t.table_col::$l_shipmode, sum(t.table_col::%l_quantity) as quantity FROM orders JOIN TABLE(JSON_TO_ARRAY(lineitems_json)) t GROUP BY t.table_col::$l_shipmode; ``` ```output +--------------------------+-----------+ | t.table_col::$l_shipmode | quantity | +--------------------------+-----------+ | MAIL | 218541962 | | FOB | 218469194 | | RAIL | 218553230 | | AIR | 218525124 | | REG AIR | 218617001 | | SHIP | 218576988 | | TRUCK | 218454537 | +--------------------------+-----------+ ``` * The following query calculates the total number of line items for each order priority: ```sql SELECT o_orderpriority as priority, sum(t.table_col::$l_quantity) as quantity FROM orders JOIN TABLE(JSON_TO_ARRAY(lineitems_json)) t GROUP BY o_orderpriority; ``` ```output +-----------------+-----------+ | priority | quantity | +-----------------+-----------+ | 3-MEDIUM | 305731841 | | 5-LOW | 305656541 | | 2-HIGH | 306112515 | | 1-URGENT | 306300507 | | 4-NOT SPECIFIED | 305936632 | +-----------------+-----------+ ``` * The following query performs an intricate aggregation operation over multiple fields: ```sql SELECT t.table_col::$l_returnflag as r, t.table_col::$l_linestatus as s, sum(t.table_col::%l_quantity) as sum_qty, sum(t.table_col::%l_extendedprice) as sum_base_price, sum(t.table_col::%l_extendedprice * (1 - t.table_col::%l_discount)) as sum_disc_price, avg(t.table_col::%l_quantity) as avg_qty FROM orders JOIN TABLE(JSON_TO_ARRAY(lineitems_json)) t GROUP by r, s ``` ```output +---+---+-----------+--------------------+--------------------+--------------------+ | r | s | sum_qty | sum_base_price | sum_disc_price | avg_qty | +---+---+-----------+--------------------+--------------------+--------------------+ | A | F | 377518399 | 566065727797.2633 | 537759104278.07465 | 25.500975103007097 | | R | F | 377732830 | 566431054976.0034 | 538110922664.7574 | 25.50838478968014 | | N | F | 9851614 | 14767438399.169971 | 14028805792.211407 | 25.522448302840946 | | N | O | 764635193 | 1146548935600.9531 | 1089215873201.9531 | 25.498214140059787 | +---+---+-----------+--------------------+--------------------+--------------------+ ``` ## Limitation * The optimization is only applied to the first dimension of an array. ## Implementation Considerations This section describes some unique behaviors in SingleStore Helios’s implementation of the JSON standard (RFC 4627). ## Infinities and NaNs SingleStore Helios does not attempt to support entities such as the `nan` (“not a number”) entity, as in `{"value":nan}`. Although the JavaScript language supports `nan` and positive and negative infinities, neither the JSON standard nor SQL’s `DOUBLE` type provide any way to represent such non-finite values. If your application requires these special values, you might try using `null`, `0.0`, or strings (such as `"nan"` and `"inf"`) in their place. ## Unicode Support JSON strings can contain any of the 65536 Unicode characters in the Basic Multilingual Plane (BMP), either directly as themselves or escaped with JSON’s `\uABCD` notation. JSON normalization converts all such ASCII escape sequences into UTF-8 byte sequences, except for those characters that must remain `\u`-encoded to keep the string valid JSON. > **⚠️ Warning**: Characters outside of the Basic Multilingual Plans in JSON strings are not supported with escaped notation and may result in incorrect results. In SingleStore, a backslash (`\`) literal in a string must be escaped with a backslash. Therefore, pattern strings containing backslashes will have two backslash characters (`\\`). Here is an example of how to use escaped notation for characters in the BMP: ```sql SELECT '{"a":"\\u00F9"}' :> JSON; ``` ```output +---------------------------+ | '{"a":"\\u00F9"}' :> JSON | +---------------------------+ | {"a":"ù"} | +---------------------------+ ``` Like MySQL, SingleStore Helios supports characters outside the Basic Multilingual Plane (characters whose codepoints are in the range 0x10000 to 0x10FFFF). This includes some uncommon Chinese characters and symbols such as emojis. The `json_collation` engine variable must be set to `utf8mb4_bin` for these characters to work. Refer to [Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-2.md) for more information. ## Character Encodings and Collation `Collation` provides a set of rules to use when comparing characters in a character set. By default, JSON columns use the collation specified in the `json_collation` engine variable ([Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-3.md)). The collation of a JSON column affects the following: * Output of `SELECT DISTINCT`, `ORDER BY`, and other queries that compare entire values. * Identification of duplicate keys inside a single JSON object during normalization. For example, whether the string `{"Schlüssel":1,"Schluessel":2}` is normalized to `{"Schluessel":2}`. * Sorting of keys inside a single JSON object. For example, whether the string `{"Schlüssel":1,"Schluff":2}` is normalized to `{"Schluff":2,"Schlüssel":1}`. The default collation generally provides the desired behavior. However, you can override the default at the table or column level or by using the `json_collation` engine variable. > **📝 Note**: SingleStore recommends that you use the `utf8mb4_bin` collation for all JSON columns. ## Table-level Override In the following example, for the `users` table, both `userdata` and `lastname` use the table collation (which has been specified as `utf8_unicode_ci`). ```sql CREATE TABLE users ( uid INT AUTO_INCREMENT PRIMARY KEY, userdata JSON, lastname AS data::name::$last PERSISTED TEXT) COLLATE=utf8_unicode_ci; ``` ## Column-level Override In the following example, for the `orders` table: * The `data` column uses `utf8_unicode_ci`. * The `productdetails` column uses `utf8_bin`. * Both the `productname` column and the `comments` column use `utf8_general_ci`, which is SingleStore Helios’s default database collation ```sql CREATE TABLE orders ( oid INT AUTO_INCREMENT PRIMARY KEY, data JSON COLLATE utf8_unicode_ci, productname AS data::product::$name PERSISTED TEXT, productdetails AS data::product::$details PERSISTED TEXT COLLATE utf8_bin, comments VARCHAR(256)); ``` ## Expression-level Override The `:>` operator allows you to specify a collation for different expressions in a query. ```sql :> text collate ``` The following example uses a binary collation (`utf8_bin`) for the first expression and a case-insensitive collation (`utf8_general_ci`) for the second expression. ```sql SELECT * FROM sets WHERE sets.json_field::$x :> text collate utf8_bin = "string1" AND sets.json_field::$y :> text collate utf8_general_ci = "string2"; ``` ## Maximum JSON Value Size Under the hood, JSON data is stored in `LONGBLOB` columns. While the DDL will allow you to specify a length of up to 4GB, there is an internal limit applied when assigning a value to a string or JSON field that caps the maximum size of a single value to `max_allowed_packet`. This is 100MB by default and can be set to up to 1GB. ## Performance Impact on Parquet Encoding for JSON When using seekable JSON/JSON Parquet encoding a major impact on performance is if a schema is dense or sparse. A node in a tree is considered to be dense if it occurs in most JSON rows. A schema is said to be dense if most of the nodes in the entire schema are dense. Otherwise, the schema is considered to be sparse. Consider the following JSON data table: | JSON Data (dense) | | ----------------------- | | `{"a":1, "b":1}` | | `{“a”:2, “c”:{“d”: 1}}` | | `{“c”:{“d”:2, “e”:3}}` | The schema will infer the JSON rows as follows: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4c1dfd3f6ed174a9/6a7ada7e6c6eac552ef08cd8/Dense_Schema-672ujG.png) Using the image above the leaves of the tree become internal columns in the JSON encoding. This is an example of a dense schema. | a | b | c::d | c::e | | ---- | ---- | ---- | ---- | | 1 | 1 | NULL | NULL | | 2 | NULL | 1 | NULL | | NULL | NULL | 2 | 3 | Using the following JSON data table: | JSON Data (sparse) | | ------------------ | | `{“a”: 1}` | | `{“b”: 1}` | | `{“c”: 1}` | | `{“d”: 1}` | | `{“e”: 1}` | The JSON rows will be encoded as follows: | a | b | c | d | e | | ---- | ---- | ---- | ---- | ---- | | 1 | NULL | NULL | NULL | NULL | | NULL | 1 | NULL | NULL | NULL | | NULL | NULL | 1 | NULL | NULL | | NULL | NULL | NULL | 1 | NULL | | NULL | NULL | NULL | NULL | 1 | The preceding table represents a sparse schema caused by the NULLs not being part of the original JSON strings. This results in poor performance since the NULLs will need to be counted which increases the execution time and memory usage. To prevent an overly sparse schema from being inferred, SingleStore uses a method where if the average of a key’s children is too low (<1%) in relation to the key itself, the key is stored as a string without inferring its children. ## Related Topics * See: [JSON\_AGG](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-agg/#section-idm4605787636892833501997993222.md) * [JSON\_COLUMN\_SCHEMA](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/json-column-schema.md) * Training: [Working with JSON](https://training.singlestore.com/learn/course/internal/view/elearning/632/working-with-json-data) *** Modified at: July 14, 2026 Source: [/cloud/create-a-database/using-json/](https://docs.singlestore.com/cloud/create-a-database/using-json/) (An index of the documentation is available at /llms.txt) --- # Using Persistent Computed Columns A computed column is a column defined by an expression that uses other columns in the table. SingleStore Helios allows users to create persistent computed columns defined by an expression that combines other columns, constants, built-in functions, and operators. SingleStore Helios’s computed columns are fully materialized and can be indexed like a standard column. There are storage and performance tradeoffs to consider when using computed columns. At a high level, computed columns are a way to optimize computationally expensive read queries that use built-in functions or require additional data processing. They allow users to precompute values that would otherwise have to be computed as part of the execution of a read query. However, computed columns also consume additional storage and require more computation on writes. Some example use cases for computed columns include: * Parsing JSON objects for improved read performance * Precomputing a value using an expression that includes values from other columns in the table * Extracting values from a column, such as a year from a timestamp or a domain from a URL * Pre-materializing common expressions in queries allowing them to be used in high-order operations, such as segment elimination or encoded `GROUP BY`s (see [Encoded Data in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore/encoded-data-in-columnstores.md) for more information) * Using computed columns as sort keys, such as a timestamp type computed column (see [Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md) for more information on sort keys) * When filtering on a query, using computed column names isn't necessary to get faster performance. If there is an index or a shard key on the computed column, then the optimizer will match the expression in the `WHERE` clause and use the computed column to execute the query. An index on a computed column can speed up an existing query by orders of magnitude without rewriting the query. See an example below: > **📝 Note**: It is a best practice to use large storage size data types for persistent computed columns, e.g. `BIGINT` or `LONGTEXT`. This ensures the expression result will fit into the computed column. ```sql CREATE TABLE t (a INT, b INT, c as a + b PERSISTED BIGINT, sort key(c)); SELECT * FROM t WHERE c = ; SELECT * FROM t WHERE a + b = ; ``` Either query is acceptable. The second query will also use column c to filter and does not have to compute a + b. * Data type mismatch within the computed column and or parameterization of a computed column can prevent matching. Warnings will inform you if there is an issue with computed column matching: ```sql COMPILE ; SHOW WARNINGS; ``` ```output An example warning is: SHOW WARNINGS; +---------+------+---------------------------------------------------------------------------+ | Level | Code | Message | +---------+------+---------------------------------------------------------------------------+ | Warning | 2626 | Prospect computed column: . of type text | | | | CHARACTER SET utf8 COLLATE utf8_general_ci NULL cannot suit expression of | | | | type longtext CHARACTER SET utf8 COLLATE utf8_general_ci NULL. | +---------+------+---------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` In the above case, change the `column_name` type to `LONGTEXT` and the matching will work as expected. * Using computed columns helps to maintain data independence by keeping the physical structure of data independent from application logic. For example, if your application requires data that is a combination of two or more columns within a table, a persistent computed column plus computed column matching can be used to speed up a query. This can be done without having to modify the query to refer to the computed column (a physical structure created to optimize performance) by name. * Promoting and indexing JSON fields is the main use case of persistent computed columns. See [Using JSON](https://docs.singlestore.com/cloud/create-a-database/using-json.md) for more information. Persistent computed columns can be created as part of a [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) statement, or can be added later using [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md). ## Computed Column Examples The following statement creates a table `col_test` with an `INT` column `col_a` and a persistent computed column `col_b` that is defined using an expression that takes the value in `col_a` and adds `1` to it. ```sql CREATE TABLE cal_test (col_a INT PRIMARY KEY, col_b AS col_a + 1 PERSISTED BIGINT, SORT KEY(col_b)); ``` Inserting values into column `col_a` will automatically populate column `col_b` using the column definition expression. ```sql INSERT INTO cal_test (col_a) VALUES (1), (2); SELECT * FROM cal_test; ``` ```output +-------+-------+ | col_a | col_b | +-------+-------+ | 2 | 3 | | 1 | 2 | +-------+-------+ ``` It is possible to create a table with computed columns initialized using a `NOT NULL` constraint, just like a non-computed column. ```sql CREATE TABLE cal_test (col_a int NOT NULL, col_b AS col_a+1 PERSISTED BIGINT NOT NULL); DESC cal_test; ``` ```output +-------+---------+------+------+---------+----------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+------+---------+----------+ | col_a | int(11) | NO | | NULL | | | col_b | int(11) | NO | | NULL | computed | +-------+---------+------+------+---------+----------+ ``` It is also possible to add a computed column to an existing table using [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md). ```sql CREATE TABLE cal_test (col_a INT, col_b INT); INSERT INTO cal_test (col_a, col_b) VALUES (15, 9), (45, 81); ALTER TABLE cal_test ADD col_c AS (col_a + col_b/2) PERSISTED BIGINT; SELECT * FROM cal_test; ``` ```output +---+-------------------+ | col_a | col_b | col_c | +-------+-------+-------+ | 15 | 9 | 20 | | 45 | 81 | 86 | +-------+-------+-------+ ``` ```sql DESC cal_test; ``` ```output +-------+---------+------+-----+---------+----------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+----------+ | col_a | int(11) | YES | | NULL | | | col_b | int(11) | YES | | NULL | | | col_c | int(11) | YES | | NULL | computed | +-------+---------+------+-----+---------+----------+ ``` Adding a computed column with a `NOT NULL` constraint using `ALTER TABLE` will result in an error. ```sql ALTER TABLE cal_test ADD col_c AS (col_a + col_b/2) PERSISTED BIGINT NOT NULL; ``` ```output ERROR 1064 ER_PARSE_ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NOT NULL' at line 1 ``` Attempts to insert values directly into a computed column will result in an error, even if the values are “correct” with respect to the column definition. ```sql INSERT INTO cal_test (col_a, col_b, col_c) VALUES (3, 4, 2); ``` ```output ERROR 1822 ER_MEMSQL_COMPUTED_INSERT: Cannot insert into computed column 'col_c' ``` A computed column can be created using any expression that outputs a value that is a valid SingleStore Helios data type, composed of operators and built-in functions. For example, computed columns allow you to extract values from a JSON blob so they can be scanned and read more quickly. However, note that aggregations, user-defined functions, and non-deterministic functions such as `rand()` and `now()` cannot be used in computed columns. ```sql SELECT * FROM j; ``` ```output +---+-------------------------------------------------+ | a | b | +---+-------------------------------------------------+ | 1 | {"name":"Sandy Beach","occupation":"lifeguard"} | +---+-------------------------------------------------+ ``` ```sql ALTER TABLE j ADD COLUMN occupation AS b::$occupation PERSISTED LONGTEXT; SELECT * FROM j; ``` ```output +---+-------------------------------------------------+------------+ | a | b | occupation | +---+-------------------------------------------------+------------+ | 1 | {"name":"Sandy Beach","occupation":"lifeguard"} | lifeguard | +---+-------------------------------------------------+------------+ ``` Modifying values in a column used to define a computed column will also update the affected values in the computed column. ```sql UPDATE j SET b::$occupation = 'circus clown'; SELECT * FROM j; ``` ```output +---+----------------------------------------------------+--------------+ | a | b | occupation | +---+----------------------------------------------------+--------------+ | 1 | {"name":"Sandy Beach","occupation":"circus clown"} | circus clown | +---+----------------------------------------------------+--------------+ ``` ## Computed Column Matching The keyword AUTO has been added when using computed columns. This addition allows the engine to infer the type of expression used by the computed column. Instead of using the syntax: ```sql CREATE TABLE test_cal(col_a int, col_b AS col_a * col_a PERSISTED int); ``` Use the word `AUTO` in place of the expression type: ```sql CREATE TABLE auto_cal(col_a int, col_b AS col_a * col_a PERSISTED AUTO); ``` This example uses the table created above and specifies the expression type. ```sql EXPLAIN SELECT col_a, col_a*col_a FROM test_cal; ``` ```output +-------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------+ | WARNING: Computed column test_cal.col_b of type [int(11) NULL] | | does not match similar expression's type. Consider changing the | | computed column's type to that of the expression [bigint(21) | | NULL] | | Project [test_cal.col_a, test_cal.col_a * test_cal.col_a AS | | 'col_a*col_a'] | | | | ColumnStoreScan ticket_test.test_cal, KEY __UNORDERED () USING | | CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +-------------------------------------------------------------------+ ``` The following uses the table created above and uses `AUTO` for the expression type. ```sql EXPLAIN SELECT col_a, col_a*col_a FROM auto_cal; ``` ```output +-------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------+ | Project [auto_cal.col_a, auto_cal.col_b AS `col_a*col_a`] | | ColumnStoreScan ticket_test.auto_cal, KEY __UNORDERED () USING | | CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +-------------------------------------------------------------------+ ``` Use `SHOW CREATE TABLE ` to see the persisted column expression when the expression type is specified. ```sql SHOW CREATE TABLE test_cal; ``` ```output +-----------+------------------------------------------------------+ | Table | Create Table | +-----------+------------------------------------------------------+ | test_cal | CREATE TABLE `test_cal` (`col_a` int(11) DEFAULT | | | NULL,`col_b` as col_a * col_a PERSISTED int(11), | | | KEY `__UNORDERED` () USING CLUSTERED COLUMNSTORE, | | | SHARD KEY () )AUTOSTATS_CARDINALITY_MODE=INCREMENTAL | | | AUTOSTATS_HISTOGRAM_MODE=CREATE AUTOSTATS_SAMPLING=ON| | | SQL_MODE='STRICT_ALL_TABLES' | +-----------+------------------------------------------------------+ ``` Use `SHOW CREATE TABLE ` to see the persisted column and when using `AUTO`. ```sql SHOW CREATE TABLE auto_cal; ``` ```output +-----------+------------------------------------------------------+ | Table | Create Table | +-----------+------------------------------------------------------+ | auto_cal | CREATE TABLE `auto_cal` (`col_a` int(11) DEFAULT | | | NULL,`col_b` as col_a * col_a PERSISTED bigint(21), | | | KEY `__UNORDERED` () USING CLUSTERED COLUMNSTORE, | | | SHARD KEY () )AUTOSTATS_CARDINALITY_MODE=INCREMENTAL | | | AUTOSTATS_HISTOGRAM_MODE=CREATE AUTOSTATS_SAMPLING=ON| | | SQL_MODE='STRICT_ALL_TABLES' | +-----------+------------------------------------------------------+ ``` > **📝 Note**: **Known Limitation**If you define a persistent computed column with `PERSISTED AUTO NOT NULL`, the `NOT NULL` constraint is not enforced. The column is created as nullable, and the `NOT NULL` constraint is silently dropped. If you need the computed column to be `NOT NULL`, specify an explicit type instead of `AUTO`. For example, using `PERSISTED DOUBLE NOT NULL` over `PERSISTED AUTO NOT NULL`. *** Modified at: July 10, 2026 Source: [/cloud/create-a-database/using-persistent-computed-columns/](https://docs.singlestore.com/cloud/create-a-database/using-persistent-computed-columns/) (An index of the documentation is available at /llms.txt) --- # Developer Resources Information developers need to know to create applications using SingleStore, from getting started, to migrating applications, to connecting with various application development languages and tools: * Connect to application development tools such as C/C++, Java, Ruby, ODBC/JDBC, Perl, Python, and many more. * Use SingleStore's Data API to develop custom applications and build seamless integrations with applications. * Use `Management` API to create and manage clusters. * View samples of concurrent multi-inserts for Bash, C, C# / .NET Core, Java, Node.js, and Python. * Perform operations based on features like time series data analysis, full-text search, geospatial features, and window functions. * View the [SQL command reference](https://docs.singlestore.com/cloud/reference/sql-reference/sql-command-syntax.md). * Create custom programs such as SPs, UDFs, TVFs, and UDAFs. ## See Also * Training: [SingleStore Helios Procedural SQL](https://training.singlestore.com/learn/course/internal/view/elearning/636/procedural-sql) * [Procedural SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference.md) * [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) ## In this section * [Try SingleStore Helios for Free](https://docs.singlestore.com/cloud/developer-resources/try-singlestore-helios-for-free.md) * [Connect with Application Development Tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md) * [Database Branching](https://docs.singlestore.com/cloud/developer-resources/database-branching.md) * [Connect with Data API](https://docs.singlestore.com/cloud/developer-resources/connect-with-data-api.md) * [Management API](https://docs.singlestore.com/cloud/developer-resources/management-api.md) * [Porting Tables to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/porting-tables-to-singlestore-helios.md) * [Concurrent Multi-Insert Examples](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples.md) * [Unsupported MySQL Features](https://docs.singlestore.com/cloud/developer-resources/unsupported-mysql-features.md) * [Transition from MySQL to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/transition-from-mysql-to-singlestore-helios.md) * [Functional Extensions](https://docs.singlestore.com/cloud/developer-resources/functional-extensions.md) * [Procedural Extensions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions.md) * [Secrets](https://docs.singlestore.com/cloud/developer-resources/secrets.md) * [Guides](https://docs.singlestore.com/cloud/developer-resources/guides.md) *** Modified at: July 15, 2026 Source: [/cloud/developer-resources/](https://docs.singlestore.com/cloud/developer-resources/) (An index of the documentation is available at /llms.txt) --- # Concurrent Multi-Insert Examples To perform a trickle load into SingleStore, you can run concurrent processes that each load data in batches using the `INSERT` command to insert up to several thousand rows in each statement. When loading a large volume of data, loading a row at a time is a resource-intensive and time-consuming process due to per-statement overhead, so batching is preferred. For bulk loading into SingleStore, you can use the `LOAD DATA` command or [pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md). `LOAD DATA` is preferable for a small number of files that are not extremely large, loaded in an initial step before working with the data. For very large data sets where parallel loading is important, pipelines are preferred. Moreover, pipelines can perform continuous loads as new files or messages arrive. The following examples demonstrate how to perform concurrent multi-inserts with different tools/languages. ## In this section * [Bash](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/bash.md) * [C](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/c.md) * [C# / .NET Core](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/c-net-core.md) * [Java](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/java.md) * [Node.js](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/node-js.md) * [Python](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/python.md) *** Modified at: March 14, 2023 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/) (An index of the documentation is available at /llms.txt) --- # Bash **Dependencies** * `singlestore` client * Bourne-Again Shell (`bash`) **Code** ```shell #!/bin/bash MHOST="127.0.0.1" MPORT="3306" MUSER="root" MDB="" NUM_WORKERS=128 BATCH_SIZE=256 memsql_exec() { singlestore -h $MHOST -P $MPORT -u $MUSER $MDB -e "$1" } memsql_exec_multi() { singlestore -h $MHOST -P $MPORT -u $MUSER $MDB -e \ "$(for ((b = 0; b < $BATCH_SIZE; b++)); do echo "$1;" done)" } echo "Creating database test" memsql_exec "CREATE DATABASE IF NOT EXISTS test" MDB="test" echo "Creating table tbl" memsql_exec "CREATE TABLE IF NOT EXISTS tbl (id INT AUTO_INCREMENT PRIMARY KEY)" echo "Launching $NUM_WORKERS workers" sleep 1 declare -a WORKERS for ((worker = 0; worker < $NUM_WORKERS; worker++)); do (while [ 1 ]; do echo "Worker $worker inserting" memsql_exec_multi "INSERT INTO tbl VALUES (NULL)" done) & WORKERS[$worker]=$! done sleep 10 for ((worker = 0; worker < $NUM_WORKERS; worker++)); do echo "Killing worker $worker" kill ${WORKERS[$worker]} wait ${WORKERS[$worker]} 2>/dev/null done echo "Cleaning up" sleep 1 memsql_exec "DROP DATABASE test" ``` *** Modified at: May 14, 2026 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/bash/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/bash/) (An index of the documentation is available at /llms.txt) --- # C# / .NET Core **Dependencies** * [SingleStore Connector for .NET and .NET Core](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core.md) **Code** ```C# using System; using System.Collections.Generic; using System.Data; using System.Diagnostics; using System.Threading; using System.Threading.Tasks; using SingleStoreConnector; namespace SingleStoreDBTest { public class SingleStoreDBTest { /** * Tweak the following globals to fit your environment * ################################################### */ public const string HOST = "svc-3nb226d8-ee13-47f0-8ca4-2dc820773442-dml.aws-oregon-2.svc.singlestore.com"; public const int PORT = 3306; public const string USER = "admin"; public const string PASSWORD = "passkey"; // Specify which database and table to work with. // Note: this database will be dropped at the end of this script public const string DATABASE = "test"; public const string TABLE = "tbl"; // The number of workers to run public const int NUM_WORKERS = 20; // Run the workload for this many seconds public const int WORKLOAD_TIME = 10; // seconds // Batch size to use public const int BATCH_SIZE = 5000; /** * Internal code starts here * ######################### */ private IDbCommand dbCommand; private string insertCommand; private void GetDbCommand() { IDbConnection conn = new SingleStoreConnection(); conn.ConnectionString = $"Server={HOST};Port={PORT};Uid={USER};Pwd={PASSWORD};"; conn.Open(); dbCommand = conn.CreateCommand(); string[] _batch = new string[BATCH_SIZE]; Array.Fill(_batch, "(DEFAULT)"); insertCommand = $"INSERT INTO {TABLE} VALUES {string.Join(",", _batch)}"; } private void SetupTestDb() { dbCommand.CommandText = $"CREATE DATABASE IF NOT EXISTS {DATABASE}"; dbCommand.ExecuteNonQuery(); dbCommand.CommandText = $"USE {DATABASE}"; dbCommand.ExecuteNonQuery(); dbCommand.CommandText = $"CREATE TABLE {TABLE} (id int primary key auto_increment)"; dbCommand.ExecuteNonQuery(); } private void Warmup() { Console.WriteLine("Warming up workload"); dbCommand.CommandText = insertCommand; dbCommand.ExecuteNonQuery(); // FRAGILE: included in count, not included in time } private void DoBenchmark() { Console.WriteLine($"Launching {NUM_WORKERS} workers for {WORKLOAD_TIME} sec"); Thread[] workers = new Thread[NUM_WORKERS]; for(int i = 0; i < NUM_WORKERS; i++) { workers[i] = new Thread(new ThreadStart(Worker)); workers[i].Start(); } Console.WriteLine($"{workers.Length} workers running..."); for(int i = 0; i < NUM_WORKERS; i++) { workers[i].Join(); } } private void Worker() { // Create another connection per thread using (IDbConnection conn = new SingleStoreConnection()) { conn.ConnectionString = $"Server={HOST};Port={PORT};database={DATABASE};Uid={USER};Pwd={PASSWORD};SslMode=None;"; conn.Open(); using (IDbCommand dbCommand = conn.CreateCommand()) { dbCommand.CommandText = insertCommand; Stopwatch stop = new Stopwatch(); stop.Start(); while(stop.ElapsedMilliseconds < WORKLOAD_TIME*1000) { dbCommand.ExecuteNonQuery(); } } } } private void ShowStats() { dbCommand.CommandText = $"USE {DATABASE}"; dbCommand.ExecuteNonQuery(); dbCommand.CommandText = $"SELECT COUNT(*) FROM {TABLE}"; using (IDataReader reader = dbCommand.ExecuteReader()) { long count = 0; while(reader.Read()) { count = (long)reader["COUNT(*)"]; } Console.WriteLine($"{count} rows inserted using {NUM_WORKERS} workers"); Console.WriteLine($"{count / WORKLOAD_TIME} rows per second"); } } private void CleanupTestDb() { if (dbCommand != null) { Console.WriteLine("Cleaning up"); dbCommand.CommandText = $"USE `information_schema`"; dbCommand.ExecuteNonQuery(); dbCommand.CommandText = $"DROP DATABASE IF EXISTS {DATABASE}"; dbCommand.ExecuteNonQuery(); dbCommand = null; } } public static int Main(string[] args) { SingleStoreDBTest tester = new SingleStoreDBTest(); try { tester.GetDbCommand(); tester.SetupTestDb(); tester.Warmup(); tester.DoBenchmark(); tester.ShowStats(); tester.CleanupTestDb(); return 0; } catch (Exception ex) { Console.WriteLine($"ERROR: {ex.Message}, {ex.GetType()}, {ex.StackTrace}"); try { tester.CleanupTestDb(); } catch { // ignore error } return 1; } } } } ``` *** Modified at: September 7, 2023 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/c-net-core/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/c-net-core/) (An index of the documentation is available at /llms.txt) --- # C **Dependencies** * C compiler (e.g. gcc) * `pthreads` library (present on most Linux distributions) * `mysqlclient` library, available from the `libmysqlclient-dev` package on Debian-based distributions. **Code** ```C /* Compile with: * * cc multi_threaded_inserts.c -lmysqlclient -pthread -o mti */ #include #include #include const static char *host = "127.0.0.1"; const static char *user = "root"; const static char *passwd = ""; const static size_t port = 3306; #define NUM_WORKERS 20 static volatile int keep_going = 1; void *insert_worker(void *worker_id); int main() { my_init(); MYSQL conn; mysql_init(&conn); printf("Connecting to SingleStore Helios...\n"); if (mysql_real_connect(&conn, host, user, passwd, NULL, port, NULL, 0) != &conn) { printf("Could not connect to the SingleStore Helios database!\n"); goto failure; } printf("Creating database 'test'...\n"); if (mysql_query(&conn, "create database test") || mysql_query(&conn, "use test")) { printf("Could not create 'test' database!\n"); goto failure; } printf("Creating table 'tbl' in database 'test'...\n"); if (mysql_query(&conn, "create table tbl (id bigint auto_increment primary key)")) { printf("Could not create 'tbl' table in the 'test' database!\n"); goto failure; } printf("Launching %lu insert workers...\n", NUM_WORKERS); pthread_t workers[NUM_WORKERS]; size_t i; for (i = 0; i < NUM_WORKERS; ++i) { pthread_create(&workers[i], NULL, &insert_worker, (void *)i); } printf("Running inserts for %lu seconds...\n", 10); sleep(10); keep_going = 0; size_t rows_inserted = 0; for (i = 0; i < NUM_WORKERS; ++i) { size_t rows_i; pthread_join(workers[i], &rows_i); rows_inserted += rows_i; } printf("Inserted %lu rows. Cleaning up...\n", rows_inserted); if (mysql_query(&conn, "drop database test")) { printf("Could not drop the testing database 'test'!\n"); } mysql_close(&conn); return 0; failure: mysql_close(&conn); return 1; } void *insert_worker(void *worker_id) { size_t id = (size_t) worker_id; MYSQL conn; mysql_init(&conn); if (mysql_real_connect(&conn, host, user, passwd, "test", port, NULL, 0) != &conn) { printf("Worker %lu could not connect to the SingleStore Helios database! Aborting...\n", id); exit(1); } size_t i; for (i = 0; keep_going; i += 8) { if (mysql_query(&conn, "insert into tbl values (null), (null), (null)," "(null), (null), (null), (null), (null)")) { printf("Worker %lu failed to insert data, aborting...\n", id); exit(1); } } mysql_close(&conn); return (void *)i; } ``` *** Modified at: July 26, 2023 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/c/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/c/) (An index of the documentation is available at /llms.txt) --- # Java **Dependencies** * JDBC library (package `libmysql-java` on Debian-based distributions) **Code** ```Java import java.sql.*; import java.util.Properties; import java.util.concurrent.*; public class Sample { private static final String dbClassName = "com.singlestore.jdbc.Driver"; private static final String CONNECTION = "jdbc:singlestore://127.0.0.1:3306/"; private static final String USER = "root"; private static final String PASSWORD = ""; private static void executeSQL(Connection conn, String sql) throws SQLException { try (Statement stmt = conn.createStatement()) { stmt.execute(sql); } } private static void ResetEnvironment() throws SQLException { Properties p = new Properties(); p.put("user", USER); p.put("password", PASSWORD); try (Connection conn = DriverManager.getConnection(CONNECTION, p)) { for (String query: new String[] { "DROP DATABASE IF EXISTS test", "CREATE DATABASE test", "USE test", "CREATE TABLE tbl (id INT AUTO_INCREMENT PRIMARY KEY)" }) { executeSQL(conn, query); } } } private static void worker() { Properties properties = new Properties(); properties.put("user", USER); properties.put("password", PASSWORD); try (Connection conn = DriverManager.getConnection(CONNECTION, properties)) { executeSQL(conn, "USE test"); while (!Thread.interrupted()) { executeSQL(conn, "INSERT INTO tbl VALUES (NULL)"); } } catch (SQLException e) { e.printStackTrace(); } } public static void main(String[] args) throws ClassNotFoundException, SQLException, InterruptedException { Class.forName(dbClassName); ResetEnvironment(); ExecutorService executor = Executors.newFixedThreadPool(20); for (int i = 0; i < 20; i++) { executor.submit(new Runnable() { @Override public void run() { worker(); } }); } Thread.sleep(20000); executor.shutdownNow(); if (!executor.awaitTermination(5, TimeUnit.SECONDS)) { System.err.println("Pool did not terminate"); } } } ``` *** Modified at: May 18, 2026 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/java/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/java/) (An index of the documentation is available at /llms.txt) --- # Node.js **Dependencies** * [singlestore-nodejs](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js.md) dependency: run `npm install --save singlestore-nodejs`. **Code** ```javascript const singlestore = require("singlestore-nodejs"); // npm install --save singlestore-nodejs const util = require('util'); /** * Tweak the following globals to fit your environment * ################################################### */ const HOST = '127.0.0.1'; const PORT = 3306; const USER = 'root'; const PASSWORD = ''; // Specify which database and table to work with. // Note: this database will be dropped at the end of this script const DATABASE = 'test'; const TABLE = 'tbl'; // The number of workers to run const NUM_WORKERS = 20; // Run the workload for this many seconds const WORKLOAD_TIME = 10; // Batch size to use const BATCH_SIZE = 5000; /** * Internal code starts here * ######################### */ let isDone = false; // await-able setTimeout() function timeout(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } // Pre-generate the insert query const _batch = new Array(BATCH_SIZE).fill().map(_ => '()').join(','); const insertQuery = `INSERT INTO ${TABLE} VALUES ${_batch}`;; function getConnection(dbName) { return new Promise(function(resolve, reject) { const conn = singlestore.createConnection({ host: HOST, port: PORT, user: USER, password: PASSWORD, database: dbName }); conn.connect(err => { if (err) { reject(err); } else { conn.query = util.promisify(conn.query); resolve(conn); } }); }); }; async function setupTestDb() { const conn = await getConnection('information_schema'); await conn.query(`CREATE DATABASE IF NOT EXISTS ${DATABASE}`); await conn.query(`USE ${DATABASE}`); await conn.query(`CREATE TABLE IF NOT EXISTS ${TABLE} (id INT AUTO_INCREMENT PRIMARY KEY)`); } async function insertWorker() { const conn = await getConnection(DATABASE); while (true) { // await will process.nextTick() await conn.query(insertQuery); if (isDone) { break; } } } async function warmup() { console.log('Warming up workload'); const conn = await getConnection(DATABASE) await conn.query(insertQuery); // FRAGILE: included in count, not included in time } async function doBenchmark() { console.log(`Launching ${NUM_WORKERS} workers for ${WORKLOAD_TIME} sec`); const workers = []; for (let i = 0; i < NUM_WORKERS; ++i) { workers.push(insertWorker()); } console.log(`${workers.length} workers running...`); await timeout(WORKLOAD_TIME * 1000); console.log('Stopping workload'); isDone = true; await Promise.all(workers); } async function printStats() { const conn = await getConnection(DATABASE); const rows = await conn.query(`SELECT COUNT(*) AS count FROM ${TABLE}`); const count = rows[0].count; console.log(`${count} rows inserted using ${NUM_WORKERS} workers`); console.log(`${count / WORKLOAD_TIME} rows per second`); } async function cleanupTestDb() { console.log('Cleaning up'); const conn = await getConnection('information_schema'); await conn.query(`DROP DATABASE ${DATABASE}`); } async function main() { try { await setupTestDb(); await warmup(); await doBenchmark(); await printStats(); await cleanupTestDb(); } catch (err) { console.error('ERROR', err); try { await cleanupTestDb(); } catch (err2) { console.error(err2); } process.exit(1); } process.exit(0); // releases all connections } main(); ``` *** Modified at: May 18, 2026 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/node-js/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/node-js/) (An index of the documentation is available at /llms.txt) --- # Python **Dependencies** * [SingleStore Python Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) **Code** > **📝 Note**: The following script is only a proof of concept. For production purposes or loading large amounts of data, use [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) or [pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md). ```python #!/usr/bin/env python3 import os import sys import time import threading import argparse import singlestoredb as s2 parser = argparse.ArgumentParser() parser.add_argument("--host", default=None, help="The hostname of the SingleStore Helios node to connect to") parser.add_argument("--port", default=None, type=int, help="The port of the SingleStore Helios node to connect to") parser.add_argument("--user", default="root", help="The user of the SingleStore Helios node to connect to") parser.add_argument("--password", default="", help="The password of the SingleStore Helios node to connect to") parser.add_argument("--database", default="simple_benchmark", help="The database to use - note: this database should not exist") parser.add_argument("--num-workers", type=int, default=10, help="The number of insert threads") parser.add_argument("--time", type=int, default=30, help="The number of seconds to run the benchmark for") options = parser.parse_args() HOST = None PORT = None TABLE = "tbl" BATCH_SIZE = 5000 # Pre-generate the workload query QUERY_TEXT = "INSERT INTO %s (val) VALUES %s" % (TABLE, ",".join(["(1)"] * BATCH_SIZE)) def get_connection(host=None, port=None, db=options.database): """ Returns a new connection to the database. """ if host is None: host = HOST if port is None: port = PORT out = s2.connect( host=host, port=port, user=options.user, password=options.password, database=db) out.autocommit(True) return out class InsertWorker(threading.Thread): """ A simple thread which inserts empty rows in a loop. """ def __init__(self, stopping): super(InsertWorker, self).__init__() self.stopping = stopping self.daemon = True self.exception = None def run(self): with get_connection() as conn: with conn.cursor() as cur: while not self.stopping.is_set(): cur.execute(QUERY_TEXT) def test_connection(): try: with get_connection(db="information_schema") as conn: conn.is_connected() except s2.Error: print("Unable to connect to SingleStore Helios with provided connection details.") print("Please verify that SingleStore Helios is running @ %s:%s" % (HOST, PORT)) sys.exit(1) def setup_test_db(): """ Create a database and table for this benchmark to use. """ with get_connection(db="information_schema") as conn: with conn.cursor() as cur: print('Creating database %s' % options.database) try: # note: the following query will fail if there is an existing database cur.execute('CREATE DATABASE %s' % options.database) except s2.Error: print("Database %s already exists - since we drop the database at" % options.database) print("the end of this script, please specify an un-used database") print("with the --database flag.") sys.exit(1) cur.execute('USE %s' % options.database) cur.execute('CREATE TABLE IF NOT EXISTS %s (id INT AUTO_INCREMENT PRIMARY KEY, val INT)' % TABLE) def warmup(): print('Warming up workload') with get_connection() as conn: with conn.cursor() as cur: cur.execute(QUERY_TEXT) def run_benchmark(): """ Run a set of InsertWorkers and record their performance. """ stopping = threading.Event() workers = [ InsertWorker(stopping) for _ in range(options.num_workers) ] print('Launching %d workers' % options.num_workers) print('Workload will take approximately %d seconds.' % options.time) [ worker.start() for worker in workers ] time.sleep(options.time) print('Stopping workload') stopping.set() [ worker.join() for worker in workers ] with get_connection() as conn: with conn.cursor() as cur: cur.execute("SELECT COUNT(*) AS count FROM %s" % TABLE) count = cur.fetchall()[0][0] print("%d rows inserted using %d threads" % (count, options.num_workers)) print("%.1f rows per second" % (count / float(options.time))) def cleanup(): """ Cleanup the database this benchmark is using. """ try: with get_connection() as conn: with conn.cursor() as cur: cur.execute('DROP DATABASE IF EXISTS %s' % options.database) except s2.Error: pass if __name__ == '__main__': HOST = options.host or "127.0.0.1" PORT = options.port or 3306 cleanup() try: test_connection() setup_test_db() warmup() run_benchmark() except KeyboardInterrupt: print("Interrupted... exiting...") ``` *** Modified at: March 26, 2025 Source: [/cloud/developer-resources/concurrent-multi-insert-examples/python/](https://docs.singlestore.com/cloud/developer-resources/concurrent-multi-insert-examples/python/) (An index of the documentation is available at /llms.txt) --- # Connect with Application Development Tools SingleStore supports connecting your cluster via various application development tools and programming languages. The support level for various third-party application development tools/programming languages may differ. SingleStore provides the following levels of support: * **GA**: SingleStore regularly tests the compatibility with this tool/language, against the latest version specified in the compatibility matrix. * **Preview**: SingleStore may not yet support complete integration with this tool/language, and it may require independent troubleshooting. If you face issues in connecting your SingleStore cluster with the tools/languages/frameworks specified in the following compatibility matrix, contact [SingleStore Support](https://www.singlestore.com/support/). | Language/Framework | Connector/Driver | Status | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- | ------- | | [C/C++](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-c-c.md) | MariaDB Connector/C (C/C++) | Preview | | [C#/.NET](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core.md) | SingleStoreConnector for .NET and .NET Core | GA | | [Drizzle ORM](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/connect-with-drizzle-orm.md) | MySQL2 | GA | | [Django](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-django.md) | Django-SingleStoreConnector | Preview | | [Go](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-go.md) | Go MySQL Driver | Preview | | [Haystack](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-haystack.md) | SingleStoreHaystack Library | GA | | [Hibernate](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/hibernate-singlestore-dialect.md) | SingleStoreJDBC Driver | GA | | [Java/JDBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) | SingleStoreJDBC Driver | GA | | [LangChain](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-langchain.md) | LangChain-SingleStoreConnector | GA | | [Laravel](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-laravel.md) | MySQL2 | Preview | | [Node.js](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js.md) | MySQL | Preview | | [ODBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) | SingleStoreODBC Driver | GA | | [Perl](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl.md) | MySQL DBI and DBM Drivers | Preview | | [PHP](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-php.md) | PDO\_MySQL Driver/MySQLi | Preview | | [Python](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) | SingleStorePython Client | GA | | [R2DBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-r-2-dbc.md) | SingleStoreR2DBC Connector | GA | | [Ruby](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby.md) | RubyGems MySQL2 | Preview | | [Rust](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-rust.md) | SQLx | Preview | | [Spring Boot](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/connect-with-spring-boot.md) | SingleStoreJDBC Driver | GA | For information about connecting with various IDEs/environments (or any MySQL compatible client), refer to [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md). You can also connect to your cluster via the [Data API](https://docs.singlestore.com/cloud/reference/data-api.md). ## In this section * [Connect with C/C++](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-c-c.md) * [Connect with Go](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-go.md) * [Connect with Haystack](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-haystack.md) * [Connect with Java/JDBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc.md) * [Connect with Laravel](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-laravel.md) * [Connect with .NET and .NET Core](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core.md) * [Connect with Node.js](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js.md) * [Connect with ODBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc.md) * [Connect with Perl](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl.md) * [Connect with PHP](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-php.md) * [Connect with Python](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python.md) * [Connect with R2DBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-r-2-dbc.md) * [Connect with Ruby](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby.md) * [Connect with Rust](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-rust.md) * [Extended Protocol Packet Metadata](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/extended-protocol-packet-metadata.md) * [Using Prepared Statements](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/using-prepared-statements.md) *** Modified at: June 15, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/) (An index of the documentation is available at /llms.txt) --- # Connect with C/C++ You can connect to your SingleStore database from C/C++ based applications using the MariaDB Connector (C/C++) (LGPLv2.1). [Download the MariaDB connector](https://mariadb.com/downloads/connectors/) version compatible with your operating system. > **📝 Note**: As of MariaDB Connector/C++ version 1.1.6, connections configured with `rewriteBatchedStatements=true` may not work as expected. To perform multi-row inserts, ensure that your application generates a single SQL statement containing multiple rows. Refer to MariaDB [Connector/C](https://mariadb.com/docs/connectors/mariadb-connector-c) and [Connector/C++](https://mariadb.com/docs/connectors/mariadb-connector-cpp) for more information. ## Examples Update the connection configuration of your SingleStore Helios deployment before running the code. * `endpoint`: IP address or hostname of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `database`: Name of the SingleStore database to connect with. * `username`: Username of the SingleStore database user with which to connect to SingleStore. * `password`: Password of the SingleStore database user. ## Example: Connect using C++ This example creates a table in SingleStore, inserts data into the table, and queries the data using C++. > **📝 Note**: The following commands may require `sudo` access (`root` privileges). 1. Install the required dependencies. This example uses a Debian-based Linux OS. ```shell apt-get install cmake libmariadb-dev libssl-dev ``` 2. Download and install the MariaDB C++ connector. 1. Clone the connector repository. ```shell git clone https://github.com/mariadb-corporation/mariadb-connector-cpp.git cd mariadb-connector-cpp ``` 2. Build and install the connector. ```shell cmake . make make install ``` The `mariadbapp` library is installed in the `/usr/local/lib` directory. 3. Add a symbolic link for `libmariadbcpp.so`. ```shell ln -s /usr/local/lib/mariadb/libmariadbcpp.so /usr/local/lib ldconfig ``` 3. Add the following code to a **s2example.cpp** file. Update the connection configuration of your SingleStore deployment in the code. ```C++ #include #include int main() { try { // Connection details const std::string url = "tcp://:3306/"; const std::string user = ""; const std::string password = ""; // Connect sql::Driver* driver = sql::mariadb::get_driver_instance(); sql::SQLString connection_url(url); sql::Properties properties({{"user", user}, {"password", password}}); std::unique_ptr conn(driver->connect(connection_url, properties)); std::unique_ptr stmt(conn->createStatement()); // Create operation stmt->execute("CREATE TABLE IF NOT EXISTS Stock (ID INT PRIMARY KEY,Code VARCHAR(10),Quantity INT)"); // Insert operation stmt->execute("INSERT INTO Stock (ID, Code, Quantity) VALUES (3, 'cdq3', 25)"); stmt->execute("INSERT INTO Stock (ID, Code, Quantity) VALUES (1, 'xvf1', 40)"); stmt->execute("INSERT INTO Stock (ID, Code, Quantity) VALUES (2, 'gwl2', 15)"); // Read operation std::unique_ptr res(stmt->executeQuery("SELECT * FROM Stock ORDER BY ID")); // Display results while (res->next()) { std::cout << "ID: " << res->getInt("ID") << ", Code: " << res->getString("Code") << ", Quantity: " << res->getInt("Quantity") << std::endl; } } catch (sql::SQLException &e) { std::cerr << "SQL Exception: " << e.what() << std::endl; return 1; } return 0; } ``` 4. Run the following command to compile the code. Run this command in the same directory as **s2example.cpp** (or append the path to the file). ```shell g++ s2example.cpp -o singlestore_mariadb \ -I/usr/local/include/mariadb \ -L/usr/local/lib/mariadb \ -lmariadbcpp ``` 5. Run the code. Run the following command in the same directory as the **singlestore\_mariadb** file generated in the previous step. ```shell ./singlestore_mariadb ``` ```output ./singlestore_mariadb ID: 1, Code: xvf1, Quantity: 40 ID: 2, Code: gwl2, Quantity: 15 ID: 3, Code: cdq3, Quantity: 25 ``` ## Example: Connect with C This example creates a table in SingleStore, inserts data into the table, and queries the data using C. > **📝 Note**: The following commands may require `sudo` access (`root` privileges). 1. Install the MariaDB C connector. This example uses a Debian-based Linux OS. ```shell apt-get install libmariadb-dev ``` 2. Add the following code to a **s2example.c** file. ```C #include #include #include int main() { // Connection parameters -- REPLACE these with your actual values const char *host = ""; const char *user = ""; const char *pass = ""; const char *database = ""; unsigned int port = 3306; MYSQL *conn = mysql_init(NULL); // Connect to SingleStore if (!mysql_real_connect(conn, host, user, pass, database, port, NULL, 0)) { fprintf(stderr, "Connection error: %s\n", mysql_error(conn)); mysql_close(conn); return 1; } // Create a table const char *create_table_sql = "CREATE TABLE IF NOT EXISTS Inventory (ID INT PRIMARY KEY,Code VARCHAR(4),Quantity INT)"; if (mysql_query(conn, create_table_sql)) { fprintf(stderr, "Create table error: %s\n", mysql_error(conn)); mysql_close(conn); return 1; } // Insert data const char *insert1 = "INSERT INTO Inventory (ID, Code, Quantity) VALUES (3, 'cdq3', 25)"; const char *insert2 = "INSERT INTO Inventory (ID, Code, Quantity) VALUES (1, 'xvf1', 40)"; const char *insert3 = "INSERT INTO Inventory (ID, Code, Quantity) VALUES (2, 'gwl2', 15)"; if (mysql_query(conn, insert1) || mysql_query(conn, insert2) || mysql_query(conn, insert3)) { fprintf(stderr, "Insert error: %s\n", mysql_error(conn)); mysql_close(conn); return 1; } // Perform SELECT query if (mysql_query(conn, "SELECT * FROM Inventory")) { fprintf(stderr, "SELECT error: %s\n", mysql_error(conn)); mysql_close(conn); return 1; } MYSQL_RES *result = mysql_store_result(conn); if (result == NULL) { fprintf(stderr, "Result error: %s\n", mysql_error(conn)); mysql_close(conn); return 1; } // Print results MYSQL_ROW row; printf("ID\tCode\tQuantity\n"); while ((row = mysql_fetch_row(result))) { printf("%s\t%s\t%s\n", row[0], row[1], row[2]); } // Clean up mysql_free_result(result); mysql_close(conn); return 0; } ``` 3. Compile the code. Run this command in the same directory as **s2example.c** (or append the path to the file). ```shell gcc s2example.c -o singlestore_mariadb -lmariadb ``` 4. Run the code. Run the following command in the same directory as the **singlestore\_mariadb** file generated in the previous step. ```shell ./singlestore_mariadb ``` ```output ID Code Quantity 3 cdq3 25 1 xvf1 40 2 gwl2 15 ``` *** Modified at: September 26, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-c-c/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-c-c/) (An index of the documentation is available at /llms.txt) --- # Connect with Go Use the Go SingleStore driver (`go-singlestore-driver`), a Go driver for the `database/sql` driver package, to connect Go applications to SingleStore databases. This driver includes features and improvements specific to SingleStore. Refer to the [Go-SingleStore-Driver](https://github.com/singlestore-labs/go-singlestore-driver) GitHub repository for more information on supported features and enhancements. ## Prerequisites * [Install](https://go.dev/doc/install) the latest version of Go. * SingleStore 8.7 or later. * (Optional) [Install](https://git-scm.com/install/linux) `git`. ## Install the Go SingleStore Driver Install the `go-singlestore-driver` package using the Go CLI utility: ```shell go get -u github.com/singlestore-labs/go-singlestore-driver/v2 ``` ## Configure the Connection to SingleStore Use the following connection configuration parameters in the standard MySQL DSN (Data Source Name) or other applicable methods: * `protocol`: Network transport type used to establish the database connection, which determines how the accompanying address is interpreted. Use the `tcp` protocol for connecting with SingleStore Helios. * `hostname`: IP address or hostname of the SingleStore cluster. Specified when the protocol is `tcp`. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `port`: Port of the SingleStore cluster. The default is `3306`. Specified when the protocol is `tcp`. * `username`: Username of the SingleStore database user. * `password`: Password for the SingleStore database user. * `database`: Name of the SingleStore database to connect with. * `param=value`: (Optional) Additional connection parameters. The driver supports the following DSN format: ```DSN :@tcp()/[?param=value] ``` For more information on the DSN and supported parameters, refer to [DSN (Data Source Name)](https://github.com/singlestore-labs/go-singlestore-driver#dsn-data-source-name). ## Example The following example connects to a SingleStore database and performs basic CRUD operations. 1. Create a project folder and initialize the Go module. ```shell mkdir ssGo cd ssGo go mod init ssGo ``` 2. Install the `go-singlestore-driver` package. ```shell go get -u github.com/singlestore-labs/go-singlestore-driver/v2 ``` 3. Create a **main.go** file and add the following code to this file. Update the connection configuration of your SingleStore database in the code. ```go package main import ( "database/sql" "fmt" "log" _ "github.com/singlestore-labs/go-singlestore-driver/v2" ) func main() { // Connection settings host := "svc-xxxx" port := "3306" user := "s2user" password := "p455w1kd" database := "demo" // SingleStore DSN dsn := fmt.Sprintf( "%s:%s@tcp(%s:%s)/%s?parseTime=true", user, password, host, port, database, ) // Open connection using the SingleStore driver db, err := sql.Open("singlestore", dsn) if err != nil { log.Fatal(err) } defer db.Close() // Verify connection if err := db.Ping(); err != nil { log.Fatal(err) } fmt.Println("Connected to SingleStore") // Create table createTableSQL := ` CREATE TABLE IF NOT EXISTS users ( id BIGINT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100), email VARCHAR(255) ) ` _, err = db.Exec(createTableSQL) if err != nil { log.Fatal(err) } fmt.Println("Table created") // Insert rows insertSQL := ` INSERT INTO users (name, email) VALUES (?, ?) ` _, err = db.Exec(insertSQL, "Alice", "alice@example.com") if err != nil { log.Fatal(err) } _, err = db.Exec(insertSQL, "Bob", "bob@example.com") if err != nil { log.Fatal(err) } fmt.Println("Rows inserted") // Update a row updateSQL := ` UPDATE users SET email = ? WHERE name = ? ` result, err := db.Exec(updateSQL, "alice@newdomain.com", "Alice") if err != nil { log.Fatal(err) } rowsUpdated, _ := result.RowsAffected() fmt.Printf("Rows updated: %d\n", rowsUpdated) // Delete a row deleteSQL := ` DELETE FROM users WHERE name = ? ` result, err = db.Exec(deleteSQL, "Bob") if err != nil { log.Fatal(err) } rowsDeleted, _ := result.RowsAffected() fmt.Printf("Rows deleted: %d\n", rowsDeleted) // Query remaining rows rows, err := db.Query(` SELECT id, name, email FROM users `) if err != nil { log.Fatal(err) } defer rows.Close() fmt.Println("\nRemaining users:") for rows.Next() { var id int var name string var email string if err := rows.Scan(&id, &name, &email); err != nil { log.Fatal(err) } fmt.Printf("ID=%d Name=%s Email=%s\n", id, name, email) } if err := rows.Err(); err != nil { log.Fatal(err) } } ``` 4. Run the application. ```shell go run main.go ``` ```output Connected to SingleStore Table created Rows inserted Rows updated: 1 Rows deleted: 1 Remaining users: ID=1 Name=Alice Email=alice@newdomain.com ``` ## References * [Go-SingleStore-Driver](https://github.com/singlestore-labs/go-singlestore-driver) GitHub repository * Additional examples: * [Getting started with SingleStore and Go](https://github.com/singlestore-labs/start-with-singlestore-go/) * [Getting started with SingleStore stored procedures and Go](https://github.com/singlestore-labs/start-with-singlestore-go-stored-procedure) *** Modified at: June 16, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-go/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-go/) (An index of the documentation is available at /llms.txt) --- # Connect with Haystack [Haystack](https://docs.haystack.deepset.ai/) by [Deepset](https://www.deepset.ai/) is an open-source framework for building search and retrieval-augmented generation (RAG) applications. The `singlestore-haystack` library enables you to integrate your SingleStore database as a [Document Store](https://docs.haystack.deepset.ai/docs/document-store) in Haystack to store and index documents and their metadata. Haystack retrieves these documents during queries and provides them to the Retriever for additional processing. The `singlestore-haystack` library uses the [SingleStore Python client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) to interact with the SingleStore database. Refer to the [singlestore-haystack](https://github.com/singlestore-labs/singlestore-haystack) GitHub repository for its source code and related information. ## Install singlestore-haystack The `singlestore-haystack` library can be installed using the standard Python package installation process: ```shell pip install singlestore-haystack ``` ## Data Storage Model `SingleStoreDocumentStore` stores documents as rows in a SingleStore table. Vector embeddings are stored in a [VECTOR](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) type column in the table. `SingleStoreDocumentStore` automatically creates the required vector and full-text indexes if they do not already exist. When using `SingleStoreEmbeddingRetriever`, documents must be embedded before they are written to the database. Use a Haystack embedder to generate these embeddings. For example, use the `SentenceTransformersDocumentEmbedder` in an indexing pipeline to generate document embeddings before storing them in SingleStore. The following is a visual representation: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt14a3939312b8a042/6a7ada9dc2cc0998aa23db6b/haystack-LBa8f6.png) In this infographic: * **Haystack table** is a SingleStore table used by `SingleStoreDocumentStore` to persist Haystack Document objects as rows. * **embedding** is a property of the document, which is stored as a vector of type `VECTOR(n, F32)`. * **content** is a property of the document. * **vector indexes** are SingleStore vector indexes created on the embedding column to enable efficient search for dense retrieval. * **fulltext index** is a SingleStore full-text index created on the content column to support BM25-based sparse retrieval. * `write_documents` represents the insert operation where `SingleStoreDocumentStore` stores documents in the table. * `retrieve_documents` represents the retrieval operations run by retrievers, such as `SingleStoreEmbeddingRetriever` (for vector search) and `SingleStoreBM25Retriever` (for full-text search). For example, consider the following code: ```python from haystack import Document from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore # Initialize the document store (uses S2_CONN_STR by default) document_store = SingleStoreDocumentStore( database_name="haystack_db", table_name="haystack_documents", embedding_dimension=384, ) # Each Document becomes a row in the SingleStore table documents = [ Document( content="SingleStore is a distributed SQL database built to power intelligent applications.", embedding=[0.1] * 384, # VECTOR(384, F32) column meta={ "num_of_years": 3, # stored as JSON/metadata column }, ) ] # Insert documents into SingleStore document_store.write_documents(documents) ``` ## Supported Components This library implements the [DocumentStore protocol](https://docs.haystack.deepset.ai/docs/document-store#documentstore-protocol) methods; import the `SingleStoreDocumentStore` implementation as follows: ```python from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore ``` In addition to `SingleStoreDocumentStore`, the `singlestore-haystack` library includes the following Haystack [Retriever](https://docs.haystack.deepset.ai/docs/retrievers) components that can be used in a pipeline: * `SingleStoreEmbeddingRetriever`: Queries SingleStore vector index and finds semantically related documents. This component uses `SingleStoreDocumentStore` to perform vector similarity search over stored vector embeddings. ```python from haystack_integrations.components.retrievers.singlestore_haystack import SingleStoreEmbeddingRetriever ``` * `SingleStoreBM25Retriever`: Performs sparse retrieval using the BM25 ranking algorithm. It leverages SingleStore full-text search (FTS) capabilities to retrieve documents based on keyword relevance instead of vector similarity (embeddings). This component uses `SingleStoreDocumentStore` to execute BM25 queries. SingleStore recommends using this component for keyword-based and hybrid search scenarios. ```python from haystack_integrations.components.retrievers.singlestore_haystack import SingleStoreBM25Retriever ``` You can specify either of the following scoring functions: * BM25 * BM25\_GLOBAL Refer to [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) for more information. For example: ```python retriever = SingleStoreBM25Retriever(document_store=document_store) results = retriever.run( query="database", top_k=2, bm25_function="BM25", )["documents"] ``` ## Use SingleStore as a Document Store ## Prerequisites Ensure the following are met before running examples in this section: * An active SingleStore cluster. * Install the `singlestore_haystack` package. * (Optional) Install the `sentence-transformers` Python library. It provides pre-trained models used in this example to generate vector embeddings. ```python pip install sentence-transformers ``` ## Configure the Connection to SingleStore To keep the credentials out of the source code, assign the connection string to the `S2_CONN_STR` environment variable in the following format: ```shell export S2_CONN_STR="singlestoredb://:@:/[]" ``` where, * `hostname`: IP address or hostname of the SingleStore cluster. * `port`: Port of the SingleStore cluster. The default is `3306`. * `username`: Username of the SingleStore database user. * `password`: Password for the SingleStore database user. * `database`: (Optional) Name of the SingleStore database to connect with. Alternatively, specify the connection configuration while instantiating the class: ```python document_store = SingleStoreDocumentStore( host="", port=, username="", password="", database_name="", table_name="
" # Name of SingleStore the table used to store Documents ) ``` ## Configure Indexes `SingleStoreDocumentStore` supports creating and customizing indexes on the SingleStore table. Refer to [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) for more information. Based on the retrieval strategy, enable or disable specific index types and configure the index accordingly. For example: ```python from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore document_store = SingleStoreDocumentStore( database_name="haystack_db", table_name="haystack_documents", embedding_dimension=768, # Enable FULLTEXT index for keyword/BM25 search use_fulltext_index=True, fulltext_index_options={ "analyzer": "standard", }, # Enable vector index optimized for dot product similarity use_dot_product_vector_index=True, dot_product_vector_index_options={ "nlist": 128, }, # Optionally disable Euclidean-distance index if not needed use_euclidian_distance_vector_index=False, ) ``` Specify the following options as applicable when instantiating a `SingleStoreDocumentStore` object: ## Dot Product Optimized Vector Index | Option | Description | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `use_dot_product_vector_index` | Creates a vector index using dot product similarity. | | `dot_product_vector_index_options` | Specifies a dictionary that contains options for configuring the vector index that uses dot product similarity. These options are forwarded toSingleStore. Refer to[Vector Index Options](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing/#section-idm457710817120003408966071884.md)for information on supported options. | ## Euclidean Distance Optimized Vector Index | Option | Description | | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `use_euclidian_distance_vector_index` | Creates a vector index using Euclidean (L2) distance similarity. | | `euclidian_distance_vector_index_options` | Specifies a dictionary that contains additional options for configuring the vector index that uses Euclidean distance similarity. These options are forwarded toSingleStore. Refer to[Vector Index Options](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing/#section-idm457710817120003408966071884.md)for information on supported options. | ## Full Text Index > **📝 Note**: The full-text index is required for keyword-based retrieval using the `SingleStoreBM25Retriever`. | Option | Description | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `use_fulltext_index` | Creates a full-text index (version 2). | | `fulltext_index_options` | Specifies a dictionary that contains additional options for configuring the full-text index. These options are forwarded toSingleStore. Refer to[Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md)for information on supported options. | ## Hybrid Retrieval To support hybrid retrieval scenarios, both vector and full-text indexes can be enabled at the same time (used together). For example, to combine dense (semantic) and sparse (keyword-based) search techniques within the same Haystack pipeline. ## Write Documents To write documents to SingleStore, use either of the following: * `SingleStoreDocumentStore.write_documents()` method * [DocumentWriter](https://docs.haystack.deepset.ai/docs/documentwriter) component ## write\_documents() Example The following example generates the embeddings using `SentenceTransformersDocumentEmbedder` and then writes the document to SingleStore. ```python from haystack import Document from haystack.components.embedders import SentenceTransformersDocumentEmbedder from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore # Initialize the document store document_store = SingleStoreDocumentStore( database_name="haystack_db", # SingleStore database table_name="haystack_documents", # SingleStore table for Documents embedding_dimension=384, # Dimension of embeddings ) # Create documents documents = [ Document(content="SingleStore is a distributed SQL database built to power intelligent applications.") ] # Create the document embedder document_embedder = SentenceTransformersDocumentEmbedder( model="sentence-transformers/all-MiniLM-L6-v2" ) # Download the model and prepare it (first run only) document_embedder.warm_up() # Generate embeddings result = document_embedder.run(documents) documents_with_embeddings = result["documents"] # Write documents (with embeddings) to SingleStore document_store.write_documents(documents_with_embeddings) ``` ## DocumentWriter Example The following example creates a Haystack pipeline to write documents to SingleStore: ```python from haystack import Document, Pipeline from haystack.components.embedders import SentenceTransformersDocumentEmbedder from haystack.components.writers import DocumentWriter from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore # Input documents documents = [ Document(content="SingleStore is a distributed SQL database built to power intelligent applications."), Document(content="SingleStore is delivered as a SaaS data platform (SingleStore Helios) and is available in AWS, Azure, and GCP."), ] # Initialize the document store document_store = SingleStoreDocumentStore( table_name="haystack_documents", embedding_dimension=384, recreate_table=True, # Recreate the table if it already exists ) # Components embedder = SentenceTransformersDocumentEmbedder( model="sentence-transformers/all-MiniLM-L6-v2" ) writer = DocumentWriter(document_store=document_store) # Build the pipeline pipeline = Pipeline() pipeline.add_component(instance=embedder, name="embedder") pipeline.add_component(instance=writer, name="writer") pipeline.connect("embedder", "writer") # Run the indexing pipeline result = pipeline.run({"embedder": {"documents": documents}}) print(result) # {'writer': {'documents_written': 2}} ``` ```output `{'writer': {'documents_written': 2}}` ``` ## Retrieve Documents Use the `SingleStoreEmbeddingRetriever` component to retrieve documents from SingleStore. For example, consider the following Haystack pipeline that finds documents using vector index and [metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering): ```python from typing import List from haystack import Document, Pipeline from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder from haystack_integrations.components.retrievers.singlestore_haystack import SingleStoreEmbeddingRetriever from haystack_integrations.document_stores.singlestore_haystack import SingleStoreDocumentStore # Initialize the document store document_store = SingleStoreDocumentStore( database_name="haystack_db", # The name of the database in SingleStore table_name="haystack_documents", # The name of the table to store Documents embedding_dimension=384, # The dimension of the embeddings being stored recreate_table=True, ) # Sample documents with metadata documents = [ Document(content="My name is Morgan and I live in Paris.", meta={"num_of_years": 3}), Document(content="I am Susan and I live in Berlin.", meta={"num_of_years": 7}), ] # The same model is used for both query and Document embeddings model_name = "sentence-transformers/all-MiniLM-L6-v2" # Embed and write documents document_embedder = SentenceTransformersDocumentEmbedder(model=model_name) document_embedder.warm_up() documents_with_embeddings = document_embedder.run(documents) document_store.write_documents(documents_with_embeddings.get("documents")) print("Number of documents written: ", document_store.count_documents()) # Build the retrieval pipeline pipeline = Pipeline() pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder(model=model_name)) pipeline.add_component("retriever", SingleStoreEmbeddingRetriever(document_store=document_store)) pipeline.connect("text_embedder.embedding", "retriever.query_embedding") # Run a query with metadata filtering result = pipeline.run( data={ "text_embedder": {"text": "What cities do people live in?"}, "retriever": { "top_k": 5, "filters": {"field": "meta.num_of_years", "operator": "==", "value": 3}, }, } ) documents: List[Document] = result["retriever"]["documents"] print(documents) ``` ```output [Document(id=4014455c3be5d88151ba12d734a16754d7af75c691dfc3a5f364f81772471bd2, content: 'My name is Morgan and I live in Paris.', meta: {'num_of_years': 3}, score: 0.339349627494812, embedding: vector of size 384)] ``` ## Examples Refer to the [singlestore-haystack](https://github.com/singlestore-labs/singlestore-haystack?tab=readme-ov-file#more-examples) GitHub repository for more examples. ## References * [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) * [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) * [Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) * [DOT\_PRODUCT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/dot-product.md) * [EUCLIDEAN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/euclidean-distance.md) *** Modified at: April 24, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-haystack/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-haystack/) (An index of the documentation is available at /llms.txt) --- # Connect with Java/JDBC You can connect from Java/JDBC-based applications to SingleStore using the following drivers/connectors: * The SingleStore JDBC Driver * MariaDB Connector/J (JDBC) For most JDBC use cases, SingleStore recommends [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). ## The SingleStore JDBC Driver The SingleStore JDBC driver is compatible with JDBC 4.2. You can use this driver to connect applications developed in Java 21, Java 17, Java 11, and Java 8 to SingleStore databases. Refer to [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) for detailed information on how to install, configure, and connect using this driver. ## Hibernate SingleStore Dialect Refer to [Hibernate SingleStore Dialect](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/hibernate-singlestore-dialect.md) for information on connecting to your SingleStore deployments from Hibernate applications using the SingleStore dialect. ## MariaDB Connector/J (JDBC) Version 2.4.0 (LGPLv2.1) You can also use the MariaDB Connector/J v2.4.0 to connect to your SingleStore database. > **📝 Note**: To enable/promote connector compatibility, SingleStore identifies itself as MySQL version x.x.x compatible. The compatibility version may vary depending on the SingleStore version in use.To specify a MySQL compatibility version, use the `compat_version` variable. For example,```sql > SET GLOBAL compat_version = '5.6.4'; > ``` Refer to [Connector/J 2.4.0](https://mariadb.com/docs/release-notes/connectors/java/2.4/2.4.0) for its binary and source code. ## In this section * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) * [Hibernate SingleStore Dialect](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/hibernate-singlestore-dialect.md) * [Connect with Spring Boot](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/connect-with-spring-boot.md) *** Modified at: November 7, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/) (An index of the documentation is available at /llms.txt) --- # Connect with Laravel You can connect Laravel to SingleStore using the SingleStore Driver for Laravel ("the driver"). Refer to the [SingleStore Driver for Laravel](https://github.com/singlestore-labs/singlestoredb-laravel-driver) Github repository for its source code and related information. Prerequisites * Install [PDO\_MYSQL](https://www.php.net/manual/en/ref.pdo-mysql.php). * Install the driver via composer: ```shell composer require singlestoredb/singlestoredb-laravel ``` ## Connect to SingleStore To connect Laravel to SingleStore, enable the driver and configure the connection. To enable the driver, edit your `config/database.php` file and create a new entry for SingleStore in the connection configuration. Update the `default` option to point to the new connection or set `DB_CONNECTION=singlestore` in your environment file. Here's a sample configuration: ```php [ 'default' => env('DB_CONNECTION', 'singlestore'), 'connections' => [ 'singlestore' => [ 'driver' => 'singlestore', 'url' => env('DATABASE_URL'), 'host' => env('DB_HOST'), 'port' => env('DB_PORT'), 'database' => env('DB_DATABASE'), 'username' => env('DB_USERNAME'), 'password' => env('DB_PASSWORD'), 'unix_socket' => env('DB_SOCKET'), 'charset' => 'utf8mb4', 'collation' => 'utf8mb4_unicode_ci', 'prefix' => '', 'prefix_indexes' => true, 'strict' => true, 'engine' => null, 'options' => extension_loaded('pdo_mysql') ? array_filter([ PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'), PDO::ATTR_EMULATE_PREPARES => true, PDO::ATTR_PERSISTENT => true ]) : [], ], ] ] ``` The SingleStore driver is an extension of the MySQL driver, so you can also use the driver from `mysql`. **Connect using SSL** To configure SSL connections between SingleStore Helios and Laravel, download the `singlestore_bundle.pem` [certificate file](https://portal.singlestore.com/static/ca/singlestore_bundle.pem) and update your `MYSQL_ATTR_SSL_CA` environment variable to point to it. Set `PDO::MYSQL_ATTR_SSL_VERIFY_SERVER_CERT` to `"true"`, as shown below. ```php 'options' => extension_loaded('pdo_mysql') ? array_filter([ PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'), PDO::MYSQL_ATTR_SSL_VERIFY_SERVER_CERT => true, PDO::ATTR_EMULATE_PREPARES => true, PDO::ATTR_PERSISTENT => true ]) : [], ``` For more details, refer to the [SingleStore Driver for Laravel](https://github.com/singlestore-labs/singlestoredb-laravel-driver) and [Laravel docs](https://laravel.com/docs/database). *** Modified at: June 11, 2024 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-laravel/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-laravel/) (An index of the documentation is available at /llms.txt) --- # Connect with .NET and .NET Core The SingleStore Connector for .NET and .NET Core (`SingleStoreConnector`) is an ADO.NET data provider for SingleStore. It implements the following classes: `DbConnection`, `DbCommand`, `DbDataReader`, and `DbTransaction`. Refer to the [GitHub repository](https://github.com/memsql/SingleStoreNETConnector) and [SingleStoreConnector on NuGet](https://www.nuget.org/packages/SingleStoreConnector/) for more information. This library is licensed under the MIT license. The `SingleStoreConnector` supports .NET 10. ## Prerequisites Download and install the latest stable version of [.NET Core](https://dotnet.microsoft.com/en-us/download). ## Install SingleStoreConnector To install the `SingleStoreConnector` in a new project, run the following command: ```shell dotnet add package SingleStoreConnector ``` You can also install the `SingleStoreConnector` using the NuGet Package Manager in Visual Studio. Refer to [SingleStoreConnector](https://www.nuget.org/packages/SingleStoreConnector/) on NuGet for more information. ## Configure the Connection You need a connection string to connect your SingleStore cluster to .NET. The connection string uses the following format: ```C# host=;port=;userid=;password=;database=; ``` Here's a sample connection string: ```C# host=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;userid=s2user;password=pass23key;database=s2_dbtest; ``` Refer to [Connection String Options](https://mysqlconnector.net/connection-options/) for more options. ## Example The following example creates a new project in Visual Studio Code and performs CRUD operations in SingleStore using C#. Create a SingleStore cluster. For this example, we'll use the following connection string: ```C# "Server=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;User ID=admin;Password=tK_,mh&Hq-EnN;Database=dbtest" ``` In Visual Studio Code, open the terminal, and create a project template: ```shell dotnet new console -o dbTestNet cd dbTestNet ``` Install the required dependencies, for example `SingleStoreConnector`: ```shell dotnet add package SingleStoreConnector ``` ## Create Operation Example Add the following code to the `Program.cs` file of your project: ```C# using SingleStoreConnector; var connStr = "Server=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;User ID=admin;Password=tK_,mh&Hq-EnN;Database=dbtest"; var connection = new SingleStoreConnection(connStr); connection.Open(); using var command = new SingleStoreCommand("CREATE TABLE testID (ID INT PRIMARY KEY, Code VARCHAR(4));INSERT INTO testID values(1, 'SamK');INSERT INTO testID values(2, 'JoeR');INSERT INTO testID values(3, 'BriA');",connection); using var reader = command.ExecuteReader(); connection.Close(); ``` In the terminal, run the following command: ```shell dotnet run ``` On the Cloud Portal command line, execute the following command to verify that the `testID` table is created: ```sql DESC testID; ``` ```output +-------+------------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+------------+------+------+---------+-------+ | id | int(11) | NO | UNI | NULL | | | code | varchar(4) | YES | | NULL | | +-------+------------+------+------+---------+-------+ ``` ## Read Operation Example Add the following code to the `Program.cs` file of your project: ```C# using SingleStoreConnector; var connStr = "Server=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;User ID=admin;Password=tK_,mh&Hq-EnN;Database=dbtest"; var connection = new SingleStoreConnection(connStr); connection.Open(); using var command = new SingleStoreCommand("SELECT * FROM testID",connection); using var reader = command.ExecuteReader(); while (reader.Read()) { Console.Write(reader.GetInt32(0)); Console.Write(" "); Console.WriteLine(reader.GetString(1)); } connection.Close(); ``` In the terminal, run the following command: ```shell dotnet run ``` ```output 1 SamK 2 JoeR 3 BriA ``` ## Update Operation Example Add the following code to the `Program.cs` file of your project: ```C# using SingleStoreConnector; var connStr = "Server=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;User ID=admin;Password=tK_,mh&Hq-EnN;Database=dbtest"; var connection = new SingleStoreConnection(connStr); connection.Open(); using var command = new SingleStoreCommand("UPDATE testID SET Code = 'BenW' WHERE ID =3;SELECT * FROM testID",connection); using var reader = command.ExecuteReader(); while (reader.Read()) { Console.Write(reader.GetInt32(0)); Console.Write(" "); Console.WriteLine(reader.GetString(1)); } connection.Close(); ``` In the terminal, run the following command: ```shell dotnet run ``` ```output 1 SamK 2 JoeR 3 BenW ``` ## Delete Operation Example Add the following code to the `Program.cs` file of your project: ```C# using SingleStoreConnector; var connStr = "Server=svc-xxxx-ddl.aws-oregon-2.svc.singlestore.com;port=3306;User ID=admin;Password=tK_,mh&Hq-EnN;Database=dbtest"; var connection = new SingleStoreConnection(connStr); connection.Open(); using var command = new SingleStoreCommand("DELETE FROM testID WHERE ID = 3;SELECT * FROM testID;",connection); using var reader = command.ExecuteReader(); while (reader.Read()) { Console.Write(reader.GetInt32(0)); Console.Write(" "); Console.WriteLine(reader.GetString(1)); } connection.Close(); ``` In the terminal, run the following command: ```shell dotnet run ``` ```output 1 SamK 2 JoeR ``` ## In this section * [Connect using Entity Framework Core](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core/connect-using-entity-framework-core.md) *** Modified at: July 28, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-net-and-net-core/) (An index of the documentation is available at /llms.txt) --- # Connect with Node.js Use the SingleStore Node.js driver (`singlestore-nodejs` dependency) to connect Node.js applications to SingleStore databases, whether using an ORM or straight SQL. The `singlestore-nodejs` dependency supports configuring the connection behavior, prepared statements, compression, SSL, and more. Refer to [SingleStore-Nodejs](https://singlestore-labs.github.io/singlestore-nodejs/docs) for more information related to the driver's implementation, supported features, and usage patterns. Refer to the [SingleStore Node.js Driver](https://github.com/singlestore-labs/singlestore-nodejs) GitHub repository for its source code and related information. To connect from Node.js to SingleStore using SSL, refer to [Connect with Node.js using SSL](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/connect-with-node-js-using-ssl.md). SSL is required for connections to a Shared cluster. ## Install the SingleStore Node.js Driver Install the `singlestore-nodejs` dependency to use the SingleStore Node.js driver: ```sql npm install --save singlestore-nodejs ``` To add TypeScript type definitions, run the following command: ```sql npm install --save-dev @types/node ``` ## Configure the Connection to SingleStore The driver supports both Promise-based and callback-based programming models for handling asynchronous operations. Use the following connection configuration parameters with the respective model to configure the connection to SingleStore: * `hostname`: IP address or hostname of the SingleStore cluster. * `port`: Port of the SingleStore cluster. The default is `3306`. * `username`: Username of the SingleStore database user. * `password`: Password for the SingleStore database user. * `database`: (Optional) Name of the SingleStore database to connect with. ## Using Promises ```javascript import singlestore from "singlestore-nodejs/promise"; const connection = await singlestore.createConnection({ host: "", port: 3306, user: "", password: "", database: "", }); ``` ## Using Callbacks ```javascript const singlestore = require("singlestore-nodejs"); const connection = singlestore.createConnection({ host: "", port: 3306, user: "", password: "", database: "", }); connection.connect((err) => { if (err) { console.error("Connection error:", err); return; } console.log("Connected to SingleStore."); ``` ## Example The following example connects to a SingleStore database and performs basic CRUD operations. 1. Create a Node.js project. ```shell mkdir nodeJs cd nodeJs npm init -y ``` 2. Install the `singlestore-nodejs` dependency. ```shell npm install singlestore-nodejs ``` 3. Enable ES modules. Add `"type": "module"` to `package.json`. For example: ```json { "name": "nodejs", "version": "1.0.0", "description": "", "type": "module", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "singlestore-nodejs": "^1.0.1" } } ``` 4. Add the following code to the `index.js` file. Update the connection configuration of your SingleStore database in the code. ```javascript import singlestore from "singlestore-nodejs/promise"; // == Update the SingleStore database connection config) == const config = { host: "svchost-xxxx", user: "s2user", password: "p455w1kd", database: "dbTest", port: 3306, }; async function main() { let conn; try { // 1. CONNECT conn = await singlestore.createConnection(config); console.log("Connected to SingleStore."); // 2. CREATE TABLE await conn.execute(` CREATE TABLE IF NOT EXISTS users ( id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100) )`); console.log("Table ready."); // 3. INSERT const [insertResult] = await conn.execute( "INSERT INTO users (name) VALUES (?)", ["Green"]); console.log("Inserted ID:", insertResult.insertId); const userId = insertResult.insertId; // 4. READ const [rows] = await conn.execute( "SELECT * FROM users WHERE id = ?", [userId]); console.log("Read:", rows); // 5. UPDATE await conn.execute( "UPDATE users SET name = ? WHERE id = ?", ["Mink", userId]); console.log("Updated user."); // 6. VERIFY UPDATE const [updatedRows] = await conn.execute( "SELECT * FROM users WHERE id = ?", [userId]); console.log("After update:", updatedRows); // 7. DELETE await conn.execute("DELETE FROM users WHERE id = ?", [userId]); console.log("Deleted user."); } catch (err) { console.error("Error:", err); } finally { if (conn) { await conn.end(); console.log("Connection closed."); } } } main(); ``` 5. Run the application. ```shell node index.js ``` ```output Connected to SingleStore. Table ready. Inserted ID: 1 Read: [ { id: 1, name: 'Green' } ] Updated user. After update: [ { id: 1, name: 'Mink' } ] Deleted user. Connection closed. ``` ## Additional Examples * Training: [Building a SingleStore Application Using Javascript and Node.js](https://training.singlestore.com/learn/course/internal/view/elearning/689/building-a-singlestore-application-using-javascript-and-nodejs). * [Getting started with SingleStore stored procedures and Node.js](https://github.com/singlestore-labs/start-with-singlestore-node-stored-procedure). * [Getting started with SingleStore and Node.js Sequelize ORM](https://github.com/singlestore-labs/start-with-singlestore-node-orm). ## References * [SingleStore Node.js Reference ](https://singlestore-labs.github.io/singlestore-nodejs/docs) * [singlestore-nodejs](https://github.com/singlestore-labs/singlestore-nodejs) GitHub repository ## In this section * [Connect with Node.js using SSL](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/connect-with-node-js-using-ssl.md) * [Connect with Drizzle ORM](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/connect-with-drizzle-orm.md) *** Modified at: May 15, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js/) (An index of the documentation is available at /llms.txt) --- # Connect with ODBC You can connect to SingleStore from ODBC-based applications using the following drivers/connectors: * The SingleStore ODBC Driver * MariaDB Connector/ODBC * MySQL Connector/ODBC For most ODBC use cases, SingleStore recommends [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md). ## The SingleStore ODBC Driver You can connect to your SingleStore Helios clusters from ODBC-based applications using the SingleStore ODBC driver. It supports both Unicode and ANSI modes. You can download the SingleStore ODBC binaries from its [GitHub](https://github.com/memsql/singlestore-odbc-connector/releases/tag/v1.1.1) repository. See [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) for detailed Information on how to install, configure, and connect using the driver. ## MariaDB Connector/ODBC Version 3.0.8 (LGPLv2.1) You can use the MariaDB Connector/ODBC v3.0.8 to connect to your SingleStore Helios database. Refer to [Connector/ODBC 3.0.8](https://mariadb.com/docs/release-notes/connectors/odbc/3.0/3.0.8) for its binary and source code. ## MySQL Connector/ODBC Version 8.0.26 (GPLv2) You can also use the MySQL Connector/ODBC v8.0.26 to connect to your SingleStore Helios database. You can download the connector binaries for the supported operating systems from [MySQL Community Downloads](https://dev.mysql.com/downloads/connector/odbc/). ## In this section * [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) *** Modified at: November 7, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/) (An index of the documentation is available at /llms.txt) --- # Connect with Perl SingleStore supports the basic Perl drivers for MySQL using the standard DBI and DBM drivers. You may install and configure Perl using any of the supported methods, but you must have the MySQL driver to get Perl to work with SingleStore. You can also use the [MariaDB Perl library](https://docs.singlestore.com/#section-idm4547852021628833399531792337.md). ## Install Drivers for Perl The following example installs the Perl drivers with the [basic CPAN method](https://dev.mysql.com/doc/refman/8.0/en/perl-installation.html): ```shell $ perl -MCPAN -e shell cpan> install DBI cpan> install DBD::mysql ``` This command installed the following version of DBI and DBD: ```shell perl -MDBI -e 'print $DBI::VERSION."\n"' # 1.642 perl -MDBD::mysql -e 'print $DBD::mysql::VERSION."\n"' # 4.050 ``` Here's a sample script to test connectivity using the installed drivers. The following script connects to a cluster and retrieves some basic information about the cluster. ```perl #!/usr/bin/perl use strict; use DBI; my $host = "connect("DBI:mysql:database=$database;host=$host",$user, $pw) or die "Cannot connect to MySQL server\n"; my $sql = 'select @@memsql_version'; my $sth = $dbh->prepare($sql); $sth->execute(); while (my @row = $sth->fetchrow_array) { print "MemSQL_Version: $row[0] \n"; } my $sql = 'select count(*) from aggregators'; my $sth = $dbh->prepare($sql); $sth->execute(); while (my @row = $sth->fetchrow_array) { print "Aggregators: $row[0] \n"; } my $sql = 'select count(*) from leaves'; my $sth = $dbh->prepare($sql); $sth->execute(); while (my @row = $sth->fetchrow_array) { print "Leaves $row[0] \n"; } my $sql = 'select variable_name, variable_value from information_schema.global_status where variable_name = "uptime"'; my $sth = $dbh->prepare($sql); $sth->execute(); while (my @row = $sth->fetchrow_array) { #print "variable_name $row[0] variable_value $row[1]\n"; print "Cluster has been up for $row[1] seconds! \n"; } ``` Replace `ip_address_or_hostname` in the code above with the IP address or the hostname of your cluster. Here's a sample output: ```shell $ ./show_cluster.pm MemSQL_Version: 6.7.14 Aggregators: 1 Leaves 2 Cluster has been up for 16025 seconds! ``` ## DBD-MariaDB Perl Library SingleStore also supports the MariaDB Perl Library v1.11 (GPLv2). | Binary | URL | | ------------------------------------------ | ------------------------------------------------------------------------------------------------- | | Platform independent, includes source code | [DBD-MariaDB-1.11.tar.gz](https://cpan.metacpan.org/authors/id/P/PA/PALI/DBD-MariaDB-1.11.tar.gz) | *** Modified at: April 7, 2023 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl/) (An index of the documentation is available at /llms.txt) --- # Connect with PHP You can connect to SingleStore via PHP using [PHP Data Objects (PDO)](https://www.php.net/manual/en/book.pdo.php) or [MySQLi](https://www.php.net/manual/en/book.mysqli.php). To connect using PDO, you need to create a DSN from the host, port, and database name, and then pass the DSN to the PDO constructor along with the database username and password. For example, > **📝 Note**: To read the connection information from the environment, use the `$_SERVER['VAR_NAME']` notation. ```php ``` To connect using MySQLi, pass the host, username, password, database name, and port to the `mysqli` constructor. For example, ```php $link = new mysqli($_SERVER['HOSTNAME'], $_SERVER['USERNAME'], $_SERVER['PASSWORD'], $_SERVER['DB_NAME'], $_SERVER['PORT']); ``` *** Modified at: March 3, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-php/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-php/) (An index of the documentation is available at /llms.txt) --- # Connect with Python For most Python use cases, SingleStore recommends using the [SingleStore Python client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md). The SingleStore Python client is a Python Database API compliant database connector that supports Python version 3.9+. This client supports the [Data API](https://docs.singlestore.com/cloud/reference/data-api.md). It also includes objects for interfacing with the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md). The [SingleStore Python Client API Reference](https://singlestoredb-python.labs.singlestore.com/api.html) lists the Python Database API objects, functions, and properties that the SingleStore Python client supports. You can also connect to your SingleStore clusters using the following Python client/libraries: * [SQLAlchemy](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-sqlalchemy.md) * [mysqlclient](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/supported-python-libraries.md) * [PyMySQL](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/supported-python-libraries.md) * [MySQL Connector/Python](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/supported-python-libraries.md) ## In this section * [Connect using the SingleStore Python Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) * [Connect with Django](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-django.md) * [Connect with LangChain](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-langchain.md) * [Connect with SQLAlchemy](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-with-sqlalchemy.md) * [Supported Python Libraries](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/supported-python-libraries.md) *** Modified at: April 22, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/) (An index of the documentation is available at /llms.txt) --- # Connect with R2DBC R2DBC (Reactive Relational Database Connectivity) is a specification and set of APIs for connecting to relational databases from Java and other JVM languages. The SingleStore R2DBC connector ("the connector") is a non-blocking, asynchronous Java client that enables reactive Java applications to connect to SingleStore using the R2DBC specification. Instead of the traditional JDBC "one thread per blocking query" model, R2DBC uses asynchronous I/O and reactive streams (for example, Publisher, Flux, and Mono) to integrate seamlessly with reactive frameworks, such as [Spring WebFlux](https://docs.spring.io/spring-framework/reference/web/webflux.html) and [Project Reactor](https://projectreactor.io/docs). ## Driver Compatibility * Java 8+ JDK * SingleStore version 9.0+ ## Install the SingleStore R2DBC Connector To install the connector using Maven, include the following dependency in the `pom.xml` file of your project. Replace `x.x.x` with the connector version you want to use. ```xml com.singlestore r2dbc-singlestore x.x.x ``` > **📝 Note**: To find the latest version of the driver, refer to the [R2DBC Driver for SingleStore](https://mvnrepository.com/artifact/com.singlestore/r2dbc-singlestore) Maven repository. ## Use the SingleStore R2DBC Connector Create a `ConnectionFactory` to configure the connection to your SingleStore deployment. Use either of the following methods: * **Builder API**: Use `SingleStoreConnectionConfiguration` and `SingleStoreConnectionFactory` for an explicit, type-safe configuration. For example: ```java import com.singlestore.r2dbc.SingleStoreConnectionConfiguration; import com.singlestore.r2dbc.SingleStoreConnectionFactory; import io.r2dbc.spi.ConnectionFactory; SingleStoreConnectionConfiguration conf = SingleStoreConnectionConfiguration.builder() .host("") .port() .username("") .password("") .database("") .build(); // Implementation of io.r2dbc.spi.ConnectionFactory ConnectionFactory factory = new SingleStoreConnectionFactory(conf); ``` * **Connection URL**: Use the standard SPI (Service Provider Interface) factory lookup and a R2DBC URL. The connection string must have the following format: `r2dbc:singlestore:[sequential:|loadbalancing:]//:@:/[database][?connection_options=value]`  For example: ```java import io.r2dbc.spi.ConnectionFactories; import io.r2dbc.spi.ConnectionFactory; ConnectionFactory factory = ConnectionFactories.get("r2dbc:singlestore://s2user:p455w04d@svchost:3306/dbTest"); ``` Refer to [Connection Options](https://docs.singlestore.com/#section-id235395480457129.md) for the options supported in the connection configuration. ## Failover and Load-Balancing Modes Failover occurs when the connection to the primary host becomes unavailable and the connector tries reconnecting to another host. During failover, the failed host is blacklisted for 60 seconds. The connector always attempts to connect to the non-blacklisted hosts first. However, if all the hosts are blacklisted, the connector can retry connecting to a blacklisted host before the 60-second cooldown. Upon failover, the connector attempts the following: 1. **Connection recovery**: The connector tries reconnecting to another host in the host list. 2. **Re-run the command** (transaction): After a connection is re-established successfully, the connector attempts to re-run the affected query. After a successful reconnection: * If the connector can replay the query successfully, the query is run without returning any connectivity-related errors. * If the connector is unable to transparently handle the failure scenario, it returns an error. The connector supports the following load-balancing modes: * `sequential`  * `loadbalancing`  Load balancing enables the connector to distribute the load across multiple hosts. While initializing a connection or after a connection failure, the connector randomly selects a host and attempts to establish a connection. Once the connection is established, all the subsequent queries are run on this host until the connection is closed or fails. Refer to the [SingleStore R2DBC Connector](https://github.com/singlestore-labs/singlestore-connector-r2dbc) GitHub repository for more information. ## Connection Options | Option | Description | Default Value | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | | `host` | Comma-separated list of endpoints or IP addresses of theSingleStoredeployment. If the first host is unreachable, the connector uses the next specified host, and so on.This option is non-operational if the`socket`option is set. | | | `port` | Port of theSingleStoredeployment. | `3306` | | `username` | Username of theSingleStoredatabase user with which to connect. | | | `password` | Password for theSingleStoredatabase user. | | | `database` | Name of theSingleStoredatabase to connect with. | | | `connectTimeout` | Specifies the connection timeout value (in seconds). Upon timeout, attempts to establish a connection to the current host are aborted. | `10`(seconds) | | `tcpKeepAlive` | Specifies whether the underlying network connection uses TCP keepalive probes. If enabled, idle TCP connections remain active. | `FALSE` | | `tcpAbortiveClose` | If enabled, resets TCP connections using an abortive (or hard) close instead of an orderly (graceful) close.This option is useful in environments where connections are opened and closed in rapid succession, where new sockets may eventually fail to be created because all local ephemeral ports are consumed by TCP connections in the`TCP_WAIT`state. | `FALSE` | | `socket` | Enables the use of a Unix domain socket for faster database connections where the database is locally deployed. | | | `allowMultiQueries` | Enables the connector to submit multiple queries in a single call.Because it allows for SQL injection, this may pose a security risk. SingleStore does not recommend enabling this option. | `FALSE` | | `connectionAttributes` | Specifies additional client information to send to the database. This enables the database to identify the client or application associated with each connection.Query the[MV\_CONNECTION\_ATTRIBUTES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-connection-attributes.md)information schema view for this information. | | | `sessionVariables` | Specifies the session variables to set upon a successful connection. | | | `tlsProtocol` | Force enables TLS/SSL protocol for the connection, specified as a comma-separated list. | Java default | | `serverSslCert` | Use this option to specify the server's certificate in DER format or specify the server's CA certificate. This allows the connection to trust a self-signed certificate.You can specify the server certificate in one of the following formats:
  • Specify the complete path: serverSslCert=/full_path/cert.pem
  • Specify a relative path: serverSslCert=classpath:relative/cert.pem
  • Specify a DER-encoded certificate string: "------BEGIN CERTIFICATE-----"
| | | `clientSslCert` | Use this option to specify the client's certificate in DER format. Use this option exclusively for mutual authentication.You can specify the client certificate in one of the following formats:
  • Specify the complete path: clientSslCert=/full_path/cert.pem
  • Specify a relative path: clientSslCert=classpath:relative/cert.pem
  • Specify a DER-encoded certificate string: "------BEGIN CERTIFICATE-----"
| | | `clientSslKey` | The client's private key path for mutual authentication. | | | `clientSslPassword` | Password of the client private key. | | | `sslMode` | Specifies the SSL/TLS mode. It can have one of the following values:
  • DISABLE: Disable the use of SSL/TLS.
  • TRUST: Use SSL/TLS only for encryption, and do not perform certificate or hostname verification. SingleStore does not recommend this mode for production environments.
  • VERIFY-CA: Use SSL/TLS for encryption and perform certificate verification, but do not perform hostname verification.
  • VERIFY-FULL: Use SSL/TLS for encryption, certificate verification, and hostname verification.
  • TUNNEL: Connect using a pre-established SSL tunnel. Refer to sslContextBuilderCustomizer and sslTunnelDisableHostVerification options for related information.
| `DISABLE` | | `sslContextBuilderCustomizer` | Enables customization of SSL context builder. | | | `sslTunnelDisableHostVerification` | Disables hostname verification during SSLHandshake when`sslMode`is set to`TUNNEL`. | | | `useServerPrepStmts` | Enables using prepared statements. Prepared statements are pre-compiled on the server-side before executing. Applications that use the same queries repeatedly can enable this option. In general, it's preferable to leave this parameter as`FALSE`, becauseSingleStorehas wider support of queries in text protocol and has internal mechanisms of query parameterization. | `FALSE` | | `prepareCacheSize` | Specifies the prepared statement cache size (number of saved statements).If`useServerPrepStmts`is set to`TRUE`, it caches prepared statements in a LRU cache to avoid preparing the query again, which allows the server to avoid re-parsing a query. For subsequent runs of a command, only the prepared identifier and parameters (if any) are sent to the server. | `256` | | `pamOtherPwd` | Specifies an additional password for PAM authentication in multi-step authentication. If multiple passwords are used, the value must be URL-encoded. | | | `autoCommit` | Sets the`autocommit`value on connection initialization. | | | `loopResources` | Enables sharing a Netty`EventLoopGroup`among multiple asynchronous libraries or frameworks. | | | `skipPostCommands` | Specifies whether initialization commands sent after a connection is established are skipped. This helps avoid unnecessary commands upon connection creation and prevents session pinning when using an RDV proxy.The connector expects the server to meet the following conditions:
  • Connection exchanges use UTF-8 (mb3/mb4).
  • Autocommit is enabled.
| | | `enableExtendedDataTypes` | Enables extended data types, by enabling the`enable_extended_types_metadata`engine variable, that allow the connector to support extended data types, such as`VECTOR`and`BSON`. | | ## Example The following example connects to a SingleStore deployment, inserts data into a table, and then queries the data. 1. Create a Maven project. ```shell mvn archetype:generate \ -DgroupId=com.example \ -DartifactId=singlestore-r2dbc-demo \ -DarchetypeArtifactId=maven-archetype-quickstart \ -DinteractiveMode=false ``` 2. Add the required dependencies and plugins to the `pom.xml` file of the project. This example includes the following: ```xml com.singlestore r2dbc-singlestore 1.0.0 io.projectreactor reactor-core 3.6.5 org.slf4j slf4j-simple 2.0.9 junit junit 3.8.1 test org.apache.maven.plugins maven-compiler-plugin 3.11.0 org.codehaus.mojo exec-maven-plugin 3.1.0 com.example.R2dbcExample ``` 3. Create a main class. This example builds a `SingleStoreConnectionConfiguration` and `SingleStoreConnectionFactory` using the connector’s builder API. ```java package com.example; import com.singlestore.r2dbc.SingleStoreConnectionConfiguration; import com.singlestore.r2dbc.SingleStoreConnectionFactory; import io.r2dbc.spi.Connection; import io.r2dbc.spi.ConnectionFactory; import reactor.core.publisher.Flux; import reactor.core.publisher.Mono; public class R2dbcExample { public static void main(String[] args) { // 1) Configure connection (update these with your actual values) SingleStoreConnectionConfiguration conf = SingleStoreConnectionConfiguration.builder() .host("svchost") .port(3306) .username("s2user") .password("passkey") .database("dbTest") .build(); // builder style // 2) Create a ConnectionFactory ConnectionFactory factory = new SingleStoreConnectionFactory(conf); // 3) Use R2DBC reactively: create table, insert, and query Mono.from(factory.create()) .flatMapMany(connection -> // use a helper that ensures the connection is closed runExampleQueries(connection) .doFinally(signal -> Mono.from(connection.close()).subscribe()) ) .doOnNext(rowString -> System.out.println("Row: " + rowString)) .blockLast(); // block to keep the JVM alive for this simple demo } private static Flux runExampleQueries(Connection connection) { // Simple schema + data String createTableSql = "CREATE TABLE IF NOT EXISTS demo_r2dbc (" + " id INT PRIMARY KEY, " + " name VARCHAR(64)" + ")"; String insertSql = "INSERT INTO demo_r2dbc (id, name) VALUES (1, 'Alice'), (2, 'John'), (3, 'Jane') " + "ON DUPLICATE KEY UPDATE name = VALUES(name)"; String selectSql = "SELECT id, name FROM demo_r2dbc"; // Execute steps in sequence: return Mono.from( connection.createStatement(createTableSql) .execute() ) .flatMapMany(result -> result.getRowsUpdated()) .thenMany( Mono.from( connection.createStatement(insertSql) .execute() ).flatMapMany(r -> r.getRowsUpdated()) ) .thenMany( Mono.from( connection.createStatement(selectSql) .execute() ) .flatMapMany(result -> result.map((row, meta) -> { Integer id = row.get("id", Integer.class); String name = row.get("name", String.class); return "id=" + id + ", name=" + name; }) ) ); } } ``` Alternatively, you can construct the `ConnectionFactory` with a R2DBC URL instead of the builder. For example: ```java ConnectionFactory factory = ConnectionFactories.get( "r2dbc:singlestore://s2user:password@svchost:3306/dbtest" ); ``` 4. Build and run the application. Run the following command from the project root directory. ```shell mvn -q clean compile exec:java ``` ```output Row: id=1, name=Alice Row: id=2, name=John Row: id=3, name=Jane ``` A similar output indicates that the application ran as expected. *** Modified at: January 30, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-r-2-dbc/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-r-2-dbc/) (An index of the documentation is available at /llms.txt) --- # Connect with Ruby SingleStore is wire-compliant with MySQL. To connect to your database with Ruby, you can use: * Standard Ruby * Ruby on Rails ## Connect with Standard Ruby For a complete example, including samples of CRUD operations, refer to [Getting started with SingleStore and Ruby](https://github.com/singlestore-labs/start-with-singlestore-ruby) GitHub repository. ## Prerequisites * [Ruby](https://www.ruby-lang.org/en/downloads/) (version 2.6.x or 3.x) * `mysql2` gem ## Connection Details To connect your Ruby application to your cluster, you'll need the following: * **Host**: Endpoint or IP Address of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Default is `3306` * **User**: Username of the SingleStore database user with which to connect. * **Password**: Password for the SingleStore database user . * **Database**: Name of the SingleStore database to connect with. Here is an example of a Ruby connection for a SingleStore Helios cluster: ```ruby client = Mysql2::Client.new( :host => "svc-1234ebc1-ab6f-123d-8e67-cd5d066ca0e8-ddl.aws-region-1.svc.singlestore.com", :username => "admin", :password => "", :database => "test" ) ``` For more examples, refer to [Getting started with SingleStore stored procedures and Ruby](https://github.com/singlestore-labs/start-with-singlestore-ruby-stored-procedure). ## Connect with Ruby on Rails To connect with Ruby on Rails, configure your SingleStore Helios. Open `config/database.yml` and edit the development adapter to use SingleStore. You will need to have the SingleStore socket which can be found using the query `SHOW VARIABLES LIKE 'socket'`. For example: ```shell singlestore -u 's2user' -p -h 'svchost-xxxx' -P 3306 -e "SHOW VARIABLES LIKE 'socket'" ``` ```output +---------------+---------------------------------------+ | Variable_name | Value | +---------------+---------------------------------------+ | socket | /var/lib/memsql/xxxx/data/memsql.sock | +---------------+---------------------------------------+ ``` SingleStore can run queries using the `mysql2` connector in controller code. Calling execute returns a `mysql2` result, which can be converted to an array and easily displayed: ```ruby class WelcomeController < ApplicationController def index render plain: ActiveRecord::Base.connection.execute('show databases').to_a.join(' ') end end ``` *** Modified at: December 14, 2022 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby/) (An index of the documentation is available at /llms.txt) --- # Connect with Rust To connect SingleStore with Rust, use [SQLx](https://github.com/launchbadge/sqlx) with its MySQL feature. Refer to the following **Cargo.toml** file: ```toml [package] name = "rust-singlestore-connect" version = "0.1.0" edition = "2021" [dependencies] sqlx = { version = "0.5", features = ["runtime-tokio-native-tls", "mysql"] } tokio = { version = "1", features = ["full"] } ``` For more information, see [SQLx Docs](https://github.com/launchbadge/sqlx). Here's a sample **main.rs** file: ```rust use sqlx::mysql::{MySqlConnectOptions, MySqlPoolOptions}; #[tokio::main] async fn main() -> Result<(), sqlx::Error> { let opts = MySqlConnectOptions::new() .host("svc-ab8077f6-7b03-4ba3-b557-063c53eff943-ddl.aws-oregon-2.svc.singlestore.com") .username("admin") .password("PASSWORD"); let pool = MySqlPoolOptions::new().connect_with(opts).await?; let row: (i64,) = sqlx::query_as("SELECT ?") .bind(150_i64) .fetch_one(&pool) .await?; assert_eq!(row.0, 150); Ok(()) } ``` *** Modified at: May 12, 2026 Source: [/cloud/developer-resources/connect-with-application-development-tools/connect-with-rust/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-rust/) (An index of the documentation is available at /llms.txt) --- # Extended Protocol Packet Metadata SingleStore uses the same format of [column definition packet](https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response_text_resultset_column_definition.html) when communicating between the client and the server as the [MySQL protocol](https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html). By default, packet metadata sent to the client consists of 12 bytes, which does not allow distinguishing non-MySQL protocol-compatible data types. Extending the packet metadata size allows the client to distinguish between data types that are not compatible with MySQL, such as `BSON` and `VECTOR`. Refer to [enum\_field\_types Reference](https://dev.mysql.com/doc/dev/mysql-server/latest/field__types_8h.html#a69e798807026a0f7e12b1d6c72374854) for a list of MySQL supported data types. By default, SingleStore does not send extended metadata in the protocol packet for backward compatibility. When extended metadata in the protocol packet is disabled (default behavior), `BSON` and `VECTOR` type columns are sent to the clients as `LONGBLOB` and `VARCHAR`, respectively. Refer to [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) for more information on the respective data types. Extended protocol packet metadata is supported in SingleStore version 8.5.28 and later. **Note**: The MariaDB C library does not support extended metadata format. ## Enable Extended Metadata To enable sending extended metadata in the protocol packet for `BSON` and `VECTOR` data types: ```sql SET SESSION enable_extended_types_metadata = TRUE; ``` `enable_extended_types_metadata` is a session variable that can be set globally. SingleStore recommends configuring this engine variable for each session. ## Extended Metadata Format The packet metadata for extended types has the following structure: | Type | Name | Description | | ---------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `string` | catalog | Catalog name. Currently, this value is always`"def"`. | | `string` | database | Database name | | `string` | table | Virtual table name | | `string` | org\_table | Physical table name | | `string` | name | Virtual column name | | `string` | org\_name | Physical column name | | `int` | | Length of the fixed length fields. This value is greater than or equal to 12 (0x0C) bytes. | | `int<2>` | character\_set | The column[character set](https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_character_set.html). | | `int<4>` | column\_length | Maximum length of the field in bytes. | | `int<1>` | type | Column[type](https://dev.mysql.com/doc/dev/mysql-server/latest/field__types_8h.html#details). | | `int<2>` | flags | [Column definition flags](https://dev.mysql.com/doc/dev/mysql-server/latest/group__group__cs__column__definition__flags.html). | | `int<1>` | decimals | Number of decimal places. | | `int<2>` | unused | Two unused zero-bytes. | | `int<1>` | type code of extended datatype | Type code of the extended type. This value is`2`for`VECTOR`data types and`1`for`BSON`types. | where, `lenenc` represents a length-encoded string. For `VECTOR` types, the protocol packet metadata contains the following additional bytes: | Type | Name | Description | | -------- | ------------------------- | ---------------------------------------- | | `int<4>` | number of vector elements | Dimension of the`VECTOR`type. | | `int<1>` | type of vector elements | The type of elements in the vector data. | ## For `VECTOR` Types For `VECTOR` types, the protocol packet metadata contains the following **6 bytes** in addition to the default 12 bytes: 1. **1 byte**: Specifies the type code. This value is `2` for `VECTOR` data types. 2. **4 bytes**: Specifies the dimension (number of elements) of the vector data. 3. **1 byte**: Specifies the type of elements in the vector data. It can have the following values based on the type of the vector elements: | Value | Vector Element Type | | ----- | ------------------- | | `1` | `F32` | | `2` | `F64` | | `3` | `I8` | | `4` | `I16` | | `5` | `I32` | | `6` | `I64` | For example, the protocol packet metadata for a `VECTOR(23)` type column (starting from the "Length of the fixed length field" byte) is: | Extended Types Disabled | Extended Types Enabled | | -------------------------------------- | ----------------------------------------------------------- | | 0c 3f 00 ff ff ff ff fd 90 00 00 00 00 | 12 3f 00 ff ff ff ff fd 90 00 00 00 00**02 17 00 00 00 01** | where the extended bytes represent the following: * `02`: Type code for `VECTOR` type. * `17 00 00 00`: Dimension (length) of the vector data, i.e., `23`. * `01`: Vector element type, i.e., `F32`. ## For `BSON` Types For `BSON` types, the protocol packet metadata contains **1 byte** in addition to the default 12 bytes. This additional byte specifies the type code, which is `1` for `BSON` types. *** Modified at: February 25, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/extended-protocol-packet-metadata/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/extended-protocol-packet-metadata/) (An index of the documentation is available at /llms.txt) --- # Using Prepared Statements > **📝 Note**: For SingleStore Helios, use the endpoint (available in the Cloud Portal) to run the commands discussed in this topic. If you are using an older version of SingleStore Helios, or using SingleStore Helios with Audit Logging enabled, SingleStore does not support server-side prepared statements. Instead, many client drivers support a configuration option to allow client-side prepared statements. This configuration option may also be referred to as client-side prepared statement emulation or parameter interpolation. You can use the same code in your application, but change a flag in the client driver configuration. For example: * In the [MySQL Connector/J (JDBC)](https://dev.mysql.com/doc/connector-j/en/connector-j-connp-props-prepared-statements.html) driver, set `useServerPrepStmts=false`, which is the default configuration. * In [MariaDB Connector/J](https://mariadb.com/kb/en/about-mariadb-connector-j/), set `useServerPrepStmts=false`. * In [ODBC](https://dev.mysql.com/doc/connector-odbc/en/connector-odbc-configuration-connection-parameters.html), set `no_ssps=1`. * In the [Go MySQL](https://github.com/go-sql-driver/mysql#interpolateparams) driver, set `interpolateParams=true`. * In [PHP PDO](https://www.php.net/manual/en/pdo.setattribute.php), set `PDO::ATTR_EMULATE_PREPARES=TRUE`. In many other databases, server-side prepared statements provide performance advantages, but SingleStore Helios already compiles and caches query plans internally, so SingleStore Helios does not need server-side prepared statements to get most of those performance advantages. *** Modified at: February 20, 2025 Source: [/cloud/developer-resources/connect-with-application-development-tools/using-prepared-statements/](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/using-prepared-statements/) (An index of the documentation is available at /llms.txt) --- # Connect with Data API SingleStore provides the Data API to execute SQL statements against your database over an HTTP connection. The Data API can be used to integrate SingleStore with serverless architecture, develop custom applications, and build seamless integrations with applications. SingleStore's Data API is authenticated to the server using the HTTP Basic and Bearer Authentication standards. To authenticate using JWTs, use the Bearer authentication standard. The Data API uses standard HTTP methods and response codes and supports URI versioning. It also uses JSON-encoded responses with support for conventional data type conversions. The Data API is accessible via a built-in HTTP server. The Data API supports all SQL statements that can be run in a direct database connection with SingleStore Helios. To learn more about SingleStore's Data API, refer to the following topics. * [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) * [Data API Authentication](https://docs.singlestore.com/cloud/reference/data-api/data-api-authentication.md) * [Access the Data API](https://docs.singlestore.com/cloud/reference/data-api/access-the-data-api.md) * [Enable the Data API](https://docs.singlestore.com/cloud/reference/data-api/enable-the-data-api.md) * [Data API Data Type Conversion](https://docs.singlestore.com/cloud/reference/data-api/data-api-data-type-conversion.md) * [Data API Error Handling](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling.md) * [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md) *** Modified at: October 18, 2024 Source: [/cloud/developer-resources/connect-with-data-api/](https://docs.singlestore.com/cloud/developer-resources/connect-with-data-api/) (An index of the documentation is available at /llms.txt) --- # Database Branching Database branching lets you quickly create private, independent instances of your database for testing, development, and data recovery. Branches share the same history as their parent, but once created, they diverge and become independent databases. This means you can insert, update, and delete data in a branch without impacting the parent database’s performance or stability. ## Benefits * It is a cost-effective solution that saves on infrastructure and time in maintaining duplicate environments. * You can create multiple isolated datasets of your production database, each containing data from a specific time. ## Use Cases * **Development and Testing** As a developer, you can instantly spin up isolated branches with the latest replica of the production database or at a specific point in time. This enables efficient testing and iterative development without impacting the production environment. * **Performance Tuning** You can evaluate multiple versions of your application on a branch, isolating and resolving any issues while the production database remains active. Any performance bugs can be investigated and tuned in the branch before implementing the solutions in the production environment. ## Creating a Branch Using SQL The following syntax creates a branch on the same cluster as the parent database: ```sql ATTACH DATABASE [AS ] ``` For all options available with this command refer [ATTACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database.md) ## Example Scenarios The following examples use a database named `sales`, and a table named `orders`. Create a database named `sales`: ```sql CREATE DATABASE sales; ``` Create a table named `orders`: ```sql CREATE TABLE orders (order_id bigint(11) NOT NULL, customer_id int(11) NOT NULL, order_date date NOT NULL, order_status char(1) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, total_price decimal(15,2) NOT NULL, SHARD KEY (order_id), SORT KEY (order_date)); ``` **Scenario 1**: As a developer, you want to test a new business logic before implementing the query on a production dataset. In the Portal UI, connect to the SQL Editor or the database endpoint and run the command: ```sql ATTACH DATABASE sales AS branch_sales; ``` You can navigate to the `orders` table within the `branch_sales` database and test your new business logic without impacting the production dataset. You can insert, update or delete rows, test the performance of your queries etc. Once done testing, you can simply drop the branch database: ```sql DROP DATABASE branch_sales; ``` **Scenario 2**: Branching can use PITR and this requires the database to be deployed using an Enterprise project. In the following case, a faulty query has deleted 1000 rows from the `orders` table. You can recover the data by simply creating a branch of your database at a point in time before the bad query was executed. Suppose the bad query was executed on January 2nd. You may branch the `sales` database at a timestamp before the query was executed and recover the data. ```sql ATTACH DATABASE sales AS recover_sales AT TIME '2024-01-02 21_57_31'; ``` To recover the data, you may query the missing rows and copy them to the production `orders` table. Alternatively, you may drop the `sales` database and then rename `recover_sales` as `sales`. Note: when you do this you will have to reconnect this newly renamed `sales` database to your application. To drop your existing `sales` database: ```sql DROP DATABASE IF EXISTS sales; ``` To rename `recover_sales` as `sales`: ```sql DETACH DATABASE recover_sales; ATTACH DATABASE recover_sales AS sales; ``` **Scenario 3**: To rename an existing database, create a new database branch from the original database. This branch serves as a point-in-time copy, containing all data from the original database exactly as it existed at the moment of branching. ``` ATTACH DATABASE x_db AS x_db_new_name; ``` After the branch is successfully created, you can detach or drop the original database. Finally, rename the newly created branch to the original database name. ## Using Branches You can create a branch at the current time or at a previous point in time. Branches are read/write by default. The branch database automatically inherits user permissions from the parent database. Users with either the `CREATE DATABASE` or `ATTACH DATABASE` permission can create branches. Users with `CREATE DATABASE` permission will automatically get the `ATTACH DATABASE` permission. However, users with only the `ATTACH DATABASE` permission can also create a branch. Branches are independent, so any updates to the branch database are not propagated to the parent database, and updates to the parent database are not propagated to the branch databases. Similarly dropping the branch database does not impact the parent database and vice versa. Creating a branch database duplicates the in-memory data and blob cache but not the data stored in the object store. Hence, storage used before the branch was created is not counted again and only new or updated data in the branch increases storage usage. *** Modified at: August 5, 2026 Source: [/cloud/developer-resources/database-branching/](https://docs.singlestore.com/cloud/developer-resources/database-branching/) (An index of the documentation is available at /llms.txt) --- # Functional Extensions ## In this section * [Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) * [Full Text VERSION 2 Custom Analyzers](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers.md) * [Configuring Full Text and Vector Indexes](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes.md) * [Working with Geospatial Features](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-geospatial-features.md) * [Loading Geospatial Data into SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/loading-geospatial-data-into-singlestore-helios.md) * [Hybrid Search - Reranking Full-Text and Vector Search Results](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md) * [Analyzing Time Series Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/analyzing-time-series-data.md) * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) * [How to Bulk Load Vectors](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/how-to-bulk-load-vectors.md) * [Tuning Vector Indexes and Queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries.md) * [Working with Window Functions](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-window-functions.md) *** Modified at: March 26, 2026 Source: [/cloud/developer-resources/functional-extensions/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/) (An index of the documentation is available at /llms.txt) --- # Analyzing Time Series Data Time series data consists of sequences of events, where each event includes a timestamp. Examples include events generated by utilities, energy production infrastructure, financial applications, software services, and Internet of Things (IoT) devices. This topic explains how to ingest, structure, and query time series data in SingleStore. ## Storing Time Series Data SingleStore stores time series data in [rowstore](https://docs.singlestore.com/cloud/create-a-database/rowstore.md) or [columnstore](https://docs.singlestore.com/cloud/create-a-database/columnstore.md) tables. Each row must include a time-based attribute that records when the event occurred. * Use `DATETIME(6)` to define the time attribute when fractional-second precision is required. * Use `DATETIME`, if fractional-second precision is not required. * Use `DATETIME(6)` or `DATETIME` instead of `TIMESTAMP`, because timestamp values are limited to dates through 2038. * Applications must supply timestamp values directly. Do not rely on server-side defaults such as `NOW()`. * By default, the time attribute uses the column name `ts`. The following example creates a table that stores time series events generated by a wind turbine. ```sql CREATE TABLE turbine_reading( tid int NOT NULL, -- turbine ID ts datetime(6) NOT NULL, rpm double, temperature double, vibration double, output double, wind_direction double, wind_speed double, SHARD(tid), SORT KEY(ts) ); ``` This table uses a columnstore table by default. Columnstore tables store data on disk, scale efficiently to large data sets, and provide the best performance for analytical queries that process large volumes of data. For most time series workloads, SingleStore recommends columnstore tables. If a time series table contains a large number of attributes and queries frequently retrieve individual rows, consider adding a [column group index](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works/#section-idm234822422484816.md) to the columnstore table. A column group index improves row lookup performance without sacrificing the scalability, compression, and scan performance benefits of columnstore storage. Consider using a rowstore table if either of the following conditions apply: * Your workload has a high update rate and the entire dataset fits in available memory. * You need low-latency point-in-time lookups using `AS OF` queries. Refer to [Finding a Row current AS OF a Point in Time](https://docs.singlestore.com/#section-id235648447697182.md) for more information. Rowstore query processing can efficiently locate a small number of rows within a narrow time range and return rows with many attributes more efficiently than columnstore processing for this access pattern. SingleStore recommends creating a `SORT KEY` on the `ts` column because time series queries filter on ranges of timestamp values. For columnstore tables, a `SORT KEY` stores the data in timestamp order, which allows the query engine to process range filters efficiently through [segment elimination](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments.md). For rowstore tables, a `KEY` on the `ts` column creates an index that enables efficient processing of range filters. For time series data sets with a large number of attributes, where applications retrieve all table attributes and the data fits within the available table memory, use a rowstore table to store the data. Rowstore query processing can efficiently locate a small number of rows within a narrow time range and return rows with many attributes more efficiently than columnstore query processing. ## Descriptive Data Store descriptive properties that remain unchanged across time series events in a separate table. For example, store information about individual turbines in a table such as the following: ```sql CREATE REFERENCE TABLE turbine( tid int, name varchar(60), model varchar(60), max_output double, latitude double, longitude double, PRIMARY KEY(tid) ); ``` Use a [reference table](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts/#UUID-712159bb-3006-3563-f42d-6188153d809f.md) for small collections of descriptive properties. For larger collections, use a standard (partitioned) table. The descriptive data table described in the previous section serves as a dimension table that links to the fact table containing the time series events. Dimensional modeling concepts commonly used in data warehouses also apply to time series data. The following example uses the sample turbine data in the table `turbine`: ```sql INSERT INTO turbine VALUES (1, 'Hood River A', 'Volkswind Mega 5', 5.0, 47.130, 113.187), (2, 'Hood River B', 'Volkswind Mega 5+', 5.3, 47.141, 113.199); ``` ## Ingesting Time Series Data SingleStore supports multiple methods for ingesting time series data. Select the method that matches with your workload: * [Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md): Use pipelines for continuous ingestion from files or Kafka topics and for high-volume data loads. * `LOAD DATA`:  Use `LOAD DATA` for bulk loading historical time series data from files. * `INSERT`: Use `INSERT` for ingesting individual rows or small batches directly from an application. You can use any combination of these methods. SingleStore optimizes data ingestion across all supported ingestion paths. ## Time Series Functions SingleStore provides a set of built-in functions designed to work together for time series analysis: | Function | Type | Purpose | | ------------- | --------- | ------------------------------------------------------------------- | | `FIRST` | Aggregate | Returns the value associated with the earliest timestamp in a group | | `LAST` | Aggregate | Returns the value associated with the latest timestamp in a group | | `TIME_BUCKET` | Scalar | Groups timestamps into fixed-width intervals | In addition to time series functions, you can use the `SERIES TIMESTAMP` option in the `CREATE TABLE` command to designate a table column as the default timestamp for time series functions.These features eliminate the need for complex window functions and CTEs when performing common time series operations such as downsampling, open/close calculations, and time-based grouping. The following example demonstrates the use of `FIRST`, `LAST`, `TIME_BUCKET`, and `SERIES TIMESTAMP` in a single query that downsamples 5-minute candlestick bars from raw trade data: ```sql CREATE TABLE tick ( ts DATETIME(6) SERIES TIMESTAMP, symbol VARCHAR(5), price NUMERIC(18,4), volume INT, SORT KEY (ts), SHARD KEY (symbol) ); INSERT INTO tick VALUES ('2024-01-15 09:30:15.000000', 'AAPL', 185.20, 1500), ('2024-01-15 09:31:45.000000', 'AAPL', 185.45, 800), ('2024-01-15 09:33:20.000000', 'AAPL', 185.10, 1200), ('2024-01-15 09:35:50.000000', 'AAPL', 185.80, 2000), ('2024-01-15 09:37:10.000000', 'AAPL', 186.00, 900), ('2024-01-15 09:38:30.000000', 'AAPL', 185.90, 1100), ('2024-01-15 09:40:05.000000', 'AAPL', 186.20, 1800), ('2024-01-15 09:42:45.000000', 'AAPL', 186.50, 700), ('2024-01-15 09:44:15.000000', 'AAPL', 186.10, 1300); SELECT TIME_BUCKET('5m') AS bucket, symbol, FIRST(price) AS open, LAST(price) AS close, MAX(price) AS high, MIN(price) AS low, SUM(volume) AS total_volume FROM tick WHERE symbol = 'AAPL' AND ts >= '2024-01-15 09:30:00' AND ts < '2024-01-15 16:00:00' GROUP BY bucket, symbol ORDER BY bucket; ``` ```output +----------------------------+--------+----------+----------+----------+----------+--------------+ | bucket | symbol | open | close | high | low | total_volume | +----------------------------+--------+----------+----------+----------+----------+--------------+ | 2024-01-15 09:30:00.000000 | AAPL | 185.2000 | 185.1000 | 185.4500 | 185.1000 | 3500 | | 2024-01-15 09:35:00.000000 | AAPL | 185.8000 | 185.9000 | 186.0000 | 185.8000 | 4000 | | 2024-01-15 09:40:00.000000 | AAPL | 186.2000 | 186.1000 | 186.5000 | 186.1000 | 3800 | +----------------------------+--------+----------+----------+----------+----------+--------------+ ``` ## Querying Time Series Data Continuing the wind turbine example from the previous section, assume the following data is inserted into the `turbine_reading` table: ```sql INSERT INTO turbine_reading VALUES (1, '2020-03-14 13:00:33', 10, 33, 100, 1000000, 90, 15), (1, '2020-03-14 13:00:34', 10, 33, 100, 1000000, 90, 15), (1, '2020-03-14 13:00:35', 11, 33, 105, 1050000, 91, 16), (1, '2020-03-14 13:00:36', 11, 33.1, 104, 1000000, 90, 16), (2, '2020-03-14 13:00:33', 18, 30, 170, 2000000, 0, 23), (2, '2020-03-14 13:00:34', 18, 30, 170, 2000000, 0, 23), (2, '2020-03-14 13:00:35', 18.5, 30, 176, 2050000, 0, 23.5), (2, '2020-03-14 13:00:36', 19, 30.1, 174, 2070000, 1, 23.6), (1, '2020-03-15 13:00:33', 11, 32, 99, 1010000, 45, 15.1), (1, '2020-03-15 13:00:34', 11, 32, 99, 1020000, 45, 15.2), (1, '2020-03-15 13:00:35', 12, 32.1, 101, 1030000, 45, 15.2), (1, '2020-03-15 13:00:36', 13, 32.15, 102, 1030000, 46, 15.2); ``` The following query demonstrates how to calculate the average of all time series values in the table. ```sql -- average RPM by turbine SELECT tid, AVG(rpm) FROM turbine_reading GROUP BY tid; ``` ```output +-----+----------+ | tid | AVG(rpm) | +-----+----------+ | 2 | 18.375 | | 1 | 11.125 | +-----+----------+ ``` ## Time Bucketing Use the `TIME_BUCKET` function to group timestamps into fixed intervals and return the start time of each bucket. The following example uses `TIME_BUCKET` to group time series data into 5-day intervals and calculate the average value for each interval: ```sql SELECT tid, TIME_BUCKET("5d", ts), AVG(output) FROM turbine_reading GROUP BY 1, 2 ORDER BY 1, 2; ``` ```output +-----+----------------------------+-------------+ | tid | TIME_BUCKET("5d", ts) | AVG(output) | +-----+----------------------------+-------------+ | 1 | 2020-03-13 00:00:00.000000 | 1017500 | | 2 | 2020-03-13 00:00:00.000000 | 2030000 | +-----+----------------------------+-------------+ ``` To bucket data by day, one approach is to cast a high-resolution `DATETIME(6)` value to the `DATE` data type. To bucket data by a fixed interval of `N` seconds, convert the timestamp to a `unix_timestamp`, divide the result by `N` using the `DIV` operator, multiply the result by `N`, and then convert it back to a timestamp value. Dividing by `N` and then multiplying by `N` removes the remainder and produces a value that is evenly divisible by `N`. This value represents the start of the time bucket and provides a standardized timestamp for grouping data. ```sql -- Find high, low, and average output for each turbine, bucketed by day, -- sorted by day. SELECT tid, ts :> date, MIN(output), MAX(output), AVG(output) FROM turbine_reading GROUP by 1, 2 ORDER BY 1, 2; ``` ```output +-----+------------+-------------+-------------+-------------+ | tid | ts :> date | MIN(output) | MAX(output) | AVG(output) | +-----+------------+-------------+-------------+-------------+ | 1 | 2020-03-14 | 1000000 | 1050000 | 1012500 | | 1 | 2020-03-15 | 1010000 | 1030000 | 1022500 | | 2 | 2020-03-14 | 2000000 | 2070000 | 2030000 | +-----+------------+-------------+-------------+-------------+ ``` ```sql -- Find high, low, and average output for each turbine, -- bucketed by three second intervals, sorted by interval start time. SELECT tid, from_unixtime(unix_timestamp(ts) DIV 3 * 3) as ts, MIN(output), MAX(output), AVG(output) FROM turbine_reading GROUP by 1, 2 ORDER BY 1, 2; ``` ```output +-----+---------------------+-------------+-------------+--------------------+ | tid | ts | MIN(output) | MAX(output) | AVG(output) | +-----+---------------------+-------------+-------------+--------------------+ | 1 | 2020-03-14 13:00:33 | 1000000 | 1050000 | 1016666.6666666666 | | 1 | 2020-03-14 13:00:36 | 1000000 | 1000000 | 1000000 | | 1 | 2020-03-15 13:00:33 | 1010000 | 1030000 | 1020000 | | 1 | 2020-03-15 13:00:36 | 1030000 | 1030000 | 1030000 | | 2 | 2020-03-14 13:00:33 | 2000000 | 2050000 | 2016666.6666666667 | | 2 | 2020-03-14 13:00:36 | 2070000 | 2070000 | 2070000 | +-----+---------------------+-------------+-------------+--------------------+ ``` ## Gap Filling When you aggregate time series data into buckets, some intervals may not contain data points. These missing intervals create gaps in the output, which can cause issues for dashboards, alerting systems, and analytics pipelines that expect a continuous sequence of time buckets. For example, if you bucket turbine readings into 1-second intervals over a 10-second window, but data exists only for seconds 33 through 36, the result contains 4 rows instead of 10. Gap filling ensures that every expected time bucket appears in the result set. This approach uses three components: 1. A reference table of time buckets: A table that contains one row per time interval and covers the full range of interest without gaps. 2. A `LEFT JOIN`: A left outer join between the reference table and the aggregated data that preserves all time buckets in the output. 3. `NVL` for default values: A function that replaces `NULL` values (where no data exists) with a default value, such as `0`. Create a table that contains one row per time interval for the required time range. In a performance-intensive workload, consider creating `time_days` as a reference table to avoid any need to shuffle or broadcast during the `LEFT JOIN`. Because reference tables are replicated to all nodes, joins against them generally avoid network-intensive data redistribution of fact tables. For example, the following table stores one row per day: ```sql CREATE REFERENCE TABLE time_days ( ts DATETIME(6) NOT NULL, PRIMARY KEY (ts) ); ``` Populate the table with rows that cover the desired time range. ``` INSERT INTO time_days (ts) VALUES ('2020-03-14 00:00:00'), ('2020-03-15 00:00:00'), ('2020-03-16 00:00:00'); ``` Use the `turbine_reading` table defined in [Storing Time Series Data](https://docs.singlestore.com/#section-id235648365943266.md) without gap filling, a standard bucketed aggregation returns only intervals where data exists: ```sql SELECT TIME_BUCKET('1d', ts) AS bucket, tid, SUM(output) AS total_output, AVG(output) AS avg_output FROM turbine_reading WHERE ts >= '2020-03-14 00:00:00' AND ts < '2020-03-17 00:00:00' GROUP BY bucket, tid ORDER BY tid, bucket; ``` ```output +----------------------------+-----+--------------+------------+ | bucket | tid | total_output | avg_output | +----------------------------+-----+--------------+------------+ | 2020-03-14 00:00:00.000000 | 1 | 4050000 | 1012500 | | 2020-03-15 00:00:00.000000 | 1 | 4090000 | 1022500 | | 2020-03-14 00:00:00.000000 | 2 | 8120000 | 2030000 | +----------------------------+-----+--------------+------------+ ``` Note that turbine 2 is missing a row for `2020-03-15`, and neither turbine has a row for `2020-03-16`. A dashboard plotting this data would show discontinuous lines or missing data points for those intervals. The following query fills the gaps using a `LEFT JOIN` to the reference table: ```sql WITH filtered AS ( SELECT * FROM turbine_reading WHERE ts >= '2020-03-14 00:00:00' AND ts < '2020-03-17 00:00:00' ), turbines AS ( SELECT DISTINCT tid FROM filtered ), agg AS ( SELECT TIME_BUCKET('1d', ts) AS bucket, tid, SUM(output) AS total_output, AVG(output) AS avg_output FROM filtered GROUP BY 1, 2 ), ref AS ( SELECT TIME_BUCKET('1d', td.ts) AS bucket, t.tid FROM time_days td CROSS JOIN turbines t WHERE td.ts >= '2020-03-14 00:00:00' AND td.ts < '2020-03-17 00:00:00' ) SELECT ref.bucket, ref.tid, NVL(agg.total_output, 0) AS total_output, NVL(agg.avg_output, 0) AS avg_output FROM ref LEFT JOIN agg ON agg.bucket = ref.bucket AND agg.tid = ref.tid ORDER BY ref.tid, ref.bucket; ``` ```output +----------------------------+-----+--------------+------------+ | bucket | tid | total_output | avg_output | +----------------------------+-----+--------------+------------+ | 2020-03-14 00:00:00.000000 | 1 | 4050000 | 1012500 | | 2020-03-15 00:00:00.000000 | 1 | 4090000 | 1022500 | | 2020-03-16 00:00:00.000000 | 1 | 0 | 0 | | 2020-03-14 00:00:00.000000 | 2 | 8120000 | 2030000 | | 2020-03-15 00:00:00.000000 | 2 | 0 | 0 | | 2020-03-16 00:00:00.000000 | 2 | 0 | 0 | +----------------------------+-----+--------------+------------+ ``` Every turbine now has a row for every day in the range. Intervals without data show 0 instead of being absent from the result. This query works as follows: * The `ref` subquery generates every combination of time bucket and turbine ID by cross-joining the reference table (bucketed using `TIME_BUCKET`) with the distinct turbine IDs in the data range. This ensures a row exists for every turbine in every time bucket in the interval. * The `agg` subquery computes the actual aggregates (`SUM`, `AVG`) grouped by bucket and turbine. * The `LEFT JOIN` preserves all rows from ref, including the intervals without readings. The `NVL` function replaces the resulting `NULL` values with `0`. Gap filling is useful in the following scenarios: * When visualizing time series in charts where gaps would break line continuity or mislead viewers. * When feeding data to monitoring or alerting systems that expect values at regular intervals. * When performing calculations (such as moving averages or cumulative sums) that require a complete sequence of equally-spaced data points. * When exporting data to external systems that do not handle missing intervals gracefully. The gap-filling approach uses `NVL` to replace missing intervals with a default value of `0`. Use interpolation to estimate values from surrounding data points instead of using a fixed default. ## Interpolation A time series may contain gaps that you want to fill so that a data point exists at every interval for a chosen time granularity. Gaps commonly occur when you convert a time series with data points at irregular intervals into one with data points at regular intervals by bucketing the data. For example, if data points arrive at random intervals approximately every half second, some one-second intervals may contain no data points. When you bucket the data into one-second intervals, those intervals appear as gaps in the resulting time series. You can interpolate missing data points by using a stored procedure. The following example demonstrates this approach with a simple set of stock ticks. In this example, the procedure interpolates data points that are missing from a time series that has already been bucketed into one-second intervals. ```sql DROP TABLE IF EXISTS tick_interpolation; CREATE TABLE tick_interpolation(ts datetime(6), symbol varchar(5), price numeric(18,4)); INSERT INTO tick_interpolation VALUES ('2019-02-18 10:55:36.000000', 'ABC', 100.00), ('2019-02-18 10:55:37.000000', 'ABC', 102.00), ('2019-02-18 10:55:40.000000', 'ABC', 103.00), ('2019-02-18 10:55:42.000000', 'ABC', 104.00); DELIMITER // CREATE OR REPLACE PROCEDURE driver() AS DECLARE q query(ts datetime(6), symbol varchar(5), price numeric(18,4)); BEGIN q = SELECT ts, symbol, price FROM tick_interpolation ORDER BY ts; ECHO SELECT 'Input time series' AS message; ECHO SELECT * FROM q ORDER BY ts; ECHO SELECT 'Interpolated time series' AS message; CALL interpolate_ts(q); END // DELIMITER ; DELIMITER // CREATE OR REPLACE PROCEDURE interpolate_ts( q query(ts datetime(6), symbol varchar(5), price numeric(18,4))) -- Important: q must produce sorted output by ts AS DECLARE c array(record(ts datetime(6), symbol varchar(5), price numeric(18,4))); r record(ts datetime(6), symbol varchar(5), price numeric(18,4)); r_next record(ts datetime(6), symbol varchar(5), price numeric(18,4)); n int; i int; _ts datetime(6); _symbol varchar(5); _price numeric(18,4); time_diff int; delta numeric(18,4); BEGIN DROP TABLE IF EXISTS tmp; CREATE TEMPORARY TABLE tmp LIKE tick_interpolation; c = collect(q); n = length(c); IF n < 2 THEN ECHO SELECT * FROM q ORDER BY ts; return; END IF; i = 0; r = c[i]; r_next = c[i + 1]; WHILE (i < n) LOOP -- IF at last row THEN output it and exit IF i = n - 1 THEN _ts = r.ts; _symbol = r.symbol; _price = r.price; INSERT INTO tmp VALUES(_ts, _symbol, _price); i += 1; CONTINUE; END IF; time_diff = unix_timestamp(r_next.ts) - unix_timestamp(r.ts); IF time_diff <= 0 THEN RAISE user_exception("time series not sorted or has duplicate timestamps"); END IF; -- output r _ts = r.ts; _symbol = r.symbol; _price = r.price; INSERT INTO tmp VALUES(_ts, _symbol, _price); IF time_diff = 1 THEN r = r_next; -- advance to next row ELSIF time_diff > 1 THEN -- output time_diff-1 rows by extending current row and interpolating price delta = (r_next.price - r.price) / time_diff; FOR j in 1..time_diff-1 LOOP _ts += 1; _price += delta; INSERT INTO tmp VALUES(_ts, _symbol, _price); END LOOP; r = r_next; -- advance to next row ELSE RAISE user_exception("time series not sorted"); END IF; i += 1; IF i < n - 1 THEN r_next = c[i + 1]; END IF; END LOOP; ECHO SELECT * FROM tmp ORDER BY ts; DROP TABLE tmp; END // DELIMITER ; ``` The `driver()` procedure produces the following output: ```sql CALL driver(); ``` ```output +-------------------+ | message | +-------------------+ | Input time series | +-------------------+ 1 row in set (0.02 sec) +----------------------------+--------+----------+ | ts | symbol | price | +----------------------------+--------+----------+ | 2019-02-18 10:55:36.000000 | ABC | 100.0000 | | 2019-02-18 10:55:37.000000 | ABC | 102.0000 | | 2019-02-18 10:55:40.000000 | ABC | 103.0000 | | 2019-02-18 10:55:42.000000 | ABC | 104.0000 | +----------------------------+--------+----------+ 4 rows in set (0.06 sec) +--------------------------+ | message | +--------------------------+ | Interpolated time series | +--------------------------+ 1 row in set (0.16 sec) +----------------------------+--------+----------+ | ts | symbol | price | +----------------------------+--------+----------+ | 2019-02-18 10:55:36.000000 | ABC | 100.0000 | | 2019-02-18 10:55:37.000000 | ABC | 102.0000 | | 2019-02-18 10:55:38.000000 | ABC | 102.3333 | | 2019-02-18 10:55:39.000000 | ABC | 102.6666 | | 2019-02-18 10:55:40.000000 | ABC | 103.0000 | | 2019-02-18 10:55:41.000000 | ABC | 103.5000 | | 2019-02-18 10:55:42.000000 | ABC | 104.0000 | +----------------------------+--------+----------+ 7 rows in set (0.16 sec) ``` The output fills the gaps between 37 and 40 seconds and between 40 and 42 seconds with linearly interpolated values. ## Smoothing Smoothing reduces noise in volatile signals so trends are easier to see. Use `AVG` as a window function to smooth time series data. The following query returns the output value and the moving average calculated over the current row and the previous row for a specified date. ```sql SELECT tid, ts, output, AVG(output) OVER w FROM turbine_reading WHERE DATE(ts) = '2020-03-14' WINDOW w as (PARTITION BY tid ORDER BY ts ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) ORDER BY 1, 2; ``` ```output +-----+----------------------------+---------+--------------------+ | tid | ts | output | AVG(output) OVER w | +-----+----------------------------+---------+--------------------+ | 1 | 2020-03-14 13:00:33.000000 | 1000000 | 1000000 | | 1 | 2020-03-14 13:00:34.000000 | 1000000 | 1000000 | | 1 | 2020-03-14 13:00:35.000000 | 1050000 | 1025000 | | 1 | 2020-03-14 13:00:36.000000 | 1000000 | 1025000 | | 2 | 2020-03-14 13:00:33.000000 | 2000000 | 2000000 | | 2 | 2020-03-14 13:00:34.000000 | 2000000 | 2000000 | | 2 | 2020-03-14 13:00:35.000000 | 2050000 | 2025000 | | 2 | 2020-03-14 13:00:36.000000 | 2070000 | 2060000 | +-----+----------------------------+---------+--------------------+ ``` ## Finding a Row Current AS OF a Point in Time A common time series operation is to retrieve the row that is current `AS OF` a specific point in time. The following query uses `ORDER BY` and `LIMIT` to identify that row: ```sql -- find turbine reading for tid 1 that is current -- AS OF 2020-03-14 13:00:35.5 SELECT * FROM turbine_reading WHERE ts <= '2020-03-14 13:00:35.5' AND tid = 1 ORDER BY ts DESC LIMIT 1; ``` ```output +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ | tid | ts | rpm | temperature | vibration | output | wind_direction | wind_speed | +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ | 1 | 2020-03-14 13:00:35.000000 | 11 | 33 | 105 | 1050000 | 91 | 16 | +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ ``` You can use `EXPLAIN` to view the query plan for this query. The query plan uses the index on `ts` to seek directly to the target timestamp and scan matching rows in reverse order, returning the row current as of a specified timestamp in `O(log n)` time. This optimization requires a rowstore table because columnstore tables do not support index-based top-1 seeks. If your workload relies heavily on `AS OF` lookups, consider using a rowstore table instead of the default columnstore table. You can implement equivalent `AS OF` queries on columnstore tables by using `MAX(ts)` with a window function or a correlated subquery. However, these approaches typically scan more data and may have higher query latency. To find the current row for each turbine as of a specific point in time, use the stored procedure shown in the following example. ```sql DELIMITER // CREATE OR REPLACE PROCEDURE get_turbine_readings_as_of(_ts datetime(6)) AS DECLARE q_turbines QUERY(tid int) = SELECT tid FROM turbine; a ARRAY(RECORD(tid int)); _tid int; BEGIN DROP TABLE IF EXISTS r; CREATE TEMPORARY TABLE r LIKE turbine_reading; a = COLLECT(q_turbines); FOR x IN a LOOP _tid = x.tid; INSERT INTO r SELECT * FROM turbine_reading t WHERE t.tid = _tid AND ts <= _ts ORDER BY ts DESC LIMIT 1; END LOOP; ECHO SELECT * FROM r ORDER BY tid; DROP TABLE r; END // DELIMITER ; CALL get_turbine_readings_as_of('2020-03-14 13:00:35.5'); ``` ```output +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ | tid | ts | rpm | temperature | vibration | output | wind_direction | wind_speed | +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ | 1 | 2020-03-14 13:00:35.000000 | 11 | 33 | 105 | 1050000 | 91 | 16 | | 2 | 2020-03-14 13:00:35.000000 | 18.5 | 30 | 176 | 2050000 | 0 | 23.5 | +-----+----------------------------+------+-------------+-----------+---------+----------------+------------+ ``` ## Managing the Life Cycle of Time Series Data Manage the lifecycle of time series data by moving older data from a rowstore table to a columnstore table when the data grows beyond available memory. When the data is no longer needed, remove it with the `DELETE` statement. Bulk `DELETE` of large sets of rows is extremely fast in SingleStore, unlike legacy B-tree based database systems. Refer to this [blog](https://www.singlestore.com/blog/table-range-partitioning-is-a-crutch-heres-why-singlestore-doesnt-need-it/) for related information. ## Related Topics * [FIRST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/first.md) * [LAST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/last.md) * [TIME\_BUCKET](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/time-bucket.md) * [SERIES TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#section-id235648339580205.md) * Training: [Time Series Data Capture and Analysis](https://training.singlestore.com/learn/course/internal/view/elearning/637/time-series-data-capture-and-analysis) * Blog: [It’s About Time: Getting More from Your Time-Series Data With SingleStoreDB Self-Managed 7.0](https://www.singlestore.com/blog/7-0-release-time-series-functionality/). *** Modified at: July 2, 2026 Source: [/cloud/developer-resources/functional-extensions/analyzing-time-series-data/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/analyzing-time-series-data/) (An index of the documentation is available at /llms.txt) --- # Configuring Full Text and Vector Indexes Configuring SingleStore for high-performance vector and full-text search is important for achieving maximum performance from these searches. This page describes how to configure SingleStore for vector and full text search, focusing on configuration for high queries per second, followed by adjustments for other types of workloads and suggested optimizations. This document is intended for readers with an understanding of SingleStore parallel query execution. See the list of resources at the end of this document for information about the concepts in this document. ## Configure for High Throughput / High QPS For high-throughput workloads where the goal is to maximize Queries Per Second (QPS), the key configuration is to have **fewer, bigger segments, and make the segment size even**. In addition, to get high queries per second (QPS), it helps to change the flexible parallelism settings to use only one thread per leaf node per query, which will use one thread per partition when there is one segment per partition. ## Concept Vector and full-text search performance is highly related to the number of index segments scanned. Hence, to optimize performance, queries should scan the fewest index segments which translates to maximizing index segment size. Conceptually, to minimize the number of segments scanned by each query, the configuration should use one database partition on each leaf node, the minimum number of segments within that partition, and one thread on each leaf node scanning that partition. To do so, the following configurations are needed: * **Use large segments**: Allow columnstore segments to be as large as possible (by setting `internal_columnstore_max_uncompressed_blob_size` and `columnstore_segment_rows` to their maximum possible values). * **Use few partitions and eliminate subpartitions**: Use one partition per leaf node, or two partitions if using a small, single-leaf or two-leaf system and eliminate subpartitions. * **Evenly distribute data among partitions**: Choose a shard key that will evenly distribute data among partitions. * **Turn down flexible parallelism**: Set flexible parallelism so that one thread will scan each partition. This setting reduces per-query overhead and helps performance when the majority of the query execution work is for index searches. * **Enable the full-text index cache.** * `SET GLOBAL fts2_cache_enable = ON` * Consider increasing the value of `fts2_cache_blob_memory_mb` which can improve search performance at the expense of more memory consumption. * Stop the FTS service using the `FULLTEXT SERVICE STOP` command after changing these settings to ensure these changes take effect. The FTS service will restart automatically when needed again. ## Recommended Configuration **Note**: It is important to create indexes and set the configuration variables before loading data. To obtain the behavior described above and minimize the number of segments scanned, the recommended configuration settings are as follows: * Use few partitions and eliminate subpartitions: * `CREATE DATABASE PARTITIONS SUB_PARTITIONS 0` * Use large segments: * `SET GLOBAL internal_columnstore_max_uncompressed_blob_size = 10737418240` * 10,737,418,240 is the maximum possible value of  `internal_columnstore_max_uncompressed_blob_size` * **Caution**: It can take up to 3 times the amount of memory specified in this variable to create a blob. If you get OOM errors, reduce this value or scale up your RAM (say with a 2x or 4x cloud scaled instance). * `SET GLOBAL columnstore_segment_rows = 10000000` * 10,000,000 is the maximum value of `columnstore_segment_rows`. * Evenly distribute data: * Select a shard key that will evenly distribute data across partitions. * Turn down flexible parallelism: * `SET query_parallelism_per_leaf_core = 0.01` * With this configuration, the engine uses one thread per leaf node. * This is a session variable and will need to be set for each connection. ## Verification and Optimization * Run the `OPTIMIZE TABLE FLUSH` command after the index is built to flush all index segments to disk. * After the initial configuration is set, verify, using the command below, that there are no or few small segments. If there are small segments, consider a change to the shard key or the number of partitions. ```sql SELECT DATABASE_NAME, TABLE_NAME, ORDINAL AS PARTITION_ID, ROWS, MEMORY_USE FROM INFORMATION_SCHEMA.TABLE_STATISTICS WHERE TABLE_NAME = ''; ``` ## Data Distribution Notes When tuning for QPS, do not increase the partition count if doing so shrinks the segment size. However, when the data is very large, that is the number of rows is much greater than (number of partitions) \* (segment size), there will be many segments in each partition. In this case, if you run the `OPTIMIZE TABLE FULL` command, the segments will have roughly even sizes except for the tail segment. Given that there are a lot of segments, having one small segment is not very significant in percentage terms. Having no tiny stray segments is more important for small data sets with just one or a few segments per partition. Avoid compaction during data ingestion by running the `OPTIMIZE TABLE FULL` command after data has been loaded into the table. Thus, if the data size is large enough so that there is more than one maximum-sized segment per partition, you can increase the partition count without effect on QPS. Increasing the partition count in this case will not reduce the number of segments. ## Workload The configuration above is designed for workloads in which: * Multiple queries are run in parallel. * When flexible parallelism is turned down so that there is one thread per leaf node, a concern is that there are extra VCPUs which will not be used. When multiple queries are running in parallel, these extra VCPUs are used. * Queries do not have highly selective predicates. * When queries do not have highly selective predicates, segment-elimination is not effective, therefore a 'small' number of 'large' segments is a reasonable configuration. ## Additional Configurations and Workload Considerations ## Segment Size Tradeoffs The index configuration (segment size) needs to be matched to the workload. In SingleStore, vector and full-text indexes are built per (columnstore) segment. Vector and full-text indexes span all the sub-partitions in a partition. Large segments typically improve search speed because the cost of searching a large segment is only slightly more than the cost of searching a small segment (search time increases logarithmically with segment size). While large segment sizes are good for QPS and for search performance, smaller segments may be appropriate for workloads with updates or workloads with highly selective queries. Queries with highly selective `WHERE` filters may benefit from smaller segments because segment elimination is applied before vector or full-text index searches. When indexes are updated regularly, when latency is important and high parallelism is useful, or when there are highly selective queries, appropriate segment size needs to be considered. ## Segment Size Tradeoffs for Update Performance There is a tradeoff between full-text and vector index search performance and index update rate. Large segments improve index search performance while small segments improve index update rate. Thus there is a tradeoff between search speed and update speed. This tradeoff can be managed by adjusting the segment size. To increase the index update rate, consider reducing the segment size or increasing the number of partitions. ## Segment Size Tradeoffs for Highly Selective Queries Similarly, there is a tradeoff between full-text and vector index search performance and segment elimination. Large segments improve index search performance; small segments improve segment elimination. Segment elimination is the process whereby the system avoids scanning segments that do not meet the filter in the query. With highly selective filters, segment elimination will result in much less data being scanned and can significantly improve performance. In this case, smaller segments mean less data is scanned (small segments are more likely to be able to be 'eliminated' and a smaller number of rows are read from the segments that pass segment elimination). Thus, smaller segments are good for performance for queries with highly selective filters that are not for full-text search. An example of such a filter is a date range filter in the `WHERE` clause of a query. That is, queries with highly selective predicates may benefit from an increased number of partitions and segments. The increased number of smaller partitions will enable more segment elimination. Refer to [Choosing a Columnstore Key](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key.md) for information on segment elimination. ## Adjust the Number of Segments and Partitions Segment size in SingleStore is managed through the variables: * `internal_columnstore_max_uncompressed_blob_size` (max value 10,737,418,240) * `columnstore_segment_rows` (max value 10,000,000) As noted above, it's often best to set these to larger-than-default values (e.g. the maximum values) to improve vector and text search performance, especially if you don't have selective standard `WHERE` clause filters. You can also choose a smaller-than-default number of partitions in your `CREATE DATABASE` statement as recommended earlier. ## Parallel Scan and Flexible Parallelism SingleStore is a powerful parallel database which supports a spectrum of parallelism; different levels of parallelization are effective for different types of workloads. In general, to optimize for throughput or high queries per second and for workloads with many relatively small queries, flexible parallelism should be turned down. To optimize for response time, flexible parallelism should be turned up. ## Flexible Parallelism Flexible parallelism allows multiple threads to scan a single partition, increasing the parallelism (number of cores) used for a query. Flexible parallelism allows all cores to work on a particular query even if there are more cores than partitions. This decreases query latency. More specifically, flexible parallelism allows each core to scan a sub-partition, so multiple threads per query per leaf. This is great for certain workloads, but not for all workloads. If you have multiple queries running concurrently, parallelism may not help. Multiple queries can use multiple cores, so there is less need for parallelism. However, if you have fewer queries running, parallelism can help improve performance. At a high level, if you have multiple concurrent queries, turn down the parallelism, but if you have few large queries running, turn up the parallelism. Flexible parallelism is managed with the engine variable `query_parallelism_per_leaf_core`. Turning down parallelism by decreasing `query_parallelism_per_leaf_core` helps throughput by reducing overhead for starting and synchronizing threads. More importantly, turning down parallelism can also reduce total work by avoiding having multiple threads process the same segment, potentially each doing an index search independently, or doing `ORDER BY ... LIMIT` queries on more threads which could increase the total number of rows decoded and transmitted. This extra decoding and transmission can drive up the total CPU cycles needed to run a query. ## Flexible Parallelism Configuration When doing a parallel (non-indexed) scan: * If (flexible) parallelism is set to one (`query_parallelism_per_leaf_core = 1`), then there will be one thread per CPU (vCPU) core and each thread will scan different sub-partitions in parallel. * If (flexible) parallelism is set lower (`query_parallelism_per_leaf_core < 1`), then fewer threads will be used and each thread will scan multiple sub-partitions. Setting `query_parallelism_per_leaf_core = .01` will give one thread per leaf node, which is ideal for high-throughput search workloads. ## Configure for Large Data For huge data, you may want more than one thread per leaf node to balance response time versus throughput. If throughput is still your top concern, you can still limit the threads per leaf node by setting `query_parallelism_per_leaf_core = 0.01`. ## Configure for Response Time To obtain the best response times for individual queries, use multiple threads per query. Set `query_parallelism_per_leaf_core` at its default value of `1.0` or some relatively high value such as `0.5` or `0.25`. Increasing query parallelism increases parallelism at the cost of thread overhead. Thus, there is a tradeoff between increased parallelism and thread overhead which can be managed by setting query parallelism. ## Benchmarking with Fixed Size Data If your data is fixed size, for example, if you are benchmarking, follow this procedure to get data evenly distributed among partitions. * Calculate an even distribution of data among partitions and set `columnstore_segment_rows` to enforce that distribution. * `SET GLOBAL columnstore_segment_rows = ((#rows in data set)/(#partitions)) + buffer` * Suggest value of \~1000 for buffer. * OR if the dataset is larger: ((#rows in data set)/(#partitions)) > 10,000,000 * `SET GLOBAL columnstore_segment_rows = 10000000` * 10,000,000 is the maximum value of `columnstore_segment_rows` The buffer value will prevent inadvertent creation of a tiny segment containing the remainder of the data if data is not split perfectly evenly. A tiny segment can hurt throughput since it will have to be searched independently by a thread, and that could take noticeable time for index search startup. In addition, index search time is logarithmic, so more segments (and thus more indexes) are virtually always worse for throughput. ## Full-Text Considerations and Optimizations Below are a series of considerations and optimizations to consider for full-text search performance. ## Indexing Limited Full-Text Columns For full-text indexes, the number of columns that are indexed can affect search performance. Only index the columns that will be searched. ## Full-Text ORDER BY … LIMIT Optimization There is a significant optimization that pushes an `ORDER BY…LIMIT` clause down into the Java Lucene process if a query is structured in a certain way.  Depending on the size of the data, this optimization can reduce a query's time from minutes to seconds. For this optimization to work, the query must follow these rules: * The `ORDER BY` clause must use `DESC` sort order. * There must be only one full-text search function (`MATCH`, `BM25`, or `BM25_GLOBAL`) in the query. You can often "refactor" the FTS part out of a query to meet these rules and employ the optimization as shown in the following example. **Original query:** ```sql SELECT ucid, company_name, fc.company_description, MATCH (TABLE fc) AGAINST ('company_technologies:("font awesome" OR "open graph")') AS _match FROM Company c JOIN FTS_Company_new fc USING (ucid) JOIN Person p USING(ucid) ANTI_JOIN GlobalExclude_global ge ON ge.PersonId = p.upid WHERE _match GROUP BY ucid ORDER BY _match DESC LIMIT 25 OFFSET 0; ``` Rewritten query that benefits from the optimization: ```sql WITH _fts AS ( SELECT ucid, company_description, MATCH (table fc) AGAINST ('company_technologies:("font awesome" OR "open graph")') AS _match FROM FTS_Company_new fc WHERE _match ORDER BY _match desc limit 25 ) SELECT ucid, company_name, fc.company_description, fc._match FROM Company c JOIN _fts fc USING (ucid) JOIN Person p USING(ucid) ANTI_JOIN GlobalExclude_global ge ON ge.PersonId = p.upid WHERE fc._match GROUP BY ucid ORDER BY fc._match DESC LIMIT 25 OFFSET 0; ``` Factoring the full-text search away from the joins into a CTE allows the optimization to work. ## FTS V2 Architecture and Performance In FTS v2, the Lucene engine is decoupled from the main database engine and runs in a co-process.  While this architecture allows SingleStore to surface the latest and greatest FTS features available in Lucene, it limits the ability to optimize filtering when both FTS matches and other non-FTS criteria are present. This behavior is because the Lucene co-process isn't deeply integrated into the SingleStore engine. Lucene must first return all matches, to which other filters can be applied and limits that may be present in the query.  This causes overhead, particularly if there are a lot of FTS matches. In queries with filters on multiple attributes, consider adding attributes that are used in filters to the full-text index so that the full `WHERE` clause can be moved into the Lucene expression. This enables optimization (within Lucene) and eliminates the overhead of passing all matches from the Lucene process back to the SingleStore process. ## Statistics Cache in FTS Service The statistics cache in the FTS V2 service is designed to store frequently-accessed term and collection statistics used during full-text scoring computations, particularly for BM25 and BM25\_GLOBAL queries. These scoring models depend heavily on term-level and document-level statistics that are typically computed on-the-fly. When workloads involve repetitive or similar query patterns, recomputing these statistics repeatedly can become a performance bottleneck. By enabling and tuning the FTS statistics cache, the system can reuse previously computed statistics, leading to improved query throughput and lower response latency. However, as with any cache mechanism, the FTS statistics cache requires careful configuration to balance performance benefits against memory usage. Enable the FTS statistics cache with the following command: ```sql SET GLOBAL fts2_stats_cache_enable = TRUE; ``` The following engine variables are used to configure the FTS statistics cache: * `fts2_stats_cache_max_size`: Specifies  the maximum number of index blobs (segments) that can be stored in the statistics cache at any given time. Helps manage memory usage by limiting the number of cached statistics ensuring the system doesn’t overuse memory. * `fts2_stats_cache_max_terms`: Specifies the maximum number of term statistics that can be cached for each index blob (segment). Controls the granularity of cached data, balancing memory consumption against query performance for term-based scoring models like BM25. ## Throttling Full-Text Search If full-text search is taking too much CPU time, full-text search can be throttled by reducing the value of the `fts2_max_connections` engine variable. The value of this variable, that is the number of connections, is roughly equal to the number of CPUs on each leaf to use. That is, if `fts2_max_connections` is set to `6`, that means that approximately 6 CPUs on each leaf will be used. ## Warm the Cache for Full-Text Search The disk cache can be warmed for a full-text search by executing a query such as the following which will cause the full-text indexes to be downloaded into the local disk cache. ```sql SELECT COUNT(*) FROM table_name WHERE MATCH(TABLE table_name) AGAINST ('valid_column:random_term'); ``` ## Other Strategies To get the best performance, get creative with application and even data changes. For example, can the database size be reduced? You may be able to eliminate older or less important data. Other ways to improve performance may be to add more hardware, or to right-size the hardware to the application and data needs. Another is to change the queries, for example, simplify the FTS portion of the queries, or break the queries into pieces. ## Resources * [How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) * [Flexible Parallelism](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/flexible-parallelism.md) * [Choosing a Columnstore Key](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/choosing-a-columnstore-key.md) * [Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) * [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) * [Tuning Vector Indexes and Queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries.md) *** Modified at: April 1, 2026 Source: [/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes/) (An index of the documentation is available at /llms.txt) --- # Full Text VERSION 2 Custom Analyzers SingleStore supports custom analyzers for full-text VERSION 2 search. Users can customize full-text search by: * Using built-in analyzers for a variety of languages. The built-in analyzers can be customized with custom stop-word lists. * Using custom analyzers in which a user can specify a tokenizer, optional token and character filters, and an optional stop-word list. In general, an analyzer contains three components: a tokenizer, character filters, and token filters. An analyzer must have exactly one tokenizer; it can have zero or more character filters and zero or more token filters. A tokenizer takes a stream of characters and breaks that stream into individual tokens, for example split on whitespace characters. A character filter takes the stream of text data and transforms it in a pre-defined way, for example, removing all HTML tags. Token filters receive a stream of tokens and may add, change, or remove tokens, for example, lowercase all tokens or remove stop words.  Users can choose from a list of pre-configured analyzers and use them without any modifications. Users can also create their own analyzers by specifying a tokenizer, character filters, and token filters to obtain a fully customized search experience.  Refer to [Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) for more information on full-text search. ## Specify an Analyzer Specify an analyzer by passing an analyzer configuration in JSON format to `INDEX_OPTIONS`, which is a JSON string that contains the index configuration. In this JSON, the analyzer key is a string or a nested JSON value. * Specify the name of a built-in analyzer (e.g.: `standard`, `cjk`, etc.) as a string. * Specify a customized built-in analyzer or a custom analyzer as a nested JSON value. The three examples below show a built-in analyzer with no customizations, a built-in analyzer with a customized set of stop words, and a custom analyzer. A full set of examples can be found in [Examples](https://docs.singlestore.com/#UUID-cd288605-d189-0467-47a1-357217158b48.md). Refer to [Analyzers](https://docs.singlestore.com/#section-idm234616810166095.md) for details on analyzers. Specify the built-in analyzer for Chinese, Japanese, and Korean characters, called the `cjk` analyzer, with no customizations. ```sql CREATE TABLE t ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{ "analyzer": "cjk"}' ); ``` Specify the built-in `cjk` analyzer with a customized set of stop words. Built-in analyzers can be customized with custom stop word lists; no other customizations for built-in analyzers are supported. ```sql CREATE TABLE t ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{"analyzer": { "cjk": { "stopset": [ "这", "那" ] } } }' ); ``` > **📝 Note**: In addition to the `cjk` analyzer, the Korean `nori` analyzer is also supported. Specify a custom analyzer, which uses the `whitespace` tokenizer, the `html_strip` character filter, and the `lower_case` token filter. The analyzer name must be `custom`. Additional character and token filters can be specified by adding additional `char_filters` and `token_filters` key pairs. ```sql CREATE TABLE t ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": "whitespace", "char_filters": ["html_strip"], "token_filters": ["lower_case"], } } }' ); ``` ## Analyzers There are two types of analyzers: built-in analyzers and custom analyzers. > **📝 Note**: The examples in this section show only the `INDEX_OPTIONS` string (JSON) and omit the rest of the index creation command. ## Built-in Analyzers Built-in analyzers are pre-configured analyzers including the standard analyzer and language-specific analyzers and do not require configuration. Built-in analyzers may be customized with custom stop-word lists. The default analyzer is the [Apache Lucene standard analyzer](https://lucene.apache.org/core/8_8_1/core/org/apache/lucene/analysis/standard/StandardAnalyzer.html), which uses the Apache Lucene standard tokenizer, lowercase token filters, and no stop words. Specify a built-in analyzer, without customizations, by specifying the name of the analyzer as the value of the `analyzer` key. The following example specifies the use of the `spanish` language analyzer. ```sql INDEX_OPTIONS '{"analyzer" : "spanish"}' ``` A custom stop word list can be specified for a built-in analyzer by specifying a `stopset` in the JSON as shown in the following example. A custom stop word list is the only customization supported for built-in analyzers. The following example specifies a custom stop word list for the standard analyzer. The value of the `analyzer` key is a nested JSON value consisting of a key-value pair with key being the name of the analyzer (`spanish` in this example), and the value being another key-value pair consisting of the key `stopset`, and the value a JSON array of stop words. ```sql INDEX_OPTIONS '{ "analyzer": { "spanish": { "stopset": [ "el", "la" ] } } }' ``` SingleStore recommends using the default language analyzer, without stop word customization, in most cases, e.g. `'{"analyzer" : "catalan"}'`. Refer to [Supported Language Analyzers](https://docs.singlestore.com/#section-idm234616824800932.md) for links to the default list of stop words for each analyzer. ## Custom Analyzers Create a custom analyzer by using the analyzer name `custom` and by specifying a tokenizer and optional token and character filters. A custom analyzer must specify: * A *required* `tokenizer` - A tokenizer breaks up incoming text into tokens. In many cases, an analyzer will use a tokenizer as the first step in the analysis process. However, to modify text prior to tokenization, use `char_filters` (see below). * An *optional array* of `token_filters`: A `token_filter` modifies tokens that have been created by the tokenizer. Common modifications performed by a `token_filter` are deletion, stemming, and case folding. * An *optional array* of `char_filters`: A `char_filter` transforms the text before it is tokenized, while providing corrected character offsets to account for these modifications. The example below shows the use of all three components, `tokenizer`, `char_filters`, and `token_filters`. ```sql INDEX_OPTIONS '{ "analyzer" : { "custom": { "tokenizer": "whitespace", "char_filters": ["html_strip"], "token_filters": ["lower_case"], } } }' ``` Each of these three components (`tokenizer`, `char_filters`, `token_filters`) can be specified as a string with the name of the component or as a nested JSON with a configuration for the component. The example below specifies a custom analyzer that uses the `whitespace` tokenizer, with a maximum length of 256 characters. ```sql INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": { "whitespace": { "maxTokenLen": 256 } } } } }' ``` ## Common Tokenizers Common tokenizers that are supported are listed in the table below. Refer to [Supported Tokenizers](https://docs.singlestore.com/#section-idm234616825064604.md) for a full list of supported tokenizers. | **"tokenizer" (case-sensitive)** | **Parameters** | **Description** | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `whitespace` | `rule`(Optional, string). Defaults to`"unicode"`.`maxTokenLen`(Optional, integer). Defaults to`256`.[WhitespaceTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/WhitespaceTokenizerFactory.html) | Divides text at whitespace characters as defined by[Character.isWhitespace(int)](https://docs.oracle.com/javase/8/docs/api/java/lang/Character.html?is-external=true#isWhitespace-int-). This definition excludes non-breaking spaces from whitespace characters.[WhitespaceTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/WhitespaceTokenizer.html) | | `standard` | `maxTokenLength`(Optional, integer). Defaults to 255.[StandardTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/StandardTokenizerFactory.html) | Implements Word Break rules from Unicode Text Segmentation:[Unicode Standard Annex #29](http://unicode.org/reports/tr29/).[StandardTokenizer (Lucene 6.6.0 API)](https://lucene.apache.org/core/8_3_0/core/org/apache/lucene/analysis/standard/StandardTokenizer.html?is-external=true) | | `n_gram` | `minGramSize`(Optional, integer). Defaults to`1`.`maxGramSize`(Optional, integer). Defaults to`2`.[NGramTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizerFactory.html) | Tokenizes the input into n-grams of the specified size(s).[NGramTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizer.html) | | `uax_url_email` | `maxTokenLength`(Optional, integer). Defaults to`255`.[UAX29URLEmailTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerFactory.html) | Implements Word Break rules from Unicode Text Segmentation:[Unicode Standard Annex #29](http://unicode.org/reports/tr29/). URLs and email addresses are also tokenized.[UAX29URLEmailTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizer.html) | ## Common Token Filters Common token filters that are supported are listed in the table below. Refer to [Supported Token Filters](https://docs.singlestore.com/#section-idm234616825771108.md) for a full list of supported token filters. | **"token\_filters" (Case-Sensitive)** | **Parameters** | **Description** | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `shingle` | `minShingleSize`(Optional, integer). Defaults to`2`.`maxShingleSize`(Optional, integer). Defaults to`2`.[ShingleFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilterFactory.html) | Constructs shingles (token n-grams), that is it creates combinations of tokens as a single token.[ShingleFilter](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilter.html) | | `lower_case` | No parameters.[LowerCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/core/LowerCaseFilterFactory.html) | Normalizes token text to lower case.[LowerCaseFilter](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/core/LowerCaseFilter.html) | | `snowball_porter` | `protected`(Optional, string). Defaults to`"protectedkeyword.txt"`.`language`(Optional, string). Defaults to`"English"`.[SnowballPorterFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/snowball/SnowballPorterFilterFactory.html) | Stems words using a Snowball-generated stemmer. Available stemmers are listed in`org.tartarus.snowball.ext`.[SnowballFilter](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/snowball/SnowballFilter.html) | | `n_gram` | `minGramSize`(Optional, integer). Defaults to`1`.`maxGramSize`(Optional, integer). Defaults to`2`.`preserveOriginal`(Optional, boolean). Defaults to`"true"`.[NGramFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramFilterFactory.html) | Tokenizes the input into n-grams of the given size(s).[NGramTokenFilter](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenFilter.html) | | `stop` | `words`(Optional, array of stop words)`ignoreCase`(Optional, boolean). If true, all words are lower-cased first. Defaults to`false`. | Custom stop words token filter.Removes stop words from a token stream. | ## Custom Stop Words SingleStore provides a custom token filter named `stop` which allows a set of custom stop words to be specified. The stop token filter works with any custom analyzer. The stop token filter has two parameters `words` and `ignoreCase`. * `words`: An optional parameter containing a list of stop words. The list of stop words must be specified as a JSON array. * `ignoreCase`: An optional boolean parameter indicating if case should be ignored. If set to `true`, all words are lower-cased before tokenization. Defaults to `false`. Sample syntax for this custom token filter is as follows. Refer to [Example 11: Standard Tokenizer with Custom Stop Words](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234955757677161.md) for additional examples of using the `stop` token filter. ```sql FULLTEXT USING VERSION 2 KEY (text) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", token_filters: [{"stop": {"ignoreCase": false, "words": ["the"]}} ]} }}'); ``` ## Common Character Filters Common character filters that are supported are listed in the table below. Refer to [Supported Character Filters](https://docs.singlestore.com/#section-idm234616826086158.md) for a full list of supported character filters. | **"char\_filters" (case-sensitive)** | **Parameters (includes Lucene Link)** | **Description (includes Lucene Link)** | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `html_strip` | `escapedTags`(Optional, string). Defaults to`"a, title"`.[HTMLStripCharFilterFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/charfilter/HTMLStripCharFilterFactory.html) | Wraps another Reader and attempts to strip out HTML.[HTMLStripCharFilter](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/charfilter/HTMLStripCharFilter.html) | ## Custom Column Mappings Custom column mappings allow columns in a table and fields in a JSON document to be indexed using custom analyzers. This functionality allows you to index columns or fields in different languages with analyzers specific to each language. For the most meaningful search results, it is important to use an analyzer specific to a language. Language-specific analyzers allow the use of language-aware stop lists, stemming, and word breaking. Applying a generic analyzer to text from multiple languages may result in lower-quality search results. ## Use Per-Column Analyzers Per-column analyzers are defined using `INDEX_OPTIONS` in the table or index creation command. You can specify a custom analyzer for each column in a table and for keypaths in JSON and BSON columns. The following command creates a table with a full-text index that indexes the `french_content` column with the `french` analyzer, the `english_content` column with the `english` analyzer, and all other columns (`title`) with the `standard` analyzer. ```sql CREATE TABLE custom_col_analyzers ( title VARCHAR(200), french_content VARCHAR(200), english_content VARCHAR(200), FULLTEXT USING VERSION 2 (title, french_content, english_content) INDEX_OPTIONS '{ "analyzer": "standard", "mappings": { "french_content": { "analyzer": "french" }, "english_content": { "analyzer": "english" } } }' ); ``` ## JSON and BSON Keypath Analyzers Analyzers for keypaths in JSON and BSON columns are defined using `INDEX_OPTIONS` in the table or index creation command. The following command specifies that the `json_column$english_content` field be indexed using the `english` analyzer, and the `json_column$french_content` field be indexed using the `french` analyzer. The mappings object, highlighted in bold, provides this specification. ```sql CREATE TABLE json_keypath_analyzers ( json_column JSON, FULLTEXT USING VERSION 2 KEY(json_column) INDEX_OPTIONS '{ "mappings": { "json_column$english_content": { "analyzer": "english" }, "json_column$french_content": { "analyzer": "french" } } }' ); ``` ## Example of Using Parameters Specify a default `uax_url_email` tokenizer: ```sql INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": "uax_url_email" } } }' ``` Specify a `uax_url_email` tokenizer with custom parameters: ```sql INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": { "uax_url_email" : { "maxTokenLength": 300 } } } } }' ``` ## Stemming Stemming transforms words to their root form, often by removing suffixes and prefixes. In English, for example, the words "dressing" and "dressed", can be stemmed to "dress". This allows a search for one form of a verb (e.g. "dressing") to return documents containing other forms of the verb (e.g. "dressed" or "dress"). Stemming is language specific. In SingleStore, stemming can be handled in two ways: * Use a [built-in analyzer](https://docs.singlestore.com/#section-idm234616818255209.md) available from JLucene that incorporates stemming. * Many of the language-specific analyzers from JLucene do stem; however, the `standard` analyzer from JLucene does not stem. * Use a [custom analyzer](https://docs.singlestore.com/#section-idm234616823004247.md) with a token filter such as `elision` or `snowball_porter` to customize stemming. The following `CREATE TABLE` statement creates a full-text index using the `spanish` analyzer, which stems for the Spanish language. ```sql CREATE TABLE spanish_lang ( text VARCHAR(200) FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{ "analyzer": "spanish"}' ); ``` The following `CREATE TABLE` statement uses a custom analyzer with custom token filters to stem for Italian text. Refer to [Example 6](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234616974238486.md) for the full example. ```sql CREATE TABLE italian_architecture ( architecture VARCHAR(400), description VARCHAR(400), SORT KEY (architecture), FULLTEXT USING VERSION 2 KEY(description) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", "token_filters": ["elision", {"snowball_porter" : {"language": "Italian"}}]}}}}' ); ``` ## NGrams NGram tokenizers split words into small pieces and are good for fast "fuzzy-style" matching using a full-text index. The minimum and maximum gram length is customizable. Refer to [Example 7](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234616974310271.md), [Example 8](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234616976167337.md), and [Example 12](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm235055586170482.md) for examples of using a ngram tokenizer. ## Examples ## Example 1: Custom Analyzer with Whitespace tokenizer Use a `custom` analyzer and a `whitespace` tokenizer to search for text with a hyphen in queries. Create a table, insert data, and optimize the table to ensure all data is included in results. ```sql CREATE TABLE medium_articles ( title VARCHAR(200), summary TEXT, FULLTEXT USING VERSION 2 (summary) INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": "whitespace" } } }' ); INSERT INTO medium_articles (title, summary) VALUES ('Build Real-Time Multimodal RAG Applications Using SingleStore!','This guide teaches you how to build a multimodal Retrieval-Augmented Generation (RAG) application using SingleStore, integrating various data types for enhanced AI responses.'), ('Building Production-Ready AI Agents with LangGraph: A Real-Life Use Case','This guide offers a solution for creating a scalable, production-ready multi-modal chatbot using LangChain, focusing on dividing tasks for improved control and efficiency.'), ('Scaling RAG from POC to Production','This guide explains Retrieval-Augmented Generation (RAG) for building reliable, context-aware applications using large language models (LLMs) and emphasizes the importance of scaling from proof of concept to production.'), ('Tech Stack For Production-Ready LLM Applications In 2024','This guide reviews preferred tools for the entire LLM app development lifecycle, emphasizing simplicity and ease of use in building scalable AI applications.'), ('LangGraph + Gemini Pro + Custom Tool + Streamlit = Multi-Agent Application Development','This guide teaches you to create a chatbot using LangGraph and Streamlit, leveraging LangChain for building stateful multi-actor applications that respond to user support requests.'); OPTIMIZE TABLE medium_articles FLUSH; ``` Observe the difference between the results of the two search queries below. ```sql SELECT * FROM medium_articles WHERE MATCH(TABLE medium_articles) AGAINST ("summary:multimodal"); ``` ```output +----------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | title | summary | +----------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Build Real-Time Multimodal RAG Applications Using SingleStore! | This guide teaches you how to build a multimodal Retrieval-Augmented Generation (RAG) application using SingleStore, integrating various data types for enhanced AI responses. | +----------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ```sql SELECT * FROM medium_articles WHERE MATCH(TABLE medium_articles) AGAINST ("summary:multi-modal"); ``` ```output +--------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | title | summary | +--------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Building Production-Ready AI Agents with LangGraph: A Real-Life Use Case | This guide offers a solution for creating a scalable, production-ready multi-modal chatbot using LangChain, focusing on dividing tasks for improved control and efficiency. | +--------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Example 2: Custom Tokenizer, Character Filter, and Token Filter to Search for HTML Entities Use a `custom` analyzer, a `whitespace` tokenizer, `html_strip` as a character filter, and `lower_case` as a token filter to search for HTML entities in queries. A character filter receives the original text data and converts it into a predefined format. A token filter receives a stream of tokens and can add, change, or remove tokens as needed. In this example, `html_strip` as a character filter removes HTML tags and `lower_case` as a token filter lowercases the tokens. Create a table, insert data, and optimize the table to ensure all data is included in results. Search for HTML entities in queries. ```sql CREATE TABLE html_table ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (title, content) INDEX_OPTIONS '{ "analyzer": { "custom": {"char_filters": ["html_strip"], "tokenizer": "whitespace", "token_filters":["lower_case"] } } }' ); INSERT INTO html_table (title, content) VALUES ('Exciting News', 'We're thrilled to announce our new project!

'), ('Learning Journey', 'Learning is a never-ending journey & I'm excited!

'), ('Success Story', 'Our team has achieved great things & we're proud!

'), ('Grateful Heart', 'Thank you for being a part of our journey & supporting us!

'), ('Future Goals', 'We're looking forward to achieving even more!

'); OPTIMIZE TABLE html_table FLUSH; ``` Search for HTML entity, and observe the result of the search query. ```sql SELECT * FROM html_table WHERE MATCH(TABLE html_table) AGAINST("content:we're"); ``` ```output +---------------+----------------------------------------------------------------+ | title | content | +---------------+----------------------------------------------------------------+ | Success Story | Our team has achieved great things & we're proud!

| | Exciting News | We're thrilled to announce our new project!

| | Future Goals | We're looking forward to achieving even more!

| +---------------+----------------------------------------------------------------+ ``` ## Example 3: Custom Analyzer, standard Tokenizer, and Custom Token Filter (`cjk_width`) to Search Japanese Text Use a `custom` analyzer, a `standard` tokenizer, and `cjk_width` as a token filter to search for a Japanese text in queries. In this example, `cjk_width` as a token filter normalizes the width differences in CJK (Chinese, Japanese, and Korean) characters. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE japanese_novels ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (title, content) INDEX_OPTIONS '{ "analyzer": { "custom": {"tokenizer": "standard", "token_filters":["cjk_width"] } } }' ); INSERT INTO japanese_novels (title, content) VALUES ('ノルウェイの森', '村上春樹の代表作で、愛と喪失をテーマにしています。'), ('吾輩は猫である', '夏目漱石による作品で、猫の視点から人間社会を描いています。'), ('雪国', '川端康成の作品で、美しい雪景色と切ない恋を描いています。'), ('千と千尋の神隠し', '宮崎駿の作品で、少女が異世界で成長する物語です。'), ('コンビニ人間', '村田沙耶香の作品で、現代社会の孤独と適応を描いています。'); OPTIMIZE TABLE japanese_novels FLUSH; ``` Observe the result of the search query for the Japanese text below. ```sql SELECT * FROM japanese_novels WHERE MATCH(TABLE japanese_novels) AGAINST("content: 夏"); ``` ```output +-----------------------+-----------------------------------------------------------------------------------------+ | title | content | +-----------------------+-----------------------------------------------------------------------------------------+ | 吾輩は猫である | 夏目漱石による作品で、猫の視点から人間社会を描いています。 | +-----------------------+-----------------------------------------------------------------------------------------+ ``` ## Example 4: Korean (nori) Analyzer Use the `korean` analyzer to search for Korean text in queries. This analyzer is also known as the nori analyzer. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE k_drama ( genre VARCHAR(200), movie_name TEXT, cast TEXT, FULLTEXT USING VERSION 2 (genre) INDEX_OPTIONS '{ "analyzer": "korean" }' ); INSERT INTO k_drama (genre, movie_name, cast) VALUES ('로맨스', '사랑의 불시착', '현빈, 손예진'), ('액션, 스릴러', '빈센조', '송중기, 전여빈'), ('드라마, 로맨스', '도깨비', '공유, 김고은'), ('사극, 드라마', '미스터 션샤인', '이병헌, 김태리'), ('코미디, 로맨스', '김비서가 왜 그럴까', '박서준, 박민영'); OPTIMIZE TABLE k_drama FLUSH; ``` Observe the result of the search query for the Korean text below. ```sql SELECT * FROM k_drama WHERE MATCH(TABLE k_drama) AGAINST("genre:로맨스"); ``` ```output +----------------------+----------------------------+----------------------+ | genre | movie_name | cast | +----------------------+----------------------------+----------------------+ | 드라마, 로맨스 | 도깨비 | 공유, 김고은 | | 코미디, 로맨스 | 김비서가 왜 그럴까 | 박서준, 박민영 | | 로맨스 | 사랑의 불시착 | 현빈, 손예진 | +----------------------+----------------------------+----------------------+ ``` ## Example 5: Korean (nori) Analyzer with User Dictionary Use the `korean` analyzer (also known as the nori analyzer) with and without a user dictionary to search for a Korean text in queries. This example demonstrates how a user dictionary can be used to add words, specifically compound words, to the dictionary used by the `korean` analyzer. Note that the user dictionary is augmentative, meaning that it adds the specified words to the existing dictionary, it does not replace the existing dictionary. Create two tables with a full text index with the `korean` analyzer, one with and one without a user dictionary. The Korean compound word 수영장, which translates to "swimming pool" in English is inserted in the user dictionary. ```sql CREATE TABLE korean_user_dict (id INT, phrase VARCHAR(400), FULLTEXT USING VERSION 2 (phrase) INDEX_OPTIONS '{"analyzer" : {"custom": {"tokenizer": {"korean": {"userDictionary": ["수영장"]} } } } }'); ``` ```sql CREATE TABLE korean (id INT, phrase varchar(400), FULLTEXT USING VERSION 2 (phrase) INDEX_OPTIONS '{ "analyzer": "korean" }' ); ``` Insert data into the tables and optimize them to ensure all data is included in results. ```sql INSERT INTO korean_user_dict VALUES (1, "수영장"), (2, "수영"), (3, "장"); OPTIMIZE TABLE korean_user_dict FLUSH; ``` ```sql INSERT INTO korean VALUES (1, "수영장"), (2, "수영"), (3, "장"); OPTIMIZE TABLE korean FLUSH; ``` When 수영장 is inserted into the `korean_user_dict` table, because 수영장 is in the user dictionary, 수영장 is tokenized as a single, atomic token and will not be further tokenized. In contrast, when 수영장 is inserted into the `korean` table, 수영장 is tokenized into 수영 and 장. By default, compound words are decomposed, and the original form is discarded (`decompoundMode` is `discard` by default). ## Example 5a - Search for 수영장 with and without User Dictionary The following queries search for the compound word 수영장 in both the `korean_user_dict` and `korean` tables. This and the following example demonstrate searching with and without a user dictionary. The `ORDER BY` clause is included to ensure consistent ordering of results. ```sql SELECT id, phrase, BM25(korean_user_dict, 'phrase:(수영장)') AS score FROM korean_user_dict ORDER BY id; ``` ```output +------+-----------------+--------------------+ | id | text | score | +------+-----------------+--------------------+ | 1 | 수영장 | 0.4458314776420593 | | 2 | 수영 | 0 | | 3 | 장 | 0 | +------+-----------------+--------------------+ ``` ```sql SELECT id, phrase, BM25(korean, 'phrase:(수영장)') AS SCORE FROM korean ORDER BY id; ``` ```output +------+-----------------+---------------------+ | id | text | score | +------+-----------------+---------------------+ | 1 | 수영장 | 0.35471969842910767 | | 2 | 수영 | 0.23797652125358582 | | 3 | 장 | 0.23797652125358582 | +------+-----------------+---------------------+ ``` When searching the `korean_user_dict` table, 수영장 matches only 수영장 and not 수영 or 장 because 수영장 is tokenized as a single, atomic token. In contrast, when searching the `korean` table, 수영장 matches 수영장, 수영, and 장 because 수영장 is tokenized as two tokens: 수영, and 장, hence 수영장 partially matches all three rows in the table. Finally, the score for 수영장 is higher when searching the table `korean_user_dict` than when searching the table `korean` because 수영장 matches only a single row in `korean_user_dict`. ## Example 5b - Search for 수영 with and without User Dictionary The following queries search for the word 수영 in the tables with and without user dictionary. The `ORDER BY` clause is included to ensure consistent ordering of results. ```sql SELECT id, phrase, BM25(korean_user_dict, 'phrase:(수영)') AS SCORE FROM korean_user_dict ORDER BY id; ``` ```output +------+-----------------+--------------------+ | id | text | SCORE | +------+-----------------+--------------------+ | 1 | 수영장 | 0 | | 2 | 수영 | 0.4458314776420593 | | 3 | 장 | 0 | +------+-----------------+--------------------+ ``` ```sql SELECT id, phrase, BM25(korean, 'phrase:(수영)') AS SCORE FROM korean ORDER BY id; ``` ```output +------+-----------------+---------------------+ | id | text | SCORE | +------+-----------------+---------------------+ | 1 | 수영장 | 0.17735984921455383 | | 2 | 수영 | 0.23797652125358582 | | 3 | 장 | 0 | +------+-----------------+---------------------+ ``` When searching the `korean_user_dict` table, 수영 matches only itself, because, as described earlier, 수영장 has been inserted in the user dictionary and 수영장 is tokenized as a single token, so it does not match 수영. In contrast, when searching the `korean` table, 수영 matches 수영장 and 수영 because 수영장 is tokenized as two tokens: 수영, and 장. In addition, the score for 수영 is higher when searching the table `korean_user_dict` than when searching the table `korean` because 수영 matches only a single row in `korean_user_dict`. ## Example 6: Custom Analyzer, standard Tokenizer, Custom Token Filters (`elision`, `snowball_porter`) to Search Italian Text Use a `custom` analyzer, a `standard` tokenizer, `elision` and `snowball_porter` as token filters for the `Italian` language to search for an Italian text. In this example, `elision` as a token filter removes specific elisions from the input token. Using `snowball_porter` as a token filter stems the words using the Lucene Snowball stemmer tokenization. The `snowball_porter` token filter requires a language parameter to control the stemmer. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE italian_architecture ( architecture VARCHAR(400), description VARCHAR(400), SORT KEY (architecture), FULLTEXT USING VERSION 2 KEY(description) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", "token_filters": ["elision", {"snowball_porter" : {"language": "Italian"}}]}}}}' ); INSERT INTO italian_architecture (architecture, description) VALUES ('Colosseo', 'Un antico anfiteatro situato a Roma, noto per i combattimenti dei gladiatori.'), ('Torre Pendente di Pisa', 'Un campanile famoso per la sua inclinazione non intenzionale.'), ('Basilica di San Pietro', 'Una chiesa rinascimentale in Vaticano, famosa per la sua cupola progettata da Michelangelo.'), ('Duomo di Milano', 'L’architettura di Milano, nota per la sua straordinaria architettura gotica e le guglie.'), ('Palazzo Ducale', 'Il Palazzo dei Dogi a Venezia, che mostra l’architettura gotica e una ricca storia.'); OPTIMIZE TABLE italian_architecture FLUSH; ``` Observe the result of the search query for the Italian text below. ```sql SELECT * FROM italian_architecture WHERE MATCH(TABLE italian_architecture) AGAINST("description:l’architettura"); ``` ```output +-----------------+--------------------------------------------------------------------------------------------+ | architecture | description | +-----------------+--------------------------------------------------------------------------------------------+ | Duomo di Milano | L’architettura di Milano, nota per la sua straordinaria architettura gotica e le guglie. | | Palazzo Ducale | Il Palazzo dei Dogi a Venezia, che mostra l’architettura gotica e una ricca storia. | +-----------------+--------------------------------------------------------------------------------------------+ ``` Use a `custom` analyzer, a `standard` tokenizer, `snowball_porter` as token filters for the `Italian` language without elision token filter to search for an Italian text in queries. Create a second table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE italian_architecture_2 ( architecture VARCHAR(400), description VARCHAR(400), SORT KEY (architecture), FULLTEXT USING VERSION 2 KEY(description) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", "token_filters": {"snowball_porter" : {"language": "Italian"}}}}}}' ); INSERT INTO italian_architecture_2 (architecture, description) VALUES ('Colosseo', 'Un antico anfiteatro situato a Roma, noto per i combattimenti dei gladiatori.'), ('Torre Pendente di Pisa', 'Un campanile famoso per la sua inclinazione non intenzionale.'), ('Basilica di San Pietro', 'Una chiesa rinascimentale in Vaticano, famosa per la sua cupola progettata da Michelangelo.'), ('Duomo di Milano', 'L’architettura di Milano, nota per la sua straordinaria architettura gotica e le guglie.'), ('Palazzo Ducale', 'Il Palazzo dei Dogi a Venezia, che mostra l’architettura gotica e una ricca storia.'); OPTIMIZE TABLE italian_architecture_2 FLUSH; ``` Observe the result of the search query for the Italian text without elision token filter below. ```sql SELECT * FROM italian_architecture_2 WHERE MATCH(TABLE italian_architecture_2) AGAINST("description:l’architettura"); ``` ```output +----------------+---------------------------------------------------------------------------------------+ | architecture | description | +----------------+---------------------------------------------------------------------------------------+ | Palazzo Ducale | Il Palazzo dei Dogi a Venezia, che mostra l’architettura gotica e una ricca storia. | +----------------+---------------------------------------------------------------------------------------+ ``` ## Example 7: Custom Analyzer and N Gram Tokenizer Use a `custom` analyzer and a `n_gram` tokenizer to search for misspelled text in queries. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE university (name VARCHAR(400), admission_page VARCHAR(400), SORT KEY (name), FULLTEXT USING VERSION 2 KEY(admission_page) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "n_gram"}}}}'); INSERT INTO university (name, admission_page) VALUES ('Harvard University', 'college.harvard.edu/admissions'), ('Stanford University', 'stanford.edu/admission/'), ('Massachusetts Institute of Technology (MIT)', 'mitadmissions.org/'), ('California Institute of Technology (Caltech)', 'admissions.caltech.edu/'), ('University of Chicago', 'uchicago.edu/en/admissions'); OPTIMIZE TABLE university FLUSH; ``` Observe the result of the search query for the misspelled text and compare the search result with the score below. ```sql SELECT name,admission_page, MATCH(TABLE university) AGAINST("admission_page:cattec") AS score FROM university WHERE score ORDER BY score DESC; ``` ```output +----------------------------------------------+--------------------------------+---------------------+ | name | admission_page | score | +----------------------------------------------+--------------------------------+---------------------+ | California Institute of Technology (Caltech) | admissions.caltech.edu/ | 2.4422175884246826 | | University of Chicago | uchicago.edu/en/admissions | 0.8550153970718384 | | Harvard University | college.harvard.edu/admissions | 0.6825864911079407 | | Stanford University | stanford.edu/admission/ | 0.5768249034881592 | | Massachusetts Institute of Technology (MIT) | mitadmissions.org/ | 0.26201900839805603 | +----------------------------------------------+--------------------------------+---------------------+ ``` ## Example 8: Custom Analyzer, n\_gram Tokenizer, Custom Character Filter (`html_strip`), and Custom Token Filter (`lower_case`) to Search for HTML entities Use a `custom` analyzer, `n_gram` tokenizer, `html_strip` as character filter, and `lower_case` as token filter to search for HTML entities in queries. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE html_table_n_gram ( title VARCHAR(200), content VARCHAR(200), FULLTEXT USING VERSION 2 (title, content) INDEX_OPTIONS '{ "analyzer": { "custom": {"char_filters": ["html_strip"], "tokenizer": "n_gram", "token_filters":["lower_case"] } } }' ); INSERT INTO html_table_n_gram (title, content) VALUES ('Exciting News', 'We're thrilled to announce our new project!

'), ('Learning Journey', 'Learning is a never-ending journey & I'm excited!

'), ('Success Story', 'Our team has achieved great things & we're proud!

'), ('Grateful Heart', 'Thank you for being a part of our journey & supporting us!

'), ('Future Goals', 'We're looking forward to achieving even more!

'); OPTIMIZE TABLE html_table_n_gram FLUSH; ``` Observe the result of the search query for the misspelled HTML entity and compare the search result with the score below. ```sql SELECT title,content, MATCH(TABLE html_table_n_gram) AGAINST("content:I',") AS score FROM html_table_n_gram WHERE score ORDER BY score DESC; ``` ```output +------------------+--------------------------------------------------------------------+---------------------+ | title | content | score | +------------------+--------------------------------------------------------------------+---------------------+ | Learning Journey | Learning is a never-ending journey & I'm excited!

| 0.5430432558059692 | | Success Story | Our team has achieved great things & we're proud!

| 0.31375283002853394 | | Exciting News | We're thrilled to announce our new project!

| 0.26527124643325806 | | Future Goals | We're looking forward to achieving even more!

| 0.2177681028842926 | | Grateful Heart | Thank you for being a part of our journey & supporting us!

| 0.1819886565208435 | +------------------+--------------------------------------------------------------------+---------------------+ ``` ## Example 9: Portuguese Analyzer with score Use a `portuguese` analyzer to search for a Portuguese text in queries. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE portuguese_news ( headline VARCHAR(200), content TEXT, FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{ "analyzer": "portuguese" }' ); INSERT INTO portuguese_news (headline, content) VALUES ('Cenário Econômico Brasileiro', 'O Brasil enfrenta desafios econômicos com a inflação em alta e a taxa de desemprego ainda elevada.'), ('Mercado de Ações em Alta', 'As ações brasileiras registraram ganhos significativos, impulsionadas por resultados financeiros positivos de grandes empresas.'), ('Nova Política Monetária do Banco Central', 'O Banco Central do Brasil anunciou mudanças na política monetária para conter a inflação e estimular o crescimento econômico.'), ('Investimentos Estrangeiros no Brasil', 'O país atraiu um aumento de investimentos estrangeiros diretos, especialmente em setores de tecnologia e energia renovável.'), ('Tendências do Mercado Imobiliário', 'O mercado imobiliário brasileiro mostra sinais de recuperação, com aumento nas vendas de imóveis e novos lançamentos.'); OPTIMIZE TABLE portuguese_news FLUSH; ``` Observe the result of the search query for the Portuguese text and compare the search result with the score below. ```sql SELECT content, MATCH(TABLE portuguese_news) AGAINST ("content:Brasil") AS score FROM portuguese_news WHERE score ORDER BY score DESC; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | content | score | +-------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | O Brasil enfrenta desafios econômicos com a inflação em alta e a taxa de desemprego ainda elevada. | 0.22189012169837952 | | O Banco Central do Brasil anunciou mudanças na política monetária para conter a inflação e estimular o crescimento econômico. | 0.2059776782989502 | +-------------------------------------------------------------------------------------------------------------------------------------+---------------------+ ``` ## Example 10: Spanish Analyzer with custom stop words Use a `spanish` analyzer with custom stop words to search for a Spanish text in queries. Create a table, insert data and optimize the table to ensure all data is included in results. ```sql CREATE TABLE spanish_news ( headline VARCHAR(200), content TEXT, FULLTEXT USING VERSION 2 (content) INDEX_OPTIONS '{ "analyzer": {"spanish": {"stopset": ["descubrimiento", "tratamiento", "nuevo"]}} }' ); INSERT INTO spanish_news (headline, content) VALUES ('Descubrimiento de un nuevo tratamiento para la diabetes', 'Investigadores han desarrollado un tratamiento innovador que mejora el control del azúcar en sangre en pacientes diabéticos.'), ('Avances en la detección temprana del cáncer', 'Un nuevo método permite detectar el cáncer en etapas más tempranas, aumentando las posibilidades de tratamiento exitoso.'), ('Nuevo enfoque para tratar enfermedades cardíacas', 'Se ha introducido un nuevo enfoque terapéutico que reduce significativamente el riesgo de ataques cardíacos.'), ('Investigación sobre un gen relacionado con el Alzheimer', 'Científicos han identificado un gen que podría estar vinculado a la enfermedad de Alzheimer, lo que abre nuevas posibilidades para el tratamiento.'), ('Desarrollo de una vacuna contra COVID-19', 'Un equipo de investigadores ha anunciado resultados prometedores en la efectividad de una nueva vacuna contra COVID-19.'); OPTIMIZE TABLE spanish_news FLUSH; ``` Observe the results of two search queries below: one for the defined Spanish stop word and another for the actual Spanish stop word. The defined stop words above overwrite the actual stop words. ```sql SELECT * FROM spanish_news WHERE MATCH(TABLE spanish_news) AGAINST("content:nuevo"); ``` ```output Empty set ``` ```sql SELECT * FROM spanish_news WHERE MATCH(TABLE spanish_news) AGAINST("content:el"); ``` ```output +----------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | headline | content | +----------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | Investigación sobre un gen relacionado con el Alzheimer | Científicos han identificado un gen que podría estar vinculado a la enfermedad de Alzheimer, lo que abre nuevas posibilidades para el tratamiento. | | Avances en la detección temprana del cáncer | Un nuevo método permite detectar el cáncer en etapas más tempranas, aumentando las posibilidades de tratamiento exitoso. | | Descubrimiento de un nuevo tratamiento para la diabetes | Investigadores han desarrollado un tratamiento innovador que mejora el control del azúcar en sangre en pacientes diabéticos. | | Nuevo enfoque para tratar enfermedades cardíacas | Se ha introducido un nuevo enfoque terapéutico que reduce significativamente el riesgo de ataques cardíacos. | +----------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Example 11: Standard Tokenizer with Custom Stop Words Use a `standard` tokenizer with custom stop words. Create a procedure to insert data into a table named `t`. This procedure will be used to generate data for the examples in this section. ```sql DELIMITER // CREATE OR REPLACE PROCEDURE insert_flush() AS BEGIN INSERT INTO t VALUES( 1, "On April 23, 2013, SingleStore launched its first generally available version of the database to the public as MemSQL."), (2, "Early versions only supported row-oriented tables, and were highly optimized for cases where all data can fit within main memory."), (3, "This design was based on the idea that the cost of RAM would continue to decrease exponentially over time, in a trend similar to Moore's law."), (4, "This would eventually allow most use cases for database systems to store their data exclusively in memory not on disk."), (5, "Shortly after launch, MemSQL added general support for an on-disk column-based storage format to work alongside the in-memory rowstore."), (6, "The decreases in cost of memory slowed over time, and the market for purely in-memory database systems largely failed to materialize, with increasing demand for disk-based OLAP workloads."), (7, "Thus, over time, MemSQL's columnstore became a major focus and a crucial feature for customers."), (8, "On October 27, 2020, MemSQL rebranded to SingleStore to reflect a shift in focus away from exclusively in-memory workloads."), (9, "The new name highlights the goal of achieving a universal storage format capable of supporting both transactional and analytical use cases."), (10, "In its current product release, v.7.5, SingleStore became the first and only database to combine separation of storage and compute plus system of record into a single platform."), (11, "Headquartered in San Francisco, California, in June 2021 singlestore.com opened an office in Raleigh, North Carolina. As part of the office opening, SingleStore launched Launch Pad, a center for innovation to incubate and prototype solutions."), (12, "Its other offices include Sunnyvale, California, seattle@singlestore.com, Washington, and Lisbon, Portugal."), (13, "seattle@singlestore.com"); OPTIMIZE TABLE t FLUSH; END;// DELIMITER ; ``` ## Example 11a: No Stop Words Create a table without stop words and insert data into that table. ```sql CREATE TABLE t ( id INT,text VARCHAR(400), SORT KEY (id), FULLTEXT USING VERSION 2 KEY(text) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard"}} }'); CALL insert_flush(); ``` The two queries below query the table for the words `The` and `the`, respectively. Since there are no stop words, results will be returned for both queries. ```sql SELECT text, MATCH(table t) AGAINST ('text:The') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output *** 1. row *** text: The new name highlights the goal of achieving a universal storage format capable of supporting both transactional and analytical use cases. SCORE: 0.5822426080703735 *** 2. row *** text: The decreases in cost of memory slowed over time, and the market for purely in-memory database systems largely failed to materialize, with increasing demand for disk-based OLAP workloads. SCORE: 0.4953887462615967 2 rows in set (0.09 sec) ``` ```sql SELECT text, MATCH(TABLE t) AGAINST ('text:the') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output *** 1. row *** text: Shortly after launch, MemSQL added general support for an on-disk column-based storage format to work alongside the in-memory rowstore. SCORE: 0.5361358523368835 *** 2. row *** text: On April 23, 2013, SingleStore launched its first generally available version of the database to the public as MemSQL. SCORE: 0.216837078332901 *** 3. row *** text: This design was based on the idea that the cost of RAM would continue to decrease exponentially over time, in a trend similar to Moore's law. SCORE: 0.19964565336704254 *** 4. row *** text: The new name highlights the goal of achieving a universal storage format capable of supporting both transactional and analytical use cases. SCORE: 0.1568010002374649 *** 5. row *** text: In its current product release, v.7.5, SingleStore became the first and only database to combine separation of storage and compute plus system of record into a single platform. SCORE: 0.13726447522640228 *** 6. row *** text: The decreases in cost of memory slowed over time, and the market for purely in-memory database systems largely failed to materialize, with increasing demand for disk-based OLAP workloads. SCORE: 0.1351594626903534 *** 7. row *** text: Headquartered in San Francisco, California, in June 2021 singlestore.com opened an office in Raleigh, North Carolina. As part of the office opening, SingleStore launched Launch Pad, a center for innovation to incubate and prototype solutions. SCORE: 0.12553386390209198 ``` Drop the table so that the tablename `t` can be used with the subsequent examples. ```sql DROP TABLE t; ``` ## Example 11b: Default English Stop Words Create a table using default English stop words and query that table for the words `The` and `the`. The words `The` and `the` are included in default English stop words. Thus, neither query returns results. ```sql CREATE TABLE t ( id INT, text VARCHAR(400), SORT KEY (id), FULLTEXT USING VERSION 2 KEY (text) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", token_filters: ["stop"]} }}'); CALL insert_flush(); ``` ```sql SELECT text, MATCH(table t) AGAINST ('text:The') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output Empty set (0.01 sec) ``` ```sql SELECT text, MATCH(TABLE t) AGAINST ('text:the') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output Empty set (0.01 sec) ``` ```sql DROP TABLE t; ``` ## Example 11c: Custom English Stop Words Create a table using custom English stop words and insert the word `the` as a stop word. And query the table for the words `The` and `the`. By default, when using the custom stop words token filter, case is ignored, so no results are returned for either query. ```sql CREATE TABLE t ( id INT, text VARCHAR(400), SORT KEY (id), FULLTEXT USING VERSION 2 KEY (text) index_options '{"analyzer" : {"custom" : {"tokenizer" : "standard", token_filters: [{"stop": {"words": ["the"]}}]} }}'); CALL insert_flush(); ``` ```sql SELECT text, MATCH(table t) AGAINST ('text:The') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output Empty set (0.06 sec) ``` ```sql SELECT text, MATCH(TABLE t) AGAINST ('text:the') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output Empty set (0.05 sec) ``` ```sql DROP TABLE t; ``` ## Example 11d: Custom English Stop Words - Do Not Ignore Case In the following, the query is modified so case is not ignored. In this example, results are returned for the first query, but not the second. ```sql CREATE TABLE t ( id INT, text VARCHAR(400), SORT KEY (id), FULLTEXT USING VERSION 2 KEY (text) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "standard", token_filters: [{"stop": {"ignoreCase": false,"words": ["the"]}}]} }}'); CALL insert_flush(); ``` ```sql SELECT text, MATCH(table t) AGAINST ('text:The') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output *** 1. row *** text: The new name highlights the goal of achieving a universal storage format capable of supporting both transactional and analytical use cases. SCORE: 0.7002022862434387 *** 2. row *** text: The decreases in cost of memory slowed over time, and the market for purely in-memory database systems largely failed to materialize, with increasing demand for disk-based OLAP workloads. SCORE: 0.6494264602661133 2 rows in set (0.06 sec) ``` ```sql SELECT text, MATCH(TABLE t) AGAINST ('text:the') AS SCORE FROM t WHERE score ORDER BY score DESC; ``` ```output Empty set (0.03 sec) ``` ```sql DROP TABLE t; ``` ## Example 12 - Use ngrams to Find a String that Contains a Substring The `n_gram` tokenizer can be used to search for strings that contain a specific substring. That is, you can use ngrams to do a search that is equivalent to having wildcards at the beginning and end of a search term, as shown in the following example. Create a table with a full-text index that uses the `n_gram` tokenizer and insert data into that table. ```sql CREATE TABLE university (name VARCHAR(400), admission_page VARCHAR(400), SORT KEY (name), FULLTEXT USING VERSION 2 KEY(admission_page) INDEX_OPTIONS '{"analyzer" : {"custom" : {"tokenizer" : "n_gram"}}}}'); INSERT INTO university (name, admission_page) VALUES ('Harvard University', 'college.harvard.edu/admissions'), ('Stanford University', 'stanford.edu/admission/'), ('Massachusetts Institute of Technology (MIT)', 'mitadmissions.org/'), ('California Institute of Technology (Caltech)', 'admissions.caltech.edu/'), ('University of Chicago', 'uchicago.edu/en/admissions'); OPTIMIZE TABLE university FLUSH; ``` The following statements and query perform the equivalent of a match against `%arvar%`. The substring `arvar` is matched because the `n_gram` tokenizer splits words into small pieces. As a result, the name Harvard matches `arvar` with the highest ranked score. ```sql SET sql_mode = pipes_as_concat; SET @q = "admission_page:" || "arvar"; SELECT *, MATCH(TABLE university) AGAINST(@q) as score FROM university WHERE MATCH(TABLE university) AGAINST(@q) ORDER BY score DESC; ``` ```output +----------------------------------------------+--------------------------------+---------------------+ | name | admission_page | score | +----------------------------------------------+--------------------------------+---------------------+ | Harvard University | college.harvard.edu/admissions | 2.921722650527954 | | Massachusetts Institute of Technology (MIT) | mitadmissions.org/ | 0.5870963335037231 | | Stanford University | stanford.edu/admission/ | 0.5820327997207642 | | University of Chicago | uchicago.edu/en/admissions | 0.2338818907737732 | | California Institute of Technology (Caltech) | admissions.caltech.edu/ | 0.16432610154151917 | +----------------------------------------------+--------------------------------+---------------------+ ``` In addition, you can use a common table expression (CTE) and `LIKE`, to obtain partial string and substring matches with the speed of a full-text index. The query below runs faster than a query with only a `LIKE` expression. ```sql WITH matches AS ( SELECT *, MATCH(TABLE university) AGAINST(@q) AS score FROM university WHERE MATCH(TABLE university) AGAINST(@q) ORDER BY score DESC LIMIT 10 ) SELECT * FROM matches WHERE admission_page LIKE '%arvar%'; ``` ```output +--------------------+--------------------------------+-------------------+ | name | admission_page | score | +--------------------+--------------------------------+-------------------+ | Harvard University | college.harvard.edu/admissions | 2.921722650527954 | +--------------------+--------------------------------+-------------------+ ``` ## Example 13: Custom Column Mappings ## Example 13a: Per-Column Analyzers Create a table that uses the `french` analyzer for the `french_content` column, the `english` analyzer for the `english_content` column, and the `standard` analyzer for all other columns (`title`). ```sql CREATE TABLE custom_col_analyzers ( title VARCHAR(200), french_content VARCHAR(200), english_content VARCHAR(200), FULLTEXT USING VERSION 2 (title, french_content, english_content) INDEX_OPTIONS '{ "analyzer": "standard", "mappings": { "french_content": { "analyzer": "french" }, "english_content": { "analyzer": "english" } } }' ); ``` Insert data into the table and optimize the table to ensure all the data is indexed. ```sql INSERT INTO custom_col_analyzers VALUES ("fast", "Nous sommes la base de données la plus rapide.", "We are the fastest database."), ("slow", "Cette base de données est un peu lente.", "This database is kind of slow."), ("slowest", "Cette base de données est la plus lente.", "This database is the slowest."); OPTIMIZE TABLE custom_col_analyzers FLUSH; ``` The following query searches the columns using the per-column analyzers defined earlier and returns any row in which the `french_content` column contains `rapide` or the `english_content` column contains `fastest`. ```sql SELECT title, MATCH(TABLE custom_col_analyzers) AGAINST ('french_content:(rapide) OR english_content:(fastest)') AS score FROM custom_col_analyzers WHERE score > 0 ORDER BY score DESC; ``` ```output +-------+--------------------+ | title | score | +-------+--------------------+ | fast | 0.6301337480545044 | +-------+--------------------+ ``` In this query, the `french_content` column is searched for `rapide` using the index built with the `french` analyzer while the `english_content` column is searched for `fast` using the index built with the `english` analyzer. The following query returns any row in which the `french_content` column contains `rapide` or the `english_content` column contains `slowest`. ```sql SELECT title, MATCH(TABLE custom_col_analyzers) AGAINST ('french_content:(rapide) OR english_content:(slowest)') AS score FROM custom_col_analyzers WHERE score > 0 ORDER BY score DESC; ``` ```output +---------+---------------------+ | title | score | +---------+---------------------+ | slowest | 0.49662238359451294 | | fast | 0.4458314776420593 | +---------+---------------------+ ``` Every `MATCH` query requires a prefix that specifies which column to search, thus each column is searched using the appropriate analyzer. ## Example 13b: JSON and BSON Keypath Analyzers Create a table that indexes the `json_column$english_content` field with the `english` analyzer and indexes the `json_column$french_content` field with the `french` analyzer. ```sql CREATE TABLE json_keypath_analyzers ( json_column JSON, FULLTEXT USING VERSION 2 KEY(json_column) INDEX_OPTIONS '{ "mappings": { "json_column$english_content": { "analyzer": "english" }, "json_column$french_content": { "analyzer": "french" } } }' ); ``` Insert data into the table and optimize the table to ensure all the data is indexed. ```sql INSERT INTO json_keypath_analyzers VALUES ('{"english_content": "We are the fastest database."}'), ('{"french_content": "Nous sommes la base de données la plus rapide."}'), ('{"english_content": "This database is kind of slow."}'), ('{ "french_content": "Cette base de données est un peu lente."}'), ('{"english_content": "his database is the slowest."}'), ('{"french_content": "Cette base de données est la plus lente."}'); OPTIMIZE TABLE json_keypath_analyzers FLUSH; ``` The following query searches the `english_content` field using the `english` analyzer, and the `french_content` field using the `french` analyzer. The query returns all the rows with `fastest` in the `english_content` field and `rapide` in the `french_content` field. ```sql SELECT json_column, (MATCH (TABLE json_keypath_analyzers) AGAINST ('json_column$english_content:fastest OR json_column$french_content:rapide') ) AS score FROM json_keypath_analyzers WHERE score > 0 ORDER BY score DESC; ``` ```output +--------------------------------------------------------------------+--------------------+ | json_column | score | +--------------------------------------------------------------------+--------------------+ | {"english_content":"We are the fastest database."} | 0.3150668740272522 | | {"french_content":"Nous sommes la base de donnes la plus rapide."} | 0.3150668740272522 | +--------------------------------------------------------------------+--------------------+ ``` The following query searches only for the word `fastest` in the `english_content` field. ```sql SELECT json_column, (MATCH(TABLE json_keypath_analyzers) AGAINST ('json_column$english_content:fastest') ) AS score FROM json_keypath_analyzers WHERE score > 0 ORDER BY score DESC; ``` ```output +----------------------------------------------------+--------------------+ | json_column | score | +----------------------------------------------------+--------------------+ | {"english_content":"We are the fastest database."} | 0.3150668740272522 | +----------------------------------------------------+--------------------+ ``` ## Supported Language Analyzers The following table lists the supported language analyzers. | **Language** | **Default Stop Word List Link** | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `arabic` | [Apache Lucene Arabic Stop Words](https://github.com/apache/lucene/blob/cfdd20f5bc8387ba24653ca2ba15aa5be10d0ae0/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt) | | `bulgarian` | [Apache Lucene Bulgarian Stop Words](https://github.com/apache/lucene/blob/cfdd20f5bc8387ba24653ca2ba15aa5be10d0ae0/lucene/analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt) | | `bengali` | [Apache Lucene Bengali Stop Words](https://github.com/apache/lucene/blob/cfdd20f5bc8387ba24653ca2ba15aa5be10d0ae0/lucene/analysis/common/src/resources/org/apache/lucene/analysis/bn/stopwords.txt) | | `brazilian_portuguese` | [Apache Lucene Brazilian, Portuguese Stop Words](https://github.com/apache/lucene/blob/cfdd20f5bc8387ba24653ca2ba15aa5be10d0ae0/lucene/analysis/common/src/resources/org/apache/lucene/analysis/br/stopwords.txt) | | `catalan` | [Apache Lucene Catalan Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ca/stopwords.txt) | | `cjk` | [Apache Lucene CJK Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/cjk/stopwords.txt) | | `sorani_kurdish` | [Apache Lucene Sorani, Kurdish Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ckb/stopwords.txt) | | `czech` | [Apache Lucene Czech Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/cz/stopwords.txt) | | `danish` | [Apache Lucene Danish Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/danish_stop.txt) | | `german` | [Apache Lucene German Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/german_stop.txt) | | `greek` | [Apache Lucene Greek Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/el/stopwords.txt) | | `english` | "a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with" | | `spanish` | [Apache Lucene Spanish Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/spanish_stop.txt) | | `estonian` | [Apache Lucene Estonian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/et/stopwords.txt) | | `basque` | [Apache Lucene Basque Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/eu/stopwords.txt) | | `persian` | [Apache Lucene Persian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt) | | `finnish` | [Apache Lucene Finnish Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/finnish_stop.txt) | | `french` | [Apache Lucene French Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/french_stop.txt) | | `irish` | [Apache Lucene Irish Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/irish_stop.txt) | | `galician` | [Apache Lucene Galician Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/gl/stopwords.txt) | | `hindi` | [Apache Lucene Hindi Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt) | | `hungarian` | [Apache Lucene Hungarian Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/hungarian_stop.txt) | | `armenian` | [Apache Lucene Armenian Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/hy/stopwords.txt) | | `indonesian` | [Apache Lucene Indonesian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/indonesian_stop.txt) | | `italian` | [Apache Lucene Italian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/italian_stop.txt) | | `korean` | This is Apache Lucene's Korean (Nori) Analyzer. Filters tokens based on part-of-speech tags: EF, EC, ETN, ETM, IC, JKS, JKC, JKG, JKO, JKB, JKV, JKQ, JX, JC, MAG, MAJ, MM, SP, SSC, SSO, SC, SE, XPN, XSA, XSN, XSV, UNA, NA, VSV.[Part of speech tags.](https://lucene.apache.org/core/10_0_0/analysis/nori/org/apache/lucene/analysis/ko/POS.Tag.html)Custom stop word lists are not supported with the`korean`analyzer[Lucene nori API](https://lucene.apache.org/core/10_0_0/analysis/nori/).[Lucene Analyzer for Korean](https://lucene.apache.org/core/10_0_0/analysis/nori/org/apache/lucene/analysis/ko/package-summary.html). | | `lithuanian` | [Apache Lucene Lithuanian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/lt/stopwords.txt) | | `latvian` | [Apache Lucene Latvian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/lv/stopwords.txt) | | `nepali` | [Apache Lucene Nepali Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ne/stopwords.txt) | | `dutch` | [Apache Lucene Dutch Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/dutch_stop.txt) | | `norwegian` | [Apache Lucene Norwegian Stop Words](https://github.com/apache/lucene/blob/539cf3c9a335bccb50a0bddbf8cabd2738727528/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/norwegian_stop.txt) | | `portuguese` | [Apache Lucene Portuguese Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/portuguese_stop.txt) | | `romanian` | [Apache Lucene Romanian Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt) | | `russian` | [Apache Lucene Russian Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/russian_stop.txt) | | `serbian` | [Apache Lucene Serbian Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/sr/stopwords.txt) | | `swedish` | [Apache Lucene Swedish Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/swedish_stop.txt) | | `tamil` | [Apache Lucene Tamil Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ta/stopwords.txt) | | `telugu` | [Apache Lucene Telugu Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/te/stopwords.txt) | | `thai` | [Apache Lucene Thai Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/th/stopwords.txt) | | `turkish` | [Apache Lucene Turkish Stop Words](https://github.com/apache/lucene/blob/13285279c2d193fe6ad3f323046dd53bbdc8dd4a/lucene/analysis/common/src/resources/org/apache/lucene/analysis/tr/stopwords.txt) | ## Supported Tokenizers The table below lists supported tokenizers. These tokenizers may have custom parameters, which can be obtained and used as described below. ## Get Parameters The parameters and description of each of these tokenizers can be obtained from the links included in the table. ## Example: Get parameters for the `uax_url_email` tokenizer To obtain the parameters for the `uax_url_email` tokenizer, follow the tokenizer factory link for the `uax_url_email` tokenizer, which can be found in the middle column of the table below. The following is the tokenizer factory from the `uax_url_email` tokenizer, which has been obtained from the tokenizer factory link. This tokenizer has one parameter `maxTokenLength`, which defaults to 255. ```sql ``` The `INDEX_OPTIONS` string to create a full-text index with the `uax_url_email` tokenizer specifying a `maxTokenLength` of `300` is shown below. ```sql INDEX_OPTIONS '{ "analyzer": { "custom": { "tokenizer": { "uax_url_email" : { "maxTokenLength": 300 } } } } }' ``` ## List of Supported Tokenizers | **"tokenizer" (Case-Sensitive)** | **Tokenizer Factory Link (Includes Parameters)** | **Tokenizer Class Link (Includes Description)** | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `uax_url_email` | [UAX29URLEmailTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizerFactory.html) | [UAX29URLEmailTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/UAX29URLEmailTokenizer.html) | | `whitespace` | [WhitespaceTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/WhitespaceTokenizerFactory.html) | [WhitespaceTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/WhitespaceTokenizer.html) | | `classic` | [ClassicTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/ClassicTokenizerFactory.html) | [ClassicTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/ClassicTokenizer.html) | | `simple_pattern` | [SimplePatternTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/SimplePatternTokenizerFactory.html) | [SimplePatternTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/SimplePatternTokenizer.html) | | `standard` | [StandardTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/standard/StandardTokenizerFactory.html) | [StandardTokenizer](https://lucene.apache.org/core/8_3_0/core/org/apache/lucene/analysis/standard/StandardTokenizer.html?is-external=true) | | `keyword` | [KeywordTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/KeywordTokenizerFactory.html) | [KeywordTokenizer](https://lucene.apache.org/core/8_3_0/core/org/apache/lucene/analysis/standard/StandardTokenizer.html?is-external=true) | | `letter` | [LetterTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/LetterTokenizerFactory.html) | [LetterTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/core/LetterTokenizer.html) | | `simple_pattern_split` | [SimplePatternSplitTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/SimplePatternSplitTokenizerFactory.html) | [SimplePatternSplitTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/SimplePatternSplitTokenizer.html) | | `pattern` | [PatternTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/PatternTokenizerFactory.html) | [PatternTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/PatternTokenizer.html) | | `thai` | [ThaiTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/th/ThaiTokenizerFactory.html) | [ThaiTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/th/ThaiTokenizer.html) | | `edge_n_gram` | [EdgeNGramTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/EdgeNGramTokenizerFactory.html) | [EdgeNGramTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/EdgeNGramTokenizer.html) | | `n_gram` | [NGramTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizerFactory.html) | [NGramTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramTokenizer.html) | | `wikipedia` | [WikipediaTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/wikipedia/WikipediaTokenizerFactory.html) | [WikipediaTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/wikipedia/WikipediaTokenizer.html) | | `path_hierarchy` | [PathHierarchyTokenizerFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/path/PathHierarchyTokenizerFactory.html) | [PathHierarchyTokenizer](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/path/PathHierarchyTokenizer.html) | | `korean` | *Description*: Tokenizer for Korean that uses morphological analysis.*Supports the following attributes*:
  • userDictionary (JSON array of strings): A JSON array of strings; each string is a term in the dictionary.
  • decompoundMode (JSON string): determines how the tokenizer handles POS.Type.COMPOUND, POS.Type.INFLECT, and POS.Type.PREANALYSIS tokens. Values can be 'none', 'discard', 'mixed', the default is 'discard'.
  • outputUnknownUnigrams (JSON boolean value): If "true" outputs unigrams for unknown words.
  • discardPunctuation (JSON boolean value): If "true", punctuation tokens are dropped from the output.
| | ## Supported Token Filters This table lists the supported token filters, the filter name and a link for the token filter factory documentation which provides parameters and description for the token filter. | **"token\_filters" (Case-Sensitive)** | **Lucene Link for Parameters and Description** | | ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `russian_light_stem` | [RussianLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ru/RussianLightStemFilterFactory.html) | | `scandinavian_normalization` | [ScandinavialnNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/ScandinavianNormalizationFilterFactory.html) | | `decimal_digit` | [DecimalDigitFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/core/DecimalDigitFilterFactory.html) | | `ascii_folding` | [ASCIIFoldingFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilterFactory.html) | | `german_stem` | [GermanStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/de/GermanStemFilterFactory.html) | | `bulgarian_stem` | [BulgarianStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/bg/BulgarianStemFilterFactory.html) | | `codepoint_count` | [CodepointCountFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/CodepointCountFilterFactory.html) | | `pattern_replace` | [PatternReplaceFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/pattern/PatternReplaceFilterFactory.html) | | `persian_normalization` | [PersianNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/fa/PersianNormalizationFilterFactory.html) | | `limit_token_position` | [LimitTokenPositionFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenPositionFilterFactory.html) | | `porter_stem` | [PorterStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/en/PorterStemFilterFactory.html) | | `greek_stem` | [GreekStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/el/GreekStemFilterFactory.html) | | `finnish_light_stem` | [FinnishLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/fi/FinnishLightStemFilterFactory.html) | | `fingerprint` | [FingerprintFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/FingerprintFilterFactory.html) | | `cjk_width` | [CJKWidthFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/cjk/CJKWidthFilterFactory.html) | | `reverse_string` | [ReverseStringFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/reverse/ReverseStringFilterFactory.html) | | `common_grams` | [CommonGramsFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/commongrams/CommonGramsFilterFactory.html) | | `delimited_boost_token` | [DelimitedBoostTokenFilterFactory](https://lucene.apache.org/core/8_9_0/analyzers-common/org/apache/lucene/analysis/boost/DelimitedBoostTokenFilterFactory.html) | | `scandinavian_folding` | [ScandinavianFoldingFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/ScandinavianFoldingFilterFactory.html) | | `hindi_stem` | [HindiStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/hi/HindiStemFilterFactory.html) | | `spanish_plural_stem` | [SpanishPluralStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/es/SpanishMinimalStemFilterFactory.html) | | `indonesian_stem` | [IndonesianStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/id/IndonesianStemFilterFactory.html) | | `trim` | [TrimFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/TrimFilterFactory.html) | | `french_light_stem` | [FrenchLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/fr/FrenchLightStemFilterFactory.html) | | `classic` | [ClassicFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/standard/ClassicFilterFactory.html) | | `fixed_shingle` | [FixedShingleFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/shingle/FixedShingleFilterFactory.html) | | `english_possessive` | [EnglishPossessiveFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/en/EnglishPossessiveFilterFactory.html) | | `german_normalization` | [GermanNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/de/GermanNormalizationFilterFactory.html) | | `keyword_repeat` | [KeywordRepeatFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/KeywordRepeatFilterFactory.html) | | `min_hash` | [MinHashFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/minhash/MinHashFilterFactory.html) | | `remove_duplicates_token` | [RemoveDuplicatesTokenFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/RemoveDuplicatesTokenFilterFactory.html) | | `snowball_porter` | [SnowballPorterFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/snowball/SnowballPorterFilterFactory.html) | | `german_minimal_stem` | [GermanMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/de/GermanMinimalStemFilterFactory.html) | | `norwegian_light_stem` | [NorwegianLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/no/NorwegianLightStemFilterFactory.html) | | `english_minimal_stem` | [EnglishMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/en/EnglishMinimalStemFilterFactory.html) | | `norwegian_minimal_stem` | [NorwegianMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/no/NorwegianMinimalStemFilterFactory.html) | | `czech_stem` | [CzechStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/cz/CzechStemFilterFactory.html) | | `sorani_stem` | [SoraniStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ckb/SoraniStemFilterFactory.html) | | `limit_token_offset` | [LimitTokenOffsetFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenOffsetFilterFactory.html) | | `persian_stem` | [PersianStemFilterFactory](https://lucene.apache.org/core/9_9_1/analysis/common/org/apache/lucene/analysis/fa/PersianStemFilterFactory.html) | | `common_grams_query` | [CommonGramsQueryFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/commongrams/CommonGramsFilterFactory.html) | | `sorani_normalization` | [SoraniNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ckb/SoraniNormalizationFilterFactory.html) | | `swedish_light_stem` | [SwedishLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/sv/SwedishLightStemFilterFactory.html) | | `k_stem` | [KStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/en/KStemFilterFactory.html) | | `french_minimal_stem` | [FrenchMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/fr/FrenchMinimalStemFilterFactory.html) | | `hyphenated_words` | [HyphenatedWordsFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/HyphenatedWordsFilterFactory.html) | | `capitalization` | [CapitalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/CapitalizationFilterFactory.html) | | `lower_case` | [LowerCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/core/LowerCaseFilterFactory.html) | | `hungarian_light_stem` | [HungarianLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/hu/HungarianLightStemFilterFactory.html) | | `telugu_stem` | [TeluguStemmer](https://lucene.apache.org/core/9_8_0//analysis/common/org/apache/lucene/analysis/te/TeluguStemmer.html) | | `italian_light_stem` | [ItalianLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/it/ItalianLightStemFilterFactory.html) | | `limit_token_count` | [LimitTokenCountFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenCountFilterFactory.html) | | `swedish_minimal_stem` | [SwedishLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/sv/SwedishLightStemFilterFactory.html) | | `galician_minimal_stem` | [GalicianMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/gl/GalicianMinimalStemFilterFactory.html) | | `portuguese_minimal_stem` | [PortugueseMinimalStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/pt/PortugueseMinimalStemFilterFactory.html) | | `bengali_normalization` | [BengaliNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/bn/BengaliNormalizationFilterFactory.html) | | `galician_stem` | [GalicianStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/gl/GalicianStemFilterFactory.html) | | `turkish_lower_case` | [TurkishLowerCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/tr/TurkishLowerCaseFilterFactory.html) | | `bengali_stem` | [BengaliStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/bn/BengaliStemFilterFactory.html) | | `indic_normalization` | [IndicNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/in/IndicNormalizationFilterFactory.html) | | `keep_word` | [KeepWordFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/KeepWordFilterFactory.html) | | `drop_if_flagged` | [DictionaryCompoundWordTokenFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/compound/DictionaryCompoundWordTokenFilterFactory.html) | | `latvian_stem` | [LatvianStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/lv/LatvianStemFilterFactory.html) | | `portuguese_light_stem` | [PortugueseLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/pt/PortugueseLightStemFilterFactory.html) | | `apostrophe` | [ApostropheFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/tr/ApostropheFilterFactory.html) | | `arabic_stem` | [ArabicStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ar/ArabicStemFilterFactory.html) | | `delimited_term_frequency_token` | [DelimitedTermFrequencyTokenFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/DelimitedTermFrequencyTokenFilterFactory.html) | | `irish_lower_case` | [IrishLowerCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ga/IrishLowerCaseFilterFactory.html) | | `edge_n_gram` | [EdgeNGramFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ngram/EdgeNGramFilterFactory.html) | | `german_light_stem` | [GermanLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/de/GermanLightStemFilterFactory.html) | | `pattern_capture_group` | [PatternCaptureGroupFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/pattern/PatternCaptureGroupFilterFactory.html) | | `spanish_light_stem` | [SpanishLightStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/es/SpanishLightStemFilterFactory.html) | | `hindi_normalization` | [HindiNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/hi/HindiNormalizationFilterFactory.html) | | `norwegian_normalization` | [NorwegianNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/no/NorwegianMinimalStemFilterFactory.html) | | `shingle` | [ShingleFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/shingle/ShingleFilterFactory.html) | | `telugu_normalization` | [TeluguNormalizationFilter](https://lucene.apache.org/core/8_11_4/analyzers-common/org/apache/lucene/analysis/te/TeluguNormalizationFilter.html) | | `date_recognizer` | [DateRecognizerFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/DateRecognizerFilterFactory.html) | | `n_gram` | [NGramFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ngram/NGramFilterFactory.html) | | `upper_case` | [UpperCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/core/UpperCaseFilterFactory.html) | | `brazilian_stem` | [BrazilianStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/br/BrazilianStemFilterFactory.html) | | `cjk_bigram` | [CJKBigramFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/cjk/CJKBigramFilterFactory.html) | | `truncate_token` | [TruncateTokenFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/TruncateTokenFilterFactory.html) | | `greek_lower_case` | [GreekLowerCaseFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/el/GreekLowerCaseFilterFactory.html) | | `length` | [LengthFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LengthFilterFactory.html) | | `arabic_normalization` | [ArabicNormalizationFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/ar/ArabicNormalizationFilterFactory.html) | | `portuguese_stem` | [PortugueseStemFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/pt/PortugueseStemFilterFactory.html) | | `elision` | [ElisionFilterFactory](https://lucene.apache.org/core/8_4_0/analyzers-common/org/apache/lucene/analysis/util/ElisionFilterFactory.html) | | `korean_part_of_speech` | [KoreanPartOfSpeechStopFilterFactory](https://lucene.apache.org/core/10_0_0/analysis/nori/org/apache/lucene/analysis/ko/KoreanPartOfSpeechStopFilterFactory.html)A token filter that removes tokens that match a set of part-of-speech tags | | `korean_reading_form` | [KoreanReadingFormFilterFactory](https://lucene.apache.org/core/10_0_0/analysis/nori/org/apache/lucene/analysis/ko/KoreanReadingFormFilterFactory.html)A token filter that rewrites tokens written in Hanja to their Hangul form. | | `korean_number` | [KoreanNumberFilterFactory](https://lucene.apache.org/core/10_0_0/analysis/nori/org/apache/lucene/analysis/ko/KoreanNumberFilterFactory.html)A token filter that normalizes Korean numbers to Arabic decimal numbers in half-width characters. | | `stop` | A custom token filter that removes stop words from a token stream.[Custom Stop Words](https://docs.singlestore.com/#section-idm234955753778211.md) | ## Supported Character Filters This table lists the supported character filters, the name, and a link for the parameters. | **"char\_filters" (case-sensitive)** | **Lucene Link for Parameters** | | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `persian` | [PersianCharFilterFactory](https://lucene.apache.org/core/8_8_0//analyzers-common/org/apache/lucene/analysis/fa/PersianCharFilterFactory.html) | | `cjk_width` | [CJKWidthCharFilterFactory](https://lucene.apache.org/core/8_8_0//analyzers-common/org/apache/lucene/analysis/cjk/CJKWidthCharFilterFactory.html) | | `html_strip` | [HTMLStripCharFilterFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/charfilter/HTMLStripCharFilterFactory.html) | | `pattern_replace` | [PatternReplaceCharFilterFactory](https://lucene.apache.org/core/8_3_0/analyzers-common/org/apache/lucene/analysis/pattern/PatternReplaceCharFilterFactory.html) | *** Modified at: July 17, 2026 Source: [/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/) (An index of the documentation is available at /llms.txt) --- # How to Bulk Load Vectors SingleStore recommends loading larger vector data sets using a binary format, such as [Apache Parquet](https://parquet.apache.org/) or [Apache Avro™](https://avro.apache.org/), and using [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) and [PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md)s. The benefits of using a binary format include: * Faster load speed. * Loading from binary file formats uses less CPU than loading from JSON array of numbers format. * Smaller input file size. * Vectors stored in binary format are typically much smaller than vectors stored in JSON array of numbers format. * No loss of precision. * Using the JSON array of numbers format, you may lose precision due to rounding errors. Thus, for high performance, SingleStore recommends using Parquet or Avro formats to load large vector data sets, if possible. If using a binary format is not feasible, SingleStore supports loading vectors encoded in hexadecimal, which is more efficient than JSON array format, but less efficient than Avro or Parquet. SingleStore also supports input and output of vectors in JSON array of numbers format. This format is convenient and user-readable but has performance impacts. Example 1 provides an end-to-end example of generating and loading binary vectors using Parquet file format and Python. Examples 2, 3, and 4 demonstrate loading vector data using Parquet, hexadecimal, and JSON array format, respectively. Using Avro is similar to using Parquet, except that the format should be specified as Avro and not Parquet. Example 5 shows exporting vector data using Parquet and hexadecimal encodings. ## Output Format for Examples Vectors can be output in JSON or binary format. Use JSON format for examples and for output readability. For production, use the default binary for efficiency. Use the following command to output vectors in JSON format. ```sql SET vector_type_project_format = JSON; ``` Use the following command to set the output format back to binary. ```sql SET vector_type_project_format = BINARY; ``` ## Example 1 - Generate and Load Binary Vector Data in Parquet using Python This example first generates a file containing binary vectors using [Parquet](https://parquet.apache.org/) file format. That file is loaded into a SingleStore table in two different ways: first, from S3 first using a [PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) command and second, using a [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) command. ## Generate a Parquet File containing Binary Vector Data Vector data in Parquet file format can be generated using Python and [pyarrow](https://arrow.apache.org/docs/python/index.html). Install the `pyarrow` library, using `pip3` or using an installation command appropriate to your system. ```shell pip3 install -q pyarrow numpy ``` Start the Python interpreter. ```shell python3 ``` The Python code below will generate a Parquet file that matches the schema of the `comments` table with the data shown in the table. ```sql CREATE TABLE comments(id INT, comment TEXT, comment_embedding VECTOR(4), category VARCHAR(256)); ``` | id | comment | comment\_embedding | category | | -- | ---------------------------------------------------- | ------------------------- | -------- | | 1 | "The cafeteria in building 35 has a great salad bar" | \[0.45, 0.55, 0.495, 0.5] | "Food" | | 2 | "I love the taco bar in the B16 cafeteria." | \[0.01, 0.01, 0.1, 0.99] | "Food" | | 3 | "The B24 restaurant salad bar is quite good." | \[0.1, 0.8, 0.2, 0.555] | "Food" | Note: In this table, the vector data is shown as a JSON array for readability; in the example, the data will be generated and loaded in binary format. Run the following code in the python interpreter. This code will create a file named `comments.parquet`. Names ending in `_pqt` are used for column names in the parquet file so that the parquet column names can be distinguished from SingleStore column names. ```python import pyarrow as pa import pyarrow.parquet as pq import numpy as np # Define column names for Parquet file columns = ['Id_pqt', 'Comment_pqt', 'Comment_embedding_pqt', 'Category_pqt'] # create some example binary vectors emb1 = np.array([0.45, 0.55, 0.495, 0.5], dtype=np.float32).tobytes() emb2 = np.array([0.01, 0.01, 0.1, 0.99], dtype=np.float32).tobytes() emb3 = np.array([0.1, 0.8, 0.2, 0.555], dtype=np.float32).tobytes() # Create the data arrays ids = pa.array([1,2,3], type=pa.int8()) comments = pa.array(["The cafeteria in building 35 has a great salad bar", "I love the taco bar in the B16 cafeteria.", "The B24 restaurant salad bar is quite good."], type=pa.string()) binary_vectors = pa.array([emb1, emb2, emb3], type=pa.binary()) categories = pa.array(["Food","Food","Food"],type=pa.string()) table = pa.Table.from_arrays([ids,comments,binary_vectors,categories],columns) pq.write_table(table, 'comments.parquet') ``` ## Load the Parquet File using a Pipeline Create the comments table in SingleStore. The `CREATE TABLE` statement for the comments table is repeated below for convenience. ```sql CREATE TABLE comments(id INT, comment TEXT, comment_embedding VECTOR(4), category VARCHAR(256)); ``` Create a pipeline to load the `comments.parquet` file into the `comments` table. The following pipeline accesses a publicly available bucket which contains the `comments.parquet` file and can be run as is (without credentials). ```sql CREATE PIPELINE parquet_comments_pipe AS LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/comments.parquet' CONFIG '{"region":"us-east-1"}' INTO TABLE comments (id <- Id_pqt, comment <- Comment_pqt, @var <- Comment_embedding_pqt, category <- Category_pqt) FORMAT PARQUET SET comment_embedding = @var:>BLOB:>VECTOR(4, F32); ``` The `SET` statement converts the binary vector from the parquet file into a SingleStore `VECTOR` type. The `SET` statement does not do data conversion; it casts the data to a `VECTOR(4, F32)`, validates that the number of elements is as expected, and that the data does not contain any NAN/INF values. Run the pipeline and verify the import succeeded. ```sql START PIPELINE parquet_comments_pipe FOREGROUND; ``` View the values in the table to confirm the import succeeded. ```sql SET vector_type_project_format = JSON; /* to make vector output human-readable */ SELECT * FROM comments; ``` ```output *** 1. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar comment_embedding: [0.449999988,0.550000012,0.495000005,0.5] category: Food *** 2. row *** id: 3 comment: The B24 restaurant salad bar is quite good. comment_embedding: [0.100000001,0.800000012,0.200000003,0.555000007] category: Food *** 3. row *** id: 2 comment: I love the taco bar in the B16 cafeteria. comment_embedding: [0.00999999978,0.00999999978,0.100000001,0.99000001] category: Food ``` ## Load the Parquet File using LOAD DATA Load the `comments.parquet` file into the `comments` table using a [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) command. `LOAD DATA` may be used to load Parquet and Avro files from S3. The following command accesses a publicly available bucket which contains the `comments.parquet` file and can be run as is (without credentials). ```sql LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/comments.parquet' CONFIG '{"region":"us-east-1"}' INTO TABLE comments (id <- Id_pqt, comment <- Comment_pqt, @emb <- Comment_embedding_pqt, category <- Category_pqt) FORMAT PARQUET SET comment_embedding = @emb:>BLOB:>VECTOR(4, F32); ``` View the values in the table to confirm the import succeeded. ```sql SET vector_type_project_format = JSON; /* to make vector output human readable */ SELECT * FROM comments; ``` ```output *** 1. row *** id: 2 comment: I love the taco bar in the B16 cafeteria. comment_embedding: [0.00999999978,0.00999999978,0.100000001,0.99000001] category: Food *** 2. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar comment_embedding: [0.449999988,0.550000012,0.495000005,0.5] category: Food *** 3. row *** id: 3 comment: The B24 restaurant salad bar is quite good. comment_embedding: [0.100000001,0.800000012,0.200000003,0.555000007] category: Food ``` The [LOAD DATA LOCAL](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data/#UUID-ad611e13-3393-3958-01c4-68bbdd3f9404.md) command may be used to load from a local file. ## Example 2 - Bulk Loading Vector Data using Parquet SingleStore supports loading vector data from Parquet files. SingleStore recommends loading from binary formats such as Parquet or Avro when possible. Create a table with a column of type `VECTOR` with 4 elements (dimension 4) and an element type of 32-bit floating-point number (`F32`). Note that in practice vectors typically have significantly more dimensions than 4, vectors with dimension 4 are used for readability. ```sql CREATE TABLE vectors(id INT, vec VECTOR(4, F32) NOT NULL); ``` This example uses the following data. The vectors are shown in JSON array format for readability. ``` 1,'[0.45, 0.55, 0.495, 0.5]' 2,'[0.1, 0.8, 0.2, 0.555]' 3,'[-0.5, -0.03, -0.1, 0.86]' 4,'[0.5, 0.3, 0.807, 0.1]' ``` ## Example 2a - Using LOAD DATA to Load Parquet Data from S3 The following command loads a parquet file stored in S3 to the `vectors` table. This example uses [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) , which is more efficient than loading data using [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) statements. This command can be run as is because `s3://singlestore-docs-examples-datasets` is a public bucket. ```sql LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/vec_f32_data.parquet' CONFIG '{"region":"us-east-1"}' CREDENTIALS '{}' INTO TABLE vectors FORMAT PARQUET (id <- id, @v <-vec) SET vec = @v:>BLOB:>VECTOR(4, F32); ``` Note the last line of this command which uses a set of casts to load Parquet data into the `VECTOR` attribute: `SET vec = @v:>BLOB:>VECTOR(4, F32)`. By default, `LOAD DATA` expects vector data to be in textual JSON array notation. The casts in the `SET` clause override this and allow more efficient assignment directly from binary data. Run the following SQL statement to verify that the data was loaded. ```sql SET vector_type_project_format = JSON; /* to make vector output readable */ SELECT id, vec FROM vectors ORDER BY id; ``` ```output +------+---------------------------------------------------+ | id | vec | +------+---------------------------------------------------+ | 1 | [0.449999988,0.550000012,0.495000005,0.5] | | 2 | [0.100000001,0.800000012,0.200000003,0.555000007] | | 3 | [-0.5,-0.0299999993,-0.100000001,0.860000014] | | 4 | [0.5,0.300000012,0.806999981,0.100000001] | +------+---------------------------------------------------+ ``` The results do not exactly match the vector values listed at the beginning of this example because elements in the `VECTOR` data type are stored as floating-point numbers and the vector values are not perfectly representable in floating-point representation. To experiment further and use your own data file, follow the instructions in the [Example 5 - Export Vector Data in Parquet and Hexadecimal](https://docs.singlestore.com/#section-idm4627199473104034253762196604.md) section. ## Example 2b - Using a Pipeline to Load Parquet Data from S3 You can also use a pipeline to load data from S3. This example is the same as Example 1a, except that a pipeline is used to load the data. Create a pipeline to load the Parquet data into the `vectors` table. ```sql CREATE PIPELINE vec_pipeline_pqt AS LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/vec_f32_data.parquet' CONFIG '{"region":"us-east-1"}' CREDENTIALS '{}' INTO TABLE vectors FORMAT PARQUET (id <- id, @v <-vec) SET vec = @v:>BLOB:>VECTOR(4, F32); ``` This command uses `CREATE PIPELINE` and will result in an error if the pipeline already exists. If you want to update a pipeline that exists and may be running, use the syntax `CREATE OR REPLACE PIPELINE`. Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for more information. Start the pipeline. ```sql START PIPELINE vec_pipeline_pqt FOREGROUND; ``` The command above starts a pipeline in the `FOREGROUND` so that errors will be displayed in the client. If you are using pipelines that will run continuously, consider starting pipelines in the background by omitting the `FOREGROUND` keyword. Refer to [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) for more information. Once the pipeline has completed, verify that the data has been loaded. ```sql SET vector_type_project_format = JSON; /* to make vector output readable */ SELECT id, vec FROM vectors ORDER BY id; ``` ```output +------+---------------------------------------------------+ | id | vec | +------+---------------------------------------------------+ | 1 | [0.449999988,0.550000012,0.495000005,0.5] | | 2 | [0.100000001,0.800000012,0.200000003,0.555000007] | | 3 | [-0.5,-0.0299999993,-0.100000001,0.860000014] | | 4 | [0.5,0.300000012,0.806999981,0.100000001] | +------+---------------------------------------------------+ ``` Once a pipeline is no longer needed, it can be dropped. ```sql DROP PIPELINE vec_pipeline_pqt; ``` If you have started a pipeline in the background (by omitting the `FOREGROUND` keyword), you can stop the pipeline instead of, or before, dropping it. Stopping a pipeline allows the pipeline to be used later, dropping a pipeline removes the pipeline from the system. Stopping is applicable for background pipelines only. Refer to [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) and [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) for more information. ## Example 3 - Bulk Loading Vector data using Hexadecimal SingleStore supports loading vector data from CSV files containing the hexadecimal encodings of vectors. SingleStore recommends this option when writing vectors into a binary format like Parquet or Avro is inconvenient. Data for the vectors from Example 1 in CSV format with the vectors encoded in hexadecimal is shown below. The examples below demonstrate how to load this data into SingleStore. ``` 1, 6666E63ECDCC0C3FA470FD3E0000003F 2, CDCCCC3DCDCC4C3FCDCC4C3E7B140E3F 3, 000000BF8FC2F5BCCDCCCCBDF6285C3F 4, 0000003F9A99993E8D974E3FCDCCCC3D ``` ## Example 3a - Using a Pipeline to Load Data in Hexadecimal from S3 This example uses the `vectors` table created earlier, so before running this example, delete data from the `vectors` table to prevent duplicate records. ```sql TRUNCATE vectors; ``` The following shows loading the data with vectors encoded in hexadecimal into the `vectors` table. > **📝 Note**: For loading data in CSV format from S3, a pipeline must be used with `LOAD DATA`. `LOAD DATA S3` works without a pipeline only for Parquet or Avro formats. The command below can be run as is because `s3://singlestore-docs-example-datasets/` is a public bucket. ```sql CREATE PIPELINE vec_pipeline_hex AS LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/vec_f32_data_hex.csv' CONFIG '{"region":"us-east-1"}' CREDENTIALS '{}' INTO TABLE vectors FORMAT CSV(id, @v) SET vec = UNHEX(@v):>VECTOR(4,F32); ``` Note the last line of this command uses the [UNHEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/unhex.md) function and a cast to load hexadecimal data into a `VECTOR` attribute. `FORMAT CSV` is the default and is not required in this query. We include it for clarity and to differentiate from the query for loading Parquet data. In addition, note that the column labelling format for CSV, `(id, @v)` in this case, is different than the column labelling format for Parquet `(id ->- id, @v -> vec)` in the previous example. Refer to [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) for more detail. Start the pipeline. ```sql START PIPELINE vec_pipeline_hex FOREGROUND; ``` Once the data load has completed, verify that the data was loaded using the following SQL. ```sql SET vector_type_project_format = JSON; /* to make vector output readable */ SELECT *; FROM vectors ORDER BY id; ``` ```output +------+---------------------------------------------------+ | id | vec | +------+---------------------------------------------------+ | 1 | [0.449999988,0.550000012,0.495000005,0.5] | | 2 | [0.100000001,0.800000012,0.200000003,0.555000007] | | 3 | [-0.5,-0.0299999993,-0.100000001,0.860000014] | | 4 | [0.5,0.300000012,0.806999981,0.100000001] | +------+---------------------------------------------------+ ``` The results do not exactly match the vector values listed, because elements in the `VECTOR` data type are stored as floating-point numbers and the vector values are not perfectly representable in floating-point representation. Once a pipeline is no longer needed it can be dropped. ```sql DROP PIPELINE vec_pipeline_hex; ``` If you have started a pipeline in the background (by omitting the `FOREGROUND` keyword), you can stop the pipeline instead of, or before, dropping it. Stopping a pipeline allows the pipeline to be used later, dropping a pipeline removes the pipeline from the system. Stopping is applicable for background pipelines only. Refer to [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) and [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) for more information. To experiment with your own data files and bucket, refer to [Example 5 - Export Vector Data in Parquet and Hexadecimal](https://docs.singlestore.com/#section-idm4627199473104034253762196604.md) and [Example 6 - Generate Vector Data](https://docs.singlestore.com/#section-idm234545992427085.md). ## Example 4 - Loading Vector Data in JSON Array Format Create a table with an attribute of type `VECTOR` with 3 elements and element type 16-bit integer (`I16`). ```sql CREATE TABLE vectors_i16 (id INT,vec VECTOR(3, I16)); ``` Sample data for the `vectors_i16` table in JSON array format is below. ``` 1,'[1,2,3]' 2,'[4,5,6]' 3,'[1,4,8]' ``` ## Example 4a - Using a PIPELINE to Load Vector Data in JSON Format from S3 The `PIPELINE` below loads this data into the `vectors_i16` table. The command can be run as is because `s3://singlestore-docs-example-datasets/` is a public bucket. ```sql CREATE PIPELINE vec_i16_pipeline AS LOAD DATA S3 's3://singlestore-docs-example-datasets/vecs/vec_i16_data.csv' CONFIG '{"region":"us-east-1"}' CREDENTIALS '{}' INTO TABLE vectors_i16 FIELDS TERMINATED BY ',' ENCLOSED BY "'" FORMAT CSV; ``` Start the pipeline. ```sql START PIPELINE vec_i16_pipeline FOREGROUND; ``` Once the pipeline has completed, validate that the data loaded using the following SQL. ```sql SET vector_type_project_format = JSON; /* to make vector output readable */ SELECT vec FROM vectors_i16; ``` ```output +---------+ | vec | +---------+ | [1,2,3] | | [4,5,6] | | [1,4,8] | +---------+ ``` Once a pipeline is no longer needed it can be dropped. ```sql DROP PIPELINE vec_i16_pipeline; ``` If you have started a pipeline in the background (by omitting the `FOREGROUND` keyword), you can stop the pipeline instead of, or before, dropping it. Stopping a pipeline allows the pipeline to be used later, dropping a pipeline removes the pipeline from the system. Stopping is applicable for background pipelines only. Refer to [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) and [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) for more information. ## Example 5 - Export Vector Data in Parquet and Hexadecimal To experiment further with loading data in Parquet and hexadecimal format, use these instructions in this section to generate a data file and place that file in cloud storage location such as S3. Then, run the pipelines above by by updating the S3 location and adding your config and credentials. First create the `vectors` table and insert data using the commands below. ```sql CREATE TABLE vectors (id int, vec VECTOR(4) not null); INSERT INTO vectors VALUES (1, '[0.45, 0.55, 0.495, 0.5]'); INSERT INTO vectors VALUES (2, '[0.1, 0.8, 0.2, 0.555]'); INSERT INTO vectors VALUES (3, '[-0.5, -0.03, -0.1, 0.86]'); INSERT INTO vectors VALUES (4, '[0.5, 0.3, 0.807, 0.1]'); ``` ## Example 5a - Export Vector Data in Parquet Format When exporting vector data in Parquet, or other binary format, set `vector_type_project_format` to `BINARY`. To export into a Parquet file on S3, use the following command. You will need to replace the S3 URI and the configuration and credentials with your own values. Refer to [SELECT](https://docs.singlestore.com/cloud/query-data/select.md) for additional details on AWS configuration and credentials in SingleStore. ```sql SET vector_type_project_format = BINARY; SELECT id, vec FROM vectors INTO S3 's3://vec_data_folder/vec_f32_data.parquet' CONFIG '{"region":"us-west-2"}' CREDENTIALS '{ "aws_access_key_id":"your_access_key_id", "aws_secret_access_key":"your_secret_access_key", "aws_session_token":"your_session_token"}' FORMAT PARQUET; ``` ## Example 5b - Export Vector Data in Hexadecimal Format To export the `vectors` table with vectors encoded in hexadecimal on S3, use the command below. Again, you will need to replace the S3 URI, configuration, and credentials with your own values. ```sql SELECT id, HEX(vec) AS vec FROM vectors INTO S3 's3://vec_data_folder/vec_f32_data_hex.csv' CONFIG '{"region":"us-west-2"}' CREDENTIALS '{ "aws_access_key_id":"your_access_key_id", "aws_secret_access_key":"your_secret_access_key", "aws_session_token":"your_session_token"}'; ``` ## Example 6 - Generate Vector Data Vector data in Parquet file format can be generated using Python and [pandas](https://pandas.pydata.org/). The `pandas` library provides a function for exporting a `pandas` `DataFrame` to a Parquet file. To encode a vector in hexadecimal format, convert the little-endian binary representation of each floating-point element of the vector to its 8-character hexadecimal encoding and concatenate the results. For example, the following Python code produces the hexadecimal encoding of a vector represented as a `numpy` array: ```python v = numpy.array([0.45, 0.55, 0.495, 0.5], dtype=numpy.float32) v.tobytes(order='C').hex() '6666e63ecdcc0c3fa470fd3e0000003f' ``` To experiment further with loading hexadecimal data, use this code to generate the hexadecimal data and place the data in cloud storage location such as S3. The pipelines above can be run with data you generate data by updating the S3 location and adding your config and credentials. > **❗ Important**: String functions, such as [CONCAT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/concat.md), [LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/length.md), [SUBSTRING](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/substring.md), and [HEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/hex.md) operate on the binary representation of a vector. To use a string function on the JSON string representation of a vector, write code as follows, where `vec` is the name of a column of type `VECTOR`.```sql > STRINGFUNC(vec :> JSON) > ``` ## Related Topics [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) *** Modified at: August 10, 2026 Source: [/cloud/developer-resources/functional-extensions/how-to-bulk-load-vectors/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/how-to-bulk-load-vectors/) (An index of the documentation is available at /llms.txt) --- # Hybrid Search - Reranking Full-Text and Vector Search Results Hybrid search combines multiple search methods into a single ranked result set. A common use case combines **full-text search**, which finds keyword matches, with **vector search**, which finds semantically similar content. Hybrid search is useful when exact terminology matters but semantic relevance is also important. For example, a search application might require documents containing a particular product name or technical term while also returning documents that discuss the same concept using different wording. This retrieval pattern is widely used in retrieval-augmented generation (RAG) pipelines. SingleStore provides both full-text and vector indexes that can accelerate each retrieval method independently. The ranked results from each search method can be combined into a single final ranking using techniques such as **reciprocal rank fusion (RRF)**. ## Sparse and Dense Retrieval Hybrid search, also referred to as hybrid retrieval, combines **sparse** and **dense** retrieval methods: * Full-text search is a form of sparse retrieval, where documents are represented by the terms they contain and matched on exact keywords and phrases. * Vector search is a form of dense retrieval, where documents are represented as embedding vectors and matched by semantic similarity. These embeddings are dense because they have no zero entries. Refer to [Working with Full-text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) and [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) for more information. ## Reciprocal Rank Fusion A common approach for combining two or more ranked result sets, such as full-text search and vector search rankings, is reciprocal rank fusion (RRF). RRF is a rank-based reranking method: it uses the rank position of each result rather than the raw scores produced by each search method. This avoids the need to directly combine full-text search scores and vector similarity scores, which are calculated differently and may use different scoring ranges. The RRF formula for `N` different ranked lists, where `r` is the rank of an item in list `i`, is shown as follows: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltd3784a50f66f592e/6a7adaab6c6eac1f8ef08ce7/img_docs_reciprocal-rank-fusion-formula_inline-dark_smaller-xl8paT.png) In this formula, `k` is a smoothing factor that reduces the impact of rank differences by making the decrease in weights more gradual. Without a smoothing factor (where k = 0), RRF gives the following weights: * Rank 1: 1.0 * Rank 2: 0.5 * Rank 3: 0.33 Using a smoothing factor, such as k = 60, produces a more gradual decline: * Rank 1: 1/61 ≈ 0.0164 * Rank 2: 1/62 ≈ 0.0161 * Rank 10: 1/70 ≈ 0.0143 RRF can be further refined by weighting each ranking differently. For example, weighting full-text search results at 0.7 and vector search results at 0.3 makes exact keyword matches contribute more to the combined score. ## Examples The following examples show how to implement hybrid search in SQL by combining [full-text search (VERSION 2](https://www.singlestore.com/blog/full-text-search-version-2/)), vector search, and RRF. Hybrid search queries generally follow the same pattern: 1. Retrieve candidate results using full-text (sparse) retrieval. 2. Retrieve candidate results using vector (dense) retrieval. 3. Rank each result set independently. 4. Rerank the combined results using the RRF formula. 5. Return the highest-ranked combined results. In these examples, Common Table Expressions (CTEs, `WITH` clauses) perform each retrieval method separately, and `FULL OUTER JOIN` combines their rankings. Example 1 shows an indexed hybrid search over Wikipedia articles. Example 2 combines full-text search, vector search, and a SQL filter. You can extend this approach using user-defined functions (UDFs), such as SingleStore external functions or Wasm UDFs, or external application code, to implement additional ranking techniques, including late interaction models, cross-encoders, or other reranking approaches. Refer to [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md) for more information. ## Example 1 - Hybrid Search of Wikipedia Articles This example demonstrates indexed hybrid search over a dataset of [Wikipedia](https://en.wikipedia.org/wiki/Main_Page) articles using [full-text search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) and [vector search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md). SingleStore has created a dataset of 160M vectors and associated paragraphs to simulate vector-based semantic search over all 6.7 million articles in Wikipedia. The video game subset used in this example has real data, with approximately 41,000 vectors from 1,800 articles. The remaining data is sample data. The full dataset, including how to load and generate the vectors, is described in the [blog post on ANN search](https://www.singlestore.com/blog/singlestore-indexed-ann-vector-search/). The dataset is available under the [Creative Commons Attribution-ShareAlike License 4.0](https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_International_License). Create a table to store the vectors and paragraphs. ```sql CREATE TABLE vecs( id BIGINT(20), url TEXT DEFAULT NULL, paragraph TEXT DEFAULT NULL, v VECTOR(1536) NOT NULL, SHARD KEY(id), KEY(id) USING HASH ); ``` Load data into this table using the following pipeline. ```sql -- since the bucket is open, you can leave the credentials clause as-is CREATE OR REPLACE PIPELINE wiki_pipeline AS load data S3 's3://singlestore-docs-example-datasets/wikipedia-video-game-data/video-game-embeddings.csv' config '{"region":"us-east-1"}' credentials '{"aws_access_key_id": "", "aws_secret_access_key": ""}' SKIP DUPLICATE KEY ERRORS INTO TABLE vecs FORMAT csv FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\r\n'; START PIPELINE wiki_pipeline FOREGROUND; ``` Add a full-text index and a vector index on the table. The vector index is optional – vector similarity queries work with or without indexes. Creating the indexes after loading data is more efficient  than creating them before loading data. ```sql ALTER TABLE vecs ADD FULLTEXT USING VERSION 2 ft_para(paragraph); ALTER TABLE vecs ADD VECTOR INDEX ivf_v(v) INDEX_OPTIONS '{"index_type":"IVF_FLAT"}'; ``` Run `OPTIMIZE TABLE` to ensure the values are indexed. ```sql OPTIMIZE TABLE vecs FLUSH; ``` The following query searches for rows with information about **Super Mario Kart** using full-text and vector search: * Full-text search finds paragraphs that match the string "`super mario kart`". * Vector search finds paragraphs semantically similar to the introduction of the [Super Mario Kart Wikipedia](https://en.wikipedia.org/wiki/Super_Mario_Kart) page by comparing embedding vectors. The variables are set prior to defining the query. * `@ft_q` stores the full-text match condition. * `@v_mario_kart` stores the vector embedding for the Super Mario Kart Wikipedia page. At a high-level, the query is structured as follows: * Full-text search and vector search retrieve candidate results separately using their respective indexes. * Each result set is ranked independently. * The rankings are combined using the RRF formula, and the top five results are returned. The query uses CTEs to create intermediate result sets for each stage. ```sql SET @ft_q = 'paragraph:"super mario kart"'; SET @v_mario_kart = (SELECT v FROM vecs WHERE URL = "https://en.wikipedia.org/wiki/Super_Mario_Kart" ORDER BY id LIMIT 1); WITH fts AS ( /* Find top full-text matches. */ SELECT id, paragraph, MATCH(TABLE vecs) AGAINST(@ft_q) AS SCORE FROM vecs WHERE MATCH(TABLE vecs) AGAINST(@ft_q) ORDER BY SCORE desc LIMIT 200 ), vs AS ( /* Find top vector search matches. <*> is dot product. */ SELECT id, paragraph, v <*> @v_mario_kart AS SCORE FROM vecs ORDER BY score DESC LIMIT 200 ), fts_ranked as ( SELECT *, RANK() OVER (ORDER BY SCORE DESC) AS fts_rank FROM fts ), vs_ranked as ( SELECT *, RANK() OVER (ORDER BY SCORE DESC) as vs_rank FROM vs ) SELECT *, 0.7 * (1.0 / (NVL(fts_rank, 1000.0) + 60)) + 0.3 * (1.0 / (NVL(vs_rank, 1000.0) + 60)) as combined_score FROM fts_ranked f FULL OUTER JOIN vs_ranked v ON f.id = v.id ORDER BY combined_score DESC LIMIT 5; ``` In more detail, this query works as follows: * The first two CTEs in the query (`fts` and `vs`) create ordered lists of 200 rows that best match the full-text and vector search criteria. * The full-text search block (`fts`) uses inverted index search based on [Java Lucene](https://lucene.apache.org/). * The vector search block (`vs`) uses approximate nearest neighbor (ANN) indexing. * The next two CTEs use the `RANK` window function to create ranked lists (`fts_ranked` and `vs_ranked`) of these 200 rows. * The ranked lists are created in separate CTEs to allow the initial CTEs to use indexed search. * If the `RANK` calculation were performed directly inside the initial `fts` and `vs`, it would prevent those CTEs from using index search. * The final query block combines the ranked result sets: * A `FULL OUTER JOIN` includes all items appearing in either ranked list. Note that a standard `INNER JOIN` would discard items that do not appear in both lists. * The `SELECT` clause calculates the final `combined_score` using RRF and returns the five highest-scoring results. This query uses a smoothing factor of 60 and weights full-text results at 0.7 and vector search results at 0.3. Any `NULL` ranks, which can arise from the outer join, are given an artificially high rank of 1000 so they don't contribute meaningfully to the score. RRF allows strong performance in one search method to offset weaker performance in the other. For example, a result with rank pair (2, 1) receives a strong combined score because it ranks highly in both lists. With the weights used in this query, a result with rank pair (3, 11) still ranks higher overall than one with (5, 8) because the full-text score is weighted more heavily (0.7 versus 0.3). ## Example 2 - Hybrid Search of Comments This example combines hybrid search with a SQL filter. The query retrieves candidate results using full-text search and vector search, restricts both searches to comments in the "`Food`" category, and then combines the rankings using RRF. Consider a table of comments that contains: * `id` – an `INT` id * `comment` – a `TEXT` field that stores the text of the comment * `comment_embedding` – a VECTOR that stores an embedding capturing the meaning of the comment. as described in [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) * `category` – a `VARCHAR` that stores a category assigned to the comment Create the `comments` table and insert insert sample data. ```sql CREATE TABLE comments(id INT, comment TEXT, comment_embedding VECTOR(4), category VARCHAR(256)); ``` ```sql INSERT INTO comments VALUES (1, "The cafeteria in building 35 has a great salad bar", '[0.45, 0.55, 0.495, 0.5]', "Food"), (2, "I love the taco bar in the B16 cafeteria.", '[0.01111, 0.01111, 0.1, 0.999]', "Food"), (3, "The B24 restaurant salad bar is quite good.", '[0.1, 0.8, 0.2, 0.555]', "Food"); ``` Add a [full-text](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) version 2 index to the table and run `OPTIMIZE TABLE` to ensure the values are indexed. ```sql ALTER TABLE comments ADD FULLTEXT USING VERSION 2 ft_comment(comment); OPTIMIZE TABLE comments FLUSH; ``` The following SQL query searches for comments that match the word "`restaurant`" and are similar to a query vector (`'[0.44, 0.554, 0.34, 0.62]'`) which is intended to represent an embedding of the user's request. In addition, search is restricted to comments in the category `"Food"`. The following variables are set prior to defining the query. * `@query_comment` stores the full-text match condition  that requires the word "`restaurant`". * `@query_vec` stores the vector embedding from the user's request. Similar to Example 1, the query is structured as follows: * Full-text search and vector search retrieve candidate comments separately using their respective indexes. * The search is restricted to comments in the "`Food`" category. * Each result set is ranked independently. * The rankings are combined using the RRF formula, and the top three results are returned. The `@query_vec` variable is cast to `VECTOR` to ensure that `@query_vec` is a valid `VECTOR` and to improve performance. The `@query_vec` variable is cast to a `VECTOR` to ensure that `@query_vec` is a valid `VECTOR` and to improve performance. ```sql SET @query_comment = 'comment:restaurant'; SET @query_vec = ('[0.44, 0.554, 0.34, 0.62]'):>VECTOR(4); WITH fts AS ( /* Find top full-text matches. */ SELECT id, comment, MATCH(TABLE comments) AGAINST (@query_comment) AS ft_score FROM comments WHERE MATCH(TABLE comments) AGAINST(@query_comment) ORDER BY ft_score desc LIMIT 200 ), vs AS ( /* Find top vector search matches. <*> is dot product. */ SELECT id, comment, @query_vec <*> comment_embedding AS vec_score FROM comments WHERE category = "Food" ORDER BY vec_score desc LIMIT 200), fts_ranked as ( SELECT *, RANK() OVER (ORDER BY ft_score DESC) AS fts_rank FROM fts ), vs_ranked as ( SELECT *, RANK() OVER (ORDER BY vec_score DESC) as vs_rank FROM vs ) SELECT *, 0.7 * (1.0 / (NVL(fts_rank, 1000.0) + 60)) + 0.3 * (1.0 / (NVL(vs_rank, 1000.0) + 60)) AS combined_score FROM fts_ranked f FULL OUTER JOIN vs_ranked v ON f.id = v.id ORDER BY combined_score DESC LIMIT 3; ``` ```output *** 1. row *** id: 3 comment: The B24 restaurant salad bar is quite good. ft_score: 0.46706151962280273 fts_rank: 1 id: 3 comment: The B24 restaurant salad bar is quite good. vec_score: 0.8993000388145447 vs_rank: 2 combined_score: 0.016314 *** 2. row *** id: NULL comment: NULL ft_score: NULL fts_rank: NULL id: 1 comment: The cafeteria in building 35 has a great salad bar vec_score: 0.9810000061988831 vs_rank: 1 combined_score: 0.005578 *** 3. row *** id: NULL comment: NULL ft_score: NULL fts_rank: NULL id: 2 comment: I love the taco bar in the B16 cafeteria. vec_score: 0.6644233465194702 vs_rank: 3 combined_score: 0.005422 ``` In the results, the row with id = 3 has the word "`restaurant`", so it receives a strong full-text ranking. Because full-text search is weighted more heavily than vector search in this example (0.7 vs. 0.3), this higher full-text ranking contributes more to the final RRF score. If only vector search were used, the comment with id = 1 would rank higher since it has the highest vector similarity score. ## Ranking Functions RRF requires a ranking function in the CTEs. Three ranking functions are available: `RANK()`, `DENSE_RANK()`, and `ROW_NUMBER()`. These functions differ in how they handle ties. Tied scores are common in full-text search, particularly for short or common query terms. When using RRF, it is important to consider how ties are ranked. > **📝 Note**: SingleStore recommends using `RANK()` for the ranking function when using RRF. The following table describes the ranking functions and their usage. | Function Name | Functionality | Usage | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | `RANK()` | Returns the rank of the current row as specified by the`ORDER BY`clause.Assigns the same rank to tied rows. Skips subsequent ranks to account for the number of ties. | When ties matter and gaps in ranks are acceptable. | | `DENSE_RANK()` | Returns the rank of the current row as specified by the`ORDER BY`clause.Assigns the same rank to tied rows. Does not skip any ranks. | When ties matter and gaps in ranks are not acceptable. | | `ROW_NUMBER()` | Returns the rank of the current row as specified by the`ORDER BY`clause.The assignment of row numbers among ties is non-deterministic. | When a unique ranking number on each output row is needed. | The following table shows an example of the results of the three functions. | Score | `RANK()` | `DENSE_RANK()` | `ROW_NUMBER()` | | ----- | -------- | -------------- | -------------- | | 100 | 1 | 1 | 1 | | 90 | 2 | 2 | 2 | | 90 | 2 | 2 | 3 | | 80 | 4 | 3 | 4 | ## Related Topics * Blog Post: [Why Your Vector Database Should Not Be a Vector Database](https://www.singlestore.com/blog/why-your-vector-database-should-not-be-a-vector-database/) * Blog Post: [Announcing SingleStore Indexed ANN Vector Search](https://www.singlestore.com/blog/singlestore-indexed-ann-vector-search/) * Blog Post: [Hybrid Search: Vector + Full-Text Search](https://www.singlestore.com/blog/hybrid-search-vector-full-text-search/) * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) * [Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) * [Tuning Vector Indexes and Queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries.md) * [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) * [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) *** Modified at: July 23, 2026 Source: [/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results/) (An index of the documentation is available at /llms.txt) --- # Loading Geospatial Data into SingleStore Helios SingleStore Helioscan load geographic data (points, paths, and polygons) that are represented in [“Well-Known-Text” (WKT)](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry) format. This topic explains how to download map data that is available in the public domain, convert the map data into WKT format, create a SingleStore table that will store the map data, and use the `LOAD DATA` statement to load the downloaded map data into the table. Let us look at an example of how to load geographic data of different countries into SingleStore Helios. 1. Navigate to the [Natural Earth](https://www.naturalearthdata.com/downloads) website and download the dataset for country boundaries. The files are downloaded in the shapefile (SHP) format, which is not supported by SingleStore. Therefore, you need to convert the files to the WKT format, as shown in the steps that follow. 2. Navigate to the [MyGeodata Converter](https://mygeodata.cloud/converter/shp-to-wkt) tool and convert the SHP file to a WKT file. It generates a CSV file in the WKT format. The CSV file contains all the polygons, along with other data about each country. **Note:** SingleStore Helios does not support `MULTIPOLYGON`. Therefore, if the CSV file contains `MULTIPOLYGON`, then convert it into multiple single `POLYGON`s, as shown in the steps that follow. 3. You can visualize a `MULTIPOLYGON` as separate `POLYGON`s by using the [Wicket library](https://arthur-e.github.io/Wicket/sandbox-gmaps3.html) website. After navigating to the website, copy the `MULTIPOLYGON` set from the CSV file and paste it to the Wicket box. Click **Map It**: In this example, the map displays the country `Fiji` with three polygons. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltd57c452dc0c08673/6a7adad7437e0f34d7dcfc48/16083b57784404-DUq8xi.png) 4. Navigate to the CSV file and separate the row containing `MULTIPOLYGON` into three individual rows, with each row containing a `POLYGON`. Copy the other column data in the original `MULTIPOLYGON` row to each of the three `POLYGON` rows. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltef0bdf639874400c/6a7ada60de2315b2cffcc48a/16083b5778fbac-kuJuS2.png) All the data is now available in the CSV file in the WKT format to load into SingleStore. 5. Create a table `Countries`: ```sql CREATE TABLE Countries ( boundary GEOGRAPHY, name_short VARCHAR(3), name VARCHAR(50), name_long VARCHAR (50), abbrev VARCHAR (10), postal VARCHAR (4), iso_a2 VARCHAR (2), iso_a3 VARCHAR(3), name_formal VARCHAR(100), SHARD KEY(name) ); ``` In this table, all polygons will be loaded into the `boundary` column, which is of the data type `GEOGRAPHY`. 6. Load the data from the CSV file to the `Countries` table: ```sql LOAD DATA INFILE '/data/natural_earth_countries_110m-1.csv' INTO TABLE Countries (boundary, @, @, @, @, @, @, @, @, @, @, name_short, @, name, name_long, abbrev, postal, name_formal, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, iso_a2, iso_a3, @, @, @, @, @, @, @ @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @, @,) FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES; ``` > **📝 Note**: In the [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) syntax, the `@` symbol is used to ignore unwanted columns. The `FIELDS TERMINED BY ','` and `OPTIONALLY ENCLOSED BY '*'` clauses define the column delimiters and ensure that the commas in the polygon data are not treated as a separate field. 7. To see the results of loading the data, select the data from the `Countries` table: ```sql SELECT * FROM Countries ORDER BY name DESC; ``` You can now use this data for all [Geospatial Functions](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions.md). *** Modified at: April 3, 2023 Source: [/cloud/developer-resources/functional-extensions/loading-geospatial-data-into-singlestore-helios/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/loading-geospatial-data-into-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Tuning Vector Indexes and Queries SingleStore supports indexed vector search which can be used to improve the performance of similarity searches over larger vector data sets. Similarity searches find a set of nearest neighbors to a query vector. Exact k-nearest neighbor search (kNN) finds the (exact) top k nearest neighbors. Indexed vector search finds an approximate top N nearest neighbors, and is known as Approximate Nearest Neighbor (ANN) search. Vector indexes can significantly improve the speed of similarity search. However, the vector indexes perform ANN search and thus return approximate results and, in addition, require memory to store the indexes. There is a tradeoff between faster queries and, on the other side, approximate results and memory use. To obtain the performance improvement from vector indexes, it is important to only use vector indexes in specific cases, pick the right index type, and set the index-building and search parameters to meet your application's needs. For background on vector indexes, also refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md), [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md), [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md), and [Hybrid Search - Reranking Full-Text and Vector Search Results](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md). ## Choose a Vector Index Vector indexes can significantly improve the speed of query execution over larger vector data sets. This section addresses when to use an index and choosing a vector index to use. ## When to Use a Vector Index SingleStore recommends using vector indexes for searches over very large sets of vectors (e.g. billions of vectors) when there are no other filters in the query that can be used to reduce the number of vectors being searched. That is, if your data set is only millions of vectors, or if there is a `WHERE` clause (filter) in the query which will eliminate many of the rows in the table, you may have acceptable performance without using an index. As vector indexes use memory, if you can achieve acceptable performance without an index, SingleStore recommends not creating an index and using brute-force (scan) for your queries. ## Types of Vector Indexes SingleStore supports several types of vector indexes, but recommends using IVF\_PQFS and HNSW\_FLAT. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for more information on types of indexes and configuration parameters supported by SingleStore. ## Example Table The following example table represents product reviews from a web site. The table contains the text of the reviews as well as a vector embedding that captures the meaning of the review. Embeddings are a common use of vectors as is described in [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md). The columns of the example table are: * `id`: an `INT` id * `review`: a `TEXT` field which stores the text of the comment * `review_embedding`: a `VECTOR` of 1024 dimensions which stores a vector embedding of the review * `category`: a `VARCHAR` which stores a category assigned to the review Below is the SQL that creates the example `reviews` table. ```sql CREATE TABLE reviews ( id INT OPTION 'Integer', review TEXT OPTION 'SeekableString', review_embedding VECTOR(1024) NOT NULL OPTION 'SeekableString', category VARCHAR(256)); ``` Note that the `id` column is qualified `OPTION 'Integer'`, and the review and `review_embedding` columns are qualified as `OPTION 'Seekable String'`. The `OPTION` clauses specify how SingleStore should encode the data in those columns and will improve query performance. The `reviews` table will be used to demonstrate creating and tuning indexes and to discuss advantages, disadvantages, and tradeoffs during index creation and tuning. ## General Advantages and Disadvantages There are several index types and a variety of configuration parameters. Vector indexes can be chosen and tuned to provide the accuracy, memory size, and search time that your application requires. In general, IVF\_PQFS has lower index build time and index size and HNSW\_FLAT has lower search time and higher accuracy (recall). IVF\_PQFS uses a fraction of the memory (\~10x reduction or more) of HNSW and has a faster index build. The disadvantage of IVF\_PQFS is potentially lower accuracy (recall) and longer search time; however, the recall and search time of IVF\_PQFS can be improved by tuning the index building and search parameters as described in the section [Tune IVF\_PQFS Parameters](https://docs.singlestore.com/#section-idm4630743721939234278251338644.md) below. HNSW gives excellent performance (search time) and accuracy (recall); however, the size of an HNSW\_FLAT index is the total size of the vectors. SingleStore keeps vector indexes in memory, so to use HNSW\_FLAT, you will need to have enough memory to store the index. As the size of an IVF\_PQFS index is significantly smaller than an HNSW\_FLAT index, tuning an IVF\_PQFS index to give good performance and recall can be a good solution for many situations. SingleStore recommends IVF\_PQFS unless your application requires high recall and very fast search speed; use HNSW\_FLAT only if you cannot obtain good enough performance (search time) and accuracy (recall) by tuning IVF\_PQFS. ## Where to Start This section is intended to provide a starting point for adding an index to your vector data. The section provides suggestions for initial parameter settings, selected performance pointers, and suggestions for testing index performance. The section assumes that the `reviews` table has been created and data has been loaded into that table. Creating the index after loading the data is more efficient than loading data after creating the index. ## Create the Vector Index SingleStore recommends starting with an IVF\_PQFS index due to the reduced memory required to store IVF\_PQFS indexes. When building the IVF\_PQFS index, SingleStore recommends setting the index building parameter `m` to be the number of dimensions of the vector column divided by 4. The following command creates such an index for the `reviews` table, where `m` is set to 256 (which is 1024/4) and `nlist` is set to 1024 and `nprobe` is set to 20. Refer to [nlist](https://docs.singlestore.com/#section-idm4546886446915234279966497722.md) and [nprobe](https://docs.singlestore.com/#section-idm4654928651059234279964805811.md) for more information on setting `nlist` and `nprobe`. The `metric_type` is set to `DOT_PRODUCT`, which is also the default. The parameter `m` is an index building parameter that determines the number of subquantizers used in product quantization, a method for compressing vectors. The parameter `nlist` specifies the number of clusters created during index build. The parameter `nprobe` specifies the number of probes to be used at query time. ```sql ALTER TABLE reviews ADD VECTOR INDEX ivf_pqfs(review_embedding) INDEX_OPTIONS'{"index_type":"IVF_PQFS", "metric_type":"DOT_PRODUCT", "m":256, "nlist":1024, "nprobe":20}'; ``` After the index is loaded, run the `OPTIMIZE` command for best performance. ```sql OPTIMIZE TABLE reviews FLUSH; ``` ## Perform a Similarity Search Query This example searches for the top 5 reviews that are most similar to a particular phrase. The query below first sets up a `@query_vec`, which is a vector that represents the phrase to be searched on. In the example below, the `@query_vec` is created from a JSON string; more commonly the `@query_vec` would be obtained from an API or from querying a different table. Refer to [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) for more information on vector embeddings. The query below uses an `ORDER BY … LIMIT` query to look for the top 5 reviews that are most similar to `@query_vec`. `DESC` ordering is used in the `ORDER BY` clause as the metric used to create the vector index was `DOT_PRODUCT`. ```sql SET @query_vec=('[0.002, 0.01, 0.001, ...]'):>VECTOR(1024); SELECT id, review, review_embedding <*> @query_vec AS score FROM reviews ORDER BY score USE INDEX (ivf_pqfs) SEARCH_OPTIONS '{"k":50}' DESC LIMIT 5; ``` Key points to note about this query: 1. The `@query_vec` variable is cast to a `VECTOR` to ensure that `@query_vec` is a valid `VECTOR` and to improve performance. 2. The `SEARCH_OPTIONS` clause sets the search parameter `k` to 50. A setting of `k` equal to the `LIMIT` value \* 10 is a good starting point. In this example `LIMIT` is 5, so `k` is set to 5\*10 = 50. The `USE INDEX` clause in the query above specifies that the index named `ivf_pqfs` should be used in the query. `USE INDEX` is included for completeness, but is not necessary in this query. The index will be used regardless of whether the clause is included or not. ## Verify a Vector Index is Being Used It is important to verify that your query is using a vector index. A vector index can be used in the following conditions: * The metric in the query (dot product vs. euclidean distance) matches the metric used to create the index. * The order in the` ORDER BY` clause (`DESC` vs. `ASC`) in the search query matches the metric. You can verify that a vector index is being used in a query with `EXPLAIN` as shown below. ```sql SET @query_vec = ('[0.002, 0.01, 0.0001, ...]'):>VECTOR(1024); EXPLAIN SELECT id, review, review_embedding <*> @query_vec AS score FROM reviews ORDER BY score USE INDEX (ivf_pqfs) SEARCH_OPTIONS '{"k":50}' DESC LIMIT 5; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------------------------+ | Project [remote_0.id, remote_0.review, remote_0.score] | | TopSort limit:5 [remote_0.score DESC] | | Gather partitions:all alias:remote_0 parallelism_level:segment | | Project [reviews.id, reviews.review, DOT_PRODUCT(reviews.review_embedding, (@query_vec:>vector(1024, F32) NOT NULL)) AS score] | | TopSort limit:5 [DOT_PRODUCT(reviews.review_embedding, (@query_vec:>vector(1024, F32) NOT NULL)) DESC] | | ColumnStoreFilter [INTERNAL_VECTOR_SEARCH(0, (@query_vec:>vector(1024, F32) NOT NULL), 5, '{\"k\":50}') index] | | ColumnStoreScan dbname.reviews, SORT KEY __UNORDERED () table_type:sharded_columnstore | +-------------------------------------------------------------------------------------------------------------------------------+ ``` The ColumnStoreFilter line of the `EXPLAIN` output above shows `INTERNAL_VECTOR_SEARCH(...)` which indicates that the vector index is being used. If `INTERNAL_VECTOR_SEARCH` is not in the `ColumnStoreFilter` line, the vector index is not being used. ## Test Index Performance To understand if an index meets the needs of your application, you will want to test the query accuracy (recall) and search time. ## Test Accuracy via Recall As described previously, recall measures the accuracy of the results returned. Specifically, recall is the percentage of relevant results that are returned by the query. To check the recall of a query with a vector index, you need to run the query with and without the vector index and compare the results returned. The USE INDEX clause can be used for this purpose. The query below contains an empty USE INDEX clause which will disable the use of the vector index. Note that the SEARCH\_OPTIONS clause has also been removed. ```sql SET @query_vec = ('[0.002, 0.01, 0.0001, ...]'):>VECTOR(1024); SELECT id, review, review_embedding <*> @query_vec AS score FROM reviews ORDER BY score USE INDEX() DESC LIMIT 5; ``` To obtain the recall of the indexed query, perform the following steps: 1. Run the query without the index using the empty `USE INDEX()` clause as shown above and note the results. This will give you the actual top N results to your query. (N=5 in the query above.) 2. Run the query with the index and note the results. This gives you the top N results as determined by the indexed search. Remember that the indexed search returns approximate results. 3. Determine how many of the top N results from the indexed query results are in the actual top N results. For example, if the query with the index returned 4 of the 5 actual results, the recall is 4 / 5 or 80%. Note that the index query will return 5 results. For determining recall, the question is how many of those 5 results are in the results from the query without the index. ## Measure Search Time When measuring search time, it is important to exclude the run time of the first query and to exclude the round-trip time between the client and server. The runtime of the first query includes query compile time and the time to load the vector index into memory and is not indicative of the performance of the query. The round-trip time includes communication between client and server and time for the server to package the result tuples and ship them to the client. Including round-trip time will make the search time artificially high. To measure search time, use the [Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) feature of the SingleStore portal. Ignore the first profile result, then run the query several times and take the average of the Execution Time values shown in the Summary. ## Tune IVF\_PQFS Parameters This section discusses tuning an IVF\_PQFS index for good performance (search time) and accuracy (recall). IVF\_PQFS has three index building parameters: `m`, `nprobe`, and `nlist` and one search parameter `k`. Each of these are discussed in turn. The ordering of the sections below indicates the recommended ordering of tuning. That is, when tuning, start with `m`, move on to `k` and `nprobe`, finally `nlist`. ## m The parameter `m` is an index building parameter that determines the number of subquantizers used in product quantization. Product quantization is used to compress high-dimensional vectors so they use less memory. The memory reduction in using product quantization is significant and is the reason that IVF\_PQFS indexes use significantly less memory than HNSW\_FLAT indexes. The parameter `m` effectively trades off index size and accuracy of results. A higher `m` gives greater accuracy (recall) but the index size is larger. A lower value of `m` gives a smaller index, but the results will have lower accuracy. The parameter `m` must be set so that dimensions % `m` = 0 and m defaults to 32. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for details. Consider values for m such as: dimensions/2, dimensions/4, dimensions/8. SingleStore recommends starting with dimensions/4. ## k The parameter `k` is an index search parameter which can be specified at query time. An example query using `k` is shown below. The following query uses the `reviews` table defined above. As before, the `@query_vec` is a `VECTOR` representing a phrase the user is interested in and the query searches for reviews in the `reviews` table that are similar to `@query_vec`. Specifically, the query searches for the top five comments which are most similar to `@query_vec`. ```sql SET @query_vec = ('[0.002, 0.01, 0.0001, ...]'):>VECTOR(1024); SELECT id, review, review_embedding <*> @query_vec AS score FROM reviews ORDER BY score USE INDEX (ivf_pqfs) SEARCH_OPTIONS '{"k":50}' DESC LIMIT 5; ``` In this query, `k` determines the number of rows output by the vector index scan, that is `k` determines the number of rows that the vector index scan produces before the `ORDER BY … LIMIT` clause is applied. Larger values of `k` give more accurate results (better recall), but increase search times. The parameter `k` must be >= limit, where limit is from `ORDER BY … LIMIT` clause and the default for `k` is the `LIMIT` value. SingleStore suggests starting with a `k` of 10 \* `LIMIT` and adjusting to get the recall and search time needed for your application. > **📝 Note**: A common cause of poor recall with IVF\_PQFS is not setting a good value for `k`. ## nprobe nprobe is a search parameter which specifies the number of probes to be used at query time. During index build, vectors are clustered into `nlist` clusters and then the query search is done by searching `nprobe` of the nearest clusters. SingleStore recommends setting `nprobe` to 20; `nprobe` cannot be greater than `nlist`. A query using a vector index will search an `nprobe`/`nlist` fraction of the data. Increasing nprobe will improve accuracy at the cost of increased query time. ## nlist `nlist` is an index-building parameter which specifies the number of inverted lists created when the index is built. That is, `nlist` controls the number of centroids generated by k-means clustering. SingleStore recommends starting with `nlist` = 1024; `nlist` is the last of the parameters you should try tuning. If you fix `nprobe` and increase `nlist`, you get faster queries with worse accuracy; however, the results depend on data distribution and may not change monotonically. A general recommendation is to set `nlist` to be of the same order as the square root of the number of rows in a segment. The following query finds the segment size in number of rows for the `reviews` table. ```sql SELECT database_name, table_name, AVG(rows_count) AS average_segment_size_in_rows FROM information_schema.columnar_segments WHERE database_name = 'dbname' AND table_name = 'reviews' GROUP BY ALL; ``` ## Tune HNSW\_FLAT Parameters HNSW\_FLAT indexes can be used when IFV\_PQFS indexes cannot be tuned to meet application needs. HNSW\_FLAT indexes use approximately the same amount of memory as the vectors themselves, be aware of the memory requirements of this type of index. HNSW\_FLAT has two index-building parameters, `M` and `efConstruction`, and one search parameter, `ef`. For tuning HNSW\_FLAT indexes, the primary parameter to tune is the search parameter, `ef`, which trades recall for performance. A larger `ef` gives better accuracy (higher recall), but slower search. That is, increasing `ef` increases exhaustiveness of graph search which gives better accuracy (recall) at the cost of slower search performance. SingleStore recommends starting with `ef` set to 120. If recall is too low, you may wish to increase `ef`. SingleStore recommends starting with setting `M` to 12 and `efConstruction` to 120. ## Track Vector Index Memory Use Vector indexes are stored in memory in SingleStore and can be large. HSNW\_FLAT indexes take the same amount of space as the vectors themselves. IVF\_PQFS indexes use much less memory. Thus, understanding how much memory is taken by a vector index is important. Refer to [Tracking Vectory Index Memory Use](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing/#section-idm234597445043844.md) in [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for more information. ## Related Topics [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) [Hybrid Search - Reranking Full-Text and Vector Search Results](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md) *** Modified at: July 10, 2026 Source: [/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries/) (An index of the documentation is available at /llms.txt) --- # Working with Full-Text Search Full-text search allows you to search for words or phrases in a large body of text efficiently. The search can be exact or “fuzzy” and performed over the text types `JSON`, `CHAR`, `VARCHAR`, `TEXT`, and `LONGTEXT`. ## Overview SingleStore provides full-text search compatible with Apache's Java Lucene ([Apache Lucene Core](https://lucene.apache.org/core/)) and which uses [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) scoring. Below are select examples of this `VERSION 2` full-text search. These examples are intended to provide a high-level introduction to SingleStore's full-text search. Details and additional examples are provided in the following sections. The following `articles` table is used for the examples. ```sql CREATE TABLE articles ( id INT UNSIGNED, year int UNSIGNED, title VARCHAR(200), body TEXT, SORT KEY (id), FULLTEXT USING VERSION 2 art_ft_index (title, body)); ``` This example searches for articles containing the term `Database` in the `title` or the phrase `"Business Intelligence"` in the `body` of the article: ```sql SELECT title, body FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:Database OR body:("Business Intelligence")'); ``` A proximity search can be used to find articles that contain the phrase `"SQL databases"` in the `body` of the article, with the specification that `SQL` and `databases` must appear within 5 words of each other. ```sql SELECT title, body FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:"SQL databases"~5'); ``` A fuzzy search can be used to find articles with words in the `body` of the article that are within an edit distance of 2 of the term `dtabase`, which in the following example is deliberately misspelled. ```sql SELECT title, body FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:dtabase~2'); ``` Regular expressions can be used in full-text searches. The following query uses a regular expression for articles with words that begin with `data` in the `body` of the article. ```sql SELECT title, body FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:/data.*/'); ``` Finally, full-text search can also be used on JSON columns; refer to [Create a Version 2 Full-Text Index over JSON](https://docs.singlestore.com/#section-idm4593657285028834222863631365.md) examples. The section [Related Topics](https://docs.singlestore.com/#section-idm4542112525337632485207490414.md) contains additional resources on full-text search including using full-text search in hybrid search and configuring full-text indexes for high performance. ## Version 2 SingleStore Process SingleStore's `VERSION 2` full-text search system uses a JLucene service. The JLucene service is a Java process that provides an interface for the SingleStore engine. This interface allows the engine to perform full-text searches and create full-text search indexes for future use. The JLucene service uses software from the Apache Lucene project. The SingleStore engine and the JLucene service run on the same machine. Communication between the two processes occurs through domain sockets and shared memory. Typically, communication occurs on a per-segment basis. The default analyzer for `VERSION 2` is the StandardAnalyzer from Apache Lucene configured to use the StandardTokenizer from Apache Lucene, a lower-case filter, and a set of [stopwords](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match/#UUID-3ba95e6d-0a5b-7e67-43b4-fbf42b975d93.md). The StandardTokenizer is a grammar-based tokenizer which uses the word break rules from the Unicode Text Segmentation, as specified in [Unicode Standard Annex #29](https://unicode.org/reports/tr29/). ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1935f451f5e7ad46/6a7adb3b33fa8a8f5c1f6f39/full-text-search-diagram_staticA-1120x333-light_-BXF8B4.png) ## Create, Add, and Drop Full-Text Indexes SingleStore recommends using `VERSION 2` full-text search for new development. SingleStore's [Legacy (Version 1)](https://docs.singlestore.com/#section-idm234388647864144.md) full-text search has been deprecated. ## Create a Version 2 Full-Text Index Create a version 2 full-text index with a [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) statement using the `FULLTEXT USING VERSION 2` index type. The `USING VERSION 2` syntax must be used in the `CREATE TABLE` command to utilize the `VERSION 2` process. ```sql CREATE TABLE ( , FULLTEXT USING VERSION 2 [] (,..., ) [INDEX_OPTIONS '{...}'] ); ``` Add a version 2 full-text index to a table [ALTER TABLE ADD FULLTEXT](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) statement. ```sql ALTER TABLE ADD FULLTEXT USING VERSION 2 [fts_index_name] (,..., ) [INDEX_OPTIONS '{...}'] ; ``` To add a column to a full-text index, drop and recreate the index. ## Drop a Full-Text Index Drop a full-text index using a [DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-index.md) or an [ALTER TABLE DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) statement. ```sql DROP INDEX | ON ; ``` ```sql ALTER TABLE DROP INDEX | ; ``` If an index name was not designated when creating the table, the full-text index `index_key_name` must be used when dropping the index. The full-text index `index_key_name` is displayed when the `SHOW INDEXES FROM ` command is executed. ## Remarks The following apply to `VERSION 2`: * Full-text indexes are only supported on columnstore tables ([How the Columnstore Works](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md)). * Only one full-text index is supported per table. * That one index can index multiple columns. * The different columns can have different tokenizers, stemming, and analyzers. * Full-text indexes are not supported on Views. * During indexing, column values are split into tokens, which are turned into indexed terms.  The maximum length of an indexed term is 32766 bytes. * If a column value is longer than that limit, you will see a message similar to: `Forwarding Error (): Leaf Error (): Document contains at least one immense term in field= (whose UTF8 encoding is longer than the max length 32766), all of which were skipped...` * A `MATCH … AGAINST` clause may not refer to a CTE ([WITH (Common Table Expressions)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md)) because the CTE produces a dynamic table which does not have a full-text index. Similarly, `MATCH … AGAINST` clauses may not refer to derived tables. * These restrictions apply to `MATCH`, `BM25`, and `BM25_GLOBAL`. * New inserts and updates into columnstore tables may initially be stored in a hidden rowstore table before being flushed to a segment file. The affected segment is re-indexed when the background flusher runs. * In that case, the full-text index in the columnstore will be updated asynchronously for new inserts and updates. Inserts and updates from this rowstore table can be force-pushed to the columnstore table by using the `OPTIMIZE TABLE FLUSH` command. * Since an index is created for each segment file, the distribution of words within the segment may affect the score of full-text queries, especially when the segments have very few rows and the columns have very few words. ## Upgrade to Full-Text Version 2 A table can be upgraded from legacy full-text search to `VERSION 2` full-text search. To do so: * Drop the existing full-text index using the `DROP INDEX` command. * Use the `ALTER TABLE` command with the `FULLTEXT USING VERSION 2` argument to create a `VERSION 2` full-text index. Once you have upgraded your table to use a `VERSION 2` full-text index, you will also need to change your queries to use the `VERSION 2` query syntax. ## Query Full-Text Indexes A full-text index search matches a search term or terms to content in a table that has been full-text indexed. The following query which uses the `articles` table from the [Overview](https://docs.singlestore.com/#section-idm234905408733053.md) finds all articles with the word `Database` in the `title`. The `title` column has a full-text index on it. ```sql SELECT title, body FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:Database'); ``` Terms can be single terms or phrases, can be modified with wildcards or boosted, can be combined with boolean operators, and more as described in the following sections. The [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md), [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md), or [BM25\_GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) functions can be used to search full-text indexed content. These functions provide different tradeoffs between efficiency and accuracy with `MATCH` being the most efficient. SingleStore recommends using `MATCH` for most applications. Refer to [MATCH and BM25 Scoring](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match/#section-idm23455788535454.md) and [Comparison of BM25 and BM25\_GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25/#section-idm234863501492724.md) for more details. SingleStore supports [custom analyzers](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers.md) for full-text `VERSION 2` search. Users can customize full-text search by: * Using built-in analyzers for a variety of languages. The built-in analyzers can be customized with custom stop-word lists. * Using custom analyzers in which a user can specify a tokenizer, optional token and character filters, and an optional stop-word list. [Custom column mappings](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234993094025062.md) can be used to index different columns in a table with different analyzers. For example, you can index columns with text in different languages with analyzers specific to those languages. Refer to [Example 13 Custom Column Mappings](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234993108252177.md). JSON columns can be searched using full-text search as shown in [Example 7: Score Over JSON](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25/#section-idm234557974454307.md). In addition, custom column mappings can be used to index fields in a JSON document with different analyzers. Refer to [Example 13b JSON and BSON Keypath Analyzers](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm234993129355599.md). ## Remarks * Each `MATCH`, `BM25`, or `BM25_GLOBAL` clause applies to only one table. * To search against multiple tables, specify multiple `MATCH`, `BM25`, or `BM25_GLOBAL` clauses. ## Terms in Full-Text Searches There are two types of search terms: single terms and phrases. A single term is a single word such as *test* or *hello* which does not require quotes. A phrase is a group of words surrounded by double quotes such as "hello SingleStore". Multiple search terms can be combined with Boolean operators to form more complex queries. Search terms can be modified to provide a wide range of search options as described below. ## Wildcard Support Single and multiple character wildcard searches within single terms are supported, but not within search phrases. * Use the `?` symbol to perform a single character wildcard search. * Use the `*` symbol to perform a multi character wildcard search. > **❗ Important**: Neither `?` or `*` are supported at the beginning of a term. For example, searching for *?ello* or *\*ello* will generate an error.Refer to [Example 12](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers/#section-idm235055586170482.md) for an example of using a `n_gram` tokenizer to emulate using a wildcard at the start and end of a string and search for a substring. A single character wildcard search matches words based on a single character. For example, to search for “text” or “test”, use the search term: `te?t`. A multiple character wildcard search matches words based on zero or more characters. For example, to search for “test”, “tests”, or “tester”, use the search term `test*`. Wildcard searches in the middle of a search term can also be used, such as `te*t`. The following examples demonstrate how these wildcard searches appear in sample queries: ```sql SELECT * FROM wilsearch1 WHERE MATCH (TABLE wilsearch1) AGAINST ('col1:te?t'); ``` ```sql SELECT * FROM wilsearch1 WHERE MATCH (TABLE wilsearch1) AGAINST ('col1:te*t'); ``` ## Boosting a Term Boosting a search term means increasing the relevance or importance of that search term in the search results. To boost a search term, use the caret ("^") symbol with a boost factor (a number) at the end of the search term. The higher the boost factor, the more relevant the search term will be. For example, if you are searching for `Single Store` and you want the term `Store` to be more relevant, boost it using the ^ symbol along with the boost factor next to the term, such as you could type `Single Store^4`. This will make rows with the term Store appear more relevant. You can also boost phrase as in the example: `"Single Store"^4 "MySQL"`. > **📝 Note**: The boost factor is 1 by default. Although the boost factor must be positive, it can be less than 1 (e.g. 0.2), which leads to the term or phrase have less relevance. The following examples show the use of boosting: ```sql SELECT * FROM bstsearch1 WHERE MATCH (TABLE bstsearch1) AGAINST ('col1:"Single Store"^4 "MySQL"'); ``` ```sql SELECT * FROM bstsearch1 WHERE MATCH (TABLE bstsearch1) AGAINST ('col1:SingleStore MySQL^0.02'); ``` ## Operators in Full-Text Search Full-text version 2 supports operators listed on the [Java Lucene full-text search string syntax](https://lucene.apache.org/core/10_1_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package.description) page. ## Grouping Terms in a Query ## Grouping Single Terms Parentheses can be used to group terms to form subqueries. This can be useful for controlling the boolean logic for a query. For example, to search for either "Single" or "Store" and "MemSQL", use the query `(Single OR Store) AND MemSQL`. This ensures that "MemSQL" exists with either the "Single" or "Store" search terms. ## Grouping Multiple Terms into a Single Field Parentheses can be used to group multiple clauses into a single field. The following query can be used to search for "SingleStore" in the `col1` full-text column or for "MemSQL" in the `col2` full-text column. ```sql SELECT * FROM grpsearch1 WHERE MATCH (TABLE grpsearch1) AGAINST ('col1:SingleStore OR col2:MemSQL'); ``` ## Escaping Special Characters Special characters that are part of the query syntax must be escaped to directly match them. These special characters are: ```sql + - && || ! ( ) { } [ ] ^ " ~ * ? : \ / ``` Use the `\` character before the special character to escape it. For example, to search for `(1+1):2` ,use the query` \(1\+1\)\:2`. ## Regular Expression Searches Regular expression searches that match a pattern between forward slashes ("/") are supported. Regular expression grammar follows [Apache Lucene RegExp](https://lucene.apache.org/core/10_1_0/core/org/apache/lucene/util/automaton/RegExp.html) grammar. For example, to find rows containing "moat" or "boat", use `/[mb]oat/`: ```sql SELECT * FROM rexsearch1 WHERE MATCH (TABLE rexsearch1) AGAINST ('col1:/[mb]oat/'); ``` The following example searches for dates with the format of month/day/year where the month and day can have one or two digits and year can have two or four digits. ```sql SELECT * FROM rexsearch1 WHERE MATCH (TABLE rexsearch1) AGAINST ('col1:/\d{1,2}\/\d{1,2}\/\d{2,4}/'); ``` ## Fuzzy Searches Fuzzy searches based on the [Damerau-Levenshtein Distance](https://en.wikipedia.org/wiki/Damerau-Levenshtein_distance) are supported. To perform a fuzzy search, use the tilde ("\~") symbol at the end of a single term. For example, to search for a term similar in spelling to "roam" use the fuzzy search: `roam~`, which will find terms like "foam" and "roams". An *edit* is a change made to a search term to transform it into another string. An optional “edit” parameter can be used after the tilde to specify the maximum number of edits permitted. This value can be 0, 1, or 2. If not specified, the default value is 2 edits. > **📝 Note**: Only integer (non-fractional) values are permitted. The following is an example of a fuzzy search of the term "roam" without the optional parameter: ```sql SELECT * FROM fuzsearch1 WHERE MATCH (TABLE fuzsearch1) AGAINST ('col1:roam~'); ``` The following is an example of a fuzzy search of the term "roam" with the optional parameter: ```sql SELECT * FROM fuzsearch1 WHERE MATCH (TABLE fuzsearch1) AGAINST ('col1:roam~1'); ``` ## Fuzzy Search OPTIONS Clause Fuzzy searches can be augmented by using an optional `OPTIONS` clause within the `MATCH` syntax. The syntax for adding options is as follows: ```sql MATCH (TABLE ) AGAINST ( OPTIONS ); ``` The set of supported options are: * `fuzzy_prefix_length`: This is the number of characters at the start of a search term that must be identical (not fuzzy) to the query term if the query is to match the search term. The default value is `0`. * `fuzzy_max_expansions` : This is the maximum number of terms to match against. The default value is `50`. * `fuzzy_transpositions`: This allows transpositions with adjacent characters. This can capture some errors more efficiently, particularly mistyped adjacent characters. When transposition is allowed, matching against "ca\~" will be returned "ac" also. The default value is `TRUE`. All of the options are case-sensitive, optional, and can be combined as required. For example: ```sql SELECT * FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:roam~' OPTIONS '{"fuzzy_prefix_length": 2}'); ``` ```sql SELECT * FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:roam~' OPTIONS '{"fuzzy_prefix_length": 1, "fuzzy_transpositions": false}'); ``` ## Proximity Searches Finding words within a specified distance away is supported. To perform a proximity search, use the tilde ("\~") symbol at the end of a search phrase. For example, to search for a "Single" and "Store" within 10 words of each other in a row, use the search `"Single Store"~10`: ```sql SELECT * FROM prxsearch1 WHERE MATCH (TABLE prxsearch1) AGAINST ('col1:"Single Store"~10'); ``` ## Range Searches Range searches match rows where the column(s) values are between the lower and upper bound specified by the query. Range searches can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically or by value depending on whether the search is over string or numeric values. Square brackets ("\[]") indicate an inclusive search, curly brackets ("{}") indicate an exclusive search. A range clause of \[123 TO 345] implies a numeric range search, a range clause of \[abc TO def] implies a lexicographic range search, and \[$123$ TO $345$] forces a lexicographic range despite the numeric range values. Specific examples are provided below. An inclusive range search matches all rows where the values are between the search terms, including values equal to the search terms. Inclusive range queries are denoted by square brackets (“\[ ]”). For example, the following query searches for rows where the numeric value of `col1` is 2008 through 2020: ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:[2008 TO 2020]'); ``` An exclusive range search matches all rows where the values are between the search terms, but are not equal to the search terms. Exclusive range queries are denoted by curly brackets (“{ }”). For example, the following query searches for rows where the numeric value of `col1` is between 2008 and 2020, but is not equal to 2008 or 2020: ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:{2008 TO 2020}'); ``` Range searches may be lexicographic or numeric. Typically searches over strings are expected to order lexicographically while searches over numeric values are expected to order by numeric value. The type of range search to use (lexicographic or numeric) is determined by the values entered in the range clause in the query. If both the upper and lower bounds of the range are numeric, then a numeric range search will be performed. If either value is not numeric, then a lexicographic query will be used. The following query will use a numeric search as both the values 2008 and 2020 are numeric. ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:{2008 TO 2020}'); ``` The following query will use a lexicographic search as both of the range values are strings that do not contain numeric values. Note that this query is inclusive of A, but exclusive of B. ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:[A TO B}'); ``` This query will use a numeric search as both the values "2008" and "2020" can be parsed numerically. ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:{"2008" TO "2020"}'); ``` It is possible to force a lexicographic search on a numeric-style range clause. To do so, surround the numeric upper and lower bounds with the '$' character. The following query will use a lexicographic search. ```sql SELECT * FROM rngsearch1 WHERE MATCH (TABLE rngsearch1) AGAINST ('col1:[$2008$ TO $2020$]') ``` ## ORDER BY ... LIMIT Optimization Full-text search queries can take advantage of an optimization that pushes down the value of `LIMIT` in an `ORDER BY … LIMIT` query into the full-text index search. This optimization will reduce the number of results returned from the full-text index search which can increase query performance. For this optimization to work, the query must follow these rules: * The `ORDER BY` clause must use `DESC` sort order. * There must be only one full-text search function (`MATCH`, `BM25`, or `BM25_GLOBAL`) in the query. Refer to [Example ORDER BY … LIMIT](https://docs.singlestore.com/#section-idm234621702469151.md) and [Full-text ORDER BY LIMIT Optimization](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes/#section-idm234742933008124.md) for examples. ## Relevancy Score The relevancy score of an expression in a `MATCH` statement denotes the ranking of the expression based on the following factors: * Number of times an expression appears in a column. More occurrences of an expression in the matched column(s) increases its relevancy score. * Rarity of the expression. Rare words have a higher relevancy score than commonly used words. * The length of the column containing the expression. A column with a short expression has a higher relevancy score than a column with a long expression. ## BM25 Scoring Full-text search version 2 supports [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) scoring. Refer to [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) for more information. ## Full-Text Index Management In SingleStore, a full-text index is built for each columnstore segment. A full-text index for a segment may be created either when a segment in a table with a full-text index is flushed to disk, or when a full-text index is created with an `ALTER` command. ## Memory Usage The `Alloc_fts2_svc` status variable is the number of allocated bytes (out of the total `max_memory`) that is attributable to the next-generation (`VERSION 2`) full-text subprocess. View the value of `Alloc_fts2_svc` on the leaf nodes with the `SHOW LEAF STATUS EXTENDED` command. ```sql SHOW LEAF STATUS EXTENDED LIKE '%fts%'; ``` The value of `Alloc_fts2_svc` on the aggregator nodes can be viewed with the `SHOW STATUS EXTENDED` command. The `Alloc_fts2_svc` variable will have a value on the aggregator nodes if a search has been run using the `BM25_GLOBAL` search function. Thus, the following command will return results only if a `BM25_GLOBAL` search has been used. ```sql SHOW STATUS EXTENDED LIKE '%fts%'; ``` ## Configurations A set of global variables is available to configure full-text search version 2. Refer to the full-text variables sections of [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for details. Also, refer to [Configuring Full Text and Vector Indexes](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes.md) for information on configuring the engine for high performance for full-text indexes. > **📝 Note**: If your system is experiencing high load due to full-text index builds, SingleStore recommends reducing the value of `fts2_max_connections` to `16` or `8`. This change will reduce load, but will slow down indexing. ## Examples ## Create a Version 2 Full-Text Search Index and Query Using MATCH This example creates a `FULLTEXT` index for both the title column and the body column. Either column can be queried separately using `MATCH (TABLE ) AGAINST ()`, and the index on the column will be applied. ```sql CREATE TABLE articles ( id INT UNSIGNED, year int UNSIGNED, title VARCHAR(200), body TEXT, SORT KEY (id), FULLTEXT USING VERSION 2 art_ft_index (title, body)); ``` ```sql INSERT INTO articles (id, year, title, body) VALUES (1, 2021, 'Introduction to SQL', 'SQL is a standard language for accessing and manipulating databases.'), (2, 2022, 'Advanced SQL Techniques', 'Explore advanced techniques and functions in SQL for better data manipulation.'), (3, 2020, 'Database Optimization', 'Learn about various optimization techniques to improve database performance.'), (4, 2023, 'SQL in Web Development', 'Discover how SQL is used in web development to interact with databases.'), (5, 2019, 'Data Security in SQL', 'An overview of best practices for securing data in SQL databases.'), (6, 2021, 'SQL and Data Analysis', 'Using SQL for effective data analysis and reporting.'), (7, 2022, 'Introduction to Database Design', 'Fundamentals of designing a robust and scalable database.'), (8, 2020, 'SQL Performance Tuning', 'Tips and techniques for tuning SQL queries for better performance.'), (9, 2023, 'Using SQL with Python', 'Integrating SQL with Python for data science and automation tasks.'), (10, 2019, 'NoSQL vs SQL', 'A comparison of NoSQL and SQL databases and their use cases.'), (11, 2020, 'Real-time Data Analysis', 'An introduction to real-time analytics.'), (12, 2021, 'Analysis for Beginners', 'Simple examples of real time analytics.'), (13, 2023, 'Data-Dictionary Design', 'Create and maintain effective data dictionaries.'), (14, 2024, 'Scalable Performance', 'Designing for scalability.'); OPTIMIZE TABLE articles FLUSH; ``` ## Search for a Single Word Search for rows with the word `database` in the `body` column. ```sql SELECT * FROM articles WHERE MATCH (TABLE articles) AGAINST ('body:database'); ``` ```output +----+------+---------------------------------+-----------------------------------------------------------------------------+ | id | year | title | body | +----+------+---------------------------------+-----------------------------------------------------------------------------+ | 7 | 2022 | Introduction to Database Design | Fundamentals of designing a robust and scalable database. | | 3 | 2020 | Database Optimization | Learn about various optimization techniques to improve database performance.| +----+------+---------------------------------+-----------------------------------------------------------------------------+ ``` ## Boolean OR Search Search for rows with the word `Database` in the title column or the phrase `Business Intelligence` in the body column. ```sql SELECT title FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:Database OR body:("Business Intelligence")'); ``` ```output +---------------------------------+ | title | +---------------------------------+ | Introduction to Database Design | | Database Optimization | +---------------------------------+ ``` ## Boolean AND Search Search for rows with the word `SQL` in the `title` column and the phrase `"Data Security"` in the `title` column. ```sql SELECT title FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:SQL AND title:("Data Security")'); ``` ```output +----------------------+ | title | +----------------------+ | Data Security in SQL | +----------------------+ ``` ## Multiple MATCH Clauses Use two `MATCH` clauses to search for rows with the word `SQL` in the `title` or the phrase `Business Intelligence` in the `title`, and that also have the word `development` in the `body`. In this example, the `+` indicates that the search term is required. ```sql SELECT title FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:SQL OR body:("Business Intelligence")') AND MATCH (TABLE articles) AGAINST ('body:web+'); ``` ```output +------------------------+ | title | +------------------------+ | SQL in Web Development | +------------------------+ ``` ## Use +, \*, and ? Search for rows that have a word starting with `Data` and followed by an arbitrary number of characters in the `title`, or words like `function` followed by a single character (e.g. functions, but not functional) in the title. In this example, the `+` indicates that `Data` should appear at the beginning of the word. ```sql SELECT title FROM articles WHERE MATCH (TABLE articles) AGAINST ('title:(+Data*) OR title:function?'); ``` ```output +---------------------------------+ | title | +---------------------------------+ | Introduction to Database Design | | SQL and Data Analysis | | Database Optimization | | Data Security in SQL | | Data-Dictionary Design | | Real-time Data Analysis | +---------------------------------+ ``` ## Create a Version 2 Full-Text Index over JSON A full-text index can be created over a JSON column in the same manner it can be created over any other text-type column. ```sql CREATE TABLE ft_records ( id INT UNSIGNED, title VARCHAR(200), records JSON, SORT KEY(id), FULLTEXT USING VERSION 2 rec_ft_index (title, records)); ``` The full-text index is created over a JSON column by concatenating all leaf string values in the JSON as a multi-valued field. The engine variable `fts2_position_increment_gap` defines the logical spacing between concatenated leaf string values to prevent matching across different leaf string values. The default value is 100. In addition to having a field for each column in the full-text index, there will also be additional fields for each unique keypath in the JSON document. ```sql INSERT INTO ft_records VALUES ( 1, 'document', '{ "k1": "cucumber", "k2": ["dragonfruit", "eggplant"], "k3": [ {"k3_1": "fig", "k3_2": "grape"}, {"k3_1": ["huckleberry", "iceberg lettuce"]}, "jicama" ] }'); OPTIMIZE TABLE ft_records FLUSH; ``` ```sql SELECT title, records FROM ft_records WHERE id = 1; ``` ```output +----------+--------------------------------------------------------------------------------------------------------------------------------------------+ | title | records | +----------+--------------------------------------------------------------------------------------------------------------------------------------------+ | document | {"k1":"cucumber","k2":["dragonfruit","eggplant"],"k3":[{"k3_1":"fig","k3_2":"grape"},{"k3_1":["huckleberry","iceberg lettuce"]},"jicama"]} | +----------+--------------------------------------------------------------------------------------------------------------------------------------------+ ``` The JSON document for the `records` column for the row inserted above is shown below. ```json { "title": "document", "records": ["cucumber", "dragonfruit", "eggplant", "fig", "grape", "huckleberry", "iceberg lettuce", "jicama"], "records$k1": "cucumber", "records$k2": ["dragonfruit", "eggplant"], "records$k3": "jicama", "records$k3.k3_1": ["fig", "huckleberry", "iceberg lettuce"], "records$k3.k3_2": "grape" } ``` Here the `records` column is a multi-valued field of all leaf string values in JSON. This field can be queried like any other full-text indexed field. An exception is that matching will not occur over separate values unless the maximum number of positions allowed between matching phrases, or the “slop,” exceeds the value of `fts2_position_increment_gap`. Along with `records`, other fields like `records$k1` and `records$k3.k3_1` are created to allow searching at each unique keypath present in the JSON document. The dollar sign ("$") is used as a delimiter between the SQL column name and the JSON keypath. ## Query over a JSON column Querying over the entire JSON column can be performed in the same way as with any other column that is part of the full-text index. ```sql SELECT (MATCH (TABLE ft_records) AGAINST ('records:/.*cumber/')) AS cumber FROM ft_records; ``` ```output +----------+ | cumber | +----------+ | 1 | +----------+ ``` ## Query over a JSON keypath The following example shows how you can search for the string `fig` at the keypath `k3.k3_1` in the `records` column using the field grouping syntax. ```sql SELECT (MATCH (TABLE ft_records) AGAINST ('records$k3.k3_1:fig')) AS fig FROM ft_records; ``` ```output +------------------------+ | fig | +------------------------+ | 0.13076457381248474 | +------------------------+ ``` ## Query for Two Terms Enclose the terms `cucumber` and `raspberry` in parentheses to do a boolean `OR` search for the terms (`cucumber` and `raspberry`) in the document. The document matches because the term `cucumber` appears in the document. ```sql SELECT id FROM ft_records WHERE MATCH (TABLE ft_records) AGAINST ('records:(cucumber raspberry)'); ``` ```output +------+ | id | +------+ | 1 | +------+ ``` ## Phrase Query and Proximity Search In the example below, quotes are placed around the words `cucumber` and `dragonfruit` to search for the phrase `"cucumber dragonfruit"`. ```sql SELECT id FROM ft_records WHERE MATCH (TABLE ft_records) AGAINST ('records:"cucumber dragonfruit"'); ``` ```output Empty set (0.008 sec) ``` The phrase `"cucumber dragonfruit"` does not match the JSON document in the `ft_records` table because the words `cucumber` and `dragonfruit` belong to different leaf strings in that document. When a proximity search query for the phrase `"cucumber dragonfruit"` with a slop of `~100` is used, the JSON document matches. ```sql SELECT id FROM ft_records WHERE MATCH (TABLE ft_records) AGAINST ('records:"cucumber dragonfruit"~100'); ``` ```output +------+ | id | +------+ | 1 | +------+ ``` Slop indicates the maximum number of words allowed between words in a phrase for the phrase to be considered a match. That is, a slop of `~100` means if there are 100 words or less between `cucumber` and `dragonfruit` in the document, the document will be considered a match. Further, a proximity search is only done on multi-valued fields such as JSON (or BSON) columns when the slop value is greater than or equal to `fts2_position_increment_gap`. In this example slop is `100` and `fts2_position_increment_gap=100` (100 is the default value of `fts2_position_increment_gap`). Since slop is equal to `fts2_position_increment_gap` matching across separate values occurs. And since the words `cucumber` and `dragonfruit` are within 100 words of each other in the document, a match is returned. ## Use of \~ in Fuzzy Search and Proximity Search As described above, the \~ symbol is used for both proximity searches and fuzzy searches. If the \~ appears after a phrase (multiple words delimited by quotes "), this indicates that a proximity search with a slop value of the number appearing after the \~ should be performed. If the \~ appears after a single word, this indicates that an edit distance (Levenshtein) comparison should be performed on that word (also called a fuzzy search). The following example shows a fuzzy search. ```sql SELECT id FROM ft_records WHERE MATCH (TABLE ft_records) AGAINST ('records:dronfruit~2'); ``` ```output +------+ | id | +------+ | 1 | +------+ ``` A match is returned because `dronfruit`, which is intentionally misspelled, is within an edit distance of 2 of `dragonfruit`. ## Phrase Search vs. Boolean Term Search It is important to understand the difference between a phrase search and a boolean term search as it relates to JSON fields. Words in quotes (like `"cucumber dragonfruit"`)  are searched for as a single phrase in a single JSON field. Words that appear in parentheses (like `(cucumber dragonfruit)` are searched as if there is a boolean `OR` between the words. When terms are separated by logical operators, they may be matched in different JSON fields. In the following example, `cucumber` and `dragonfruit` appear in fields `k1` and `k2` respectively, and thus a match occurs. ```sql SELECT id FROM ft_records WHERE MATCH (TABLE ft_records) AGAINST ('records:(cucumber dragonfruit)'); ``` ```output +------+ | id | +------+ | 1 | +------+ ``` ## Example - `ORDER BY … LIMIT` The query below will take advantage of the `ORDER BY...LIMIT` pushdown. In this query there is one `MATCH` function, named `match_res`, the columns in the `WHERE` and `ORDER BY` clauses are the same, and the sort order is `DESC`. ```sql SELECT id, title, MATCH (TABLE articles) AGAINST ('body:database') AS match_res FROM articles WHERE match_res ORDER BY match_res DESC LIMIT 25; ``` ## Index Repair Full-text index creation failure is rare. However, if full-text index creation fails, you will receive the error `ER_FTS_INDEX_NEEDS_REPAIR_ON_SEGMENT`. The index can be repaired by running `OPTIMIZE TABLE FIX_FULLTEXT`. ## Legacy (Version 1) SingleStore Process SingleStore's legacy full-text search system uses a CLucene service which is embedded in the SingleStore database engine. The legacy (`VERSION 1`) full-text search has been deprecated, SingleStore recommends using `VERSION 2` full-text search for new development. Version 1 full-text indexes are only supported on columnstore tables. They can only be enabled as part of a `CREATE TABLE` statement using the [FULLTEXT](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) index type. This means version 1 full-text indexes cannot be altered after the table is created. If the table is dropped, then the index is deleted automatically. ```sql CREATE TABLE (... FULLTEXT USING VERSION 1 [] ()) ``` Content in columns that are full-text indexed can be searched using the [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) function. Each `MATCH` clause applies to only one table. To search against multiple tables, specify multiple `MATCH` clauses. [Upgrade to Full-Text Version 2](https://docs.singlestore.com/#section-idm235055478427269.md) describes how to upgrade to full-text `VERSION 2`. > **📝 Note**: New inserts and updates into columnstore tables may initially be stored in a hidden rowstore table before being flushed to a segment file. The affected segment is re-indexed when the background flusher runs.In that case, the full-text index in the columnstore will be updated asynchronously for new inserts and updates. Inserts and updates from this rowstore table can be force-pushed to the columnstore table by using the `OPTIMIZE TABLE FLUSH` command.Since an index is created for each segment file, the distribution of words within the segment may affect the score of full-text queries, especially when the segments have very few rows and the columns have very few words. The following example illustrates how to create a table with a legacy version full-text search index and how to query from that table.SingleStore recommends explicitly using the `VERSION 1` syntax. ```sql CREATE TABLE articles ( id INT UNSIGNED, year int UNSIGNED, title VARCHAR(200), body TEXT, SORT KEY (id), FULLTEXT USING VERSION 1 (title, body)); ``` ```sql SELECT * FROM articles WHERE MATCH (title,body) AGAINST ('database'); ``` Refer to the [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) page for more examples. ## Related Topics * [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md)- Allows for semantic searching, which is searching based on meanings, not keywords. * [Hybrid Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md) - Allows full-text and vector search methods in one query. Full-text and vector search ranking can be combined. * [Configuring Full Text and Vector Indexes](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/configuring-full-text-and-vector-indexes.md) * [BM25 and BM25\_GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) * [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) * [HIGHLIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/highlight.md) * Training: [Full-Text Index and Search](https://training.singlestore.com/learn/course/internal/view/elearning/635/full-text-index-and-search) *** Modified at: July 24, 2026 Source: [/cloud/developer-resources/functional-extensions/working-with-full-text-search/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search/) (An index of the documentation is available at /llms.txt) --- # Working with Geospatial Features SingleStore supports geospatial queries. The term “geospatial” covers a very broad range of features. There is a formal standardization for some of those features, known as SQL-MM or OpenGIS. The features in SingleStore are not a complete implementation of that standard, but follow the same structure. SingleStore uses the [S2Geometry](https://s2geometry.io/) library. At a basic level, geospatial support requires three things: data types, topological functions, and measurement functions. ## Geospatial Types ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta875008a9e0f465d/6a7adb9af6825778ec58d989/polygons_paths_points-iBhw2C.png) The three main geospatial object types are points, paths, and polygons. SingleStore supports a subset of the standard “Well-Known Text” (WKT) syntax to describe them. All geospatial functions can accept either a geospatial object or the equivalent WKT string as arguments. SingleStore does not support `MULTIPOLYGON`, `GEOMETRY`, `CURVE`, or other keywords. ## Point A POINT is a simple longitude / latitude pair. Note the lack of a comma between the numbers. Also note that the ordering in the pair is significant, the first value is the longitude and the second value is the latitude. ``` "POINT(-74.044514 40.689244)" ``` ## Path or Linestring A linestring or path object is an ordered list of points. The points (vertices) in a linestring (or polygon) object are connected by edges. ``` "LINESTRING(0 0, 0 1, 1 1)" ``` ## Polygon A polygon is similar to a linestring, except that it is “closed.” That is, it must have at least three unique points, and the first and last point-pairs must be equal: ``` "POLYGON((0 0, 0 1, 1 1, 0 0))" ``` When defining a polygon (or loop), the area inside the polygon's edges is the "smaller" portion of the globe. For example, if you define a polygon that outlines the shape of Australia, there could be two possible sides: All of the globe except Australia, or Australia itself. Because the side with the smaller area contains Australia, that is what is considered inside the polygon. Because of this, it is not possible to select more than half of the globe inside a polygon. You can also describe multiple shapes (or rings) inside a polygon object, i.e., a polygon object can also be represented as a set of rings. For example, a set of islands, or a lake inside a country. Each ring is represented as a set of points. The first ring (external ring) in the set represents the shape of the polygon. The subsequent rings in the set (internal rings) represent the holes in the shape. The rings within the boundary of a polygon (internal rings) can never cross. They may however intersect at a point, but only as a tangent. Each respective shape embedded within a polygon object is represented using the following notation. ``` "POLYGON(Ring 0, Ring 1, ... Ring N)" ``` Note the parenthesis around pairs of numbers, which are separated by commas. ``` "POLYGON((0 0, 0 10, 10 10, 10 0, 0 0), (5 5, 6 5, 5 6, 5 5))" ``` In the above example, rings `(0 0, 0 10, 10 10, 10 0, 0 0)` and `(5 5, 6 5, 5 6, 5 5)` represent Ring 0 and Ring 1, respectively. ## Edges Edges in SingleStore are "spherical geodesics", i.e. they are the shortest paths on the sphere and therefor they do not always follow raw longitude/latitude lines like "plate carrée" projection edges do. This can lead to unexpected results when looking for intersection, etc. as the shapes formed from these edges in SingleStore may differ from those drawn in other applications and viewer/visualization tools that use "plate carrée" projection edges. For that reason, when testing it is best to use a viewer/application that uses spherical geodesics. Note that using a viewer or other application is not necessary -- it's used here just to show the difference in behavior between geodesics and plate projection edges. For example, in the two images below we have a polygon outlining the country of Sweden and a second polygon checking for intersection. The same polygon definitions are used in both. We are using WKT viewers to demonstrate the difference in behavior based on the edge algorithm being used. Here is the polygon definition for the intersection shape: ```sql POLYGON ((-0.1 66.41326, 90.1 66.41326, 90.1 85.15113, -0.1 85.15113, -0.1 66.41326)) ``` In the first image, the two polygons intersect. That is because the viewer is using "plate carrée" projection edges. ![Two polygons that intersect. The first is the outline of the country of Sweden and the second is a square that crosses the northern half of Sweden.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2ad7f7c1cff1ae72/6a7adb7bde231548f6fcc4f8/geo_plate_edge2-4FnwSA.png) In this next view, the polygons do not intersect. The viewer is using geodesic edges: ![Two polygons that do not intersect. One is the outline of the country of Sweden and the second is a trapezoid shape that passes north of the first polygon.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt89ba6426b17ee85a/6a7ada84e88c6552b4fffdcd/geo_geodesic_edge-bswz3p.png) Because SingleStore uses geodesic edges, there would not be an intersection between the two polygons. ## Allowed Numerical Values Values allowed for geospatial types are integers and floating point values. Scientific notation (e.g. 1.2e-12) is not allowed for values of geospatial types. ## Creating a Geospatial Table Geospatial objects are just another data type in SingleStore and can be used right alongside numbers, strings, and JSON. To create a column that holds point data, use the `GeographyPoint` type. For path and polygon data, use the `Geography` type. > **❗ Important**: It is possible to store POINTs in the `Geography` type, but is not recommended. The `GeographyPoint` type was designed specifically for point data, and is highly optimized for memory efficiency and speed. In this example, we’ll create a table of neighborhoods. The “centroid (A point used in clustering algorithms to represent the center or mean of a cluster, a group of vectors near each other.)” field is the point that is roughly in the middle of the borders of the neighborhood. The “shape” field is the polygon describing those borders. All geospatial data can be indexed for very high query performance. ```sql CREATE ROWSTORE TABLE neighborhoods ( id INT UNSIGNED NOT NULL PRIMARY KEY, name VARCHAR(64) NOT NULL, population INT UNSIGNED NOT NULL, shape GEOGRAPHY NOT NULL, centroid GEOGRAPHYPOINT NOT NULL, index (shape) WITH (resolution = 8), index (centroid) ); INSERT INTO neighborhoods VALUES (1, "Hamilton", 12346, "POLYGON((1 1,2 1,2 2, 1 2, 1 1))","POINT(1.5 1.5)"), (2, "Probability Bay", 263951, "POLYGON((5 1,6 1,6 2,5 2,5 1))", "POINT(5.5 1.5)"), (3, "Decentralized Park", 29265, "POLYGON((5 5,6 5,6 6,5 6,5 5))", "POINT(5.5 5.5)"), (4, "Axiom Township", 845696, "POLYGON((1 5,2 5,2 6,1 6,1 5))", "POINT(1.5 5.5)"), (5, "Elegant Island ", 987654, "POLYGON((3 3,4 3,4 4,3 4,3 3))", "POINT(3.5 3.5)"); ``` ## Querying Geospatial Data ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltaa5c1f1036681e56/6a7adbb59f5251f68565e92a/geospatial_structures-7kHoRJ.png) SingleStore provides a SQL function for each of these types of queries: * Area ([GEOGRAPHY\_AREA](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-area.md)) * Distance ([GEOGRAPHY\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-distance.md)) * Length ([GEOGRAPHY\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-length.md)) * Contains ([GEOGRAPHY\_CONTAINS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-contains.md)) * Intersects ([GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-intersects.md)) * Approx. Intersects ([APPROX\_GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/approx-geography-intersects.md)) * Within Distance ([GEOGRAPHY\_WITHIN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-within-distance.md)) You can run queries over the neighborhood data to see how they relate to each other spatially. For example, you can join the table to itself and measure distances between neighborhoods: ```sql SELECT b.name AS town, ROUND(GEOGRAPHY_DISTANCE(a.centroid, b.centroid), 0) AS distance_from_center, ROUND(GEOGRAPHY_DISTANCE(a.shape, b.shape), 0) AS distance_from_border FROM neighborhoods a, neighborhoods b WHERE a.id = 2 ORDER BY 2; ``` ```output +--------------------+----------------------+----------------------+ | town | distance_from_center | distance_from_border | +--------------------+----------------------+----------------------+ | Probability Bay | 0 | 0 | | Elegant Island | 314174 | 157090 | | Hamilton | 444379 | 333195 | | Decentralized Park | 444531 | 333399 | | Axiom Township | 628012 | 471030 | +--------------------+----------------------+----------------------+ 5 rows in set (0.00 sec) ``` > **❗ Important**: Distance is calculated using the standard metric for [distance on a sphere](https://en.wikipedia.org/wiki/Great-circle_distance). You can also find out where you are with `GEOGRAPHY_INTERSECTS`: ```sql SELECT id, name FROM neighborhoods WHERE GEOGRAPHY_INTERSECTS("POINT(2 1)", shape); ``` ```output +----+----------+ | id | name | +----+----------+ | 1 | Hamilton | +----+----------+ 1 row in set (0.00 sec) ``` ## Spatial Joins Geospatial objects are first-class data types in SingleStore, which means that you can use spatial relationships to join tables. For example, to find all of the businesses in your neighborhood, you join the businesses table with the neighborhoods table on whether the business location (a point) is contained by the neighborhood's shape (a polygon). Create the table: ```sql CREATE ROWSTORE TABLE geo_businesses ( id INT UNSIGNED NOT NULL PRIMARY KEY, name VARCHAR(64) NOT NULL, revenue INT UNSIGNED NOT NULL, location GEOGRAPHYPOINT NOT NULL, INDEX (location) ); INSERT INTO geo_businesses VALUES (1,"Touring Machines Bike Repair", 10000, "POINT(1.25 1.25)"), (2,"Manhattan Boat Club", 20000, "POINT(5.75 5.25)"), (3,"The Philospher's Diner", 12345, "POINT(1.25 5.25)"), (4,"Push & Pop Cafe", 23456, "POINT(3.25 3.75)"), (5,"Lil Mac's Subs", 76543, "POINT(5.75 5.75)"), (6,"Touring Machines Bike Repair", 21012, "POINT(5.25 1.75)"), (7,"Random Slice Pizza", 11111, "POINT(1.75 1.75)") ; ``` Find businesses contained within a specific neighborhood: ```sql SELECT b.id, b.name FROM geo_businesses b, neighborhoods n WHERE n.name="Elegant Island" AND GEOGRAPHY_CONTAINS(n.shape, b.location); ``` ```output +----+-----------------+ | id | name | +----+-----------------+ | 4 | Push & Pop Cafe | +----+-----------------+ ``` ```sql SELECT b.id, b.name FROM geo_businesses b, neighborhoods n WHERE n.name="Hamilton" AND GEOGRAPHY_CONTAINS(n.shape, b.location) ORDER BY 1; ``` ```output +----+------------------------------+ | id | name | +----+------------------------------+ | 1 | Touring Machines Bike Repair | | 7 | Random Slice Pizza | +----+------------------------------+ ``` These functions and many more are documented in [Geospatial Functions](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions.md). ## Geospatial Model SingleStore uses a spherical model similar to that used in Google Earth. It assumes a perfectly spherical Earth with a radius of 6,367,444.66 meters. The real Earth is not a sphere. It resembles an ellipsoid slightly wider than it is tall, with many local variations. In a spherical model, measurements of distance and area may vary from reality. For example, a 10,000 meter line drawn vertically over San Francisco would measure about 10,020 meters in this model. Topological functions, which describe the relationships between shapes and points, are less affected by this difference. ## Columnstore Geospatial In SingleStore, geospatial data and functions are supported in columnstore (i.e. on-disk) tables. While the native `GeographyPoint` type is supported, the `Geography` type is not. Columnstore tables do not support indexes on geo data. However, you can store path and polygon data in a plain text field, and run geospatial functions over them just as you do with rowstore (in-memory) tables. ```sql CREATE TABLE neighborhoods_colstore( id INT UNSIGNED NOT NULL, name VARCHAR(64) NOT NULL, population INT UNSIGNED NOT NULL, shape TEXT NOT NULL, centroid GEOGRAPHYPOINT NOT NULL, sort key (name), shard key (id)); INSERT INTO neighborhoods_colstore VALUES (1, "Hamilton", 12346, "POLYGON((1 1,2 1,2 2, 1 2, 1 1))","POINT(1.5 1.5)"), (2, "Probability Bay", 263951, "POLYGON((5 1,6 1,6 2,5 2,5 1))", "POINT(5.5 1.5)"), (3, "Decentralized Park", 29265, "POLYGON((5 5,6 5,6 6,5 6,5 5))", "POINT(5.5 5.5)"), (4, "Axiom Township", 845696, "POLYGON((1 5,2 5,2 6,1 6,1 5))", "POINT(1.5 5.5)"), (5, "Elegant Island ", 987654, "POLYGON((3 3,4 3,4 4,3 4,3 3))", "POINT(3.5 3.5)"); ``` ```sql SELECT id, name FROM neighborhoods_colstore WHERE GEOGRAPHY_INTERSECTS("POINT(2 1 )", shape); ``` ```output +----+----------+ | id | name | +----+----------+ | 1 | Hamilton | +----+----------+ ``` ## Special Spatial Tuning Parameters Spatial indices on `Geography` columns may be given an optional `RESOLUTION` parameter. You can choose a resolution between 6 and 32. This controls how many pieces the polygons and linestrings are broken up into for indexing. As a general rule, a lower number means less memory consumption, faster inserts and updates, but slower query times. A higher number may increase query performance at a cost of memory and insert performance. > **❗ Important**: These tuning parameters should only be adjusted if you have a *measurable need* for more performance and have worked through the tradeoffs between ingest, computational load, and memory consumption.For example, it may be useful to compare the percentage difference in the number of rows returned by [GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-intersects.md) and [APPROX\_GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/approx-geography-intersects.md).“This difference is the”false positive" rate of the index for a given resolution. For example, at resolution 16, suppose that 10% more records are returned by the approximate function. At resolution 8, the rate only grows to 11%. In that case you should be able to save memory and increase ingest throughput by using the lower number, without significantly harming query performance. Again, this kind of tuning is highly dependent on your particular dataset and workload. The parameters obtained for one application may not apply to another. Your mileage may vary. You can also change the resolution of polygons passed into geospatial functions at query time. In this example, the resolution of the “shape” column is unchanged, but the POLYGON in the second argument is broken into 16 parts. Each part is queried separately. A lower number will result in fewer index lookup operations, but may return more false positive candidates from the index which are then filtered out of the result set in the second phase of query execution. ```sql SELECT * FROM neighborhoods WITH (index = shape, resolution = 16) WHERE geography_intersects(shape,"POLYGON(...)"); ``` ## Using GeoJSON Data SingleStore does not have native GeoJSON support. However, we have generic JSON support as well as computed columns. Combining these features, you can import a subset of GeoJSON data using SingleStore's built-in JSON type and `GEOGRAPHYPOINT` type. For example, given a set of GeoJSON points like this: ```json { "type": "Feature", "geometry": { "type": "Point", "coordinates": [123.456, 78.901] }, "properties": { "id": 123456, "name": "Probability Bay Naval Observatory" } } ``` You can construct a table like this: ```sql CREATE ROWSTORE TABLE geostuff ( id AS dataz::properties::%id persisted BIGINT unsigned, name AS dataz::properties::$name persisted VARCHAR(128), location AS geography_point( dataz::geometry::coordinates::%`0`, dataz::geometry::coordinates::%`1` ) persisted geographypoint, dataz JSON NOT NULL, index (location) ); ``` And then insert the JSON string: ```sql INSERT INTO geostuff (dataz) VALUES ('{"type": "Feature", "geometry": {"type": "Point","coordinates": [123.456, 78.901]},"properties": {"id": 123456, "name": "Probability Bay Naval Observatory" }}'); ``` And query: ```sql SELECT id, name, location FROM geostuff; ``` ```output +--------+-----------------------------------+---------------------------------+ | id | name | location | +--------+-----------------------------------+---------------------------------+ | 123456 | Probability Bay Naval Observatory | POINT(123.45600012 78.90100000) | +--------+-----------------------------------+---------------------------------+ ``` *** Modified at: July 2, 2025 Source: [/cloud/developer-resources/functional-extensions/working-with-geospatial-features/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-geospatial-features/) (An index of the documentation is available at /llms.txt) --- # Working with Vector DataSingleStore supports vector database processing, which allows you to store and search vector data. A typical vector search locates the set of vectors that most closely match a query vector. Vectors usually come from objects: text, images, video, audio, etc. Vector database searches find data based on its content or meaning, even without exact matches. For example, vector search can allow a [semantic search of text](https://www.singlestore.com/blog/ai-powered-semantic-search-in-singlestoredb-/), where a query about "meals" could return information about "lunch" and "dinner" without using those words because they are similar in meaning.SingleStore supports a native vector data type and indexed approximate-nearest-neighbor (ANN) search that provide high-performance vector search and easier building of vector-based applications. See [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md).Some benefits of using SingleStore for vector database processing, as opposed to a specialized vector database system, are:- A broad array of standard modern database capabilities are available in SingleStore. These include SQL, fast distributed and parallel query processing, full-text search, extensibility, ACID transactions, high availability, disaster recovery, point-in-time recovery, broad connectivity support, etc. - Less data movement is needed between different data subsystems (e.g., caches, text search systems, SQL databases, and specialized vector databases) when all the data, including vector data, is stored in SingleStore. - Operational costs may be reduced since fewer data management tools and fewer copies of the data are needed. - Less specialized skills and reduced labor may be needed to run an application environment.## Vector DataVector data consists of arrays of numbers. Vector data can be stored in SingleStore using the native [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md). Vector embeddings, which are vectors describing the meaning of objects, can be obtained from many sources. For example, [OpenAI](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings) has APIs that will give vector embeddings for text, and [facenet](https://github.com/davidsandberg/facenet) is an open-source software package with a pre-trained neural network that can provide embeddings for face images.Many large language models (LLMs) are now available and can be used to provide vector embeddings for language to help implement semantic search, chatbots, and other applications. Some of the more well-known LLMs are:- GPT models from OpenAI - BERT by Google - LaMDA by Google - PaLM by Google - LLaMA by Meta AILLM technology is evolving quickly, and new sources of embeddings for language are rapidly becoming available. Moreover, it is possible to train your own models that produce embeddings for database information. Sources for models include [Ollama](https://ollama.com/), [voyageAI](https://www.voyageai.com), and [huggingface](https://huggingface.co/spaces/mteb/leaderboard). Regardless of the embedding source, these embeddings can be used for vector similarity search in SingleStore.Vector embeddings can be stored as vector data in SingleStore. Vector embeddings are typically high-dimensional, with anywhere from a hundred to a few thousand dimensions.## Vector Similarity SearchA similarity search is the most common vector data operation. A vector similarity search in SingleStore uses `SELECT…ORDER BY…LIMIT`… queries that use vector similarity functions, including [DOT\_PRODUCT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/dot-product.md) and [EUCLIDEAN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/euclidean-distance.md). `DOT_PRODUCT` is the most commonly used similarity metric.If the vectors are normalized to length one before saving to a database, and the query vector is also normalized to length one, then `DOT_PRODUCT` gives the cosine of the angle between the two vectors. `DOT_PRODUCT` will produce what is known as the cosine similarity metric for its two arguments. If the cosine is close to one, then the angle between the vectors is close to zero. A cosine value closer to one shows that the vectors are pointing in almost the same direction, so the objects they represent are similar. Cosine similarity values range from 1 (vectors are similar) to -1 (vectors are dissimilar).The figure below illustrates when vectors are similar (cosine close to 1) and dissimilar (cosine close to 0): ![Vector angles close to 1 and 0](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte47f6c43d2400d40/6a7ada4096b5a6bc5a86fd8c/vector_angles-a2wJpX.png)> **📝 Note**: When using high-dimensional vectors the cosine similarity concept is still applicable.Many vector models that produce vector embeddings will already have the vectors normalized to length one. In this case, when using `DOT_PRODUCT`, it is not necessary to normalize the vectors again.Vector indexes can be used to improve performance of search over vectors. ANN searches give a fast, approximate answer to nearest neighbor queries. If exact answers are required, an exact k-Nearest Neighbor (KNN) search can be performed simply by not using a vector index. See [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for information on ANN search.The `EUCLIDEAN_DISTANCE` function calculates the Euclidean distance between two vectors. Euclidean distance measures the distance between vectors taking into account the length of the vectors. The results of `EUCLIDEAN_DISTANCE` range from 0 to infinity; values closer to 0 indicate that the vectors are similar, values closer to infinity indicate that vectors are less similar.## Loading, Inserting, and Updating VectorsVector data can be added to a database using the `VECTOR` data type. For example, using this table format:```sql CREATE TABLE comments(id INT, comment TEXT, comment_embedding VECTOR(4), category VARCHAR(256)); ```The default element type for `VECTOR` is a 32-bit floating point number (`F32`). The supported element types are `I8`, `I16`, `I32`, `I64`, `F32`, and `F64`. Refer to [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) for more information.## Bulk Loading Binary VectorsSingleStore recommends loading large vector data sets using a binary format such as [Apache Parquet](https://parquet.apache.org/) or [Apache Avro™](https://avro.apache.org/) with [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) and [PIPELINES](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md). Binary formats have advantages including faster load speed, smaller input file size, and no loss of precision. For high performance, using a binary format such as Parquet, Avro is necessary.```sql CREATE TABLE comments(id INT, comment TEXT, comment_embedding VECTOR(4), category VARCHAR(256)); ```[How to Bulk Load Vectors](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/how-to-bulk-load-vectors.md) provides information on generating and loading large vector data sets using binary formats using `LOAD DATA` and `PIPELINES`.Binary vectors can also be loaded through a standard client API. Consult the corresponding API documentation for details.Finally, SingleStore supports input and output of vectors in JSON array of numbers format. This format is convenient and user-readable but has performance and precision impacts.## Initial ExamplesThis section provides examples of loading vector data using JSON and hexadecimal formats. These formats are used for readability. SingleStore recommends using binary vector formats for production workloads.Consider the following information, where `@emb` is a JSON array of numbers representing the vector embedding for the phrase "The cafeteria in building 35 has a great salad bar." Fictional vectors are used here to keep the examples small enough to read easily.```sql SET @_id = 1; SET @cmt = "The cafeteria in building 35 has a great salad bar"; SET @emb = '[0.45, 0.55, 0.495, 0.5]'; SET @cat = "Food"; ```This data can be inserted into the comments table with a standard `INSERT` statement:```sql INSERT INTO comments VALUES (@_id, @cmt, @emb, @cat); ```If you have vector data that is already in binary format, that data should be inserted into the database in binary format. Converting such data to JSON array of numbers format and inserting that data into the database will result in significantly reduced performance.In addition, it is important to convert vectors into binary vector format before inserting them into a database. Storing vectors as strings and converting them to binary during querying will deliver much slower performance and is not recommended.Binary data may be inserted in hexadecimal string format and converted using `UNHEX` before inserting it in a table. For example, suppose that client application software produces this hex string `B806363CE90A363CCDCCCC3D77BE7F3F` for the vector `'[0.01111, 0.011111, 0.1, 0.999]'`.`UNHEX` can be used to convert a hex string that will work for insert into the database. For example:```sql SET @_id = 2; SET @cmt = "I love the taco bar in the B16 cafeteria."; SET @hs = "B806363CE90A363CCDCCCC3D77BE7F3F"; SET @cat = "Food"; ``````sql INSERT INTO comments VALUES (@_id, @cmt, UNHEX(@hs), @cat); ```Insert a third tuple into this table.```sql SET @_id = 3; SET @cmt = "The B24 restaurant salad bar is quite good."; SET @emb = '[0.1, 0.8, 0.2, 0.555]'; SET @cat = "Food"; ``````sql INSERT INTO comments VALUES (@_id, @cmt, @emb, @cat); ```View the contents of the table using this query.```sql SET vector_type_project_format = JSON; /* to make vector output readable */ SELECT id, comment, comment_embedding, category FROM comments ORDER BY id; ``````output *** 1. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar comment_embedding: [0.449999988,0.550000012,0.495000005,0.5] category: Food *** 2. row *** id: 2 comment: I love the taco bar in the B16 cafeteria. comment_embedding: [0.0111100003,0.0111109996,0.100000001,0.999000013] category: Food *** 3. row *** id: 3 comment: The B24 restaurant salad bar is quite good. comment_embedding: [0.100000001,0.800000012,0.200000003,0.555000007] category: Food ```If you are using a command line tool, you can use `\G` at the end of the query to get the results formatted as above.## Output Format for ExamplesVectors can be output in JSON or binary format. Use JSON format for examples and for output readability. For production, use the default binary for efficiency.Use the following command to output vectors in JSON format.```sql SET vector_type_project_format = JSON; ```Use the following command to set the output format back to binary.```sql SET vector_type_project_format = BINARY; ```## Example Search Based on Vector SimilarityTo find the most similar vectors in a query vector, use an `ORDER BY… LIMIT`… query. `ORDER BY` will arrange the vectors by their similarity score produced by a vector similarity function, with the closest matches at the top.Suppose that the query is "restaurants with good salad," and for this query, the vector embedding API returned the vector '\[0.44, 0.554, 0.34, 0.62]'. To find the top two matches for this query vector use the following SQL.In this query, the infix operator `<*>` represents `DOT_PRODUCT`. The infix operator `<->` can be used for `EUCLIDEAN_DISTANCE`.The `@query_vec` variable is cast to a `VECTOR` to ensure that `@query_vec` is a valid `VECTOR` and to improve performance.```sql SET @query_vec = ('[0.44, 0.554, 0.34, 0.62]'):>VECTOR(4); SELECT id, comment, category, comment_embedding <*> @query_vec AS score FROM comments ORDER BY score DESC LIMIT 2; ``````output *** 1. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar category: Food score: 0.9810000061988831 *** 2. row *** id: 3 comment: The B24 restaurant salad bar is quite good. category: Food score: 0.8993000388145447 ```- Search queries must use the same metric used in the creation of a vector index in order to use that index. That is, a query that uses `DOT_PRODUCT` can only use a vector index created with `DOT_PRODUCT`. - Vectors must be normalized to length 1 before using the `DOT_PRODUCT` function to obtain the cosine similarity metric. SingleStore recommends vectors be normalized to length 1 before they are saved to the database. Many models that produce vector embeddings produce vectors normalized to length one. In this case, it is not necessary to normalize the vectors again. - Using `EUCLIDEAN_DISTANCE` over vectors which have been normalized to length 1 may result in poor recall.## Hybrid Filtering or Metadata FilteringWhen building vector search applications, you may wish to filter on the fields of a record, with simple filters or via joins, in addition to applying vector similarity operations. Filtering on fields of a record using simple filters or joins along with applying vector similarity operations is often referred to as hybrid filtering or metadata filtering. SingleStore can handle this filtering using standard SQL operations.For example, given the `comments` table, you can get the top three matches for a query vector that is in the category "Food" using this SQL:```sql SET @query_vec = ('[0.44, 0.554, 0.34, 0.62]'):>VECTOR(4); SELECT id, comment, category, comment_embedding <*> @query_vec AS score FROM comments WHERE category = "Food" ORDER BY score DESC LIMIT 3; ``````output *** 1. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar category: Food score: 0.9810000061988831 *** 2. row *** id: 3 comment: The B24 restaurant salad bar is quite good. category: Food score: 0.8993000388145447 *** 3. row *** id: 2 comment: I love the taco bar in the B16 cafeteria. category: Food score: 0.6644238829612732 ```Any SQL feature can be used along with vector similarity calculation using `DOT_PRODUCT()` and `EUCLIDEAN_DISTANCE()`. These include filtering, ordering, grouping, aggregation, window functions, full-text search, and more.## Create and Use a Vector IndexThe command below creates a vector index on the `comment_embedding` field of the `comments` table. Vector indexes can be used to improve performance of queries over large vector data sets. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for information on creating and using vector indexes.```sql ALTER TABLE comments ADD VECTOR INDEX ivf(comment_embedding) INDEX_OPTIONS '{"index_type":"IVF_PQFS"}'; ```Optionally optimize the table for best performance.```sql OPTIMIZE TABLE comments FLUSH; ```The following query will use the vector index.```sql SET @query_vec = ('[0.44, 0.554, 0.34, 0.62]'):>VECTOR(4); SELECT id, comment, category, comment_embedding <*> @query_vec AS score FROM comments ORDER BY score DESC LIMIT 2; ``````output *** 1. row *** id: 1 comment: The cafeteria in building 35 has a great salad bar category: Food score: 0.9810000061988831 *** 2. row *** id: 3 comment: The B24 restaurant salad bar is quite good. category: Food score: 0.8993000388145447 ```The `EXPLAIN` command can be used to see the query plan and verify that the vector index is being used. In the example below, you can see `INTERNAL_VECTOR_SEARCH` in the `ColumnStoreFilter` row. This tells you that the vector index is being used.```sql SET @query_vec = ('[0.09, 0.14, 0.5, 0.05]'):>VECTOR(4); EXPLAIN SELECT id, comment, category, comment_embedding <*> @query_vec AS score FROM comments ORDER BY score DESC LIMIT 2; ``````output +--------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------+ | Project [remote_0.id, remote_0.comment, remote_0.category, remote_0.score] | | TopSort limit:2 [remote_0.score DESC] | | Gather partitions:all alias:remote_0 parallelism_level:segment | | Project [comments.id, comments.comment, comments.category, | | DOT_PRODUCT(comments.comment_embedding,(@query_vec:>vector(4, F32))) AS score] | | TopSort limit:2 [DOT_PRODUCT(comments.comment_embedding,(@query_vec:>vector(4, F32))) DESC] | | ColumnStoreFilter [INTERNAL_VECTOR_SEARCH(0, (@query_vec:>vector(4, F32)), 2, '') index] | | ColumnStoreScan test.comments, SORT KEY __UNORDERED () table_type:sharded_columnstore | +--------------------------------------------------------------------------------------------------------+ ```## Hybrid SearchHybrid Search combines multiple search methods in one query and blends full-text search (which finds keyword matches) and vector search (which finds semantic matches) allowing search results to be (re-)ranked by a score that combines full-text and vector rankings. [Hybrid Search - Reranking Full-Text and Vector Search Results](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md) provides two examples of hybrid search including an example of indexed hybrid search using SingleStore's [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) and full-text indexing ([Working with Full-Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md)). See also this [blog post on Hybrid Search](https://www.singlestore.com/blog/hybrid-search-vector-full-text-search/).## Retrieval-Augmented Generation (RAG)Retrieval-Augmented Generation (RAG) is a method for enhancing the quality of results for text-based Generative AI (GenAI) applications. To do RAG with SingleStore:In advance:- split up relevant text into chunks - get a vector for each chunk from your chosen LLM and place the chunks in SingleStoreThen, when a question comes in for which you wish to generate an answer:- get a vector for the question from the same LLM - search the stored vectors to find the top k matching vectors compared with the query vector - for each of these top vectors, get the associated text chunks - pass these chunks to the LLM’s question-answering API as context, along with the original questionThis method helps the LLM generate high-quality results for a specialized topic, beyond what it was trained for. RAG is a recommended approach for use with SingleStore to generate high-quality answers via your LLM for a specific situation or context.Tools such as Langchain, or a Natural Language Processing (NLP) library like spaCy or NLTK can be used to make splitting documents into chunks take less time for the application developer.## Tracking Vector Index Memory UseVector index memory use is managed by the vector index cache and can be tracked using [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md). Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing/#section-idm234597445043844.md) for more information.## Related Topics- Blog Post: [Why Your Vector Database Should Not Be a Vector Database](https://www.singlestore.com/blog/why-your-vector-database-should-not-be-a-vector-database/) - [Announcing SingleStore Indexed ANN Vector Search](https://www.singlestore.com/blog/singlestore-indexed-ann-vector-search/) - [How to Bulk Load Vectors](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/how-to-bulk-load-vectors.md) - [Hybrid Search - Reranking Full-Text and Vector Search Results](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md) - [Tuning Vector Indexes and Queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries.md) - [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) - [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md)***Modified at: August 5, 2026Source: [/cloud/developer-resources/functional-extensions/working-with-vector-data/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data/)(An index of the documentation is available at /llms.txt) --- # Working with Window Functions Window functions perform ranking and aggregate calculations across sets of rows that bear some relation to the current row. Unlike aggregates, which also act on multiple rows, window functions return a result for every row, instead of aggregating values from all rows into a single value. ## Syntax Window functions in SingleStore use the following syntax: ``` function ([expr]) OVER {window_name | (window_definition)} window_definition: [partition_by_clause] [order_by_clause] [frame_clause] partition_by_clause: PARTITION BY {col | expr} order_by_clause: ORDER BY {col | expr} frame_clause: {ROWS | RANGE} {UNBOUNDED PRECEDING | int PRECEDING | CURRENT ROW} | {BETWEEN {UNBOUNDED PRECEDING | int {PRECEDING | FOLLOWING} | CURRENT ROW} AND {UNBOUNDED FOLLOWING | int {PRECEDING | FOLLOWING} | CURRENT ROW} } ``` ## Syntax Remarks ## window\_definition Window definition clauses are similar to standard `GROUP BY` and `ORDER BY` clauses with the following exception: you cannot refer to the projections list using aliases or position, because literal integers are treated as integers instead of positional references. ## frame\_clause The `frame_clause` is used with an aggregate window function, and it specifies a subset of rows in a partition to evaluate. Ranking, value, and percentile window functions are not affected by the `frame_clause` because they are intended to operate on the entire set of rows in each partition. In the `frame_clause`, `int PRECEDING` and `int FOLLOWING` are referred to as frame offsets. `RANGE` cannot be used with frame offsets. Additionally, frame offsets must be non-negative integers. Any non-integer is automatically rounded. If `BETWEEN` is not specified, the default upper bound is `CURRENT ROW`. For example, `ROWS UNBOUNDED PRECEDING` is equivalent to `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. If no frame is specified, the default is `RANGE UNBOUNDED PRECEDING`, i.e. `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. `RANGE` includes all rows whose elements in the `order_by_clause` are equal. If the `order_by_clause` is empty, `RANGE` will include all rows in the partition. The `order_by_clause` can include optional `NULLS FIRST` or `NULLS LAST` clauses that sort NULL values to the beginning or end of the results set. Ascending order is assumed for the non-NULL values. `DESC` can be added to the clause to order the non-NULL values in descending order. `NULLS FIRST` and `NULLS LAST` clauses are supported by all window functions. Consider the following examples: ```sql CREATE TABLE wf_t (a INT); INSERT INTO wf_t VALUES (1),(2),(2),(3),(3),(3),(4); ``` ```sql SELECT COUNT(1) OVER (ORDER BY a ROWS UNBOUNDED PRECEDING) AS col1, COUNT(1) OVER (ORDER BY a RANGE UNBOUNDED PRECEDING) AS col2 FROM wf_t; ``` ```output +------+------+ | col1 | col2 | +------+------+ | 1 | 1 | | 2 | 3 | | 3 | 3 | | 4 | 6 | | 5 | 6 | | 6 | 6 | | 7 | 7 | +------+------+ ``` ```sql INSERT INTO wf_t VALUES (null), (null); ``` ```sql SELECT a, RANK() OVER (ORDER BY a NULLS FIRST) AS col1 FROM wf_t; ``` ```output +------+------+ | a | col1 | +------+------+ | NULL | 1 | | NULL | 1 | | 1 | 3 | | 2 | 4 | | 2 | 4 | | 3 | 6 | | 3 | 6 | | 3 | 6 | | 4 | 9 | +------+------+ ``` ```sql SELECT a, RANK() OVER (ORDER BY a NULLS LAST) AS col1 FROM wf_t; ``` ```output +------+------+ | a | col1 | +------+------+ | 1 | 1 | | 2 | 2 | | 2 | 2 | | 3 | 4 | | 3 | 4 | | 3 | 4 | | 4 | 7 | | NULL | 8 | | NULL | 8 | +------+------+ ``` ```sql SELECT a, RANK() OVER (ORDER BY a DESC NULLS FIRST) AS col1 FROM wf_t; ``` ```output +------+------+ | a | col1 | +------+------+ | NULL | 1 | | NULL | 1 | | 4 | 3 | | 3 | 4 | | 3 | 4 | | 3 | 4 | | 2 | 7 | | 2 | 7 | | 1 | 9 | +------+------+ ``` ```sql SELECT a, RANK() OVER (ORDER BY a DESC NULLS LAST) AS col1 FROM wf_t; ``` ```output +------+------+ | a | col1 | +------+------+ | 4 | 1 | | 3 | 2 | | 3 | 2 | | 3 | 2 | | 2 | 5 | | 2 | 5 | | 1 | 7 | | NULL | 8 | | NULL | 8 | +------+------+ ``` ## Supported Functions The following built-in functions are supported in a window function: Ranking functions: * [RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/rank.md) * [DENSE\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/dense-rank.md) * [ROW\_NUMBER](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/row-number.md) * [NTILE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/ntile.md) * [PERCENT\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percent-rank.md) Value functions: * [LAG](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lag.md) * [LEAD](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lead.md) * [FIRST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/first-value.md) * [NTH\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/nth-value.md) * [LAST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/last-value.md) Aggregate functions: * [AVG](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/avg.md) [COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/count.md) * [MIN](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/min.md) * [MAX](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/max.md) * [STD](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/std.md) * [STDDEV\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev-samp.md) * [SUM](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/sum.md) * [VAR\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/var-samp.md) * [VARIANCE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/variance.md) Percentile functions: * [PERCENTILE\_DISC](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-disc.md) * [PERCENTILE\_CONT and MEDIAN](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-cont-and-median.md) ## Named Window Definition `window_name` is a reference to a named window definition given in a `WINDOW` clause, such as: ```sql SELECT RANK() OVER window_name FROM table WINDOW window_name AS (window_definition) ``` If using multiple window functions on the same window frame, you may see improved performance by setting and using a window\_name. Using the same window\_name across multiple functions allows SingleStore Helios code generation to make execution more efficient. Consider the following examples. The statement below does not use a `window_name` for two identical window definitions. ```sql SELECT website, timestamp, AVG(clicks) OVER (PARTITION BY website ORDER BY timestamp ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING), SUM(visits) OVER (PARTITION BY website ORDER BY timestamp ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING) FROM t; ``` The statement above can be improved by using `window_name` syntax, which will improve performance: ```sql SELECT website, timestamp, AVG(clicks) OVER w, SUM(visits) OVER w FROM t WINDOW w AS (PARTITION BY website ORDER BY timestamp ROWS BETWEEN 3 PRECEDING AND 3 FOLLOWING); ``` The second version of the query has fewer parameters, so the generated code can be smaller and more efficient. ## Alternatives to Common Window Function Patterns The following functions can be used instead of common window function patterns: * Use `FIRST()` and `LAST()` instead of `FIRST_VALUE()` and `LAST_VALUE()` when aggregating time series data with `GROUP BY`. These functions return the first or last value by timestamp without the need for `OVER()`, frame clauses, or `ORDER BY` within a window. They use the `SERIES TIMESTAMP` column implicitly. * Use `TIME_BUCKET()` instead of manual time bucket arithmetic. For example, replace `FLOOR(UNIX_TIMESTAMP(ts)/300)*300` with `TIME_BUCKET('5m', ts)`. * Use `APPROX_PERCENTILE()` instead of `PERCENTILE_CONT` and `PERCENTILE_DISC` to compute percentile values more efficiently. Results are approximate but significantly faster on large datasets. Refer to [Analyzing Time Series Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/analyzing-time-series-data.md) for more information on time series functions. ## Examples In the following simple examples, the table `t` contains four uniquely identified values divided into two categories. ```sql CREATE TABLE wft (id int, category char(1), value int); INSERT INTO wft VALUES (1, 'a', 1), (2, 'a', 3), (3, 'b', 2), (4, 'b', 4); SELECT * FROM wft ORDER BY id; ``` ```output +----+----------+-------+ | id | category | value | +----+----------+-------+ | 1 | a | 1 | | 2 | a | 3 | | 3 | b | 2 | | 4 | b | 4 | +----+----------+-------+ ``` Selecting the average of a column with `GROUP BY` performs an aggregation that returns a single row per group. Selecting (projecting over) `id` would be meaningless in this case – SingleStore Helios will return values from an arbitrary row. ```sql SELECT category, avg(value) FROM wft GROUP BY category; ``` ```output +----------+------------+ | category | avg(value) | +----------+------------+ | a | 2.0000 | | b | 3.0000 | +----------+------------+ ``` In contrast, using `avg` as a window function with `OVER` and partitioning by category will compute the average value for each partition of category, then append that category-wise average to each row. ```sql SELECT id, category, value, avg(value) over (partition by category) FROM wft; ``` ```output +----+----------+-------+-----------------------------------------+ | id | category | value | avg(value) over (partition by category) | +----+----------+-------+-----------------------------------------+ | 2 | a | 3 | 2.0000 | | 1 | a | 1 | 2.0000 | | 4 | b | 4 | 3.0000 | | 3 | b | 2 | 3.0000 | +----+----------+-------+-----------------------------------------+ ``` Here is another example where we order by `value` within each partition, and use the `rank()` window function to produce the rank of each row within its partition. ```sql SELECT id, category, value, rank() OVER (partition by category order by value) AS rank FROM wft; ``` ```output +------+----------+-------+------+ | id | category | value | rank | +------+----------+-------+------+ | 1 | a | 1 | 1 | | 2 | a | 3 | 2 | | 3 | b | 2 | 1 | | 4 | b | 4 | 2 | +------+----------+-------+------+ ``` *** Modified at: July 14, 2026 Source: [/cloud/developer-resources/functional-extensions/working-with-window-functions/](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-window-functions/) (An index of the documentation is available at /llms.txt) --- # Guides Solution guides for some common tasks with the SingleStore Helios. ## In this section * [Augmenting Your Data Warehouse to Accelerate BI](https://docs.singlestore.com/cloud/developer-resources/guides/augmenting-your-data-warehouse-to-accelerate-bi.md) * [Building a New Application](https://docs.singlestore.com/cloud/developer-resources/guides/building-a-new-application.md) * [Designing for Multi-Tenant Applications](https://docs.singlestore.com/cloud/developer-resources/guides/designing-for-multi-tenant-applications.md) * [Migrating Data from Another Database](https://docs.singlestore.com/cloud/developer-resources/guides/migrating-data-from-another-database.md) *** Modified at: July 19, 2022 Source: [/cloud/developer-resources/guides/](https://docs.singlestore.com/cloud/developer-resources/guides/) (An index of the documentation is available at /llms.txt) --- # Augmenting Your Data Warehouse to Accelerate BI Welcome to SingleStore Helios! We put together a guide to help you augment your data warehouse by migrating some of your BI workloads to SingleStore. Use this guide to follow a top-down approach to: * Preparing your data from your existing data warehouse * Interacting with SingleStore * Designing your SingleStore tables to accelerate your dashboards * Incorporating streaming data with SingleStore * Optimizing your queries for your dashboards * Connecting your dashboard to SingleStore ## Tutorial Outline This tutorial is divided into the following sections: **Getting Started** * Benefits of leveraging SingleStore Helios for your BI dashboard * Why users leverage SingleStore to augment their existing data warehouse technologies * What to consider when bringing your data from: * Snowflake, Redshift, BigQuery, Synapse * AWS S3, Google CS, Azure, Kafka * What to consider when designing your database coming from: * MySQL, MariaDB, Postgres **Sample Architecture** **Preparing Your Data for SingleStore** **Database Administration** * Cloud Portal * Drivers * MySQL Workbench * MySQL Command Line **Designing Your Tables** * Data types * Designing a schema for an application * Sharding for application workloads * Sort keys for application workloads * Reference Tables **Data Ingest** * How this may be different from ingesting directly to the data warehouse * Using SingleStore Pipelines * Using INSERT Statements **Testing Your Queries and Performance** * Run queries * Visual Explain * Concurrency test your queries using `dbench` based on app expectations **Writing Back to Your Application** **Connecting Your BI Tools** * Connectivity samples for Tableau, Power BI, Looker ## Getting Started We are excited for you to get started accelerating your BI dashboards by augmenting your cloud data warehouse with SingleStore Helios (SSMS). We have many users that have started their BI projects with data warehouses like Redshift, Snowflake and BigQuery, and realized that due to the need for fast ingest, performance, and high concurrency, they needed to augment their architecture with a highly performant, scalable cloud-native database. Some terminology used in this guide: * App/Application: Web- or mobile-based, customer or internal facing application. * Dashboard: Visual analytics displayed either through commercial business intelligence tools or custom-built solutions. * Object storage: Cloud-based data repository. Data warehouse users often find improved query latency and concurrency support using SingleStore Helios as the data store directly fueling their dashboards, while retaining their data warehouse for long term storage. Not only that, but data ingested directly into SingleStore is immediately able to be queried, rather than waiting minutes or hours for batch ingest into the data warehouse to complete. Let’s get started by discussing some of the basic things to consider when bringing your data over from one of these data warehouses. ## Things to Consider For users starting with a database already hosted in a cloud provider like AWS, Azure, or GCP, you may already have your data sitting in object storage (if not, we walk through how to do this later). SingleStore has a feature called [Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md), which allows you to bring in data from any of these places very quickly. We’ll go through how to do this in a bit. Now, of course, every database is different. Let’s talk a bit about database design differences. Things to know, regardless of your current platform: * SingleStore Helios can be deployed in any of the three major clouds, in any region – so you don’t have to worry about moving regions. * Cloud Enterprise Data Warehouses (EDWs) typically leverage ANSI SQL, and SingleStore does too. However, SingleStore leverages MySQL syntax so there may be some modifications required. SingleStore is also ACID compliant. Things to know for specific databases: **Snowflake** Users often use SingleStore Helios in conjunction with Snowflake to achieve improve concurrency and ingest SLAs at a lower overall TCO. **Redshift** * Users often use SingleStore Helios in conjunction with Redshift to improve overall query latency for workloads with transactions and analytics. * Many users come to SingleStore Helios after they have reached the Redshift column limitation as well. * Redshift’s "leader nodes" and "compute nodes" correspond to SingleStore Helios’s aggregator nodes and leaf nodes. Aggregators handle client communication and query orchestration, and leaf nodes manage data and compute power. You don’t need to know too much here given that this is a managed service, but you’ll see the node types denoted within the "Nodes" section of the Cloud Portal. **BigQuery** Users typically find SingleStore Helios to be faster in overall ingest speeds and query latency with concurrency, but may find BigQuery simpler to implement with the overall GCP stack. **Azure Synapse** Synapse’s "control nodes" and "compute nodes" correspond to SingleStore Helios’s "aggregator nodes" and "leaf nodes". Aggregators handle client communication and query orchestration, and leaf nodes manage data and compute power. You don’t need to know too much here given that this is a managed service, but you’ll see the node types denoted within the "Nodes" section of the Cloud Portal. ## Sample Architectures Many of our users experience slow dashboard response times when overwhelming their data warehouses with various different speeds and sizes of data. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcb0d104d84917592/6a7adb55e88c657e5ffffe01/16083b57a2711d-sv7XmI.png) Keeping those data warehouses in place and augmenting them with a high performance database built for fast ingest and concurrency helps our users not only save money, but deliver a stellar user experience. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt96535c2572f369a9/6a7adb6042a117127b950957/16083b57a2ee4f-eOSb7y.png) ## Preparing Your Data for SingleStore SingleStore has simple, powerful methods of bringing data in from object storage. Here are some options for getting your existing databases exported to object storage in places like S3, GCS and Azure Blob Storage. Is your data already in cloud storage? Feel free to move on to the next section. Existing Managed Cloud Databases: * [Snowflake to S3](https://docs.snowflake.com/en/user-guide/data-unload-s3.html) * [Redshift to S3](https://docs.aws.amazon.com/redshift/latest/dg/t_Unloading_tables.html) * [BigQuery to GCS](https://cloud.google.com/bigquery/docs/exporting-data) * Synapse to Blob Storage For other databases that you’re looking to migrate, you can still export that data to CSV, JSON, etc. and then upload it to object storage. Typically this would involve something like a `SELECT ... INTO OUTFILE ...` from your existing database. From there, you can upload those files to object storage. If you are importing data from MySQL or MariaDB, you can perform a simple `mysqldump` as listed in [Transition from MySQL to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/transition-from-mysql-to-singlestore-helios.md). **Don’t want to do a bulk import? That’s fine too! After we discuss schema design, we’ll walk you through connecting directly from your application and writing data.** ## Database Administration Now that we have identified our data source, let’s talk a bit about how we’ll interact with SingleStore. When you signed up in Cloud Portal, you got access to the SQL Editor and Visual Explain tools. The Cloud Portal is the best place to interact with your SingleStore data, build new data pipelines, and test out queries. Make sure you explore the features in the Cloud Portal, as well as the Tutorials available through the **Help** button. Ensure that you have the appropriate [client drivers](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads.md) prior to using other database administration tools. ## MySQL Workbench If you’re coming from MariaDB or MySQL, you may already be comfortable with Workbench. You can download MySQL Workbench [here](https://dev.mysql.com/downloads/workbench/). **Note**: When defining your connection, you will need to go to the Advanced tab and insert `defaultAuth=mysql_native_password` in the "Others:" field to ensure proper authentication. ## MySQL Command Line Within your Cloud Portal, your cluster details will include a MySQL Command section. This gives you an easy way to use the command line to connect to SingleStore. You can find the download for the MySQL shell [here](https://dev.mysql.com/downloads/shell/). **Note**: When defining your connection, you will need to enter this at the MySQL command line: `--defaultAuth=mysql_native_password` to ensure proper authentication. ## Designing Your Tables At this point, you should be using the SQL Editor or some other MySQL client to work with your cloud database. Before checking out how fast SingleStore can bring in your data, let’s make sure your tables are designed optimally for your application workload. By default within SingleStore Helios, database tables are created using our Universal Storage format (i.e., disk-based columnstore). There are a few important things to consider when designing tables to house your data: ## Data Types If you’re familiar with relational databases, you may not need too much guidance on data types. In addition to data types traditionally supported in relational databases, SingleStore also supports JSON and geospatial data types. Read about [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) in SingleStore. ## Shard Key This key determines how data is distributed across the database cluster, and is critical to ensure that your data isn’t skewed. Skewed data can lead to longer query times. Data contained within unique values of your shard key will reside in individual partitions of the database. Well distributed data will ensure scalability and parallelism when fulfilling diverse requests concurrently. * **Snowflake** users may be familiar with micro-partitioning and clustering to distribute data. If you’ve perhaps found these difficult to maintain on ever-changing data, you are in the right place. * **Redshift** users would be familiar with the various mechanisms listed here for data distribution across a cluster. * **BigQuery** users will be familiar with the concept of sharding from “partitioned tables”. * **Synapse** users will be familiar with the concept of sharding, though SingleStore Helios gives you a bit more flexibility here on data distribution. So how do I pick a shard key best for my application workload? * If you have a primary key, make sure the shard key is a subset of it (because cardinality matters!). * If your BI queries include frequent joins or filters on a specific set of columns, make sure the shard key is a subset of those (in both tables, for joins). * For highly concurrent workloads, make sure your shard key allows your queries to be single partition (i.e., join or filter columns within shard key). In this example, we use `user_id` as our shard key, which works nicely given its high cardinality as a part of this dataset. All records with the same `user_id` will be maintained together, which will improve query response time. ```sql CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT, user_id INT, page_id INT, ts TIMESTAMP, SHARD KEY (user_id), SORT KEY (click_id, user_id) ); ``` ## Columnstore Key In addition to identifying your shard key, it’s important to tell SingleStore how you would like to sort your data within each data segment. This helps SingleStore enable segment elimination, which ensures a minimal amount of data needs to be read for each query. This also helps SingleStore presort data for your queries. If you’re familiar with BigQuery, you may note that this is similar to the concept of [clustering](https://cloud.google.com/bigquery/docs/clustered-tables). So how do I pick a columnstore key best for my BI workload? * BI dashboards often require lots of filtering. If you have common filter columns, make sure those are in the columnstore key. * If you’re inserting in order by some column, it’s best to put that column first in the columnstore key. * Lower cardinality columns should be first in the columnstore key. In this example, we use `price` as our sort key, so items are sorted in order of that column when queried. ```sql CREATE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, Qty INT, SORT KEY (Price), SHARD KEY (ProductId) ); ``` ## Reference Tables If you have small, infrequently changing table(s) that are required for joins, consider making them [Reference Tables](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts/#UUID-712159bb-3006-3563-f42d-6188153d809f.md). * Reference tables are a convenient way to recreate dimension tables that you may use in MySQL, MariaDB, or PostgreSQL. * Reference tables are replicated to each leaf in the cluster ensuring data does not need to go across the network between partitions to join data. * Reference table commands need to be run from the endpoint listed in the Cloud Portal. ## Manage Database Users ## Add a Database User Database users are created automatically for all SingleStore Helios users who are granted access to a cluster group. These users can access clusters in the Portal or the SQL Editor and can also be used for third-party SQL clients or development tools. Refer [Authenticate via Browser-based SSO using JWTs](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-browser-based-sso-using-jwts.md) for more information about authentication from third-party clients. Database users can also be added by using the [`CREATE USER`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) command. The [`GRANT`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) command cannot be used to add a new user since auto user creation by using the `GRANT` command is deprecated and the `NO_AUTO_CREATE_USER` variable is enabled by default. If you have invited another SingleStore Helios user to join your [organization](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations.md), refrain from adding a database user with the same SingleStore Helios email address (i. e. the username) until after the invitation has been accepted. As the admin user is not always a cluster log-in option for all organization members, SingleStore recommends adding a separate database user for each organization member. ## Change a Database User Password The database `admin` password is configured when the cluster is first created. This is also referred to as the “Master Username” in the Cloud Portal, which is `admin` by default. To change this password, navigate to **Clusters** in the left navigation pane, and then select **Connect > CLI Client** for the cluster. Under **User Credentials**, select **Reset Password** next to the **Password** field and follow the provided instructions. Use the `SET PASSWORD` command to change a database user's password. A database user can also use this command to change their own password. ```sql SET PASSWORD FOR 'username'@'host' = PASSWORD('password'); ``` ## Remove a Database User To remove a database user, use the [DROP USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-user.md) command. ```sql DROP USER ''@'' ``` ## Inspect Database User Permissions You can view grants and permissions by querying `information_schema.user_privileges`. You can also view grants for a user by running [SHOW GRANTS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-grants.md): ```sql SHOW GRANTS FOR user@domain; ``` ## Set a Login Attempt Lockout Policy for a Database User You can specify the number of times a user can enter an incorrect password before they are locked out of the system. When a user reaches this limit, their account is locked for the specified number of seconds. This feature can be enabled per user or per role, in which case every user belonging to that role will be subject to failed login attempt lockout. ## Enable the Lockout Policy To enable the lockout policy: Set *both*`FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` for the user or role. `FAILED_LOGIN_ATTEMPTS` is the number of failed attempts before the account is locked, for example: `4`. `PASSWORD_LOCK_TIME` is the number of seconds a locked out account must wait before reattempting to log in. > **📝 Note**: You must set both `FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` to enable the feature. Enable the lockout feature at 4 failed attempts, with a lockout time of 4 hours (14400 seconds) when creating a user: ```sql CREATE USER user1 WITH FAILED_LOGIN_ATTEMPTS = 4 PASSWORD_LOCK_TIME = 14400; ``` Enabling the feature for a role: ```sql CREATE ROLE general WITH FAILED_LOGIN_ATTEMPTS = 4 PASSWORD_LOCK_TIME = 14400; ``` If a user is associated with more than one role with different password lock times, the larger `PASSWORD_LOCK_TIME` value is applied. If a user and a role the user is tied to have conflicting `FAILED_LOGIN_ATTEMPTS` settings, the lower value is applied. ## Update Lockout Settings If the `PASSWORD_LOCK_TIME` value is updated for a role or user, the new setting applies to currently locked accounts. For example, if a locked out user’s lockout time setting is 1 day, and `PASSWORD_LOCK_TIME` is then set to 4 hours, the new limit is enforced and the account will be unlocked 4 hours after it was locked. If a user’s lockout time setting is 4 hours, and the setting is increased to 1 day, the user will remain locked out for 1 day. If the `FAILED_LOGIN_ATTEMPTS` setting for a locked out user is updated to be higher than the current setting, the user is unlocked. If the new setting is lower than the current number of failed login attempts, and also higher than the user’s current number of failed login attempts, the new setting is ignored until the user successfully logs in. The user is still subject to the original `FAILED_LOGIN_ATTEMPTS` setting. ## Unlock a Locked Account To unlock a locked account: Use the `ALTER USER` command and specify `ACCOUNT UNLOCK`. ```sql ALTER USER user ACCOUNT UNLOCK; ``` If an account is locked either manually using the `ALTER USER … ACCOUNT LOCK` command or automatically has a result of exceeding the values specified for `FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` arguments, restarting the cluster removes the lock and returns the account’s status to ONLINE. The account lock needs to be explicitly re-applied after the cluster is back online and/or automated as part of the customer’s operational procedures. Users can confirm the lock state before and after a restart by viewing the `ACCOUNT_STATUS` column in the information\_schema.USERS view. ## Testing Your Queries Here are some helpful hints for testing your queries. ## Running Queries Hopefully at this point you have your data in SingleStore. You can check this by running some basic SELECT statements within the SQL Editor. Next, you may want to try out some queries that you were running with your last database. Generally, you will want to run queries twice to get a true understanding of the runtime as the first run must create and cache the query plan. ## Visual Explain One great feature of SingleStore Helios is Visual Explain. If you encounter a situation in which your query is taking longer than expected, highlight the desired query in the SQL Editor and click the Visual Explain icon to the left of the Run button. The icon resembles a tree. After clicking, you may then choose between [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) and [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md). Once you identify a bottleneck, you should be able to make changes either to your schema or to your query itself in order to improve speed. You can manually (non-visually) run `EXPLAIN` or `PROFILE` from any client; see the links above for details on the commands. ## Benchmarking At SingleStore, we've developed an easy-to-use tool for benchmarking called `dbbench`. You can check that out [here](https://github.com/memsql/dbbench). Once you've installed the packages to your host machine, you can walk through this [tutorial](https://github.com/memsql/dbbench/blob/master/TUTORIAL.md). All you will have to do is change the host from 127.0.0.1 to the endpoint listed for your cluster in the Cloud Portal (it should look something like: `svc-xxx-dml.aws-virginia-1.db.memsql.com`). The port will remain `3306`, as listed. You can then move onto testing based on your application workload. ## Writing Back to Your Application In the Sample Architectures section, we shared some general practices that our users have employed when augmenting their data warehouse environment with SingleStore. Given the nature of data warehouses and how they may serve other applications, you may want to write your data back to them even after putting them in SingleStore. We understand that, and have methods to do so. The best way to write to a cloud data warehouse from SingleStore is again via object storage (i.e., S3, GCS, Azure Blob). SingleStore Helios enables users to `SELECT… INTO…` in order to take the output of a given query and drop it into object storage. This can also be bundled within a stored procedure for continuous updates. See [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) for more information. ## Connecting Your BI Tools You’ve likely already selected an analytics partner, so we won’t go too deep into the similarities and differences between them. Based on our experience with BI and analytics technologies, we’ve linked to detailed guides below on how to get started on what to look out for when connecting them to SingleStore. In these examples, you’ll be using your endpoint here provided in the Cloud Portal, along with the credentials you used when spinning up your cluster. You will want to make sure that the inbound IP whitelist within the “Firewall” section of the Cloud Portal allows communication between your BI and SingleStore Helios environments. ## Tableau Documentation: [Connect with Tableau Desktop](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau.md) **Notes**: Tableau has a limitation of auto updates at most one per hour, so you may not see your real-time data updating within your dashboard by the second. However, you can manually refresh in Tableau to see freshly ingested data. ## Power BI Documentation: [Connect with Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi.md) **Notes**: Make sure you are using DirectQuery mode to truly harness the power of SingleStore and all of the data you have stored. Otherwise, you will be using Import mode to bring only a subset of data into PowerBI. This will also help you harness the power of your real-time data. ## Looker Documentation: [Connect with Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker.md) **Notes**: Looker allows dashboards to update up to the second, so be sure to watch the video in (b) to see how to configure that. This is very beneficial to showcase the speed of data ingest with SingleStore. ## Additional Analytics and BI Tools [Connect to Analytics and BI Tools](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) *** Modified at: November 18, 2025 Source: [/cloud/developer-resources/guides/augmenting-your-data-warehouse-to-accelerate-bi/](https://docs.singlestore.com/cloud/developer-resources/guides/augmenting-your-data-warehouse-to-accelerate-bi/) (An index of the documentation is available at /llms.txt) --- # Building a New Application Welcome to SingleStore Helios! We put together a guide to help you build a new application using SingleStore. Use this guide to follow a top-down approach to: * Preparing your data for SingleStore * Interacting with SingleStore * Designing your SingleStore tables to be optimized for your applications * Importing data from your originating database * Optimizing your queries for your applications * Connecting your application to SingleStore ## Tutorial Outline This tutorial is divided into the following sections: **Getting Started** * Benefits of leveraging S2MS for your application * Why SingleStore is a developer’s first choice for applications * What kinds of apps are developers building? **Preparing Your Data for SingleStore** **SingleStore Database Administration** * Cloud Portal * Drivers * MySQL Workbench * MySQL Command Line **Designing Your Tables** * Data Types * Designing a schema for an application * Sharding for application workloads * Sort keys for application workloads * Reference Tables **Ingesting Data** * Using SingleStore Pipelines * Using `INSERT` Statements **Testing Your Queries** * Run queries * Visual Explain * Concurrency test your queries **Connecting to Your Application Development Tools** * Code samples to connect to SingleStore ## Getting Started We are excited for you to get started building your first application with SingleStore Helios (S2MS). Hundreds of developers have built apps using S2MS to power their new startup idea, pet project, or even school projects! There are many reasons people come to us, so read on as we discuss the simple steps to getting speed. Some terminology used in this guide: * App/Application: Web- or mobile-based, customer or internal facing application. * Object storage: Cloud-based data repository. There are a few key steps to building the database for your first app, and here's how SingleStore can help: * Design your schemas - Make a few simple, straightforward selections on how your data is distributed and sorted to set yourself up for millisecond response * Load your data - Five lines of SQL to ingest thousands of records per second * Test your queries - Simple Cloud Portal interface to run queries, with in-depth visual profiling tool to make them fast * Connecting to your app - We offer many connectivity options, no matter where you're building your first app These steps are applicable to all of the applications that have been built on SingleStore. Here are a few ideas to help you brainstorm on some concepts to embed within your app: * Geospatial apps for real-time tracking and fencing using points, paths, and polygons * Analytical apps with complex aggregations and real-time updates * Mobile apps with rapid queries and high concurrency Let's get started by discussing some of the basic things to consider when bringing your data over from one of these databases. ## Sample Architecture Here's example of how SingleStore is used by many to give immersive, responsive experiences to their customers. This can be done in many ways, and you'll see below how one user leveraged SingleStore Helios not just as a fast query layer for an app, but as the data hub to send information downstream to other apps. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2237ab539aff3143/6a7ada5705b7b54d3817ffd1/16083b57a37f35-Q3O8au.png) Users just getting started may already have data being generated by their application. If not, maybe you just have some synthetic data. Regardless of what kind of data you have or where it's coming from, there is sure to be a way to ingest it within SingleStore and test out queries. ## Preparing Your Data for SingleStore SingleStore has simple, powerful methods of bringing data in from object storage and streaming data sources. If your data is already in object storage like S3, Azure Blob or GCS, you will be set for the data import section. Also, if your data is in more of a real-time store you will be able to ingest data easily. That said, we realize if you're just getting started you may not have bulk or streaming data sources set up. If you'd like to write directly from your application, that's fine too! After we discuss schema design, we'll walk you through connecting directly from your application and writing data. ## Database Administration Now that we have identified our data source, let's talk a bit about how we'll interact with SingleStore. **Cloud Portal** When you signed up for SingleStore, you got access to our Cloud Portal, including tools such as the SQL Editor and Visual Explain. The Cloud Portal is the best place to interact with your SingleStore data, build new data pipelines, and test out queries. Click [here](https://www.singlestore.com/resources/youtube-memsql-studio) to get a tour of the features in the Cloud Portal, as well as the Tutorials available through the **Help** button. If you to use third-party tools, make sure that you have the appropriate [client drivers](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads.md) prior to using other database administration tools. **MySQL Workbench** If you're coming from MariaDB or MySQL, you may already be comfortable with Workbench. You can download MySQL Workbench [here](https://dev.mysql.com/downloads/workbench/). > **📝 Note**: When defining your connection, you will need to go to the Advanced tab and insert `defaultAuth=mysql_native_password` in the "Others:" field to ensure proper authentication. **MySQL Command Line** Within your Cloud Portal, your cluster details will include a MySQL Command section. This gives you an easy way to use the command line to connect to SingleStore. You can find the download for the MySQL shell [here](https://dev.mysql.com/downloads/shell/). > **📝 Note**: When defining your connection, you will need to enter this at the MySQL command line: `--defaultAuth=mysql_native_password` to ensure proper authentication. ## Designing Your Tables At this point, you should be using the SQL Editor or some other MySQL client to work with your cloud database. Before checking out how fast SingleStore can bring in your data, let's make sure your tables are designed optimally for your application workload. By default within SingleStore Helios, database tables are created using our Universal Storage format (i.e., disk-based columnstore). There are a few important things to consider when designing tables to house your data. ## Data Types If you're familiar with relational databases, you may not need too much guidance on data types. In addition to data types traditionally supported in relational databases, SingleStore also supports JSON and geospatial data types. Read about [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) in SingleStore. ## Shard Key This key determines how data is distributed across the database cluster, and is critical to ensure that your data isn't skewed. Skewed data can lead to longer query times. Data contained within unique values of your shard key will reside in individual partitions of the database. We offer [free training on sharding](https://training.singlestore.com/learn/course/internal/view/elearning/680/sharding-and-shard-keys) if you'd like to learn more! So how do I pick a shard key best for my application workload? * If you have a primary key, make sure the shard key is a subset of it (because cardinality matters!). * If your application queries include frequent joins or filters on a specific set of columns, make sure the shard key is a subset of those. * Concurrency is very important with application workloads, so make sure your shard key allows your queries to be single partition, as explained below. In this example, we use `user_id` as our shard key, which works nicely given its high cardinality as a part of this dataset. All records with the same `user_id` will be maintained together, which will improve query response time. ```sql CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT, user_id INT, page_id INT, ts TIMESTAMP, SHARD KEY (user_id), SORT KEY (click_id, user_id) ); ``` ## Columnstore Key In addition to identifying your shard key, it's important to tell SingleStore how you would like to sort your data within each data segment. This helps SingleStore enable segment elimination, which ensures a minimal amount of data needs to be read for each query. This also helps SingleStore presort data for your queries So how do I pick a columnstore key best for my application workload? * If you have common filter columns, make sure those are in the columnstore key. * If you're inserting in order by some column, it's best to put that column first in the columnstore key. * Lower cardinality columns should be first in the columnstore key. In this example, we use `price` as our sort key, so items are sorted in order of that column when queried. ```sql CREATE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, Qty INT, SORT KEY (Price), SHARD KEY (ProductId) ); ``` ## Reference Tables If you have small, infrequently changing table(s) that are required for joins, consider making them [Reference tables](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md). * Reference tables are replicated to each leaf in the cluster ensuring data does not need to go across the network between partitions to join data. * Reference table commands need to be run from the endpoint listed in the Cloud Portal. ## Data Ingest Now for the fun part, ingesting data! This is where things may look a bit different to you compared to other databases, because SingleStore has this unique ingest capability called Pipelines that supports high-frequency, parallel ingest of data from sources like S3, Azure Blob, GCS, Kafka, etc. Skip-list indexes and concurrent versioning technologies allow these writes to not lock tables, allowing reads to continue unimpacted during ingest. ## SingleStore Pipelines To use a Pipeline to import data into SingleStore, write a `CREATE PIPELINE` statement using the SQL Editor or a MySQL client. A few things to consider: * Make sure that your security settings in your blob storage will allow for access from SingleStore. For example, AWS S3 security settings can be found [here](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-permissions.html). * Make sure your buckets are not public, but you should be able to obtain an access and secret key using the AWS doc [here](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html). * You can use wildcard notation when identifying your files from within the bucket. Here's an example of a CREATE PIPELINE statement: ```sql CREATE PIPELINE library AS LOAD DATA S3 'my-bucket-name' CONFIG '{"region": "us-west-1", "suffixes": ["csv"]}' CREDENTIALS '{"aws_access_key_id": "your_access_key_id", "aws_secret_access_key": "your_secret_access_key"}' INTO TABLE `classic_books` FIELDS TERMINATED BY ','; ``` GCP Documentation: [Load Data from Google Cloud Storage (GCS) Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline.md) [Bucket Security](https://cloud.google.com/storage/docs/access-control) Azure Documentation: [Load Data from Azure Blob Storage Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline.md) [Bucket Security](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-secure-access-application) ## Insert Statements Perhaps instead of bulk loading data, you'd like to write directly from your application. That's great! Many of our users do this with the [MySQL JDBC driver](https://dev.mysql.com/downloads/connector/j/). You can find examples on writing your insert statements [here](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md). ## Testing Your Queries Here are some helpful hints for testing your queries. ## Running Queries Hopefully at this point you have your data in SingleStore. You can check this by running some basic `SELECT` statements within the SQL Editor. Next, you may want to try out some queries that you were running with your last database. Generally, you'll want to run queries twice to get a true understanding of the runtime as the first run must create and cache the query plan. ## Visual Explain One great feature of Cloud Portal is our Visual Explain functionality. If you encounter a situation in which your query is taking longer than expected, highlight the desired query in the SQL Editor and click the Visual Explain icon to the left of the Run button. The icon resembles a tree. After clicking, you may then choose between [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) and [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md). the visual explanation or profiling of your query. Once you identify a bottleneck, you should be able to make changes either to your schema or to your query itself in order to improve speed. You can manually (non-visually) run `EXPLAIN` or `PROFILE` from any client; see the links above for details on the commands. ## Benchmarking At SingleStore, we've developed an easy-to-use tool for benchmarking called `dbbench`. You can check that out [here](https://github.com/memsql/dbbench). Once you've installed the packages to your host machine, you can walk through this [tutorial](https://github.com/memsql/dbbench/blob/master/TUTORIAL.md). All you will have to do is change the host from 127.0.0.1 to the endpoint listed for your cluster in the Cloud Portal (it should look something like: `svc-xxx-dml.aws-virginia-1.db.memsql.com`). The port will remain `3306`, as listed. You can then move onto testing based on your application workload. ## Connecting to Your Application Development Tools We've created tutorials on how to connect to SingleStore using a variety of different frameworks, which you can find in the list below. The tutorials below show how to leverage both SingleStore Helios and SingleStore Self-Managed (our self-managed product). You can skip to the part of each that details how to make the connection and then get started. Again, you'll be using your endpoint here provided in the Cloud Portal, along with the credentials you used when spinning up your cluster. **JavaScript / Node** SQL: Stored Procedures: **C#** SQL: Stored procedures: **Java** SQL: Stored procedures: **Go** SQL: Stored procedures: **Ruby** SQL: Stored procedures: *** Modified at: May 22, 2026 Source: [/cloud/developer-resources/guides/building-a-new-application/](https://docs.singlestore.com/cloud/developer-resources/guides/building-a-new-application/) (An index of the documentation is available at /llms.txt) --- # Designing for Multi-Tenant Applications SingleStore is a great solution for multi-tenant analytical applications because of its unique architecture which provides ultra fast ingest, super low latency queries, and high concurrency. This article is for developers looking to create secure, fast, scalable, and cost effective applications on top of SingleStore. It contains design patterns you can use to get the most out of SingleStore for your multi-tenant apps. Definition: A *multi-tenant application* is one where a software-as-a-service (SaaS) application provider has many customers of their own. Each of their customers is a tenant of the application. While the typical design for a multi-tenant application assumes a Cloud-based platform, and that is the general approach described here, the principles and approach discussed herein also work well with a self-hosted environment. The diagrams below mention clusters in the Cloud but the same concepts can be applied to self-managed clusters. SaaS application providers typically have several goals, primarily: * create a compelling application * make the application fast and scalable * keep the cost per tenant reasonable * isolate tenants from one another SingleStore can allow you to achieve all these goals. ## Multi-Tenancy Approaches There are the following main approaches to design your databases and table schemas to handle multi-tenancy: * **tenant\_id-column**: This approach puts all tenants' data together in one database (DB), and puts a tenant\_id column in every table to differentiate data for each tenant. * **tables-per-tenant**: This approach puts all tenant data in one DB, and gives each tenant their own set of tables. More than one tenant's tables may be placed in one database. All data in one table is for one tenant. * **DB-per-tenant**: This approach gives each tenant their own database. All data for a tenant is in tables in one database. * **hybrid**: Hybrid approaches combine elements of two or more of the above. Here's a quick summary of the benefits and drawbacks of each and our suggestions of when each is appropriate. | Approach | Benefits | Drawbacks | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | tenant\_id column | Low disk and memory overhead because only one DB and relatively few tables. Easy to aggregate data across many tenants.Use this if you have a very large number of tenants (say more than a thousand) or if you want to have an individual query aggregate data across many tenants. | Queries have to include tenant\_id column filter on each table. Query plans may not be optimal for every tenant. | | tables-per-tenant | Low disk overhead because creating a database and its associated log files is what takes the most disk space, at least initially. | Requires a table naming convention such as tenantID\_LogicalTableName; application must substitute tenantID into actual table name at runtime. | | DB-per-tenant | Easy naming convention – queries can look the same for each tenant. "USE db\_for\_tenant" sets the context DB for a tenant. Tenants are isolated from each other. Backup and restore, disaster recovery, resource governance, and other DB-level features work separately for each tenant. | Requires a lot of disk for each tenant for preallocated log files for each partition. Requires dedicated threads for each tenant DB (which may hit system limits with many DBs).Each tenant's data must be backed up separately. | ## Multi-Tenant Design Approaches Customers have a few options when designing their databases with multi-tenancy in mind. The following sections describe various possible approaches. ## Tenant\_id Column Approach The most scalable and cost effective model is one database and set of tables for all tenants, with a tenant\_id column in each table used to identify data for different tenants. You could use customer\_id or a similar column name to identify each tenant's data; we'll use tenant\_id in the rest of this topic. Highlights for tenant\_id column approach: * The best system in terms of scalability, manageability, and cost control. * Keep customer data separate via addition of filters on tenant\_id into SQL statements generated by the application. * Use the tenant\_id column approach unless you have a good reason not to. Reasons to not use tenant\_id column approach: * Your customers have different schemas - For this, use tables-per-tenant or DB-per-tenant approaches. * For compliance reasons, some customers must not be in the same database as other customers - Go to DB-per-tenant, or a hybrid of tables-per-tenant and DB-per-tenant. * For compliance reasons, each customer must have its own database - Go to DB-per-tenant. Here's an example schema that uses a tenant\_id column to distinguish data among tenants "acme" (1) and "ajax" (2): ```sql /* this table has one row per tenant to define tenant names and IDs */ CREATE TABLE tenant(id int, name varchar(80)); INSERT INTO tenant VALUES (1,"acme"), (2,"ajax"); /* data for each tenant in these tables tied to the owning tenant by tenant_id */ CREATE TABLE sales(tenant_id int, product_id int, amount numeric(18,2)); CREATE TABLE product(tenant_id int, product_id int, product_name varchar(80), list_price numeric(18,2)); ``` Now, to retrieve all sales including the name of the item sold for tenant "ajax", you would write a SQL query like: ```sql SELECT s.product_id, p.product_name, s.amount FROM sales s, product p WHERE s.tenant_id = 2 AND p.tenant_id = 2 AND s.product_id = p.product_id; ``` To make the SQL more dynamic so you don't have to substitute the tenants in with application code all the time, you could use a session variable: ```sql SET @tenant_id = (SELECT id FROM tenant WHERE name = "ajax"); SELECT s.product_id, s.amount FROM sales s, product p WHERE s.tenant_id = @tenant_id AND p.tenant_id = @tenant_id AND s.product_id = p.product_id; ``` ## Security in the Tenant\_id-Column Approach In the tenant\_id-column approach, the application can keep data secure by (1) inserting the correct tenant\_id in each new row and (2) putting filters like "table\_name.tenant\_id = @tenant\_id"  in the `WHERE` clause of every query, for every table referenced in the query. Normally, all data access must be done via the application. Direct access to the data must not be granted to individual tenants. If direct read-only SQL access to the data is required, it is possible to create a view for each tenant for each table, restricting access to only their data, and grant `SELECT` access on the views to the appropriate tenant. It's also possible to use [Row-Level Security](https://docs.singlestore.com/cloud/security/administration/row-level-security-rls-deployment-guide.md) so only one view will be needed and the appropriate tenant ID filter is put in at runtime ## Tables-Per-Tenant Approach Another great approach is to put all customers in the same database, but use separate sets of tables for each customer. We'll call this tables-per-tenant for short. Highlights for tables-per-tenant approach: * This system is good at reducing total resources used (disk, threads), but you will need to carefully manage your memory use as you add customers. Even small tables take a certain amount of memory, which can become a limiting factor for tables-per-tenant. * This system is significantly more scalable than the DB-per-tenant approach so you should use it unless you strongly prefer DB-per-tenant for isolation of tenant data, for example, or you have few tenants or a big enough budget to provision enough disk for each tenant database needed in DB-per-tenant. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6f09b2af1d12cda2/6a7adaf5c2cc0981fc23db91/multi_planb2-baohyS.png) Figure 1. The tables-per-tenant approach, with a set of tables for each tenant, all in one database. ## Memory Management Using Tables-Per-Tenant Memory use is proportional to: `# of tables in set * # of sets of tables * overhead per table+ (number of frequent queries per tenant * number of active tenants)` Use the following strategies to minimize the above formula: * Use views instead of tables for each tenant (a hybrid tenant\_id-column approach). * Use fewer unique queries (queries that require individual query plans to be compiled). * Decrease the number of tables via denormalization and/or avoiding vertical partitioning (wider tables are ok). Another memory minimization strategy is: * Turn off [table autostats](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-tasks.md) (generally not recommended because it can affect query plan quality). Consider this strategy to minimize disk usage: * Use fewer partitions (you can choose the number of partitions when you run `CREATE DATABASE` – but be aware that it is difficult to increase the number of partitions later). Tenants that you know will be small for a long time can have, say, only 2 or 4 partitions instead of the default value which is many times more. The `enable_idle_table_eviction` engine variable defaults to `Full` mode. This table eviction setting ensures efficient memory use in multi-tenancy scenarios by prioritizing active tenants and queries, while minimizing resource usage for less active tenants. Refer to the [Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) page for more information. You can [view your memory usage](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md) in the SingleStore Helios Portal. ## Security in Tables-Per-Tenant Approach The typical way to secure data in the tables-per-tenant approach is to perform all data access through applications and not grant direct SQL access to the tenant. If direct SQL access to the tables by the tenant is required, a user\_id can be created for each tenant and that user can be granted access to their tables with `GRANT` statements. Typically you'd only grant read-only (`SELECT`) access. Granting `UPDATE` access would risk corrupting data and causing your application to fail. ## DB-Per-Tenant Approach It’s also possible to put each customer in its own database: DB-per-tenant. This is easy to set up and great for prototyping, but for many budgets it will not scale to a significant number of tenants due to the resources used for databases and tables. Highlights for DB-per-tenant approach: * If every customer must be isolated at the database level for compliance reasons, the DB-per-tenant approach is your best option. * This option will work well for small numbers of customers, but because each database has significant local disk space overhead, you will need to scale up your cluster group (or cluster for self-managed) as you add customers and databases. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt777c15fa6ecdce16/6a7adb0e6c6eac2d5af08d09/multi_pland-3mimi9.png) ## Local Disk Use in DB-Per-Tenant [Unlimited Storage](https://docs.singlestore.com/cloud/manage-data/benefits-of-unlimited-storage.md) provides great benefits, but is distinct from the [Local Disk](https://support.singlestore.com/hc/en-us/articles/4406838603156-Understanding-Disk-Usage) (a.k.a. Persistent Cache), which is limited and holds cached data as well as other types of data, such as transaction logs, snapshots, and plan cache. By default, an 8-partition database, on average, will use 4 GB for transaction logs for each partition when you create the database, so if you have a lot of databases, this can add up quickly. This means that if you plan to have a lot of customers, you will need a large amount of local disk using the DB-per-tenant approach, which can require scaling up your cluster. ## Reducing Disk Usage Per DB For smaller tenants, you can reduce the amount of disk needed per database by: * reducing the number of partitions * reducing the `log_file_size_partitions` and `snapshot_trigger_size` engine variables Having less partitions means you'll need to allocate fewer log files (and thus less disk space) since each partition has a log file active and one pre-allocated, at time of DB creation. Reducing `log_file_size_partitions` will proportionally reduce storage for pre-allocated log files. Reducing `snapshot_trigger_size` reduces the total amount of log bytes that have to be kept around on disk in a steady state. Aggressive settings for these two variables are: | Variable | Value | Formatted Value | | --------------------------- | -------- | --------------- | | log\_file\_size\_partitions | 8388608 | 8,388,608 | | snapshot\_trigger\_size | 67108864 | 67,108,864 | E.g. the following would create a 4-partition database with the log file size and snapshot trigger size shown: ```sql SET GLOBAL log_file_size_partitions = 8388608; SET GLOBAL snapshot_trigger_size = 67108864; CREATE DATABASE db_tenant1 PARTITIONS = 4; ``` These values may work for small tenants but not be appropriate for large tenants. You can use different settings when creating different databases. For small tenants and a DB-per-tenant model, these settings can let you have many times more databases on the same size cluster. Over time, if you start with small settings like this and a tenant grows to be large and needs to run on a large cluster, you may be required to to create a new database with more partitions and move the data over to it, which is an offline operation. It's best to set these values appropriately in advance for each database. Large tenants should have larger values for these, when using a DB-per-tenant approach. ## Isolating Sets of Customers with Cluster Groups (Hybrid) Some groups of tenants might need to be physically isolated from other tenants. You can achieve this with separate cluster groups: * If some sets of customers must be totally walled off, cluster groups (or separate clusters in a self-hosted environment) provide the greatest isolation. For example, if all your customers in AWS Europe needed to be separated, you could have an AWS Europe cluster group. * You would need to use this in conjunction with a tenant\_id-column or tables-per-tenant approach as having every customer in its own cluster group will not scale well in terms of operational complexity or cost. ![Diagram showing an organization consisting of three tenants: AWS European Tenants, Azure European Tenants, and AWS Australian Tenants](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta9afd9d801ead906/6a7adba480ee382ad7607923/multi_planc-my8RRa.png) ## Providing Capacity and Isolation for Tenants Clusters are how SingleStore achieves separation of storage and compute, which makes them ideal for use in multi-tenant applications. ## Example Let's assume that one of SingleStore's customers has 10,000 customers with free, standard and enterprise offerings. Clusters can be used to create different service expectations between these offerings. Each group of tenants can be placed on a separate cluster, where the number of users and the size of the compute cluster could be adjusted to provide higher or lower quality of service per tenant. Example offerings: * Free - S-2 Cluster - 9,000 tenants with low performance requirement and infrequent usage. * Standard - S-4 Cluster - 900 tenants, medium performance requirement with infrequent usage. * Enterprise - S-16 Cluster - 99 tenants with high performance requirements and frequent usage. Here, a tenant\_id-column approach would be used throughout, so 9,000 tenants in the "Free" group could fit on one S-2 cluster. The same approach would be used for the higher-level groups so the application code would only have to be written once and not be different for different tiers. ## Summary When deciding which multi-tenancy approach is best for your particular circumstances, consider which factors including cost, scaling tenants over time, and isolation are most important to you. One of these approaches can work well for you: * tenant\_id column * tables-per-tenant * DB-per-tenant * a hybrid of two or more of the above Hybrid approaches can help you balance the relative strengths of the three foundational approaches. Moreover, placing databases containing different sets of tenants on different cluster groups (clusters) can give you additional scalability and isolation across your set of tenants. *** Modified at: July 23, 2026 Source: [/cloud/developer-resources/guides/designing-for-multi-tenant-applications/](https://docs.singlestore.com/cloud/developer-resources/guides/designing-for-multi-tenant-applications/) (An index of the documentation is available at /llms.txt) --- # Migrating Data from Another Database Welcome to SingleStore Helios! We put together a guide to help you migrate your analytical application to SingleStore. Use this guide to follow a top-down approach to: * Extracting your data from your existing database * Interacting with SingleStore * Designing your tables to be optimized for your applications * Importing data from your originating database * Optimizing your queries for your applications * Connecting your application to SingleStore ## Tutorial Outline This tutorial is divided into the following sections: **Getting Started** * Benefits of leveraging S2MS for your application * What to consider when bringing your application data from: * MySQL, MariaDB, Postgres * AWS, GCP, Azure * What to consider when designing your database coming from: * MySQL, MariaDB, Postgres * AWS, GCP, Azure **Preparing Your Data for SingleStore** **Database Administration** * Cloud Portal * Drivers * MySQL Workbench * MySQL Command Line **Designing Your Tables** * Data types * Designing a schema for an application * Sharding for application workloads * Sort keys for application workloads * Reference Tables **Ingesting Data** * Using SingleStore Pipelines * Using INSERT Statements **Testing Your Queries and Performance** * Run queries * Visual Explain * Concurrency test your queries **Connecting Your Application Development Tools** * Code samples to connect to SingleStore ## Getting Started We are excited for you to get started migrating your application from your existing database to SingleStore Helios (S2MS). We have many users that have migrated internal and customer-facing applications to S2MS after they found that other solutions (MySQL, MariaDB, Postgres, etc.) were not meeting their needs. Users often find improved data ingestion speed and query latency, and better concurrency support using S2MS as their backend database instead of those platforms. Some terminology used in this guide: * App/Application: Web- or mobile-based, customer or internal facing application. * Object storage: Cloud-based data repository. Let’s get started by discussing some of the basic things to consider when bringing your data over from one of these databases. ## Things to Consider For users starting with a database already hosted in a cloud provider like AWS, GCP, or Azure, you may already have your data sitting in object storage (if not, we walk through how to do this later). The feature called [Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md), allows you to bring in data from any of these places very quickly. We’ll go through how to do this in a bit. Alternatively, you can use `mysqldump`, since SingleStore follows the MySQL wire protocol. Now, of course, every database is different. Let’s talk a bit about database design differences. Things to know, regardless of your current platform: * SingleStore Helios can be deployed in any of the three major clouds, in any region – so you don’t have to worry about moving regions. **MySQL or MariaDB** (for example, open source, AWS RDS MariaDB, Google Cloud SQL, Azure DB for MySQL, etc.) * S2MS is MySQL wire-compatible, making it very easy to transition any of these above databases. * S2MS is a distributed database system offering simple, powerful sharding capabilities through a shard key, which we will explain later. * S2MS will not validate whether foreign keys exist when your data changes, though you can still have foreign keys in your app. **PostgreSQL** (for example, open source, EnterpriseDB, CitusDB) * S2MS primarily follows MySQL syntax, so you will have to re-write queries. * S2MS is distributed, and provides robust support for all data types including JSON. * S2MS will not validate whether foreign keys exist when your data changes, though you can still have foreign keys in your app. ## Preparing Your Data for SingleStore SingleStore has simple, powerful methods of bringing data in from object storage. Here are some options for getting your existing databases exported to object storage in places like S3, GCS and Azure Blob Storage. Is your data already in cloud storage? Feel free to move on to the next section. Existing Managed Cloud Databases: * [AWS RDS to AWS S3](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html) * [Google Cloud SQL to GCS](https://cloud.google.com/sql/docs/mysql/import-export/exporting) * Azure Database to Blob Storage For other databases that you’re looking to migrate, you can still export that data to CSV, JSON, etc. and then upload it to object storage. Typically this would involve something like a `SELECT ... INTO OUTFILE ...` from your existing database. From there, you can upload those files to object storage. If you are importing data from MySQL or MariaDB, you can perform a simple `mysqldump` as listed in [Transition from MySQL to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/transition-from-mysql-to-singlestore-helios.md). **Don’t want to do a bulk import? That’s fine too! After we discuss schema design, we’ll walk you through connecting directly from your application and writing data.** ## Database Administration Now that we have identified our data source, let’s talk a bit about how we will interact with SingleStore. **Cloud Portal** When you signed up in the Cloud Portal, you got access to the SQL Editor and Visual Explain tools. The Cloud Portal is the best place to interact with your SingleStore data, build new data pipelines, and test out queries. Click [here](https://www.singlestore.com/resources/youtube-memsql-studio) to explore the features in the Cloud Portal, as well as the Tutorials available through the **Help** button. If you plan to use third party tools, make sure that you have the appropriate [client drivers](https://docs.singlestore.com/cloud/connect-to-singlestore/client-and-driver-downloads.md) prior to using other database administration tools. ## MySQL Workbench If you’re coming from MariaDB or MySQL, you may already be comfortable with Workbench. You can download MySQL Workbench [here](https://dev.mysql.com/downloads/workbench/). **Note**: When defining your connection, you will need to go to the Advanced tab and insert `defaultAuth=mysql_native_password` in the “Others:” field to ensure proper authentication. ## MySQL Command Line Within your Cloud Portal, your cluster details will include a MySQL Command section. This gives you an easy way to use the command line to connect to SingleStore. You can find the download for the MySQL shell [here](https://dev.mysql.com/downloads/shell/). **Note**: When defining your connection, you will need to enter this at the MySQL command line: `--defaultAuth=mysql_native_password` to ensure proper authentication. ## Designing Your Tables At this point, you should be using the SQL Editor or some other MySQL client to work with your cloud database. Before checking out how fast SingleStore can bring in your data, let’s make sure your tables are designed optimally for your application workload. By default within SingleStore Helios, database tables are created using our Universal Storage format (i.e., disk-based columnstore). There are a few important things to consider when designing tables to house your data: ## Data Types If you’re familiar with relational databases, you may not need too much guidance on data types. In addition to data types traditionally supported in relational databases, SingleStore also supports JSON and geospatial data types. Read about [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md). ## Shard Key This key determines how data is distributed across the database cluster, and is critical to ensure that your data isn’t skewed. Skewed data can lead to longer query times. Data contained within unique values of your shard key will reside in individual partitions of the database. * Users of commercial Postgres offerings like Citus will find this familiar to, but not exactly the same as, distribution keys. * Users of **MySQL** in [AWS RDS](https://aws.amazon.com/blogs/database/sharding-with-amazon-relational-database-service/) or [GCP Cloud SQL](https://cloud.google.com/sql/docs/mysql/operational-guidelines) will also be familiar with the concept of sharding. * We offer [free training on sharding](https://training.singlestore.com/learn/course/internal/view/elearning/680/sharding-and-shard-keys) if you’d like to learn more! So how do I pick a shard key best for my application workload? * If you have a primary key, make sure the shard key is a subset of it (because cardinality matters!). * If your application queries include frequent joins or filters on a specific set of columns, make sure the shard key is a subset of those. * Concurrency is very important with application workloads, so make sure your shard key allows your queries to be single partition, as explained below. In this example, we use `user_id` as our shard key, which works nicely given its high cardinality as a part of this dataset. All records with the same `user_id` will be maintained together, which will improve query response time. ```sql CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT, user_id INT, page_id INT, ts TIMESTAMP, SHARD KEY (user_id), SORT KEY (click_id, user_id)) ); ``` ## Columnstore Key In addition to identifying your shard key, it’s important to tell SingleStore how you would like to sort your data within each data segment. This helps enable segment elimination, which ensures a minimal amount of data needs to be read for each query. This also helps presort data for your queries. So how do I pick a columnstore key best for my application workload? * If you have common filter columns, make sure those are in the columnstore key. * If you’re inserting in order by some column, it’s best to put that column first in the columnstore key. * Lower cardinality columns should be first in the columnstore key. In this example, we use `price` as our sort key, so items are sorted in order of that column when queried. ```sql CREATE TABLE products ( ProductId INT, Color VARCHAR(10), Price INT, Qty INT, SORT KEY (Price), SHARD KEY (ProductId) ); ``` ## Reference Tables If you have small, infrequently changing table(s) that are required for joins, consider making them [Reference Tables](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts/#UUID-712159bb-3006-3563-f42d-6188153d809f.md). * Reference tables are a convenient way to recreate dimension tables that you may use in MySQL, MariaDB, or PostgreSQL. * Reference tables are replicated to each leaf in the cluster ensuring data does not need to go across the network between partitions to join data. * Reference table commands need to be run from the endpoint listed in the Cloud Portal. ## Data Ingest Now for the fun part, ingesting data! This is where things may look a bit different to you compared to other databases, because SingleStore has this unique ingest capability called Pipelines that supports high-frequency, parallel ingest of data from sources like S3, Azure Blob, GCS, Kafka, etc. Skip-list indexes and concurrent versioning technologies allow these writes to not lock tables, allowing reads to continue unimpacted during ingest. Previously, we discussed how to bring your data from MySQL, MariaDB, etc. into object storage. Now it’s time to go fetch that data from your object storage and import it into SingleStore. ## SingleStore Pipelines To use a Pipeline to import data into SingleStore, write a `CREATE PIPELINE` statement using the SQL Editor or a MySQL client. A few things to consider: * Make sure that your security settings in your blob storage will allow for access from SingleStore. For example, AWS S3 security settings can be found [here](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-permissions.html). * Make sure your buckets are not public, but you should be able to obtain an access and secret key using the AWS doc [here](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html). * You can use wildcard notation when identifying your files from within the bucket. Here’s an example of a `CREATE PIPELINE` statement: ```sql CREATE PIPELINE library AS LOAD DATA S3 'my-bucket-name' CONFIG '{"region": "us-west-1", "suffixes": ["csv"]}' CREDENTIALS '{"aws_access_key_id": "your_access_key_id", "aws_secret_access_key": "your_secret_access_key"}' INTO TABLE `classic_books` FIELDS TERMINATED BY ','; ``` GCP Documentation: [Load Data from Google Cloud Storage (GCS) Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline.md) [Bucket Security](https://cloud.google.com/storage/docs/access-control) Azure Documentation: [Azure Blob Pipelines Quickstart](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline.md) [Bucket Security](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-secure-access-application) ## INSERT Statements Perhaps you already have an application that is writing to your existing database, and you simply want to redirect the writes from that application to SingleStore. That’s great! Many of our users do this with the [MySQL JDBC driver](https://dev.mysql.com/downloads/connector/j/). See [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) for more examples on writing your insert statements. ## Testing Your Queries and Performance ## Running Queries Hopefully at this point you have your data in SingleStore. You can check this by running some basic SELECT statements within the SQL Editor. Next, you may want to try out some queries that you were running with your last database. Generally, you’ll want to run queries twice to get a true understanding of the runtime as the first run must create and cache the query plan. ## Visual Explain One great feature of the Cloud Portal is our Visual Explain functionality. If you encounter a situation in which your query is taking longer than expected, highlight the desired query in the SQL Editor and click the Visual Explain icon to the left of the Run button. The icon resembles a tree. After clicking, you may then choose between [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) and [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md). Once you identify a bottleneck, you should be able to make changes either to your schema or to your query itself in order to improve speed. You can manually (non-visually) run `EXPLAIN` or `PROFILE` from any client; see the links above for details on the commands. ## Benchmarking If you’re moving over from any of the databases we’ve talked about so far, you are probably interested in improving performance. We have developed an easy-to-use tool for benchmarking called dbbench. You can check that out [here](https://github.com/memsql/dbbench). Once you’ve installed the packages to your host machine, you can walk through this [tutorial](https://github.com/memsql/dbbench/blob/master/TUTORIAL.md). All you will have to do is change the host from 127.0.0.1 to the endpoint listed for your cluster in the Cloud Portal (it should look something like: `svc-xxx-dml.aws-virginia-1.db.memsql.com`). The port will remain `3306`, as listed. You can then move onto testing based on your application workload. ## Connecting to Your Application Development Tools We've created tutorials on how to connect to SingleStore using a variety of different frameworks, which you can find in the list below. The tutorials below show how to leverage both SingleStore Helios and SingleStore Self-Managed (our self-managed product). You can skip to the part of each that details how to make the connection and then get started. Again, you'll be using your endpoint here provided in the Cloud Portal, along with the credentials you used when spinning up your cluster. **JavaScript / Node** SQL: Stored Procedures: **C#** SQL: Stored procedures: **Java** SQL: Stored procedures: **Go** SQL: Stored procedures: **Ruby** SQL: Stored procedures: *** Modified at: May 22, 2026 Source: [/cloud/developer-resources/guides/migrating-data-from-another-database/](https://docs.singlestore.com/cloud/developer-resources/guides/migrating-data-from-another-database/) (An index of the documentation is available at /llms.txt) --- # Management API The SingleStore `Management` API provides a REST-based interface for provisioning, configuring, operating, and managing cloud resources within SingleStore Helios. The API is designed to support infrastructure automation, lifecycle management, and seamless integration with DevOps, CI/CD, and infrastructure-as-code workflows, enabling you to automate deployment pipelines, dynamically scale environments, enforce governance policies, manage operational workflows, and more. This RESTful API allows you to perform the following operations: * **Cluster groups**: Create, update, or delete cluster groups, request information on available cluster groups, list the cluster groups * **Clusters**: Create, update, suspend, resume, or delete clusters, request information on clusters, list the clusters, get the outbound allowlist for a cluster * **Projects**: List the projects accessible to a user * **Private connections**: Create, update, or delete private connections, request information on private connections * **Flow**: Create or terminate a Flow instance, request information on a Flow instance, list the Flow instances * **Invitations**: Invite a user, revoke an invitation, list the open invitations, request information on an invitation * **Teams**: Create, update, or delete a team for the current organisation, request information on a team, list the teams * **Users**: Add an existing user to the current organization, delete a user, request information on a user, list the users * **Roles**: Create, update, or delete a custom role, request information on a role, list the roles * **Jobs**: Create, update, or delete a job, request information on jobs, list the jobs * **Stage**: * Create, update, or delete a folder in a Stage, request information on a folder * Upload, download, modify, or delete a file in a Stage * **Files**: Create, upload, update, download, or delete a file in the personal, shared, or model space, list the files, request information on a file * **Secrets**: Create, update, or delete a secret, request information on a secret, list the secrets * **Audit logs**: Request Control Plane audit log entries * **Organizations**: Request information on current organization * **Billing**: Get raw compute and storage usage information for your organization * **Shared Edition** (Shared Tier): * Create or delete a Shared Edition deployment (starter cluster), request information on a starter cluster, list the starter clusters * Create, update, or delete a user in a Shared Edition deployment * **Cloud workload identity and delegated entities**: Request information on the cloud workload identity of a cluster, add or remove delegated entities, list the configured delegated entities * **Regions**: List the regions for a user The `Management` API supports [HTTP Bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/), which must be used in conjunction with SSL (HTTPS) for secure communication over the network. API calls made over HTTP will fail. The `Management` API authenticates requests using a unique, hex-encoded alphanumeric key, which can be generated on the [Cloud Portal](https://portal.singlestore.com/). For information on the `Management` API, refer to [Management API Overview](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md). You can also interface with the `Management` API from [SingleStore Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) using the [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) commands. *** Modified at: May 26, 2026 Source: [/cloud/developer-resources/management-api/](https://docs.singlestore.com/cloud/developer-resources/management-api/) (An index of the documentation is available at /llms.txt) --- # Porting Tables to SingleStore Helios ## Shard keys An important decision to make when porting your schema is setting up the appropriate shard keys, which determine how rows are partitioned (see [Sharding](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md) for more details). You can choose how to shard a table by adding a `SHARD KEY`. This functions like a normal table index, and can contain any number of columns. For example, the following table is hash partitioned based on the values of `a` and `b` - all rows with the same values for both `a` and `b` will be on the same partition: ```sql CREATE TABLE t1(a INT, b INT, c INT, SHARD KEY(a, b)); ``` If your table has a `PRIMARY` key, SingleStore Helios will by default choose that key as the shard key. For example, both of the following table definitions will shard the table by `a`: ```sql CREATE TABLE t1(a INT, b INT, PRIMARY KEY(a)); CREATE TABLE t1(a INT, b INT, SHARD KEY(a), PRIMARY KEY(a)); ``` If your table does not have a primary key, then SingleStore Helios will by default shard rows uniformly across the partitions. You can also choose to shard your table this way by entering a blank shard key. For example, both of the following will shard the table this way: ```sql CREATE TABLE t1(a INT, b INT); CREATE TABLE t1(a INT, b INT, SHARD KEY()); ``` If you aren’t sure yet what your workload will look like, these defaults will allow you to continue developing your application and do not impose any functional restrictions. Once you understand how your workload performs in SingleStore Helios better, you can recreate your table with a different chosen shard key. ## Choosing a Shard Key There are two major factors to take into account when defining a shard key: * Using a column or set of columns unique enough to minimize skew. * Sharding on columns which you expect to filter or join on often. This allows the optimizer to minimize network traffic during the execution of the query (see [Query Execution](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution.md) for more details). Let’s say we have the following table: ```sql CREATE TABLE events(event_id INT, user_id INT, time TIMESTAMP); ``` If we aren’t sure yet what our workload will look like, just using the first create table statement without specifying a shard key will shard rows uniformly. This will allow you to get started. It will ensure that there is no skew, but may not yield optimal query performance. Let’s say we have this query which we would like to be extremely performant: ```sql SELECT COUNT(DISTINCT first.user_id) FROM events AS first JOIN events AS second WHERE first.user_id = second.user_id AND TIMESTAMPDIFF(MINUTE, first.time, second.time) < 20; ``` In this case, we’ll want to modify our table schema to shard on `user_id`. When we run our query, we’ll be able to take advantage of locality when performing the join and when computing the distinct aggregate value: The advantage with the join is that it will happen on the leaves without having to repartition or move data between leaves. ```sql CREATE TABLE events(event_id INT, user_id INT, time TIMESTAMP, SHARD KEY(user_id)); ``` *** Modified at: July 29, 2024 Source: [/cloud/developer-resources/porting-tables-to-singlestore-helios/](https://docs.singlestore.com/cloud/developer-resources/porting-tables-to-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Procedural Extensions SingleStore Helios Procedural Extensions (PSQL) are available beginning in MemSQL version 6.0. These extensions provide familiar mechanisms for SQL developers and database administrators to encapsulate custom programmatic logic, namely: * [Stored Procedures (SPs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/stored-procedures.md) * [User-Defined Scalar-Valued Functions (UDFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-scalar-valued-functions-udfs.md) * [User-Defined Table-Valued Functions (TVFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-table-valued-functions-tvfs.md) * [User-Defined Aggregate Functions (UDAFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-aggregate-functions-udafs.md) * [External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/external-functions.md) Within stored procedures, you can write flexible queries that use [static parameterized SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/static-parameterized-sql.md) and [dynamic SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql.md). ## Wasm-based Functions SingleStore also supports Wasm-based functions. Refer to [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) for information on creating Wasm-based UDFs and TVFs. To create Wasm-based UDAFs, refer to [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md). If your logic and data structures can be implemented in PSQL with reasonable effort in a way that performs well, use PSQL. For integrating more complex logic and data structures, or using existing C/C++ or Rust code, SingleStore recommends using Wasm-based functions. For information on when to use PSQL, Wasm-based, or external functions, refer to [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md). ## Extensions Extensions in SingleStore allow you to combine user-defined objects, such as UDFs or UDAFs, into a packaged archive and then create, manage, and deploy these objects and other resources using the `CREATE EXTENSION` command. Extensions support both PSQL and Wasm-based functions. Refer to [Extensions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/extensions.md) for more information. ## Related Topics * [Procedural SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference.md) * [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) ## In this section * [Control Flow](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/control-flow.md) * [Additional Data Types](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/additional-data-types.md) * [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md) * [Stored Procedures](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/stored-procedures.md) * [User-Defined Scalar-Valued Functions (UDFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-scalar-valued-functions-udfs.md) * [User-Defined Table-Valued Functions (TVFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-table-valued-functions-tvfs.md) * [User-Defined Aggregate Functions (UDAFs)](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-aggregate-functions-udafs.md) * [External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/external-functions.md) * [Static Parameterized SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/static-parameterized-sql.md) * [Dynamic SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql.md) * [Procedural Extensions Security](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/procedural-extensions-security.md) * [See Also](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/see-also.md) *** Modified at: June 13, 2025 Source: [/cloud/developer-resources/procedural-extensions/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/) (An index of the documentation is available at /llms.txt) --- # Additional Data Types Stored procedures and UDFs can use `ARRAY` and `RECORD` types by accepting them as input parameters, creating and manipulating them in the procedure or UDF definition, and optionally returning them as a return type. Stored procedures can also use `QUERY` types. *** Modified at: April 24, 2021 Source: [/cloud/developer-resources/procedural-extensions/additional-data-types/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/additional-data-types/) (An index of the documentation is available at /llms.txt) --- # Choose Between PSQL, Wasm-based, and External Functions The decision whether to use Procedural SQL (PSQL) functions, Wasm-based functions, and external functions depends on the specific functionality and performance requirements of each implementation. SingleStore recommends using PSQL functions to implement database logic and Wasm-based functions for complex or existing C/C++ or Rust code. Use external functions to run functions in languages that are not supported by Wasm, implement highly complex C/C++ functions which require native code, or interact with external services or APIs. ## Using PSQL Functions For standard SQL operations, procedural logic, and performance-critical implementations, SingleStore recommends using PSQL functions. PSQL functions are run within the SingleStore engine and offer high performance for operations that can be expressed in SQL or Procedural SQL. PSQL functions are ideal for encapsulating reusable business logic that can be expressed in SQL (or PSQL) and run within the database engine, thereby avoiding the use of external compute resources. ## Using Wasm-based Functions Use Wasm-based functions to implement complex logic, data structures, or existing C/C++ or Rust code. Wasm-based functions leverage existing code libraries to perform computational tasks within the database engine that may be challenging to express in SQL. In many cases, Wasm-based functions offer significant performance improvements over external functions. Wasm-based functions in SingleStore do not support runtime capabilities that the database does not provide out of the box, such as making system calls, opening files, opening sockets, sending network messages, creating processes, or creating threads. SingleStore does not support these operations to protect the security and integrity of the system. ## Using External Functions Use external functions to execute logic that cannot be implemented within the database, when the language isn't supported by Wasm (C#, Java, Python, etc.), or where only native code assures high performance. You can also use external functions to integrate with external services or APIs, implement runtime capabilities, or perform computational tasks using libraries or resources that are not available within the SingleStore engine. For example, use external functions to implement a large C/C++ application with multiple dynamic link libraries (DLLs) that cannot be compiled using Wasm. External functions may have a higher overhead (higher resource costs or lower performance) than PSQL or Wasm-based functions because of network latency and the resource costs associated with serializing and deserializing data between SingleStore and external services. Conversely, external functions written in native C/C++ or Rust environments may provide performance improvements over Wasm for computation-intensive applications (such as mathematical software) because they can run directly as machine code instead of being interpreted by the Wasm runtime. ## Summary The following table summarizes the recommended use case and limitation of each function type: | Function Type | Description | Recommended Use Case | Limitation | | --------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Procedural SQL (PSQL) | Runs SQL or PSQL code within theSingleStoreengine | Standard SQL operations, procedural logic, and performance-critical implementations |
  • Implementing complex logic or data structures is not efficient
  • Cannot run code from other languages
| | Wasm-based | Imports existing C/C++ or Rust code and runs the code within theSingleStoreengine | Implement complex data structures using C/C++ or Rust |
  • Does not have runtime capabilities, such as, making system calls, opening files, opening sockets, etc.
  • Only supports C/C++ and Rust languages currently
| | External | Calls a function that runs outside of theSingleStoreengine |
  • Native code or environment delivers performance improvements
  • Run functions in languages not supported by Wasm
  • Connect with external APIs and services
|
  • Higher overhead, resource costs
  • Lower performance because of network latency
| *** Modified at: August 14, 2024 Source: [/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions/) (An index of the documentation is available at /llms.txt) --- # Control Flow All common control flow statements are supported, including: * Conditional control, such as `IF`, `ELSE`, and `ELSIF` * Iterative control, such as `LOOP`, `FOR`, `WHILE`, `EXIT`, `CONTINUE`, and loop labels For more information, see the [Control Flow Statements](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/control-flow-statements.md) topic. *** Modified at: April 28, 2021 Source: [/cloud/developer-resources/procedural-extensions/control-flow/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/control-flow/) (An index of the documentation is available at /llms.txt) --- # Dynamic SQL Dynamic queries give you the most flexibility in defining SQL statements. Before proceeding to the rest of this section, see [Static Parameterized SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/static-parameterized-sql.md) to understand how static parameterized queries work. Using PSQL variables, you can substitute strings anywhere at run time. In the following example, the query `SELECT id, name FROM customer ORDER BY ` is built dynamically using a string. The [EXECUTE IMMEDIATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/execute-immediate.md) statement uses the string to run the query. ```sql SET sql_mode = 'PIPES_AS_CONCAT'; DROP DATABASE IF EXISTS singlestore_docs_example; CREATE DATABASE singlestore_docs_example; use singlestore_docs_example; CREATE TABLE product(id INT, description TEXT); INSERT INTO product VALUES (1, 'white paper'), (2, 'blue pen'), (3, 'black pen'); DELIMITER // CREATE PROCEDURE get_product_data(col TEXT) AS DECLARE str TEXT; BEGIN str = 'ECHO SELECT id, description FROM product ORDER BY ' || col; EXECUTE IMMEDIATE str; END // DELIMITER ; CALL get_product_data('description'); ``` Output: ``` +------+-------------+ | id | description | +------+-------------+ | 3 | black pen | | 2 | blue pen | | 1 | white paper | +------+-------------+ ``` More examples of dynamic queries are shown below. These queries cannot be written as static parameterized queries because column names and table names are not constant values. ```sql SELECT FROM location` ``` ```sql INSERT INTO
(id) VALUES (1),(2),(3) ``` With dynamic queries, you’re not limited to using PSQL variables that represent column and table names. For instance, you can substitute a PSQL variable for a `WHERE` clause: ```sql SELECT id, name FROM part ``` You can also substitute a PSQL variable for a `SELECT` statement: ```sql SELECT ``` ## In this section * [Example Dynamic SQL Use Cases](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql/example-dynamic-sql-use-cases.md) * [Writing Dynamic SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql/writing-dynamic-sql.md) *** Modified at: June 12, 2023 Source: [/cloud/developer-resources/procedural-extensions/dynamic-sql/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql/) (An index of the documentation is available at /llms.txt) --- # External Functions An external function calls code that is executed outside of a SingleStore database. See the [CREATE EXTERNAL FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-external-function.md) topic for more information. *** Modified at: November 2, 2021 Source: [/cloud/developer-resources/procedural-extensions/external-functions/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/external-functions/) (An index of the documentation is available at /llms.txt) --- # Procedural Extensions Security ## Security Models Used by Procedural Extensions ## The Definer Security Model An object that uses the definer security model is executed using the security permissions of the user that created the object. [Stored procedures](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md) use the definer security model, by default. [Table valued functions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md), and [views](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-view.md) always use the definer security model. If such an object's definer does not have the needed permissions to execute all of the commands in the object's body, the object will fail to execute. The definer’s permissions are not checked when the object is created; the permissions are checked only when the object is executed. Once the object has been created, a user only needs the `EXECUTE` security permission to execute that object. If the object’s definer is deleted, then the object will no longer be able to be executed. If the object’s definer ever has a permission `REVOKED` that is required to execute the object’s body, then the object can no longer be executed. If a procedure `p2` uses the current user security model (see the next section), but was called by a procedure `p1` that uses the definer security model, `p2` will run as the user who defined `p1`. Although stored procedures use the definer security model by default, you can indicate in a stored procedure definition that definer security be used. This is shown in the following stored procedure definition, where `AUTHORIZE AS DEFINER` is specified. ``` DELIMITER // CREATE PROCEDURE p() AUTHORIZE AS DEFINER AS BEGIN ... END // DELIMITER ; ``` Note that this stored procedure definition could have included a `DECLARE` block (as is the case for any stored procedure). ## The Current User Security Model Stored procedures can be configured to use the current user security model. When the current user executes a stored procedure that uses the current security model, the stored procedure is executed using the security permissions of that user. If such a stored procedure does not have the needed permissions to execute all of the commands in the object's body, the object will fail to execute. The current user's permissions are not checked when the object is created; the permissions are checked only when the object is executed. If a procedure `p2` uses the current user security model, but was called by a procedure `p1` that uses the definer security model (see the previous section), `p2` will run as the user who defined `p1`. To create a stored procedure that uses current user security, specify the `AUTHORIZE AS CURRENT_USER` clause in a `CREATE PROCEDURE` statement, as shown in the following definition. ``` DELIMITER // CREATE PROCEDURE p() AUTHORIZE AS CURRENT_USER AS BEGIN ... END // DELIMITER ; ``` Note that this stored procedure definition could have included a `DECLARE` block (as is the case for any stored procedure). See the [CURRENT\_USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-user.md) page for more information. ## Application-Based User Security in Stored Procedures Within stored procedures, you can handle scenarios where your application’s users are given access to data based on an access level. For example, consider a reporting application for an educational institution. When this application’s users are provisioned, they are assigned an access level of `1`,`2` or `3`. When users access a feature to view a student’s data, the application calls the `get_student_info` procedure, which is shown below. This procedure returns only the data that the users’ access level allows. ```sql SET sql_mode = 'PIPES_AS_CONCAT'; DROP DATABASE IF EXISTS singlestore_docs_example; CREATE DATABASE singlestore_docs_example; use singlestore_docs_example; CREATE TABLE student(first_name TEXT, last_name TEXT, address TEXT, gpa FLOAT); INSERT INTO student VALUES ('John', 'Smith', '10 Anywhere St', 3.5), ('Sally', 'Smith', '20 Anywhere Lane', 4.0), ('Ann', 'Brown', '30 Anywhere Dr', 3.8); DELIMITER // CREATE PROCEDURE get_student_info(user_access_level INT, last_name TEXT) AS DECLARE cols_to_return TEXT; BEGIN IF user_access_level = 1 THEN cols_to_return = 'last_name, first_name'; ELSIF user_access_level = 2 THEN cols_to_return = 'last_name, first_name, address'; ELSIF user_access_level = 3 THEN cols_to_return = 'last_name, first_name, address, gpa'; END IF; EXECUTE IMMEDIATE 'ECHO SELECT ' || cols_to_return || ' FROM student WHERE last_name = ' || QUOTE(last_name) || ' ORDER BY last_name'; END // DELIMITER ; ``` Call `get_student_info` with access level `1`: ```sql CALL get_student_info(1, 'Smith'); ``` ```output +-----------+------------+ | last_name | first_name | +-----------+------------+ | Smith | John | | Smith | Sally | +-----------+------------+ ``` Call `get_student_info` with access level `2`: ```sql CALL get_student_info(2, 'Smith'); ``` ```output +-----------+------------+------------------+ | last_name | first_name | address | +-----------+------------+------------------+ | Smith | John | 10 Anywhere St | | Smith | Sally | 20 Anywhere Lane | +-----------+------------+------------------+ ``` Call `get_student_info` with access level `3`: ```sql CALL get_student_info(3, 'Smith'); ``` ```output +-----------+------------+------------------+------+ | last_name | first_name | address | gpa | +-----------+------------+------------------+------+ | Smith | John | 10 Anywhere St | 3.5 | | Smith | Sally | 20 Anywhere Lane | 4 | +-----------+------------+------------------+------+ ``` *** Modified at: November 29, 2022 Source: [/cloud/developer-resources/procedural-extensions/procedural-extensions-security/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/procedural-extensions-security/) (An index of the documentation is available at /llms.txt) --- # See Also * Training: [SingleStore Helios Procedural SQL](https://training.singlestore.com/learn/course/internal/view/elearning/636/procedural-sql) * [Procedural SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference.md) * [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) *** Modified at: August 6, 2024 Source: [/cloud/developer-resources/procedural-extensions/see-also/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/see-also/) (An index of the documentation is available at /llms.txt) --- # Static Parameterized SQL Within stored procedures, you can write queries that contain PSQL variables representing parameters. These queries are known as static parameterized queries. In the following example, `SELECT id, description FROM product ORDER BY id LIMIT row_count;` is a static parameterized query. `row_count` is an PSQL variable representing a parameter. ```sql DROP DATABASE IF EXISTS singlestore_docs_example; CREATE DATABASE singlestore_docs_example; use singlestore_docs_example; CREATE TABLE product(id INT, description TEXT); INSERT INTO product VALUES (1, 'white paper'), (2, 'blue pen'), (3, 'black pen'); DELIMITER // CREATE PROCEDURE get_product_data(row_count INT) RETURNS QUERY (id INT, description TEXT) AS DECLARE qry QUERY(id INT, description TEXT) = SELECT id, description FROM product ORDER BY id LIMIT row_count; BEGIN RETURN qry; END // DELIMITER ; ECHO get_product_data(2); ``` Output: ``` +------+-------------+ | id | description | +------+-------------+ | 1 | white paper | | 2 | blue pen | +------+-------------+ ``` Within stored procedures, you define static queries using the [QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/query.md) data type. Parameters in static queries always represent constants. In the above example, the variable `row_count` evaluates to the constant value `2`. Other examples of static parameterized queries are shown below. ```sql SELECT * FROM order WHERE amount >= ``` ```sql INSERT INTO item (name) VALUES () ``` *** Modified at: June 5, 2023 Source: [/cloud/developer-resources/procedural-extensions/static-parameterized-sql/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/static-parameterized-sql/) (An index of the documentation is available at /llms.txt) --- # Stored Procedures Stored procedures can accept input parameters, query tables using SQL statements, call UDFs, define custom logic using control flow statements and variable assignment, and optionally return a value. Stored procedures can also be called across databases. See the [CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-procedure.md) topic for more information. *** Modified at: April 28, 2021 Source: [/cloud/developer-resources/procedural-extensions/stored-procedures/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/stored-procedures/) (An index of the documentation is available at /llms.txt) --- # User-Defined Aggregate Functions (UDAFs) UDAFs support creation of custom aggregation logic, beyond the built-in aggregate functions supplied by SingleStore Helios. See the [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md) topic for more information. *** Modified at: June 22, 2022 Source: [/cloud/developer-resources/procedural-extensions/user-defined-aggregate-functions-udafs/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-aggregate-functions-udafs/) (An index of the documentation is available at /llms.txt) --- # User-Defined Scalar-Valued Functions (UDFs) UDFs can accept input parameters, call other UDFs, define custom logic using control flow statements and variable assignment, and return a value. See the [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) topic for more information. *** Modified at: April 28, 2021 Source: [/cloud/developer-resources/procedural-extensions/user-defined-scalar-valued-functions-udfs/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-scalar-valued-functions-udfs/) (An index of the documentation is available at /llms.txt) --- # User-Defined Table-Valued Functions (TVFs) TVFs can accept input parameters, execute a single `SELECT` statement, and return the result as a table-typed value. For more information, see [CREATE FUNCTION (TVF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md). *** Modified at: April 28, 2021 Source: [/cloud/developer-resources/procedural-extensions/user-defined-table-valued-functions-tvfs/](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/user-defined-table-valued-functions-tvfs/) (An index of the documentation is available at /llms.txt) --- # Secrets SingleStore Helios Secrets is a cloud service that allows you to manage and use your sensitive data (e.g., access tokens, API Keys). Instead of hard-coding your sensitive information directly in your SingleStore Notebooks or other Python environments in order to connect to your ecosystem, create a new secret and access the sensitive information by referencing the secret by name. Using secrets helps you avoid exposing or hard-coding the actual credential, key, or other sensitive information. > **📝 Note**: SingleStore does not have a direct connector to integrate AWS Secrets Manager. If you are deploying on AWS, you should use Helios, where secrets are automatically handled securely on AWS. ## Manage Secrets Each secret is a name, value pair. You can create and manage secrets using any of the following: * Cloud Portal UI * `Management` API ## Using the Cloud Portal You can create, edit, delete, and share secrets in SingleStore via the **Secrets** tab available on the **Editor** page. ## Create a Secret To create a secret: 1. On the Cloud Portal, select **Editor > Secrets**. 2. On the **Secrets** tab, select **New Secret**. 3. Enter a **Name** and **Value** for the secret. ![A dialog named New Secret with two boxes to enter secret and value, respectively.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt72622255a2d18341/6a7adbd614d87c00903c7705/secretsS2-dED6Rw.png) 4. Select **Create Secret**. The secret is added to the list of secrets displayed on the **Secrets** page in the Cloud Portal. ## Edit a Secret To edit a secret, select **Edit** from the **Actions** column for the secret you want to edit. Make your changes, and select **Update**. ## Delete a Secret To delete a secret, select **Delete** from the **Actions** column for the secret you want to delete. Confirm and select **Delete**. ## Using the Management API Use the `Secrets` path (`/v1/secrets` endpoint) in the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) to create and manage secrets. Refer to [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. ## Use Secrets You can access the secrets in SingleStore Notebooks or other Python environments using the `get_secret()` function from the [SingleStore Python SDK](https://singlestoredb-python.labs.singlestore.com/index.html) without the need to install additional libraries. ## SingleStore Notebooks To read/access a secret, run the following command in your notebook. ```Python from singlestoredb.management import get_secret secret = get_secret('') ``` ## Other Python Environments To read/access a secret externally from other Python environments, connect to your SingleStore cluster using the Management API objects in the SingleStore Python SDK. [Generate an API key](https://docs.singlestore.com/cloud/reference/management-api/#section-idm4495199953840032756423936306.md) for your organization on the [Cloud Portal](https://portal.singlestore.com/) to authenticate your connection. The following example accesses a secret named **secretExample**: ```Python from singlestoredb import manage_workspaces singlestoreAPIkey = '' org = manage_workspaces(singlestoreAPIkey).organizations.current print(org.get_secret('secretExample').value) ``` ## Share Secrets All secrets are only accessible by the user who created the secret by default. To share a secret with other members or teams of your organization: 1. Select **Share** from the **Actions** column for the secret to share. 2. From the **Share \** dialog, you can invite individual users or teams in the organization to have access to your secret. There are two access levels: * **Owner**: These users can edit the value of the secret, share it, and delete it. * **Reader**: These users have read-only access to the secret. Select a user or team from the list and then specify the access level (**Owner** or **Reader**). A secret may have more than one owner. To remove a user's or team's access to a secret, select **Remove Access** from the **Access** list. 3. Select **Save** to share the secret. ## Remarks * All secrets within an organization share the same namespace. Use a unique name for your secret to avoid conflicts with identically named secrets in the organization. * Each secret must have at least one user with Owner access. Any secret without an Owner is automatically removed. For example, if a secret has only one owner and the owner leaves the organization, the secret gets deleted. However, it remains accessible if another user is given Owner access to the secret. * A Secret’s name can only contain letters, numbers, and underscores (“\_") and its value must not exceed 5MB in size. ## Examples The following examples demonstrate how to securely retrieve and use a secret to access sensitive information. ## Build an AWS S3 Client The following example shows how to build an AWS S3 client in a notebook: ```Python import boto3 from singlestoredb.management import get_secret aws_access_key_id = get_secret('AWS_ACCESS_KEY_ID') aws_secret_access_key = get_secret('AWS_SECRET_KEY') s3_client = boto3.client('s3', aws_access_key_id = aws_access_key_id, aws_secret_access_key = aws_secret_access_key) ``` ## Use Hugging Face Token to Download an LLM The following example shows how to download Google’s Gemma-7B model in a Python environment: ```Python from singlestoredb import manage_workspaces from transformers import AutoTokenizer, AutoModelForCausalLM singlestoreAPIkey = '' org = manage_workspaces(singlestoreAPIkey).organizations.current hf_token = org.get_secret('HF_TOKEN').value tokenizer = AutoTokenizer.from_pretrained('google/gemma-7b',token=hf_token) model = AutoModelForCausalLM.from_pretrained('google/gemma-7b', token = hf_token) ``` *** Modified at: November 21, 2025 Source: [/cloud/developer-resources/secrets/](https://docs.singlestore.com/cloud/developer-resources/secrets/) (An index of the documentation is available at /llms.txt) --- # Transition from MySQL to SingleStore Helios SingleStore Helios is wire-compatible with MySQL, which makes it easy to transition your data and application from MySQL to SingleStore. To migrate your existing MySQL data, SingleStore recommends using Flow. [SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md) enables you to ingest large datasets with a guided, no-code interface that simplifies the migration process. It eliminates the need for manual schema conversion, custom scripting, and additional infrastructure. Unlike traditional migration methods, which can be resource intensive and operationally complex, Flow automates key migration steps to reduce overall migration time and complexity. To transition your data and application from MySQL to SingleStore Helios: 1. Migrate your data with SingleStore Flow. 2. Transition your application. ## 1. Migrate your Data with Flow Use SingleStore Flow to migrate your data from MySQL to SingleStore. Refer to [Load Data with SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md) for more information. SingleStore intentionally does not support all the MySQL features. Refer to [Unsupported MySQL Features](https://docs.singlestore.com/cloud/developer-resources/unsupported-mysql-features.md) for more information. You can also migrate your data using `mysqldump`, although it may require manual scripting and operational effort. * *Load Data with mysqldump* ## Migrating Data with mysqldump`mysqldump` is a popular tool packaged with the MySQL client infrastructure. It queries the database to produce a series of `CREATE TABLE` and `INSERT` statements that can be replayed to restore the database.For more information on loading data from MySQL using `mysqldump`, refer to [Load Data from MySQL using mysqldump](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql/load-data-from-mysql.md).> **📝 Note**: If you are running SingleStore Helios, use the endpoint to run the commands discussed in this topic.While moving your data from MySQL to SingleStore Helios, there are a few considerations to keep in mind:- For most MySQL storage engines (MyISAM, InnoDB, etc), indexes are stored as B-trees. In SingleStore Helios, indexes can be unidirectional lock-free skip lists (ascending or descending) or lock-free hash tables. Picking the right data structure for your index can have a significant impact on the performance of your application. While hash tables are optimized for key-value look ups, skip lists are extremely flexible for complex range scans and sorts (`ORDER BY`). While transferring your schema, you should audit your table definitions and investigate whether your indexes can be optimized for SingleStore Helios. The default `BTREE` notation is converted into a skip list (see [Skip List Indexes](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts/#UUID-d944e5f7-f309-7eab-50b1-66cf3f18dcef.md)) . - `mysqldump` will generate a few queries that are unsupported by SingleStore Helios. For example, SingleStore Helios does not support disabling `UNIQUE_CHECKS`. To make it easier to work with `mysqldump`, unsupported features are by default reported as warnings instead of errors. This functionality can be controlled by adjusting the `warn_level` variable. See [Unsupported MySQL Features](https://docs.singlestore.com/cloud/developer-resources/unsupported-mysql-features.md) for more details. Some components of a `CREATE TABLE` statement might be blocked completely. If you run into this issue while loading a schema into SingleStore Helios, you can manually massage the schema definition into something supported by SingleStore Helios. - If the machine running SingleStore Helios does not have enough memory to support the data you’re loading, the server will issue an error on offending `INSERT` statements indicating its out-of-memory state. In this case, you should upgrade your machine to one with more memory. If you copy your existing `memsqlbin` directory to the new machine, SingleStore Helios will be able to reuse the schema definitions and `INSERT` statements that have already compiled. - Definer error (error:1227) gets triggered when MySQL attempts to create an object under a database user, and that database user does not exist on the destination database. Refer [https://aws.amazon.com/premiumsupport/knowledge-center/definer-error-mysqldump/ ](https://aws.amazon.com/premiumsupport/knowledge-center/definer-error-mysqldump/) for a work around.You should separate your schema and data into separate files, so that you can easily review and modify your schema if necessary. To produce a dump of your database, run something like:```shell mysqldump -h 127.0.0.1 -u root -B [database name] --no-data -r schema.sql ``````shell mysqldump -h 127.0.0.1 -u root -B [database name] --no-create-info -r data.sql ```You can then replay these files directly into SingleStore Helios by running:```shell singlestore -h 127.0.0.1 -u root -P 3307 < schema.sql ``````shell singlestore -h 127.0.0.1 -u root -P 3307 < data.sql ```While this step runs, you can observe the `memsql.log` file to see which unsupported features have been ignored. After the import is completed, you can connect to SingleStore Helios and start querying the tables directly. ## 2. Transition your Application Update the connection configuration and credentials in your application to connect to a SingleStore Helios deployment. Refer to [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md) for related information. A key consideration to note is [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md), which plays a significant role in SingleStore's superior query execution performance. The first time SingleStore encounters a table schema, it generates and compiles code that implements the infrastructure around the table (memory allocation, inserts, deletes, iterations, etc.). Therefore, code generation makes loading a schema into SingleStore for the first time slower than with MySQL. Once a table is compiled, SingleStore is able to reuse it for the lifetime of your application - even if you restart the server or drop (and recreate) the table. Similarly, when using `mysqldump` to migrate your data, the `INSERT` queries generated by `mysqldump` also have to be compiled exactly once. When transitioning your application: * Queries that throw errors most likely correspond to unsupported syntax (refer to [SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference.md) for supported SQL surface area). *** Modified at: September 10, 2025 Source: [/cloud/developer-resources/transition-from-mysql-to-singlestore-helios/](https://docs.singlestore.com/cloud/developer-resources/transition-from-mysql-to-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Try SingleStore Helios for Free The fastest and easiest way to get started with SingleStore is to create a cluster in SingleStore Helios. Go [here](https://www.singlestore.com/managed-service-trial/) to try SingleStore Helios for free. ## In this section * [Sign Up to try SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/try-singlestore-helios-for-free/sign-up-to-try-singlestore-helios.md) *** Modified at: April 22, 2026 Source: [/cloud/developer-resources/try-singlestore-helios-for-free/](https://docs.singlestore.com/cloud/developer-resources/try-singlestore-helios-for-free/) (An index of the documentation is available at /llms.txt) --- # Sign Up to try SingleStore Helios With SingleStore Helios you can deploy a cluster in seconds and experience the performance, scalability, and multimodality of our database engine. Try the various features of SingleStore Helios by registering for a free trial: You may also log into [training](https://training.singlestore.com) and our [forums](https://www.singlestore.com/forum/) using your SingleStore account. ## Registering for a Free Trial Visit to get started. You can create a SingleStore account using your email address, or via your Google or Microsoft login. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt70022815a967f97c/6a7adbdabdcff03964c3784c/signup-BijxwG.png) * To sign up using your Google login, click the **Sign up with Google** button and follow the steps to sign into your Google account. * To sign up using your Microsoft login, click on the **Sign up with Microsoft** button and follow the steps to sign into your Microsoft account. After signing up, you will have successfully created a SingleStore account, and will be able to deploy a SingleStore Helios trial in your cloud platform of choice. ## Getting Started with Your Cluster There are two options to try SingleStore Helios for free: 1. **[Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition.md)** (Public Preview) – based on a shared-compute resource for small and performant prototype apps. In the Shared Edition, you create a Shared cluster that does not consume any credits. This option is not for use in production applications. 2. **Standard Edition** – based on a dedicated compute resource with, SLAs, larger resource limits, and more configurability and control. A Standard cluster will consume credits – SingleStore initially provides trial users with some free credits to experience a Standard cluster. In the case of a POC, or if you need to upgrade to SingleStore Helios Enterprise Edition, contact us through the internal chat app in the Cloud Portal. ## Auto-Suspension of Clusters If all clusters in a trial [cluster group](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/cluster-architecture.md) have not had a query run against them for two days, the organization’s contact email will receive a warning that the clusters will be suspended if no further action is taken. To prevent the clusters from being auto-suspended, run any query against any cluster in a cluster group within 24 hours after this first warning email. If no further action is taken within 24 hours after the first warning email, the clusters in the cluster group will be suspended, and a second email will be sent notifying you about the change. You can resume a suspended cluster via the Cloud Portal or the [Management API](https://docs.singlestore.com/cloud/user-and-cluster-administration/management-api.md). Alternatively, you may spin up a Shared cluster (Shared Edition) which does not have an auto-suspension policy based on query inactivity. ## Trial Completion Trial clusters that are suspended for 30 days will be terminated automatically. To prevent a trial cluster from being terminated, simply ensure that there is activity on the cluster. If you wish to evaluate SingleStore beyond the trial period, contact [Support](https://support.singlestore.com) to extend your product evaluation. *** Modified at: October 27, 2025 Source: [/cloud/developer-resources/try-singlestore-helios-for-free/sign-up-to-try-singlestore-helios/](https://docs.singlestore.com/cloud/developer-resources/try-singlestore-helios-for-free/sign-up-to-try-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Unsupported MySQL Features SingleStore intentionally does not support all of the features of MySQL. In some cases, SingleStore can safely ignore unsupported features in a SQL statement and continue executing only the supported features. ## Unsupported Feature List * **warn\_level Engine Variable** - how SingleStore behaves when it encounters unsupported functionality is controlled via the `warn_level` engine variable. The variable has two settings: * Warnings (default) - Permits SQL statements that aren’t supported, but whose unsupported features can safely be ignored. Warnings will be issued when such queries are used. Use this mode when porting an existing application to SingleStore and to avoid having to change the application to not use unsupported features. * Errors - SQL statements with unsupported features will be rejected as errors. This is the most strict warn level. Use this level when you are developing a new application on SingleStore and want to only use features fully supported by SingleStore. The `warn_level` variable is set just like other global engine variables: `SET GLOBAL warn_level= "errors";`. * `getProcedures()` and `getProcedureColumns()` MySQL built-ins * **User-Defined Variables**. SingleStore does not fully support user-defined variables. See [User-Defined Variables](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables.md). * **Triggers (other than the TIMESTAMP type)** * **Foreign keys and referential integrity** * **Prepared statements** - SingleStore does not fully support server-side prepared statements. See [Using Prepared Statements](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/using-prepared-statements.md). * SingleStore geospatial is incompatible with MySQL geospatial. * The comparison operator `LIKE` does not support the `ESCAPE` clause. However, you can use the backslash character `\` as an escape character in `LIKE` patterns. * `get_lock()` and related functions (`release_lock()`, etc.). * Certain SingleStore engine variables exist for backwards compatibility with MySQL and are non-operational in SingleStore. These engine variables are: * `character_set_client` * `character_set_connection` * `character_set_database` * `character_set_filesystem` * `character_set_results` * `character_set_system` * `character_sets_dir` Refer to [Reserved Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/reserved-engine-variables.md) for the complete list. ## Behavior Differences SingleStore differs from MySQL query behavior in a few ways, mostly in cases where MySQL behavior is officially undefined, and the observed behavior depends on the details of implementation. * **No implicit ordering of results by primary key:** MySQL (and other single-machine databases) may return data in order of the primary key of the table, even when the query does not specify an ordering. In SingleStore, a `SELECT *` query without an order clause may return data in any order, and the ordering may even be different between runs of the same query. This is because partitions stream their results to the aggregator in parallel, and the results are forwarded to the client as they arrive. To retrieve rows in order, you must specify an explicit order clause. ```sql SELECT * FROM messages; ``` ```output +----+-------+ | id | msg | +----+-------+ | 2 | ohai! | | 3 | kthx | | 1 | yo | +----+-------+ 3 rows in set (0.00 sec) ``` ```sql SELECT * FROM messages ORDER BY id; ``` ```output +----+-------+ | id | msg | +----+-------+ | 1 | yo | | 2 | ohai! | | 3 | kthx | +----+-------+ 3 rows in set (0.00 sec) ``` * **`UNION ALL` with `ORDER BY` behavior inconsistent with MySQL:** SingleStore parses UNION ALL followed by ORDER BY commands differently from MySQL. In SingleStore, the following query is valid. In MySQL, it is invalid. ```sql SELECT * FROM table1 UNION ALL SELECT * FROM table2 ORDER BY table2.col1 ``` *** Modified at: January 20, 2023 Source: [/cloud/developer-resources/unsupported-mysql-features/](https://docs.singlestore.com/cloud/developer-resources/unsupported-mysql-features/) (An index of the documentation is available at /llms.txt) --- # Getting Started with SingleStore Helios SingleStore allows you to build your applications for free with the Free Shared Edition. To register, go to [Cloud Portal](https://portal.singlestore.com/). The Free Shared Edition offers a free Shared cluster with one attached database. Refer to [Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition.md) for more information on Shared cluster. **Disclaimer:** Shared clusters are not intended for Production environments. ## Shared Cluster Compute and Storage A Shared cluster uses a shared compute environment with a compressed storage limit of 1 GB. Refer to [Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition/#section-idm4594624572019234098224277745.md) for more information. ## Connection Options SingleStore Helios supports various connection options to connect and query your data. ## Connect to SingleStore You can connect to SingleStore using your cluster via various application development tools. To connect using the Shared cluster: 1. Select **Clusters** in the left navigation pane. 2. Select **\** from the list of clusters, and then select **Connect**. You can connect to your Shared cluster using any of the supported tools. ## CLI Client Select **\ > Connect > External Tools > CLI Client**. Use any of the following tools to connect: | Tool | How to Connect | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MySQL Client | Select**CLI Client > MySQL Client**. Refer to[Connect with MySQL Client](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client.md)for more information. | | SingleStoreEndpoints | Select**CLI Client >SingleStoreEndpoints**. Refer to[SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md)for more information. | For more information on supported CLI clients, refer to [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md). ## SQL IDE Select **\ > Connect > External Tools > SQL IDE**. Use any of the following tools to connect: | Tool | How to Connect | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Sequel Pro | Select**SQL IDE > Sequel Pro**. Refer to[Connect with Sequel Pro](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sequel-pro.md)for more information. | | SQL Workbench | Select**SQL IDE > SQL Workbench**. Refer to[Connect with SQL Workbench](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-sql-workbench.md)for more information. | For more information on supported SQL IDEs, refer to [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md). ## BI Tools Select **\ > Connect > External Tools > BI Tools**. Use any of the following tools to connect: | Tool | How to Connect | | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tableau | Select**BI Tools > Tableau**. Refer to[Connect with Tableau](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau.md)for more information. | | Looker | Select**BI Tools > Looker**. Refer to[Connect with Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker.md)for more information. | | Power BI | Select**BI Tools > Power BI**. Refer to[Connect with Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi.md)for more information. | | Other | Select**BI Tools > Others**. Refer to[Connect with Analytics and BI Tools](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools.md)for more information. | ## Your App Select **\ > Connect > External Tools > Your App**. Use any of the following application tools to connect: | Tool | How to Connect | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Python | Select**Your App > Python**. Refer to[Connect using the SingleStore Python Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md)for more information. | | Node.js | Select**Your App > Node.js**. Refer to[Connect with Node.js](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-node-js.md)for more information. | | Ruby on Rails | Select**Your App > Ruby on Rails**. Refer to[Connect with Ruby](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-ruby.md)for more information. | | Perl | Select**Your App > Perl**. Refer to[Connect with Perl](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-perl.md)for more information. | | Java/JDBC | Select**Your App > Java/JDBC**. Refer to[Connect with Java/JDBC](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc.md)for more information. | | HTTP | Select**Your App > HTTP**. Refer to[Data API](https://docs.singlestore.com/cloud/reference/data-api.md)for more information. | For more information on supported application development tools and programming languages, refer to [Connect with Application Development Tools](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools.md). ## MongoDB® Client You can connect to your Shared cluster from supported MongoDB® tools and drivers using the `mongodb://` endpoint,  select **\ > Connect > External Tools > MongoDB Client**. Here's a sample connection string: ``` mongodb://^:@svc-XXXX-shared-mongo.XXXX.svc.singlestore.com:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true&dbName= ``` For more information on supported tools and drivers, refer to [Supported Tools](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai/#section-idm4635272429529633674406799576.md). ## Query Data from the Cloud Portal You can query your SingleStore database from the [Cloud Portal](https://portal.singlestore.com/). ## SQL Editor You can query the attached database of your Shared cluster using the SQL Editor. Select **Editor** in the left navigation, and then select **Open SQL Editor** on the right. Alternatively, you can select **\ > Connect > SQL Editor**. Refer to [Connect with the SQL Editor](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-the-sql-editor.md) for more information. ## Kai Shell You can run MongoDB® commands on your Shared cluster using [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai.md). Select **Editor** in the left navigation, and then select **Open Kai Shell**. Alternatively, you can select **\ > Connect > Kai Shell**. Refer to [Kai Shell](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai/#section-idm4599119856966433938380761075.md) for more information. ## Notebooks SingleStore Helios offers Notebooks in the Shared cluster that supports SQL and Python development within a secure and seamless connection environment. Refer to [Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) for more information. [Gallery](https://docs.singlestore.com/cloud/container-services/notebooks/using-notebooks/#section-idm4501416375564833538413771196.md) is a collection of notebooks designed for use with SingleStore. You can also access this collection externally through [Spaces](https://www.singlestore.com/spaces/). You can contribute to Spaces by following the steps in the [SingleStore Spaces](https://github.com/singlestore-labs/spaces-notebooks/blob/master/CONTRIBUTING.md) GitHub repository. ## In this section * [About SingleStore Helios](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios.md) * [Compute Clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters.md) * [Cluster Architecture](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/cluster-architecture.md) * [Migrating to Clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/migrating-to-clusters.md) * [SingleStore Helios Sizing Guidelines](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/singlestore-helios-sizing-guidelines.md) * [Migrate from Rockset to SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/migrate-from-rockset-to-singlestore.md) * [SQrL](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/sqrl.md) * [Next Steps and Examples](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples.md) *** Modified at: July 22, 2026 Source: [/cloud/getting-started-with-singlestore-helios/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # About SingleStore Helios SingleStore Helios is a database platform built to enable developers to build and power intelligent applications.  From customer-facing SaaS platforms to internal operational intelligence, SingleStore is specifically designed to deliver low latency, high concurrency, and scalability to many of the world's largest enterprises. This is possible because of a Distributed SQL architecture, support for relational and multi-model data, and a unified storage engine able to handle transactional and analytical workloads simultaneously. SingleStore Helios can be deployed in AWS, Azure, or GCP. ## Design Principles SingleStore Helios is a SaaS data platform designed to deliver extremely low latency and high concurrency to power many of the world's most scalable applications. Using a modern Distributed SQL architecture, SingleStore adheres to structured [design principles](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/design-principles.md) to ensure seamless and highly performant operation. ## Query Execution SingleStore parameterizes and compiles queries to machine code on primary execution, allowing repeat operations to deliver extreme performance. For more details on [query execution](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution.md) take a look at how common execution patterns are handled in a distributed system. ## High Availability & Data Sharding SingleStore handles data availability by ensuring no single point of failure. [Data is sharded](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md) within the cluster, and the storage engine ensures [high availability](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/high-availability.md) and automatic failover in the event underlying cloud instances fail. All of this operates transparently, to ensure data is always available to mission-critical applications. ## Shared Responsibility Model As a SaaS service, SingleStore Helios operates using a [shared responsibility model](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/shared-responsibility.md). Users are responsible for their applications, users, and using best practices for securing access to their data. SingleStore is responsible for the deployment, operation, and support of the data platform. ## Security SingleStore Helios was designed for [security](https://docs.singlestore.com/cloud/security.md) from the ground up, providing advanced features for authentication, encryption, data governance, and compliance. SingleStore has industry-leading security certifications including ISO/IEC 27001, SOC Type 2, and Privacy Shield. It is also fully compliant with CCPA, GDPR, and HIPAA requirements. This allows organizations to simplify data security, reduce operational expenses, and eliminate insider threats. ## FAQs Common [questions and answers](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs.md) about the SingleStore Helios data platform. ## Additional Information [Quickstart: Loading Data and Running Queries](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-data-and-run-queries-into-singlestore.md) - walks you through using the Cloud Portal to create a database, load sample data quickly, and then run some queries against that data. [Designing for Multi-tenant Applications](https://docs.singlestore.com/cloud/developer-resources/guides/designing-for-multi-tenant-applications.md) - explains the various models and trade-offs available for a multi-tenancy SaaS application. Visit for more information about SingleStore Helios. ## In this section * [Design Principles](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/design-principles.md) * [Sharding](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding.md) * [Query Execution](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution.md) * [High Availability](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/high-availability.md) * [SingleStore Helios Region Types](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-region-types.md) * [SingleStore Helios Editions](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md) * [Shared Responsibility](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/shared-responsibility.md) * [SingleStore Helios FAQs](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs.md) *** Modified at: May 11, 2026 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Design Principles SingleStore distributed architecture is designed to be straightforward, simple, and fast. The architecture is designed around a few key principles: | Principle | Description | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Performance first | SingleStore is designed to enable high throughput on concurrent workloads, and is a highly scalable distributed system. Thecluster (A collection of SingleStore aggregator and leaf nodes.)can be scaled out at any time to provide increased storage capacity and processing power. SingleStore can process billions of rows per second onclusters of commodity hardware. | | Built for online database operations | This enables you to scaleclustercapacity and query performance online by adding additionalnodes to acluster. SingleStoreclusteroperations are dynamic - you will never have to restart anynodes while managing thecluster. | | No single point of failure | By running theclusterwith redundancy, you ensure that every partition of data is backed up to a hot standby. In the event of anynode failure, SingleStore will automatically fail over replica partitions. | | Simple and powerfulclusterscaling | The distributed system allows you to seamlessly scale deployments up or down while powerful automation handles the underlying infrastructure transparently. | | No implicit data movement | In the event of a failure, SingleStore promotes just enough replica partitions to put theclusterback online. SingleStore will never rebalance or move data around implicitly. | *** Modified at: June 22, 2022 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/design-principles/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/design-principles/) (An index of the documentation is available at /llms.txt) --- # High Availability SingleStore is highly available by default. It ensures high availability (HA) by storing data redundantly in a set of nodes, called *availability groups*. SingleStore supports two availability groups. Each availability group contains a copy of every partition in the system—some as primaries and some as replicas. As a result, SingleStore has two copies of your data in the system to protect the data against single node failure. The primary partitions are distributed evenly on nodes across the cluster. The primary partitions on every node in an availability group have their replicas spread evenly among a set of nodes in the opposite availability group. The even distribution of replicas ensures that a failover distributes the additional load from the node failure uniformly across the cluster. As read queries are sent to the primary partitions, balanced load distribution prevents the overloading of a single node with the newly promoted primary partitions. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0d85dc1e7ba3891c/6a7adb90bdcff0b1e4c3782e/diagram_load-balanced-partition-placement-RD0MCS.png) In the event of a node failure, SingleStore automatically promotes the appropriate replica partitions on the node’s pair into primary partitions, so that the databases remain online. The additional workload from the node failure is spread evenly among multiple other nodes, which contain the replica copies of the primary partitions in the failed node. However, if all of the machines fail, then data will be unavailable until enough machines are recovered or until the cluster is recreated from scratch. The following diagrams illustrate the partition distribution before and after a cluster failover. In the first diagram, the primary partitions are distributed evenly across nodes. Replica copies of the primary partitions in an availability group are placed evenly across the nodes in the opposite availability group. For example, `db_0` has a replica on Node 2, while `db_1` has a replica on Node 4. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt13ccd63bdfe44268/6a7adac45967e510cd5d1e11/load-balanced-failover-ie241d.png) If Node 1 fails in this setup, SingleStore promotes the replica of `db_0` on Node 2 to primary and the replica of `db_1` on Node 4 to primary. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0e590fa463ca6878/6a7adae7e88c655c08fffde7/load-balanced-failed-node-fLTKuB.png) When a node comes back online, it will be automatically reintroduced to the cluster. As for the partitions on the node, they are either caught up or rebuilt from scratch. In SingleStore Helios, one load balancer is set up for the Master Aggregator (MA) and then a second load balancer for the child aggregators (CAs). The load balancer for the CAs does equal distribution of traffic between CAs. **Note:** In AWS, the Network Load Balancer (NLB) works better than the classic Elastic Load Balancing (ELB). You can use a single NLB, but different ports for MA and CA targets. ## Availability Zone (AZ) An availability zone (AZ) is a data center with its own power, cooling, and network connections. An AZ is physically separated from other AZs, so local failures, like a fire that destroys a whole AZ, will not affect other AZs. AWS, Azure, and Google Cloud Platform (GCP) all provide multiple AZs in each region where they operate. ## Single and Multi-AZ High Availability High availability is supported in Single-AZ and Multi-AZ configurations, which determine whether SingleStore is deployed in one or multiple cloud availability zones. Multi-AZ failover is supported only in SingleStore Helios. * **Single-AZ**: By default, SingleStore is deployed with high availability in a single cloud availability zone, providing data redundancy and automatic recovery from cloud instance failures within that zone. * **Multi-AZ**: When enabled, Multi-AZ deploys SingleStore across multiple cloud availability zones, with each availability group placed in a separate zone. This makes data resilient to both cloud instance failures and the loss of an entire availability zone. *** Modified at: May 18, 2026 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/high-availability/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/high-availability/) (An index of the documentation is available at /llms.txt) --- # Query Execution ## Distributed DML Query Execution In this topic we will look at common DML query patterns and how they are executed through the distributed system. You can use the [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) command to examine a query’s node-level query plans. Let’s assume the following schema: ```sql CREATE TABLE a ( a1 int, a2 int, a3 int, SHARD KEY (a1, a2), KEY (a3) ); CREATE TABLE b ( b1 int, b2 int, b3 int, SHARD KEY (b1, b2) ); CREATE REFERENCE TABLE r ( r1 int, r2 int, PRIMARY KEY (r1), KEY (r2) ); ``` ## Index Matching Matching on the shard key or the secondary index. ## Matching the Shard Key If you specify an equality on every column in the shard key, then the node will direct the query to exactly one partition. Most queries do not fall into this pattern; instead, the node must send queries to every partition in the cluster for intermediate results and then stitch them together. These queries are sent to one partition: ```sql SELECT * FROM a WHERE a1 = 4 AND a2 = 10; SELECT a3, count(*) FROM a WHERE a1 = 4 AND a2 = 10 GROUP BY a3; ``` These queries are sent to all partitions: ```sql SELECT * FROM a WHERE a1 = 4; SELECT * FROM a WHERE a1 = 4 OR a2 = 10; SELECT * FROM a WHERE a1 IN (4, 5) AND a2 IN (10); ``` ## Secondary Index Matching If your query uses a secondary (non-shard) index, then the node must send the query to every partition in the cluster. Locally, each partition’s table will use its part of the secondary index to speed up the query. While the overall performance of the query is dictated by the seek and scan time of these indexes, the fact that the query must be sent everywhere in the cluster can increase the variance (and therefore overall latency) of the query. This query matches the secondary index on the column a3: ```sql SELECT * FROM a WHERE a3 = 5; ``` ## No Index Matching Queries that do not match any index perform a full table scan on all partitions. From the perspective of the node, these queries are the same as queries that match a secondary index. ## Distributed Joins ## Aligning Shard Keys for Performance Aligning the shard keys of large tables enables more efficient joining. It is possible to perform arbitrary distributed joins across any tables and along any column. However, if you join two tables with identical shard key signatures along that shard key, the joins will be performed local to the partitions, reducing network overhead. ```sql CREATE TABLE users ( id BIGINT AUTO_INCREMENT, user_name VARCHAR(1000), account_id BIGINT, PRIMARY KEY (id) ); CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT, account_id BIGINT, user_id BIGINT, page_id INT, ts TIMESTAMP, SHARD KEY (user_id), PRIMARY KEY (click_id, user_id) ); ``` In this example, `id` is the shard key of the `users` table, and the shard key on the `clicks` table has the same signature (a single BIGINT). These queries join locally without network overhead: ```sql SELECT * FROM users INNER JOIN clicks ON users.id = clicks.user_id WHERE clicks.page_id = 10; SELECT avg(c1.ts - c2.ts) FROM clicks c1 INNER JOIN clicks c2 ON c1.user_id = c2.user_id WHERE c1.page_id > c2.page_id; ``` Whereas this query will stream rows between nodes: ```sql SELECT u.account_id, count(distinct user_id), count(1) FROM users u INNER JOIN clicks c ON u.account_id = c.account_id GROUP BY u.account_id; ``` If you identify your data layout and join patterns in advance, this technique can be an extremely effective way to run performant joins between distributed tables. For more information about how queries execute, see [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md). ## Write Queries ## `UPDATE` and `DELETE` Queries The `WHERE` clause in an `UPDATE` or `DELETE` query is optimized the same way as the `WHERE` clause in a `SELECT` query. If the predicate matches the shard key exactly then the query is routed to a single partition. ## `INSERT` Queries SingleStore Helios executes `INSERT` queries by analyzing the insert values relevant to the shard key and routing the query to the corresponding partition. For example, `INSERT INTO a (a1, a2, a3) VALUES (1, 2, 3)` would compute the hash value of `(1, 2)` and map this value to the appropriate partition. If you are bulk inserting data with `INSERT` queries, then you should take advantage of the multi-insert syntax: `INSERT INTO a (a1, a2, a3) VALUES (1, 2, 3), (2, 3, 4), ...`. The node will chop up the multi-insert into single-partition insert queries and run them in parallel across the cluster. This technique enables your application to combat the inherent latency of running in a distributed system. *** Modified at: September 24, 2021 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution/) (An index of the documentation is available at /llms.txt) --- # Sharding Every distributed table has exactly one shard key. A shard key can contain any number of columns. When you run `CREATE TABLE` (this command creates a columnstore table which is the default table type for SingleStore Helios) or `CREATE ROWSTORE TABLE` (this command creates a rowstore table) to create a table, you can specify a shard key for the table. If you choose not to specify a shard key: * If the table has a primary key, then the shard key will be whatever the primary key is. * If the table has neither a shard key nor a primary key, then the engine will perform keyless sharding (A type of database partitioning that divides a database into smaller more easily manageable parts.) for that table, which means the engine randomly distributes rows of the table across database partitions (A partition contains a subset (a shard) of a database’s data. Each partition holds a vertical slice of data, distributed as per a hashing algorithm on the primary key or randomly for keyless sharded databases.). A table’s shard key determines in which partition a given row in the table is stored. When you run an`INSERT` query, the node computes a hash function of the values in the column or columns that make up the shard key, which produces the partition number where the row should be stored. The node then directs the `INSERT` operation to the appropriate node machine and partition. ![SingleStore Sharding](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltd3059278c94e2610/6a7ada8afc63ab6889641813/sharding-xywila.png) [DML](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/query-execution.md) queries that fully match the shard key can be routed directly to a single partition on a single node server. Group-by queries where the set of keys are guaranteed to not overlap between partitions can be executed in parallel on the node, with the results streamed back without any additional processing on the node. Any two rows with the same shard key value are guaranteed to be on the same partition. SingleStore requires that the set of columns in the `PRIMARY` or `UNIQUE` index needs to be a superset of the shard key columns, meaning that the primary / unique key contains all columns in the shard key. Rows that might have a primary key conflict thus map to the same partition so uniqueness can be enforced locally. Choosing an appropriate shard key is important for minimizing [data skew](https://docs.singlestore.com/#section-idm4501210432494432654843416521.md). ## Types of Shard Keys ## Default Shard Key If no shard key is specified, the primary key is used as the shard key. If no primary key or shard key is specified for a given table, the shard key defaults to being keylessly sharded. To accomplish this, you can either specify no keys, or use `shard key()` with nothing between the parenthesis as follows. ```sql CREATE TABLE t1(a INT, b INT); CREATE TABLE t1(a INT, b INT, SHARD KEY()); ``` In most cases, keyless sharding will result in uniform distribution of rows across partitions. The exception to this is `INSERT … SELECT` statement involving sharded tables, in which case, data is inserted locally into the same partition as the source in order to avoid data redistribution. ## Primary Key as the Shard Key If you create a table with a primary key and no explicit shard key, the PK will be used as the shard key by default. ```sql CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT PRIMARY KEY, user_id INT, page_id INT, ts TIMESTAMP ); ``` ## Non-Unique Shard Key The explicit syntax for a shard key is `SHARD KEY (column_list)`, where column\_list is zero or more columns separated by commas. For example: ```sql CREATE TABLE clicks ( click_id BIGINT AUTO_INCREMENT, user_id INT, page_id INT, ts TIMESTAMP, SHARD KEY (user_id), PRIMARY KEY (click_id, user_id) ); ``` In this example, any two clicks by the same user will be guaranteed to be on the same partition. You can take advantage of this property in query execution for efficient `COUNT(DISTINCT user_id)` queries, which knows that any two equal `user_id` values will never be on different partitions. Note that even though `click_id` will be unique, we still have to include `user_id` in the primary key. ## Choosing a Shard Key See the [Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md) guide for information on how to choose a shard key. ## Data Skew Data skew occurs when data is unevenly distributed across partitions. It's important to choose a shard key that minimizes data skew in order to get the fastest query performance. Refer to [Detecting and Resolving Data Skew](https://docs.singlestore.com/cloud/create-a-database/detecting-and-resolving-data-skew.md) for more information. ## Related Topics * Training: [SingleStore Sharding and Shard Keys](https://training.singlestore.com/learn/course/internal/view/elearning/680/sharding-and-shard-keys) * [Shard Keys](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures/shard-keys.md) * [Understanding Shard Key Selection](https://docs.singlestore.com/cloud/create-a-database/understanding-shard-key-selection.md) *** Modified at: September 16, 2025 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/sharding/) (An index of the documentation is available at /llms.txt) --- # Shared Responsibility SingleStore Helios has built in security controls that make it a secure environment to run customer workloads. However the responsibility of keeping it secure is shared between the user and SingleStore. SingleStore Helios is designed with strong security by default so that there is minimal overhead on the user. The default configuration includes encryption at rest, encryption in transit, removal of public access, and deployment within strong network boundaries. Users are responsible for configuring the necessary levels of control which is based on the security posture of their organization. ## Shared Responsibility Model The following table outlines the responsibilities of the customer and SingleStore for a SingleStore Helios deployment in Managed regions: ## Cloud Infrastructure Physical Security | **Customer**
  • Select the cloud provider and the region of choice.
| **SingleStore**
  • Provision the requested clusters in a private network.
  • Provision all additional configurations described by users.
  • Secure the infrastructure and networks using best practices.
| | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ## Customer Data, Accounts, and Identities | **Customer**
  • Create and manage customer data.
  • Add user accounts and access using identities.
| **SingleStore**
  • Provide secure access and storage to customer data.
  • Provide secure connectivity to the platform to ensure confidentiality, integrity, and authentication for customer data in motion.
| | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Network Isolation and Connectivity | **Customer**
  • Configure the network connectivity, including Firewall, DNS, Private Networking, and IP allowlisting between the user and SingleStore account.
| **SingleStore**
  • Enforce network security restrictions as per configurations made by the customer.
  • Provision resource for private networking.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ## SingleStore Database Access | **Customer**
  • Configure user authentication.
  • Add roles and privileges for users.
  • Manage certifications and JWKS setups for clusters.
  • Manage IAM roles on cloud resources to be used by SingleStore Helios.
| **SingleStore**
  • Provide Role-Based Access Control (RBAC) as part of the platform.
  • Provide integration with MFA and other SSO tools.
  • Provide secure identity management capabilities and access to user accounts on the platform.
  • Support secure token-based authentication/authorization.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## API Controls/Access | **Customer**
  • Manage and configure API keys.
| **SingleStore**
  • Generate API keys.
  • Implement API access.
| | -------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ## Data Encryption (in Transit and at Rest) Customer-Managed Encryption Keys (CMEK) is only supported on Managed regions. | **Customer**
  • Set the TLS version to be used.
  • For CMEK: Configure cloud provider KMS and key policy according to the customer’s own requirements, and then manually configure CMEK on SingleStore Helios.
| **SingleStore**
  • Enable default encryption of data at rest and in motion with cloud provider managed keys.
  • Connect to the KMS and use keys for encryption at rest.
  • For CMEK: Connect to the customer-specified KMS and use keys for encryption of data at rest.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Granular Auditing | **Customer**
  • Configure audit levels and audit log destinations.
| **SingleStore**
  • Stream audit logs to external resources based on user configuration.
  • Enable audit logging for the database automatically.
  • Monitor the platform's audit logs.
| | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Performance Monitoring/Alerting | **Customer**
  • Configure real-time alerts and performance thresholds.
  • Configure external tools for monitoring and alerting.
  • Access to metrics and logs via Grafana dashboards.
| **SingleStore**
  • Configure performance analysis and monitoring capabilities.
  • Monitor the platform’s performance logs and alerts.
| | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Security Patches and Maintenance | **Customer**
  • Ensure that the client software used to interact with the platform is up-to-date and patched.
| **SingleStore**
  • Automatically apply security patches and updates.
  • Run internal vulnerability and patch management processes.
| | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ## High Availability and Disaster Recovery | **Customer**
  • Can create and manage own custom backups in accordance with internal backup and disaster recovery policy.
  • Configure backup and recovery capabilities and provisions supported by the platform.
| **SingleStore**
  • SingleStore stores data in durable object storage for recovery in case of unexpected disaster.
  • SingleStore provides self-serve recovery steps (based on the purchased edition).
  • Implement automated failover and replication mechanisms.
| | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Application Security | **Customer**
  • Validate and check user-defined functions (UDFs) and code written to interface with external functions for security issues.
  • Validate the security of third-party services to leverage on SingleStore Helios computing capabilities or through integrations.
  • Secure system access for users both inside and outside the customer's environment.
| **SingleStore**
  • Provide a secure operating and computing environment.
  • Run incident detection and response mechanisms internally.
  • Manage network egress and ingress at the network layer and control access to data.
  • Validate the security of the software supply chain used by CI/CD procedures and tools.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Secrets | **Customer**
  • Ensure proper access control to secrets configured within the platform.
  • Manage the lifecycle of secrets as well as their end-to-end distribution.
| **SingleStore**
  • Securely store and encrypt customer secrets.
| | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | ## Compliance | **Customer**
  • Configure the environment(s) to meet the requirements for the customer’s own compliance and regulatory needs.
  • If the customer needs to store and manage PHI data on SingleStore Helios, a BAA must be set up with SingleStore.
| **SingleStore**
  • Maintain compliance and uphold Information Security and Data Protection standards and requirements that apply to our product and business (namely ISO27001 and SOC 2 Type II).
  • Support compliance inheritance of HIPAA.
| | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## AI Usage | **Customer**
  • Ensure secure deployment of Generative AI applications and the responsible use of data leveraged in AI-powered features provided by SingleStore.
  • Implement human oversight in AI-enabled business workflows.
  • Where selection of underlying AI models is required, it is the customer's responsibility to make and validate the choice.
| **SingleStore**
  • Provide a secure platform for enabling Generative AI applications that seamlessly integrate with SingleStore.
  • Ensure compliance with data protection and regulatory standards, with continuous monitoring and adaptation as AI-related regulations evolve.
  • Evaluate Generative AI technology to assess bias, security vulnerabilities, accuracy, and safety.
| | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ## Responsibility Matrix The following can be used as a quick reference to the shared responsibilities of the customer and SingleStore. ## Cloud Management | **Action** | **SingleStore** | **Customer** | | ----------------------- | --------------- | ------------ | | VPC | ✔ | | | EC2 instance management | ✔ | | | Kubernetes management | ✔ | | | S3 buckets management | ✔ | | | SingleStoreprovisioning | ✔ | | ## Upgrades and Security | **Action** | **SingleStore** | **Customer** | | ---------------------------------------- | --------------- | ------------ | | SingleStoreupgrades | ✔ | | | Software vulnerability remediation | ✔ | | | Infrastructure vulnerability remediation | ✔ | | | Scaling | ✔ | | ## Networking | **Action** | **SingleStore** | **Customer** | | ------------------- | --------------- | ------------ | | External Routing | ✔ | | | K8 internal Routing | ✔ | | | Firewall | | ✔ | | DNS | ✔ | | | Load Balancer | ✔ | | ## Access Control | **Action** | **SingleStore** | **Customer** | | --------------------------- | --------------- | ------------ | | IAM role, service accounts | ✔ | | | Access control and auditing | ✔ | ✔ | ## Availability | **Action** | **SingleStore** | **Customer** | | ------------------ | --------------- | ------------ | | DR | ✔ | | | Availability (SLA) | ✔ | | ## Support | **Action** | **SingleStore** | **Customer** | | ------------- | --------------- | ------------ | | Logging | ✔ | | | Audit logging | ✔ | ✔ | | Monitoring | ✔ | | | Break glass | ✔ | | *** Modified at: April 22, 2026 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/shared-responsibility/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/shared-responsibility/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Editions SingleStore provides the following editions of SingleStore Helios to meet the unique needs of a variety of customer workloads. * Shared Edition * Standard Edition * Enterprise Edition Refer to [Helios Pricing](https://www.singlestore.com/pricing/) to view and compare the features supported by each Edition. > **📝 Note**: To purchase a different edition or upgrade/downgrade your current SingleStore Helios edition, please contact your sales representative. ## Shared Edition Shared Edition provides a basic entry-point to get started for low-throughput and prototype applications for evaluation, development, POCs, or non-production testing. Refer to [Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition.md) for information on Shared Edition and the configuration and operational limits for Shared clusters. ## Standard Edition Standard Edition is designed for general-purpose production workloads that demand high performance, scalability, and access to all the core features provided by SingleStore. Standard Edition can be deployed across multiple cloud providers and regions. ## Key Capabilities of Standard Edition * **Full relational and multi-model storage engine capabilities**, including MySQL compatibility, JSON, Time-series, Full-text Search, Vector, and Geospatial. * **High Availability** with load-balanced failover enables transparent operation even in the event of underlying cloud resource failures. * **Multi-AZ failover** support enables deployments to remain operational even if an entire cloud availability zone becomes unavailable. * **Continuous backups** enables reliable online Point-in-Time Recovery (PITR) to exactly 00:00 UTC (by default) on any day within the retention window in the event of failure. * **Resource governance** provides user-configurable control for resource management and workload isolation. ## Enterprise Edition Enterprise Edition is designed for mission-critical workloads that require advanced recovery features, in addition to high performance and scalability, to ensure continuous operation. Enterprise Edition can be deployed across multiple cloud providers and regions, and it supports all the core features provided by SingleStore. ## Key Capabilities of Enterprise Edition All Standard Edition features, plus: * **Continuous backups** the online PITR in this edition allows data restoration to any point in time, down to the microsecond, with configurable retention periods. * **Smart Disaster Recovery (SmartDR)** enables continuous, asynchronous replication between geographically distributed primary and secondary regions, ensuring resilience against regional outages. * **Customer-Managed Encryption Keys (CMEK)** enables complete control over database encryption by integrating SingleStore with the cloud provider’s native Key Management Service (KMS). * **Audit Logging** provides detailed logs of both Control Plane and database-level operations, supporting compliance, security audits, and operational transparency for mission-critical environments. ## Projects Projects are used to organize clusters and databases for a specific workload or set of workloads. This allows users with multiple workloads to easily group and manage resources. Clusters and databases are assigned to a project when they are created and cannot be moved between projects. Edition is set at a project level, and an organization may have multiple projects, each with a unique edition. All clusters and databases in a project include features available to the selected edition. For example, databases in a project configured with Enterprise Edition have Point in Time Recovery (PITR) available, but databases in a project configured with Standard Edition only support restoring from daily continuous backup checkpoints. ## In this section * [Shared Edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/shared-edition.md) *** Modified at: November 27, 2025 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios FAQs ## In this section * [General](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/general.md) * [Clustering](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/clustering.md) * [Query Compilation](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/query-compilation.md) * [Query Execution](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/query-execution.md) * [Durability](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/durability.md) * [Import and Backup](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/import-and-backup.md) * [SingleStore Helios and Spark](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/singlestore-helios-and-spark.md) * [Miscellaneous](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/miscellaneous.md) *** Modified at: April 5, 2023 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-faqs/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Region Types SingleStore provides flexible deployment options across multiple regions to meet your infrastructure, security, and compliance needs. "Regions" refer to geographical locations and infrastructure zones provided by AWS, Google Cloud (GCP), and Microsoft Azure where SingleStore can be deployed. SingleStore Helios can be deployed in the following region types: * Managed * Dedicated Refer to [Helios Pricing](https://www.singlestore.com/pricing/) for a list of features supported by Managed region types. Refer to [SingleStore Helios Editions](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md) for more information on Editions. ## Managed Regions Managed regions are the most secure, fully managed, and feature-rich way to deploy SingleStore. These regions deliver the best overall TCO (Total Cost of Ownership) and provide full access to all the SingleStore capabilities. Managed regions are available across AWS, Azure, and GCP. Managed regions support Shared, Standard, and Enterprise editions of SingleStore. Refer to [SingleStore Helios Service Level Agreement](https://docs.singlestore.com/cloud/support/singlestore-helios-service-level-agreement.md) and [Certifications](https://docs.singlestore.com/cloud/security/#section-idm4586200488270432631191056934.md) for the SLA and certifications provided by SingleStore for Managed regions. If a region is not currently available or supported, contact your SingleStore account representative or SingleStore Support. ## Dedicated Regions Dedicated regions support the same feature set and functionality as Managed regions, but they are specifically designed for organizations that require a single-tenant data plane. Dedicated regions offer the same level of security as Managed regions. Dedicated regions maintain the same Helios availability SLAs and compliance certifications as Managed Regions, but consume compute credits at a faster rate. *** Modified at: April 21, 2026 Source: [/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-region-types/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-region-types/) (An index of the documentation is available at /llms.txt) --- # Cluster Architecture ## Unique Design SingleStore is a real-time hybrid transaction/analytics processing (HTAP) database designed with a distributed SQL architecture. This means that compute can be scaled out using a clustered architecture, rather than only scaling up using larger machines. Clusters further enhance this distributed architecture by freeing databases from the confines of a single workload, delivering true separation of compute and storage. ## Cluster Architecture Clusters are built using the native data replication engine built into SingleStore. The isolated pools of compute are clustered on top of cloud hardware. These compute pools have dedicated vCPU, memory, and persistent cache to deliver immediate query responsiveness, while operating on top of scale-out object storage. Combined with SingleStore's query code generation and tiered Universal Storage architecture, this allows clusters to deliver extremely low latency query response, highly concurrent access and fast parallel streaming ingest while automating the movement of data across workloads. ## Implementation Each cluster is a part of a Group, and Databases can be attached or detached from any cluster within a group. Each group can contain up to five clusters. Groups are deployed in a cloud provider and region. An organization can have many groups deployed across multiple cloud providers and regions. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8027a0af35c9d938/6a7ada55c2cc09de1c23db41/workspace-groups-txT45N.png) Databases can be attached to multiple clusters simultaneously. Each database can have one R/W attachment and many R/O attachments. This allows data to be shared across workloads, providing scalability and isolation. The R/W attachment writes logs/blobs to shared storage and sends logs to the R/O attached clusters asynchronously. The R/O attached clusters replay these logs, creating local blobs. For example, a customer-facing SaaS application is running on cluster 1 and writing data into DB01. An internal telemetry application running on cluster 2 is reading data from DB01. In this scenario, DB01 has an R/W attachment to cluster 1 and a R/O attachment to cluster 2. Attachments are created on each database, and it is possible for a cluster to have a combination of R/W and R/O attachments. Writes made to a database from a cluster with an R/W attachment will appear on clusters with R/O attachments near instantaneously. The transaction log tail is sent to the replica and applied there asynchronously in real time. Thus, R/W-R/O sync is eventually made consistent. All new writes to a database are typically replicated immediately, allowing clusters with R/O attachments to serve real-time workloads without waiting for data to load or going through complex ETL processes. There are two situations in which replay latency can happen: * Load replay needs the CPU to go through the log, so if this part is slowed down, replay will fall behind. * SingleStore processes and saves transactions in a certain order, but it does not always commit them in that order. For example, if you have a big load followed by a small load, the system might finish the small task first. However, when it needs to replay or sync this on a replica, the replica has to wait for the big load to finish before replaying the small one. When attaching a database to a cluster as R/O, all new data written will be replicated to the new cluster. When sizing the cluster ensure that the selected size can accommodate the size of the working data set. A very small cluster attaching a database with a large working set will not have sufficient resources to process the incoming replication stream or reads against the database. For example, if a rowstore table is replicated to a small R/O cluster, replay on the R/O cluster will stop once the memory limit is reached, and the database can become unrecoverable. If replay stops, then eventually the write-side may also stop. An atomic write for two databases within a single cluster is not supported. *** Modified at: September 11, 2025 Source: [/cloud/getting-started-with-singlestore-helios/cluster-architecture/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/cluster-architecture/) (An index of the documentation is available at /llms.txt) --- # Compute Clusters Clusters are compute pools that allow customers to run multiple isolated workloads while providing low-latency access to shared data. This ensures applications are operating on real-time data. ## Cluster Benefits * Allows granular scalability and isolation of compute resources * Eliminates the cost and latency of moving data between multiple workloads * Removes consistency challenges by eliminating data silos ## Managing Clusters A cluster is owned by an organization, and members of the organization can create, resize, and terminate clusters if they have the appropriate permissions. ## Create a Cluster Cluster s can be created through the [Cloud Portal](https://portal.singlestore.com/) UI or the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md). The following functions are available via these management interfaces. * Create * Connect * Create or drop a database * Attach or detach a database > **📝 Note**: The cluster `admin` password is configured when the cluster is first created. SingleStore recommends that you save this password as you will need it to log into the database outside of the Cloud Portal, such as when using a SQL client. This is also referred to as the “Master Username” in the Cloud Portal, which is `admin` by default. ## Connecting to a cluster Each cluster provides a connection endpoint that you can use to connect from a client, an IDE, or an application. You can also use the SQL Editor in the Cloud Portal to connect and run queries directly. SingleStore Helios provides a Classless Inter-Domain Routing (CIDR) allowlist to restrict the networks that can access a cluster. This can be defined by editing the IP Allowlist for a cluster. Refer to [IP Address Allowlisting](https://docs.singlestore.com/cloud/security/firewall.md) for more information. ## Resize or Scale a Cluster Cluster can be resized or scaled by changing the size or scale factor in the Cloud Portal or Management API. Resizing a cluster changes the base compute configuration to provision greater or fewer resources (vCPU & memory). This option should be used when the change is expected to be persisted for a period of time. Data is redistributed within the cache for optimal performance. When the operation is running there may be a temporary reduction in performance due to resources being dynamically added or removed. For large deployments with heavy active workloads the time required to complete the resizing operation may increase as a larger volume of active data needs to be redistributed within the deployment. The resize operation and data rebalancing will continue in the background while the workload is running, so no extra steps need to be taken to ensure the job completes.Resizing can be performed through the Management API by using `WorkspaceUpdate` size. Scaling a cluster adds additional compute and memory without changing the base configuration. This is ideal for situations where additional compute is needed for shorter periods of time. No data is moved in this configuration, allowing operations to complete more quickly. Scaling can be performed through the Management API by using `WorkspaceUpdate` `scaleFactor`. ## Autoscaling Autoscaling is ideal for dynamic workloads where the user does not know when peaks in workload may occur and can be turned on or off for each compute deployment independently. Autoscaling tracks the active workload and automatically scales based on compute and memory usage. When the workload requests more vCPU or memory than is available, autoscaling will automatically add compute resources on the fly. If the workload decreases and the additional compute is no longer needed, autoscaling will return to the base size. While many databases limit autoscaling to read-replicas, SingleStore has implemented autoscaling to provide both enhanced write and read performance. When configuring autoscaling users can turn the feature on or off, and set the maximum amount of vCPU and Memory to be provisioned (2x or 4x of the base amount). If set to 4x,  autoscaling increases resources incrementally, first scaling to 2x, and if necessary, then to 4x. The scaling threshold checks are the same (for example, 1x & 2x checks are exactly the same as 2x & 4x). The same applies to scaling down. It first scales from 4x to 2x and then to 1x. Autoscaling support varies by cluster size: * S-00 to S-32: Supports autoscaling up to 4x. * S-40: Supports autoscaling up to 2x. * S-40 to S-72: Supports autoscaling up to 2x. * S-80 and above: Does not support autoscaling. The default settings for autoscaling may not fit every workload. Some workloads can benefit from a shorter sample duration, while others benefit from a longer duration to prevent unnecessary scaling. Autoscaling provides three sensitivity levels to handle a workload. The default recommended setting is Normal. 1. Low: This is a more conservative scaling, which uses 15 min sampling and 30 minute cooldown. 2. Normal: This is the standard configuration, which uses 5 min sampling and 10 minute cooldown. 3. High: This level is the most sensitive scaling, which uses 3 min sampling and 5 minute cooldown. The Low setting can be used when scaling should happen only after sufficient workload has been running for the 15-minute sampling period. This prevents the occurrence of scaling up when there are short spikes of load followed by a drop in CPU utilization. The High setting is recommended when workloads are most dynamic and scaling should occur as often as needed, with minimal cooldown before returning to the base scale factor. ## Cache Configuration The Cache Configuration allows SingleStore to leverage greater volumes of Persistent Cache to increase the amount of data (the working set) that can be accessed with extremely low latency.Increasing the cache configuration, for example from “1x” to “2x” or “4x”), will increase the overall volume of the cache, and automatically distribute data within the cache. As data is automatically redistributed for optimal performance, typically this operation will take between minutes and hours, depending on the volume of data. This operation runs online and data is available throughout the reconfiguration process. Cache configurations can be increased or decreased as desired. Cache Configuration can be updated through the Management API with `WorkspaceUpdate` `cacheConfig`. Refer to [Management API Overview](https://docs.singlestore.com/cloud/reference/management-api.md) for more information. ## Suspend and Resume a Cluster Clusters do not need to be running when there is no active workload querying a database. To save cost, clusters can be suspended when inactive. Options: * Suspend: Compute resources will not be billed while a cluster is suspended * Resume: Resume re-provisions compute resources for the cluster * Auto-suspend: Suspend after a set period of time, or after a period of inactivity Databases are retained even when clusters are suspended. ## Terminate a Cluster Clusters can be terminated when they are no longer needed. ## Post-Termination Data Retention Encrypted data from terminated deployments is automatically and securely purged from all systems on termination, and any administrative backup data is purged within 14 days. Hence, it is not possible to recover data from a terminated deployment. If you need to retain data post-termination, SingleStore recommends that you back up your data to an external object store (such as a bucket). ## Updates Scheduled updates are applied to each cluster based on the update window assigned to the group. Updates are applied to all clusters in a group simultaneously during the scheduled window. Workloads that receive updates independently should be run in separate cluster groups to ensure independent schedules. For example, development and production clusters should run in separate groups, thereby ensuring updates are applied to development before production. ## Opt-in to Preview Features & Updates When creating a cluster, this option enables the latest features and updates to be applied as soon as they are available. The configuration is suitable only for non-production deployments and cannot be changed after creation. ## Usage Examples ## Example Workloads * Run a SaaS application and operational analytics application on shared production data * Deploy a cluster for batch reporting or one-time jobs on existing data, and terminate it when the work is complete * Use application data from production to power new AI applications ## Sharing Databases Across Applications ## Example: Customer App Plus Operational Analytics Reporting In the following examples, S-XX indicates the cluster size; the number of vCPUs and amount of memory available for a workload. Additional clusters can be quickly created and attached to the same database whenever there is a need. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9894862842612b4d/6a7adb6de88c6575d3fffe07/workspace-example-one-gm8I97.png) * Serve the customer application from "App Cluster (App WS)" (R/W). * Create a temporary "Reporting Cluster (Reporting WS) " to generate real-time insights. * Attach the database for reporting (R/O). * Run the reporting application. *** Modified at: May 22, 2026 Source: [/cloud/getting-started-with-singlestore-helios/compute-clusters/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters/) (An index of the documentation is available at /llms.txt) --- # Migrate from Rockset to SingleStore ## Overview Rockset is a real-time analytics database that was recently acquired by OpenAI. As a result of this acquisition, all existing customers will be transitioned off of Rockset and will need to find an alternative solution as detailed in the [Rockset FAQ](https://docs.rockset.com/documentation/docs/faq). Rockset is popular due to its ability to store schemaless JSON data, offer fast ingestion, run real-time analytics on immense volumes of data, handle upserts, and support complex queries like JOINS. SingleStore checks all of these boxes while providing  an even greater ability to build intelligent applications with powerful vector and search capabilities, and manage enterprise data with key security and manageability features. SingleStore is a great alternative for your application needs and this guide will help you through the migration process. > **📝 Note**: Please [fill out this form](https://www.singlestore.com/contact/) to have SingleStore assist you with your Rockset migration. ## Rockset Key Concepts and How They Map to SingleStore ## Ingestion for Data Sources Rockset supports ingestion from a variety of data sources including streaming services, databases, and object stores. SingleStore supports most of the sources supported by Rockset along with additional sources. SingleStore provides native high-speed ingestion with [pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md), which support one-time loads up to massively-parallel incremental updates. | **Data Source** | **Rockset** | **SingleStore** | | -------------------- | ----------- | --------------- | | Apache Kafka | ✔ | ✔ | | MySQL | ✔ | ✔ | | MongoDB® | ✔ | ✔ | | Amazon S3 | ✔ | ✔ | | Amazon MSK | ✔ | ✔ | | Google Cloud Storage | ✔ | ✔ | | Azure Blob storage | ✔ | ✔ | | Confluent Cloud | ✔ | ✔ | | HDFS | | ✔ | | Iceberg + Catalog | | ✔ | Partner integration solutions for: * Azure Event Hubs * Microsoft SQL Server * Oracle * PostgreSQL ## Ingest Transformation Rockset transforms data during ingestion. SingleStore supports data transformation through [SQL-based procedures](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/writing-efficient-stored-procedures-for-pipelines.md), which can effectively achieve the same behavior. These transformations are applied to both the initial data load and to new data that is ingested into SingleStore. ## Collections A set of Rockset documents is a collection, which is similar to tables in a relational database like SingleStore. Rockset allows querying this data either via SQL or through Query Lambdas. Similarly, SingleStore allows the data stored in relational tables to be queried via SQL, [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md) (for MongoDB® data sources), or a [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) that allows HTTP API access to SingleStore databases. ## Aliases Aliases are used to associate multiple names with Rockset collections, which is a concept that  SingleStore does not support. ## Views Views are virtual collections defined by SQL queries. A view’s SQL query may reference nothing, or may reference other views, collections, or aliases. SingleStore also supports [views](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md). ## Workspaces Workspaces are containers that hold Rockset resources such as collections, Query Lambdas, views, aliases,  and even other workspaces. SingleStore has a highly advanced version of [clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters.md) that are elastic, with support for auto-scaling, and that provide complete role-based access control (RBAC), security, and isolation. Clusters in SingleStore can be created from either a UI or via the [Management API](https://docs.singlestore.com/cloud/reference/management-api/reference.md). Refer to [Rockset vs. SingleStore](https://www.singlestore.com/comparisons/rockset/#technical-comparison) for a comprehensive technical comparison between these two platforms. ## Key Workload Considerations Some of the key features for Rockset can be mapped to similar features in SingleStore which may also have additional capabilities. ## Full-Text Search Rockset supports text search as part of its product offering, and provides full-text search via the [SEARCH](https://docs.rockset.com/documentation/reference/text-search-functions) function. SingleStore not only matches these features but provides much richer search functionality with support for Java-based Lucene [full-text search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) with the recent release of SingleStore v8.7. ## Vector Search Rockset added support for [vector search](https://docs.rockset.com/documentation/docs/vector-search) using a similarity index, which is used for indexing the embeddings. However, there is no choice of indexing algorithm, which is an IVF\_PQ index with support for dot, cosine, and Euclidean distance metrics. SingleStore supports vector indexing with advanced algorithms like HSNW and IVF\_PQ, and you can [combine indexed vector search with full-text indexing](https://www.singlestore.com/blog/hybrid-search-vector-full-text-search/). You can also [combine vector search with queries over other data types](https://www.singlestore.com/blog/why-your-vector-database-should-not-be-a-vector-database/), including JSON, time-series, full-text, spatial, and key-value data. SingleStore includes both exact K Nearest Neighbor (KNN) and indexed Approximate Nearest Neighbor (ANN). Provided distance metrics include dot product and Euclidean distance. * [Vector search](https://www.singlestore.com/blog/singlestore-high-performance-vector-search/) * [Enhanced full text search](https://www.singlestore.com/blog/unfreeze-your-data-lakehouse-power-intelligent-applications/#build-intelligent-applications-with-faster-vector-search-enhanced-full-text-search) * [Hybrid search](https://www.singlestore.com/hybrid-vector-search-full-text-search/) * [Vector range search](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-range-search.md) ## JSON Data Support JSON-based storage and query execution capabilities and compatibility with MySQL queries make it easy to migrate and translate Rockset queries over to SingleStore. SingleStore provides a rich set of capabilities in JSON data processing for analytics and excels at delivering analytics on immense volumes of data, as evidenced by the performance of [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md). SingleStore supports seekable columnstore (universal storage) and columngroup indexes to allow combined row and columnar access to large tables. Indexes can be added to increase query performance and text indexes added on the JSON column to match for query strings over any field on JSON data. SingleStore supports MySQL-style queries on [JSON data](https://docs.singlestore.com/cloud/create-a-database/using-json.md) which helps ease the migration of Rockset queries into SingleStore, thereby allowing further access to the MySQL ecosystem and MySQL-based tooling. ## Transactions Rockset is not a transactional database and is not designed to support OLTP workloads. Rockset provides atomic writes at the document level. While multiple fields can be updated atomically within a single document, atomic updates to more than one document are not supported. All writes are asynchronous, where the period from when data is written to the database to when the data is visible to queries varies between 1 and 5 seconds, which affects real-time responsiveness. Conversely, SingleStore is an ACID-compliant database with support for transactions. Consolidating the entire data stack while providing  analytics on the same data storage is achievable. ## Security and Enterprise Readiness When considering where to run production workloads, it is critical to select a platform that is both secure and that can run these workloads at scale. | **Category** | **Rockset** | **SingleStore** | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | --------------- | | Continuous backups with data stored on SSDs for performance, and cloud storage for durability with disaster recovery | ✔ | ✔ | | Multi-region hot-cold and hot-hot deployment options | ✔ | ✔ | | Enterprise SLAs | ✔ | ✔ | | Security Compliance
  • SOC 2 Type II
  • HIPAA
  • GDPR
  • CCPA
| ✔ | ✔ | | Data security and networking
  • Single Sign-On with Okta and custom SSO
  • Data encrypted at rest and in transit
  • Role-Based Access Control (RBAC)
  • IP Allowlisting
  • AWS PrivateLink
  • Virtual Private Rockset in your VPC
| ✔ | ✔ | | Performance metric endpoint | ✔ | ✔ | *** Modified at: May 13, 2026 Source: [/cloud/getting-started-with-singlestore-helios/migrate-from-rockset-to-singlestore/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/migrate-from-rockset-to-singlestore/) (An index of the documentation is available at /llms.txt) --- # Migrating to Clusters ## Overview This page describes the general process for migrating to SingleStore Helios. Migration is typically performed by following the documented steps to: * Prepare the target Helios deployment * Back up or export existing databases and objects * Restore or import the databases and objects into Helios * Validate the migrated environment The exact steps may vary depending on your source environment and migration goals. ## Migrate to Helios The following table outlines the tasks that must be performed to migrate from a self-managed cluster or a non-SingleStore platform to Helios. | Step | Task | Self-Managed Cluster | Non-SingleStorePlatform | | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ----------------------- | | 1 | **Create an inventory**of all objects, databases, tables, records, functions, stored procedures, views, aggregations, pipelines, database variables, etc. in your current production environment that need to be migrated. | ✔ | ✔ | | 2 | **Calculate the size**required to store all the existing objects and data. Refer to the[SingleStore Helios Sizing Guidelines](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/singlestore-helios-sizing-guidelines.md)for more information. | ✔ | ✔ | | 3 | **Create test plans for the following**:*Connectivity*
  • Upstream / downstream dataflows
  • Application
  • End-user
*Functionality*
  • Data accuracy
  • Custom objects
  • Jobs
  • Permissions
*Performance*
  • Ingest
  • Query
  • Skew
*Rollback* | ✔ | ✔ | | 4 | **Create theclusterin theCloud Portal**Refer to[Compute Clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters.md)for more information.Ensure there are enough credits to create an appropriately sizedcluster. | ✔ | ✔ | | 5 | **Generate SQL statements**that can be executed to reproduce all the required objects. Use a third-party tool such as`mysqldump`.Refer to[Transition from MySQL to SingleStore Helios](https://docs.singlestore.com/cloud/developer-resources/transition-from-mysql-to-singlestore-helios.md)for more information. | | ✔ | | 6 | **Create the objects inSingleStore Helios**. Use the`.sql`dump file created in Step 5.Copy and paste the contents of this file into theSingleStore HeliosSQL Editorand run the SQL statements to recreate all objects in thecluster.**Note**: While self-managedSingleStoreStudio does not work withclusters, you can use theCloud Portalorthird-party tools. | | ✔ | | 7 | **Back up**all of your databases. Use the[BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md)command to back up the data in production or take a backup from daily backups.For non-SingleStoredatabases, refer to[Migrating Data from Another Database](https://docs.singlestore.com/cloud/developer-resources/guides/migrating-data-from-another-database.md)for more information. | ✔ | ✔ | | 8 | **Restore the backed-up data to thecluster**. Use the[RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md)command to restore the database backups from the cloud object store into thecluster.**Note**: If you are moving from aSingleStore Self-Managedcluster to aSingleStore Helioscluster, first copy the backup to the proper cloud service and then restore.For non-SingleStoredatabases, refer to[Migrating Data from Another Database](https://docs.singlestore.com/cloud/developer-resources/guides/migrating-data-from-another-database.md)for more information. | ✔ | ✔ | | 9 | **Validate inventory and objects inSingleStore Helios**. Confirm that everything exists in theclusterenvironment. This should include everything gathered in Step 1. | ✔ | ✔ | | 10 | **Add**users, groups, roles, permissions, resource governor pools, database variables, and global system variables settings by raising a[Support ticket](https://support.singlestore.com). | ✔ | ✔ | | 11 | **Set the firewall rules**for the newcluster. Refer to[Firewall](https://docs.singlestore.com/cloud/security/firewall.md)for more information. | ✔ | ✔ | | 12 | **Select the endpoint URL**. Theclusterwill have a different endpoint URL than the prior cluster.Only DML endpoints are currently offered. The DDL can be provided if some operations require it. Refer to[SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md)for more information. | ✔ | ✔ | | 13 | **Perform application-level validation testing**. Connect your application and run benchmark queries. | ✔ | ✔ | *** Modified at: July 22, 2026 Source: [/cloud/getting-started-with-singlestore-helios/migrating-to-clusters/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/migrating-to-clusters/) (An index of the documentation is available at /llms.txt) --- # Next Steps and Examples ## In this section * [Purchase SingleStore Helios](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/purchase-singlestore-helios.md) * [Sample Data](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data.md) *** Modified at: October 28, 2024 Source: [/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/) (An index of the documentation is available at /llms.txt) --- # Purchase SingleStore Helios To extend your SingleStore Helios trial cluster evaluation period or to purchase SingleStore Helios, contact the SingleStore sales team by clicking the **Upgrade or Request a Quote** link in the top nav of SingleStore Helios, or via the [Request a Quote](https://www.singlestore.com/quote/) page. [SingleStore Helios pricing information and FAQs](https://www.singlestore.com/managed-service-pricing/) are available on the SingleStore Web site. *** Modified at: June 22, 2022 Source: [/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/purchase-singlestore-helios/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/purchase-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Sample Data ## In this section * [Load Data and Run Queries into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-data-and-run-queries-into-singlestore.md) * [Load and Analyze AdTech Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-and-analyze-adtech-data-into-singlestore.md) * [Load TPC-H Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-tpc-h-data-into-singlestore.md) * [Load MarTech Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-martech-data-into-singlestore.md) * [Load Stock Trading Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-stock-trading-data-into-singlestore.md) *** Modified at: October 10, 2024 Source: [/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Sizing Guidelines ## Overview This document is designed to help users with capacity planning to size a SingleStore Helios deployment appropriately. ## Size a Workload | **ClusterSizing Considerations** | | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | **Description and How to Measure** | | **Data Size** | It is important to understand how much data will be stored inSingleStore Helios.The most critical element for sizing is to understand how much of the data is the “working set,” which equates to how much data is required to meet the query latency requirements/SLA of an application. For optimal performance, it is imperative that the working set of data fits in the persistent cache.For example: A Fintech application displays stock performance over the last 30 days and stores the historical data for a full year for other types of analysis. There is a specific SLA for how quickly the dashboard must be visualized when working with data from the last 30 days, but the SLA for the analysis over the year’s worth of data is more flexible.
  • The market performance data for the last 30 days is considered the working set of data
  • The full year’s worth of data is the total data set
Even if the back-end database stores the previous year’s worth of market performance data (which is periodically queried but does not have an SLA), this would not impact the functionality of the application, and thus would not factor into sizing, as this data is stored in the object storage and not in the local cache storage.Data size can be used as a baseline/lower bound for sizing. To calculate data size:
  1. Determine your working set size in TB (the data that must meet your SLA)
  2. SingleStore typically sees 75% compression on average, depending on the data set. Apply Compression to the working set size data
  3. Multiply by two (2) to account for High Availability (HA) replication
  4. Add 25% for overhead (logs, snapshots, etc.)
This can be expressed formulaically as:`WorkingSetStorageInTBs = WorkingSetRawDataInTBs × (1 - CompressionRatio) × 2 × 1.25`Example: 4TB working set raw data`4 × (1 - 0.75) × 2 × 1.25 = 4 × 0.25 × 2 × 1.25 = 2.5TB`storage needed

📝 Note:

The total dataset can be larger than the working set. Data outside the working set is stored in object storage and hydrated on-demand when queried, so it doesn't impact cluster sizing.

**Questions**
  1. How large is the working set of data?
  2. How much data does the application need to perform?
  3. In lieu of an application, how many distinct data sets are there?
  4. How many tables? What are the core tables? How large are they?
  5. What types of data are primarily being stored? Strings? INTs? Floats? JSON? etc.
  6. Is a single tenant or multi-tenant structure being maintained? If multi-tenant, how many tenants are there? What is the approximate size of each?
  7. Is the data set growing? If so, by how much? How frequently?
  8. For those SLAs that are not as stringent for latency and concurrency, an important factor in determining cluster size largely depends on data size.
| | **Data Ingest** | Data ingest is about how much, in both rows and bytes, and how fast data is written into the database. This is typically inserts, but could also be updates and/or deletes.**Questions**
  1. What is the data size? The number of rows is not as helpful as the overall size of the data.
  2. What is the average ingest rate in bytes/second? What is the peak?
  3. How frequently is this volume of data ingested into the database? Does it stream in continuously, in small batches (every hour, etc.), or infrequently (once a day, etc.)
  4. How is the data currently ingested? What is the data source and format? Is it performed via an ELT, an ETL, or other transformation, like a stored procedure? The goal is to understand how much work will be performed during ingestion. For example, is data being inserted straight into a table or is there significant work performed in a stored procedure to process and transform the data?
  5. How is the ingest rate growing? By what percentage over what time?
| | **Query Shapes** | The “core queries” of an application are required to calculate the appropriate cluster size. Core queries are the queries most commonly run in the application and which require a specific SLA in order to meet the desired/required user experience.Some core queries are run many times, possibly with different parameters (literals) but otherwise the same structure.It is important to understand the shapes of the core queries that are running as they can impact the performance of the database differently.A query shape (type) is the form of the query plan that determines how much work a query needs to do. These are the different query shapes.[Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md)can be used to profile these queries.
  • Selective queries: Queries that return a small number of rows. SELECT * FROM table WHERE id = 5
  • Aggregation queries: SELECT AVG(col1), SUM(col2) FROM table GROUP BY (col3)
  • Joins: SELECT Count(*) FROM lineitem JOIN orders ON o_orderkey = l_orderkey;
It is also important to identify queries that can run on one partition vs. queries that must run on every partition. Selective queries typically hit one or a small number of partitions. Aggregations often touch all the partitions.Queries that run on one partition will tend to have lower total CPU usage.For example: Filtering on`id`if`id`is a primary key (which is a shard key and will have a unique index) can enable`SELECT * FROM table WHERE id = 5`to run on one leaf node. If`id`is not a shard key, then the query must touch every partition, and thus every leaf node.Graphical profile plans as well as JSON profile plans will reveal the indexes being used.Consider`CREATE TABLE t(id int, id2 int, primary key(id));`and query profile`SELECT * FROM t WHERE id2 = 1;`The following in the JSON profile means it is touching all shards and scanning:`"index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "storage":"columnar", "table_type":"sharded_columnstore", "columnstore_in_memory_scan_type":"TableScan", "columnstore_in_memory_scan_index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE",`For profile`SELECT * FROM t WHERE id = 1;`means it is seeking one shard and seeking:`"index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "storage":"columnar", "table_type":"sharded_columnstore", "columnstore_in_memory_scan_type":"IndexSeek", "columnstore_in_memory_scan_index":"UNIQUE KEY PRIMARY (id) USING HASH",`**Questions**
  1. What are the core queries?
  2. What are the query shapes?
  3. What is the total CPU of each core query? All core queries?
  4. What is the response time of each core query? All core queries?
| | **Latency** | Typically, customers will provide one or more latency targets for their core queries, such as a 5-second response time, or a range of response times for each core query.**Questions**
  1. What are the latency expectations for each core query?
  2. Is there a latency expectation for all core queries? If so, what is it?
| | **Concurrency** | Concurrency refers to the number of queries hitting the data at a given moment, including both typical and peak traffic. Ideally, this concurrency can be measured per query shape. Review the concurrency of each of the core queries.General numbers can be used to record the concurrency: 1, 5, 10, 20, 50, 100, 500, 1000, etc.**Questions**
  1. When a user logs into the application, how many queries are triggered?
  2. On average, how many of the core queries are running in the application concurrently?
  3. Do large spikes in traffic occur? If so, what is the minimum number of queries? The maximum?
  4. How is concurrency growing? By what percentage over what time?
| | **Partitions** | As part of capacity planning, you need to determine how many partitions are required for optimal performance.SingleStoreis a distributed cluster of nodes that talk to each other. To ensure maximum performance, you should have multiple cores to handle queries against the same database partitions on a node.The performance also depends on the types of queries run, whether they implement distributed joins, and how many are run concurrently.The general recommendation for most clusters is to have 4 CPU cores per database partition on each leaf. This is based on the fact that every query is a single-threaded operation.For example, if you have a cluster of 4 leaf nodes (a.k.a. “leaves”) with 16 cores on each leaf (64 CPU cores in total across all leaves), you should have 4 partitions on each leaf, that is, 16 partitions in total across the cluster.It is also important to choose the number of partitions that are divisible by the number of leaves you have. In the above example, a database with 16 partitions within a cluster of 4 leaves will divide those 16 partitions evenly across the 4 leaves (`16 / 4 = 0`).However, if you set 18 partitions instead of 16 then you will not have evenly distributed data. 18 (partitions) / 4 (leaves) = 2, which means two leaves would have an extra partition each. The number of partitions modulo the number of leaves should always equal zero.You should consider the workload when deciding on the number of partitions per leaf. For instance, if you have many parallelized queries that access the same information, it is advisable to have more cores per partition. But if the scale of your application is smaller, and queries can be run sequentially or less often, you can consider fewer cores per partition.By default, the partition count and core count match 1:1, and you need to manually change the number of partitions to maximize the performance. The number of partitions on a database is determined by either of the two ways:
  • Explicitly stating the number of partitions when creating the database. This can be done by adding PARTITIONS=X where X equals the number of partitions you want for that database.
  • By multiplying the number of leaves in the cluster with the value of the default_partitions_per_leaf variable.
**Questions**
  1. How frequently are queries with distributed joins run?
  2. How many queries are run concurrently?
  3. Is the chosen partition count divisible by the number of leaf nodes in the cluster?
  4. How frequently are parallelized queries that access the same information run?
| ## Cluster Size Capabilities After gathering the sizing considerations outlined above, use the comparison table below to select an initial cluster size. Test your core queries against the selected size and adjust up or down based on actual performance. | ClusterSize | vCPU | Memory (GB) | Storage (TB) | Optimal Working Set | Target Query Latency | Max Concurrent Queries | Typical Use Cases | | ----------- | ---- | ----------- | ------------ | ------------------- | -------------------- | -------------------------------- | ----------------------------------------------------------------------- | | S-4 | 32 | 256 | 1–2 | <= 1 TB | \~ 100 - 500 ms | Up to \~10 concurrent queries | Small-scale analytics, development / testing, low concurrency workloads | | S-8 | 64 | 512 | 2–4 | <= 2 TB | \~ 100 - 500 ms | Up to \~100 concurrent queries | Medium analytics, real-time dashboards, moderate concurrency | | S-16 | 128 | 1024 | 4–8 | <= 3 TB | \~ 100 - 500 ms | Up to \~1000 concurrent queries | Large analytics, high-performance applications, higher concurrency | | S-32 | 256 | 2048 | 8–16 | <= 4 TB | \~ 100 ms | Up to \~10000 concurrent queries | Enterprise analytics, ultra-fast queries, very high concurrency | ## Comparative Performance for Capacity Planning To estimate a cluster size, review the examples below and select the one that most closely resembles your requirements. To determine the optimal cluster size, test the core queries with the selected cluster size and either increase or decrease the size until it fulfills your latency requirements. | ClusterSizing Examples | | | | | | | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------- | | **Workload Size** | Small Analytics Workload | Medium Analytics Workload | Large Analytics Workload | Small Transactional Workload | Large Transactional Workload | | **Data Size \*** | 500GB | 7TB | 25TB | 2 TB (working set) | 1 TB (working set) | | **Data Ingest** | < 5,000 RPS | < 5,000 RPS | > 25,000 RPS | \~5,000 inserts/sec from application | \~10,000 inserts/sec from application | | **Query Latency** | < 1s |
  • P50 query latency of < 1 s
  • P95 query latency of < 5 s
  • P99 query latency of < 10 s
(Refer to[latency](https://readosapien.com/what-is-p50-p90-p99-latency/)for more information) | < 30 s | < 500 ms | < 100 ms | | **Query Shapes** | Analytical queries and summary statistics on fact and reference tables | Analytical queries powering nearline analytics and user-defined metrics | Majority analytical queries with large table scans | Transactional queries with fast analytical smaller queries | Large table scans and aggregations. | | **Concurrency tested (QPS)** | 10 - 50 | < 50 | 100 | 100 | 100 | | **ProposedClusterSize**(About[clustersizes](https://www.singlestore.com/cloud-pricing/)) | S-1 | S-8 | S-12 | S-8 | S-20 | **\*** Data sizes shown are compressed, where the compression rate can vary up to 80% based on the type of data. > **📝 Note**: Working set refers to the data that needs to be queryable in a timely manner (typically within 5–10 minutes). All other data is stored in object storage and hydrated to leaves when queried. *** Modified at: May 28, 2026 Source: [/cloud/getting-started-with-singlestore-helios/singlestore-helios-sizing-guidelines/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/singlestore-helios-sizing-guidelines/) (An index of the documentation is available at /llms.txt) --- # SQrL SQrL (pronounced "squirrel") is an AI-powered co-pilot built by SingleStore that answers your questions and helps you code faster (in the context of SingleStore Helios). It is trained using content from SingleStore Helios documentation, [GitHub](https://github.com/memsql/), [Forums](https://www.singlestore.com/forum/), and other internal sources, to provide product-related information on SingleStore Helios. In the [Question and Answer mode](https://docs.singlestore.com/#section-idm4545101682140833798654309115.md), SQrL can answer questions and provide code suggestions for both the SingleStore Helios and SingleStore Self-Managed deployments. ## Why Use SQrL SQrL is designed to provide immediate and relevant responses to SingleStore Helios-related questions. It can assist you with deployments, code optimization, integrations, resource management, troubleshooting, etc. SQrL is also integrated into SingleStore [Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md), which allows you to optimize your code using code recommendations and debug errors in your code using potential resolutions in the context of your code. > **📝 Note**: SQrL may sometimes return reasonable-sounding but inaccurate responses. To help us improve SQrL's accuracy, provide feedback using **Thumbs down** next to a response. ## Getting Started with SQrL You can access SQrL from any of the following platforms: ## a. SQrL Website You can use SQrL in the [Question and Answer mode](https://docs.singlestore.com/#section-idm4545101682140833798654309115.md) on the [SQrL website](https://www.singlestore.com/sqrl/) even if you are not a member of any SingleStore Helios organization. SingleStore users on SingleStore Self-Managed deployments who do not have access to the Cloud Portal can also use this platform. ## b. Cloud Portal SQrL is accessible on the Cloud Portal in the following locations: * On the Cloud Portal homepage, select **Ask SQrL**. You can also access SQrL from other screens on the Cloud Portal using the floating **?** (question) icon. Select **Ask SQrL** from the list. These options allow you to access SQrL in the [Question and Answer mode](https://docs.singlestore.com/#section-idm4545101682140833798654309115.md) only. Note that the **Chat with us** option is not the same as using SQrL. ![Access SQrL in the Cloud Portal.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt34ac43bcc3ac921c/6a7adbdb14d87c49fb3c7709/sqrl-cl-new-2nJ0Kk.png) * **With Notebooks**: On the Cloud Portal, under **Notebooks**, open a notebook. Once the notebook is open, enable the **Code with SQrL** toggle switch on the upper-right corner. This option allows you to use SQrL in [code-only mode](https://docs.singlestore.com/#section-idm4538780119100833798656998424.md). ![Code with SQrL in Notebooks.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb9421cd6fe733059/6a7adb13bdcff06c3bc377fe/sqrl-nb-new-V1u3L9.png) ## c. SingleStore Documentation You can access SQrL in the [Question and Answer mode](https://docs.singlestore.com/#section-idm4545101682140833798654309115.md) from any page in the Documentation site. Select the (SQrL) icon on the side of the page to open the SQrL pane. ![Access SQrL from the Documentation site.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt62e07c1472bec8d0/6a7ada3b227b1c06c458cdfa/sqrl-position-0K7gq0.png) ## SQrL Usage Modes You can use SQrL in any of the following modes. ## a. Question and Answer Mode In the Question and Answer mode, SQrL *answers* the user's queries (questions). You can ask questions about learning new features, troubleshooting specific issues, comparing approaches to a specific use case, and more. The answers may include suggestions to related questions and a list of sources from where the response is derived. After you enter your query, select **Enter**/**Return** to submit it. To add another line to your query (add a new line), use **Shift+Enter** or **Shift+Return**. You can also provide feedback on SQrL responses to help improve its accuracy. Select **Thumbs up** for positive responses and **Thumbs down** for negative responses and provide feedback. To share a SQrL discussion, select **Share**. It shares the entire SQrL discussion. When you select **Share**, a unique URL is generated for the discussion session. Anyone with access to the URL can view the contents of the discussion. You can also copy a specific code snippet or other blocks from the discussion with SQrL without using the Share feature. Select **Clear** to clear the current discussion. To stop a SQrL response, select **Stop**. ## b. Code-only Mode The code-only mode is only available in SingleStore notebooks (on the Cloud Portal). In this mode, SQrL provides code suggestions that can be integrated into your notebooks. To use SQrL in the code-only mode, enable the **Code with SQrL** toggle switch and enter a query in the **Connect with SQrL** box. After you enter your query, select **Generate** to submit your query. Use **Enter**/**Return** to add a new line to your query. In this mode, SQrL does not preview live responses line-by-line. Hence, the response time may seem longer. After SQrL has generated a code snippet, you can either select **Add Cell** to add it to a new cell at the end of your notebook, or select **Add Cell & Run** to add it to a new cell and run that cell. If SQrL is unable to generate a code snippet relative to the query, it returns an error. Select **Regenerate** to try again, either with the same query or a different query. ## c. Code Generator Mode SQrL Code Generator helps you generate SQL queries based on your prompts, keeping your table schemas in mind. To use Code Generator mode, navigate to SQrL in the Portal and select "Code Generator". Next, you can select which tables are relevant in generating your SQL. SingleStore does not access any of the personal information in these tables –– it just passes the schema of this table and exposes it to the LLM as context. This ensures that the SQL that is generated understands the relationships and columns between tables and that is more accurate. Once you have selected your tables, you can ask SQrL multiple questions without having to reselect your tables again. To refresh context and select new tables, close and reopen the window. ## SQrL and Privacy This section focuses on privacy practices, data collection, data security, and data retention in the context of SQrL only. Refer to [Privacy Notice](https://www.singlestore.com/privacy-notice/) for information about SingleStore's privacy policy. ## Data Collected by SQrL SingleStore collects and processes data from SQrL to provide the service. SingleStore continuously improves SQrL by leveraging data collected from SQrL to provide more accurate responses to your queries. This data includes prompts and responses, which are always collected, and user engagement metadata (optional). SingleStore retains some of the collected data for analysis and improvements. The collected data is **not** used to train large language models (LLMs). SingleStore collects prompts, answers, and user engagement metadata from SQrL as follows: * **Prompts** A prompt is the contextual information that SQrL gets when a user asks a question/query. Prompts are transmitted in real-time and retained to improve our product and documentation on questions asked by users. * **Responses** SingleStore retains responses/answers from SQrL to improve the user experience of our product. * **User Engagement Metadata** When you use SQrL, it collects user engagement data. This data consists of usage information and events generated while interacting with SQrL. These events include, but not limited to: user actions like Thumbs up or Thumbs down, feedback, and feature engagement. You can enable or disable user engagement metadata collection within your organization on the Cloud Portal, go to **\ > Settings**. Under **SQrL Metadata**, use the toggle switch to enable or disable metadata collection. Note that data from prompts and responses is always collected. ## Sharing Collected Data SingleStore shares the data collected from the SQrL discussions with the following AI service providers (via API): * Anthropic * Amazon Bedrock The data shared with these AI providers is NOT used to train their AI models. For information on privacy policies of each of these providers, refer to the following links: * [Anthropic Console](https://legal.anthropic.com) * [Amazon Bedrock](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_bedrock.html) ## Data Security Refer to [Security](https://www.singlestore.com/privacy-notice/#security) for information. *** Modified at: March 2, 2026 Source: [/cloud/getting-started-with-singlestore-helios/sqrl/](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/sqrl/) (An index of the documentation is available at /llms.txt) --- # Glossary##### AggregateThe task of collecting a set of values to return a single value. When data is aggregated, data rows are replaced with totals or summary statistics.##### Aggregator nodeA node that routes queries to the leaf nodes, aggregates intermediate results, and sends the results back to a client. There are two types of aggregators: master and child.##### Approximate Nearest Neighbor A technique used in computational geometry and machine learning to  find the approximate nearest neighbors in high-dimensional spaces.##### Background mergerAn algorithm used by SingleStore that allows columnstore segments to maintain sort order (as close as possible), while data is being ingested or updated.##### Binary Javascript Object Notation A binary-encoded serialization of JSON-like documents. It is designed to be efficient in space but also rich in its ability to represent more data types than JSON.##### CentroidsA point used in clustering algorithms to represent the center or mean of a cluster, a group of vectors near each other.##### Child aggregatorA node that can be promoted to the role of Master Aggregator in the event that the existing Master Aggregator’s host fails. Depending on the query volume, a cluster may contain zero or more child aggregators.##### ClusterA collection of SingleStore aggregator and leaf nodes.##### Code generationAn industrial compiler to produce highly efficient machine code that enables low-level optimizations, which are not possible when executing queries via interpretation alone. By default, queries are interpreted first and then asynchronously compiled in the background for use in later executions. This speeds up query execution time for long and complex queries, while at the same time providing efficient query plans for later use.##### Common Table ExpressionA named temporary result set that exists within the scope of a single statement and that can be referred to later within that statement, possibly multiple times.##### Database branchingA feature that creates private, independent copies of a database including all of its data.##### Database userA user that resides in the cluster. A database user’s lifetime is bound to that cluster such that, when a cluster is terminated, all of the users, permissions, and groups are permanently removed as well. Database users can connect to a cluster via SQL client and run SQL queries against their data. Unlike organization users, database users must be managed via SQL statements.##### DeadlockA situation when two or more transactions mutually hold and request a table write that the other transaction needs.##### DeterministicAn operation or function that will always produce the same result for the same input values.##### Full backupA stored and complete copy of database.##### Garbage Collection The process where unneeded versioned nodes for multi-version concurrency control (MVCC) are eliminated. These versioned nodes may be part of skip list indexes or hash indexes for in-memory rowstore tables or columnstore segments stored in memory.##### GlobbingGlobbing is commonly used in data ingest to read or select a subset of files based on a naming pattern. Data ingest uses globbing to process large numbers of files quickly and efficiently, especially when the file names follow a predictable naming pattern.##### Hadoop Distributed File SystemA distributed file system by Apache Hadoop. It is highly fault tolerant and designed to run on COTS (Commercial Off the Shelf) or low cost out-of-the-box hardware.##### Hash indexA data structure optimized for fast equality lookups by a key.##### Hierarchical dataA set of data items that are related to each other by hierarchical relationships. Hierarchical relationships exist where one item of data is the parent of another item(s).##### Hierarchical Navigable Small World An algorithm used for approximate nearest neighbor (ANN) search, particularly in high-dimensional spaces. This algorithm is well-suited for applications requiring fast approximate nearest neighbor queries, such as similarity search in large-scale datasets. HNSW is known for its efficiency in handling high-dimensional data, making it a good choice for machine learning, data mining, information retrieval, etc.##### High-dimensional spacesDatasets with a large number of features or attributes, where each feature represents a separate dimension.##### HostA hardware or virtual machine which holds the aggregator and leaf nodes that comprise a SingleStore Helios cluster.##### Incremental backupA stored copy of data that only the data stores what has been modified since the most recent backup.##### Information schemaHolds the information or metadata for all the databases in a cluster.##### Inline viewA `SELECT` statement embedded in the `FROM` or `WITH` clause of another `SELECT` statement that creates a temporary table that is operated on by the outer query.##### In-place changeAn in-place change in a database refers to modifying the data or schema of the database without requiring a full data migration or recreation of the database. It allows you to make changes to the database while minimizing downtime and preserving existing data.##### Inverted file with product quantization A method used for approximate nearest neighbor search in large-scale datasets, particularly in high-dimensional spaces.##### Javascript Object NotationAn open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays.##### JSON Web Key SetsA set of keys containing public keys that can be used to authenticate any JWT.##### JSON Web TokenAn open, industry standard typically used for authorization and information exchange.##### Leaf nodeA node that stores a subset of a cluster’s data. A leaf node functions as a storage and compute node. To optimize performance, SingleStore automatically distributes data across leaf nodes into partitions. Each leaf node contains several partitions.##### Lock-free backupsBackup operations that do not block `INSERT`, `UPDATE`, and `DELETE` operations during the backup.##### Low Level Virtual Machine It is a collection of modular and reusable compiler and toolchain technologies used for developing compiler frontends and backends. SingleStore includes an LLVM-based code generation framework that is used to compile queries to machine code.The project has outgrown the original name, and now LLVM is just its name, not an acronym.##### Master aggregatorA specialized node that’s responsible for cluster monitoring and failover. It orchestrates basic cluster operations and all DDL operations.##### MetadataInformation about a database's schema, access to the database, storage, built-in programs, and or other information about the data such as date created, file size, and etc.##### Multi-Version Concurrency Control A method used to increase transaction concurrency and reduce response time for read-only transactions by maintaining a history of versions of each row in a table.##### NodeA SingleStore server. A host may contain one or more SingleStore server instances.##### Non-Uniform Memory AccessA computer memory design used in multiprocessing. Through NUMA, a processor can access memory that’s considered “local memory” faster than it can access memory that is local to another processor, or memory that is shared between processors.##### NormalizedOrganizing data to appear similar across all records and fields.##### Object storeA data storage architecture that manages data as objects, as opposed to other storage architectures like file systems which manages data as a file hierarchy, and block storage which manages data as blocks within sectors and tracks.##### Online Analytical ProcessingA data processing type that is designed to analyze data dimensions concurrently.##### Online Transaction ProcessingA data processing type that executes a number of transactions occurring concurrently.##### OrganizationAllows shared access to clusters and related resources within a company or group.##### Organization userAn organization user resides within the “control plane” of SingleStore Helios and can perform those actions that are available on the Cloud Portal, such as manage organization users, manage clusters, and run SQL queries against cluster data via the Cloud Portal SQL Editor.##### PartitionA partition contains a subset (a shard) of a database’s data. Each partition holds a vertical slice of data, distributed as per a hashing algorithm on the primary key or randomly for keyless sharded databases.##### Persisting the name of a filePersisting the name of a file is storing the file name so it can be retrieved and used again later in a program or system.##### PipelinesA feature that continuously loads data as it arrives from external sources. As a built-in component of the database, Pipelines can extract, shape (modify), and load external data without the need for third-party tools or middleware.##### Point-in-time recovery A user-initiated operation that allows a set of data in a database to be recovered to a specific timestamp in the past.##### Procedural SQL A set of programming extensions for SingleStore that allow developers to write code in a procedural format.##### Product Quantization A technique used for vector compression. It is very effective in compressing high-dimensional vectors for nearest neighbor search.#####An application that allows you to design, manage, and monitor your SingleStore Helios instances.##### userA user that can log into the Cloud Portal and access portal services. Each Cloud Portal user is associated with a default organization and automatically has access to all of the clusters and related resources within it.##### Query shapePatterns or structures in query. Some query shapes are unsupported in SingleStore.##### Random-Access MemoryA computer's short-term memory. It is where the data that the processor is currently using is stored temporarily. RAM can be accessed much faster than data on a hard disk, solid-state disk, or another long-term storage device, which is why RAM capacity is so important for system performance.##### ReplicationEnsures redundancy in a cluster. There are two types of replication: high availability - replicating partitions between the leaf nodes and cluster replication - replicating partitions between clusters.##### Segment eliminationA process where metadata stored for columnstore segments is used to determine where a segment can match a filter queried at execution time.##### Serializable isolationAn isolation level provides the strictest transaction isolation. This level guarantees transactions that are executed in parallel produce the same result as if they were executed serially (one at a time).##### ShardA subset of a databases's data.##### Smart Disater Recovery A process that handles the continuous asynchronous replication of data between a primary and a secondary region. A primary region is the main geographic location where your database(s) currently reside and operate, while a secondary region is an additional, geographically separate location to which your database(s) are replicated for disaster recovery purposes.##### Solid-State DriveA solid-state drive is a storage device. It is a non-volatile medium that stores persistent data on solid-state flash memory.##### Single sign-onSingle sign-on, where one can log into SingleStore Helios via Azure AD, Okta, and PingOne.##### Skiplist indexA data structure optimized for ordered data that allows for queries to quickly seek data by binary searching.##### SQL surface areaThe amount of components installed and or configuration options that are enabled.##### ShardingA type of database partitioning that divides a database into smaller more easily manageable parts.##### Unlimited storageAn unlimited amount of storage space in the cloud that can be used to move data to seamlessly between memory, persistent cache, and storage.#####A database whose size is not limited by the size of the persistent cache, but only by available external object storage.##### Unlimited storage databaseA database whose size is not limited by the size of the persistent cache, but only by available external object storage.##### User-defined aggregate function A callable routine that accepts input parameters, executes programmatic logic in the function body, and returns a scalar-type value.##### User-defined function A callable routine that accepts input parameters, executes programmatic logic, and returns a value (or values).##### User-defined table-valued function A callable routine that accepts input parameters, executes a single SELECT statement in the function body, and returns a single table-type value (similar to a view).##### WorkspaceSingleStore Helios’s version of a cluster.***Modified at: May 22, 2026Source: [/cloud/glossary/](https://docs.singlestore.com/cloud/glossary/)(An index of the documentation is available at /llms.txt) --- # Load Data SingleStore provides several options for loading data from various sources such as files, cloud repositories, external databases, and third-party ETL or CDC tools. ## Data Loading Methods Available in SingleStore * **SingleStore Flow**: A GUI-based tool designed for database migration and change data capture (CDC) replication. It supports a wide range of enterprise databases. * **SingleStore Pipelines**: A built-in, real-time ingestion mechanism ideal for streaming and ETL scenarios, allowing continuous loading and transformation of data from file storages and streaming platforms. * **LOAD DATA SQL Command**: A SQL-based method for directly loading data from local or remote files into SingleStore tables. All of these methods are accessible through the Helios Portal Load Data UI for easy setup and management. ## Supported File Formats SingleStore supports loading data from the following file formats: * [CSV](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files.md) * [JSON](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files.md) * [BSON](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files.md) * [Avro](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files.md) * [Parquet](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files.md) ## Data Sources And the Corresponding Loading Methods in SingleStore **Note**: Click a check mark (**✓**) to view the associated guide. | Data Sources | SingleStoreFlow | Pipelines | LOAD DATA Command | Notes/Special Features | | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **FIle Systems** | | | | | | Amazon Web Services (AWS) S3 | | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3.md) | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3.md) | | | Azure Blob Storage | | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline/connect-to-azure-blob-storage-from-singlestore.md) | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline/connect-to-azure-blob-storage-from-singlestore.md) | | | Google Cloud Storage (GCS) | | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline.md) | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline.md) | | | Hadoop Distributed File System (HDFS) | | **[✓](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline.md)** | | | | Apache Iceberg | | **[✓](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md)** | | | | Filesystem paths (local or network drives) | | **[✓](https://docs.singlestore.com/cloud/load-data/data-sources/#load-data-from-the-filesystem-using-a-pipeline.md)** | [**✓**](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-local-files.md) | | | Stage | | [**✓**](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) | [**✓**](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) | SingleStoremanaged storage | | **Streaming Systems** | | | | | | Kafka | | [**✓**](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka.md) | | | | **Database Sources** | | | | | | Oracle & Oracle RAC | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | | | | | Microsoft SQL Server | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | | | | | MySQL | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | **[✓](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/aggregator-pipelines.md)** | | When using pipelines; there is a cap of 16 pipelines for the MySQL source.Also supports[mysqlimport](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql/load-data-from-mysql.md)utility. | | PostgreSQL | [**✓**](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md) | | | | | Snowflake | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | | | Data migration only. | | Any JDBC-compliant database | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | | | Limited to non-production preview, available for testing. | | MongoDB® | **[✓](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md)** | **[✓](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/aggregator-pipelines.md)** | | When using pipelines; there is a cap of 16 pipelines for MongoDB® source. | | **Third-party Integration Tools** | | | | | | Apache Beam/Flink, AWS Glue, Azure Data Factory, dbt, Fivetran, Informatica, Qlik, StreamSets and similar platforms. | | | | Supports JDBC, ODBC, REST APIs.[Third-party and Custom Connectors](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) | ## In this section * [Load Data with SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md) * [Load Data with SingleStore Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios.md) * [About SingleStore Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) * [Data Sources](https://docs.singlestore.com/cloud/load-data/data-sources.md) * [Load Data from Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files.md) * [Integrate with SingleStore Helios](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) *** Modified at: July 14, 2026 Source: [/cloud/load-data/](https://docs.singlestore.com/cloud/load-data/) (An index of the documentation is available at /llms.txt) --- # About SingleStore Pipelines A SingleStore pipeline is a mechanism for continuously loading data into SingleStore from external sources including Apache Kafka, Amazon S3, Azure Blob Storage, Google Cloud Storage, and the local file system. Pipelines can extract, shape (modify), and load external data without the need for additional third-party tools. Pipelines are robust, scalable, highly performant, and support fully distributed workloads. The following features make pipelines a powerful alternative to third-party ETL middleware in many scenarios: * **Easy continuous loading**: Pipelines monitor their source folder or Kafka queue and, when new files or messages arrive, automatically load them. This simplifies the job of the application developer. * **Scalability**: Pipelines inherently scale with SingleStore clusters as well as distributed data sources like Kafka and cloud data stores like Amazon S3. * **High performance**: Pipelines data is loaded in parallel from the data source directly to the SingleStore leaf nodes, in most situations; this improves throughput by bypassing an aggregator. Additionally, pipelines have been optimized for low lock contention and concurrency. * **Exactly-once semantics**: The architecture of pipelines ensures that transactions are processed exactly once, even in the event of failover. * **Debugging**: Pipelines makes it easier to debug each step in the ETL process by storing exhaustive metadata about transactions, including stack traces and stderr messages. * **Concurrency**: Multiple pipelines can insert data into a single table. This ability is similar to using multiple write queries. See Sync Variables Lists for more information. * **Backup**: Database backups preserve the state of all pipelines (offsets, etc.) in that database. When a backup is restored, all pipelines in that database will revert to the state (offsets, etc.) they were in when the target backup was generated. Pipelines support Avro, CSV, JSON, and Parquet. ## Related Topics * [The Lifecycle of a Pipeline](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/the-lifecycle-of-a-pipeline.md) * [Pipeline Troubleshooting](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting.md) ## In this section * [Pipeline Concepts](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts.md) * [Pipeline Troubleshooting](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting.md) *** Modified at: August 28, 2025 Source: [/cloud/load-data/about-singlestore-pipelines/](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/) (An index of the documentation is available at /llms.txt) --- # Pipeline Concepts ## Create a Pipeline A core `CREATE PIPELINE` command contains: * A **Pipeline Declaration** with the pipeline name. * A **Data Source Specification** based on where the data is stored (AWS S3, Azure, Kafka, etc.). * A **Data File Mapping** which maps the data file to a SingleStore table and which is based on the data file format. Below is a simple pipeline that loads a CSV file from Amazon S3. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb6a9c86624fcd231/6a7adbab889c374933057cb4/pipeline_concepts_figure1-q6nCp5.png) In the pipeline above, the structure of the Data Source Specification is determined by the location of the data file, Amazon S3 in this example. The structure of the Data File Mapping is determined by the data file format, CSV in this example. The following pipeline loads a Parquet file from Amazon S3. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2281c5a407bd8bc3/6a7adac62f00b21c6cef363b/pipeline_concepts_figure2-mPMUoQ.png) In this pipeline, the Data Source Specification is the same as in the pipeline above (except for the file name), but the Data File Mapping is different as both the file format and the data file format are different (CSV in the prior example, Parquet in this example). Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for a full description of the `CREATE PIPELINE` command. ## Supported Data Stores Pipelines support the following data stores: * Amazon S3 - [S3 Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-e9b729f5-e821-593f-4780-33e798409694.md) * Google Cloud Services - [GCS Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-157713ba-2165-ec8f-6291-47e4c3e65737.md) * HDFS - [HDFS Pipeline Syntax (Version 2.2.x or newer)](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a8df0412-4c5b-75f4-e6ce-753934798422.md) * Kafka - [Kafka Pipeline Syntax (Version 0.8.2.2 or newer)](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-4605223c-017e-44d6-7b51-ec3b59137e07.md) * Link - [Creating a Pipeline Using a Connection Link](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a3b0c2e9-827c-3e03-4fd6-ea2c57d954f1.md) * Microsoft Azure - [Azure Blob Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a57594c8-138d-3bb1-d25b-aebef5185c17.md) * MongoDB - [Replicate MongoDB® Collections using SQL](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-sql.md) * MySQL - [Replicate Data from MySQL](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql.md) ## Supported Data File Formats * Avro - [Kafka Pipeline Using Avro Format](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#section-idm234619121847915.md) * CSV - [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) * Iceberg - [Create an Iceberg Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md) * JSON - [Kafka Pipeline Using JSON Format](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#section-idm4529487879993633828365570841.md) * Parquet - [Create a Parquet Pipeline](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files/create-a-parquet-pipeline.md) ## In this section * [The Lifecycle of a Pipeline](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/the-lifecycle-of-a-pipeline.md) * [Pipelines Scheduling](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipelines-scheduling.md) * [Parallelized Data Extraction with Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/parallelized-data-extraction-with-pipelines.md) * [Data Shaping with Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/data-shaping-with-pipelines.md) * [Aggregator Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/aggregator-pipelines.md) * [Kafka Connect Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/kafka-connect-pipelines.md) * [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md) * [Pipeline Built-in Functions](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipeline-built-in-functions.md) * [Pipeline Retry Options](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipeline-retry-options.md) * [Writing Efficient Stored Procedures for Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/writing-efficient-stored-procedures-for-pipelines.md) * [Best Practices for Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/best-practices-for-pipelines.md) *** Modified at: July 14, 2026 Source: [/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/) (An index of the documentation is available at /llms.txt) --- # Pipeline Troubleshooting ## Concepts This topic requires an understanding of pipeline batches, which are explained in [The Lifecycle of a Pipeline](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/the-lifecycle-of-a-pipeline.md). ## Monitoring Users can monitor the status and performance of pipelines in SingleStore via a dashboard. The *Pipeline Summary* section of this dashboard provides a high-level overview of the state of all pipelines, including the number and percentage of pipelines in various states including running, stopped, and errored. The *Pipeline Performance* section also provides insights into the operation of pipelines. For example, metrics such as execution count, average CPU time per execution, average elapsed time per execution, and others, can aid in identifying and optimizing aspects of pipeline performance. ## Address specific errors The following table lists errors that can occur when running a pipeline, such as `CREATE PIPELINE`, and errors that can occur while a pipeline is extracting, shaping, or loading data. | Error | Resolution | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Syntax error when running`CREATE PIPELINE`. | Both`CREATE PIPELINE`and`LOAD DATA`(which is part of the`CREATE PIPELINE`syntax) have many options. Verify that the included options are specified in the correct order. | | Error`1970: Subprocess timed out` | The master aggregator is likely unable to connect to the pipeline's data source. Check the connection parameters, such as`CONFIG`and`CREDENTIALS`, that specify how to connect to the data source. | | `CREATE PIPELINE ... S3`returns an error that the bucket cannot be located. | The bucket name is case-sensitive. Verify that the S3 bucket names are in the same case as the bucket names specified in your`CREATE PIPELINE ... S3`statement. | | Error`1953: exited with failure result (8 : Exec format error)`or`No such file or directory` | This error can occur when a pipeline attempts to run a transform. Check the following:
  • Verify that the first line of your transform contains a shebang. This specifies the interpreter (such as Python) to use to run the script.
  • Is the interpreter (such as Python) installed on all leaves?
  • If the transform was written on a Windows machine, do the newlines use \r\n?
| | `CREATE PIPELINE ... WITH TRANSFORM`fails with a`libcurl`error. | An incorrect path to the transform was likely specified. If the path to the transform is correct, then running`curl`with the path to the transform will succeed. | | Error:`java.lang.OutOfMemoryError: Java heap space` | This error may occur when the heap memory usage exceeds the value of`java_pipelines_heap_size`variable. Increase the value of this engine variable to potentially resolve this error. | | S3 pipeline create/start delays (approximately 60 seconds) or “Subprocess timed out” outside AWS | Reduce the value of the`subprocess_ec2_metadata_timeout_ms`engine variable (for example, to`1000`) or provide explicit S3`CREDENTIALS`.Newclusters set this timeout to`1`(millisecond) to avoid this behavior.Refer to[Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md)for more information. | ## Error: An error that isn't associated with any specific source partition of the pipeline occurred during the batch loading process. The whole batch will be failed. **Issue** The batch loading process was able to load the data from the source, but it failed to ingest the data into the SingleStore database. This error is caused by a secondary error which is the root cause of the pipeline failure. The secondary error may be caused by resource overload, lock wait timeouts, etc. **Solution** Address the secondary error to solve the issue. Query the [PIPELINES\_ERRORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-errors.md) information schema table for more information on the error that caused the pipeline failure. For example, consider the following error: ``` Error,2790,"An error that isn't associated with any specific source partition of the pipeline occurred during the batch loading process. The whole batch will be failed. Error 1205 : ""Leaf Error (svchost:3306): Lock wait timeout exceeded; try restarting transaction. Unique key Row Value lock owned by connection id xxxx, query `open idle transaction`""" ``` In this case, the pipelines failed because the query was unable to acquire row locks for ingesting data. Identify the transaction that caused the timeout, and kill its connection. ## Rename a table referenced by a pipeline Trying to rename a table that is referenced by a pipeline returns the following error: ```sql ERROR 1945 ER_CANNOT_DROP_REFERENCED_BY_PIPELINE: Cannot rename table because it is referenced by pipeline ``` The following sequence demonstrates how to rename a pipeline referenced table: 1. Save the pipeline settings. ```sql SHOW CREATE PIPELINE EXTENDED; ``` 2. Stop the pipeline. ```sql STOP PIPELINE ; ``` 3. Drop the pipeline. ```sql DROP PIPELINE ; ``` 4. Change the name of the table. ```sql ALTER TABLE RENAME ; ``` 5. Recreate the pipeline with the required configuration options, and change the table name to reflect the new table name. 6. Start the pipeline. ```sql START PIPELINE ; ``` ## Pipeline errors that are handled automatically ## Typical error handing scenario In most situations, an error that occurs while a pipeline is running is handled in this way: If an error occurs while a batch is running, the batch fails and its transactions are rolled back. > **📝 Note**: Monitor the source-specific configurations that you have set to ensure that your pipelines are operating within those limits.For example, if [`operation.timeout.ms`](https://docs.singlestore.com/cloud/load-data/data-sources/configuration-options-for-different-sources/#section-idm234488319108397.md) is set to 10 seconds in Kafka and a Kafka offset takes 20 seconds to fetch, an error will be thrown. To avoid this error, increase the `operation.timeout.ms` limit. The following table lists events, which may or may not cause errors, and how the events are handled. | Event | How the Event is Handled | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The pipeline cannot access a file or object. | `nb`skips the file/object. | | The pipeline cannot read a file or object because it is corrupted. | `nb`skips the file or object.After fixing the issue with the corrupted file/object, you can have the pipeline reprocess the file/object by running`ALTER PIPELINE ... DROP FILE ;`. The pipeline will process the file/object during the next batch. | | A file or object is removed from the filesystem after the batch has started processing the file/object. | The batch does not fail; the file or object is processed. | | A file is removed from the filesystem (or an object is removed from an object store) after the pipeline registers the file/object in`information_schema.PIPELINES_FILES`, but before the file/object is processed. | `nb`skips the file or object. | | Theclusterrestarts while the batch is being processed. | The typical error handling scenario (mentioned earlier in this topic) applies.Once theclusteris online,`b`is retried. | | A leaf node is unavailable before the pipeline starts. | This does not cause the pipeline to fail. The pipeline will not ingest any data to the unavailable leaf node. | | A leaf node fails while the pipeline is running. | The batch fails. The batch is retried as described in this error handling scenario; that batch and all future batches no longer attempt to load data to the unavailable leaf node. | | An aggregator fails while the pipeline is running | The batch fails. When the aggregator is available, the batch is retried as described in the typical error handling scenario. | | The pipeline reaches the allocated storage space for errors. | The pipeline pauses. To resolve this issue:
  1. Increase the value of the ingest_errors_max_disk_space_mb engine variable.
  2. Run CLEAR PIPELINE ERRORS; to free up storage space for errors. (Running this command removes all the existing pipeline errors that are shown when running SHOW ERRORS;).
| ## Additional Information For information on troubleshooting pipeline errors and performance issues, refer to [Pipeline Dashboards](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md). ## In this section * [Debugging Pipeline Errors](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/debugging-pipeline-errors.md) * [Troubleshoot Pipeline Performance and Memory Usage](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/troubleshoot-pipeline-performance-and-memory-usage.md) *** Modified at: May 12, 2026 Source: [/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/) (An index of the documentation is available at /llms.txt) --- # Data Sources ## In this section * [Load Data from Kafka](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka.md) * [Load Data from Amazon Web Services (AWS) S3](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3.md) * [Load Data from Amazon Kinesis Using a Kafka Connect Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-kinesis-using-a-kafka-connect-pipeline.md) * [Iceberg Ingest](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md) * [Load Data from Azure Blob Storage Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline.md) * [Load Data from Google Cloud Storage (GCS) Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline.md) * [Load Data from HDFS Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline.md) * [Replicate Data from MongoDB®](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mongodb.md) * [Replicate Data from MySQL](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql.md) * [Configuration Options for Different Sources](https://docs.singlestore.com/cloud/load-data/data-sources/configuration-options-for-different-sources.md) * [SingleStore Snowpark App](https://docs.singlestore.com/cloud/load-data/data-sources/singlestore-snowpark-app.md) * [SingleStore Replicate](https://docs.singlestore.com/cloud/load-data/data-sources/singlestore-replicate.md) *** Modified at: July 10, 2026 Source: [/cloud/load-data/data-sources/](https://docs.singlestore.com/cloud/load-data/data-sources/) (An index of the documentation is available at /llms.txt) --- # Configuration Options for Different Sources SingleStore Helios supports a number of configuration options for different sources. These options can be used with the `CONFIG` clause in the [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-6166b957-3476-1e7b-46ae-c04322557883.md) command. ## Kafka Configurations The following table shows the SingleStore Helios-specific configurations for a Kafka environment. | Parameter | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `spoof.dns` | Used while connecting to Kafka via a proxy, for example, when connecting across multiple cloud services. Use`spoof.dns`to re-route the connections to the proxy without modifying the Kafka broker configuration. | | `operation.timeout.ms` | Specifies a timeout for operations such as metadata requests and message consumption/production. This value can be adjusted based on the size of the consumed/produced dataset.**Default**: 10 seconds`CONFIG '{"operation.timeout.ms" : "10000"}'` | | `sasl.kerberos.ccache` | Specifies the Kerberos credential cache location used by Kafka pipelines. This value gets set as the`KRB5CCNAME`environment variable for the pipeline extractor process.If this option is not specified, thenSingleStorecreates a per-pipeline credential cache directory at`/`.The default value of`sasl.tmpdir`is`/tmp`.For example,`CONFIG '{"sasl.kerberos.ccache" : "FILE:/var/lib/memsql/krb5cc_kafka"}'` | | `sasl.kerberos.disable.kinit` | Disables the internal`kinit`invocation used by the pipeline extractor. Use this option when an external process, such as a cron job or sidecar, manages the Kerberos ticket cache.Set the value to the string`"true"`(not a JSON boolean). When enabled,`sasl.kerberos.kinit.cmd`is internally overridden with a no-op command.For example,`CONFIG '{"sasl.kerberos.disable.kinit" : "true"}'` | The `CONFIG` clause of a Kafka pipeline can accept a `spoof.dns` element as an alternative to configuring Kafka brokers. The `spoof.dns` element must be a JSON object consisting of an arbitrary number of key-value pairs with URL string values. When the pipeline attempts to connect to a Kafka broker whose URL matches one of the keys, the pipeline will connect to the corresponding URL value, effectively remapping the broker URLs inside the pipeline Kafka client. This `CREATE PIPELINE` command will let you set the AWS private link configuration for Kafka Brokers with AWS MSK. ```sql CREATE PIPELINE AS LOAD DATA KAFKA ':/' CONFIG '{ "spoof.dns": { ":":":", ":":":", ":":":", } }' INTO TABLE ; ``` There are a few more configuration options that are supported by Kafka. Consult the `CONFIGURATION.md` file in the [librdkafka](https://github.com/confluentinc/librdkafka/tree/v1.9.2) project in GitHub to see the full list. > **📝 Note**: Some configuration options are not supported in SingleStore. The client will receive a `"Forbidden Key"` error when accessing unsupported configuration options. Do not set `sasl.kerberos.kinit.cmd` directly in a Kafka pipeline `CONFIG`. SingleStore blocks this key for security reasons. To disable the internal `kinit` invocation when an external process manages the Kerberos ticket cache, set `"sasl.kerberos.disable.kinit"` to `"true"` instead. The configuration below controls some of the various aspects of the consumer's behavior (e.g., timeouts, fetching behavior, and message handling). These parameters can be adjusted to optimize the performance and reliability of the Kafka consumer based on your environment and requirements. ```sql CREATE PIPELINE p AS LOAD DATA kafka 'host.example.com:9092/whatever' CONFIG '{"fetch.max.bytes": "52428800", "topic.metadata.refresh.interval.ms": "300000", "message.max.bytes": "1000000", "fetch.wait.max.ms": "500", "session.timeout.ms": "45000", "topic.metadata.refresh.fast.interval.ms": "100", "fetch.min.bytes": "1", "max.partition.fetch.bytes": "1048576", "fetch.message.max.bytes": "1048576", "socket.keepalive.enable": "true", "fetch.error.backoff.ms": "500", "socket.timeout.ms": "60000"}' INTO TABLE t format CSV; ``` The following configuration sets some of the different communication options that are used with Kafka brokers (e.g., timeouts, batching behavior, and resource usage). These parameters should be based on your application requirements and specific Kafka deployment environment. ```sql CREATE PIPELINE p AS LOAD DATA kafka 'host.example.com:9092/whatever2' CONFIG '{"connections.max.idle.ms": "230000", "client.id": "", "fetch.max.bytes": "1000000", "operation.timeout.ms": "30000", "batch.num.messages": "1000", "socket.keepalive.enable": "false", "socket.timeout.ms": "60000"}' INTO TABLE t format CSV; ``` ## S3 Configurations The following table shows the SingleStore Helios-specific configurations for S3. | Parameter | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disable_gunzip` | When this parameter is set to`true`, files with the`.gz`extension are not decompressed.When this parameter is disabled or missing, files with the`.gz`extension are decompressed.`CONFIG '{"disable_gunzip" : true}'` | | `request_payer` | Specifies who is responsible for paying for the data transfer and request costs associated with accessing an S3 bucket.By default, the owner of an S3 bucket is responsible for paying these costs. However, when using the`request_payer`parameter, the requester will be responsible for covering the costs associated with the request. This can include costs such as`GET`,`PUT`, and`LIST`requests, as well as data transfer charges.`CONFIG '{"request_payer" : "name"}'` | | `endpoint_url` | Specifies the URL of the S3-compatible storage provider. This parameter can be used to direct requests to a non-standard endpoint, such as an S3-compatible service other than AWS. For example, MiniO, which is an S3-compatible storage provider, or a private cloud object storage which exposes an interface like S3.`CONFIG '{"endpoint_url" : "sample_url"}'` | | `compatibility_mode` | Instructs the downloader to use S3 API calls that are better supported by third parties.`CONFIG '{"compatibility_mode" : true}'` | | `file_compression` | Decompresses files with the specified extensions. It can have the following values:`"gz"`,`"lz4"`,`"auto"`, and`"disable"`. This parameter overrides`disable_gunzip`.`CONFIG '{"file_compression" : "gz"}'` | | `file_time_threshold` | If set, files last modified before the specified timestamp are not ingested. The timestamp must be specified in the[Unix Timestamp](https://www.unixtimestamp.com/)format represented as an integer value.`CONFIG '{"file_time_threshold" : 10070010}'` | No `CONFIG` clause is required to create an S3 pipeline. This clause is used to specify things like the Amazon S3 region where the source bucket is located or an entrypoint for an S3-compatible object sore. If no `CONFIG` clause is specified, SingleStore Helios will automatically use the `us-east-1` region, also known as `US Standard` in the Amazon S3 console. To specify a different region, such as `us-west-1`, include a `CONFIG` clause as shown in the example below. The `CONFIG` clause can also be used to specify the `suffixes` for files to load. These suffixes are a JSON array of strings. When specified, `CREATE PIPELINE` only loads files that have the specified suffix. Suffixes in the `CONFIG` clause can be specified without a `.` before them, for example, `CONFIG '{"suffixes": ["csv"]}'`. ```sql CREATE OR REPLACE PIPELINE AS LOAD DATA S3 'data-test-bucket' CONFIG '{"region": "us-east-1","request_payer": "requester", "endpoint_url": "https://storage.googleapis.com", "compatibility_mode": true}' CREDENTIALS '{"aws_access_key_id": "ANIAVX7U2LM9QVJMK2ZT", "aws_secret_access_key": "xxxxxxxxxxxxxxxxxxxxxxx"}' INTO TABLE 'market_data' (ts, timestamp, event_type, ticker, price, quantity, exchange, conditions); ``` ## Azure Blob Configurations The following table shows the SingleStore Helios-specific configurations for Azure Blobs. | Parameter | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disable_gunzip` | When this parameter is set to`true`, files with the`.gz`extension are not decompressed.When this parameter is disabled or missing, files with the`.gz`extension are decompressed.`CONFIG '{"disable_gunzip" : true}'` | Note that no `CONFIG` clause is required to create an Azure pipeline unless you need to specify the `suffixes` for files to load. These suffixes are a JSON array of strings. When specified, `CREATE PIPELINE` only loads files that have the specified suffix. Suffixes in the `CONFIG` clause can be specified without a `.` before them, for example, `CONFIG '{"suffixes": ["csv"]}'`. ## GCS Configurations The following table shows the SingleStore Helios-specific configurations for GCS. | Parameter | Description | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disable_gunzip` | When this parameter is set to`true`, files with the`.gz`extension are not decompressed.When this parameter is disabled or missing, files with the`.gz`extension are decompressed.`CONFIG '{"disable_gunzip" : true}'` | ## HDFS Configurations The following table shows the SingleStore Helios-specific configurations for HDFS. | Parameter | Description | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disable_partial_check` | When this parameter is set to`true`, a pipeline is created that imports Hive output files. When the pipeline runs, the extractor imports files, but does not check for additional files in the directory.`CONFIG '{"disable_partial_check" : true}'` | | `disable_gunzip` | When this parameter is set to`true`, files with the`.gz`extension are not decompressed.When this parameter is disabled or missing, files with the`.gz`extension are decompressed.`CONFIG '{"disable_gunzip" : true}'` | ## Filesystem Configurations The following table shows the SingleStore Helios-specific configurations for the filesystem. | Parameter | Description | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `disable_gunzip` | When this parameter is set to`true`, files with the`.gz`extension are not decompressed.When this parameter is disabled or missing, files with the`.gz`extension are decompressed.`CONFIG '{"disable_gunzip" : true}'` | | `process_zero_byte_files` | When this parameter is set to`true`, zero-byte files are processed.When this parameter is disabled or missing, zero-byte files are not processed.`CONFIG '{"process_zero_byte_files" : true}'` | *** Modified at: June 17, 2026 Source: [/cloud/load-data/data-sources/configuration-options-for-different-sources/](https://docs.singlestore.com/cloud/load-data/data-sources/configuration-options-for-different-sources/) (An index of the documentation is available at /llms.txt) --- # Iceberg Ingest > **📝 Note**: This is a Preview feature. [Apache Iceberg](https://iceberg.apache.org/) is an open-source table format that helps simplify analytical data processing for large datasets in data lakes. SingleStore can be used to add a speed layer to Iceberg tables. Iceberg tables can be directly ingested into SingleStore without the need for an external ETL tool and subsequently processed using SingleStore's high-performance database engine. ## Remarks The following are supported: * Iceberg Version 1 tables and Iceberg Version 2 tables with data files in [Parquet](https://parquet.apache.org/) format. * Iceberg tables stored in Amazon S3 with catalogs: GLUE, Snowflake, REST, JDBC, Hive, Hadoop, Polaris, and Unity. > **❗ Important**: The global engine variable `enable_iceberg_ingest` must be set to `ON` to use Iceberg Ingest. This variable is set to `OFF` by default.```sql > SET GLOBAL enable_iceberg_ingest = ON; > ``` ## Syntax ```sql CREATE [OR REPLACE] PIPELINE AS LOAD DATA S3 '[]' CONFIG '{"catalog_type": "GLUE|SNOWFLAKE|REST|JDBC|HIVE|HADOOP", [, "endpoint_url": ""] [, "ingest_mode": "append|upsert|one_time"] [, "catalog_name": "" ] [, “catalog.” : “property_value” [, …]] [, "iceberg_vended_credentials":true|false] }' CREDENTIALS '’ [REPLACE | MERGE] INTO TABLE FORMAT ICEBERG; :( { | @}<- [, ... ]) : {ident [::ident ...]} ``` All the data shaping options for Parquet pipelines are supported for Iceberg pipelines. Refer to [Data Shaping with Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/data-shaping-with-pipelines.md) for more information. Schema inference for Iceberg pipelines is supported. Refer to [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md) and [Infer Data from Iceberg Files](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference/#section-idm234989481470077.md) for details. ## Table Identifier * The `` identifies the Iceberg table. The `` is catalog-specific but is typically in the form: `database_name.table_name`. ## CONFIG ## Catalog Specification * The `catalog_type` is required for the catalog specification. * The `catalog_name` is a name to associate with the catalog when reading table metadata and is used internally in SingleStore for logging and metrics purposes. The `catalog_name` is required for the JDBC catalog and is optional for other catalogs. * The `catalog.property` is a list of key-value pairs for configuring the catalog connection. The property and value are passed directly to the Iceberg SDK to establish the catalog connection. Refer to [Iceberg Catalog and Security Configuration](https://docs.singlestore.com/#section-id235382377087001.md) for more information. ## S3 Specification * The `` is a JSON string for S3 configuration parameters such as `region`, `endpoint_url`, and `compatibility_mode`. Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) and [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) for more information. ## S3 Compatible Backend Storage SingleStore supports Amazon S3 compatible backend storage for all catalog types used with Iceberg tables. The `endpoint_url` is an Amazon S3 compatible storage backend, such as MinIO or IBM Storage Ceph. This value is distinct from the `endpoint_url` defined in the [S3 Specification](https://docs.singlestore.com/#section-idm234375301764122.md). ## Ingest Modes SingleStore supports multiple mechanisms for ingesting Iceberg data. SingleStore recommends using `MERGE` pipelines as the preferred method for ingesting Iceberg data. If `MERGE` pipelines do not meet performance or functional requirements, three alternative ingest modes are available using `ingest_mode:` `one_time`, `append`, and `upsert`. Refer to [Types of Ingest](https://docs.singlestore.com/#section-id235382401268725.md) for details and a comparison of the ingest mechanisms. > **❗ Important**: `MERGE` pipelines cannot be used with `ingest_mode` options. You cannot specify `MERGE` and an `ingest_mode` option in the same pipeline. ## Vended Credentials (`iceberg_vended_credentials`) SingleStore supports vended credentials for Iceberg REST catalogs with S3 storage. Refer to [Iceberg Catalog and Security Configuration](https://docs.singlestore.com/#section-id235382377087001.md) for details. ## AWS EKS IRSA (`eks_irsa`) SingleStore supports AWS Elastic Kubernetes Service (EKS) IAM vended credentials for Iceberg REST catalogs with S3 storage. Refer to [Iceberg Catalog and Security Configuration](https://docs.singlestore.com/#section-id235382377087001.md) for details. ## CREDENTIALS (``) * The `` specifies S3 credentials in JSON format. With this specification, credentials are directly included in the pipeline. While including credentials in the pipeline is simpler, it is less secure than using vended credentials or EKS IRSA. For information about the supported credential options, refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-e9b729f5-e821-593f-4780-33e798409694.md). * When using Vended Credentials or EKS IRSA, the `CREDENTIALS` clause is optional; however, the clause may be included to add additional authentication parameters. ## REPLACE INTO [REPLACE INTO](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/replace.md) is required for pipelines that use `ingest_mode: upsert`. SingleStore recommends that `REPLACE INTO` not be used with other types of pipelines as doing so causes unnecessary overhead. When you execute a `REPLACE INTO` statement on a table with a `PRIMARY KEY` or `UNIQUE` index, the engine checks for matching values in these indexes. If a match is found, it deletes the existing row and replaces it with the new row. If no match exists or if there is no `PRIMARY KEY` or `UNIQUE` index, `REPLACE INTO` operates like `INSERT`. ## MERGE INTO The `MERGE INTO` clause creates a pipeline, called a `MERGE` pipeline, that continuously ingests data from an Iceberg table into a SingleStore table. A `MERGE` pipeline continuously detects inserts, updates, and deletes to and from the Iceberg table and applies those updates to the SingleStore table. SingleStore recommends `MERGE` pipelines as the preferred method for ingesting Iceberg data. Refer to [Types of Ingest](https://docs.singlestore.com/#section-id235382401268725.md) for details on `MERGE` and a comparison of `MERGE` and `ingest_mode`. ## Subvalue Mappings The `iceberg_subvalue_mapping` assigns fields from the Iceberg table to columns in the SingleStore table or to temporary variables. A `::`-separated list of field names is used in `iceberg_subvalue_path` to look up fields in nested schemas. When the files in the Iceberg table are Parquet files, the `::`-separated list of field names is used to look up fields in nested Parquet schemas. The following rules apply: * The last field in `iceberg_subvalue_path` must be a primitive type. * All `iceberg_subvalue_path` components containing whitespace or punctuation must be surrounded by backticks (\`). * The `iceberg_subvalue_path` may not contain Parquet nested types (list or map types). Refer to [Parquet - Nested Types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types) for more information. ## Iceberg Catalog and Security Configuration ## Catalog Types SingleStore supports the following Iceberg Catalogs: Glue, Snowflake, REST, JDBC, Hive, and Hadoop. Polaris and Unity are REST catalogs and are supported. ## Vended Credentials SingleStore supports "vended credentials" for Iceberg REST catalogs with S3 storage. Vended credentials allow clients to access Iceberg tables without the Iceberg catalog credentials. The Iceberg catalog generates temporary limited credentials for a client to access the storage, improving the security and ease of catalog administration. Catalogs have catalog-specific settings for vended credentials. All such settings must be added as catalog properties using `"catalog."` in the `CONFIG` section. In addition, specify `"iceberg_vended_credentials": true` in the `CONFIG`. ## AWS Elastic Kubernetes Service (EKS) IAM Roles for Service Accounts (IRSA) AWS EKS IRSA can be used to provide credentials for a S3 pipeline. To use EKS IRSA: * Configure [delegated entities](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities/#section-id235390403966648.md) to use [Cloud Workload Identity](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities.md) instead of static credentials. * EKS IRSA must be enabled by setting the `enable_eks_irsa` engine variable to `ON`. ```sql SET GLOBAL enable_eks_irsa = ON; ``` * Include the `"creds_mode": "eks_irsa"` option in the pipeline `CONFIG`. SingleStore supports two scenarios for access with IAM roles and EKS IRSA: 1. The role assigned to a pod has direct access to S3 resources. 1. Include the `"creds_mode": "eks_irsa"` option in the pipeline `CONFIG`. ```sql CREATE PIPELINE AS LOAD DATA S3 's3://bucket_name/directory' CONFIG '{"region":"us-west-2", "creds_mode": "eks_irsa"}' CREDENTIALS '{}' INTO TABLE FORMAT ICEBERG; ``` 2. The role assigned to a pod does not have direct access to the S3 resources, but it has permissions to assume a role that has the desired access. 1. Provide the role to be assumed with `role_arn` in the pipeline `CREDENTIALS`. ```sql CREATE PIPELINE AS LOAD DATA S3 's3://bucket_name/directory' CONFIG '{"region":"us-west-2", "creds_mode": "eks_irsa"}' CREDENTIALS '{"role_arn": ""}' INTO TABLE FORMAT ICEBERG; ``` Refer to the Minimum Required S3 Pipeline Syntax and AWS Elastic Kubernetes Service (EKS) IAM Roles for Service Accounts (IRSA) Authentication examples in [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-e9b729f5-e821-593f-4780-33e798409694.md). ## AWS Secrets Manager Secrets Secrets in the AWS Secrets Manager can be used in the pipeline `CONFIG` or `CREDENTIALS`. To place a secret part of a pipeline `CONFIG` or `CREDENTIALS` into the AWS Secrets Manager: 1. Identify the part of the configuration that you want to hide. Use the secrets feature only for credentials; do not store the entire configuration into the Secrets Manager, as this may cause problems with the pipeline. 2. Add the secret part as a secret in AWS Secrets Manager. As AWS Secrets Manager stores secrets as JSON, a secret may contain multiple key-value pairs. For example, you might create a secret that contains values for `"catalog.credential"` and `"catalog.another_credential"`. 3. Ensure the pipeline's AWS credentials allow accessing the AWS Secrets Manager. Those credentials could be either static credentials, or a service role provided to the pod by EKS IRSA. 4. Remove the secret part or parts from the pipeline CONFIG and replace them with `"ref": "arn_of_your_secret"`. ```json { "catalog_type":"REST", "region":"us-east-1", "catalog.uri":"https://whatever.snowflakecomputing.com/polaris/api/catalog", "ref": "arn:aws:secretsmanager:us-east-1:188440000000:secret:test-secret-VkYPHc", ... } ``` Every `"ref"` property in the pipeline `CONFIG` or `CREDENTIALS` is replaced by the values contained in the secret mentioned in the property's value. These values are placed on the same level as the `"ref"` they replace. The AWS Secrets Manager can be accessed with a role that must be assumed before accessing the secret. Set this role in `CREDENTIALS` using `secretsmanager_role_arn`, or in `ref` by using a pair of role ARN and secret ARN separated by a pipe; for example: `"ref": "arn:aws:iam::188440000000:role/role_to_assume|arn:aws:secretsmanager:us-east-1:188440000000:secret:my-secret"`. When AWS credentials are used only to access the Secrets Manager, and not to access data on S3 (for example, when vended credentials are used to access S3) those credentials must be prefixed with the string `"secretsmanager_"`, such as, `"secretsmanager_role_arn"`. ## Types of Ingest ## Compare Types of Continuous Ingest SingleStore supports several types of continuous ingest: `MERGE` pipelines, Append-Only, and Upsert. With continuous ingest, a running pipeline automatically detects updates to the Iceberg table and ingests them into the SingleStore table. These types of ingest have different characteristics, SingleStore recommends using `MERGE` pipelines as the preferred method for ingesting Iceberg data. Append-Only and Upsert are provided as performance optimizations. * `MERGE` can process Append, Overwrite, Replace, and Delete (positional and equality) Iceberg snapshots; Append-Only and Upsert using `ingest_mode` have limitations on the types of Iceberg snapshots processed. * The `MERGE` pipeline has the broadest coverage of what can be ingested as incremental updates, merge pipelines are therefore inherently more complex. * `MERGE` requires adding three columns (`$_file`, `$_row`, `$_delete`) in the destination (SingleStore) table. These three extra columns have the following impacts: * `MERGE` is slightly slower on initial ingest compared to Append-Only mode. * `MERGE` takes slightly more space in storage. * The `MERGE` pipeline statement is more complex, but that can be mitigated with [INFER](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/infer-pipeline.md). * When using `MERGE`, users may not directly update the destination (SingleStore) table. If the destination (SingleStore) table is updated, updates made to the source Iceberg table may not propagate properly. * When using `MERGE`, the Replace type of snapshot is not a no-op. Data files will be re-ingested if the source table went through compaction. * `MERGE` implements merge-on-read (MOR). * `MERGE` requires an extra condition for `SELECT` statements to manage MOR-style deletes. * An extra entity (`VIEW`) is created by `INFER` by default. * Impacts scan performance. * `ingest_mode` updates the SingleStore table directly and does not have a read penalty. ## Snapshot Transactionality Snapshot transactionality controls whether the destination table reflects only fully ingested Apache Iceberg snapshots. When snapshot transactionality is enabled, the destination table does not expose an intermediate state after each internal pipeline batch. Instead, data becomes visible when ingestion has advanced to a fully ingested Iceberg snapshot. Snapshot transactionality can be enabled globally with the `pipelines_batch_matches_snapshot` engine variable (specified in [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md)) or per pipeline with `BATCH_MATCHES_SNAPSHOT ON` in `CREATE PIPELINE`. **Important**: If snapshot transactionality is enabled when a pipeline is created, the files discovered during pipeline creation may be ingested as a single transaction. For large tables, this transaction may be very large. SingleStore recommends creating the pipeline without snapshot transactionality, allowing the initial load to complete, and then enabling snapshot transactionality with `ALTER PIPELINE` for subsequent iterations. **Note**: For non-merge Iceberg pipelines, destination-table visibility may occur after more than one source snapshot has been fully ingested. This behavior can occur because snapshots may be batched internally. ## Merge Pipelines (`MERGE`) The `MERGE` clause creates a merge pipeline, that continuously ingests data from an Iceberg table into a SingleStore table. Append, Overwrite, Replace, and Delete (positional and equality) Iceberg V2 snapshots are supported. For a merge pipeline, the SingleStore (destination) table: * Must have the following three columns: * `` `$_file` VARCHAR(2048) COMMENT 'ICEBERG_FILE_PATH' `` * `` `$_row` BIGINT COMMENT 'ICEBERG_FILE_POS' `` * `` `$_delete` JSON DEFAULT '{}' COMMENT 'ICEBERG_DELETED_BY' `` * You may use different column names for the `` `$_file` ``, `` `$_row` ``, and `` `$_delete` `` columns as long as the columns are marked with the `COMMENT` clauses shown above. * Must have shard key defined as: * ``SHARD KEY(`$_file`, `$_row`)`` * Must not be updated or modified. If the destination table is modified, updates may be applied incorrectly. The pipeline declaration must set the `` `$_file` `` and `` `$_row` `` columns as follows: ```sql SET `$_file` = pipeline_source_file(), `$_row` = pipeline_source_file_row() ``` To ingest deletes, a merge pipeline must be created using a [VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md) over the destination table. This view must have the following `WHERE` clause: ```sql WHERE JSON_LENGTH(`$_delete`) = 0 ``` Merge pipelines perform merge-on-read on-demand during table scan when a view is used. [MERGE Pipeline](https://docs.singlestore.com/#section-idm235005322019542.md) provides a merge pipeline example. ## Append-Only (`ingest_mode`) Append-only mode is used for scenarios in which new rows are added to the Iceberg table, but no rows are deleted or modified. In append-only mode (`"ingest_mode":"append"`) the pipeline will process `APPEND` Iceberg snapshots. If a `DELETE` or `OVERWRITE` snapshot is encountered, an error is raised. Users may override those errors by specifying `"ignore_unsupported_modifications":true` in the pipeline configuration. SingleStore does not recommend setting `"ignore_unsupported_modifications":true` as doing so may lead to data inconsistency. ## Upsert (`ingest_mode`) Upsert mode is used for scenarios where updates modify non-key columns based on a specified key column(s). An upsert is an insert and update, an upsert updates a row in the table if there is a primary key match or inserts the row if there is no primary key match. In upsert mode (`"ingest_mode":"upsert"`), the pipeline will process updates to the Iceberg table as upserts to the SingleStore table. That is, the pipeline will process Iceberg `APPEND` and `OVERWRITE` snapshots of the Iceberg table as upserts to the SingleStore table. Requirements: * The SingleStore table must have a primary key or a unique index. In the ``, a column(s) in the Iceberg table must be mapped to the column(s) in SingleStore on which there is a key or unique index. Typically, the column(s) in the Iceberg table will also be declared as a key. * The pipeline must be created using `REPLACE`. Refer to [Additional CREATE PIPELINE Examples](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/additional-create-pipeline-examples/#section-idm4517954193070433070171573573.md) for more information. Limitations: * Pipelines will fail on delete snapshots. Users may override these errors by specifying `"ignore_unsupported_modifications":true` in the pipeline configuration. SingleStore does not recommend setting `"ignore_unsupported_modifications":true` as doing so may lead to data inconsistency. * Limitations are expected to be addressed in future releases. ## One-Time (ingest\_mode) A one-time ingest (`"ingest_mode":"one_time"`) is supported. In this mode, SingleStore requests Iceberg table metadata, and loads data from the latest snapshot available at that moment. Subsequent updates to the Iceberg table are not automatically ingested. ## CREATE OR REPLACE When a pipeline with a specified `pipeline_name` already exists, the `CREATE OR REPLACE` command functions similarly to `CREATE PIPELINE`, with the added benefit of preserving existing pipeline metadata, such as loaded offsets and data files. Running `CREATE OR REPLACE` on an existing pipeline initiates the Iceberg pipeline to retrieve a new snapshot, schema, and data files, and inserts data from these new files into the destination table in SingleStore. Executing `CREATE OR REPLACE` on an existing Iceberg pipeline may cause some data files to be ingested twice. To avoid this, use `CREATE OR REPLACE` only with `REPLACE` statements or in an upsert configuration. ```sql CREATE PIPELINE books_create_pipe AS LOAD DATA S3 'db.books' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog", "ingest_mode": "one_time" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' REPLACE INTO TABLE books (Id <- id, Name <- name, NumPages <- numPages, Rating <- rating) FORMAT ICEBERG; ``` Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for syntax for `CREATE OR REPLACE PIPELINE`, `REPLACE INTO TABLE`, and `ON DUPLICATE KEY UPDATE`. Refer to [Performing Upserts](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/performing-upserts.md) for more information on upserts. ## Iceberg Pipeline Configuration ## Data Type Translation The following table lists Iceberg Types, the SingleStore data type that can be used to store those types, and the recommended conversion to be applied with a `SET` clause. | **Iceberg Type** | **RecommendedSingleStoreData Type** | **Recommended Conversion** | | ---------------- | ----------------------------------- | --------------------------------------------------------------- | | boolean | `TINYINT`/`BOOL`/`BOOLEAN` | | | int | `INT` | | | long | `BIGINT` | | | float | `FLOAT` | | | double | `DOUBLE` | | | decimal(P,S) | `DECIMAL` | | | date | `DATE` | DATE\_ADD('1970-01-01', INTERVAL @date DAY) | | time | `TIME(6)` | DATE\_ADD('1970-01-01', INTERVAL @time\_value MICROSECOND) | | timestamp | `DATETIME(6)` | DATE\_ADD('1970-01-01', INTERVAL @timestamp\_value MICROSECOND) | | timestamptz | `DATETIME(6)` | DATE\_ADD('1970-01-01', INTERVAL @timestamp\_value MICROSECOND) | | string | `LONGTEXT (utf8mb4_bin)` | | | uuid | `BINARY(16)` | | | fixed (L) | `BINARY(L)` | | | binary | `LONGBLOB` | | ## Data File Filtering When ingesting from Iceberg tables, SingleStore applies file-level filtering to reduce the number of data files downloaded. File-level filtering is determined by the `WHERE` clause in the pipeline definition. * Only a subset of expressions can be used for file filtering. * Expressions that are not eligible for file-level filtering are applied later as row-level filtering after the file is downloaded. To enable file filtering before download, write the `WHERE` clause using the following rules: * Use any combination of the boolean operators `AND` and `OR`. * Use the binary operators `=`, `<`, `<=`, `>`, `>=` with a column name on one side and a literal value on the other. For example, `WHERE` `column_name_1 < 9999` * Function calls are not supported in file filtering. Pipelines always evaluate the full `WHERE` clause during ingestion, after file-level filtering. * Row-level filtering may include more complex expressions, such as function calls, that are not eligible for file-level filtering. * When the `WHERE` clause uses an `AND` operator, unsupported expressions can be discarded for file-level filtering while still being applied during row-level filtering. For example, `WHERE col1 > 1000 AND col2 = my_udf(col3)` * Per-file filtering: `col1 > 1000` * Per-row filtering: `col1 > 1000 AND col2 = my_udf(col3)` If none of the expressions in the `WHERE` clause can be applied at the file level, the `CREATE PIPELINE` statement issues the following warning: `Pipeline's where clause is not suitable for file-level filtering` File-level filtering relies on column-level metadata that Iceberg generates for each data file. Refer to [Manifest Entry Fields](https://iceberg.apache.org/spec/#manifest-entry-fields) for more information. ## Monitor Ingest The per-file status of the Iceberg table ingestion can be queried through the [PIPELINES\_ICEBERG\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-iceberg-files.md) information schema view. This view lists all files currently tracked by the pipeline and their corresponding status. Enabling `OFFSETS_METADATA_GC` on the Iceberg pipeline reduces the number of entries by tracking files only referenced by the pipeline’s current Iceberg table snapshot. The option `OFFSETS_METADATA_GC` can be set when you [CREATE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) the pipeline or modify it using [ALTER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/alter-pipeline.md). ## Configure Iceberg Ingest The global engine variable `enable_iceberg_ingest` must be set to `ON` to use Iceberg ingest. This variable is set to `OFF` by default. ```sql SET GLOBAL enable_iceberg_ingest = ON; ``` Engine variables can be used to control parameters including memory usage and timeouts for Iceberg pipelines. The default values of these engine variables are defined to work well for common Iceberg pipelines uses. Pipeline parameters including memory usage and timeouts can be controlled using global engine variables with the `pipelines_*` prefix as specified in [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md). Memory usage can be controlled using the `pipelines_iceberg_heap_size` and `pipelines_iceberg_data_workers_heap_size` engine variables. `pipelines_iceberg_heap_size` controls JVM memory used to fetch Iceberg metadata, such as connecting to the catalog and downloading or parsing table manifests. `pipelines_iceberg_data_workers_heap_size` controls memory used by extractor workers that download and fetch Parquet files. This variable is relevant only to non-merge pipelines that process merge-on-read (MOR) snapshots. If Java heap space out-of-memory errors occur while fetching Iceberg metadata, start by increasing `pipelines_iceberg_heap_size` from the default value of 96 MB, for example to 128 MB and then 256 MB, until the errors stop. The appropriate value depends on the workload and the size of the metadata being processed. Pipeline timeouts can be controlled using `pipelines_extractor_get_offsets_timeout_ms` and `pipelines_extractor_idle_timeout_ms`. For Iceberg pipelines, the values of these variables are adjusted to be a minimum of 5 minutes. Increase the values of these variables if there are timeouts in pipeline creation or while running pipeline batches. The `pipelines_batch_matches_snapshot` engine variable sets the default snapshot-transactionality behavior for Iceberg pipelines. A per-pipeline `BATCH_MATCHES_SNAPSHOT` setting overrides the global value. Refer to [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for more information. ## Examples ## Glue Catalog on Amazon S3 An Iceberg table with data files in Parquet format that is stored in an AWS S3 bucket using AWS Glue can be loaded into a SingleStore table using a pipeline ([CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md). Refer to [Apache Iceberg - Glue Catalog](https://iceberg.apache.org/docs/1.5.0/aws/#glue-catalog) for information on using a GLUE catalog with Iceberg. In this example, a table named `books` is created and data from an Iceberg table that meets this schema is loaded into the `books` table. Create the table. ```sql CREATE TABLE books( Id INT, Name TEXT, NumPages INT, Rating DOUBLE, PRIMARY KEY(Id)); ``` The following data is used for this example. ```shell (1, 'Happy Place', 400, 4.9) (2, 'Legends & Lattes', 304, 4.9) (3, 'The Vanishing Half', 352, 4.9) (4, 'The Wind Knows My Name', 304, 4.9) ``` The `PIPELINE` statement below will load data from an Iceberg table containing the data above into the `books` table. The column names on the left side of the `<-` are the column names from the SingleStore table into which the data will be loaded. The column names on the right side of the `<-` are the column names from the Iceberg table which is to be loaded into SingleStore. ```sql CREATE PIPELINE books_pipe AS LOAD DATA S3 'db.books' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE books (Id <- id, Name <- name, NumPages <- numPages, Rating <- rating) FORMAT ICEBERG; ``` Test the pipeline. ```sql TEST PIPELINE books_pipe; ``` ```output +------+------------------------+----------+--------+ | Id | Name | NumPages | Rating | +------+------------------------+----------+--------+ | 4 | The Wind Knows My Name | 304 | 4.9 | | 1 | Happy Place. | 400 | 4.9 | | 2 | Legends & Lattes | 304 | 4.9 | | 3 | The Vanishing Half | 352 | 4.9 | +------+------------------------+----------+--------+ ``` Refer to [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) for more information on starting pipelines. ## Use Subvalue Mappings This example shows the use of subvalue mappings to load nested elements from an Iceberg schema into a SingleStore table. Create a table. ```sql CREATE TABLE addresses( Id INT, Name TEXT, Street TEXT, City TEXT, Country TEXT, PRIMARY KEY(Id)); ``` The following data is used for this example. ```shell (1, 'Mylo', struct('123 Main St', 'New York', 'USA')) (2, 'Naya', struct('456 Elm St', 'San Francisco', 'USA')) ``` The `PIPELINE` statement below will load data from an Iceberg table containing the data above into the `addresses` table. The column names on the left side of the `<-` are the column names from the SingleStore table into which the data will be loaded. The column names on the right side of the `<-` are the column names from the Iceberg table which is to be loaded into SingleStore. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db2.addresses' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` Test the pipeline. ```sql TEST PIPELINE addresses_pipe; ``` ```output +------+------+-------------+---------------+---------+ | Id | Name | Street | City | Country | +------+------+-------------+---------------+---------+ | 1 | Mylo | 123 Main St | New York | USA | | 2 | Naya | 456 Elm St | San Francisco | USA | +------+------+-------------+---------------+---------+ ``` Refer to [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) for more information on starting pipelines. ## Snowflake Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 with a Snowflake catalog. Iceberg tables to be ingested in SingleStore must be created on an external volume. Refer to [Tutorial: Create your first Iceberg table](https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table), [Create an external volume](https://docs.snowflake.com/en/user-guide/tutorials/create-your-first-iceberg-table#create-an-external-volume), and [Snowflake Iceberg Catalog SDK](https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog) for more information. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.schema_name.table_name' CONFIG '{"region": "us-west-2", "catalog_type": "SNOWFLAKE", "catalog.uri": "jdbc:snowflake://tpq12345.snowflakecomputing.com", "catalog.jdbc.user":"", "catalog.jdbc.password":"", "catalog.jdbc.role":""}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` For the Snowflake catalog, the `` must consist of three parts - the database name, the schema name, and the table name, `db_name.schema_name.table_name` in the example above. The catalog.uri can be obtained from running `SELECT SYSTEM$ALLOWLIST();` in the Snowflake system. In addition, the `catalog.uri`, `catalog.jdbc.user`, `catalog.jdbc.password`, and `catalog.jdbc.role` are required when using the Snowflake catalog. ## REST Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 with REST catalog. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.table_name' CONFIG '{"region": "us-west-2", "catalog_type": "REST", "catalog.uri": "http://host.address:8181"}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` ## File Filtering with REST Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 with REST catalog using file filtering. Assume there exists an Iceberg table that stores weather sensor data and is partitioned with a day transform on the timestamp as follows. ```sql CREATE TABLE sensors.weather ( sensorid int, temperature double, humidity double, pressure double, ts timestamp) PARTITIONED BY day(ts); ``` This data is to be loaded into a SingleStore table defined as follows: ```sql CREATE TABLE weather_data( SensorId INT, Temperature DOUBLE, Humidity DOUBLE, Pressure DOUBLE, TS DATETIME, PRIMARY KEY(SensorId)); ``` The following pipeline will load all weather data for January 1, 2026 and later into the `weather_data` table. As the Iceberg table is partitioned by day(ts), SingleStore will read and load only the Iceberg files containing data for 2026 (and future years). ```sql CREATE PIPELINE weather_pipe AS LOAD DATA S3 'sensors.weather' CONFIG '{"region": "us-west-2", "catalog_type": "REST", "catalog.uri": "http://host.address:8181"}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE weather_data (SensorId <- sensorid, Temperature <- temperature, Humidity <- humidity, Pressure <- pressure, TS <- ts) FORMAT ICEBERG WHERE timestamp >= '2026-01-01'; ``` ## JDBC Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 with JDBC catalog. Refer to [Iceberg JDBC Integration](https://iceberg.apache.org/docs/1.5.0/jdbc/) for more information on using Iceberg with JDBC catalog. SingleStore supports Postgres, MySQL, and SQLite JDBC drivers by default. The following example uses JDBC with SQLite. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.table_name' CONFIG '{"region": "us-west-2", "catalog_type": "JDBC", "catalog_name": "catalog_name", "catalog.warehouse": "s3://path_to_warehouse", "catalog.uri":"jdbc:sqlite_:file:/path_jdbc"}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` The following example uses JDBC with MySQL. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.table_name' CONFIG '{"region": "us-west-2", "catalog_type": "JDBC", "catalog_name": "catalog_name", "catalog.warehouse": "s3://path_to_warehouse", "catalog.uri": "jdbc:mysql://host.address:3306/default", "catalog.jdbc.user": "", "catalog.jdbc.password": ""}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Country <- address::country) FORMAT ICEBERG; ``` The `catalog.warehouse` and `catalog_name` are required for JDBC catalogs. ## Hive Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 using Hive Catalog. Iceberg tables to be ingested in the Hive catalog must use Hive Metastore service. Refer to [Apache Hive](https://hive.apache.org/) and [Hive - Apache Iceberg](https://iceberg.apache.org/docs/latest/hive/) for more information. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.table_name' CONFIG '{"catalog_type": "HIVE", "catalog.uri": "thrift://:46590", "region": "us-east-1", "catalog.hive.metastore.client.auth.mode": "PLAIN", "catalog.hive.metastore.client.plain.username": "", "catalog.hive.metastore.client.plain.password": "", "catalog.metastore.use.SSL": "true", "catalog.hive.metastore.truststore.type": "PKCS12", "catalog.hive.metastore.truststore.path": "/path/to/your/project/hive/truststore.12" "catalog.hive.metastore.truststore.password": '' }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` The `catalog.uri` is the base URL for accessing the Hive catalog's API or service endpoint. The `catalog.hive.metastore.client.auth.mode` is the authentication mode for connecting to the Hive Metastore. The `catalog.hive.metastore.client.plain.username` is the username used to authenticate with the Hive Metastore. The `catalog.hive.metastore.client.plain.password` is the password for the authenticated user. The `catalog.metastore.use.SSL` is a boolean flag that secures communication with the Hive Metastore. The `catalog.hive.metastore.truststore.type` is the truststore format used to validate the SSL certificate. The `catalog.hive.metastore.truststore.path` is the file path that contains the SSL certificate. The `catalog.hive.metastore.truststore.password` is the password needed to access the truststore. Refer to [GitHub](https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java) for additional Hive configurations. ## Hadoop Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 with Hadoop catalog. Hadoop catalog supports S3 compatible backend storages. Refer to [Apache Hadoop](https://hadoop.apache.org/) and [Hadoop - Apache Iceberg](https://iceberg.apache.org/javadoc/nightly/org/apache/iceberg/hadoop/HadoopCatalog.html) for more information. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA s3 'db_name.table_name' CONFIG '{"catalog_type": "HADOOP", "catalog.warehouse": "s3://path_to_warehouse", "endpoint_url": "", "region": "us-east-1" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key":""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` ## Polaris Catalog on Amazon S3 Ingest an Iceberg table stored in Amazon S3 using Polaris Catalog. Polaris catalog uses the Apache Iceberg REST API. Refer to [Getting Started with Snowflake Open Catalog](https://other-docs.snowflake.com/opencatalog/tutorials/open-catalog-gs) for more information. ```sql CREATE PIPELINE addresses_pipe AS LOAD DATA S3 'db_name.table_name' CONFIG '{"catalog_type": "REST", "catalog.warehouse": "", "table_id": "db_name.table_name", "region":"us-east-1", "catalog.uri":"https://tpq12345.snowflakecomputing.com/polaris/api/catalog", "catalog.credential":"catalog.credential", "catalog.scope": "PRINCIPAL_ROLE:ALL"}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE addresses (Id <- id, Name <- name, Street <- address::street, City <- address::city, Country <- address::country) FORMAT ICEBERG; ``` The `catalog.uri` is the base URL for accessing the Polaris catalog's API or service endpoint. The `catalog.scope` defines the access permissions for the Polaris catalog. The `catalog.connection` is a secret key from Polaris catalog connection, formatted as `:`. Get this when you configure the service connection in Snowflake while creating a connection for a Polaris catalog. ## Upsert (`ingest_mode`) The example below shows a pipeline using `ingest_mode` of `upsert`. As required when `ingest_mode` is `upsert`, the books table has a primary key, `Id`, in this example. When started, this pipeline will ingest data from the latest snapshot of the Iceberg table into the SingleStore `books` table. Then, when the Iceberg table is updated, those updates will automatically be applied to the SingleStore table as upserts. ```sql CREATE PIPELINE books_upsert_pipe AS LOAD DATA S3 'db.books' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog", "ingest_mode": "upsert" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' REPLACE INTO TABLE books (Id <- id, Name <- name, NumPages <- numPages, Rating <- rating) FORMAT ICEBERG; ``` ## Append (`ingest_mode`) The example below shows a pipeline using `ingest_mode` of `append`. When started, this pipeline will ingest data from the latest snapshot of the Iceberg table into the SingleStore `books` table. Then, when the Iceberg table is updated, any appends to that table will automatically be appended to the SingleStore table. If non-append updates are encountered, an error will be thrown. ```sql CREATE PIPELINE books_append_pipe AS LOAD DATA S3 'db.books' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog", "ingest_mode": "append" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE books (Id <- id, Name <- name, NumPages <- numPages, Rating <- rating) FORMAT ICEBERG; ``` ## MERGE Pipeline The example below shows an example of a creating a merge pipeline. In this example, a destination table in SingleStore , named `_books` , is created with the necessary `$_file`, `$_row`, and `$_delete` columns. Then a view, `books` , is created on top of the `_books` table. Finally, a merge pipeline is defined. When started, this pipeline will ingest data from the latest snapshot of the source Iceberg table into the SingleStore `_books` table. When the source Iceberg table is updated, those updates (append, replace, delete) will be propagated to the `_books` table. Create the destination `_books` table. ```sql CREATE TABLE _books( Id INT, Name TEXT, NumPages INT, Rating DOUBLE, `$_file` VARCHAR(2048), `$_row` BIGINT, `$_delete` JSON DEFAULT '{}', KEY(Id), SHARD KEY(`$_file`,`$_row`)); ``` Create the `books` view. ```sql CREATE VIEW books AS SELECT Id, Name, NumPages, Rating from _books WHERE JSON_LENGTH(`$_delete`) = 0; ``` Define the pipeline. ```sql CREATE PIPELINE books_pipe AS LOAD DATA S3 'db.books' CONFIG '{"region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog" }' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' MERGE INTO TABLE _books (Id <- id, Name <- name, NumPages <- numPages, Rating <- rating) FORMAT ICEBERG SET `$_file` = pipeline_source_file(), `$_row` = pipeline_source_file_row(); ``` ## Debug Datetime Conversions Datetime value conversion can be debugged by loading raw datetime from an Iceberg file into a `BLOB`. The example below creates a `books_debug` table with a `PublishTimestamp` attribute of type `DATETIME(6)` and an extra attribute, `RawTimestamp`, into which the raw timestamp data will be loaded. If the datetimes are stored as strings, try loading the raw values into a `TEXT` attribute in addition to, or instead of, using a `BLOB` attribute. ```sql CREATE TABLE books_debug( Id INT, Name TEXT, NumPages INT, Rating DOUBLE, PublishTimestamp DATETIME(6), RawTimestamp BLOB); ``` The `PIPELINE` statement below loads the `publishTimestamp` from the Iceberg file into both the `PublishTimestamp` column and the `RawTimestamp` column. A `SET` statement is used to convert the `publishTimestamp` from Iceberg to the `PublishTimestamp` in SingleStore. The column names on the left side of the `<-` are the column names from the SingleStore table into which the data will be loaded. The column names on the right side of the `<-` are the column names from the Iceberg file which is to be loaded into SingleStore. ```sql CREATE PIPELINE books_debug_pipe AS LOAD DATA S3 '' CONFIG '{ "region":"us-west-2", "catalog_type": "GLUE", "catalog_name": "s3_catalog", "table_id": "dbtest.books" }' CREDENTIALS '{ "aws_access_key_id": "", "aws_secret_access_key": "", "aws_session_token": ""}' INTO TABLE books_debug (Id <- IdIce, Name <- Name, NumPages <- NumPages, Rating <- Rating, @ts<-PublishTimestamp, RawTimestamp<-PublishTimestamp ) FORMAT ICEBERG SET PublishTimestamp = DATE_ADD('1970-01-01', INTERVAL @ts MICROSECOND); ``` Use `TEST PIPELINE` to compare the values in the two columns. The `PublishTimestamp` column shows the converted timestamp, the `RawTimestamp` column contains the timestamp from the Iceberg file, in microseconds in this example. The combination of these two columns can be used to debug the conversion used in the `SET` statement. ```sql TEST PIPELINE books_debug_pipe; ``` ```output +------+--------------------+----------+--------+------------------+-------+---------------------+ | Id | Name | NumPages | Rating | RawTimestamp | PublishTimestamp | +------+--------------------+----------+--------+------------------+-------+---------------------+ | 1 | Happy Place | 400 | 4.9 | 1680721200000000 | 2023-04-05 12:00:00.000000 | | 2 | Legends & Lattes | 304 | 4.9 | 1669665600000000 | 2022-11-28 12:00:00.000000 | | 3 | The Vanishing Half | 352 | 4.9 | 1591124400000000 | 2020-06-02 12:00:00.000000 | +------+--------------------+----------+--------+------------------+-------+---------------------+ ``` ## Troubleshooting The following table lists errors that can occur when creating an Iceberg Ingest pipeline. Also, refer to [Debugging Pipeline Errors](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/debugging-pipeline-errors.md) for additional information on troubleshooting pipeline errors. | Catalog | Error | Cause and Resolution | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Snowflake | Certificate for <...> doesn't match any of the subject alternative names. | An incorrect URI may cause this error.Verify that the catalog.uri is valid. | | Snowflake | SEVERE: WARNING!!! Using fail-open to connect. Driver is connecting to an HTTPS endpoint without OCSP based Certificate Revocation checking as it could not obtain a valid OCSP Response to use from the CA OCSP responder. | This issue needs to be resolved on the Snowflake side. Refer to[OCSP Configuration](https://docs.snowflake.com/en/user-guide/ocsp#ocsp-certification-checks-require-port-80)for more information | | Snowflake | Parquet parsing errors such as “Dictionary encoding not implemented". | Set Snowflake table property`STORAGE_SERIALIZATION_POLICY = COMPATIBLE`as in[CREATE ICEBERG TABLE (Snowflake as the Iceberg catalog) \| Snowflake Documentation](https://docs.snowflake.com/en/sql-reference/sql/create-iceberg-table-snowflake). | | All | protocol error … Process died unexpectedly or didn't start. | An incorrect value of the engine variable`java_pipelines_java11_path`may cause this error. Ensure sure the path is valid, for example, “/usr/bin/java”. | | All | java.lang.OutOfMemoryError: Java heap space | This error occurs if:
  • Heap memory usage exceeds the pipelines_iceberg_heap_size variable limit
  • There is an excessive amount of table metadata
  • Parquet row groups are very large
To resolve:
  • If the Parquet row groups are very large, switch to a MERGE pipeline by removing the ingest_mode option from the CONFIG call and adding MERGE so the statement becomes MERGE INTO <table_name>
  • Increase the value of the pipelines_iceberg_heap_size engine variable
| | All | Unsupported snapshot with operation`%s`for table`%s`(snapshot=`%d`). | This error occurs when the Iceberg pipeline is used on an unsupported snapshot type.To resolve:
  • Set ignore_unsupported_modifications config value to true to ignore it.
  • If applicable (for example, when you want to ingest updates), set ingest_mode to upsert so that updates are processed.
  • Alternatively, remove the ingest_mode option from the CONFIG and use a MERGE pipeline instead, so the statement becomes MERGE INTO <table_name>.
| ## Related Topics * [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) * [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) *** Modified at: August 3, 2026 Source: [/cloud/load-data/data-sources/iceberg-ingest/](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest/) (An index of the documentation is available at /llms.txt) --- # Load Data from Amazon Kinesis Using a Kafka Connect Pipeline > **📝 Note**: To access this feature, enable [Opt-in to Preview Features & Updates](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters/#section-id235151491495792.md) while creating a cluster to deploy a SingleStore Helios 9.1 RC on Helios. SingleStore pipelines can extract streaming data from Amazon Kinesis Data Streams using Kafka Connect source connectors, optionally transform them, and insert them into a destination table. SingleStore Kafka Connect Pipelines leverage the Kafka Connect ecosystem to stream data from external systems into SingleStore without requiring an intermediate Kafka cluster. ## Prerequisites To complete this guide, your environment must meet the following prerequisites: * **AWS Account**: This guide uses Amazon Kinesis and requires an AWS account's access key ID and secret access key. * **SingleStore Helios installation -or- a SingleStore Helios cluster**: You will connect to the cluster and create a pipeline to pull data from your Amazon Kinesis Data Stream. * **Kafka Connect Pipelines enabled**: This is an experimental feature that must be explicitly enabled by a user with the `SUPER` permission before creating pipelines. ## Part 1: Enable Kafka Connect Pipelines Run the following command to enable this feature: ```sql SET GLOBAL experimental_features_config = "kafka_connect_enabled=true" ``` > **📝 Note**: This setting must be configured before creating Kafka Connect Pipelines and requires the `SUPER` permission. The setting persists across cluster restarts and changes take effect immediately. Verify that the feature is enabled: ```sql SHOW VARIABLES LIKE 'experimental_features_config' ``` ```output +------------------------------+----------------------------+ | Variable_name | Value | +------------------------------+----------------------------+ | experimental_features_config | kafka_connect_enabled=true | +------------------------------+----------------------------+ ``` ## Part 2: Set Up Amazon Kinesis Data Stream ## Create a Kinesis Data Stream 1. Log into the AWS Management Console. 2. Navigate to **Kinesis**. 3. Select **Data Streams** from the left navigation menu. 4. Select **Create data stream**. 5. Enter a stream name (e.g., my-kinesis-stream). 6. Select the capacity mode: * **On-demand**: Automatically scales based on throughput * **Provisioned**: Specify the number of shards 7. Select **Create data stream**. Note the following information for later use: * Stream name (e.g., `my-kinesis-stream`) * AWS Region (e.g., `us-east-1`) * Number of shards (for optimal pipeline performance) ## Generate AWS Credentials To access your Kinesis Data Stream, you need AWS credentials with appropriate permissions. ## Required IAM Permissions The following minimum permissions are required: * kinesis:GetRecords * kinesis:GetShardIterator * kinesis:DescribeStream * kinesis:ListShards ## Create an IAM Policy 1. In the AWS Management Console, select **IAM** from the list of services. 2. Under **Access Management**, select **Policies**, and then select **Create policy**. 3. Select the **JSON** tab and enter the following policy (replace `` with your stream name): ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "KinesisReadAccess", "Effect": "Allow", "Action": [ "kinesis:GetRecords", "kinesis:GetShardIterator", "kinesis:DescribeStream", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:*:*:stream/" } ] } ``` 4. Select **Next** and enter a policy name (e.g., `SingleStoreKinesisReadPolicy`). 5. Select **Create policy**. ## Assign the IAM Policy to a User 1. In the **IAM** service, select **Users** and then select **Add users**. 2. Enter a name for the new user and select **Next**. 3. Select **Attach policies directly**. 4. Search for the policy you created and select the checkbox next to it. 5. Select **Next** and then select **Create user**. ## Create Access Keys 1. In the **IAM** service, select **Users** and select the user name you created. 2. Select the **Security credentials** tab. 3. In the **Access keys** section, select **Create access key**. 4. Select **Third-party service** and select **Next**. 5. (Optional but recommended) Add a description tag. 6. Select **Create access key**. 7. Download the CSV file or copy the credentials. You will need: * Access key ID * Secret access key > **📝 Note**: If you do not download or copy the credentials before selecting **Done**, the secret key cannot be retrieved and will need to be recreated. ## Part 3: Create a SingleStore Database and Kinesis Pipeline Now that you have a Kinesis Data Stream configured, you can create a SingleStore database and pipeline to ingest the streaming data. ## Create the Database Create a new database to hold your data: ```sql CREATE DATABASE kinesis_data; USE kinesis_data; ``` ## Deploy the Kafka Connect Connector To deploy and configure custom Kafka Connect connectors, contact [SingleStore Support](http://support.singlestore.com) with connector requirements. ## Create the Kinesis Pipeline Use the following information to create your pipeline: * **Stream name**: `my-kinesis-stream` * **AWS Region**: `us-east-1` * **Access Key ID**: `` * **Secret Access Key**: `` * **Number of shards**: (match your Kinesis stream configuration) Run the following command by replacing the placeholder values with your own: ```sql CREATE INFERRED PIPELINE kinesis_pipeline AS LOAD DATA KAFKACONNECT 'kafka-connector' CONFIG '{ "connector.class": "com.github.jcustenborder.kafka.connect.kinesis.KinesisSourceConnector", "aws.access.key.id": "", "aws.secret.key.id": "", "kafka.topic": "kinesis-topic", "kinesis.stream": "my-kinesis-stream", "kinesis.region": "us-east-1", "tasks.max": 4 }' CREDENTIALS '{}' FORMAT AVRO; ``` Important configuration notes: * `connector.class`: Fully-qualified Java class name of the Kafka Connect source connector * `tasks.max`: Set this equal to the number of shards in your Kinesis stream for optimal performance. The default value is `4`. * `kafka.topic`: A logical identifier for the data source (does not require an actual Kafka topic) * `kinesis.region`: AWS region where your Kinesis stream is located * Credentials: AWS credentials must be placed in the `CONFIG` parameter for Kinesis. The `CREDENTIALS` parameter can remain empty ## Static Schema Table When an inferred Kafka Connect Pipeline is created, SingleStore automatically creates a table with a predefined structure: ```sql CREATE TABLE `kinesis_pipeline` ( `topic` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL, `id` JSON COLLATE utf8mb4_bin NOT NULL, `record` JSON COLLATE utf8mb4_bin NOT NULL, SORT KEY `__UNORDERED` (), SHARD KEY () ) ``` The table contains three columns: * `topic`: Source identifier (`TEXT`) * `id`: Unique record identifier (`JSON`) * `record`: Complete record data (`JSON`) This static schema allows SingleStore to ingest data from various sources without requiring predefined table schemas. ## Start the Pipeline You can run the pipeline in the foreground or background. ## Start in the Foreground To test the pipeline and load existing data, run the following command: ```sql START PIPELINE kinesis_pipeline FOREGROUND; ``` This command runs synchronously and returns when all available records have been loaded. ## Start in the Background For continuous streaming, run the following command: ```sql START PIPELINE kinesis_pipeline; ``` This command runs the pipeline in the background, continuously polling Kinesis for new records. ## Verify Pipeline Status Check the pipeline status: ```sql SHOW PIPELINES; ``` ```output +---------------------------+---------+ | Pipelines_in_kinesis_data | State | +---------------------------+---------+ | kinesis_pipeline | Running | +---------------------------+---------+ ``` Run the following command to query detailed pipeline information: ```sql SELECT PIPELINE_NAME, STATE, CONFIG_JSON FROM information_schema.PIPELINES WHERE PIPELINE_NAME = 'kinesis_pipeline' ``` *** Modified at: July 6, 2026 Source: [/cloud/load-data/data-sources/load-data-from-amazon-kinesis-using-a-kafka-connect-pipeline/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-kinesis-using-a-kafka-connect-pipeline/) (An index of the documentation is available at /llms.txt) --- # Load Data from Amazon Web Services (AWS) S3 SingleStore Pipelines can extract objects from Amazon S3 buckets, optionally transform them, and insert them into a destination table. To understand Amazon S3’s core concepts and the terminology used in this topic, please read the [Amazon S3 documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html). ## Prerequisites To complete this Quickstart, your environment must meet the following prerequisites: * **AWS Account**: This Quickstart uses Amazon S3 and requires an AWS account’s *access key id* and *secret access key*. * **SingleStore Helios installation –or– a SingleStore Helios cluster**: You will connect to the database or cluster and create a pipeline to pull data from your Amazon S3 bucket. ## Part 1: Creating an Amazon S3 Bucket and Adding a File 1. On your local machine, create a text file with the following CSV contents and name it `books.txt`: ``` The Catcher in the Rye, J.D. Salinger, 1945 Pride and Prejudice, Jane Austen, 1813 Of Mice and Men, John Steinbeck, 1937 Frankenstein, Mary Shelley, 1818 ``` 2. In S3 create a bucket and upload `books.txt` to the bucket. For information on working with S3, refer to the [Amazon S3 documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html). Note that the `aws_access_key_id` that your SingleStore pipeline will use (specified in the next section in `CREATE PIPELINE library ... CREDENTIALS ...`) must have read access to both the bucket and the file. Once the `books.txt` file has been uploaded, you can proceed to the next part of the Quickstart. ## Part 2: Generating AWS Credentials To be able to use an S3 bucket within the pipeline syntax, the following minimum permissions are required: * s3:GetObject * s3:ListBucket These permissions only provide for read access from an S3 bucket which is the minimum required to ingest data into a pipeline. There are two ways to create an IAM Policy: with the Visual editor or JSON. Both creation methods require obtaining the Amazon Resource Number (ARN) before the policy is created. ## Create an IAM Policy Using the Visual Editor 1. Log into the AWS Management Console. 2. Obtain the Amazon Resource Number (ARN) and region for the bucket. The ARN and region are located in the **Properties** tab of the bucket. 3. Select **IAM** from the list of services. 4. Select **Policies** under Access Management and select the **Create policy** button. 5. Using the Visual editor: 1. Select the **Service** link and select **S3** from the list or manually enter S3 into the search block. 2. Select the **S3** link from the available selections. 3. In the Action section, select the **List** and **Read** checkboxes. 4. Under Resources, select the **bucket** link and select the **Add ARN** link. Enter the ARN and bucket name and select the **Add** button. 5. Under Resources, select the **object** link and select the **Add ARN** link. Enter the ARN and object name and select the **Add** button. If no objects are added under resources, the created policy has access to all objects in the bucket’s root path. 6. Request conditions are optional. ## Create an IAM Policy Using JSON 1. To use JSON for policy creation, copy the information from following the code block into the AWS JSON tab. Make sure to change the bucket name. ```json { "Version": "2012-10-17", "Statement": [{ "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::", "arn:aws:s3:::/*" ] }] } ``` 2. Select the **Add tag** button if needed and select **Next: Review**. 3. Enter a policy name this is a required field. The description field is optional. Select **Create policy** to finish. ## Assign the IAM Policy to a New User 1. In the IAM services, select **Users** and select the **Add users** button. 2. Enter in a name for the new user and select **Next**. 3. Select the **Attach policies directly** radio button. Use the search box to find the policy or scroll through the list of available policies. 4. Select the checkbox next to the policy to be applied to the user and select **Next**. 5. Select the **Create user** button to finish. ## Create Access Keys for Pipeline Syntax Access keys must be generated for the new user. 1. In the IAM services, select **Users** and select the user name. 2. Select the **Security credentials** tab. 3. In the access keys section, select the **Create access key** button. 4. Select the **Third-party service** radio button and select **Next**. 5. Although setting a description tag is optional, SingleStore recommends doing so, especially when multiple keys are needed. Select the **Create key** button to continue. 6. Either download a `.csv` file containing the access and secret key information or copy the credentials directly. Select **Done** when finished. 7. Following is the basic syntax for using an access key and a secret access key in a pipeline: ```sql CREATE PIPELINE AS LOAD DATA S3 's3://bucket_name/' CONFIG '{"region":"us-west-2"}' CREDENTIALS '{"aws_access_key_id": "", "aws_secret_access_key": ""}' INTO TABLE FIELDS TERMINATED BY ','; ``` If creating or starting S3 pipelines takes approximately `60` seconds, or fails with a subprocess timeout when running outside AWS, or in environments where IMDS is blocked, reduce the value of the `subprocess_ec2_metadata_timeout_ms` engine variable (for example, to `1000`) or explicitly provide `CREDENTIALS`. New clusters already set this value to `1` (millisecond) to avoid delays. Refer to [Sync Variables Lists](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) for more information. > **⚠️ Warning**: If the key information is not downloaded or copied to a secure location before selecting **Done**, the secret key *cannot* be retrieved, and will need to be recreated. ## Part 3: Creating a SingleStore Database and S3 Pipeline Now that you have an S3 bucket that contains an object (file), you can use SingleStore Helios or DB to create a new pipeline and ingest the messages. Create a new database and a table that adheres to the schema contained in the `books.txt` file. At the prompt, execute the following statements: ```sql CREATE DATABASE books; ``` ```sql CREATE TABLE classic_books ( title VARCHAR(255), author VARCHAR(255), date VARCHAR(255) ); ``` These statements create a new database named `books` and a new table named `classic_books`, which has three columns: `title`, `author`, and `date`. Now that the destination database and table have been created, you can create an S3 pipeline. In Part 1 of this Quickstart, you uploaded the `books.txt` file to your bucket. To create the pipeline, you will need the following information: * The name of the bucket, such as: `` * The name of the bucket’s region, such as: `us-west-1` * Your AWS account’s access keys, such as: * *Access Key ID*: `` * *Secret Access Key*: `` * Your AWS account's session token, such as: * *Session Token*: `your_session_token` * Note that the `aws_session_token` is required only if your credentials in the `CREDENTIALS` clause are temporary Using these identifiers and keys, execute the following statement, replacing the placeholder values with your own. ```sql CREATE PIPELINE library AS LOAD DATA S3 'my-bucket-name' CONFIG '{"region": "us-west-1"}' CREDENTIALS '{"aws_access_key_id": "your_access_key_id", "aws_secret_access_key": "your_secret_access_key", "aws_session_token": "your_session_token"}' INTO TABLE `classic_books` FIELDS TERMINATED BY ','; ``` You can see what files the pipeline wants to load by running the following: ```sql SELECT * FROM information_schema.PIPELINES_FILES; ``` If everything is properly configured, you should see one row in the `Unloaded` state, corresponding to `books.txt`. The `CREATE PIPELINE` statement creates a new pipeline named `library`, but the pipeline has not yet been started, and no data has been loaded. A SingleStore pipeline can run either in the background or be triggered by a foreground query. Start it in the foreground first. ```sql START PIPELINE library FOREGROUND; ``` When this command returns success, all files from your bucket will be loaded. If you check `information_schema.PIPELINES_FILES` again, you should see all files in the `Loaded` state. Now query the `classic_books` table to make sure the data has actually loaded. ```sql SELECT * FROM classic_books; ``` ```output +------------------------+-----------------+-------+ | title | author | date | +------------------------+-----------------+-------+ | The Catcher in the Rye | J.D. Salinger | 1945 | | Pride and Prejudice | Jane Austen | 1813 | | Of Mice and Men | John Steinbeck | 1937 | | Frankenstein | Mary Shelley | 1818 | +------------------------+-----------------+-------+ ``` You can also have SingleStore run your pipeline in the background. In such a configuration, SingleStore will periodically poll S3 for new files and continuously them as they are added to the bucket. Before running your pipeline in the background, you must reset the state of the pipeline and the table. ```sql DELETE FROM classic_books; ALTER PIPELINE library SET OFFSETS EARLIEST; ``` The first command deletes all rows from the target table. The second causes the pipeline to start from the beginning, in this case, *forgetting* it already loaded `books.txt` so you can load it again. You can also drop and recreate the pipeline, if you prefer. To start a pipeline in the background, run ```sql START PIPELINE library; ``` This statement starts the pipeline. To see whether the pipeline is running, run `SHOW PIPELINES`. ```sql SHOW PIPELINES; ``` ```output +----------------------+---------+ | Pipelines_in_books | State | +----------------------+---------+ | library | Running | +----------------------+---------+ ``` At this point, the pipeline is running and the contents of the `books.txt` file should once again be present in the `classic_books` table. > **📝 Note**: Foreground pipelines and background pipelines have different intended uses and behave differently. For more information, see the [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) topic. ## Use Cloud Workload Identity with S3 Pipelines You can use [Cloud Workload Identity](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities.md) instead of static credentials to load data via S3 pipelines. Perform the following tasks to create an S3 pipeline that authenticates using the cloud workload identity: 1. [Configure delegated entities](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities/#section-id235390403966648.md). 1. **Create an IAM role** in your AWS account with the necessary privileges. You can also use an existing IAM role. 2. **Update the IAM role's trust policy** to allow the cluster's cloud workload identity to assume the role. Specify the [cloud workload identity ARN of the SingleStore cluster](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities/#section-id23539041172686.md). Alternatively, [create a CloudFormation stack](https://docs.singlestore.com/#section-id235391786003154.md) to configure the IAM roles. 2. Create an S3 pipeline. In the pipeline configuration: 1. Set `creds_mode` to `eks_irsa` in the `CONFIG` clause. 2. Specify the IAM role to assume using `role_arn` in the `CREDENTIALS` clause. The specified role ARN must match the configured delegated entities for the SingleStore cluster. For example: ```sql CREATE PIPELINE s3_pipeline AS LOAD DATA S3 's3://bucket-name/path/' CONFIG '{ "region": "us-east-1", "creds_mode": "eks_irsa" }' CREDENTIALS '{ "role_arn": "arn:aws:iam::xxxxxxxx:role/singlestore-s3-pipeline" }' INTO TABLE table_name FIELDS TERMINATED BY ','; ``` 3. Start the pipeline to ingest data. ```sql START PIPELINE s3_pipeline; ``` If delegated entities are not configured, SingleStore pipelines that attempt to use IRSA with a role ARN not present in the delegated entities list fail at runtime. ## Use CloudFormation Stack to Configure the IAM Role Use [this CloudFormation Stack template](https://singlestore-public-resources-production.s3.us-east-1.amazonaws.com/cf-templates/s3-access-role/template.yaml) to define the IAM role. Configure the following input parameters for the stack: | Parameter | Description | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RoleName` | Name of the IAM role that theSingleStoreclusterassumes to access the S3 buckets.Default:`SingleStoreS3AccessRole-01` | | `WorkspaceGroupCloudWorkloadIdentities` | Comma-separated list ofSingleStorecluster's cloud workload identity ARNs that can assume this IAM role.**Note**: Do not include a trailing comma (at the end of the list). | | `BucketNames` | Comma separated list of S3 buckets that can be accessed by assuming this role. | Refer to [Getting started with CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/GettingStarted.html) for more information. ## Next Steps See [About SingleStore Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) to learn more about how pipelines work. ## In this section * [Scanning and Loading Files in AWS S3](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/scanning-and-loading-files-in-aws-s-3.md) * [Load Data Using pipeline\_source\_file()](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/load-data-using-pipeline-source-file.md) * [Connect to AWS S3 Bucket from SingleStore](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/connect-to-aws-s-3-bucket-from-singlestore.md) * [Load Data in CSV Format from Amazon S3 Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/load-data-in-csv-format-from-amazon-s-3-using-a-pipeline.md) * [Load Data in JSON Format from Amazon S3 Using a Wildcard](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/load-data-in-json-format-from-amazon-s-3-using-a-wildcard.md) * [Configure Kinesis Event Notifications for S3 Pipelines](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/configure-kinesis-event-notifications-for-s-3-pipelines.md) * [Enable EKS IRSA](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/enable-eks-irsa.md) * [S3 Pipeline Errors](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/s-3-pipeline-errors.md) *** Modified at: July 6, 2026 Source: [/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/) (An index of the documentation is available at /llms.txt) --- # Load Data from Azure Blob Storage Using a Pipeline ## Prerequisites To complete this Quickstart, your environment must meet the following prerequisites: * **Azure Account**: This Quickstart uses Azure Blob Store. * **SingleStore Helios installation –or– a SingleStore Helios cluster**: You will connect to the database or cluster and create a pipeline to pull data from your Azure Blob Store. ## Part 1: Creating an Azure Blob Container and Adding a File 1. On your local machine, create a text file with the following CSV contents and name it *books.txt*: ``` The Catcher in the Rye, J.D. Salinger, 1945 Pride and Prejudice, Jane Austen, 1813 Of Mice and Men, John Steinbeck, 1937 Frankenstein, Mary Shelley, 1818 ``` 2. In Azure, create a container and upload `books.txt` to the container. For information on working with Azure, see the [Azure Docs](https://docs.microsoft.com/en-us/azure/). Once the *books.txt* file has been uploaded, you can proceed to the next part of the Quickstart. ## Part 2: Creating a SingleStore Database and Azure Blob Pipeline Now that you have an Azure container that contains an object (file), you can use SingleStore Helios or DB to create a new pipeline and ingest the blobs. We will create a new database and a table that adheres to the schema contained in **books.txt** file. At the MemSQL prompt, execute the following statements: ```sql CREATE DATABASE books; ``` ```sql CREATE TABLE classic_books ( title VARCHAR(255), author VARCHAR(255), date VARCHAR(255) ); ``` These statements create a new database named `books` and a new table named `classic_books`, which has three columns: `title`, `author`, and `date`. Now that the destination database and table have been created, you can create an Azure pipeline. In Part 1 of this Quickstart, you uploaded the **books.txt** file to your container. To create the pipeline, you will need the following information: * The name of the container, such as: `my-container-name` * Your Azure Storage account’s name and key, such as: * *Account Name*: `your_account_name` * *Account Key*: `your_account_key` Using these identifiers and keys, execute the following statement, replacing the placeholder values with your own: ```sql CREATE PIPELINE library AS LOAD DATA AZURE 'my-container-name' CREDENTIALS '{"account_name": "your_account_name", "account_key": "your_account_key"}' INTO TABLE `classic_books` FIELDS TERMINATED BY ','; ``` You can see what files the pipeline wants to load by running the following: ```sql SELECT * FROM information_schema.PIPELINES_FILES; ``` If everything is properly configured, you should see one row in the `Unloaded` state, corresponding to `books.txt`. The `CREATE PIPELINE` statement creates a new pipeline named `library`, but the pipeline has not yet been started, and no data has been loaded. A SingleStore pipeline can run either in the background or be triggered by a foreground query. Start it in the foreground first. ```sql START PIPELINE library FOREGROUND; ``` When this command returns successfully, all files from your bucket will be loaded. If you check `information_schema.PIPELINES_FILES` again, you should see all files in the `Loaded` state. Now query the `classic_books` table to make sure the data has actually loaded. ```sql SELECT * FROM classic_books; ``` ```output +------------------------+-----------------+-------+ | title | author | date | +------------------------+-----------------+-------+ | The Catcher in the Rye | J.D. Salinger | 1945 | | Pride and Prejudice | Jane Austen | 1813 | | Of Mice and Men | John Steinbeck | 1937 | | Frankenstein | Mary Shelley | 1818 | +------------------------+-----------------+-------+ ``` You can also have SingleStore run your pipeline in background. In such a configuration, SingleStore will periodically poll Azure Blob Storage for new files and continuously load them as they are added to the storage container. Before running your pipeline in the background, you must reset the state of the pipeline and the table. ```sql DELETE FROM classic_books; ALTER PIPELINE library SET OFFSETS EARLIEST; ``` The first command deletes all rows from the target table. The second causes the pipeline to start from the beginning, in this case, *forgetting* it already loaded `books.txt` so you can load it again. You can also drop and recreate the pipeline, if you prefer. To start a pipeline in the background, run `START PIPELINE`. ```sql START PIPELINE library; ``` This statement starts the pipeline. To see whether the pipeline is running, run `SHOW PIPELINES`. ```sql SHOW PIPELINES; ``` ```output +----------------------+---------+ | Pipelines_in_books | State | +----------------------+---------+ | library | Running | +----------------------+---------+ ``` At this point, the pipeline is running and the contents of the **books.txt** file should once again be present in the `classic_books` table. > **📝 Note**: Foreground pipelines and background pipelines have different intended uses and behave differently. For more information, see [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md). ## Next Steps See [About SingleStore Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) to learn more about how pipelines work. ## In this section * [Connect to Azure Blob Storage from SingleStore](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline/connect-to-azure-blob-storage-from-singlestore.md) *** Modified at: September 9, 2024 Source: [/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-azure-blob-storage-using-a-pipeline/) (An index of the documentation is available at /llms.txt) --- # Load Data from Google Cloud Storage (GCS) Using a Pipeline SingleStore pipelines can extract objects from Google Cloud Storage, optionally transform them, and insert them into a destination table. To understand GCS core concepts and the terminology used in this topic, please read the [Google Cloud Storage Documentation](https://cloud.google.com/storage). ## Prerequisites The following prerequisites are the needed to create a GCS pipeline. * **GCS Account**: Requires a Google `access_id` and `secret_key`. * **SingleStore Helios installation –or– a SingleStore Helioscluster**: You will connect to the database or cluster and create a pipeline to pull data from your GCS bucket. ## Part 1: Creating a GCS Bucket and Adding a File 1. On your local machine, create a text file with the following CSV contents and name it *books.txt*: ``` The Catcher in the Rye, J.D. Salinger, 1945 Pride and Prejudice, Jane Austen, 1813 Of Mice and Men, John Steinbeck, 1937 Frankenstein, Mary Shelley, 1818 ``` 2. In GCS, create a bucket and upload `books.txt` to the bucket. Also, create an HMAC key for authentication to the bucket, as SingleStore pipelines only support that type of authentication to GCS. For information on working with GCS, refer to the [Google Cloud Storage Documentation](https://cloud.google.com/storage/docs). ## Part 2: Creating a SingleStore Helios Database and GCS Pipeline Now that you have a GCS bucket that contains an object (file), you can use SingleStore Helios to create a new pipeline and ingest the messages. In this part of the Quickstart, you will create new GCS pipeline and use it to load your CSV data into the database. Create a new database and a table that adheres to the schema contained in the *books.txt* file. At the MemSQL prompt, execute the following statements: ```sql CREATE DATABASE books; ``` ```sql CREATE TABLE classic_books ( title VARCHAR(255), author VARCHAR(255), date VARCHAR(255) ); ``` These statements create a new database named `books` and a new table named `classic_books`, which has three columns: `title`, `author`, and `date`. Now that the destination database and table have been created, you can create a GCS pipeline. Use the *books.txt* file previously uploaded to your bucket. To create the pipeline, you will need the following information: * The name of the bucket, such as: my-bucket-name * Your Google account’s access HMAC keys, such as: Access Key ID: `your_access_key_id` Secret Access Key: `your_secret_access_key`. Using these identifiers and keys, execute the following statement, replacing the placeholder values with your own: ```sql CREATE PIPELINE library AS LOAD DATA GCS 'my-bucket-name' CREDENTIALS '{"access_id": "your_access_key_id", "secret_key": "your_secret_access_key"}' INTO TABLE `classic_books` FIELDS TERMINATED BY ','; ``` You can see what files the pipeline wants to load by running the following: ```sql SELECT * FROM information_schema.PIPELINES_FILES; ``` If everything is properly configured, you should see one row in the `Unloaded` state, corresponding to `books.txt`. The `CREATE PIPELINE` statement creates a new pipeline named `library`, but the pipeline has not yet been started, and no data has been loaded. A pipeline can run either in the background or be triggered by a foreground query. Start it in the foreground first. ```sql START PIPELINE library FOREGROUND; ``` When this command returns success, all files from your bucket will be loaded. If you check `information_schema.PIPELINES_FILES` again, you should see all files in the `Loaded` state. Now query the `classic_books` table to make sure the data has actually loaded. ```sql SELECT * FROM classic_books; ``` ```output +------------------------+-----------------+-------+ | title | author | date | +------------------------+-----------------+-------+ | The Catcher in the Rye | J.D. Salinger | 1945 | | Pride and Prejudice | Jane Austen | 1813 | | Of Mice and Men | John Steinbeck | 1937 | | Frankenstein | Mary Shelley | 1818 | +------------------------+-----------------+-------+ ``` You can also have SingleStore Helios run your pipeline in the background. In such a configuration, SingleStore Helios will periodically poll GCS for new files and continuously them as they are added to the bucket. Before running your pipeline in the background, you must reset the state of the pipeline and the table. ```sql DELETE FROM classic_books; ALTER PIPELINE library SET OFFSETS EARLIEST; ``` The first command deletes all rows from the target table. The second causes the pipeline to start from the beginning, in this case, *forgetting* it already loaded `books.txt` so you can load it again. You can also drop and recreate the pipeline, if you prefer. To start a pipeline in the background, run ```sql START PIPELINE library; ``` This statement starts the pipeline. To see whether the pipeline is running, run `SHOW PIPELINES`. ```sql SHOW PIPELINES; ``` ```output +----------------------+---------+ | Pipelines_in_books | State | +----------------------+---------+ | library | Running | +----------------------+---------+ ``` At this point, the pipeline is running and the contents of the *books.txt* file should once again be present in the `classic_books` table. > **📝 Note**: Foreground pipelines and background pipelines have different intended uses and behave differently. For more information, see the [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) topic. ## Related Topics See [About SingleStore Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) to learn more about how pipelines work. *** Modified at: April 27, 2026 Source: [/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-google-cloud-storage-gcs-using-a-pipeline/) (An index of the documentation is available at /llms.txt) --- # Load Data from HDFS Using a Pipeline When you use HDFS (A distributed file system by Apache Hadoop. It is highly fault tolerant and designed to run on COTS (Commercial Off the Shelf) or low cost out-of-the-box hardware.) pipelines, you extract data from an HDFS file path, optionally transform the data, and load it to a SingleStore Helios table. This topic assumes that you have set up HDFS in your cluster and you are familiar with [how HDFS works](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html). > **📝 Note**: HDFS Pipelines cannot run Hadoop jobs. ## Creating and Starting an HDFS Pipeline You create an HDFS pipeline by running the [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) statement. You start a pipeline by running [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) ## HDFS Pipeline Scenario Imagine that your organization has numerous applications running on-premises. These applications generate lengthy log files that contain possible errors the applications generate as they run. Your goal: Get a running weekly and monthly count of the number of errors per application. To accomplish this, you follow these steps sequentially. The steps are explained in detail in the sections below. 1. Copy the first day’s application log files into HDFS. 2. Run a Hadoop job that processes the log files and outputs the results to one output file. 3. Use an HDFS pipeline to extract the results and import them into a SingleStore Helios table. 4. Confirm that the SingleStore Helios table contains the data for the first day. If the data is correct, run steps one through three continuously, at the end of every day. > **📝 Note**: This scenario provides a hypothetical example of how you can use HDFS pipelines. It is a not a working example, as the application logs and the Hadoop job are not available. ## Log File Format Shown below is an example application log file containing selected errors from one day. The `...` parts indicate lengthy sections containing errors that are removed for brevity. ``` ... [2019-03-01 09:30:08]: ERR-2030: File not found: file1.txt ... [2019-03-01 12:15:35]: ERR-1010: Could not read configuration file conf_file.txt ... [2019-03-01 14:00:10]: ERR-1520: Not enough memory available to open file file2.txt ... [2019-03-01 16:40:35]: ERR-1010: Could not read configuration file conf_file10.txt ... [2019-03-01 19:20:55]: ERR-1520: Not enough memory available to open file file15.txt ... ``` Your other applications generate log files that are formatted in a similar way. The errors contain the date and time of the error, the error code and the error message. ## HDFS folder Setup The HDFS folder `/user/memsql/` stores the input files, per application, that are processed by your Hadoop job. After your job runs, it outputs the resulting `part-00000` file to the HDFS folder `/user/memsql/output`. The `part-00000` filename has the current date appended to the end. ## Running the Hadoop Job Run the Hadoop job which extracts the error date and error code for each log entry. The job will write these fields, along with the application name, to a line in the output file. An example output file is shown below. ``` App1, ERR-2030, 2019-03-01 App1, ERR-1010, 2019-03-01 App1, ERR-1520, 2019-03-01 App1, ERR-1010, 2019-03-01 App1, ERR-1520, 2019-03-01 App2, E-400, 2019-03-01 App2, E-250, 2019-03-01 App2, E-800, 2019-03-01 App2, E-400, 2019-03-01 ``` ## Creating the Destination Table Create the table where the data will be loaded from the pipeline. ```sql CREATE TABLE app_errors (app_name TEXT, error_code TEXT, error_date DATE, SORT KEY (error_date)); ``` The reason you use a columnstore table to store the application errors is because a columnstore is well suited for performing aggregate queries. Also, you will not be updating rows in the table once they are imported. ## Creating Your Pipeline Use the following statement to create a new pipeline named `my_pipeline`, where you reference the HDFS path `/memsql/output/` as the data source, and import data into the `app_errors` table once the pipeline is started. ```sql CREATE PIPELINE my_pipeline AS LOAD DATA HDFS 'hdfs://hadoop-namenode:8020/memsql/output/' INTO TABLE app_errors FIELDS TERMINATED BY ', '; ``` Should your `CREATE PIPELINE` statement fail, run `SHOW WARNINGS`. This command may provide information as to why the `CREATE PIPELINE` statement failed. ## Testing Your Pipeline After you run your `CREATE PIPELINE` statement successfully, run the tests in this section to confirm your pipeline is working. The following query should return one row for each file that the pipeline has imported. In this case, the query will only return one row, since the Hadoop job generates one output file. For example, if you ran the job on January 1, 2019, `FILE_NAME` could have the value `part-00000-20190101`. The `FILE_STATE` field value should have the value `Unloaded`. ```sql SELECT * FROM information_schema.pipelines_files WHERE pipeline_name = 'my_pipeline'; ``` Run the following command to test if the master aggregator can connect to the HDFS namenode. If the test is successful, zero rows will be returned. ```sql TEST PIPELINE my_pipeline LIMIT 0; ``` Run the following command to test if the SingleStore Helios leaf nodes can connect to the HDFS datanodes. If the test is successful, one row will be returned. ```sql TEST PIPELINE my_pipeline LIMIT 1; ``` ## Starting Your Pipeline Assuming the tests you ran in the previous section succeeded, start your pipeline in the foreground: ```sql START PIPELINE my_pipeline FOREGROUND LIMIT 1 BATCHES; ``` Starting your pipeline in the foreground allows you to see any errors, if they occur. After your pipelines ingests one batch of data into the `app_errors` table, your pipeline stops. Run the following query, which should return one row with the `part-00000-20190101` file. In this row, the `FILE_STATE` field should have the value `Loaded`, assuming no errors occurred when you started the pipeline. ```sql SELECT * FROM information_schema.pipelines_files WHERE pipeline_name = 'my_pipeline'; ``` Run `SELECT * FROM app_errors;` to view the records in the `app_errors` table. If your HDFS file `/user/memsql/output/part-00000-20190101` contains the output as shown in the [Running the Hadoop Job](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline/#UUID-5d36d75c-39cb-cb2d-2646-c6dfcce4dcfb.md) above, you will see nine records, each with an `app_name`, `error_code` and `error_date` field. ## Syncing HDFS with Your Application Logs Now that your Hadoop job generates an output file successfully and your pipeline imports the file successfully, you want to periodically copy your application log files to the `/user/memsql/` HDFS folders. To accomplish this, you write a script to automatically copy the files at the end of every day. After you start the script, start your pipeline in the background to continuously ingest data into the `app_errors` table. ```sql START PIPELINE my_pipeline; ``` > **📝 Note**: Foreground pipelines and background pipelines have different intended uses and behave differently. For more information, see [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md). ## Finding the Weekly and Monthly Error Count Recall that your goal is to find a running weekly and monthly count of the number of errors each application generates. Prior to creating a weekly and monthly query that returns these counts, you write a simpler query that returns the number of errors per application and error code, for all records in the table. ```sql SELECT COUNT(*), app_name, error_code FROM app_errors GROUP BY app_name, error_code ``` Using the previous query as a starting point, you write the monthly query: ```sql SELECT COUNT(*), app_name, error_code, MONTH(error_date) FROM app_errors GROUP BY app_name, error_code, MONTH(error_date) ``` Finally, you write the weekly query, using `WEEK(error_date, 2)` to specify that the week begins on Sunday: ```sql SELECT COUNT(*), app_name, error_code, WEEK(error_date, 2) FROM app_errors GROUP BY app_name, error_code, WEEK(error_date, 2) ``` ## Next Steps See [About SingleStore Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) to learn more about how pipelines work. ## In this section * [Enabling Wire Encryption and Kerberos on HDFS Pipelines](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline/enabling-wire-encryption-and-kerberos-on-hdfs-pipelines.md) *** Modified at: July 24, 2025 Source: [/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-hdfs-using-a-pipeline/) (An index of the documentation is available at /llms.txt) --- # Load Data from Kafka ## Securely Connect to Kafka from SingleStore Helios ## Overview When running a `CREATE PIPELINE ... KAFKA ... `statement, you may need to make a secure connection to Kafka. Use Secure Socket Layer (SSL) for the connection and Simple Authentication and Security Layer (SASL) to authenticate. Using SASL for authentication is optional. * GSSAPI (Kerberos) * PLAIN * SCRAM-SHA-256 * SCRAM-SHA-512 * OAUTHBEARER SASL This topic assumes SSL have been set up, configured, and enabled on the Kafka brokers. For information on how to enable this functionality, see the [SSL and SASL sections in the Kafka documentation](http://kafka.apache.org/documentation/#security_ssl). ## Convert Java Keystore (JKS) to Privacy Enhanced Mail (PEM) Key To use SSL encryption for SingleStore pipelines, JKS keys need to be converted to PEM keys. > **📝 Note**: In the steps below, the **< >** symbols indicate a variable and any information provided between these symbols is an example. 1. Create the key and keystore. You will be prompted to enter details such as name, organizational unit, city, state, etc. ```shell keytool -genkey -keyalg RSA -keystore .jks -storepass -alias "" ``` ```output What is your first and last name? [Unknown]:- What is the name of your organizational unit? [Unknown]:- What is the name of your organization? [Unknown]:- ``` 2. Export the client certificate from the keystore using the same password as in step 1. ```shell keytool -exportcert -rfc -file .pem -alias -keystore .jks ``` ```output Enter keystore password: Certificate stored in file .pem ``` 3. Import the client certificate to the truststore located on your Apache Server. Enter a new password for the keystore. ```shell keytool -keystore .jks -alias -import -file .pem ``` ```output Enter keystore password: Re-enter new password: Trust this certificate? [no]: yes Certificate was added to keystore ``` 4. Convert the client keystore to Public-Key Cryptography Standards (PKCS12) format. The \.jks and password are the same as in step 1. ```shell keytool -v -importkeystore -srckeystore .jks -srcalias -destkeystore .p12 -deststoretype PKCS12 ``` ```output Importing keystore .jks to .p12... Enter destination keystore password: Re-enter new password: Enter source keystore password: [Storing new-keystore-name.p12] ``` 5. Extract the client certificate key into a .pem file. Use the import password created in step 4. ```shell openssl pkcs12 -in .p12 -nocerts -nodes > .pem ``` ```output Enter Import Password: ``` ## Steps for Creating a Secure Connection To create a secure connection from SingleStore Helios to a Kafka cluster, follow these steps in order. ## Upload a Certificate to Use to Connect via TLS/SSL Use the following steps to enable TLS/SSL encryption between SingleStore Helios and Kafka. 1. On the [Cloud Portal](https://portal.singlestore.com), select **Clusters**. 2. Select the three dots under the **Actions** column for your cluster on which to enable TLS/SSL connections, and then select **Access & Security** from the list. 3. On the **Security** tab, in the **Secure Connections** section, select **Upload Certificate** to upload your CA certificate. This will make it available to all the nodes and will allow you to secure outbound connections via TLS/SSL. ## Build a String with the Connection Settings Using the following settings, create a string containing the `CONFIG` clause and optionally the `CREDENTIALS` clause of the `CREATE PIPELINE ... KAFKA ...` or `SELECT ... INTO KAFKA ...` statement that you will be running. ## SSL Connection Settings 1. In your `CONFIG` JSON, if you want to enable SSL encryption only, set `"security.protocol": "ssl"`. If you want to enable Kerberos with SSL, or otherwise want to use SASL, set `"security.protocol": "sasl_ssl"`. 2. Set the remaining SSL configuration in the `CONFIG` JSON: * `ssl.ca.location`. Value is always `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem`. 3. If your SSL certificate key is using a password, set it in your `CREDENTIALS` JSON. * `ssl.key.password`: Password for the SSL certificate key. ## SASL Connection Settings 1. In your `CONFIG` JSON, set `"security.protocol": "sasl_ssl"` for SSL connections, or `"security.protocol": "sasl_plaintext"` if you want to authenticate with Kafka without SSL encryption. 2. If your Kafka brokers do not use [SCRAM](https://en.wikipedia.org/wiki/Salted_Challenge_Response_Authentication_Mechanism) for authentication, set `"sasl.mechanism": "PLAIN"` in your `CONFIG` JSON. Otherwise, set `"sasl.mechanism": "SCRAM-SHA-256"` or `"sasl.mechanism": "SCRAM-SHA-512"`. 3. In your `CONFIG` JSON, provide the username, `"sasl.username": ""`. 4. In your `CREDENTIALS` JSON, provide the password, `"sasl.password": ""`. > **📝 Note**: `SASL_PLAINTEXT/PLAIN` authentication mode with Kafka sends your credentials unencrypted over the network. It is therefore not secure and susceptible to being sniffed.`SASL_PLAINTEXT/SCRAM` authentication mode with Kafka will encrypt the credentials information sent over the network, but transport of Kafka messages themselves is not secure. ## Configuring OAUTHBEARER Authentication Mechanism OAUTHBEARER is used to secure access to resources on a server by requiring clients to obtain a bearer token. The client presents this token with each request to the server, and the server verifies the token before granting access to the requested resource. To use SASL OAUTHBEARER authentication with Kafka, the following information is required: * `"sasl.mechanism":"OAUTHBEARER"` - Specifies the client will authenticate using an OAuth 2.0 Bearer Token. * `"sasl.oauthbearer.client.id":""` - The client ID is usually provided by the OAuth provider when the client is registered. It is a unique ID that is associated with the OAuth 2.0 Bearer Token. * `"sasl.oauthbearer.client.secret":""` The client secret is usually assigned by the OAuth provider when a client is registered and is used to authenticate the client. * `"sasl.oauthbearer.token.endpoint.url":""` - This is the endpoint URL on the authorization server that is used to obtain an OAuth 2.0 Bearer Token. The client sends a request to this endpoint to get a token, which is then used to authenticate subsequent requests to the server. Optional configurations are: * `"sasl.oauthbearer.scope":""` - Determines the permissions and resources that are available to an authorized client. This extension is optional. * `"sasl.outhbearer.extensions":""` - Can be included in the SASL/OAUTHBEARER mechanism to provide additional data or parameters for authentication. Consult [RFC-7628](https://tools.ietf.org/html/rfc7628#section-3.1) for further information on SASL extensions. * See "OAUTHBEARER Pipelines Configuration Details" section for additional details regarding "sasl.oauthbearer.ssl.ca.location" "sasl.oauthbearer.config" fields. **Prerequisites** To use SASL OAUTHBEARER authentication the following prerequisites are required: * A Kafka broker with listeners configured using OAUTHBEARER authentication. * A connection to a database where Kafka brokers are reachable to create a pipeline to pull data from the Kafka queue. * An identity service was selected (e.g., Okta, Google OAuth, Facebook OAuth, Azure AD, Keycloak, etc.). * An Oauthbearer client was created and configured with the client\_credentials grant type on the identity service. > **📝 Note**: The instructions for setting up a client on the Identity Service will vary depending on the server chosen and the type of application being built. **Syntax for SASL OAUTHBEARER Pipeline** Below is the syntax for creating a Kafka pipeline using OAUTHBEARER authentication. ```sql CREATE or REPLACE PIPELINE AS LOAD DATA KAFKA "" CONFIG '{"security.protocol":"SASL_SSL", "sasl.mechanism":"OAUTHBEARER", "sasl.oauthbearer.client.id":"", "sasl.oauthbearer.client.secret":<"CLIENT_SECRET>", "sasl.oauthbearer.token.endpoint.url":"", "sasl.oauthbearer.scope":""}' INTO TABLE ; ``` **OAUTHBEARER Pipelines Configuration Details** To review the current pipeline configuration use the command: ```sql SELECT * FROM information_schema.pipelines ``` Most token providers are assumed to work out-of-the-box. However, the implementation details below will allow troubleshooting and configuring pipelines for specific cases. **Token Request Details:** * SingleStore implements` client_credentials` grant type for OAUTHBEARER token requests. Ensure the OAuth client is created with support for the `client_credentials` grant type. * By default, SingleStore treats Oauthbearer tokens as opaque to address[privacy concerns](https://datatracker.ietf.org/doc/html/rfc9068#name-privacy-considerations)and uses the[expires\_in](https://www.rfc-editor.org/rfc/rfc6749#section-4.2.2)field from the JSON token response to determine the token's validity time. If `expires_in` is not present or `"sasl.oauthbearer.config”:"use_expires_in=false”` is set in the pipeline configuration, we fall back to decoding the Oauth token JWT and use the [exp claim](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.4) to determine its validity. * The token refresh is scheduled at 80% of the token’s lifetime and handled by SingleStore in the background. * For https Oauth token requests, by default, SingleStore will use the CA bundle specified at pipeline creation or try to find one of the system paths if none are specified. This behavior can be changed by including additional configuration settings in the pipeline CONFIG on creation. This change will affect token request logic only, not the total SSL Kafka communication. * `"sasl.oauthbearer.ssl.ca.location":"system"` - to use the system default path; * `"sasl.oauthbearer.ssl.ca.location":"/usr/lib/ssl/certs/ca-certificates.crt"` - to use a specific CA path location; * `"sasl.oauthbearer.ssl.ca.location":""` (empty) to disable SSL verification; **Token Cache Details:** * The OAUTHBEARER tokens are cached in the extractor pools for some time (along with the extractors) and the pipeline cookies (per pipeline). The Oauth server must allow multiple simultaneous active tokens due to the several nodes or extractor pools used in the SingleStore Helios architecture. * `"sasl.oauthbearer.cookie.cache":"on"` - by default, tokens are cached in the pipeline cookies to minimize the number of token requests. To disable this feature set `"sasl.oauthbearer.cookie.cache":"off"`. To reset the pipeline cookies, use `alter pipeline … set offset cursor ''`. * The pipeline may fail because the OAUTHBEARER client config on the identity server is changed or the pipeline config is altered in a way which causes previously requested cached tokens to become invalid. This issue may be avoided by resetting the respective pipeline cache. * `alter pipeline … set offset cursor ''` - to reset cookies cache; * `flush extractor pools` - to reset extractor pools; ## Avro Schema Caching When using Kafka pipelines with Avro format and the Confluent Schema Registry, SingleStore can cache schemas across batch boundaries to improve performance and reduce load on the Schema Registry. By default, schemas are fetched from the Schema Registry for every batch. Enable schema caching using the `pipelines_avro_schema_cache_enabled` global variable: ```sql SET GLOBAL pipelines_avro_schema_cache_enabled = ON; ``` When schema caching is enabled: * Each pipeline maintains a schema cache for each partition. * Schemas are fetched from the Confluent Schema Registry at most once for each schema ID per partition while the pipeline remains active. * Subsequent batches reuse cached schema definitions and avoid redundant HTTP requests. The schema cache is cleared when: * `ALTER PIPELINE ... SET CONFIG` modifies the pipeline configuration. This includes changing the Confluent Schema Registry URL, as it is part of the pipeline `CONFIG`. * `ALTER PIPELINE ... SET CREDENTIALS` updates pipeline credentials. * `CREATE OR REPLACE PIPELINE` updates a pipeline's configuration and credentials to match the new definition while preserving the pipeline state, including the pipeline ID, cursor positions, offsets, and pipeline history. SingleStore replaces all pipeline metadata with the new definition and always performs a full replacement, even when the configuration remains unchanged. * `DROP PIPELINE` removes the pipeline. * `DROP DATABASE` removes the database. The schema cache is preserved when: * `ALTER PIPELINE ... SET TRANSFORM` modifies only the transform. * `STOP PIPELINE` and `START PIPELINE` stop and restart the pipeline. Enable `pipelines_avro_schema_cache_enabled` when: * Kafka Avro pipelines process frequent or high-throughput batches. * Schemas change infrequently. * Reducing load on the Confluent Schema Registry is important. * Network latency to the Schema Registry affects pipeline performance. > **📝 Note**: Schema caching applies only to Kafka pipelines that use Avro format with the Confluent Schema Registry. It does not affect other pipeline types or data formats. ## Kafka Version Setting > **⚠️ Warning**: Using SSL and SASL with Kafka requires Kafka protocol version 0.9 or later; therefore, `CREATE PIPELINE ... KAFKA ...` and `SELECT ... INTO KAFKA ...` statements using SSL and SASL with Kafka also need to adhere to that version requirement. The Kafka protocol version can be passed in through JSON through the `CONFIG` clause, similar to this `CONFIG '{"kafka_version":"0.10.0.0"}'`. Alternatively, the `pipelines_kafka_version` engine variable controls this parameter for any pipeline without using a Kafka version configuration value in a `CREATE PIPELINE ... KAFKA ...` statement. ## Final Step: Use the Connection String in a SQL Statement Create your `CREATE PIPELINE ... KAFKA ...` or `SELECT ... INTO KAFKA ...` statement, using the string containing the connection settings that you created in the previous steps. ## Examples ## Create a Kafka Pipeline with Data from Confluent Cloud ```sql CREATE PIPELINE quickstart_kafka AS LOAD DATA KAFKA '/test' CONFIG '{"sasl.username": "", "sasl.mechanism": "PLAIN", "security.protocol": "SASL_SSL", "ssl.ca.location": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}' CREDENTIALS '{"sasl.password": ""}' INTO TABLE messages; ``` ## Publish Messages to Confluent Cloud ```sql SELECT text FROM t INTO KAFKA '/test-topic' CONFIG '{"sasl.username": "", "sasl.mechanism": "PLAIN", "security.protocol": "SASL_SSL", "ssl.ca.location": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"}' CREDENTIALS '{"sasl.password": ""}; ``` ## Configure SASL OAUTHBEARER for Use with Kafka Pipelines > **📝 Note**: SASL OAUTHBEARER is not supported with `SELECT INTO ...` ```sql CREATE or REPLACE PIPELINE AS LOAD DATA KAFKA "" CONFIG '{"security.protocol":"SASL_SSL", "sasl.mechanism":"OAUTHBEARER", "sasl.oauthbearer.client.id":"", "sasl.oauthbearer.client.secret":<"CLIENT_SECRET>", "sasl.oauthbearer.token.endpoint.url":"", "sasl.oauthbearer.scope":""}' INTO TABLE ; ``` ## Connecting to Confluent Schema Registry over SSL To enable Pipelines to connect to Confluent Schema Registry over SSL, install the registry’s certificate (ca-cert) on all nodes in your SingleStore Helios cluster. For example: ```shell cat ca-cert >> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem update-ca-trust ``` > **📝 Note**: You only need to install the registry’s certificate (ca-cert) in your SingleStore Helios cluster if you are using a self-signed certificate or a certificate signed by an internal CA. After installing the registry’s certificate on all nodes, existing Pipelines that you already created (or new Pipelines that you create) with the host name/IP address and port of Confluent Schema Registry will use SSL automatically. ## Schema Registry Configuration Options Supported configuration options include: * `schema.registry.ssl.ca.location` * `schema.registry.ssl.ca.directory` * `schema.registry.ssl.verifypeer` * `schema.registry.ssl.certificate.location` * `schema.registry.ssl.key.location` * `schema.registry.ssl.key.password` * `schema.registry.curl.timeout` * `schema.registry.curl.tcp.keepalive` * `schema.registry.curl.tcp.keepidle` * `schema.registry.curl.keepintvl` * `schema.registry.curl.connecttimeout` * `schema.registry.username` * `schema.registry.password` * `schema.registry.curl.verbose` ## Load Data using the SingleStore Kafka Sink Connector The SingleStore Kafka Sink Connector is a Kafka Connect connector that enables you to ingest AVRO, JSON, and CSV messages from Kafka topics into SingleStore Helios. It is a Sink (target) connector designed to read data from Kafka topics and write the data to SingleStore Helios databases. Refer to [Load Data using the SingleStore Kafka Sink Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector.md) for more information. ## In this section * [Load Data from Kafka Using a Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/load-data-from-kafka-using-a-pipeline.md) * [Retrieve Kafka Properties](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/retrieve-kafka-properties.md) * [Kafka Pipeline Using Avro Format](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/kafka-pipeline-using-avro-format.md) * [Kafka Pipeline Using JSON Format](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/kafka-pipeline-using-json-format.md) * [Test your Kafka Cluster using kcat (formerly kafkacat)](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/test-your-kafka-cluster-using-kcat-formerly-kafkacat.md) * [Connect to Kafka Pipelines using an Outbound Endpoint](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/connect-to-kafka-pipelines-using-an-outbound-endpoint.md) * [Kafka Pipeline Errors](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/kafka-pipeline-errors.md) *** Modified at: June 19, 2026 Source: [/cloud/load-data/data-sources/load-data-from-kafka/](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-kafka/) (An index of the documentation is available at /llms.txt) --- # Replicate Data from MongoDB® You can replicate your MongoDB® collections to your SingleStore Helios databases using Change Data Capture (CDC). Refer to [Replicate MongoDB® Collections to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore.md) for more information. *** Modified at: August 14, 2024 Source: [/cloud/load-data/data-sources/replicate-data-from-mongodb/](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mongodb/) (An index of the documentation is available at /llms.txt) --- # Replicate Data from MySQL > **📝 Note**: Replicating data from MySQL via CDC-in pipelines has been deprecated. Future application development must use [SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md), which is recommended for all workloads. Change Data Capture (CDC) pipelines enable you to ingest historical data and sync the continuous changes to data as they happen on the source MySQL database. You can replicate your MySQL databases to SingleStore using CDC pipelines. > **📝 Note**: You can only create a maximum of 16 CDC-in pipelines in total for MongoDB® and MySQL sources. Perform the following tasks: 1. Configure the source MySQL database. Refer to [Configure MySQL](https://docs.singlestore.com/#section-idm4594041876137634092850850389.md) for more information. 2. (Optional) Create a link to the MySQL database. Refer to [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) for more information. You can also specify the link configuration and credentials in the `CONFIG`/`CREDENTIALS` clause of the `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement instead of creating a link. > **📝 Note**: Create a link to the primary MySQL instance. SingleStore does not support replicating MySQL databases using CDC from secondary (replica) MySQL instances. 3. Create the required table(s), stored procedure(s), and pipeline(s) using the `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement. Refer to [Syntax](https://docs.singlestore.com/#section-idm4503128687899233950410758547.md) for more information. You can either [replicate the MySQL tables as is](https://docs.singlestore.com/#section-idm234424168959935.md) or [apply custom transformations](https://docs.singlestore.com/#section-idm234424169115312.md). > **📝 Note**: Before restarting the `INFER PIPELINE` operation, delete all the related artifacts.```sql > DROP TABLE ; > DROP PIPELINE ; > DROP PROCEDURE ; > ``` 4. Once all the components are configured, start the pipelines. * To start all the pipelines, run the `START ALL PIPELINES` SQL statement. * To start a specific pipeline, run the `START PIPELINE ` SQL statement. By default, the pipeline is named `.`. **Note**: Once the pipeline(s) (CDC operation) has started, do not run `ALTER TABLE` statements on the source MySQL database. For more information, refer to the relevant section on this page. * [Replication Strategies](https://docs.singlestore.com/#section-idm234424168393315.md) * [CDC Snapshot Strategies](https://docs.singlestore.com/#section-idm323432564239142.md) * [Configure Ingestion Speed Limit using Engine Variables](https://docs.singlestore.com/#section-idm323435536289338.md) * [Limitations](https://docs.singlestore.com/#section-idm234424135826981.md) ## Replicate MySQL Tables Example The following example shows how to replicate MySQL tables without any custom transformations. This example uses the `LINK` clause to specify the MySQL endpoint connection configuration. Perform the following tasks after configuring MySQL: 1. Create a link to the MySQL endpoint using the `CREATE LINK` command. ```sql CREATE LINK pLink AS MYSQL CONFIG '{"database.hostname": "svchost", "database.port": 3306, "database.ssl.mode":"required"}' CREDENTIALS '{"database.password": "pa55w0rd", "database.user": "repl_user"}'; ``` 2. Create tables, pipelines, and stored procedures in SingleStore based on the inference from the source tables: ```sql CREATE TABLES AS INFER PIPELINE AS LOAD DATA LINK pLink "*" FORMAT AVRO; ``` 3. Start the pipelines, and begin the replication process. ```sql START ALL PIPELINES; ``` ## Prerequisites ## Configure MySQL 1. Enable binary logging for MySQL replication by setting the `log-bin` option to `mysql-bin` in your MySQL configuration. Refer to [Enabling the binlog](https://debezium.io/documentation/reference/stable/connectors/mysql.html#enable-mysql-binlog) for more information. 2. Configure the following additional options in MySQL, as specified: ``` binlog_format=ROW binlog_row_image=FULL ``` You may need to restart MySQL in order to incorporate the configuration changes. Consult the MySQL documentation for more information. 3. Create a new database named `singlestore`. This database is required for internal usage. ```sql CREATE DATABASE IF NOT EXISTS singlestore; ``` The `singlestore` database is meant to store the pipeline metadata information to confirm the status of the pipeline and other operations. SingleStore does not allow replicating this database and the tables that are included in it. The tables in the `singlestore` database can be empty by design. 4. Create a database user for connecting to the MySQL instance: ```SQL CREATE USER IDENTIFIED BY ''; ``` You can also use an existing user. 5. Grant the following privileges to the ``: ```sql GRANT CREATE, INSERT, DELETE, DROP, SELECT ON singlestore.* TO ; GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO ; GRANT SELECT ON *.* TO ; ``` 6. For authentication, the `` must use the `mysql_native_password` plugin. Refer to [Native Pluggable Authentication](https://dev.mysql.com/doc/refman/8.0/en/native-pluggable-authentication.html) for more information. 7. The MySQL instance must allow incoming traffic from the SingleStore Helios cluster. 8. All the source MySQL tables must have primary keys. ## Syntax ```sql CREATE TABLE [IF NOT EXISTS] AS INFER PIPELINE AS LOAD DATA FORMAT AVRO; -- Use either the LINK or MYSQL clause, they are mutually exclusive -- : LINK "." | MYSQL "." CONFIG '' CREDENTIALS '' ``` ```sql CREATE TABLES [IF NOT EXISTS] AS INFER PIPELINE AS LOAD DATA FORMAT AVRO; -- Use either the LINK or MYSQL clause, they are mutually exclusive -- : LINK "*" | MYSQL "*" CONFIG '' CREDENTIALS '' ``` ## `CREATE TABLE ... AS INFER PIPELINE` Behavior The `CREATE TABLE [IF NOT EXISTS] AS INFER PIPELINE` statement, 1. Connects to the MySQL instance using the specified `LINK ` or `CONFIG`/`CREDENTIALS` clause. 2. Infers the schema of the table and creates a table (named **\**) using the inferred schema. When the `IF NOT EXISTS` clause is specified and a table with the specified name already exists, a new table is not created and the existing table is used instead. Refer to [Limitations](https://docs.singlestore.com/#section-idm234424135826981.md) for more information. 3. Creates a pipeline (named `.`) and stored procedure (named `.`) that maps the AVRO data structure to the SingleStore data structure. The `IF NOT EXISTS` clause is ignored for pipelines and stored procedures. If a pipeline or stored procedure with the same name already exists, the `CREATE TABLE ... AS INFER PIPELINE` statement returns an error. ## `CREATE TABLES AS INFER PIPELINE` Behavior The `CREATE TABLES [IF NOT EXISTS] AS INFER PIPELINE` statement creates a table in SingleStore for each table in the source MySQL database using the same set of operations as the `CREATE TABLE [IF NOT EXISTS] AS INFER PIPELINE` statement (specified above). Additionally, it discovers the available databases and tables filtered by `database.exclude.list`, `database.include.list`, `table.exclude.list`, and `table.include.list`. ## Arguments * ``: Name of the table to create in the SingleStore Helios database. You can also specify a table name that differs from the name of the source MySQL table. * ``: Name of the link to the MySQL endpoint. Refer to [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) for more information. * ``: Name of the source MySQL database. * ``: Name of the source MySQL table. * ``: Configuration parameters, including the source MySQL configuration, in the JSON format. Refer to [Parameters](https://docs.singlestore.com/#section-idm457045647466883432568358349.md) for supported parameters. * ``: Credentials to use to access the MySQL database in JSON format. For example: ``` CREDENTIALS '{"database.password": "", "database.user": ""}' ``` * `database.user`: Name of the MySQL database user. * `database.password`: Password of the MySQL database user. ## Parameters The `CREATE {TABLE | TABLES}`, `CREATE LINK`, and `CREATE AGGREGATOR PIPELINE` statements support the following parameters in the `CONFIG` clause: | Parameter | Description | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `database.hostname` | IP address or the hostname of the source MySQL instance. | | `database.port` | Port of the MySQL instance.Default:`3306` | | `database.ssl.mode` | Specifies the SSL mode. | | `max.queue.size` | Specifies the size of the queue inside the extractor process for records that are ready for ingestion. Increasing the queue size results in an increase in the memory consumption by the replication process and you may need to increase the`pipelines_cdc_java_heap_size`engine variable. Refer to[Configure Ingestion Speed Limit using Engine Variables](https://docs.singlestore.com/#section-idm323435536289338.md)for more information.Default:`1024` | | `max.batch.size` | Specifies the maximum number of rows of data fetched from the remote source in a single iteration (batch).`max.batch.size`must be lower than`max.queue.size`. Refer to[Configure Ingestion Speed Limit using Engine Variables](https://docs.singlestore.com/#section-idm323435536289338.md)for more information.Default:`512` | | `poll.interval.ms` | Specifies the interval for polling of remote sources (in milliseconds) if there were no new records in the previous iteration in the replication process.Default:`500` | | `table.include.list` | A comma-separated list of regular expressions that match fully-qualified table identifiers (in`databaseName.tableName`format) for MySQL tables to monitor. By default, changes in all the non-system tables in the database are captured. This parameter is only supported in`CREATE TABLES AS INFER PIPELINE`statement. When this option is specified, tables excluded from the list are not monitored. The`table.include.list`and`table.exclude.list`parameters are mutually exclusive, i.e., they cannot be used in the same`CREATE TABLES AS INFER PIPELINE`statement. This parameter is ignored in the`CREATE TABLE ... AS INFER PIPELINE`statement. | | `table.exclude.list` | A comma-separated list of regular expressions that match fully-qualified table identifiers (in`databaseName.tableName`format) for MySQL tables to exclude from the monitoring list. By default, this list is empty. This parameter is only supported in`CREATE TABLES AS INFER PIPELINE`statement. The`table.include.list`and`table.exclude.list`parameters are mutually exclusive, i.e., they cannot be used in the same`CREATE TABLES AS INFER PIPELINE`statement. | | `database.include.list` | A comma-separated list of regular expressions that match the names of databases to monitor. By default, all the databases are monitored. When this option is specified, databases excluded from the list are not monitored. This parameter is only supported in`CREATE TABLES AS INFER PIPELINE`statements. The`database.include.list`and`database.exclude.list`parameters are mutually exclusive, i.e., they cannot be used in the same`CREATE TABLE AS INFER PIPELINE`statement. If this option is used with the`table.include.list`or`table.exclude.list`option, it returns the intersection of the matches | | `database.exclude.list` | A comma-separated list of regular expressions that match the names of databases to exclude from monitoring. By default, this list is empty. This parameter is only supported in`CREATE TABLES AS INFER PIPELINE`statement. The`database.include.list`and`database.exclude.list`parameters are mutually exclusive, i.e., they cannot be used in the same`CREATE TABLES AS INFER PIPELINE`statement. If this option is used with the`table.include.list`or`table.exclude.list`option, it returns the intersection of the matches. | | `signal.data.collection` | A table in the remote MySQL source that is used bySingleStoreto generate special markings for snapshotting and synchronization. By default, this parameter is set to`singlestore.signals_xxxxxx`, where`xxxxxx`is an automatically generated character sequence. The default signal table is in the`singlestore`database. Once the pipelines are started, any change to the value of this parameter is ignored, and the pipelines use the latest value specified before the pipelines started. | | `snapshot.mode` | Specifies the snapshot mode for the pipeline. It can have one of the following values:
  • "initial" (Default): Perform a full snapshot first and replay CDC events created during the snapshot. Then, continue ingestion using CDC.
  • "incremental": Start the snapshot operation and CDC simultaneously.
  • "schema_only": Skip the snapshot, and ingest changes using CDC.
Refer to[CDC Snapshot Strategies](https://docs.singlestore.com/#section-idm323432564239142.md)for more information.Default:`"initial"` | ## Replication Strategies Use one of the following methods to create the required components for data ingestion. ## Replicate MySQL Tables As Is To replicate or migrate MySQL tables as is, use the `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement. This method automatically creates the required tables, pipelines, and stored procedures. Refer to [Syntax](https://docs.singlestore.com/#section-idm4503128687899233950410758547.md) for more information. ## Apply Transformations or Ingest a Subset of Columns To apply transformations or ingest only a subset of columns, manually create the required tables, stored procedure, and pipelines: 1. Run the `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement to infer the schema of the MySQL table(s) and automatically generate templates for the relevant table(s), stored procedure(s), and aggregator pipeline(s). 2. Use the automatically-generated templates as a base to create a new table(s), stored procedure(s), and pipeline(s) for custom transformations. To inspect the generated table(s), stored procedure(s), and pipeline(s), use the `SHOW CREATE TABLE` , `SHOW CREATE PROCEDURE`, and `SHOW CREATE PIPELINE` commands, respectively. After running the `SHOW` commands, you can drop the templates and then recreate the same components with custom transformations. Using the automatically-generated templates: 1. Create table(s) in SingleStore with a structure that can store the ingested MySQL table. By default, the primary key is used as the shard key for a table. You can specify a different shard key and sort key in the table definition. For an example, refer to [Example 3](https://docs.singlestore.com/#section-idm234424452975799.md). Refer to [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) for more information. 2. Create stored(s) procedure to map the MySQL table columns to the SingleStore table and implement other transformations required. Refer to [CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-procedure.md) for information on creating stored procedures. 3. Create pipeline(s) to ingest the MySQL tables using the `CREATE AGGREGATOR PIPELINE` SQL statement. Note that the pipeline definition may contain encoded names in the field mapping clause for unsupported symbols in the `_uxxxx` format because of Avro naming limitations, for example, `` `i_d` <- `payload`::`i_u005fd` ``. The table and stored procedure declarations use human readable Unicode symbols instead of the encoding. Hence, they do not require additional handling. Refer to [Parameters](https://docs.singlestore.com/#section-idm457045647466883432568358349.md) for a list of supported parameters. Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for the complete syntax and related information. **Note**: The CDC feature only supports `AGGREGATOR` pipelines. Refer to [Syntax](https://docs.singlestore.com/#section-idm4503128687899233950410758547.md) for information on `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement. ## CDC Snapshot Strategies SingleStore supports the following strategies for creating snapshots: * **Perform a full snapshot before CDC** (`"snapshot.mode":"initial"`): The pipeline captures the position in the binary log and then performs a full snapshot of the data. Once the snapshot is complete, the pipeline continues ingestion using CDC. This strategy is enabled by default. If the pipeline is restarted while the snapshot is in progress, the snapshot is restarted from the beginning. If the initial snapshot is large in size and the deployment is prone to restarts or connection issues from the source, SingleStore recommends using the `incremental` snapshot mode. Note that the `incremental` snapshot mode is slower than the `initial` mode. For faster data ingestion when the initial historical data is large in size, manually perform the snapshot and capture changes using the `schema_only` mode. To use this strategy, set `"snapshot.mode":"initial"` in the `CONFIG` JSON. **Requirement**: The binary log retention period must be long enough to maintain the records while the snapshot is in progress. Otherwise, the pipeline will fail and the process will have to be started over. * **CDC only** (`"snapshot.mode":"schema_only"`): The pipeline will not ingest existing data, and only the changes are captured using CDC. To use this strategy, set `"snapshot.mode":"schema_only"` in the `CONFIG` JSON. * **Perform a snapshot in parallel to CDC** (`"snapshot.mode":"incremental"`): The pipeline captures the position in the binary log and starts capturing the changes using CDC. In parallel, the pipeline performs incremental snapshots of the existing data and merges it with the CDC records. Although this strategy is slower than performing a full snapshot and then ingesting changes using CDC, it is more resilient to pipeline restarts. To use this strategy, set `"snapshot.mode":"incremental"` in the `CONFIG` JSON. **Requirement**: The binary log retention period must be long enough to compensate for unexpected pipeline downtime. * **Manually perform the snapshot and capture changes using the CDC pipeline**: To use this strategy, set `"snapshot.mode":"schema_only"` in the `CONFIG` JSON. 1. Create a pipeline and then wait for at least one batch of ingestion to capture the binary log position. 2. Stop the pipeline. 3. Snapshot the data using any of the suitable methods, for example, `mysqldump`. 4. Restore the snapshot in SingleStore with any of the supported methods, for example, [load data from an S3 bucket](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3.md). 5. Start the CDC pipeline. This strategy provides faster data ingestion when the initial historical data is very large in size. **Requirement**: The binary log retention period must be long enough to maintain the records while the snapshot is in progress. Otherwise, the pipeline will fail and the process will have to be started over. ## Configure Ingestion Speed Limit using Engine Variables Use the following engine variables to configure ingestion speed: | Variable Name | Description | Default Value | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | | `pipelines_cdc_row_emit_delay_us` | Specifies a forced delay in row emission while migrating/replicating your tables (or collections) to yourSingleStore Heliosdatabases. It can have a maximum value of`1000000`. | `1` | | `pipelines_cdc_java_heap_size` | Specifies the JVM heap size limit (in MBs) for CDC-in pipelines. | `128` | Refer to [Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables.md) for information on how to set engine variables. You can also use the [supported parameters](https://docs.singlestore.com/#section-idm457045647466883432568358349.md) to configure the ingestion speed. ## In-Depth Variable Definitions Use the `pipelines_cdc_row_emit_delay_us` engine variable to limit the impact of CDC pipelines on the master aggregator node. It specifies a forced delay in row emission during ingest. This variable can be set to a maximum value of `1000000`. Use the `max.queue.size` parameter in the `CONFIG` JSON to control the ingestion speed. SingleStore recommends setting `max.batch.size` to half of `max.queue.size`. Increasing the queue size requires a larger Java heap limit, adjust the `pipelines_cdc_java_heap_size` engine variable accordingly. Query the `INFORMATION_SCHEMA.PIPELINES_BATCHES_SUMMARY` table for information on pipeline batch performance. ## Optimize CDC-in Pipelines ## Pipelines and Extractors CDC-in [pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md) ("pipelines") are aggregator pipelines that run on the Master Aggregator (MA). Each pipeline extracts data from a single source table and loads data into a single SingleStore table. Extractors are subprocesses that extract data from the source and provide the data to the pipelines. The extractors are shared between the CDC-in pipelines. SingleStore recommends ingesting a limited number of tables using CDC-in pipelines. ## Memory and Resource Consumption Each extractor consumes a persistent amount of resources (approximately `pipelines_cdc_java_heap_size` per extractor). It is used to store the queue of rows for subsequent batches of ingestion. **Note**: Total memory consumption may be higher and includes memory usage by the static system memory, shared libraries, JVM heap, etc. To reduce the resource consumption on the MA reduce the JVM heap size (`pipelines_cdc_java_heap_size`). Although, reducing this limit may also reduce the ingestion speed. **Note**: If JVM heap size is reduced, you may also need to reduce the pipeline batch and queue size. ## Limitations * SingleStore intentionally does not support all of the features of MySQL. Refer to [Unsupported MySQL Features](https://docs.singlestore.com/cloud/developer-resources/unsupported-mysql-features.md) for more information. * Autoincrement properties, default values, indexes, and additional keys (excluding primary keys) are not inferred or replicated. * The Load Data workflow in the [Cloud Portal](https://portal.singlestore.com) (**Cloud Portal > Ingestion > Load Data**) is not supported when using private links. Use the SQL commands instead. * If any column in the source MySQL table has a data type that is not supported by SingleStore, all the columns in the table are ingested as `TEXT` type in SingleStore. Apply custom transformations to your stored procedures and pipelines to map the unsupported MySQL type to SingleStore type. Refer to [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) for information on supported data types. * The MySQL database, table, or column names must not contain the following characters: `/`, `\`, `` ` ``, `'`, `"`, `'\b'`, `'\t'`, `'\n'`, `'\f'`, `'\r'`, or non-printable characters. Additionally, the database name must not contain `.`. ## Replicate MySQL Tables via AWS PrivateLink SingleStore supports loading data from MySQL using CDC pipelines via AWS PrivateLink. To replicate your MySQL databases using private links: 1. Create an outbound connection from SingleStore. Refer to [Connect to SingleStore Helios using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink.md) for more information. Even though data is ingested into SingleStore, an outbound connection is used because the ingestion process is initiated by connecting from SingleStore Helios to the AWS instance/resource. 2. Attach the MySQL instance to the target group of your AWS service. 3. Configure MySQL. Refer to [Prerequisites](https://docs.singlestore.com/#section-idm234424118362126.md) for more information. 4. Copy the **Service Name** of your outbound connection from the [Cloud Portal](https://portal.singlestore.com/). 1. Select **\ > Firewall > Private Links**. 2. Under the **Actions** column of your private connection, select the three dots, and select **View Connection**. 3. From the **View Connection** dialog, copy the **VPC Endpoint Service Name**. 5. Create a `LINK` to the VPC Endpoint Service. Specify the **Service Name** copied earlier in the `database.hostname` parameter. ```sql CREATE LINK AS MYSQL CONFIG '{"database.hostname": "", "database.port": 3306 [, "database.ssl.mode":""]}' CREDENTIALS '{"database.password": "", "database.user": ""}'; ``` 6. Replicate the tables. ```sql CREATE TABLES AS INFER PIPELINE AS LOAD DATA LINK "*" FORMAT AVRO; START ALL PIPELINES; ``` You can choose any of the supported [replication strategies](https://docs.singlestore.com/#section-idm234424168393315.md) and [CDC Snapshot Strategies](https://docs.singlestore.com/#section-idm323432564239142.md). ## Troubleshooting * If the `CREATE {TABLES | TABLE} AS INFER PIPELINE` SQL statement returns an error, run the `SHOW WARNINGS` command to view the reason behind the failure. * To view the status of the pipelines, query the `information_schema.PIPELINES_CURSORS` table. Run the following SQL statement to display the status of the replication task: ```sql SELECT SOURCE_PARTITION_ID, EARLIEST_OFFSET, LATEST_OFFSET, LATEST_EXPECTED_OFFSET-LATEST_OFFSET as STATUS, UPDATED_UNIX_TIMESTAMP FROM information_schema.PIPELINES_CURSORS; ``` The value in the `STATUS` column indicates the following: * `1`: Indicates that the snapshot is in progress and the pipeline is expecting more data. * `0`: Indicates that the initial snapshot is complete and the pipeline is capturing changes using CDC. **Note**: If `snapshot.mode` is set to `incremental`, the pipeline performs incremental snapshots in parallel to capturing changes via CDC. In this case, the value `1` in the `STATUS` column indicates that the pipeline is performing incremental snapshots and capturing changes in parallel. * To view pipeline errors, run the following SQL statement: ```sql SELECT * FROM information_schema.PIPELINES_ERRORS; ``` * If a pipeline fails with an out of memory error in Java, either increase the heap size using the `pipeline_cdc_java_heap_size` engine variable or reduce the `max.queue.size` and `max.batch.size` parameters in the `CONFIG` clause. The heap size is limited by the memory available on the MA. SingleStore recommends setting the queue size as double of the batch size. ## Examples Refer to [Replicate MySQL Tables Example](https://docs.singlestore.com/#section-idm4578618939937633950439090354.md) for a sample replication example. ## Example 1 The following example shows how to specify a table name that differs from the source MySQL table name: ```sql CREATE TABLE s2table AS INFER PIPELINE AS LOAD DATA LINK pLink "db1.mtest" FORMAT AVRO; ``` This command replicates the `mtest` table from the source MySQL database to the `s2table` in SingleStore. ## Example 2 The following example shows how to specify additional parameters in a `CREATE LINK` statement: ```sql CREATE LINK pLink AS MYSQL CONFIG '{ "database.hostname": "svchost", "database.port": 3306, "database.ssl.mode":"required", "database.exclude.list": "mysql,performance_schema", "table.include.list": "db1[.]example", "max.queue.size": 1024, "max.batch.size": 512, "poll.interval.ms": 500}' CREDENTIALS '{ "database.password": "pa55w0rd", "database.user": "repl_user"}'; ``` ## Example 3 The following example shows how to specify a different sort key and shard key for the table. 1. Infer the schema of the source MySQL table and generate a template for the table: ```sql CREATE TABLE IF NOT EXISTS mtest AS INFER PIPELINE AS LOAD DATA LINK pLink "db1.mtest" FORMAT AVRO; ``` 2. Run the `SHOW CREATE TABLE` command to get the table definition: ```sql SHOW CREATE TABLE mtest; ``` ```output +-----------+-----------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-----------+----------------------------------------------------------------------------------------------------------------------------------+ | mtest | CREATE TABLE `mtest` ( `id` int(11) NOT NULL, `text` text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci, PRIMARY KEY (`id`), SHARD KEY `__SHARDKEY` (`id`), SORT KEY `__UNORDERED` () ) AUTOSTATS_CARDINALITY_MODE=INCREMENTAL AUTOSTATS_HISTOGRAM_MODE=CREATE AUTOSTATS_SAMPLING=ON SQL_MODE='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER' | +-----------+-----------------------------------------------------------------------------------------------------------------------------------+ ``` 3. Drop the existing table, and use the definition of the template as base to specify a different sort key or shard key: ```sql DROP TABLE mtest; CREATE TABLE mtest ( id INT(11) NOT NULL PRIMARY KEY, text TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci, SHARD KEY (id), SORT KEY(id, text)); ``` ## Example 4 - Use Regular Expressions to Include or Exclude Tables or Databases The names of the databases or tables to include or exclude are specified using regular expressions. For example, to include or exclude all the tables in a database named `dbTest`, use the following regular expression: ``` dbTest[.].* -- OR -- dbTest\..* ``` where, * `dbTest` matches the exact string `dbTest`. * `[.]` and `\.` match the character dot (`.`), which represents the dot (`.`) in `.` notation. Note that `.` is a special character in regular expressions, and to match the character `.` it must either be escaped (`\.`) or specified as a literal character (`[.]`). * `.*` matches any sequence of characters, because the dot (`.`) matches any single character and the asterisk (`*`) matches zero or more occurrences of any character. Here's a sample `CREATE LINK` statement to include all the tables in the `dbTest` database, for example, `dbTest.foo`, `dbTest.bar`, `dbTest.exampleTable`, etc.: ```sql CREATE LINK pLink AS MYSQL CONFIG '{ "database.hostname": "svchost", "database.port": 3306, "database.ssl.mode":"required", "database.exclude.list": "mysql,performance_schema", "table.include.list": "dbTest\..*" }' CREDENTIALS '{ "database.password": "pa55w0rd", "database.user": "repl_user"}'; ``` To specify the tables or databases to include or exclude from the replication task, use either (or a combination) of the following parameters: `table.include.list`, `table.exclude.list`, `database.include.list`, or `database.exclude.list`. Refer to [Parameters](https://docs.singlestore.com/#section-idm457045647466883432568358349.md) for more information. ## In this section * [Load Data from MySQL using mysqldump](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql/load-data-from-mysql.md) *** Modified at: May 12, 2026 Source: [/cloud/load-data/data-sources/replicate-data-from-mysql/](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql/) (An index of the documentation is available at /llms.txt) --- # SingleStore Replicate > **📝 Note**: This product is no longer supported. ## Overview SingleStore Replicate is a tool for transferring data from a wide variety of heterogeneous databases into SingleStore Helios. It provides a number of features and functions to tailor how the data is replicated, and offers: * A command-line interface for installation, configuration, and use * Importing data from certain sources, notably Oracle, via a transaction-consistent point-in-time snapshot * Customizable filters and maps to tailor the replicated data * SQL 2017, 2016, and 2014 standards compliance To best take advantage of SingleStore Helios’s flexibility and scalability, data in existing databases can now be moved into SingleStore Helios, allowing data-driven applications to work with minimal changes to either code or data. This capability is especially important when the requirements of the original application have grown beyond the ability to scale the current database, or the database system product hosting the original database has simply become too costly *to* scale. ## Supported Data Sources SingleStore Replicate supports data transfer from the following sources into SingleStore Helios: * AWS Aurora * Cassandra * DB2 (IBM) * Hana (SAP) * MemSQL v6.0 and later * MongoDB® * MySQL * Netezza * Oracle Enterprise Edition (EE) including Compression and Partitioning and Real Application Clusters (RAC) * PostgreSQL v9.4 and later * Salesforce * SQL Server (Microsoft) * Teradata * Vertica ## Capabilities and Benefits Replicate supports high-performance ingest using a parallel, distributed architecture that reduces labor and speeds data transfer, which decreases the time it takes to complete complex data transfer jobs from weeks down to days. Replicate’s benefits grow as the complexity of the data transfer increases. ## System Requirements Physical or virtual machines configured for high performance will make Replicate achieve maximum scaling and replication throughput. An example hardware configuration that typically works well for Replicate consists of: * A multi-core 32/64-bit CPU with 128GB memory * A minimum of 400-500GB (SSDs/NVMe) of free storage * A current Linux version (Red Hat/CentOS/Ubuntu) * Java 8 or later installed (via the JRE or JDK) A fast network connection with provisioned redundancies at all points of failure will offer the best performance, reliability, and lowest latency for replication. As Replicate is Java-based, it may be possible, though not guaranteed, to work on other operating systems that can run Java SE Runtime Environment 8. Support for other operating systems is planned for future releases. ## Obtain Replicate Software and Documentation SingleStore Replicate is available to all customers who have purchased the SingleStore Helios database or SingleStore Helios. To obtain the SingleStore Replicate software and full documentation, contact your SingleStore sales representative, or SingleStore support. *** Modified at: January 4, 2024 Source: [/cloud/load-data/data-sources/singlestore-replicate/](https://docs.singlestore.com/cloud/load-data/data-sources/singlestore-replicate/) (An index of the documentation is available at /llms.txt) --- # SingleStore Snowpark App Turbocharge your data with SingleStore’s Snowflake Native App on Snowpark Container Services (SPCS). Leverage the real-time analytics power of SingleStore for applications built on Snowflake, all while keeping your data within Snowflake. For more information, refer to [Get Started with the SingleStore Snowpark App](https://docs.singlestore.com/spcs/). *** Modified at: October 24, 2024 Source: [/cloud/load-data/data-sources/singlestore-snowpark-app/](https://docs.singlestore.com/cloud/load-data/data-sources/singlestore-snowpark-app/) (An index of the documentation is available at /llms.txt) --- # Integrate with SingleStore Helios ## Introduction The guides in this section explain how to integrate third-party applications with your cluster. ## AI/ML Platform * [Dataloop](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-dataloop.md) ## Analytics and BI * [Alteryx Designer](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer.md) * [Cognos Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics.md) * [Denodo](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo.md) * [Dremio](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio.md) * [GoldenGate](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate.md) * [GoodData](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata.md) * [Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker.md) * [MicroStrategy](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy.md) * [Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi.md) * [Sisense](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense.md) * [Superset](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset.md) * [Tableau](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau.md) * [ThoughtSpot](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot.md) * [Tibco Spotfire](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire.md) * [Zoho Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics.md) ## Application Platform * [Twilio Segment](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-twilio-segment.md) * [Vercel](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-vercel.md) ## Data Integration and Extract, Transform, Load (ETL) * [AWS Glue](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-aws-glue.md) * [Apache Beam](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-beam.md) * [Apache Flink](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-flink.md) * [Apache NiFi](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-apache-nifi.md) * [Azure Data Factory (ADF)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-data-factory-adf.md) * [Azure Event Hubs](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-event-hubs.md) * [Data Virtuality Pipes](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-data-virtuality-pipes.md) * [dbt](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/transform-data-with-dbt.md) * [Estuary Flow](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-estuary-flow.md) * [Fivetran](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-fivetran.md) * [Flyway](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway.md) * [HVR](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-hvr.md) * [Informatica PowerCenter](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-informatica-powercenter.md) * [Liquibase](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-liquibase.md) * [Qlik Replicate](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate.md) * [SingleStore Kafka Sink Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector.md) * [Spark](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark.md) * [StreamSets](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets.md) * [Tableau Prep](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-prep.md) ## Management * [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) * [Terraform](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-singlestore-helios-using-terraform.md) ## Monitoring * [Datadog](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-datadog.md) * [New Relic](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-new-relic.md) ## Schema Migration * [Flyway](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway.md) ## Security and Governance * [Collibra](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-collibra-platform.md) * [Teleport](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/configure-rbac-with-teleport.md) ## In this section * [Load Data from AWS Glue](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-aws-glue.md) * [Load Data from Apache Beam](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-beam.md) * [Load Data from Apache Flink](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-flink.md) * [Connect with Apache NiFi](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-apache-nifi.md) * [Load Data from Azure Event Hubs](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-event-hubs.md) * [Load Data from Azure Data Factory (ADF)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-data-factory-adf.md) * [Connect with Collibra Platform](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-collibra-platform.md) * [Connect with Dataloop](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-dataloop.md) * [Load Data from Data Virtuality Pipes](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-data-virtuality-pipes.md) * [Transform Data with dbt](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/transform-data-with-dbt.md) * [Load Data from Estuary Flow](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-estuary-flow.md) * [Load Data with Fivetran](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-fivetran.md) * [Migrate Schema with Flyway](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway.md) * [Load Data from HVR](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-hvr.md) * [Load Data from Informatica PowerCenter](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-informatica-powercenter.md) * [Load Data with Liquibase](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-liquibase.md) * [Load Data from Qlik Replicate](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate.md) * [Load Data using the SingleStore Kafka Sink Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector.md) * [Load Data from Spark](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark.md) * [Load Data from StreamSets](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets.md) * [Connect with Tableau Prep](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-tableau-prep.md) * [Configure RBAC with Teleport](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/configure-rbac-with-teleport.md) * [Connect with Twilio Segment](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-twilio-segment.md) * [Connect with Vercel](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-vercel.md) *** Modified at: December 11, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Configure RBAC with Teleport The Teleport integration provides secure and certificate-based access to SingleStore via the Teleport Database Service. This integration enables you to control access to SingleStore databases using role-based access control (RBAC) through the Teleport RBAC system. ## How the Teleport Integration Works Teleport uses a proxy service that routes traffic from database clients to the SingleStore database using mutual TLS (mTLS). Teleport maintains a certificate authority (CA) for database clients that issues short-lived certificates for user sessions. The Teleport Database Service presents certificates signed by this CA when proxying connections to the SingleStore database, which is configured to trust the Teleport database client CA. The Teleport Database Service verifies the identity of the SingleStore databases by checking their TLS certificates against either the Teleport database CA or a custom CA configured for the database. Teleport authenticates using mutual TLS (mTLS). When a user initiates a database session, the Teleport Database Service presents a Teleport-signed certificate to SingleStore. To verify the connection, SingleStore presents its own certificate signed by either Teleport CA or a custom CA. After successful mutual authentication, the Teleport Database Service routes client traffic to the SingleStore database using the same protocol and tools used to connect to MySQL Server. ## Prerequisites * Install the [tctl](https://goteleport.com/docs/reference/cli/tctl/) and [tsh](https://goteleport.com/docs/connect-your-client/tsh/) clients. * An active Teleport cluster. Verify connectivity to the Teleport cluster. To test connectivity: * Run `tsh login` to sign in. For example: ```shell tsh login --proxy= --user= ``` * Run `tctl status`. Typically, if this command runs successfully, other `tctl` commands will run as expected. * An active SingleStore deployment with "Upload a CA Bundle" feature enabled. * A host to run the Teleport Database Service. * Install the `mariadb` or `mysql` command-line client. ## Configure the Teleport Connection Perform the following tasks to configure the connection between Teleport and SingleStore: 1. Create a Teleport database token. 2. Export Teleport CA. 3. Configure SingleStore. 4. Create a local Teleport User. 5. Configure and start Teleport Database Service. 6. Connect to SingleStore. Once the connection is successfully configured, use Teleport to manage access to the SingleStore databases. ## Create a Teleport Database Token The Teleport Database Service requires a token to connect to the Teleport cluster. Run the following command to generate a valid token: ```shell tctl tokens add --type=db --format=text ``` Copy and store the generated token in `/tmp/token` on the server that will run the Teleport Database Service. ## Export Teleport CA To enable SingleStore to accept and validate client certificates issued by Teleport, export the Teleport CA and configure SingleStore to use it for client certificate verification. Run the following command to export the Teleport CA bundle to a file, say **teleport-db-client.pem**: ```shell tctl auth export --type=db-client > teleport-db-client.pem ``` ## Configure SingleStore ## Upload the Teleport CA Bundle Upload the exported Teleport CA bundle to SingleStore to enable client verification for MTLS/X509. Perform the following tasks: 1. Log in to the [Cloud Portal](https://portal.singlestore.com) and select your deployment. 2. On the **Security** tab, select **Upload CA Bundle**. 3. Upload the **teleport-db-client.pem** file exported earlier. Upon successful upload, the **CA Bundle uploaded** message is displayed. Additionally, you can view, replace, or delete a CA bundle. ## Create/Update a Database User Teleport uses a certificate to authenticate database users. Either create a new user or update an existing user to authenticate via a certificate. For example: * To create a new SingleStore database user: ```sql CREATE USER 'jane'@'%' REQUIRE SUBJECT '/CN=jane'; ``` Provide the necessary privileges to the new database user. For example: ```sql GRANT ALL PRIVILEGES ON dbTest.* TO 'jane'; ``` * Update an existing user: ```sql ALTER USER 'jane'@'%' REQUIRE SUBJECT '/CN=jane'; -- Remove the password for the user SET PASSWORD FOR 'jane'@'%' = PASSWORD(""); ``` ## Create a Local Teleport User Use the `tctl users add` command to create a local Teleport user. For example: ```shell tctl users add \ --roles=access \ --db-users="*" \ --db-names="*" \ jane ``` This command creates a local Teleport user named `jane` with the built-in `access` role and access to all the databases and database user names. Follow the instructions in the output to complete user setup. ## Configure and Start Teleport Database Service Install and configure Teleport on the Linux server where Teleport Database Service will run. ## Install on a Linux Server 1. To install a Teleport Agent on a Linux server, run the Teleport cluster's install script. The script selects the correct version, edition, and installation mode for the cluster. For example: ```shell curl "https://:/scripts/install.sh" | sudo bash ``` Update the Teleport cluster hostname and port before running the command. 2. Generate a configuration file at `/etc/teleport.yaml` for the Teleport Database Service. 1. Download the SingleStore CA from . 2. Run the following command to generate the configuration file. Specify the SingleStore CA file in the `--ca-cert-file` option. > **📝 Note**: A single Teleport process can run multiple services. The following command overwrites any existing configuration file. If you're running multiple services, add `--output=stdout` to print the configuration in the terminal, and then manually adjust `/etc/teleport.yaml`. ```shell sudo teleport db configure create \ -o file \ --token=/tmp/token \ --proxy=example.teleport.sh:443 \ --name=example-singlestore \ --protocol=mysql \ --uri=:3306 \ --ca-cert-file="/path/to/singlestore_bundle.pem" \ --labels=env=dev ``` where, * `--proxy`: Host and port of the Teleport Proxy Service. * `--uri`: Endpoint of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. 3. Configure the Teleport Database Service to automatically start when the system starts. * If Teleport is installed using a package manager, run the following commands: ```shell sudo systemctl enable teleport sudo systemctl start teleport ``` * If Teleport is installed using a tar archive, run the following command: ```shell sudo teleport install systemd -o /etc/systemd/system/teleport.service sudo systemctl enable teleport sudo systemctl start teleport ``` Run the `systemctl status teleport` command to view the status of the Teleport Database Service. ## Connect to SingleStore Once the Teleport Database Service is added to the cluster, run the following commands to log in to Teleport and view the databases accessible to the current user. For example: ```shell tsh login --proxy=teleport.example.com --user=jane tsh db ls ``` ```output Name Description Labels ------------------- ------------------- ------- example-singlestore Example SingleStore env=dev ``` To connect to a database, run the `tsh db connect` command. For example: ```shell tsh db connect --db-user=jane --db-name= example-singlestore ``` This command retrieves the credentials for a database and connects to it. Additionally, you can connect to the database using the Web UI. > **📝 Note**: The `mysql` or `mariadb` command-line client must be available in the system `PATH` to establish a connection. `mariadb` is the default command-line client for SingleStore. To log out of a database and remove the credentials, run the following command: ```shell # Remove credentials for a specific database instance. tsh db logout # Remove credentials for all database instances. tsh db logout ``` *** Modified at: July 10, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/configure-rbac-with-teleport/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/configure-rbac-with-teleport/) (An index of the documentation is available at /llms.txt) --- # Connect with Apache NiFi Apache NiFi is a data integration and automation platform designed to ingest, move, transform, and manage data across various systems in real time. You can connect to SingleStore Helios from Apache NiFi using [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). Refer to [Apache NiFi](https://nifi.apache.org/components/) documentation for more information. ## Prerequisites 1. [Download and install](https://nifi.apache.org/nifi-docs/getting-started.html#downloading-and-installing-nifi) Apache NiFi. 2. Install Java 21+. 3. [Download](https://github.com/memsql/S2-JDBC-Connector/releases/latest) the SingleStore JDBC driver `.jar` file. 4. Start Apache NiFi. Run the following command from the NiFi installation directory. ```shell ./bin/nifi.sh start ``` 5. Copy the username and password from the generated log. Use these credentials to log in to NiFi Flow. For example, ```shell tail -n 100 /logs/nifi-app.log | grep Generated ``` ```output Generated Username [eea280ec-6000-4000-a000-9aaf521ef5cf] Generated Password [He5Gd9ZHVgkA0JQR9d9uNc2A5WAHROI4] ``` ## Configure the Connection to SingleStore To configure the connection to SingleStore in NiFi Flow, perform the following: 1. Open NiFi Flow, go to . 2. Log in to NiFi Flow using the generated username and password. 3. On the **Nifi Flow** canvas, open the options menu (right-click) and select **Controller Services**. 4. Select the **+** icon. 5. On the **Add Controller Service** dialog, from the list of controllers, select **DBCPConnectionPool > Add**. 6. On the **Controller Services** page. Select the three dots for the controller added in the previous step, and then select **Edit**. 7. In the **Edit Controller Service** dialog, 1. On the **Settings** tab, enter **SingleStore** in the **Name** field. 2. On the **Properties** tab, enter or select the following: 1. **Database Connection URL**: Specify the connection string for the SingleStore deployment in the following format: `jdbc:singlestore://:/` Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. 2. **Database Driver Class Name**: Specify `com.singlestore.jdbc.Driver`. 3. **Database Driver Location**: Specify the path to the SingleStore JDBC driver `.jar` file downloaded earlier. 4. **Database User**: Specify the username of the SingleStore database user with which to connect. 5. **Password**: Specify the password for the SingleStore database user. 6. Specify other connection configuration properties as applicable. 7. Under **Verification**, select **✔ > Verify** to verify the connection configuration. 3. After the connection configuration is verified, a message similar to the following is displayed: ``` Perform Validation Component Validation passed Configure Data Source Successfully configured data source Establish Connection Successfully established Database Connection ``` 4. Select **Apply**. The **SingleStore** controller service is now configured. 8. Enable the **SingleStore** controller service. Select the three dots for the controller, and then select **Enable** from the list. 9. Close the **Enable Controller Service** dialog once the **SingleStore** controller service is enabled. The connection to the SingleStore deployment is now configured. Specify the **SingleStore** controller service in processor configuration to connect to your SingleStore database. ## Example The following example connects to a SingleStore deployment, runs a SQL query, and then stores the output in a file. 1. On the NiFi Flow canvas, add the **ExecuteSQL** processor by dragging the processor to the canvas. 2. Configure the ExecuteSQL processor. Open the options menu for the processor (right-click) and then select **Configure**. 3. On the **Edit Processor** dialog, enter or select the following: 1. On the **Settings** tab, enter a name for the processor. For example, **Query table**. 2. On the **Properties** tab, select **Database Connection Pooling Service** and then select **SingleStore** (the configured controller service) from the list. 3. In the **SQL Query** field, enter a SQL query to run. 4. Specify other connection configuration properties as applicable. 5. Select **Apply**. 4. On the NiFi Flow canvas, add the **ConvertRecord** processor. 5. Configure the ConvertRecord processor. Open the options menu for the processor (right-click) and then select **Configure**. 6. On the **Edit Processor** dialog, enter or select the following: * **Record Reader**: Select the three dots and then select **Create new service > AvroReader**. This example uses the `AvroReader` service. * **Record Writer**: Select the three dots and then select **Create new service > JSONRecordSetWriter**. This example uses the `JSONRecordSetWriter` service. 7. On the NiFi Flow canvas, add the **PutFile** processor. 8. Configure the PutFile processor. Open the options menu for the processor (right-click) and then select **Configure**. 9. In the **Edit Processor** dialog, enter or select the following: 1. On the **Properties** tab, enter the path to the directory where the output file is stored in the **Directory** field. 2. On the **Relationships** tab, select **terminate** for success and failure. 3. Select **Apply**. 10. Configure the relationships between the ExecuteSQL, ConvertRecord, and PutFile processors. 11. (If required) Enable all the controller services. Open the options menu for the canvas (right-click), and then select **Enable All Controller Services**. 12. Open the options menu for the canvas and select **Start**. 13. The output is written to a file in the specified directory. For example, > **📝 Note**: The output is formatted for readability. ```shell cat 0b570ff9-9000-4000-9000-9eff3d4dde09 ``` ```output [ { "id": 2, "created": "2025-06-05 11:58:23.978207", "payload": "{\"eventId\":202,\"tags\":[\"music\",\"concert\"]}" }, { "id": 3, "created": "2025-06-05 11:58:23.978207", "payload": "{\"eventId\":101,\"tags\":[\"sports\",\"indoor\"]}" }, { "id": 1, "created": "2025-06-05 11:58:23.978207", "payload": "{\"eventId\":101,\"tags\":[\"sports\",\"outdoor\",\"fun\"]}" }] ``` *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-apache-nifi/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-apache-nifi/) (An index of the documentation is available at /llms.txt) --- # Connect with Collibra Platform Collibra is a data intelligence platform that helps organizations streamline data management, governance, and collaboration. It enables businesses to discover, understand, and trust their data by providing tools for data cataloging, data lineage, and policy management. Collibra enhances decision-making by ensuring data quality, security, and compliance across various data sources. You can connect your SingleStore deployments to Collibra Platform using the [SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) ("the driver"). The driver enables Catalog to register database information and extract the structure of the source into its schemas, tables, and columns. The driver can retrieve the following SingleStore database components: * Schemas * Tables * Views * Columns * Primary keys ## Prerequisites * Access to an Edge site or a Collibra Cloud site. * Global System administration and Manage connections and capabilities permissions. * An active SingleStore deployment. * SingleStore JDBC driver version 1.2.6+ `.jar` file. ## Configure the Connection to SingleStore To connect using the SingleStore JDBC driver, perform the following tasks: 1. Create a JDBC connection. 2. Register a data source. ## 1. Create a JDBC Connection Refer to [Create a JDBC connection](https://productresources.collibra.com/docs/collibra/latest/Content/Edge/JDBCConnections/ta_create-jdbc-connection.htm?data-source=singlestore) for information on creating a JDBC connection to SingleStore in Collibra Platform. Specify the following connection settings while configuring the JDBC connection: * **Name**: Specify **SingleStore**. * **Driver class name**: Specify `com.singlestore.jdbc.Driver`. * **Driver jar**: Download the SingleStore JDBC driver `.jar` file from its [GitHub repository](https://github.com/memsql/S2-JDBC-Connector/releases/latest), and upload the JAR file. * **Connection String**: Enter `jdbc:singlestore://{host}:{port}/{db}`, where: * `host`: IP address or the hostname of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `port`: Default is `3306`. * `db`: Name of the SingleStore database to connect with. Specify the following connection properties to configure the connection to your SingleStore deployment: * **User**: Username of the SingleStore database user with which to access the database. * **Password**: Password for the SingleStore database user. > **📝 Note**: SingleStore does not support foreign keys. Hence, you must set the `foreign-keys-strategy` property to `SKIP` under Catalog JDBC Ingestion Edge capability. ## 2. Register a Data Source Use the JDBC connection created earlier to register a data source. Refer to [Register a data source via Edge](https://productresources.collibra.com/docs/collibra/latest/Content/Catalog/RegisterDataSource/Edge/ta_register-a-data-source-via-edge.htm) for more information. ## View Technical Lineage You can also view the technical lineage graph for SingleStore to visualize the journey of the data objects in your data sources and data transformation details. Refer to [Technical Lineage](https://productresources.collibra.com/docs/collibra/latest//Content/CollibraDataLineage/co_collibra-data-lineage.htm#technical-lineage) for more information. Here are a few sample queries: * Visualize pipeline metadata for external functions: ```sql SELECT PIPELINE_NAME, CONFIG_JSON, STATE, CREATE_TIME FROM information_schema.PIPELINES WHERE DATABASE_NAME = ''; ``` * Select and parse executed queries from history for internal relations: ```sql SELECT QUERY_TEXT FROM information_schema.MV_QUERIES WHERE ACTIVITY_NAME LIKE 'InsertSelect%'; ``` ## References * [Registering and integrating data sources in Catalog](https://productresources.collibra.com/docs/collibra/latest/Content/Catalog/to_register-integrate.htm) * [Create a JDBC connection to SingleStore](https://productresources.collibra.com/docs/collibra/latest/Content/Edge/JDBCConnections/ta_create-jdbc-connection.htm?data-source=singlestore) *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-collibra-platform/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-collibra-platform/) (An index of the documentation is available at /llms.txt) --- # Connect with Dataloop Dataloop is an end-to-end AI development platform that enables you to integrate data models and applications for building and deploying AI systems (applications). The SingleStore integration in Dataloop streamlines data processing, table updates, and data uploads between SingleStore Helios and Dataloop datasets. This integration allows you to: * Directly run queries from Dataloop in SingleStore using the integrated service. * Automatically create and update tables based on Dataloop dataset information. * Upload SingleStore query results directly to Dataloop datasets. The SingleStore integration in Dataloop uses the [SingleStore Python Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md). ## Prerequisites * An active SingleStore Helios deployment. * Access to the [Dataloop Platform](https://console.dataloop.ai/) (console). * A table in the SingleStore database with at least the following columns: * `id`: Auto-generated field. * `prompt`: Contains the prompt to create in Dataloop. * `response`: Stores the model responses. This field is automatically populated from the Reinforcement Learning Human Feedback (RLHF) pipeline. ## Pipeline Nodes The SingleStore integration in Dataloop provides the following nodes: * Import SingleStore * Export SingleStore ## Import SingleStore The Import SingleStore node retrieves the prompts from a SingleStore table, adds them to a specified dataset in Dataloop, and creates the prompt items accordingly. ## Export SingleStore The Export SingleStore node updates the best marked response, model name, and `id` from Dataloop in the corresponding row in the SingleStore table. ## Connect to SingleStore from Dataloop To connect to SingleStore: 1. [Install the SingleStore Integration](https://docs.singlestore.com/#section-idm234882358149887.md) from Dataloop Marketplace. 2. (Optional) Create a Secret and store the password of the SingleStore database user. The Secret (password) can also be created/specified while configuring the Dataloop pipelines. 3. [Create a Dataloop pipeline to SingleStore](https://docs.singlestore.com/#section-idm234882358414866.md) and specify the SingleStore connection configuration in the Import/Export node configuration. ## Install the SingleStore Integration To install the SingleStore integration in Dataloop, 1. Log in to the [Dataloop Platform](https://console.dataloop.ai/) (console). 2. On the left navigation pane, select **Marketplace**. 3. Under Dataloop Hub, select **Applications**. 4. From the list of applications, select **SingleStore Integration**. 5. Select **Install** to install the SingleStore Integration. 6. On the **Install Application** dialog, select **Proceed**. 7. From the **SingleStore Password** list, select **Add New Secret**. You can also select an existing Secret. To create or specify a Secret later, select **Set Up Later**. 8. On the **Create Secret** dialog, enter the following: 1. **Secret Name**: A name for the Secret. 2. **Secret**: The password for the SingleStore database user used to connect to your SingleStore deployment. 9. Select **Create Secret > Install Application**. The SingleStore Integration is now installed. > **📝 Note**: The SingleStore Integration only supports username/password based authentication. Specify the SingleStore connection configuration in the Import/Export node configuration. ## Create a Pipeline to SingleStore To create a pipeline in Dataloop, 1. On the Dataloop Platform, on the left navigation pane, select **Pipelines > Create Pipeline > Start from Scratch**. 2. On the **Set Pipeline Name** dialog, enter a name for the pipeline and select **Create Pipeline**. 3. From the **Node Library** pane, drag the **Import SingleStore** or **Export SingleStore** node to the canvas. 4. Select the node added to the canvas, and set the following parameters for each node: * **host**: IP address or hostname of the SingleStore Helios deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **user**: Name of the SingleStore database user with which to connect. * **database**: Name of the SingleStore database. * **table\_name**: Name of the table in the SingleStore database. 5. To specify a different access credential, select **Actions > Edit Access Credentials**, and select (or create) a Secret. 6. Configure the pipelines as required, and start the pipelines to start data processing. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-dataloop/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-dataloop/) (An index of the documentation is available at /llms.txt) --- # Connect with Tableau Prep You can connect Tableau Prep to SingleStore using [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) or [the SingleStore ODBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) to perform ETL operations. To connect using the SingleStore ODBC driver, select the **SingleStore** option from the connection list. To connect using the SingleStore JDBC driver, select the **SingleStoreDB JDBC by SingleStore** option from the connection list instead. The following certification matrix shows the latest version(s) tested for SingleStore Helios and Tableau Prep: | Certification Matrix | Versions | | ---------------------- | ------------------- | | Tableau Prep | 2023.1.3, 2023.2.0+ | | SingleStoreODBC Driver | 1.1.7+ | | SingleStoreJDBC Driver | 1.1.4+ | > **📝 Note**: SingleStore recommends using the latest version of Tableau Prep with the latest version of the SingleStore JDBC driver or the SingleStore ODBC driver. ## Prerequisites * Install the SingleStore JDBC driver or the SingleStore ODBC driver. * Create the DSN for source and destination databases. * Install Tableau Prep. **Note**: When using a source or target other than SingleStore or MySQL, you will need to install the respective JDBC/ODBC driver for that platform. ## Write to a Database Although Tableau Prep can read data from a wide array of databases and cloud storage platforms, writing to external databases is available only for a limited set of datasources. SingleStore enables writing back to a database via Tableau Prep, which allows the users to: * Connect to other data sources supported by Tableau Prep or connect to a SingleStore database. * Prepare data, including cleansing, imputation, and complex transformation. * Write changes back to the SingleStore database from the other sources, a different SingleStore database, or a different table in the same database. * Build Tableau dashboards using the data in SingleStore. ## Configure Tableau Prep to Connect to SingleStore and Run a Transform To connect to SingleStore from Tableau Prep: 1. On Tableau Prep, on the left navigation pane, select **Connections > Connect > SingleStore** (or **SingleStoreDB JDBC by SingleStore**). ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt631f42678c4640d9/6a7adb0996b5a6b66986fdc8/tableau_prep1-s1ASfz.png) 2. Enter the connection details in the **SingleStore** dialog and connect to the datasource: * **Server**: IP address or hostname of the SingleStore Helios cluster Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Default is `3306` * **Database** (Optional): The SingleStore Helios database name * **Username** and **Password**: The SingleStore database username and password 3. Select the database and tables, and configure the required transformations. 4. From the **Save output to** list, select **Database table**. 5. Under **Settings**, either select an existing connection or create a new connection to the target database. 6. From the **Write Options** list select one of the following options: **Create table**, **Append to table**, or **Replace data**. 7. Select **Run flow** to run the transform. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-tableau-prep/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-tableau-prep/) (An index of the documentation is available at /llms.txt) --- # Connect with Twilio Segment Twilio Segment enables you to collect, unify, and connect your data from various sources, and stream the data (events) into SingleStore to perform analytics and data processing operations. Refer to [Segment Documentation](http://segment.com/docs/) for more information. ## Prerequisites * An active SingleStore deployment with the Data API enabled. Refer to [Enable the Data API](https://docs.singlestore.com/cloud/reference/data-api/enable-the-data-api.md) for related information. * Configure the **Inbound IP Allowlist** of your SingleStore deployment to allow access from Twilio Segment. Refer to [Firewall](https://docs.singlestore.com/cloud/security/firewall.md) for related information. * A Twilio Segment instance with active sources. ## Connect to SingleStore from Segment To connect your SingleStore database, 1. Configure SingleStore as a destination in Segment for a specific source. 2. Configure field mappings and stream data into SingleStore. 3. (Optional) Verify that events are being streamed. ## Configure the SingleStore Destination To configure SingleStore as a destination in Twilio Segment, 1. Log in to your Segment workspace. 2. On the left navigation pane, select **Connections > Sources**. 3. From the list of Sources, select a source. 4. On the selected source's page, select **Add Destination**. 5. On the **Catalog** page, under **Destinations**, select **SingleStore**. You can also search "SingleStore" and select it from the search results. 6. On the **SingleStore** page, select **Add destination**. 7. Under **Setup**, enter a name for the destination in the **Destination name** field. 8. Select **Create destination**. The SingleStore destination is created. 9. On the **Settings** tab of your destination, configure the connection to your SingleStore database. Enter the following information: 1. **Host**: Hostname or IP address of your SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) for related information. 2. **Port**: Port of the SingleStore Data API. The default is `443`. 3. **Username**: Username of the SingleStore database user with which to connect. 4. **Password**: Password for the SingleStore database user. 5. **Database Name**: Name of the SingleStore database to connect with. 6. **Table Name**: Name of the target table in the specified database. The default is `segment_data`. 10. Select **Save Changes** to save the destination configuration. ## Configure Mappings and Stream Data To configure mappings from source event to destination field and stream data into SingleStore, 1. Select the **Mappings** tab for your destination. 2. Under **Mappings**, select **New Mapping**. 3. Under **Set up mapping**, select **Send Data**. 4. On the same page, under **Map Fields**, configure the source event to destination field mapping. 5. Select **Next**. 6. Enter a name for the mapping in the **Mapping name** field. 7. Select **Save**. The mapping is saved. 8. On the **Settings** tab for your destination, enable the **Enable Destination** toggle. Once the mappings are configured and the destination is enabled, events are streamed to the specified SingleStore table as they are received. ## Verify Event Streaming To verify that events are streamed, 1. Log in to your SingleStore deployment ([Cloud Portal](https://portal.singlestore.com)). 2. Select **Clusters** > **\ > Connect > SQL Editor**. 3. Run the following query(s) to verify that events are streamed to SingleStore: ```sql -- Count the number of events ingested -- -- Update the table name to the specified target table -- SELECT COUNT(*) FROM segment_data; ``` ```sql -- View the 10 most recent ingested events -- -- Update the table name to the specified target table -- SELECT * FROM segment_data ORDER BY timestamp DESC LIMIT 10; ``` You can also connect to your SingleStore deployment from a supported third-party client/application and run the query(s). ## Disable Event Streaming To disable event streaming, 1. Select **\ > Settings**. 2. Disable the **Event Destination** toggle. 3. Select **Save Changes**. *** Modified at: May 22, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-twilio-segment/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-twilio-segment/) (An index of the documentation is available at /llms.txt) --- # Connect with Vercel You can connect any of your Vercel projects with your SingleStore Helios clusters using the [SingleStore Helios - Vercel](https://vercel.com/integrations/singlestoredb-cloud) integration. This integration allows you to build applications using Vercel and SingleStore Helios. To connect your Vercel project: 1. [Set up a Vercel project](https://docs.singlestore.com/#section-idm4647015618432033965095627924.md). 2. [Connect the project to your SingleStore Helios cluster](https://docs.singlestore.com/#section-idm4647015777468833965097811681.md). 3. [Build and deploy your application](https://docs.singlestore.com/#section-idm4539956100372833965113447449.md). ## Prerequisites * A Vercel account. ## Connect Vercel to SingleStore Helios ## Set up a Vercel Project On the Vercel Dashboard, [create a new project](https://vercel.com/docs/projects/overview#creating-a-project). You can also leverage one of Vercel's templates, such as [Next.js Boilerplate](https://vercel.com/templates/next.js/nextjs-boilerplate), which facilitates the development of your web application. ## Connect to your SingleStore Helios Cluster Once your Vercel project is deployed, perform the following tasks: 1. Go to the [SingleStore Helios - Vercel](https://vercel.com/integrations/singlestoredb-cloud) integration page, and select **Add Integration**. 2. On the **Add SingleStoreDB Cloud** dialog, select your Vercel account in which to add the integration, and then select **Continue**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt107e4d249a83e420/6a7adb00366687cf0727e386/vercel_1-2ny8fU.png) 3. Select the project(s) to integrate with your cluster, and then select **Continue**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltec9a263eb7a7f4b8/6a7adb05e88c658bb9fffded/vercel_2-1u3gk2.png) 4. Verify the permissions, and select **Add Integration**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4800becb5836f3cb/6a7adb04437e0ffc0bdcfc52/vercel_3-77ngCo.png) 5. In the dialog that opens, log in or sign up for the Cloud Portal. 6. If you have already created a cluster, select it from the **Cluster** list, and then select **Connect**. If there aren't any active clusters, you can create a new cluster and then select it. Your Vercel project is now connected to your cluster. The project will now have three additional environment variables (go to **\ -> Settings -> Environment Variables**): * `SINGLESTORE_WORKSPACE_HOST` * `SINGLESTORE_WORKSPACE_USERNAME` * `SINGLESTORE_WORKSPACE_PASSWORD` ## Build and Deploy There are multiple options available for building and deploying your application. For example, for a basic deployment, you can use Vercel Functions and [create a serverless function](https://vercel.com/docs/functions/serverless-functions/quickstart#create-a-serverless-function). The following example connects to SingleStore Helios from a serverless function using the [Data API](https://docs.singlestore.com/cloud/reference/data-api.md): ```javascript var SingleStoreClient = require('@singlestore/http-client'); var instance = SingleStoreClient.ApiClient.instance; var BasicAuth = instance.authentications['BasicAuth']; BasicAuth.username = process.env.SINGLESTORE_WORKSPACE_USERNAME; BasicAuth.password = process.env.SINGLESTORE_WORKSPACE_PASSWORD; instance.basePath = 'https://' + process.env.SINGLESTORE_WORKSPACE_HOST; // Get the API handle. var api = new SingleStoreClient.HttpApi(); export default function handler(req, res) { api.rows({ queryInput: { database: 'mydatabase', sql: 'SELECT * FROM purchases LIMIT 200;' } }).then(dbResponse => { res.send(dbResponse.results[0].rows); }).catch(err => { res.send(err); }); } ``` Refer to [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md) for additional connectivity options that suit your application stack. *** Modified at: July 22, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/connect-with-vercel/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/connect-with-vercel/) (An index of the documentation is available at /llms.txt) --- # Load Data from Apache Beam The SingleStore I/O connector allows you to access your SingleStore databases from Apache Beam using [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). You can also export your SingleStore databases to Apache Beam using this connector. For information on Apache Beam concepts, see [Apache Beam Documentation](https://beam.apache.org/documentation/). ## Prerequisites To use SingleStore I/O connector, add the following Maven artifact dependency to your `pom.xml` file, ```xml org.apache.beam beam-sdks-java-io-singlestore {{< param release_latest >}} ``` ## Configure the Data Source You need to create a data source configuration to configure the SingleStore I/O connection. Define a `DataSourceConfiguration` object with the connection information in the following format: ```java SingleStoreIO.DataSourceConfiguration .create(":") .withDatabase("") .withConnectionProperties("") .withPassword("") .withUsername(""); ``` | Parameter | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `.create()`(Required) | Specifies the hostname or IP address of the connection in the``format. Default port is`3306`.Refer to[SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md)to determine the endpoint (`host:port`) of your deployment. | | `.withDatabase()` | Specifies the database name. If this parameter is set, the connection uses the specified database. | | `.withConnectionProperties()` | Specifies a list of[connection string parameters](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#section-idm4492642533297632740896869469.md)for the JDBC driver in the`"=[;=...]"`format. | | `.withUsername()` | Specifies the username of theSingleStoreuser. | | `.withPassword()` | Specifies the password of theSingleStoreuser. | Here's a sample configuration: ```java SingleStoreIO.DataSourceConfiguration .create("svchost:3306") .withDatabase("dbTest") .withConnectionProperties("connectTimeout=30000;useServerPrepStmts=FALSE") .withPassword("passw0rd") .withUsername("s2user"); ``` ## SingleStore I/O Transforms SingleStore supports the following `pTransforms`: * `Read`: Reads data sequentially by executing a single query. * `ReadWithPartitions`: Reads data in parallel. * `Write`: Writes data using `LOAD DATA` queries. All read transforms return bounded `PCollections`. Write transforms can write both bounded and unbounded `PCollections`. ## Read from SingleStore The SingleStore I/O connector supports the following operations to read from tables: * `.read()` for sequential data reading, and * `.readWithPartitions()` for reading data in parallel. ## Sequential Data Reading with `.read()` ```java PCollection items = pipeline.apply( SingleStoreIO.read() .withDataSourceConfiguration() .withTable("") // or .withQuery("") .withStatementPreparator() .withOutputParallelization() .withRowMapper() ); ``` > **📝 Note**: You must specify either the `withTable()` or the `withQuery()` parameter. | Parameter | Description | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.withDataSourceConfiguration()`(Required) | Specifies the`DataSourceConfiguration`object. See[Configure the Data Source](https://docs.singlestore.com/#section-idm4522194824188833503606758322.md)for more information. | | `.withTable("")` | Specifies the table to read data. | | `.withQuery("")` | Specifies the query to execute. | | `.withStatementPreparator()` | Specifies the[`StatementPreparator`](https://docs.singlestore.com/#section-idm456376309617923350368853301.md)object. | | `.withRowMapper()`(Required) | Specifies the[`RowMapper`](https://docs.singlestore.com/#section-idm4625920250897633503690110446.md)object. | | `.withOutputParallelization()` | Indicates if the resulting`PCollection`set is reshuffled. By default, it is set to`true`. | Here's an example: ```java SingleStoreIO.read() .withDataSourceConfiguration( new SingleStoreIO.DataSourceConfiguration.create("myHost:3306") .withUsername("root") .withPassword("secret") .withDatabase("db") ) .withQuery("SELECT a*a FROM t") .withStatementPreparator(statementPreparator) .withOutputParallelism(true) .withRowMapper(rowMapper) ``` ## Parallel Data Reading with `.readWithPartitions()` ```java PCollection items = pipeline.apply( SingleStoreIO.readWithPartitions() .withDataSourceConfiguration() .withTable("") // or .withQuery("") .withRowMapper() ); ``` > **📝 Note**: You must specify either the `withTable()` or the `withQuery()` parameter. Additionally, you must specify the `.withDatabase()` parameter in the `DataSourceConfiguration`. | Parameter | Description | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.withDataSourceConfiguration()`(Required) | Specifies the`DataSourceConfiguration`object. See[Configure the DataSource](https://docs.singlestore.com/#section-idm4522194824188833503606758322.md)for more information. | | `.withTable("")` | Specifies the table to read data. | | `.withQuery("")` | Specifies the query to execute. | | `.withRowMapper()`(Required) | Specifies the[`RowMapper`](https://docs.singlestore.com/#section-idm4625920250897633503690110446.md)function. | Here's an example: ```java SingleStoreIO.readWithPartitions() .withDataSourceConfiguration( new SingleStoreIO.DataSourceConfiguration.create("myHost:3306") .withUsername("root") .withPassword("secret") .withDatabase("db") ) .withTable("t1") .withRowMapper(rowMapper) ``` ## `StatementPreparator` The `StatementPreparator` object implements an interface that is used by `.read()` to set the parameters of the `PreparedStatement`. Here's an example: ```java public static class MyStatmentPreparator implements SingleStoreIO.StatementPreparator { @Override public void setParameters(PreparedStatement preparedStatement) throws Exception { preparedStatement.setInt(1, 10); } } ``` ## `RowMapper` The `RowMapper` function is used to convert each row of the `ResultSet` into an element of the resulting `PCollection`. Here's an example: ```java public static class MyRowMapper implements SingleStoreIO.RowMapper { @Override public MyRow mapRow(ResultSet resultSet) throws Exception { return MyRow.create(resultSet.getInt(1), resultSet.getString(2)); } } ``` ## Write to SingleStore The `.write()` transformation sends your `PCollection` object to your SingleStore databases. It returns the total number of rows written by each batch of elements. Use the following format: ```java data.apply( SingleStoreIO.write() .withDataSourceConfiguration() .withTable("") .withUserDataMapper() .withBatchSize() ); ``` | Parameter | Description | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.withDataSourceConfiguration()`(Required) | Specifies the`DataSourceConfiguration`object. See[Configure the DataSource](https://docs.singlestore.com/#section-idm4522194824188833503606758322.md)for more information. | | `.withTable("")`(Required) | Specifies the table to read data. | | `.withBatchSize()` | Specifies the number of rows loaded by a single`LOAD DATA`query. The default value is`100000`. | | `.withUserDataMapper()`(Required) | Specifies the[`UserDataMapper`](https://docs.singlestore.com/#section-idm4581682772566433503694428191.md)function. | Here's an example: ```java SingleStoreIO.write() .withDataSourceConfiguration( new SingleStoreIO.DataSourceConfiguration.create("myHost:3306") .withUsername("root") .withPassword("secret") .withDatabase("db") ) .withTable("t") .withBatchSize(1000) .withUserDataMapper(userDataMapper) ``` ## `UserDataMapper` This function maps data from `PCollection` objects to an array of Strings, which is eventually converted into a set of CSV values and loaded into SingleStore. Here's an example: ```java public static class MyRowDataMapper implements SingleStoreIO.UserDataMapper { @Override public List mapRow(MyRow element) { List res = new ArrayList<>(); res.add(element.id().toString()); res.add(element.name()); return res; } } ``` ## Example The following example shows how to connect to a SingleStore Helios cluster from Apache Beam using the SingleStore I/O connector and perform read/write operations. Connect to your SingleStore Helios cluster, and run the following commands: ```sql CREATE DATABASE dbExample; USE dbExample; CREATE TABLE readExample ( ID INT, Name VARCHAR(32) ); CREATE TABLE writeExample ( ID INT, Name VARCHAR(32) ); INSERT INTO readExample VALUES (1, 'Adam'), (2, 'Bill'), (3, 'Clark'), (4, 'Dennis'), (5, 'Erik'), (6, 'Finn'), (7, 'Grace'), (8, 'Henry'); ``` Create a Maven project, and add the following dependencies to your **pom.xml** file: ```xml com.s2 s2BeamExample 1.0 org.apache.beam beam-runners-direct-java 2.44.0 runtime com.singlestore singlestore-jdbc-client 1.1.4 org.apache.beam beam-sdks-java-io-singlestore 2.44.0 ``` ## Read Operation Example Add the following Java code to the **S2ReadExample** class. It defines the data source configuration for the SingleStore I/O connection and reads the values from the **readExample** table as key/value pairs. Once the data is loaded in the `PCollection`, it is converted from key/value pairs to strings and written to a `.csv` file. ```java Pipeline pipeline = Pipeline.create(); PCollection < KV < Integer, String >> data = pipeline.apply(SingleStoreIO. < KV < Integer, String >> read() .withDataSourceConfiguration(DataSourceConfiguration .create("svchost") .withUsername("s2user") .withPassword("pa55w0rd") .withDatabase("dbExample")) .withQuery("SELECT * FROM readExample WHERE ID > ?") .withStatementPreparator(new StatementPreparator() { public void setParameters(PreparedStatement preparedStatement) throws Exception { preparedStatement.setInt(1, 4); } }) .withRowMapper(new RowMapper < KV < Integer, String >> () { public KV < Integer, String > mapRow(ResultSet resultSet) throws Exception { return KV.of(resultSet.getInt(1), resultSet.getString(2)); } }) ); data .apply(MapElements .into(TypeDescriptors.strings()) .via((KV < Integer, String > kv) - > kv.getKey() + "," + kv.getValue())) .apply(TextIO .write().to("/path/to/output").withNumShards(1).withSuffix(".csv") ); ``` Run the following commands: ```shell mvn clean compile assembly:single java com.s2.beam.S2ReadExample ``` After the code runs successfully, the resulting CSV file contains the following data: ``` 5,Erik 6,Finn 7,Grace 8,Henry ``` ## Write Operation Example Add the following Java code to the **S2WriteExample** class. It defines the data source configuration for the SingleStore I/O connection, reads the data from the CSV file (created in the example above), and writes it into the **writeExample** table. ```Java Pipeline pipeline = Pipeline.create(); PCollection < String > lines = pipeline.apply( TextIO.read().from("/path/to/output.csv")); PCollection < KV < Integer, String >> keyValues = lines.apply( MapElements.into(TypeDescriptors.kvs(TypeDescriptors.integers(), TypeDescriptors.strings())) .via((String line) - > { String[] fields = line.split(","); return KV.of(Integer.parseInt(fields[0]), fields[1]); }) ); keyValues.apply(SingleStoreIO. < KV < Integer, String >> write() .withDataSourceConfiguration(DataSourceConfiguration .create("svchost ") .withUsername("s2user") .withPassword("pa55w0rd") .withDatabase("dbExample")) .withTable("writeExample") .withUserDataMapper(new UserDataMapper < KV < Integer, String >> () { public List < String > mapRow(KV < Integer, String > element) { List < String > result = new ArrayList < > (); result.add(element.getKey().toString()); result.add(element.getValue()); return result; } }) ); ``` Run the following commands: ```shell mvn clean compile assembly:single java com.s2.beam.S2WriteExample ``` After the code runs successfully, run the following SQL query: ```sql SELECT * FROM writeExample; ``` ```output +----+-------+ | ID | Name | +----+-------+ | 5 | Erik | | 6 | Finn | | 7 | Grace | | 8 | Henry | +----+-------+ ``` The contents of the CSV file are now inserted into the **writeExample** table. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-beam/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-beam/) (An index of the documentation is available at /llms.txt) --- # Load Data from Apache Flink Accelerate your workloads by combining Apache Flink's stream-processing capabilities with SingleStore's high performance ingest. Use the SingleStore JDBC driver ("the driver") to connect to SingleStore Helios from Apache Flink. The driver allows data to be ingested into, and extracted from, SingleStore databases, providing seamless integration with Apache Flink's DataStream and Table APIs. ## Prerequisites * An active SingleStore Helios deployment. * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) installed with Maven. * Java JDK ## Connect to SingleStore from Apache Flink To connect to SingleStore from Apache Flink, 1. Update the `pom.xml` file of your Maven project to include the dependency for the SingleStore JDBC driver. For example: ```XML com.singlestore singlestore-jdbc-client 1.2.4 ``` 2. Define the connection configuration for your SingleStore Helios deployment using a `JdbcConnectionOptions` object in your application code. ```Java JdbcConnectionOptions connectionOptions = new JdbcConnectionOptions.JdbcConnectionOptionsBuilder() .withUrl("jdbc:singlestore://:/") .withDriverName("com.singlestore.jdbc.Driver") .withUsername("") .withPassword("") .build(); ``` Update the following before running the application: * `hostname`: Hostname or IP address of your SingleStore Helios deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `port`: Port of the SingleStore Helios deployment. Default is `3306`. * `database`: Name of the SingleStore database to connect with. * `username`: Name of the SingleStore database user with which to access the database. * `password`: Password for the SingleStore database user. 3. Connect to your SingleStore databases using the `JdbcConnectionOptions` object. ## Example The following example connects to a SingleStore database from Apache Flink using the SingleStore JDBC driver and inserts data. 1. Connect to the target SingleStore deployment and run the following SQL commands. This example inserts data into the `dbTest.Stock` table. ```sql CREATE DATABASE dbTest; USE dbTest; CREATE TABLE Stock ( ID INT, Code VARCHAR(4), Quantity INT ); ``` 2. Create a new Maven project in your local environment. For example: ```shell mvn archetype:generate -DgroupId=SingleStore \ -DartifactId=FlinkSingleStoreConnection \ -DarchetypeArtifactId=maven-archetype-quickstart \ -DinteractiveMode=false ``` 3. Add the following dependency to the `pom.xml` file of the Maven project, including any other dependencies required. ```xml com.singlestore singlestore-jdbc-client 1.2.4 ``` 4. Build the Maven project from the parent directory using the following command. ```shell mvn clean package ``` 5. Add the following code to a `flinkExample.java` file in your Maven project (for example, in the `src/main/java/com/SingleStore` directory) . In this example, the `JdbcSink.sink()` method uses the `JdbcConnectionOptions` object to connect to the SingleStore instance and insert three rows of data. **Note**: Update the connection configuration of your SingleStore deployment in the `JdbcConnectionOptionsBuilder()` constructor. ```java import org.apache.flink.api.java.tuple.Tuple3; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.connector.jdbc.JdbcSink; import org.apache.flink.connector.jdbc.JdbcExecutionOptions; import org.apache.flink.connector.jdbc.JdbcConnectionOptions; public class flinkExample { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStream> stock = env.fromElements( Tuple3.of(101, "CVBN", 40), Tuple3.of(102, "FGTR", 25), Tuple3.of(103, "YTTK", 20) ); JdbcConnectionOptions connectionOptions = new JdbcConnectionOptions.JdbcConnectionOptionsBuilder() .withUrl("jdbc:singlestore:///dbTest") .withDriverName("com.singlestore.jdbc.Driver") .withUsername("") .withPassword("") .build(); stock.addSink(JdbcSink.sink( "INSERT INTO Stock (ID, Code, Quantity) VALUES (?, ?, ?)", // SQL Insert Query (statement, user) -> { statement.setInt(1, user.f0); statement.setString(2, user.f1); statement.setInt(3, user.f2); }, JdbcExecutionOptions.builder() .withBatchSize(1000) .withBatchIntervalMs(200) .build(), connectionOptions )); // Execute Flink job env.execute("Flink Insert Example"); } } ``` 6. Build the Maven project using the following command. This command creates a `.jar` file in the `/target` directory of your project. ```shell mvn clean package ``` 7. From the directory where Flink is extracted, run the `start-cluster.sh` script to start a Flink cluster. This example uses a local Flink cluster. Perform the following tasks: 1. Change to the directory where Flink is extracted. ```shell cd /path_to_flink_directory/flink- ``` 2. Run the following command to start a Flink cluster. ```shell ./bin/start-cluster.sh ``` 8. Use the `.jar` file created in the previous step and submit a Flink job. Run the following command from the directory where Flink is extracted. ```shell ./bin/flink run /path/to/jar_file.jar ``` 9. After the job completes, connect to the SingleStore database and run the following SQL query: ```sql SELECT * FROM Stock ORDER BY ID; ``` ```output +------+------+----------+ | ID | Code | Quantity | +------+------+----------+ | 101 | CVBN | 40 | | 102 | FGTR | 25 | | 102 | YTTK | 20 | +------+------+----------+ ``` The data has been successfully inserted. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-flink/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-apache-flink/) (An index of the documentation is available at /llms.txt) --- # Load Data from AWS Glue AWS Glue is a fully managed serverless data integration service that allows users to extract, transform, and load (ETL) data from various sources for analytics and data processing. AWS Glue runtime supports connectivity to a variety of data sources. SingleStore provides a connector for AWS Glue based on Apache Spark Datasource, available through AWS Marketplace. The connection supports VPC networking and integration with [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) for authentication credentials. The following architecture diagram shows SingleStore connecting with AWS Glue for an ETL job. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1605193beb19f1be/6a7adbeaf6825748a158d9a0/aws_glue_diagram-6OIN8B.png) ## Prerequisites * AWS Glue version 5.0+. * Admin access to the AWS account. * An active SingleStore cluster with a sample dataset. The example uses the `lineitem` table from the `tpch` database. Refer to [Load TPC-H Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-tpc-h-data-into-singlestore.md). Alternatively, you can use another data set stored in your SingleStore database. > **📝 Note**: The SingleStore AWS Glue connector returns an error if used in the following AWS regions: Hong Kong, Sao Paolo, Stockholm, Bahrain, and Cape Town. ## Configuration Overview The following steps connect a SingleStore cluster in an AWS Glue ETL job as the source, transform the data, and store it back on the following: * SingleStore database * Amazon S3 in parquet format To create an ETL job using the SingleStore connector: 1. Store authentication credentials in Secrets Manager. 2. Create an [AWS Identity and Access Management (IAM)](http://aws.amazon.com/iam) role for the AWS Glue ETL job. 3. Configure the SingleStore connector and connection. 4. Create an ETL job using the SingleStore connection in AWS Glue Studio. ## Store Authentication Credentials in Secrets Manager AWS Glue provides integration with AWS Secrets Manager to securely store connection authentication credentials. To create these credentials: 1. Log in to AWS, and open **Secrets Manager** (search Secrets Manager on the AWS console and select it from the results). 2. Select **Store a new secret**. 3. Under **Secret type**, select **Other type of secrets**. 4. Under **Key/value pairs**, set one row for each of the following parameters: * `ddlEndpoint`: IP address or hostname of the SingleStore cluster. * `database`: Name of the SingleStore database to connect with. * `user`: Username of the SingleStore database user with which to connect. * `password`: Password for the SingleStore database user. 5. Select **Next**. 6. In the **Secret name** field, enter a name for the secret, for example, `aws-glue-singlestore-connection-info`. 7. Select **Next**. 8. Disable **Automatic rotation**, and then select the **Next** button. 9. Review the secret configuration and then select **Store**. 10. Copy and securely store the **Secret ARN**. ## Create an IAM Role for the AWS Glue ETL Job To create a role with an attached policy to allow read-only access to credentials that are stored in Secrets Manager for the AWS Glue ETL job: 1. Log in to AWS, and open **IAM** (search IAM in the AWS console and select it from the results). 2. Select **Policies > Create Policy**. 3. Select **JSON**. On the **JSON** tab, enter the following JSON snippet (update the Region and account ID from the secret ARN): ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], "Resource": "arn:aws:secretsmanager:::secret:aws-glue-*" } ] } ``` 4. Select **Next**. 5. Under **Policy name**, enter a name for the policy, for example, **GlueAccessSecretValue**. 6. Select **Create policy**. 7. On the **IAM Dashboard**, select **Roles > Create role**. 8. Under **Trusted entity type**, select **AWS Service**. Under **Use case**, select **Glue**. 9. Select **Next**. 10. Find and select the following AWS managed policies: `AWSGlueServiceRole` and `AmazonEC2ContainerRegistryReadOnly`. 11. Find the policy created earlier `GlueAccessSecretValue`, and select it. 12. Select **Next**. 13. In the **Role name** field, enter a role name, for example, **GlueCustomETLConnectionRole**. 14. Confirm that the three policies are selected, and select **Create role**. ## Configure the SingleStore Connector and Connection To subscribe to the SingleStore connector and configure the connection: 1. On the AWS dashboard, select **AWS Glue Studio** (search Glue Studio on the AWS console and select it from the results). 2. In the left navigation pane, select **Data Connections**. 3. Select **Go to AWS Marketplace**. On the **AWS Marketplace**, search and select **SingleStore connector for AWS Glue**. 4. Select **View purchase options > Subscribe**. 5. Once the subscription request is successful, select **View subscription > SingleStore connector for AWS Glue > Usage instructions**. 6. On the **Usage Instructions** dialog, open the **Activate the Glue connector using AWS Glue Studio** link to activate the connector. 7. On the **AWS Glue Studio** console, select **Data connections** in the left navigation pane. 8. In the **Connections** section, select **Create connection**. 9. In the **Create connection** dialog, under **Connection properties**, enter a name for the connection in the **Name** field, for example, `SingleStore_connection`. 10. From the **AWS Secret** list, select the AWS secret value `aws-glue-singlestore-connection-info` created earlier. 11. Select **Create connection and activate connector**. ## Create an ETL Job using the SingleStore Connection in AWS Glue Studio The example uses the `lineitem` table from the `tpch` database. Refer to [Load TPC-H Data into SingleStore](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/next-steps-and-examples/sample-data/load-tpc-h-data-into-singlestore.md). Alternatively, you can use another data set stored in your SingleStore database. After authentication is set up and the SingleStore connector is configured, create an ETL job using the connection: 1. In the **AWS Glue Studio** console, select **Data Connection** in the left navigation pane. 2. In the **Connections** dialog, select **SingleStore\_connection** (the connection created earlier). Select **Create job**. 3. On the **Job details** tab, in the **Name** field, enter a name for the job, for example, **SingleStore\_transform\_job**. 4. In the **Description** field, enter a description, for example, **Glue job to transform tpch data from SingleStore Helios**. 5. From the **IAM Role** list, select **GlueCustomETLConnectionRole**. 6. From the **Glue version** list, select a Glue version. 7. Use default settings for other properties. Select **Save**. 8. On the **Visual** tab, in the workspace area, select the **SingleStore** connection. On the **Data source properties – Connector** tab, expand **Connection options**, and select **Add new option**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt80df0287b42cb2ab/6a7adba74c4bfb3055cc1f51/glue_select_data_source-0YjtHz.png) 9. Under **Connection options**, in the **Key** field, enter **dbtable**. In the **Value** field, enter `lineitem`. 10. On the **Output schema** tab, select **Edit**. Select the three dots, and select **Add root key** from the list. > **📝 Note**: In this example, AWS Glue Studio is using information stored in the connection to access the data source instead of retrieving metadata information from a Data Catalog table. Hence, you must provide the schema metadata for the data source. Use the schema editor to update the source schema. For instructions on how to use the schema editor, refer to [Editing the schema in a custom transform node](https://docs.aws.amazon.com/glue/latest/ug/edit-jobs-transforms.html). 11. Add the key and data type, which represent the name of the column in the database and its data type, respectively. Select **Apply**. 12. On the **Visual** tab workspace area, select **Add nodes**, and then select **DropFields** from the list. 13. On the **Transform** tab (for Drop Fields), from the **Node parents** list, select **SingleStore connector for AWS Glue**. 14. Select the fields to drop. 15. On the **Visual** tab workspace area, select **Add node**, and then select **Custom Transform** from the list. 16. From the **Node parents** list, select **Drop Fields**. 17. Add the following script to **Code block**: ```python def MyTransform (glueContext, dfc) -> DynamicFrameCollection: from pyspark.sql.functions import col df = dfc.select(list(dfc.keys())[0]).toDF().limit(100) df1 = df.withColumn("disc_price",(col("l_extendedprice")*(1-col("l_discount"))).cast("decimal(10,2)")) df2 = df1.withColumn("price", (col("disc_price")*(1+col("l_tax"))).cast("decimal(10,2)")) dyf = DynamicFrame.fromDF(df2, glueContext, "updated_lineitem") glueContext.write_dynamic_frame.from_options(frame = dyf, connection_type = "marketplace.spark", connection_options = {"dbtable":"updated_lineitem","connectionName":"SingleStore_connection"}) return(DynamicFrameCollection({"CustomTransform0": dyf}, glueContext)) ``` This example calculates two additional columns, `disc_price` and `price`. It uses `glueContext.write_dynamic_frame` to write the updated data back to SingleStore using the connection **SingleStore\_connection** created earlier. > **📝 Note**: You can specify any of the SingleStore Spark connector [configuration settings](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/configuration-settings.md) in `connection_options`. For example, the following statement (from the script) is extended to specify a shard key:``` > glueContext.write_dynamic_frame.from_options(frame = dyf, > connection_type = "marketplace.spark", > connection_options = {"dbtable":"updated_lineitem","connectionName":"SingleStore_connection", "tableKey.shard" : "l_partkey"}) > ``` 18. On the **Output schema** tab, select **Edit**. Add the additional columns `price` and `disc_price` with the `decimal` data type. Select **Apply**. 19. On the **Visual** tab, in the workspace area, select **Add nodes > Select from Collection**. 20. On the **Transform** tab, from the **Node parents** list, select **Custom transform**. 21. Select **Add nodes**. From the **Target** list, select **Amazon S3**. 22. From the **Node parents** list, select **Select from Collection**. 23. On the **Data target properties - S3 tab**, from the **Format** list, select **Parquet**. 24. In the S3 Target Location field, enter `s3://aws-glue-assets-{Your Account ID as a 12-digit number}-{AWS region}/output/` or select from the list. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltf4a36b93f3154ade/6a7adadad402d90311eadcf6/glue_s3_target_location-Kc2cbU.png) 25. Select **Save > Run**. The transformed data is now stored in Amazon S3 and the SingleStore database. *** Modified at: June 15, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-aws-glue/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-aws-glue/) (An index of the documentation is available at /llms.txt) --- # Load Data from Azure Data Factory (ADF) SingleStore can be configured as a “Linked service” in Azure Data Factory (ADF), which allows it to be used as a source or sink for ETL data flows. ## Prerequisites This integration supports SingleStore Helios version 7.1 and later. You will also need an Azure account with access to ADF. ## Create an ADF Factory and Add a SingleStore Datasource 1. Open the ADF main page. In the search bar, enter "Data factories", and then select "Data factories" from the drop down list. You will be taken to the Data factories page. 2. Click **Create new factory** on the Data factories page. 3. On the **Basics** tab on the Create Data Factory page, click **Create new** and create a new resource group. 4. Click **Next: Git configuration** to move to the **Git configuration** tab. 5. Configure your Git repo or click **Configure Git later**. 6. Continue through the tabs (you may need to wait for validation) until you get to the **Review + create** tab. Review your settings and click **Create**. 7. Once your data factory is deployed, click **Go to resource**. 8. In your Data factory window, click **Author & Monitor** (near the center of the page). 9. Open the **Manage** tab, click **Linked services** on the left, and then click **+ New**. 10. In the "New linked service" window, search for and select "Azure Database for MySQL". 11. Enter your database connection info and credentials, then click **Test connection**. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. 12. If your connection is successful, click **Create**. You’ve created a connection to SingleStore, which can be used in ADF activities as a data source or data sink. *** Modified at: September 27, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-data-factory-adf/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-data-factory-adf/) (An index of the documentation is available at /llms.txt) --- # Load Data from Azure Event Hubs SingleStore can be configured as an event hub in Azure Event Hubs. ## Prerequisites You need an Azure account with access to Event Hubs. You will also need to [create an Event Hubs namespace](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-create), which gives you a unique [FQDN](https://en.wikipedia.org/wiki/Fully_qualified_domain_name). ## Get Connection String from Azure Portal After creating the namespace, you’ll need a connection string to communicate with Event Hubs. 1. On the Azure portal, in the left navigation menu, select **All services**. 2. In the **Analytics** section, select **Event Hubs**. 3. From the list of event hubs, select the SingleStore event hub. 4. On the **Event Hubs Namespace** page, in the left navigation menu, select **Shared Access Policies**. 5. From the list of policies, select the relevant shared access policy. 6. In the selected policy dialog, copy the **Connection string-primary key**. Here is a sample pipeline: ```sql CREATE PIPELINE `data_ingest` AS LOAD DATA KAFKA 'front2back2eventhub.servicebus.windows.net:9093/table-update' CONFIG '{"security.protocol": "sasl_ssl","sasl.mechanism": "PLAIN","sasl.username": "$ConnectionString"}' CREDENTIALS '{"sasl.password": ""}' BATCH_INTERVAL 2500 ENABLE OUT_OF_ORDER OPTIMIZATION INTO TABLE `test`; ``` *** Modified at: July 8, 2024 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-event-hubs/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-azure-event-hubs/) (An index of the documentation is available at /llms.txt) --- # Load Data from Data Virtuality Pipes Data Virtuality Pipes allows you to load data from your favorite data sources directly into SingleStore Helios. Verify that Data Virtuality has the correct certificates and you are logged into your Data Virtuality Pipes account. Then, follow these steps to get started: 1. On the [Pipes](https://eu.pipes.datavirtuality.com/) portal, on the **Source** tab, select **+ New Source > SingleStore**. 2. On the [Cloud Portal](http://portal.singlestore.com), select **Clusters**. Select the three dots under the **Actions** column for your cluster, and then select **Access & Security** from the list. Under **IP Allowlist**, select **Allow access from anywhere**. If you select **Only allow access from specified addresses**, you need to add the Data Virtuality Pipes IP address to the allowlist. > **📝 Note**: You can find the Data Virtuality Pipes IP address on the SingleStore source screen, in the **Help** pane, under **Firewall Configuration**.![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt506f228069942c09/6a7adb2342a1173153950937/pipes_ip-BYiUle.png) 3. In the **SingleStore** source screen (on the Pipes portal), enter the following connection details: **Host** (without port), **User**, **Password**, and **Database** in the respective boxes. To use SSL, enable **SSL** on the **Advanced** tab (optional). Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. Select the **Connect** button to finalize the connection. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5a986a03e0e8f920/6a7adafac5d19302ab5db8dd/pipes_source_details-UtVyDk.png) Once the SingleStore cluster is successfully connected as a Data Virtuality Pipes storage source, you can create Jobs and manage data using Pipes. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb2b5e210a32deae5/6a7adb0cfc63abbe79641831/pipes_singlestore_source-T4JP3C.png) Refer to [Data Virtuality Resources](http://datavirtuality.com/pipes/) for more information. *** Modified at: May 12, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-data-virtuality-pipes/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-data-virtuality-pipes/) (An index of the documentation is available at /llms.txt) --- # Load Data from Estuary Flow Integrate your SingleStore databases with Estuary Flow to build and manage real-time ETL and ELT data pipelines that capture and replicate database events to downstream applications. You can configure SingleStore as a **destination** in Estuary Flow. Ingest data from Estuary Flow into SingleStore using [Materializations](https://docs.estuary.dev/concepts/materialization/). ## Configure the SingleStore Destination Use the SingleStore variant of the Kafka-compatible Dekaf connector to configure SingleStore as a destination in Estuary Flow. ## Prerequisites * An Estuary Flow collection * An active SingleStore deployment ## Connect to SingleStore Helios from Estuary Flow To configure SingleStore as a destination: 1. Log in to [Estuary Flow](https://dashboard.estuary.dev/). 2. Create a new materialization. 1. On the **Dashboard**, select **Destinations** from the left navigation pane. 2. On the **Destinations** page, select **NEW MATERIALIZATION**. 3. On the **Create Materialization** page, from the list of connectors, search and select the **SingleStore** connector. 4. Enter or select the following information: 1. **Name**: Enter a name for the materialization. 2. **Data Plane**: Select a data plane to use. 3. **Auth Token**: Specify an authentication token. This token is used to authenticate Kafka consumers to this materialization task. 5. Link a capture (collection) to this task. 6. Select **NEXT > TEST** to test the materialization. 7. Select **SAVE AND PUBLISH** to publish this task. 3. Log in to your SingleStore deployment. 4. Create a table to store the ingested data. For example: ```sql CREATE TABLE estuaryFlow (id INT, created DATETIME(6), product_ids JSON); ``` 5. Create a pipeline to ingest the data from Estuary Flow. For example: ```sql CREATE PIPELINE estuaryIngest AS LOAD DATA KAFKA "dekaf.estuary-data.com:9092/" CONFIG '{ "security.protocol":"SASL_SSL", "sasl.mechanism":"PLAIN", "sasl.username":"{}", "broker.address.family": "v4", "schema.registry.username": "{}", "fetch.wait.max.ms": "2000" }' CREDENTIALS '{ "sasl.password": "", "schema.registry.password": "" }' INTO TABLE estuaryFlow FORMAT AVRO SCHEMA REGISTRY 'https://dekaf.estuary-data.com' ( id <- id, created <- created, product_ids <- product_ids ); ``` where, * `collection_name`: Name of the collection linked to the materialization task. * `materialization_name`: Name of the Estuary Flow materialization. * `auth_token`: Authentication token specified for the materialization. 6. Start the pipeline. For example: ```sql START PIPELINE estuaryIngest; ``` The pipeline starts ingesting data from the Estuary Flow collection to the SingleStore table. Run the `SHOW PIPELINES` command to view the status of the pipeline. Refer to [SingleStore | Estuary Flow](https://docs.estuary.dev/reference/Connectors/materialization-connectors/Dekaf/singlestore/) for more information, for example, configuration properties. *** Modified at: October 10, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-estuary-flow/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-estuary-flow/) (An index of the documentation is available at /llms.txt) --- # Load Data from HVR [HVR](https://fivetran.com/docs/hvr5/introduction) enables data replication across multiple sources and targets. It uses various change data capture (CDC) methods to identify changes to data sources and replicate them to various targets. Supported sources and targets include databases and file systems. HVR can be used with SingleStore Helios for data transfer and integration, where SingleStore serves as either a source or a target. The following certification matrix shows the supported version of HVR with SingleStore: | SingleStore Helios | HVR | | ------------------ | ----- | | 7.1 and later | 5.7.5 | ## Supported Capabilities The following table specifies the [capabilities](https://fivetran.com/docs/hvr5/capabilities) supported by HVR for SingleStore location: | Capability | Supported | | -------------------------- | --------- | | Integrate | ✔ | | Refresh and Compare | ✔ | | Capture | | | Hub Database | | | Bi-directional Replication | | Refer to [Capabilities for SingleStore](https://fivetran.com/docs/hvr5/capabilities/capabilities-for-singlestore) for more information. ## References * For requirements, access privileges, and other features of HVR when using SingleStore for replication, refer to [Requirements for SingleStore](https://fivetran.com/docs/hvr5/location-class-requirements/requirements-for-singlestore). * For information about compatibility and supported versions of SingleStore with HVR platforms, refer to [Platform Compatibility Matrix](https://fivetran.com/docs/hvr5/introduction/platform-compatibility-matrix). *** Modified at: August 27, 2024 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-hvr/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-hvr/) (An index of the documentation is available at /llms.txt) --- # Load Data from Informatica PowerCenter You can connect to your SingleStore Helios cluster from Informatica PowerCenter to perform analytics and ETL operations using [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md). ## Prerequisites * An active SingleStore Helios cluster * An Informatica PowerCenter license ## Connect to SingleStore Helios from PowerCenter Perform the following tasks to connect to your SingleStore Helios database from PowerCenter: 1. Download and install Informatica PowerCenter. Refer to [Informatica Documentation](https://docs.informatica.com/data-integration/powercenter/10-5-3.html) for more information. 2. [Download and install](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) the SingleStore ODBC driver. SingleStore recommends using the ANSI version of the driver. > **📝 Note**: Informatica PowerCenter only supports 32-bit data sources on Windows. Hence, use the 32-bit version of the SingleStore ODBC driver to connect to your SingleStore database from PowerCenter on Windows. 1. Place the **libssodbca.so** and **libssodbcw\.so** files in the **\/\/ODBC7.1/lib** directory. 2. Update the path to the **libssodbca.so** and **libssodbcw\.so** files in the **odbc.ini** file used by Informatica PowerCenter. 3. Configure SingleStore Helios as an ODBC data source in Informatica PowerCenter. Refer to the [Connecting to an ODBC Data Source](https://docs.informatica.com/data-integration/powercenter/10-5/installation-for-powercenter-and-data-quality/appendix-b--connecting-to-databases-from-unix-or-linux/connecting-to-an-odbc-data-source.html) for more information. *** Modified at: July 2, 2024 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-informatica-powercenter/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-informatica-powercenter/) (An index of the documentation is available at /llms.txt) --- # Load Data from Qlik Replicate Qlik Replicate allows its users to "accelerate data replication, ingestion and streaming across a wide variety of heterogeneous databases, data warehouses, and Big Data platforms". This document provides a brief overview of integrating Qlik Replicate with SingleStore. For purposes of illustration, MySQL is used as a source database, and SingleStore Helios is the target for replication. For detailed instructions on how to configure Qlik Replicate for your particular use case, please refer to the user guide provided by Qlik or visit their [online help center](https://help.qlik.com/en-US/replicate/November2020/Content/Replicate/Main/Introduction/Home.htm). Note that SingleStore can be used as a target database but not as a source. The certification matrix below shows the latest versions of the database, driver, and connector verified to work together in a replication task: | Product | Versions | | ---------------------- | ------------- | | SingleStore Helios | 7.1 and newer | | `mysql-connector-odbc` | 5.3.11-1.el7 | | `unixODBC` | 2.3.1-14.el7 | ## In this section * [Using Qlik Replicate](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate/using-qlik-replicate.md) *** Modified at: September 27, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-qlik-replicate/) (An index of the documentation is available at /llms.txt) --- # Load Data from Spark You can use SingleStore and Spark together to accelerate workloads by taking advantage of computational power of Spark in tandem with the fast ingest and persistent storage SingleStore has to offer. The SingleStore Spark Connector allows you to connect your Spark and SingleStore environments. The connector supports both data loading and extraction from database tables and Spark DataFrames. The connector is implemented as a native Spark SQL plugin, and supports Spark’s [DataSource](https://spark.apache.org/docs/latest/sql-data-sources.html) API. Spark SQL supports operating on a variety of data sources through the DataFrame interface, and the DataFrame API is the widely used framework for how Spark interacts with other systems. In addition, the connector is a true Spark data source; it integrates with the Catalyst query optimizer, supports robust SQL pushdown, and leverages SingleStore `LOAD DATA` to accelerate ingest from Spark via compression. ## In this section * [Getting Started with the SingleStore Spark Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/getting-started.md) * [Installation](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/installation.md) * [Configuration Settings](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/configuration-settings.md) * [Examples](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/examples.md) * [Supported Operations](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/supported-operations.md) * [SQL Pushdown](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/sql-pushdown.md) * [Parallel Read Support](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/parallel-read-support.md) * [Data Type Conversions](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/data-type-conversions.md) * [Debugging](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/debugging.md) * [Security and Permissions](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/security-and-permissions.md) *** Modified at: April 14, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-spark/) (An index of the documentation is available at /llms.txt) --- # Load Data from StreamSets **Streamsets**: Streamsets is an open-source, desktop platform for hybrid cloud that enables data movement architectures at scale. For details, see the [StreamSets website](https://streamsets.com/). **SingleStore Helios Fast Loader**: The SingleStore Helios Fast Loader destination uses a JDBC connection to insert data into a SingleStore Helios or MySQL database table with a LOAD statement. In pipelines that insert data into SingleStore Helios or MySQL, you can use the SingleStore Helios Fast Loader destination rather than the JDBC Producer destination for faster performance. This topic describes how to connect StreamSet to SingleStore Helios using SingleStore Helios Fast Loader. The certification matrix below shows the latest versions tested for SingleStore Helios and StreamSets: | Certification Matrix | Versions | | ------------------------- | -------- | | StreamSets Data Collector | 3.13.0 | ## In this section * [Prerequisites](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/prerequisites.md) * [Connecting StreamSets to SingleStore Helios via Fast Loader](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/connecting-streamsets-to-singlestore-helios-via-fast-loader.md) * [SingleStore Helios Fast Loader vs the JDBC Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/singlestore-helios-fast-loader-vs-the-jdbc-connector.md) * [Error and Logs](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/error-and-logs.md) * [Frequently Asked Questions](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/frequently-asked-questions.md) *** Modified at: September 27, 2023 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-from-streamsets/) (An index of the documentation is available at /llms.txt) --- # Load Data with Fivetran Data can be ingested into SingleStore from multiple sources using Fivetran to power data applications. Fivetran's built-in connectors automatically synchronize any changes to the schema and API, ensuring consistent and reliable access to data. Refer to [Implementation & Setup Documentation](https://fivetran.com/docs) for related information. ## Prerequisites * An active SingleStore Helios deployment. * A Fivetran account with Create Destinations or Manage Destinations permissions. Refer to [Fivetran documentation](https://fivetran.com/docs/using-fivetran/fivetran-dashboard/account-settings/role-based-access-control#rbacpermissions) for more information. ## Configure SingleStore * Allow incoming connections from [Fivetran IP Addresses](https://fivetran.com/docs/using-fivetran/ips). * Grant the following privileges to the SingleStore database user: ```sql GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER ON .* to ; ``` **Note**: Additionally, grant the `CREATE DATABASE` privilege if a database is not specified in the connection configuration in Fivetran. ## Connect to SingleStore from Fivetran To connect to SingleStore from Fivetran, 1. On the [Fivetran Dashboard](https://fivetran.com/dashboard/), select **Destinations > + Add Destination**. 2. In the **Destination Name** field, enter a name for the Destination, and then select **Add**. 3. From the list of destinations, select **SingleStore**. 4. On the **SingleStore** page, select or enter the following connection configuration information: * **Host**: Hostname or IP address of the SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Port of your SingleStore deployment. Default is `3306`. * **Database** (Optional): Name of the SingleStore database to connect with. If specified, all of the tables are created in this database. * **Username**: Username of the SingleStore database user. * **Password** (Optional): Password of the SingleStore database user. * **Driver parameter** (Optional): Additional connection string parameters supported by [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) in the `'parameter1=value1; parameter2=value2;...'` format. * **Data processing location**: Location where Fivetran operates and runs data computations. * **Cloud service parameter**: Cloud service provider used for running Fivetran services. * **Timezone**: Time zone for Fivetran sync schedule. 5. Select **Save > Test**. Fivetran connects to SingleStore and validates the specified connection configuration. Once the connection is successful, Fivetran automatically configures a [Fivetran Platform Connector](https://fivetran.com/docs/logs/fivetran-platform) that transfers the connector logs and account metadata to a table in the SingleStore destination. The Fivetran Platform Connector allows you to audit changes and monitor your connectors and usage. The Fivetran Platform Connector sends this information at the destination level. ## Data Type Mapping While ingesting data from Fivetran, SingleStore maps the Fivetran data types to SingleStore data types. Any unsupported data type is automatically cast to the closest supported SingleStore data type. The following table maps the Fivetran data types to the corresponding SingleStore data types: | Fivetran Data Type | SingleStoreData Type | | ------------------ | -------------------- | | BOOLEAN | BOOLEAN | | SHORT | SMALLINT | | INT | INT | | LONG | BIGINT | | BIGDECIMAL | DECIMAL | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | LOCALDATE | DATE | | LOCALDATETIME | DATETIME(6) | | INSTANT | DATETIME(6) | | STRING | TEXT | | XML | TEXT | | JSON | JSON | | BINARY | BLOB | ## Schema Changes The following table lists the type of supported and unsupported schema changes by the SingleStore destination in Fivetran. | Change Type | Supported | Description | | ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Add column | ✔ | If a column is added to the data source, Fivetran adds that column in theSingleStoredestination. | | Change column type | ✔ | If the data type of a column in the data source is changed, Fivetran automatically changes the data type of that column in theSingleStoredestination.To change the datatype of a column inSingleStore, Fivetran performs the following tasks:
  1. Creates a new column with the updated data type.
  2. Copies the data from the existing column to the new column.
  3. Deletes the existing column.
  4. Renames the new column.
| | Change key | | SingleStoredoes not support changing the`PRIMARY KEY`column. | | Change key column type | | SingleStoredoes not support changing the data type of the`PRIMARY KEY`column. | ## Limitations Fivetran does not support [history mode](https://fivetran.com/docs/core-concepts/sync-modes/history-mode) for SingleStore destinations. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-with-fivetran/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-fivetran/) (An index of the documentation is available at /llms.txt) --- # Load Data with Liquibase Liquibase provides a database schema management solution that allows you to add version control to your SingleStore Helios databases, manage revisions of your database, and deploy scripts. You can connect your SingleStore Helios databases to Liquibase using [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). Refer to [Supported features](https://www.liquibase.com/databases/singlestoredb) for a list of Liquibase features that SingleStore supports. ## Prerequisites * A SingleStore Helios cluster running SingleStore version 8.1 or later. * [Download](https://github.com/memsql/S2-JDBC-Connector/releases/latest) the SingleStore JDBC driver `.jar` file. * [Download](https://github.com/liquibase/liquibase-singlestore/releases/latest) the Liquibase extension for SingleStore. * [Install Liquibase](https://docs.liquibase.com/). ## Connect with Liquibase from SingleStore Helios To connect to your SingleStore Helios database with Liquibase, 1. Run the following command to initialize a new Liquibase project: ```shell liquibase init project ``` You may use the default properties or customize the available parameters while initializing a new project. In this guide, we'll use **liquibase.properties** as the defaults file. 2. Place the downloaded SingleStore JDBC driver and Liquibase extension for SingleStore `.jar` files in the **liquibase/lib** directory. If you are using Maven, add the following dependencies to your **pom.xml** file: ```xml com.singlestore singlestore-jdbc-client 1.1.5 org.liquibase.ext liquibase-singlestore 1.0.0 ``` 3. Customize the following parameters in the **liquibase.properties** file: * **url**: Specify the connection string for your SingleStore Helios database in the following format: `jdbc:singlestore://:/`. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **username**: Specify the username of the SingleStore Helios database user that is used to access the database. * **password**: Specify the password for the SingleStore Helios database user. * **driver**: Specify the JDBC Driver Class as `com.singlestore.jdbc.Driver`. * **classpath**: Specify the path to the downloaded SingleStore JDBC driver `.jar` file, including the filename and the **.jar** extension. Here's a sample **liquibase.properties** file: ``` url:jdbc:singlestore://svc-XXXX.svc.singlestore.com:3306/dbTest username:admin password:pa55w0rd liquibaseProLicenseKey: licensekey liquibase.hub.APIKey: APIKey driver:com.singlestore.jdbc.Driver classpath=~/singlestore-jdbc-client-1.1.5.jar ``` 4. Update the changelog file in your project, and add a changeset. In this guide we'll use the following **changelog.sql** file: ```sql -- liquibase formatted sql -- changeset admin:1 CREATE TABLE Stock (ID INT, Code VARCHAR(4), PRIMARY KEY(ID)) ``` 5. Run the following command from the Liquibase installation directory: ```shell liquibase status --changelog-file=changelog.sql ``` ```output -- Liquibase command 'status' was executed successfully. ``` For a successful connection, the output contains a similar message. Refer to [Using Liquibase with SingleStore](https://contribute.liquibase.com/extensions-integrations/directory/database-tutorials/singlestore/) for more information. *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/load-data-with-liquibase/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/load-data-with-liquibase/) (An index of the documentation is available at /llms.txt) --- # Migrate Schema with Flyway You can integrate Flyway into your SingleStore Helios cluster to manage schema changes in your database using the natively integrated SingleStore driver. You can also use Flyway to migrate schemas. Native SingleStore support is only available in the Teams Edition of Flyway. Integration with SingleStore is supported in Flyway version 9.8.2+. ## Prerequisites * An active SingleStore Helios cluster. * Install the Teams Edition of [Flyway command-line](https://flywaydb.org/documentation/usage/commandline/). ## Connect Flyway to SingleStore Helios To integrate Flyway into your SingleStore Helios cluster, you need to update the connection parameters for your cluster and the Flyway license key in the Flyway configuration file (**flyway.conf**). This file is located in the **/\/conf/** directory. Here's a sample configuration: ``` flyway.url=jdbc:singlestore://svchost:3306/dbTest flyway.user=admin flyway.password=pa55w0rd! flyway.schemas=dbTest flyway.licenseKey= ``` Refer to [connection string parameters](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#section-idm4492642533297632740896869469.md) for more information on the supported parameters. ## Example Flyway migrations are defined in `.sql` files located in the **/\/sql/** directory. Flyway runs the SQL statements defined in these `.sql` files using the `migrate` command. The filename must comply with the [naming conventions](https://flywaydb.org/documentation/concepts/migrations.html#naming). The following example shows how to perform a SQL-based migration using Flyway. 1. Create a `.sql` file in the **/\/sql/** directory, say **V1\_Create\_table\_example.sql**, and add the following SQL statement: ```sql CREATE TABLE example ( ID INT NOT NULL, Name VARCHAR(100) NOT NULL ); ``` 2. Run the `migrate` command in the **/\** directory to apply the changes: ```shell flyway migrate ``` ```output Successfully validated 1 migration (execution time 00:00.020s) Creating Schema History table `dbTest`.`flyway_schema_history` ... Current version of schema `dbTest`: << Empty Schema >> Migrating schema `dbTest` to version "1 - Create table example" ``` The database is now migrated to version **1**. 3. Create another `.sql` file, say **V2\_Add\_column.sql**, and add the following SQL statement: ```sql ALTER TABLE example ADD COLUMN Trn DATE; ``` 4. Run the `migrate` command: ```shell flyway migrate ``` ```output Successfully validated 2 migrations (execution time 00:00.023s) Current version of schema `dbTest`: 1 Migrating schema `dbTest` to version "2 - Add column" Successfully applied 1 migration to schema `dbTest`, now at version v2 ``` This command validates and runs both the migrations, and the database is now migrated to version **2**. ## References * [Flyway Documentation](https://flywaydb.org/documentation/) * [Flyway Migrations](https://flywaydb.org/documentation/concepts/migrations.html) *** Modified at: September 26, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/migrate-schema-with-flyway/) (An index of the documentation is available at /llms.txt) --- # Load Data using the SingleStore Kafka Sink Connector The SingleStore Kafka Sink Connector is a Kafka Connect connector that enables you to ingest AVRO, JSON, and CSV messages from Kafka topics into SingleStore Helios. It is a Sink (target) connector designed to read data from Kafka topics and write the data to SingleStore Helios databases. For related information, refer to the following topics: | [How the SingleStore Kafka Sink Connector Works](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/how-the-singlestore-kafka-sink-connector-works.md) | Understand how the connector streams data, processes records, creates tables inSingleStore, etc. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | [Getting Started with Apache Kafka (Self-Managed)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-apache-kafka-self-managed.md) | Install and use the Java-based connector with open-source Apache Kafka. | | [Getting Started with Confluent Cloud (Custom Connect)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-cloud-custom-connect.md) | Use theSingleStoreKafka Sink Connector as a custom connector on Confluent Cloud. | | [Getting Started with Confluent Cloud (Self-Managed)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-cloud-self-managed.md) | Use the connector on Confluent Cloud by deploying it on a self-managed Kafka Connect cluster. | | [Getting Started with Confluent Platform](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-platform.md) | Use the connector on Confluent Platform. | | [Data Mapping](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/data-mapping.md) | Understand how the connector maps records from each Kafka topic to aSingleStoretable. | | [Data Type Mapping](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/data-type-mapping.md) | Data type mapping between Kafka Connect schema types andSingleStoredata types. | | [SingleStore Kafka Sink Connector Properties](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/singlestore-kafka-sink-connector-properties.md) | Configuration properties supported by the connector. | | [Configure mTLS for SingleStore Kafka Sink Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/configure-mtls-for-singlestore-kafka-sink-connector.md) | Configure a mutual TLS (mTLS) connection for the connector. | ## In this section * [Getting Started with Apache Kafka (Self-Managed)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-apache-kafka-self-managed.md) * [Getting Started with Confluent Cloud (Custom Connect)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-cloud-custom-connect.md) * [Getting Started with Confluent Cloud (Self-Managed)](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-cloud-self-managed.md) * [Getting Started with Confluent Platform](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/getting-started-with-confluent-platform.md) * [How the SingleStore Kafka Sink Connector Works](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/how-the-singlestore-kafka-sink-connector-works.md) * [Data Mapping](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/data-mapping.md) * [Data Type Mapping](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/data-type-mapping.md) * [SingleStore Kafka Sink Connector Properties](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/singlestore-kafka-sink-connector-properties.md) * [Configure mTLS for SingleStore Kafka Sink Connector](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/configure-mtls-for-singlestore-kafka-sink-connector.md) *** Modified at: April 24, 2026 Source: [/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/singlestore-kafka-sink-connector/) (An index of the documentation is available at /llms.txt) --- # Transform Data with dbt The `dbt-singlestore` adapter can be used to connect with your SingleStore Helios database to build data transformation pipelines using dbt. dbt provides a development environment to create transformation workflows on data that is already in SingleStore Helios, which dbt turns into tables and views through `SELECT` statements. See [dbt documentation](https://docs.getdbt.com/docs/introduction) for more information. ## Prerequisites Refer to the [dbt-singlestore](https://github.com/memsql/dbt-singlestore?tab=readme-ov-file#testing-and-supported-versions) GitHub repository for the compatibility matrix between SingleStore and dbt. ## Download and Install the SingleStore dbt Adapter You can [download](https://pypi.org/project/dbt-singlestore/#files) the `dbt-singlestore` adapter from the Python Package Index repository (PyPI). Run the following command to install the adapter available on PyPI: ```shell pip install dbt-singlestore ``` You can also install the package from GitHub: ```shell pip install git+https://github.com/memsql/dbt-singlestore.git ``` ## Configure the SingleStore Profile dbt uses the **profiles.yml** file as a standard profile for SQL data sources. Here's a sample configuration: ```YAML default: outputs: dev: type: singlestore host: port: user: password: database: # required schema: # required threads: # default: 1 target: dev ``` > **📝 Note**: Always specify the `type` as `singlestore`. You must include the `type` either in **profiles.yml** or **dbt\_project.yml**. | Configuration | Description | Required | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | | `host` | The IP address or the hostname of theSingleStore Helioscluster.Refer to[SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md)to determine the endpoint (`host:port`) of your deployment. | No | | `user` | TheSingleStore Heliosdatabase username. | No | | `password` | Password of theSingleStore Heliosuser. | No | | `database` | The name of theSingleStore Heliosdatabase. If you are using custom database names in your model configuration, you must create them before running the models. | Yes | | `schema` | Refer to[Schema and Concurrent Development](https://docs.singlestore.com/#section-idm4573377319940832872801277635.md). | Yes | | `threads` | The number of threads available to dbt. | No | ## Schema and Concurrent Development In SingleStore Helios, database and schema denote the same concepts. In dbt, a schema refers to a namespace within a database. Therefore, you must specify the `schema` in **profile.yml** for dbt to work with the project metadata. To support concurrent development, prefix the table names that dbt is building within the database with `schema`. The following macro prefixes the table names with `schema`: ``` {% macro generate_alias_name(custom_alias_name=none, node=none) -%} {%- if custom_alias_name is none -%} {{ node.schema }}__{{ node.name }} {%- else -%} {{ node.schema }}__{{ custom_alias_name | trim }} {%- endif -%} {%- endmacro %} ``` For example, if you have a model named `customers` and `schema` is set to `dev`, without this macro, dbt creates a table named `customers` in the database. When this macro is added to the project, dbt creates a table named `dev__customers`. Refer to [SingleStore Profile](https://docs.getdbt.com/reference/warehouse-profiles/singlestore-profile#schema-and-concurrent-development) for more information. *** Modified at: October 24, 2025 Source: [/cloud/load-data/integrate-with-singlestore-helios/transform-data-with-dbt/](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios/transform-data-with-dbt/) (An index of the documentation is available at /llms.txt) --- # Load Data from Files Pipelines support the following data formats: ## In this section * [Load Data from Local Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-local-files.md) * [Load Data from CSV Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files.md) * [Load Data from JSON Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files.md) * [Load Data from Avro Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files.md) * [Load Data from Parquet Files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files.md) * [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) *** Modified at: July 25, 2025 Source: [/cloud/load-data/load-data-from-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/) (An index of the documentation is available at /llms.txt) --- # Load Data from Avro Files SingleStore Helios supports loading data from AVRO files. These files can be stored in multiple data sources. You can also load data from AVRO files that are stored locally. To view the list of data sources supported by SingleStore Helios, refer to [Data Sources](https://docs.singlestore.com/cloud/load-data/data-sources.md). ## Syntax for Avro LOAD DATA Below is the basic syntax for loading data from a local Avro file into a table. ```sql LOAD DATA [LOCAL] INFILE 'file_name' WHERE/SET/SKIP ERRORS[REPLACE | SKIP { CONSTRAINT | DUPLICATE KEY } ERRORS] INTO TABLE tbl_name FORMAT AVRO SCHEMA REGISTRY {"IP" | "Hostname"} subvalue_mapping [SET col_name = expr,...] [WHERE expr,...] [MAX_ERRORS number] [ERRORS HANDLE string] [SCHEMA 'avro_schema'] subvalue_mapping: ( {col_name | @variable_name} <- subvalue_path, ...) subvalue_path: {% | [%::]ident [::ident ...] ``` ## Order of Operations for Updating an Avro Schema and its Associated Database Objects When updating an Avro schema, follow the steps below (in order) to modify the pipeline and the pipeline’s target table to match the updated schema. The schema (which resides in the schema registry) can be modified at any time during this process. 1. Run [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) to detach the Pipeline. **When adding a field to the schema, this step is not required**. 2. Using [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md), to add or remove fields from the pipeline’s target tables. 3. Using [ALTER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/alter-pipeline.md) or [CREATE OR REPLACE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md), to add or remove fields from the pipeline. 4. Run [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) to start the pipeline. **If a field is added to the schema and the pipeline is running, this step is not required.** If adding a field to the schema prior to adding a field to the pipeline and its target table, and have not stopped the Pipeline, the Pipeline continues to run. The newly added field will not contain data unless it has also been added to the pipeline's target table. If a field is added to the schema before being added to a running pipeline and its target table, the pipeline will continue to run. Before the new field is added to the schema, a default value for the field is populated in the pipeline’s target table. See [Example 1: Adding a New Field to an Avro Schema](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/avro-schema-evolution-with-pipelines/#UUID-fd9ae8c9-caf6-52dd-d4f6-2ef6d48666c2.md) below for details. ## Schema Evolution Examples ## Example 1: Adding a New Field to an Avro Schema Consider an Avro schema `product` with the fields `id` and `color`: ```json { "type": "record", "name": "product", "fields": [ { "name": "id", "type": "long" }, { "name": "color", "type": [ "null", "string" ] } ] } ``` The fields are loaded into the SingleStore Helios table `t` through the Pipeline via the following commands: ```sql CREATE TABLE t(id BIGINT DEFAULT 0, color VARCHAR(30) DEFAULT NULL, input_record JSON DEFAULT NULL); CREATE PIPELINE p AS LOAD DATA FS "/path/to/files/data.avro" INTO TABLE t FORMAT AVRO SCHEMA REGISTRY "your_schema_registry_host_name_or_ip:your_schema_registry_port" (id <- %::id, color <- %::color, input_record <- %); ``` Now, the Avro schema evolves with a new field `price`: ```json { "type": "record", "name": "product", "fields": [ { "name": "id", "type": "long" }, { "name": "color", "type": [ "null", "string" ] }, { "name": "price", "type": "float" } ] } ``` To reflect the addition of the new field, you need to first update the table `t` along with the default value of the new field `age`: ```sql ALTER TABLE t ADD field price FLOAT DEFAULT NULL; ``` Using the `CREATE OR REPLACE PIPELINE` command, update the Pipeline to load the new field. This allows the Pipeline to continue running without losing any offsets. ```sql CREATE OR REPLACE PIPELINE p AS LOAD DATA FS "/path/to/files/data.avro" INTO TABLE t FORMAT AVRO SCHEMA REGISTRY "your_schema_registry_host_name_or_ip:your_schema_registry_port" (id <- %::id, color <- %::color, price <- %::price DEFAULT NULL, input_record <- %); ``` ## Populating the Pipeline’s Target Table with a Default Value If, prior to updating the Avro schema to include the new field `price`, you updated the Pipeline and its target table to include the field, the table’s field would be populated with the `DEFAULT` value. This value can be specified either in the `ALTER` statement that alters the table (`DEFAULT NULL` in this example), or in the `CREATE OR REPLACE PIPELINE` statement that alters the Pipeline (`DEFAULT NULL` in this example). If the `DEFAULT` value is specified in both places, as in this example, the table’s `DEFAULT` value is used. If the `DEFAULT` value is specified in neither place, an error is thrown. ## Example 2: Removing a Field from an Avro Schema Consider an Avro schema `product` with the fields `id`, `color`, and `price`: ```json { "type": "record", "name": "product", "fields": [ { "name": "id", "type": "long" }, { "name": "color", "type": [ "null", "string" ] }, { "name": "price", "type": "float" } ] } ``` Load the schema into the SingleStore Helios table `t` through the Pipeline by the following command: ```sql CREATE OR REPLACE PIPELINE p AS LOAD DATA FS "/path/to/files/data.avro" INTO TABLE t FORMAT AVRO SCHEMA REGISTRY "your_schema_registry_host_name_or_ip:your_schema_registry_port" (id <- %::id, color <- %::color, price <- %::price, input_record <- %); ``` Now, the Avro schema evolves by no longer containing the field `price`: ```json { "type": "record", "name": "data", "fields": [ { "name": "id", "type": "long" }, { "name": "color", "type": [ "null", "string" ] } ] } ``` Run the following commands in sequence to evolve the schema: ```sql STOP PIPELINE DETACH p; ``` **Note:** The `DETACH` option in the `STOP PIPELINE` command allows to temporarily stop a pipeline in order to make changes to a source table to remove a field. ```sql ALTER TABLE t REMOVE field age; ``` ```sql CREATE OR REPLACE PIPELINE p AS LOAD DATA FS "/path/to/files/data.avro" INTO TABLE t FORMAT AVRO SCHEMA REGISTRY "" (id <- %::id, color <- %::color, input_record <- %); ``` ```sql START PIPELINE p; ``` ## Example 3: Connecting to Confluent Schema Registry Over SSL > **📝 Note**: This example only applies to SingleStore Helios 7.3.5 and later. The following example shows how to connect to Confluent Schema Registry over SSL, using the SSL configuration settings in the `CONFIG` and `CREDENTIALS` clauses of `CREATE PIPELINE`. ```sql CREATE OR REPLACE PIPELINE p AS LOAD DATA FS "/path/to/files/data.avro" INTO TABLE t FORMAT AVRO SCHEMA REGISTRY "" (id <- %::id, color <- %::color, input_record <- %); CONFIG '{"schema.registry.ssl.certificate.location": "/var/private/ssl/client_memsql_client.pem", "schema.registry.ssl.key.location": "/var/private/ssl/client_memsql_client.key", "schema.registry.ssl.ca.location": "/var/private/ssl/ca-cert.pem"}' CREDENTIALS '{"schema.registry.ssl.key.password": "abcdefgh"}' ``` ```sql START PIPELINE p; ``` > **📝 Note**: You can use a subset of the \`ssl.\` settings as follows:- `schema.registry.ssl.key.location`, `schema.registry.ssl.ca.location`, and `schema.registry.ssl.key.password` > - `schema.registry.ssl.certificate.location`, `schema.registry.ssl.key.location`, and `schema.registry.ssl.key.password` `schema.registry.ssl.key.password` is only required if your SSL certificate key has a password. ## In this section * [Avro Schema Evolution with Pipelines](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/avro-schema-evolution-with-pipelines.md) * [Load Avro Files with LOAD DATA](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/load-avro-files-with-load-data.md) * [Extracting and Converting Avro Values](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/extracting-and-converting-avro-values.md) * [Load Avro Files Examples](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/load-avro-files-examples.md) *** Modified at: April 3, 2025 Source: [/cloud/load-data/load-data-from-files/load-data-from-avro-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-avro-files/) (An index of the documentation is available at /llms.txt) --- # Load Data from CSV Files SingleStore Helios supports loading data from CSV files. These files can be stored in multiple data sources. You can also load data from CSV files that are stored locally. To view the list of data sources supported by SingleStore Helios, refer to [Data Sources](https://docs.singlestore.com/cloud/load-data/data-sources.md). ## In this section * [Load CSV Files with LOAD DATA](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files/load-csv-files-with-load-data.md) * [Load CSV Files Examples](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files/load-csv-files-examples.md) * [Loading JSON Data from a CSV File](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files/loading-json-data-from-a-csv-file.md) *** Modified at: April 3, 2025 Source: [/cloud/load-data/load-data-from-files/load-data-from-csv-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-csv-files/) (An index of the documentation is available at /llms.txt) --- # Load Data from JSON Files SingleStore Helios supports loading data from JSON files. These files can be stored in multiple data sources. You can also load data from JSON files that are stored locally. To view the list of data sources supported by SingleStore Helios, refer to [Data Sources](https://docs.singlestore.com/cloud/load-data/data-sources.md). ## In this section * [Load JSON Files with LOAD DATA](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/load-json-files-with-load-data.md) * [Extracting and Converting JSON Values](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/extracting-and-converting-json-values.md) * [Load JSON Files Examples](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/load-json-files-examples.md) * [Using Mapping with JSON files](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/using-mapping-with-json-files.md) *** Modified at: April 3, 2025 Source: [/cloud/load-data/load-data-from-files/load-data-from-json-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-json-files/) (An index of the documentation is available at /llms.txt) --- # Load Data from Local Files To load data from local files, enable `LOCAL INFILE` by starting the SingleStore client with the `--local-infile` option. This option allows the client to read files from the local filesystem and send them to the database for ingestion using `LOAD DATA LOCAL INFILE`. The basic format for loading data into a table using local file is: ``` LOAD DATA LOCAL INFILE '/example-directory/' INTO TABLE COLUMNS TERMINATED BY ','; ``` ## Example Using a Local CSV File Using this table format: ```sql CREATE TABLE employees(emp_id INT, emp_lastname VARCHAR(25),emp_firstname VARCHAR(25), emp_title VARCHAR(25),dept VARCHAR(25),emp_city VARCHAR(25), emp_state VARCHAR(5),emp_ccode VARCHAR(4)); ``` Load data from a local file location: ```sql LOAD DATA INFILE '/tmp/emp_data.csv' INTO TABLE employees FIELDS TERMINATED BY ',' ENCLOSED BY '"'; ``` Verify the file loaded correctly: ```sql SELECT * FROM employees; ``` ```output +--------+--------------+---------------+---------------+-----------+--------------+-----------+-----------+ | emp_id | emp_lastname | emp_firstname | emp_title | dept | emp_city | emp_state | emp_ccode | +--------+--------------+---------------+---------------+-----------+--------------+-----------+-----------+ | 102 | Karras | Damien | Doctor | R&D | NYC | NY | US | | 110 | Strode | Laurie | VP Sales | Sales | Haddonfield | IL | US | | 89 | Whateley | Wilbur | CEO | Sen_Mgmt | Dunwich | MA | US | | 817 | Eriksson | Oskar | Support | IT | Stockholm | NULL | SE | | 298 | Denbrough | Bill | Salesperson | Sales | Bangor | ME | US | | 399 | Torrance | Jack | PR Dir | PR | Estes Park | CO | US | | 410 | Wilkes | Annie | HR Mgr | HR | Silver Creek | CO | US | | 312 | Cady | Max | IT Dir | IT | New Essex | FL | US | | 691 | Grevers | Nick | Rep | PR | Grimnetz | NULL | CH | | 14 | Bateman | Patrick | Prod_Mgr | prod_dev | NYC | NY | US | | 75 | White | Carrie | Receptionist | HR | Chamberlain | ME | US | | 263 | MacNeil | Regan | R&D Mgr | R&D | Washington | DC | US | +--------+--------------+---------------+---------------+-----------+--------------+-----------+-----------+ ``` ## CSV FILE DATA Below is the contents of the csv file used in the examples above. ```csv 014, Bateman, Patrick,Prod_Mgr, prod_dev, NYC, NY, US 102, Karras, Damien, Doctor,R&D, NYC, NY, US 298, Denbrough, Bill, Salesperson,Sales, Bangor, ME, US 399, Torrance, Jack, PR Dir, PR,Estes Park,CO, US 410, Wilkes, Annie, HR Mgr,HR,Silver Creek, CO, US 110, Strode, Laurie, VP Sales,Sales, Haddonfield, IL, US 312, Cady, Max, IT Dir, IT, New Essex,FL, US 089, Whateley, Wilbur, CEO, Sen_Mgmt, Dunwich, MA, US 075, White, Carrie, Receptionist, HR,Chamberlain, ME, US 263, MacNeil, Regan, R&D Mgr,R&D, Washington, DC, US 817, Eriksson, Oskar, Support, IT, Stockholm, NULL, SE 691, Grevers, Nick, Rep, PR, Grimnetz, NULL, CH ``` *** Modified at: July 14, 2026 Source: [/cloud/load-data/load-data-from-files/load-data-from-local-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-local-files/) (An index of the documentation is available at /llms.txt) --- # Load Data from Parquet Files Parquet formatted files can be loaded from a local file or by using a pipeline. The basic syntax is provided below: ## Load Parquet Files from a Local Filesystem Parquet formatted data stored on the local filesystem can be loaded using a `LOAD DATA` query. This streamlines the process of loading cloud-stored data into tables. Other `LOAD DATA` clauses (`SET`, `WHERE`, etc.) are supported (but not shown) in the following syntax examples. Syntax for loading a Parquet file: ```sql LOAD DATA INFILE '' INTO TABLE (val1 <- source1, val2 <- source2 [ ... ] ) [COMPRESSION { AUTO | NONE | LZ4 | GZIP }] [ ... ] FORMAT PARQUET; ``` ## Example: Loading Parquet Files Create a table: ```sql CREATE TABLE assets( Product_ID INT, Category VARCHAR(50), Model VARCHAR(50), Price DECIMAL(6,2), Employee_ID NUMERIC(5) ); ``` Create the LOAD DATA statement: Verify the file's location and the column names are correct. ```sql LOAD DATA INFILE '/tmp/assets.parquet' INTO TABLE assets (Product_ID <- Product_ID, Category <- Category, Model <- Model, Price <- Price, Employee_ID <- Employee_ID) FORMAT PARQUET; ``` Verify the data loaded: ``` SELECT * FROM ASSETS; --- +------------+------------+-----------------+---------+-------------+ | Product_ID | Category | Model | Price | Employee_ID | +------------+------------+-----------------+---------+-------------+ | 58 | cell_phone | iphone_12 | 879.00 | 410 | | 1049 | laptop | mac_book_pro | 1999.00 | 14 | | 1100 | laptop | mac_book_pro | 1999.00 | 298 | | 210 | laptop | mac_book_pro | 2500.00 | 312 | | 130 | cell_phone | iphone_13 | 699.00 | 110 | | 99 | laptop | mac_book_air_M1 | 999.00 | 75 | | 140 | cell_phone | iphone_13_pro | 999.00 | 263 | | 49 | cell_phone | iphone_12 | 879.00 | 102 | | 2037 | laptop | mac_book_air_M2 | 1199.00 | 399 | | 111 | laptop | mac_book_pro | 2100.00 | 89 | +------------+------------+-----------------+---------+-------------+ ``` > **❗ Important**: Column names must be specified and they are case-sensitive. ## Load Parquet Files Using a Pipeline Parquet formatted data stored in an AWS S3 bucket can be loaded using a `LOAD DATA` query with a pipeline. Pipeline syntax: ```sql LOAD DATA S3 '' CONFIG '{"region" : ""}' CREDENTIALS '{"aws_access_key_id" : " ", "aws_secret_access_key": ""}' INTO TABLE (`` <- %, `` <- % DEFAULT NULL , ) FORMAT PARQUET; ``` ## Example: Parquet Pipeline This section will provide an example of creating a SQL table, a Parquet file for import, and a Parquet formatted pipeline. Create a table with the following structure: ```sql CREATE TABLE employees( ID NUMERIC(5), Last_Name VARCHAR(25), First_Name VARCHAR(25), Job_Title VARCHAR(25), Department VARCHAR(25), City VARCHAR(25), State VARCHAR(2), Email VARCHAR(50) ); ``` Install the `pandas` and `pyarrow` libraries for Parquet file support: ```python pip install pandas pyarrow ``` Create a file with data based on the table structure, be sure to save it as a Python file and use the correct path for the output: ```python import pandas as pd # Provided data data = [ ('014', 'Bateman', 'Patrick', 'Prod_Mgr', 'prod_dev', 'NYC', 'NY', 'pbateman@worldtech.com'), ('102', 'Karras', 'Damien', 'Doctor', 'R&D', 'NYC', 'NY', 'dkarras@worldtech.com'), ('298', 'Denbrough', 'Bill', 'Salesperson', 'Sales', 'Bangor', 'ME', 'bdenbrough@worldtech.com'), ('399', 'Torrance', 'Jack', 'PR Dir', 'PR', 'Estes Park', 'CO', 'jtorrance@worldtech.com'), ('410', 'Wilkes', 'Annie', 'HR Mgr', 'HR', 'Silver Creek', 'CO', 'awilkes@worldtech.com'), ('110', 'Strode', 'Laurie', 'VP Sales', 'Sales', 'Haddonfield', 'IL', 'lsatrode@worldtech.com'), ('312', 'Cady', 'Max', 'IT Dir', 'IT', 'New Essex', 'FL', 'mcady@worldtech.com'), ('089', 'Whateley', 'Wilbur', 'CEO', 'Sen_Mgmt', 'Dunwich', 'MA', 'wwhateley@worldtech.com'), ('075', 'White', 'Carrie', 'Receptionist', 'HR', 'Chamberlain', 'ME', 'cwhite@worldtech.com'), ('263', 'MacNeil', 'Regan', 'R&D Mgr', 'R&D', 'Washington', 'DC', 'rmacneil@worldtech.com') ] # Define column names columns = ['ID', 'Last_Name', 'First_Name', 'Job_Title', 'Department', 'City', 'State', 'Email'] # Create a DataFrame df = pd.DataFrame(data, columns=columns) # Save DataFrame to Parquet file df.to_parquet('//', index=False) ``` Once the file is saved, change the directory to the location of the Python script and run the script to generate the Parquet file: ```bash cd / ``` ```bash python .py ``` Upload the newly created Parquet file up to an S3 bucket and create the pipeline: ```sql CREATE PIPELINE emp_list AS LOAD DATA S3 's3:///employee_data.parquet' CONFIG '{"region":"us-west-2"}' CREDENTIALS '{"aws_access_key_id": "XXXXXXXXXX", "aws_secret_access_key": "XXXXXXXXXX"}' INTO TABLE employees (`ID` <- ID, `Last_Name` <- Last_Name, `First_Name` <- First_Name, `Job_Title` <- Job_Title, `Department` <- Department, `City` <- City, `State` <- State, `Email` <- Email) FORMAT PARQUET; ``` Start the pipeline: ```sql START PIPELINE emp_list; ``` Verify the data loaded correctly: ```sql SELECT * FROM employees; --- +------+-----------+------------+--------------+------------+--------------+-------+--------------------------+ | ID | Last_Name | First_Name | Job_Title | Department | City | State | Email | +------+-----------+------------+--------------+------------+--------------+-------+--------------------------+ | 263 | MacNeil | Regan | R&D Mgr | R&D | Washington | DC | rmacneil@worldtech.com | | 312 | Cady | Max | IT Dir | IT | New Essex | FL | mcady@worldtech.com | | 89 | Whateley | Wilbur | CEO | Sen_Mgmt | Dunwich | MA | wwhateley@worldtech.com | | 102 | Karras | Damien | Doctor | R&D | NYC | NY | dkarras@worldtech.com | | 14 | Bateman | Patrick | Prod_Mgr | prod_dev | NYC | NY | pbateman@worldtech.com | | 410 | Wilkes | Annie | HR Mgr | HR | Silver Creek | CO | awilkes@worldtech.com | | 75 | White | Carrie | Receptionist | HR | Chamberlain | ME | cwhite@worldtech.com | | 110 | Strode | Laurie | VP Sales | Sales | Haddonfield | IL | lsatrode@worldtech.com | | 298 | Denbrough | Bill | Salesperson | Sales | Bangor | ME | bdenbrough@worldtech.com | | 399 | Torrance | Jack | PR Dir | PR | Estes Park | CO | jtorrance@worldtech.com | +------+-----------+------------+--------------+------------+--------------+-------+--------------------------+ ``` > **❗ Important**: Column names must be specified and they are case-sensitive. ## In this section * [Create a Parquet Pipeline](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files/create-a-parquet-pipeline.md) * [Load Parquet Data using LOAD DATA](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files/load-parquet-data-using-load-data.md) *** Modified at: August 10, 2026 Source: [/cloud/load-data/load-data-from-files/load-data-from-parquet-files/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files/) (An index of the documentation is available at /llms.txt) --- # Stage Stage is a managed file storage service that helps you organize and manage local files for ingestion into your SingleStore database(s). Each cluster group has a Stage where you can create folders and upload files. Stage is also supported in the Shared Edition (Shared clusters). Through Stage, you can also save query results into files. > **📝 Note**: The cluster group must be running SingleStore version 8.1 or later. ## Why Use Stage Stage enables you to: * Manage files and folders using the [Cloud Portal](https://portal.singlestore.com) UI, [Management API](https://docs.singlestore.com/cloud/reference/management-api.md), [Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) (via the SingleStore Python SDK), and [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) commands. * Load files from a Stage into tables using pipelines, `LOAD DATA` command, or Cloud Portal UI and notebooks. * Export query results into a file using the `SELECT ... INTO STAGE` statement. ## Manage a Stage You can manage files and folders in a Stage using any of the following: * [Cloud Portal UI](https://docs.singlestore.com/#section-idm4565630576617634287605423613.md) * [Management API](https://docs.singlestore.com/#section-idm4537642325016034287607436222.md) * [Notebooks or SingleStore Python Client](https://docs.singlestore.com/#section-idm4567279659785634287607666885.md) * [Fusion SQL](https://docs.singlestore.com/#section-id235705289092485.md) ## Using the Cloud Portal UI On the Cloud Portal, select **Ingestion > Stage** in the left navigation pane. Manage files and folders in a Stage from this page. ## Upload a File To upload a file in a Stage: 1. Select **Ingestion > Stage > Upload File(s)**. 2. In the **Upload File(s)** dialog, perform the following: | **Select File(s)** | Either drag and drop file(s) to the dialog, or select**Browse Files**to select the files to upload. | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Cluster** | Select theclustergroup associated with the Stage to which the files are uploaded. | | **Folder** | Select one of the following options:
  • No folder: Uploads the file at the Stage root path (/).
  • Select folder: Uploads the file to an existing folder in the Stage; select a folder from the list.
  • Create folder: Uploads the file to a new folder in the Stage; enter a name for the new folder in the New folder name field.
| | **Stage** | (Read-only field) Specifies the cloud provider and region where the file is uploaded. | 3. Select **Upload Files** to upload the selected file(s) at the specified path in a Stage. Alternatively, select a folder in Stage, and then select **Upload File(s)** to directly upload files in this (current) folder. The file is uploaded in the Stage associated with the parent folder. ## Create a Folder Folders help organize and manage files efficiently. To create a folder at the Stage root path (`/`): 1. Select **Ingestion > Stage > Create Folder**. 2. In the **Create Folder** dialog, perform the following: | **Folder Name** | Enter a name for the folder. | | --------------- | -------------------------------------------------------------------------------------- | | **Cluster** | Select theclustergroup associated with the Stage where the folder is created. | | **Stage** | (Read-only field) Specifies the cloud provider and region where the folder is created. | 3. Select **Create Folder**. A folder is now added in the Stage associated with the selected cluster group. Alternatively, to create a subfolder: 1. Select an existing folder in Stage, and then select **Create Folder**. 2. Enter a name for the folder in the **Folder name** field and select **Create Folder**. The new folder is created inside the existing folder (as a subfolder) in the Stage associated with the parent folder. ## Manage Files and Folders To perform a specific task, select the ellipsis (three dots) under the **Actions** column of the file or folder and select the corresponding option from the list. | Action | Instruction | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Rename a file or folder | Select**Rename**, specify the file or folder name under the**Name**column, and select the**Save rename**(✔) icon. Select the**Cancel rename**(**X**) icon to cancel the rename operation. | | Download a file | Select**Download**to download the file. | | Load to database | Select**Load Data**to ingest the file in aSingleStoredatabase. This workflow generates a notebook that contains the commands to load the selected file. Refer to[Ingest a File Using theCloud Portal](https://docs.singlestore.com/#section-idm4617371648320033719737793809.md)for more information. | | Delete a file or folder | Select**Delete**, confirm the deletion and then select**Delete**again to remove the file or folder.

⚠️ Warning:

Deleting a folder removes the folder and all its contents, including any files and subfolders.

| ## Using the Management API Use the `Stage` path (`/v1/stage` endpoint) in the `Management` API to manage files and folders in a Stage. Refer to [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. For example, the following API call lists all the files and folders in the Stage attached to the cluster group with the specified ID: ```Shell curl -X 'GET' \ 'https://api.singlestore.com/v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/' \ -H 'accept: application/json' ``` ## Using Notebooks or SingleStore Python Client The [SingleStore Python SDK](https://singlestoredb-python.labs.singlestore.com/index.html) supports the [Stage](https://singlestoredb-python.labs.singlestore.com/api.html#stage) object, which can be used to manage files and folders in a Stage. You can also use the Stage object (including other objects in the SingleStore Python SDK) in a [notebook](https://docs.singlestore.com/cloud/container-services/notebooks.md). Refer to the [SingleStore Python Client](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-python/connect-using-the-singlestore-python-client.md) and [SingleStore Python SDK API Reference](https://singlestoredb-python.labs.singlestore.com/api.html) for more information. For example, the following code snippet uploads a file named **data.csv** to a Stage attached to a cluster group named **example-group**: ```Python from singlestoredb import manage_workspaces mgr = manage_workspaces('access_key_token_for_the_Management_API') wg = mgr.workspace_groups['example-group'] wg.stage.upload_file('/filepath/data.csv', '/data.csv') ``` ## Using Fusion SQL Commands Refer to [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) for the supported operations. ## Ingest a File using Stage Ingest files from Stage into a database using the Cloud Portal, `LOAD DATA` command, or pipelines. ## Using the Cloud Portal Using the Cloud Portal UI and SingleStore Notebooks, you can ingest Stage files into a SingleStore database attached to the cluster associated with the Stage. 1. Under **Stage**, select the ellipsis (three dots) in the **Actions** column of the file to upload, and then select **Load To Database**. 2. In the **Load Data** dialog, perform the following: 1. **Choose Cluster**: Select a cluster. 2. **Choose a database**: Select a database attached to the cluster. 3. **Table**: Select an existing table or enter a new table name. 3. Select **Generate Notebook**. A notebook is created, which contains the instructions and queries required to load the data. > **📝 Note**: You can edit the queries to modify table or column names, apply custom transformations, or make any other adjustments required before loading the data. 4. Select **Run > Run All Cells**. Optionally, run each notebook cell individually, in sequence. The **Check that the data has loaded** cell verifies if the data has been loaded. ## Using the LOAD DATA command Create a table with a structure that can store data from the file. Use the following `LOAD DATA` syntax to load a file from a stage: ```sql LOAD DATA STAGE 'path_in_stage/filename.extension' INTO TABLE [FORMAT {JSON | AVRO | CSV}]; ``` Refer to [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) for complete syntax and related information. The following example loads data from a CSV file from a Stage: ```sql LOAD DATA STAGE 'simple.csv' INTO TABLE simple_data FIELDS TERMINATED BY ',' IGNORE 1 LINES; ``` > **📝 Note**: `LOAD DATA STAGE` command is not supported in the Shared Edition. The cluster group must be running SingleStore version 8.9 or later. ## Using Pipelines Create a table with a structure that can store the data from the file. Use the following `CREATE PIPELINE` syntax to load a file from a Stage: ```sql CREATE PIPELINE AS LOAD DATA STAGE { } INTO TABLE { } ``` Once the table and pipeline are created, start the pipeline. Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for the complete syntax and related information. Here's a sample `CREATE PIPELINE` statement that loads data from a CSV file: ```sql CREATE PIPELINE dbTest.plTest AS LOAD DATA STAGE 'data.csv' BATCH_INTERVAL 2500 SKIP DUPLICATE KEY ERRORS INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n' STARTING BY '' FORMAT CSV; ``` ## Export SQL Results to a Stage SQL results may be exported to a Stage as follows: ```sql SELECT * FROM GROUP BY 1 INTO STAGE '' FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'; ``` Use the `GROUP BY 1` clause to avoid getting multiple files from each leaf node. ## Supported Files The Stage storage service supports the following file formats: | CSV | SQL | JSON | | ------- | --- | ---- | | Parquet | GZ | Zstd | | Snappy | | | ## Storage Limits Each Stage can have up to 10GB of storage for free. Individual files must not exceed 5GB in size. *** Modified at: July 31, 2026 Source: [/cloud/load-data/load-data-from-files/stage/](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage/) (An index of the documentation is available at /llms.txt) --- # Data Type Mapping in SingleStore Flow The following mappings define how data types from source databases are converted to their corresponding SingleStore types during ingestion with Flow. ## In this section * [MySQL to SingleStore Data Type Mapping](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow/mysql-to-singlestore-data-type-mapping.md) * [PostgreSQL to SingleStore Data Type Mapping](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow/postgresql-to-singlestore-data-type-mapping.md) *** Modified at: December 8, 2025 Source: [/cloud/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow/) (An index of the documentation is available at /llms.txt) --- # Free Trial Access to Flow SingleStore Flow is available without a license for learning and development purposes. It is not intended for production use. With the no-license operation, the total data transfer is limited to: * **Ingest**: 10 million rows or 10 GB, whichever comes first. * **XL Ingest**: 10 million rows, with up to 3 slices. * **SingleStore destination capacity**: Up to 1 unit. This limit applies to all types of data transfer, including both initial extracts and change data capture (CDC) operations. The enforcement of this limit is on a best-effort basis and may not be exact. If you require production use or higher data transfer capacity, please [contact us](https://www.singlestore.com/contact) to request a license. *** Modified at: May 27, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/free-trial-access-to-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/free-trial-access-to-flow/) (An index of the documentation is available at /llms.txt) --- # Install SingleStore Flow This section contains instructions for installing Ingest and XL Ingest, and license information. To use Ingest and XL Ingest, install the products and provide the product IDs to either [ team@singlestore.com](mailto:team@singlestore.com) or your account representative to obtain either a paid license or a Developer trial license. ## Prerequisites While the database prerequisites will vary based on the source, the following are the minimum requirements for using both Ingest and XL Ingest. | **Component** | **Requirements** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Virtual Machine (VM) | The sizing recommendation is based on the size of databases/tables being transferred.For smaller databases sized 100GB or less, a minimum of 2 vCPUs, 8 GB RAM, and 100 GB of free disk space is recommended.For large transfers, more disk space may be required. The required free space for Ingest is twice the size of the largest table to be transferred.For very large tables, where XL Ingest must be used, the amount of local disk needed is`2 x (logical partition size) x (number of concurrent partition transfers)`.More CPU cores and RAM is required for sources and destinations that contain a large amount of data and have an aggressive timeline in which to transfer it.The VM must be close to the source database to minimize latency.Refer to the[Recommended Hardware Configuration](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/#section-idm234737713970451.md)section in the Ingest documentation for more information. | | Browser | A Chromium-based browser, such as[Google Chrome](https://www.google.com/chrome/)or[Microsoft Edge](https://www.microsoft.com/en-us/edge/), installed on the VM | | Java | [Amazon Corretto Java 21](https://docs.aws.amazon.com/corretto/latest/corretto-21-ug/downloads-list.html), a no-cost distribution of the Open Java Development Kit (OpenJDK), installed on the VM | ## Set Up the Encryption Key Flow encrypts sensitive values in `config/config.xml` with AES-256-CBC. The Flow JVM reads the encryption key from the `FLOW_ENCRYPTION_KEY` environment variable. New installations must configure an encryption key before storing encrypted secrets. Use the following key requirements: * Algorithm: AES-256. The decoded key must be exactly 32 bytes. * Encoding: Base64, with optional trailing = padding. Flow validates the key at startup. If `FLOW_ENCRYPTION_KEY` is set but invalid, Flow does not start. If `FLOW_ENCRYPTION_KEY` is not set, new installations will not start, but upgraded installations may fall back to a legacy hardcoded key (insecure and deprecated). When Flow resolves the key successfully, it writes `Info(CMG500)` to the log together with a short SHA-256 fingerprint. Use this entry to confirm which key the process is using without exposing the key. **Note**: SingleStore recommends rotating the encryption key every 90 to 180 days for security. Refer to [Rotate the Encryption Key](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/rotate-the-encryption-key.md) for the key rotation procedure. ## Generate a Key on Windows Run the following PowerShell commands to generate a Base64-encoded 32-byte key. The commands use a cryptographically secure random generator. ```shell $bytes = New-Object byte[] 32 [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($bytes)[Convert]::ToBase64String($bytes) ``` If OpenSSL is available on the Windows host, run the following command: ```shell openssl rand -base64 32 ``` Store the generated value in a secret manager or another secure location. This value is required when configuring the `FLOW_ENCRYPTION_KEY`. ## Generate a Key on Linux or macOS Use OpenSSL to generate a Base64-encoded 32-byte key. OpenSSL is preinstalled on every supported Linux distribution. Run the following command: ```shell openssl rand -base64 32 ``` Store the generated value in a secret manager or another secure location. This value is required when configuring the `FLOW_ENCRYPTION_KEY`. ## Install Ingest You can install Ingest in both Windows and non-Windows environments. ## Install Ingest as a Windows Service **Note**: Filenames may change from release to release. 1. Download the following files to your host: * Ingest JAR: * Ingest Setup ZIP: 2. Unzip the Ingest setup ZIP file. This creates an `ingest` directory. 3. Move the `ingest` directory to your desired install location. For example, `C:\SingleStore\ingest`. 4. Copy the downloaded `ingest.jar` file into the **ingest directory** so that all required files are in the same location. 5. Change the directory to the `ingest` directory and open the `ingest.xml` file in a Notepad. Change the following lines as per your requirements. | `SingleConnect-Ingest` | This tag is used as a unique ID in Windows services.EachIngestpipeline has its own ID. | | ------------------------------------------------- | -------------------------------------------------------------------------------------- | | `SingleConnect Ingest` | This tag is a display name in the Windows services.It must be brief but descriptive. | | `SingleConnect Ingest` | This tag specifies a description of theIngestservice. | **Note**: When a server hosts several Java versions, relying on the system’s default Java can result in version conflicts or unexpected failures. To guarantee Ingest uses the intended Java version, refer the full path to your required Java binary directly. For example, if Java 21 has been extracted to `C:\java\java21`, specify the absolute path to `java.exe` in your `ingest.xml` file: ```xml C:\java\java21\bin\java.exe ``` 6. In `ingest.xml`, uncomment the env line and replace the placeholder with the generated Base64 key. The service wrapper exports the key to the Java process automatically. ```xml ``` To keep the secret out of `ingest.xml`, omit the env element and set `FLOW_ENCRYPTION_KEY` as a Windows system environment variable in **System Properties** > **Advanced** > **Environment Variables**. The Ingest service reads the updated value the next time it starts. 7. Grant the following permissions to the user that runs the Ingest service, on the entire Ingest directory: * Read * Write * Delete Ensure that all contained files and subfolders get the same permissions. 8. Install Ingest as a Windows service. 1. Open a command prompt as an Administrator. 2. Change the current directory to the `ingest` directory and run the `Install-Service.bat` file using the following command. ```shell Install-Service.bat ``` This batch file runs commands to install the service. Once the command is completed, close the command prompt. 9. The Ingest service must appear in the Windows services list like `SingleConnect-Ingest`, matching the ID set earlier. 10. The Ingest service starts automatically upon installation and must be running. You can start, stop, or restart the Ingest service similar to other Windows services. 11. To verify the encryption configuration, open the most recent log file under `ingest\log\` and search for `CMG500`. Verify that the log contains the following line: ```shell Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>) ``` If the log shows "`...from legacy hardcoded fallback`", Flow is using an insecure default key instead of `FLOW_ENCRYPTION_KEY`. This means the environment variable was not successfully passed to the JVM. Recheck the `ingest.xml` env entry, or the Windows system environment variable, and then restart the service. 12. Open a Chromium-based browser at `http://localhost:8081`, or `http://:8081` if the application is installed on a remote server. You can change this port from the user interface (UI). The new port setting takes effect after the next run. 13. Navigate to **Settings** > **License** and enter the license key. ## Install Ingest in Non-Windows Environment 1. Download the following files to your host: * Ingest JAR: * Ingest Setup ZIP: 2. Unzip the Ingest setup ZIP file. This creates an `ingest` directory. 3. Move the `ingest` directory to your desired install location. For example, `/opt/singlestore/ingest`. 4. Copy the downloaded `ingest.jar` file into the **ingest directory** so that all required files are in the same location. 5. Configure and start Ingest using ONE of the following methods: * Edit the launcher `ingest.sh` in the Ingest directory. Uncomment the `FLOW_ENCRYPTION_KEY` lines and replace the placeholder with the generated Base64 key. ```shell export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE java -cp ingest.jar:lib/*:. org.sirus.Run ``` To start Ingest, run: ```shell ./ingest.sh ``` * Leave `ingest.sh` unchanged, and export `FLOW_ENCRYPTION_KEY` in the shell that launches Ingest, then start Ingest. ```shell export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE java -jar ingest.jar java -jar ingest.jar ``` * Pass the key as a Java system property. ```shell java -Dflow.encryption.key=YOUR_BASE64_KEY_HERE -jar ingest.jar ``` **Note**: The `flow.encryption.key` Java system property takes precedence over the environment variable. This precedence is useful when multiple Flow instances run on the same host with different keys. 6. To include external libraries, run the following command. **Windows** ```shell java -cp "other_lib.jar;ingest.jar" org.springframework.boot.loader.JarLauncher ``` **Linux** ```shell java -cp "other_lib.jar:ingest.jar" org.springframework.boot.loader.JarLauncher ``` 7. Tail the most recent file under `log/` and search for `CMG500`: ```shell grep -F 'Info(CMG500)' log/*.log | tail ``` Expected output: ```shell Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>) ``` If the log shows "`..from legacy hardcoded fallback`", the variable was not exported into the JVM environment. Confirm it in the same shell that launched Ingest. ```shell env | grep FLOW_ENCRYPTION_KEY ``` 8. Open a Chromium-based browser at `http://localhost:8081`, or `http://:8081` if the application is installed on a remote server. You can change this port from the user interface (UI). The new port setting takes effect after the next run. 9. Navigate to **Settings** > **License** and enter the license key. ## Install XL Ingest You can install XL Ingest in both Windows and non-Windows environments. **Note**: Filenames may change from release to release. ## Install XL Ingest as a Windows Service 1. Download the following files to your host: * XL Ingest JAR: * XL Ingest Setup ZIP: 2. Unzip the XL Ingest setup ZIP file. This creates an `xlingest` directory. 3. Move the `xlingest` directory to your desired install location. For example, `C:\SingleStore\xlingest`. 4. Copy the downloaded `xlingest.jar` file into the **xlingest directory** so that all required files are in the same location. 5. Change the directory to the `xlingest` directory and open `xlingest.xml` file in a Notepad. Change the following lines as per your requirements. | `SingleConnect-XL-Ingest` | This tag is used as a unique ID in Windows services.EachXL Ingestpipeline has its own ID. | | ---------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `SingleConnect XL Ingest` | This tag is a display name in the Windows services.It must be brief and descriptive. | | `SingleConnect XL Ingest` | This tag specifies a description of theXL Ingestservice. | 6. In `xlingest.xml`, uncomment the env line and replace the placeholder with the generated Base64 key. The service wrapper exports the key to the Java process automatically. ```xml ``` XL Ingest and Ingest use the same `FLOW_ENCRYPTION_KEY` value because they share the same configuration file. The XL Ingest service reads the updated value the next time it starts. 7. Install XL Ingest as a Windows service. 1. Open a command prompt as an Administrator. 2. Navigate to `xlingest` directory and run the `Install-SingleConnect-XL-Ingest-Service.bat` file using the following command. ```shell Install-SingleConnect-XL-Ingest-Service.bat ``` This batch file runs commands to install the service. Once the command is completed, close the command prompt. 8. The XL Ingest service must appear in the Windows services list like `SingleConnect-XL-Ingest`, matching the ID you set earlier. 9. The XL Ingest service starts automatically upon installation and must be running. You can start, stop, or restart the XL Ingest service like other Windows services. 10. To verify the encryption configuration, open the most recent log file under `xlingest\log\` and search for `CMG500`. Verify that the log contains the following line: ``` Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>) ``` If the log shows "`...from legacy hardcoded fallback`", Flow is using an insecure default key instead of `FLOW_ENCRYPTION_KEY`. This means the environment variable was not successfully passed to the JVM. Recheck the `xlingest.xml` env entry, or the Windows system environment variable, and then restart the service. 11. Open a Chromium-based browser at . If the application is installed on a server, navigate to `http://:8084` instead. 12. Navigate to **Configuration** > **License**, and enter the license key. ## Install XL Ingest in Non-Windows Environment 1. Download the following files to your host: * XL Ingest JAR: * XL Ingest Setup ZIP: 2. Unzip the XL Ingest setup ZIP file. This creates an `xlingest` directory. 3. Move the `xlingest` directory to your desired install location. For example, `/opt/singlestore/xlingest`. 4. Copy the downloaded `xlingest.jar` file into the **xlingest directory** so that all required files are in the same location. 5. Configure and start XL Ingest using ONE of the following methods: * Edit the launcher `xlingest.sh` in the ingest directory. Uncomment the `FLOW_ENCRYPTION_KEY` lines and replace the placeholder with the generated Base64 key. ```shell export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE java -cp xlingest.jar:lib/*:. cirus.Run ``` To start XL Ingest, run: ```shell ./xlingest.sh ``` * Leave `xlingest.sh` unchanged, and export `FLOW_ENCRYPTION_KEY` in the shell that launches XL Ingest, then start XL Ingest. ```shell export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE java -jar xlingest.jar java -jar ingest.jar ``` * Pass the key as a Java system property. ```shell java -Dflow.encryption.key=YOUR_BASE64_KEY_HERE -jar xlingest.jar ``` **Note**: The `flow.encryption.key` Java system property takes precedence over the environment variable. This precedence is useful when multiple Flow instances run on the same host with different keys. 6. To include external libraries, use the following command. **Windows** ```shell java -cp "other_lib.jar;xlingest.jar" cirus.Run ``` **Linux** ```shell java -cp "other_lib.jar:xlingest.jar" cirus.Run ``` 7. Tail the most recent file under `log/` and search for `CMG500`: ```shell grep -F 'Info(CMG500)' log/*.log | tail ``` Expected output: ```shell Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>) ``` If the log shows "`..from legacy hardcoded fallback`", the variable was not exported into the JVM environment. Confirm it in the same shell that launched XL Ingest. ```shell env | grep FLOW_ENCRYPTION_KEY ``` 8. Open a Chromium-based browser at . If the application is installed on a server, navigate to `http://:8084` instead. 9. Navigate to **Configuration** > **License**, and enter the license key. ## Apply the License Key The Flow suite of products comes with a validity period specific to each customer's agreement. You must obtain valid license keys to begin using the software. For learning and evaluation purposes, a free license with usage limits is available, refer to [Free Trial Access to Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/free-trial-access-to-flow.md) for more details. Licenses specify a maximum capacity in units. To transfer data to a SingleStore destination, the capacity of the Ingest license must be at least equal to the number of capacity units in the destination. Each software product has its own unique product ID when set up individually. To obtain a valid license, contact [SingleStore Sales](https://www.singlestore.com/contact/) with the **Product ID**. **For** Ingest, go to **Settings >** **Licence**. In **Licence Key**, enter the license key, and then select **Apply** to save the settings. To retrieve the Product ID, go to your Ingest instance, go to **Settings** > **Licence**, and copy the ID from the **'Product Id'**. ****For** XL Ingest,** go to **Configuration**. In **Licence Key**, enter the license key, and then select **Save** to save the settings. To retrieve the Product ID, go to your XL Ingest instance, go to **Configuration** > **Licence**, and copy the ID from the **'Product Id'**. *** Modified at: July 23, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow/) (An index of the documentation is available at /llms.txt) --- # Rotate the Encryption Key Use this procedure to rotate an existing `FLOW_ENCRYPTION_KEY` used by Flow to encrypt secrets stored in `config/config.xml` after Ingest or XL Ingest is installed and configured. SingleStore recommends rotating `FLOW_ENCRYPTION_KEY` every 90 to 180 days, and immediately after any suspected compromise. Flow rotates the key transparently. At the first `loadConfig()` after restart, Flow decrypts each secret with the old key, re-encrypts it with the new key, and rewrites `config/config.xml`. No manual edits to `config/config.xml` are required. Refer to [Install SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow.md) for information on initial key setup and installation. ## Prepare for Key Rotation Before rotating the key, complete the following tasks: * Make a copy of `config/config.xml` so the configuration can be restored if needed. * Choose a maintenance window because Flow restarts during the rotation. Note that active extracts resume from the last checkpoint after the service starts again. * Generate the new Base64 encoded key using the same method used for the original key. ## Rotate the Encryption Key Perform the following steps on every Flow host, including both Ingest and XL Ingest when applicable. 1. Generate a new key using the same command used during the initial setup. Refer to [Set Up the Encryption Key](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow/#section-id235645650796082.md) for more information. 2. Set the following variables in the launcher: * `FLOW_ENCRYPTION_KEY` must contain the new key. * `OLD_FLOW_ENCRYPTION_KEY` must contain the previous key. **Linux launcher (`ingest.sh` or `xlingest.sh`):** ```shell export FLOW_ENCRYPTION_KEY= export OLD_FLOW_ENCRYPTION_KEY= java -cp ingest.jar:lib/*:. org.sirus.Run ``` **Windows service wrapper (`ingest.xml` or `xlingest.xml`) :** ```xml ``` **Windows batch launcher (`singlestore-ingest.bat` / `SingleStore-XL-Ingest.bat`):** ```shell set FLOW_ENCRYPTION_KEY= set OLD_FLOW_ENCRYPTION_KEY= java -jar ingest.jar ``` 3. Restart the Flow service so the JVM picks up the new environment. Use `services.msc` on Windows, your service manager on Linux/macOS, or stop and restart the Java process manually for non-service deployments. 4. Confirm that the rotation completed. Check the most recent log file under `log/` for the following markers in order. ```shell Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint ) Info(CMG503): Old encryption key configured from OLD_FLOW_ENCRYPTION_KEY env var (fingerprint ); rotation enabled Info(CMG502): Encryption key rotation detected; re-encrypting secrets under current key ``` `CMG502` is the success signal confirming that every secret in `config/config.xml` was re-encrypted with the new key and persisted. If `CMG502` does not appear after a normal startup, either no secrets were stored under the old key, or every secret was already encrypted with the new key. 5. After confirming successful rotation on ALL hosts (`CMG502` in logs), remove `OLD_FLOW_ENCRYPTION_KEY` from the configuration before the next restart. The old key is no longer needed once all secrets are re-encrypted. Keep `FLOW_ENCRYPTION_KEY` set. ## Understand Key Rotation Behavior The following behavior is expected during key rotation. * If `OLD_FLOW_ENCRYPTION_KEY` is set without `FLOW_ENCRYPTION_KEY`, Flow does not start. This prevents Flow from treating the previous key as the current key. * If `OLD_FLOW_ENCRYPTION_KEY` is set to the same value as `FLOW_ENCRYPTION_KEY`, Flow logs a warning and rotation does not occur. * If a secret cannot be decrypted with either the current key or the old key, Flow raises an error and stops. This indicates the secret was encrypted with a different key. To resolve: Stop Flow, restore `config/config.xml` from backup, set `FLOW_ENCRYPTION_KEY` to the correct original key, restart Flow, and retry rotation with the correct `OLD_FLOW_ENCRYPTION_KEY` value. * The `CMG500` and `CMG503` log entries include a short fingerprint of the encryption key. If two processes show the same fingerprint, they are using the same key. This helps confirm which key was active without exposing the key itself. * Rotation is applied per host. Roll out the new key to every Flow instance and confirm `CMG502` on each instance before removing `OLD_FLOW_ENCRYPTION_KEY`. ## Roll Back the Rotation If the new key is incorrect and Flow does not start, remove `FLOW_ENCRYPTION_KEY` and, optionally, `OLD_FLOW_ENCRYPTION_KEY` from the launcher, and then restart Flow. Repeat the rotation procedure with the correct key value. If Flow has already started with the new key and `config/config.xml` was rewritten, restore the pre-rotation backup of `config/config.xml` before retrying. *** Modified at: July 22, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/rotate-the-encryption-key/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/rotate-the-encryption-key/) (An index of the documentation is available at /llms.txt) --- # Run Flow as a Linux Service Flow can be set up as a `systemd` service on Linux that runs in the background, persists across reboots and user logouts, automatically restarts on crashes, and can use integrated logging via `journalctl`. **Note**: This procedure is compatible with [systemd-based](https://en.wikipedia.org/wiki/Systemd) Linux distributions. You may need to adapt it for your specific environment. Follow these steps to configure Flow to run as a service: 1. Create the `systemd` unit file. ``` sudo vi /etc/systemd/system/.service ``` Replace `` with a name of your choice, such as flow-ingest or flow-xl-ingest. 2. Add the following content into the file, and update the `User`, `WorkingDirectory`, and `ExecStart` fields to match your environment. ``` [Unit] Description=SingleStore Flow Ingest After=network.target [Service] Type=simple User= # e.g., ec2-user WorkingDirectory= # e.g., /home/ec2-user/ingest # Adjust JVM options if you need memory limits, GC tuning, etc. ExecStart=/usr/bin/java -jar # e.g., /home/ec2-user/ingest/ingest.jar # SuccessExitStatus=143 # Uncomment so Ctrl+C / SIGTERM is treated as a clean exit Restart=always RestartSec=10 LimitNOFILE=65536 # Increase the open-file limit if ingest opens many files [Install] WantedBy=multi-user.target ``` 3. Enable and start the service. ``` sudo systemctl daemon-reload sudo systemctl enable --now ``` 4. Check service status and logs. ``` sudo systemctl status # health-check sudo journalctl -u -f # live logs (Ctrl-C to quit) ``` *** Modified at: July 10, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/run-flow-as-a-linux-service/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/run-flow-as-a-linux-service/) (An index of the documentation is available at /llms.txt) --- # Secure Access to Flow To protect Flow from unauthorized access, isolate it from public networks and allow access only through controlled entry points. The best setup will depend on the underlying operating system and infrastructure. Two commonly used configurations for securing Flow are: * **Linux Setup**: Mac to Linux Flow server via a bastion host * **Windows Setup**: Windows to Windows Flow server via Remote Desktop Protocol (RDP) ## Linux Setup: Mac to Linux Flow Server via a Bastion Host This setup uses a bastion host - a specialized server used to provide access to a private network from an external network (such as the internet). It is typically placed in a demilitarized zone (DMZ) and serves as a "jump server" or intermediary through which access to a Flow server is granted, usually via SSH or RDP. ## Architecture ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5a018293bf7bcc1f/6a7adafb448e4e64105b5442/Linux_Architecture_Secure_Flow-UVK6RB.png) In this setup: * A local machine (e.g., a Mac) connects to a Linux bastion host. * The bastion host has internal network access to the Flow server. * Flow can be accessed either directly from the bastion host or via SSH tunneling. ## Steps 1. **Local Mac Setup** 1. Generate an SSH key pair (if not already created): ```shell ssh-keygen -t rsa -b 2048 -f ~/.ssh/bastion_key ``` This command generates: * A private key (`bastion_key`) to be stored securely * A public key to be copied to the bastion host 2. Install Google Chrome to access Flow. 2. **Bastion Host Setup** The bastion host can be any server exposed to the internet (e.g., a cloud instance or physical server). In cloud environments (AWS, Azure, GCP), the bastion host is typically placed in a public subnet, while the Flow server resides in a private subnet. 1. Install the public SSH key on the bastion host Copy the public key from the local machine to the bastion host, either manually or by running the following command: ```shell ssh-copy-id -i ~/.ssh/bastion_key.pub your-username@bastion-host-ip ``` 2. Disable password-based login on the bastion host ```shell sudo nano /etc/ssh/sshd_config PasswordAuthentication no # Disable password auth ``` 3. Restrict login to specific users Add the following to `/etc/ssh/sshd_config`: ```shell AllowUsers your-username # Only allow a specific user ``` 4. Restart SSH service to apply changes ```shell sudo systemctl restart ssh ``` 5. Configure firewall / security group SSH access to the bastion host can be restricted to specific IP ranges, such as an office public IP address or VPN endpoints. Tools such as `ufw` (Uncomplicated Firewall) or `iptables` can be used to enforce these restrictions. If using `ufw`, run: ```shell sudo ufw allow from to any port 22 sudo ufw enable ``` If using a cloud provider like AWS, Azure or Google Cloud, set up a Security Group (AWS) or Firewall Rules (GCP) to allow only specific IP addresses to SSH into the bastion host. 3. **Flow Server Setup** The Flow server must only accept connections from the bastion host. To enforce this, configure a firewall or security group. If using `ufw`, run the following commands on the Flow server: ```shell sudo ufw default deny incoming sudo ufw allow from to any port 22 sudo ufw enable ``` If using a cloud provider like AWS, Azure or Google Cloud, modify the Flow server’s security group/firewall rules to only accept inbound connections from the bastion host’s internal IP address. For example, for AWS, allow SSH on port 22 from the IP of the bastion host. 4. **Accessing Flow** **Option 1: Direct SSH from bastion host** On the bastion host, run: ```shell ssh @ ``` **Option 2: SSH Tunneling or Port Forwarding** On the local Mac, create an SSH tunnel: ```shell ssh -i ~/.ssh/bastion_key -L 8081:flow_server_private_ip:8081 @ ``` This command creates a local port that forwards traffic to the Flow server's HTTP ports (e.g., 8081 for Ingest, 8084 for XL Ingest). **Note**: Ensure that network and firewall rules allow the bastion host's private IP to reach the Flow server’s HTTP ports. ## Monitoring and Logging * Enable logging for SSH connections on the bastion host. This can be done by configuring `/etc/ssh/sshd_config`: ```shell LogLevel VERBOSE ``` * Ensure all connections and activities are logged (e.g., via Syslog or CloudTrail in AWS). * Consider setting up two-factor authentication (2FA) for SSH or using a VPN in conjunction with the bastion host for added security. ## Security Best Practices * **Update and Patch Regularly**: Ensure that the bastion host is always up-to-date with security patches. * **Use Intrusion Detection Systems (IDS)**: Tools like Fail2Ban can help detect and block brute force or suspicious login attempts. * **Limit SSH Access**: Limit the number of users who can access the bastion host and configure role-based access for administrative functions. * **Harden SSH Configurations**: Implement other SSH security best practices, such as disabling root login and using strong key-based authentication. ## Windows Setup: Windows to Windows Flow Server via RDP This setup uses Windows Remote Desktop Protocol (RDP) to connect to a Flow server running on Windows. ## Architecture ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7e33cf3ed7a4a631/6a7adafc594711bb254c2679/Windows_Architecture_Secure_Flow-1kQfXA.png) In this setup: * A Windows PC on the corporate network or connected via VPN is used to access the Flow server. * Connection to the Windows Flow server is established using RDP over a private network, VPN, or Remote Desktop Gateway (RD Gateway). **Important**: * RDP (TCP 3389) must never be exposed directly to the public internet. * RDP must be restricted in Windows Firewall and any external firewall to VPN address ranges or RD Gateway only. * Enforce strong passwords and multi-factor authentication (MFA) for RDP. ## Steps 1. From the Windows PC, open **Remote Desktop Connection**. 2. Connect to the Windows Server using RDP (port 3389). 3. Once logged in, access Flow on Chrome browser on the server. ## Optional: Enable Direct Access via Firewall Rules Windows Firewall can be configured to allow direct access to Flow from specific IP addresses without using RDP. **Important**: * This option is only for internal or VPN networks. * Do not open Flow’s port (e.g., 8081) to the public internet, even with IP restrictions. * If external access is required, use VPN. On the Windows Flow server: 1. **Configure the Windows Firewall** Add a rule in the Windows Firewall to allow incoming connections on the port where Flow is running. This rule can be configured to allow only one specific IP or a range of IP addresses. To allow incoming connections: 1. Open the **Control Panel** from the **Start** menu and click **Windows Firewall**. 2. In the Windows Firewall window, select **Advanced Settings**. 3. In the Windows Firewall with Advanced Security window, click **Inbound Rules** in the left-hand pane. 4. In the right-hand pane, click **New Rule…**. 5. In the **Rule Type** step, select **Port**, then click **Next**. 6. In the **Protocol and Ports** step, choose **TCP**. Under **Specific local ports**, enter the port number to be enabled - for example, 8081. Click **Next**. 7. In the **Action** step, select **Allow the connection**, then click **Next**. 8. In the **Profile** step, select only **Domain** or **Private** based on the requirement. Click **Next**. 9. In the **Name** step, enter a name for the rule and click **Finish**. The new rule will now appear in the list of inbound rules. 2. **(If applicable) Configure the External Firewall** If an external/physical firewall is in place, add a rule to allow incoming connections on the port where Flow is running. This rule can also be configured to allow only one specific IP or a range of IP addresses. *** Modified at: July 10, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/secure-access-to-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/secure-access-to-flow/) (An index of the documentation is available at /llms.txt) --- # SingleStore Flow API > **📝 Note**: This is a Preview feature. The SingleStore Flow API allows you to create, configure, and manage Flow pipelines, enabling seamless integration with third-party applications and workflow automation. It provides an HTTP interface that primarily follows REST conventions, while also supporting RPC-style command endpoints for specific operations. Flow provides two APIs: [Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/singlestore-flow-ingest-api.md) for data transfers up to 10 GB, and [XL Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/singlestore-flow-xl-ingest-api.md) for larger tables. Additionally, you can use automated scripts to set up source and destination connections, configure schedule, select tables, apply license, and run a full extract with Ingest APIs. For larger tables, the scripts automate sliced data transfers using XL Ingest APIs. Refer to the [Example Flow API Scripts](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/example-flow-api-scripts.md) for sample Python and bash applications that use the API. ## In this section * [SingleStore Flow Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/singlestore-flow-ingest-api.md) * [SingleStore Flow XL Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/singlestore-flow-xl-ingest-api.md) * [Example Flow API Scripts](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/example-flow-api-scripts.md) *** Modified at: October 20, 2025 Source: [/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api/) (An index of the documentation is available at /llms.txt) --- # SingleStore Flow Release Notes ## Maintenance Release Changelog ## SingleStore Ingest ## 2026/07/23 Version [4.5.0\_b9df345a](https://release.memsql.com/singlestoreflow/production/4.5.0-b9df345a/ingest.jar) **Features** * Added schema evolution support for PostgreSQL during CDC, including detection and application of column additions and removals. * Added schema evolution support for MySQL during CDC, including handling for column changes and ENUM value changes. * Added selected-table size and row-count statistics. * Improved Postgres and MySQL metrics. **Fixes** * Fixed a MySQL binlog column mismatch issue that could occur when multiple table-map events share a transaction. * Fixed a destination connection leak during CDC polling. * Fixed an issue where delta changes could appear as **Full Extract** after **Sync New Tables**. * Fixed 0-row tables so they are marked **No Change** and the progress timer stops. ## 2026/04/30 Version [4.4.2\_38b8f072](https://release.memsql.com/singlestoreflow/production/4.4.2-38b8f072/ingest.jar) **Features** * Tables with no primary key no longer produce errors and are skipped during delta extracts with a warning message. This applies to Oracle Full Extract, Oracle Log Miner, MySQL Fast Extractor, and Postgres drivers. * Added support for full extract on SingleStore as a source database (non-production preview). * `ENUM` data type in Postgres is now converted to VARCHAR(256) in the destination table. * UI now displays timestamps according to the browser's locale settings. **Fixes** * Fixed issues with loading `NULL` values for `INT` and `DATE` data types from Oracle source databases. * Fixed display of long table names in the UI to prevent truncation. **Known Issues** * Some data types are not handled correctly when using SingleStore as a source database. ## 2026/04/02 Version [4.4.0 (7385e0f)](https://github.com/memsql/singlestore-flow/actions/runs/23836780354/artifacts/6215388646) **Features** * Added support for MySQL Fast Extractor (Currently in Preview). * Added support for `BIT(n)` datatype in MySQL. * Connection UI enhancements – Simplified connection setup by removing the **Confirm Password** field for all source drivers and the **Database name ** field for SingleStore. **Fixes** * Fixed an issue that caused data containing backslashes to be processed incorrectly. * Fixed an issue where refreshing the UI resulted in a whitelabel error page. **Known Issues** * Invalid dates are loaded as 1970-01-01 in MySQL. ## 2026/02/19 Version [4.3.5 (4112eaa)](https://bryteflow.com/release/singlestore/ingest/435-4112eaa/ingest.zip) **Features** * Unsupported datatypes are highlighted in the UI and tables with these types can no longer be selected * Added support for partitioned tables in Postgres * Invalid dates in MySQL (like year `0000`) are converted to supported values * The test connection for Postgres provides more detailed information * The `BOOL` / `BOOLEAN` type in MySQL and Postgres is now created as `TINYINT(1)` * `NUMERIC` in Postgres is now created as `DECIMAL(65,30)` * Added support for the `ENUM` type in MySQL **Fixes** * Record count in loading logs now reflects the actual value **Known Issues** * Some data types may erroneously appear as unsupported ## 2025/12/11 Version [4.3.0 (7d351cf)](https://bryteflow.com/release/singlestore/ingest/430-7d351cf/ingest.zip) **Features** * Added support for Salesforce as a source **Fixes** * Updates made based on latest vulnerability tests * The license is now displayed in units rather than GB * Updates made to replace schema with database in reference to SingleStore * Updated the Snowflake driver **Known Issues** * Compound columns in Salesforce are not supported * Sync struct may show a mismatch for integer columns in Salesforce * CDC Delete may not be captured for some tables in Salesforce * The Snowflake driver may produce errors. The recommended workaround is to include the following JDBC options: ``` &JDBC_QUERY_RESULT_FORMAT=JSON&CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX=true&DISABLE_GCS_DEFAULT_CREDENTIALS=true ``` ## 2025/11/21 Version [4.2.1 (ef8b557)](https://bryteflow.com/release/singlestore/ingest/421-ef8b557/ingest.zip) **Features** * Re-themed the UI to closely align with SingleStore's brand * Added SSL support for MySQL * PostgreSQL TEXT\[] arrays are now handled as JSON in SingleStore * The test connection for MySQL no longer displays an "Unable to Parse" error for basic credential-related issues * Suppressed extraneous messages on console on startup ## 2025/09/13 Version [4.1.7(9f09aa2)](https://bryteflow.com/release/singlestore/ingest/417-9f09aa2/ingest.zip) **Features** * Added ability to suppress Table Structure Sync in use cases where the destination tables are manually created * Added support for Where condition for Oracle Full Extracts * Table and columns using reserved words will be retained by default **Fixes** * Null values in JSON columns are correctly handled for Postgres **Known Issues** * Dropped columns in the source are not detected on structure sync * Some structure sync operations may fail without indication of error ## 2025/09/03 Version [4.1.6 (a8a074a)](https://bryteflow.com/release/singlestore/ingest/416-a8a074a/ingest.zip) **Fixes** * Fixed the "Queue Full" error in MySQL Continuous Mining * The binlog now reconnects after exiting due to network errors in MySQL Continuous Mining * Added support for license-free trial use * Removed deprecated AWS options in the UI * Fixed an issue where only the first transaction in a batch was processed for MySQL Continuous Mining ## 2025/08/20 Version [4.1.4 (00d0934)](https://bryteflow.com/release/singlestore/ingest/414-00d0934/ingest.zip) **Features** * Added support for non-production free license **Fixes** * Nulls and zero length strings loaded with no change * Fixed out-of-memory error for MySQL full extracts * Subsequent transactions in a batch are no longer ignored * Added handling for table and column names using reserved words ## 2025/06/04 Version [4.1\_b875](https://bryteflow.com/release/singlestore/ingest/41-b875/ingest.zip) **Features** * Added support for SingleStore as a source to the Any Database driver * Real-time CDC for MySQL - Added a new dropdown “Continuous” to MySQL * Added support for multi-tenant databases **Fixes** * More consistent display of tables in databases without a schema (or database) ## 2025/05/08 Version [4.1\_b804](https://bryteflow.com/release/singlestore/ingest/41-b804/ingest.zip) **Features** * Added support for MongoDB (private preview only) * Added support for Where clause for Snowflake source **Fixes** * Handled database and table names with hyphens. Hyphen changed to an underscore in the destination * When history is stored, all timestamps will be in UTC * Vector data type extracted and saved as JSON in Postgres * Fixed Issues with Rollback **Known Issues** * Timestamp with Timezone datatypes is not converted to UTC ## 2025/04/02 Version [4.1\_b702](https://bryteflow.com/release/singlestore/ingest/41-b702/ingest.zip) **Fixes** * Fixed delta load error in PostgreSQL for masked column * Cleared out `.err` files from spool directory for MySQL * Added support for the `JSON` data type in MySQL * `DATETIME(3)` now created to retain milliseconds for MySQL to SingleStore ## 2025/03/05 Version [4.1\_b699](https://bryteflow.com/release/singlestore/ingest/41-b699/ingest.zip) **Features** * Added support for the JDBC database source (Snowflake). Databases vary considerably and this driver has only been tested with Snowflake. JDBC database source extracts do not support a CDC mechanism and are full extracts only. * Enhancements to the PostgreSQL source driver * Array data type (inserted as JSON in SingleStore) * Custom data types * `JSON` and `JSONB` data types * Column names that are reserved words * Table names with mixed case **Fixes** * Fixed MySQL delta extract errors **Known Issues** * Masked columns are not supported for PostgreSQL * The `BYTEA` data type is not handled correctly on PostgreSQL * The `BIT(size)` data type is not handled correctly on MySQL * The `TINYINT`/`BIT` data type may show a mismatch on Sync Struct ## 2025/02/17 Version [4.1\_b671](https://bryteflow.com/release/singlestore/ingest/41-b671/ingest.zip) **Fixes** * Fixed MySQL delta extract errors * Added support for PostgreSQL user-defined data types * Added support for PostgreSQL extract of tables with reserved-word columns ## 2025/01/31 Version [4.1\_b643](https://bryteflow.com/release/singlestore/ingest/41-b643/ingest.zip) **Features** * Implemented a new licensing scheme which is based wholly on the destination SingleStore instance size * Added connection attributes to SingleStore connection * Added support for loading binary data. Flow now uses Avro format internally. * Mapped the Microsoft SQL Server type `DATETIME` to `DATETIME(6)` **Fixes** * Sync Struct does not get false negatives when matching data types * Handled reserved words in column names * Fixed loading deltas for history tables **Known Issues** * Delta extracts can sometimes fail on MySQL * Timezone-based timestamps do not take the time zone into account * Masked columns can fail on PostgreSQL * Binary data may be ignored on PostgreSQL * MySQL `BIT(n)` data types are not handled correctly * Rollback may not work as expected in some cases * Load message in the logs may show incorrect record counts ## 2025/01/21 Version [4.0.2\_b587](https://bryteflow.com/release/singlestore/ingest/402_b587/ingest.zip) **Features** * Updated how reserved words are handled in column names, which can be done by either adding an underscore (default) or enclosing them in backticks * Increased the precision for `DATETIME` in the SQL Server input source ## 2024/12/24 Version [4.0.2\_b571](https://bryteflow.com/release/singlestore/ingest/402_b571/ingest.zip) **Features** * Implemented data type mapping for all four sources * Primary key and shard key are automatically created on SingleStore * Removed extraneous sources from the drop-down list **Known Issues** * Binary data not loaded * Timestamps with time zones are not converted to UTC * Masking columns may give load errors * Loading with history may fail in some cases ## 2024/12/03 Version [4.0.2\_b536](https://bryteflow.com/release/singlestore/ingest/402_b536/ingest.zip) **Features** * Full extracts provide a detailed progress bar on the Dashboard * Default transfer type is now set to not keep history * PostgreSQL databases will now show tables in the `PUBLIC` schema **Fixes** * Auto-selecting the primary key now works as expected * The Dashboard shows scheduled period for values under 1 minute * The Advanced tab for Destination now works as expected * Fixed the occasional null value error on startup * Changes in the underlying runtime metadata tables ## SingleStore XL Ingest ## 2026/07/23 Version [2.6.0-b9df345a](https://release.memsql.com/singlestoreflow/production/2.6.0-b9df345a/xl-ingest.jar) **Features** * Improved large-table onboarding so a large table can be onboarded without halting ongoing CDC. * Added XL-Ingest table status summary with **Running** and **Scheduled** states. **Fixes** * Fixed XL-Ingest handling for PostgreSQL BC, infinity, and -infinity temporal values. * Fixed XL-Ingest deletes for composite primary keys in SingleStore. ## 2026/04/30 Version [2.5.1\_38b8f072](https://release.memsql.com/singlestoreflow/production/2.5.1-38b8f072/xl-ingest.jar) **Features** * Added support for transfers of tables without primary keys. * Extracts are automatically paused when available disk space falls below 50 GB. * When source or destination session count is set to 0, the sessions are stopped. * When a slice is transferred, a JSON line is logged providing comprehensive statistics for the transfer. **Fixes** * Fixed out of memory issues that occurred when more than 2,000 slices were submitted. * Fixed handling of special characters including carriage return, new line, tab, and vertical bar. * The table name column in the UI is now resizable. * Fixed handling of invalid dates from Oracle source databases. ## 2026/04/02 Version [2.4.6\_7385e0f](https://github.com/memsql/singlestore-flow/actions/runs/23836780354/artifacts/6215354164) **Features** * Added support for MySQL Fast Extractor (Currently in Preview) * Added support for `UUID` as a slicing column for Postgres * Added Multi-tenant support for Postgres ## 2026/02/19 Version [2.4.5\_4112eaa](https://bryteflow.com/release/singlestore/ixl/245-4112eaa/xlingest.zip) **Features** * UI refreshed to reflect the SingleStore style * The `BOOL` / `BOOLEAN` type in MySQL and Postgres is now created as `TINYINT(1)` * `NUMERIC` in Postgres is now created as `DECIMAL(65,30)` **Fixes** * Fixed errors on loading BIT data type ## 2025/12/12 Version [2.4.3\_7d351cf](https://bryteflow.com/release/singlestore/ixl/243-7d351cf/xlingest.zip) **Fixes** * Implemented changes based on latest vulnerability testing ## 2025/03/05 Version [2.4\_b1257](https://bryteflow.com/release/singlestore/ixl/24_b1257/xlingest.zip) **Features** * Added support for the JDBC database source (Snowflake) * Enhancements to the PostgreSQL source driver * UUID column can be used for slicing * Array data type (added to SingleStore as JSON) * Custom data types * `JSON` and `JSONB` data types * Column names that are reserved words * Table names with mixed case ## 2025/02/17 Version [2.4\_b1227](https://bryteflow.com/release/singlestore/ixl/24_b1227/xlingest.zip) **Fixes** * Handled reserved-word column names ## 2025/01/31 Version [2.4\_b1224](https://bryteflow.com/release/singlestore/ixl/24_b1224/xlingest.zip) **Features** * Implemented a new licensing scheme which is based wholly on the destination SingleStore instance size * Added connection attributes to SingleStore connection * Added support for loading binary data. Flow now uses Avro format internally. **Fixes** * Added support for loading data for tables with masked columns **Known Issues** * Timezone-based timestamps do not take the time zone into account * BLOB data is not extracted for Oracle ## 2025/01/21 Version [2.3\_b1178](https://bryteflow.com/release/singlestore/ixl/23_b1178/xlingest.zip) **Features** * Updated how reserved words are handled in column names, which can be done by either adding an underscore (default) or enclosing them in backticks * Increased the precision for `DATETIME` in the SQL Server input source ## 2024/12/24 Version [2.3\_b1174](https://bryteflow.com/release/singlestore/ixl/23_b1174/xlingest.zip) **Features** * Implemented data type mapping for all four sources **Known Issues** * Binary data not loaded * Timestamps with time zones are not converted to UTC * Masking columns may give load errors ## ControlRoom ## 2025/05/06 Version [43-b8](https://bryteflow.com/release/singlestore/controlroom/43-b8/croom.zip) **Known Issues** * Minor cosmetic issues on dashboard * No details are shown for an XL Ingest instance *** Modified at: July 23, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-release-notes/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-release-notes/) (An index of the documentation is available at /llms.txt) --- # SingleStore Ingest Operational FAQ This document describes some of the most frequent operational user queries. ## Stop Ingest Perform the following steps to stop Ingest. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Open **Windows Services** and stop the Ingest Windows service. 3. If the Ingest Windows service stops successfully but the Ingest page is still loading, you may need to terminate the non-responding processes. 4. To terminate the non-responding Ingest process, follow these steps. 1. Open the command prompt as **Administrator**. 2. Run the following command: ```shell netstat -ano | find "8081" ``` Replace `8081` with the port number on which Ingest is running. 3. Note the process ID (PID) of the Ingest service from the above command output. 4. Open **Windows Task Manager**, navigate to the **Details** tab, and find the process with corresponding PID. 5. Right-click on the process and select **End Task**. This terminates the runaway Ingest process. ## Restart Ingest Perform the following steps to restart Ingest. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. Then, stop the Ingest Windows service. 2. Restart the Ingest Windows service and wait for the Ingest dashboard to appear, refreshing the page as needed. 3. Navigate to the **Dashboard** tab, under **Schedule**, and enable the scheduler. 4. Ingest picks up the correct processing sequence. Confirm that the last sequence number loaded correctly and that the new sequence number started correctly. If needed, [perform a rollback](https://docs.singlestore.com/#section-idm234737465163327.md) to the correct sequence number for processing. ## Perform a Full Extract Use the following steps to perform a full extract for an initial sync of all tables in a pipeline. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to **Dashboard** > **Tables**. 1. Select the tables (one or more) for the full extract. 2. Ensure that the primary key (**PKey**) column(s) are applied. 3. Enable **Delta Extract** (**With History**). 4. Select **Apply**. 3. Navigate to the **Operations** tab and select **Full Extract**. 4. Navigate to the **Logs** tab. Check the logs to confirm that the selected tables have started the full extract. ## Skip an Initial Extract Use the following steps to perform deltas without an initial sync, or when using XL Ingest. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to **Dashboard** > **Tables**. 1. Select the table(s) to perform deltas without an initial sync of the data. 2. Ensure that the primary key (**PKey**) column(s) are applied. 3. Enable **Delta Extract** (**With History**). 4. Enable **Skip Initial Extract**. 5. Select **Apply**. 3. Navigate to the **Operations** tab and select **Full Extract**. 4. Navigate to the **Logs** tab. Check the logs to confirm that the selected table(s) skipped the full extract. ## Redo an Initial Extract Use the following steps to redo an initial extract for existing tables. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to **Dashboard** > **Tables**. 1. Select the table(s) for which to redo the initial extract. 2. Ensure that the primary key (**PKey**) column(s) are applied. 3. Enable **Delta Extract** (**With History**). 4. Enable **Redo Initial Extract**. 5. Select **Apply**. 3. Navigate to the **Operations** tab and select **Sync New Tables**. 4. Navigate to the **Logs** tab. Check the logs to confirm that the selected table(s) have started the full extract. ## Perform a Rollback Use the following steps to perform a rollback in Ingest. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to the **Operations** tab, select the gear icon, and then select **Rollback**. 3. In the pop-up window, select the sequence number from which you want to replay the log files. *Confirm that the log files are available on the source server.* 4. Select **Select**. 5. Navigate to the **Dashboard** tab. Under **Schedule**, enable the scheduler. ## Add New Tables to an Existing Pipeline Use the following steps to add only new tables to existing pipelines. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to **Dashboard** > **Tables**: 1. Select one or more new tables as required. 2. Ensure the primary key (**PKey**) column(s) are applied. 3. Enable **Delta Extract** (**With History**). 4. Select **Apply**. 3. Navigate to the **Operations** tab and select **Sync New Tables**. 4. Navigate to the **Logs** tab. Confirm that the existing tables were ignored and the new table(s) started to sync. ## Create a New Pipeline Use the following steps to create an additional Ingest pipeline. 1. Unzip the Ingest ZIP file into a folder, such as `C:\singlestore-flow\pipeline\`. 2. Edit the `ingest.xml` file located in the `C:\singlestore-flow\pipeline\` folder. 3. Change the `Bryteflow-Ingest` and `BryteFlow Ingest` tags to reference the new pipeline names. 4. Check that port `8081` is free and not being used by other processes. 5. Open a command prompt as Administrator, navigate to the new pipeline folder, run `Install-Service.bat`, and then close the command prompt. 6. Open the Windows Services console and start the newly-created service. 7. Launch the pipeline at `localhost:8081`. 8. To avoid using port `8081` for future pipelines, SingleStore recommends changing the port to another preferred Web port by navigating to **Settings** > **Instance Details** > **Web Port**. 9. Restart the Windows service and launch the Ingest application on the new Web port, such as `localhost:8082`. 10. Ensure you have a valid license key to get started. ## Enable Windows Authentication for SQL Server Use the following steps to enable Windows authentication in Ingest. 1. Stop the Ingest service. 2. Download and copy the `sqljdbc_auth.dll` file to the `bin` directory of your Java installation. You can download the DLL from [here](https://bryteflow.com/release/files/sqljdbc_auth.zip) or from [Microsoft](https://learn.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server). 3. Start the Ingest service and add the following JDBC option in the **Source Database** connection settings. ``` integratedSecurity=true ``` ## Latency Calculation Latency is calculated based on the time from when the record is committed to the source until it is available at the destination. ## Prerequisites for Oracle Source The Flow installation guide provides details on how to enable supplemental logging for source table(s) for delta replication. Refer to [Install SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow.md) for more information. **Note**: * If database supplemental logging is enabled, only inserts and deletes are captured, and updates are missed. * If table-level supplemental logging is enabled, only updates are captured, and inserts and deletes are missed. Therefore, both database and table-level supplemental logging are required. ## Synchronize Schema Changes Using Sync Struct Use the following steps to perform a Sync Struct for existing tables when the source structure changes. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Navigate to the **Operations** tab and select **Sync Struct**. 3. Navigate to the **Logs** tab to check the logs. Ingest compares the structures and synchronizes any differences. ## Use of BCP by Flow is not supported on Linux for SQL Server Use the following steps to disable BCP when enabling change data capture or change tracking on SQL Server running on Linux: **Note**: It is recommended to test this in a non-production environment before deploying to production. 1. Check if Ingest is currently running any extract or loads. If no extracts are running, stop the Ingest service. 2. Navigate to the Ingest installation directory. Locate the `config.xml` file in: `/ingest/config/config.xml.` 3. Open the file in a text editor. In the `` section, and set the "`use_bcp`" parameter to `"N"`: ```xml ``` If the parameter does not exist, add the line manually. 4. Save the file. 5. Restart the Ingest service. ## Loading Data into Reference Tables Loading data into reference tables using Flow requires using the DDL endpoint for destination connection. Using the DML endpoint may result in an error like: ``` LOAD DATA into reference table on a child aggregator is not permitted on child aggregators. Try the command again on the master aggregator.Loading data into reference tables using Flow requires using the DML endpoint for destination connection. Using the DDL endpoint may result in an error like: ``` To resolve this: 1. Identify the cluster group ID of your SingleStore Helios cluster. 2. In the Flow instance, update the hostname for the SingleStore Helios destination connection by replacing `svc--dml` with `svc--ddl` so that the `id` is replaced with the cluster group ID and the `dml` tag is changed to `ddl`. Keep the rest of the hostname unchanged. Example: ``` svc-78636867-cf31-4b41-4765-24a3997fd429-ddl.aws-ireland-2.svc.singlestore.com ``` 3. Test the connection again. It should now allow ingesting data into the reference table. ## Load Data from Views Loading data from views is not supported through the Ingest UI. For self-managed deployments, you can use a `config.xml` workaround to perform a **one-time full extract** from a view. CDC (change data capture) from views is not supported. * Use a separate Flow instance dedicated to views, configured only for the one-time full extract. * As views do not define primary keys, configure the destination table with either a unique indexed column as the primary key, or multiple columns that together form a natural key. **Note**: This workaround is not recommended for production use without prior testing. To load data from a view in a full extract pipeline, add the view as a table entry in `config.xml` as follows: 1. Stop the Ingest service and ensure that no extracts are running. 2. Open the following configuration file: ```shell /ingest/config/config.xml ``` 3. In the `` section, add a `
` entry for the view using the following template: ```
``` 4. Save `config.xml`. 5. Restart the Ingest service. 6. From the Ingest UI, run a **Full Extract** operation. Flow treats the view as if it were a table and loads its result set into the destination table. *** Modified at: July 6, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/) (An index of the documentation is available at /llms.txt) --- # SingleStore Ingest ## Overview SingleStore Ingest (“Ingest”) is real-time data replication software that replicates data from various sources to SingleStore. It is one of the primary components of SingleStore Flow. Ingest offers high performance, enabling real-time Change Data Capture from sources with zero load on the source systems. It captures changes and transfers them to the target system. It automates the creation of either an exact copy or a time-series copy of the data source in the target. It first performs a full initial load from the source, then incrementally merges changes to SingleStore. The entire process is fully automated. ## Supported Source Databases Ingest supports the following database sources: * Oracle * Microsoft SQL Server * MySQL * PostgreSQL * Snowflake * Any JDBC-complaint database (non-production preview, available for testing) * MongoDB (private preview) Contact your SingleStore account team or [SingleStore Sales](https://www.singlestore.com/contact/) if you want to move data from a source not listed above. ## Ingest Architecture Ingest replicates data from any supported source to a SingleStore destination database. It is a fully self-service, automated data replication tool. SingleStore Flow, of which Ingest is a part, offers several deployment strategies for its customers, including: * Standard deployment in an AWS environment * High Availability deployment in an AWS environment * Hybrid deployment using both on-premises and cloud infrastructure * Fully on-premises deployment Flow components can be deployed in Google Cloud and Microsoft Azure as well. AWS components and services are referenced here as an illustration of a common type of deployment. Ingest uses log-based Change Data Capture for data replication. The following is the technical architecture diagram that illustrates the standard setup in an AWS environment. The following diagram serves as the reference for all setup instructions. Estimated deployment time: Approximately 1 hour ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt3208fa2b1092e81e/6a7adb25594711dd234c267d/singlestore_flow_ingest_architecture-EgW798.png) ## Ingest / AWS Service Integration The following is the Ingest architecture which showcases integration with various optional AWS services in a standard deployment. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9341d4dac76d1021/6a7adbe342a1175ec6950983/singlestore_flow_ingest_aws_service_integration-uBfRFF.png) This architecture diagram illustrates a standard deployment that highlights the following features: * AWS services running alongside Ingest. * Recommended Flow architecture for a VPC in AWS. * Data flow between the source database, AWS, and SingleStore destination database, including security and monitoring features. * Security, including IAM, organized in a separate group and integrated with Ingest. ## Ingest High Availability Architecture The following High Availability architecture explains how Ingest is deployed in a multi-AZ setup. In the event of an instance or AZ failure, it automatically scales to another AZ without incurring any data loss. Estimated deployment time: Approximately 1 day ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9313e76f226c7dff/6a7adbe2c5d19301135db91d/singlestore_flow_ingest_ha_architecture-0KXTtm.png) ## Ingest Hybrid Architecture Ingest also offers a hybrid deployment model that combines on-premises services with those in the AWS Cloud. Ingest can be easily set up on a Windows server in an on-premises environment. The SingleStore destination endpoint resides in the AWS Cloud that creates a hybrid model. SingleStore recommends secure connectivity between on-premises and AWS services, which can be achieved using a VPN connection or AWS Direct Connect. Estimated deployment time: Approximately 2 hours to 1 day ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt84486e87b38a6464/6a7adb1d77b03469dc3f3fa1/singlestore_flow_ingest_hybrid_architecture-WbOrIR.png) ## In this section * [Prerequisites](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/prerequisites.md) * [Source Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/source-database-setup.md) * [Destination Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/destination-database-setup.md) * [Start and Stop Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/start-and-stop-ingest.md) * [Configure Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/configure-ingest.md) * [Dashboard](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/dashboard.md) * [Configure Source Database](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/configure-source-database.md) * [Configure Destination Database](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/configure-destination-database.md) * [Tables](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/tables.md) * [Operations](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/operations.md) * [Settings](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/settings.md) * [Reports](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/reports.md) * [Logs](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/logs.md) * [AWS Environment Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/aws-environment-setup.md) * [Appendix](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/appendix.md) *** Modified at: September 1, 2025 Source: [/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest/) (An index of the documentation is available at /llms.txt) --- SingleStore XL Ingest ================== ## Overview SingleStore XL Ingest ("XL Ingest"), a component of SingleStore Flow, is companion software to [SingleStore Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest.md) ("Ingest"). When large tables (greater than 10GB) are included in Ingest, a full extract may not be feasible due to long processing times. This increases the likelihood of encountering issues, and if a problem occurs, the entire initial extract must be rerun. Therefore, XL Ingest is essential for working with large tables. XL Ingest handles the initial transfer of large tables by dividing them into smaller logical partitions. It then transfers multiple partitions from the source to the target in parallel. This ensures the transfer happens within a reasonable amount of time. Use XL Ingest to transfer the identified logical partitions of large tables in parallel to SingleStore for storage and processing. To transfer large tables from source database to SingleStore Helios, perform the steps outlined in [Select Tables](https://docs.singlestore.com/#section-idm234737541647291.md). Complete these steps before triggering an XL Ingest job to prevent data loss during the transition. **Note**: Pausing updates on the source database is not required during this process. Both Ingest and XL Ingest can operate independently and concurrently without disrupting the source data. ## Select Tables To ingest large data table(s) from source database to SingleStore, perform the following steps in Ingest before triggering a job in XL Ingest. 1. Navigate to **Dashboard** > **Tables** and select the gear icon. 2. Define a primary key (**Pkey**) and any necessary partitions for the table. 3. Enable **Skip Initial Extract** to bypass the initial extract and directly proceed with the delta load. 4. Select **Apply** to save the changes. 5. Navigate to the **Dashboard** tab. Under **Schedule**, disable the Ingest scheduler to ensure that all tables are moved to the destination at the same time. 6. Navigate to the **Operations** tab and initiate **Full Extract** to trigger the initial bulk load for all the selected tables, except for those tables marked as **Skip Initial Extract**. 7. Initiate **Sync New Tables** to trigger the initial bulk load for tables marked as **Redo Initial Extract** and newly-added tables in an ongoing replication. This captures the watermark for CDC and creates the table in the destination database (SingleStore). 8. Enable the ingest scheduler in Ingest after transferring tables using XL Ingest. **Note**: After marking tables with **Skip Initial Extract**, the next scheduled delta run automatically captures CDC for all tables, including those loaded with XL Ingest. XL Ingest prevents duplication during the CDC load by using a watermark to track changes. ## Split Table into Slices Large tables must be divided into notional slices based on the value of a single slice column, for example, primary key. For automatic slice determination, XL Ingest uses parameters like the number of slices needed and how many characters from the start of the slice column value must be used. **Note**: Date and time columns (for example `DATE`, `DATETIME`, `TIMESTAMP`) are not supported as slice columns in XL Ingest. Refer to [Configure Table Slicing](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/configure-table-slicing.md) for more information on slicing. ## For Large Tables XL Ingest automatically determines the slices based on the specified parameters. For example, a slice column like `names` can be divided by the first 3 characters. Alternatively, you can manually enter the slice values instead of using auto-slice. ## For Smaller Tables Slicing may not be necessary. The entire table can be processed as a single slice. ## For PostgreSQL Partitioned Tables To use XL Ingest with PostgreSQL partitioned tables: 1. In Ingest, select only the parent PostgreSQL partitioned table on the **Tables** page. Child partitions do not appear separately. 2. Configure the primary key and other table options on the parent table. 3. Set up slicing on the partition key column using Auto-Slice or by manually entering slice boundary values. 4. Sync the table in XL Ingest to load all slices in parallel. 5. After XL Ingest completes, enable the Ingest scheduler so CDC continues from the same parent table. ## In this section * [Start and Stop XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/start-and-stop-xl-ingest.md) * [Configure XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/configure-xl-ingest.md) * [Configure Table Slicing](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/configure-table-slicing.md) *** Modified at: May 28, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest/) (An index of the documentation is available at /llms.txt) --- # Upgrade SingleStore Flow The following sections contain step-by-step instructions for upgrading Ingest and XL Ingest. ## Upgrade Ingest Use the following steps to upgrade Ingest without losing connection settings, current progress, or scheduled operations. 1. Confirm that Ingest is not currently running an extract. If no extracts are running, turn off the scheduler. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Open **Windows Services** and stop the Ingest Windows service. Refer to [Stop Ingest ](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/#section-idm234737476643798.md)for more information. 2. Change to the current Ingest installation directory. 3. Rename the `ingest.jar` file to something else, such as `ingest.jar.` where ` 5. Move the `ingest.jar` file to the current Ingest folder. 6. Open **Windows Services** and start the Ingest Windows service. 7. To verify that the upgrade is successful, note the **Version** and **Build** number on the top left corner of the **Dashboard** or navigate to **Settings** > **About Ingest**. It can also be seen in the **Logs** tab, under **Others**. 8. Navigate to the **Dashboard** tab, under **Schedule**, and enable the scheduler. For more information about common operational user queries, refer to [SingleStore Ingest Operational FAQ](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq.md). ## Upgrade XL Ingest Use the following steps to upgrade XL Ingest without losing connection settings, current progress, or scheduled operations. 1. Confirm that XL Ingest is not currently running an extract. If no extracts are running, turn off the scheduler. 1. Navigate to the **Dashboard** tab. Under **Schedule**, disable the scheduler and wait for the load to complete. 2. Open **Windows Services** and stop the XL Ingest Windows service. 2. Change to the current XL Ingest installation directory. 3. Rename the `xlingest.jar` file to something else, such as `xlingest.jar.`, where `` is the date on the `.jar` file. 4. Download the latest XL Ingest JAR file: 5. Move the `xlingest.jar` file to the current XL Ingest installation folder. 6. Open **Windows Services** and start the XL Ingest Windows service. 7. To verify that the upgrade is successful, navigate to the installation directory, open the **log** folder, and check the last entry in the `sirius-yyyy-mm.log` file. 8. Navigate to the **Dashboard** tab, under **Schedule**, and enable the scheduler. *** Modified at: May 1, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/upgrade-singlestore-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/upgrade-singlestore-flow/) (An index of the documentation is available at /llms.txt) --- # Load Data with SingleStore Flow on Helios > **📝 Note**: This is a Preview feature. ## Overview SingleStore Flow (or, "Flow") is SingleStore's data migration and change data capture (CDC) solution which is comprised of two primary components: * **SingleStore Ingest**, also referred to as "Ingest," transfers the schema for all tables, moves table data for tables up to 10GB in size, and handles ongoing CDC for all tables into SingleStore. * **SingleStore XL Ingest**, also referred to as "XL Ingest," handles the initial transfer of large tables by breaking them into smaller logical partitions. It then copies multiple partitions from the source to the target in parallel. This helps transfer large tables efficiently, and is resilient to network outages and restarts of Flow, the source, or the destination. Flow is integrated into the [Cloud Portal](http://portal.singlestore.com/) under **Ingestion** > **Load Data**. This service enables loading and synchronizing data from supported databases into SingleStore, without the need to install or manage any infrastructure. Flow supports the following source databases for transfer to SingleStore: * Oracle * Oracle RAC * Microsoft SQL Server * MySQL * PostgreSQL * Snowflake * Any JDBC-compliant database (non-production preview, available for testing) * MongoDB® (private preview) ## Why Use SingleStore Flow Key benefits of using SingleStore Flow: * **No-code setup**: Configure end-to-end migration via the Flow UI. * **Automatic schema handling**: Analyze the source database and automatically create the required schema in the target SingleStore database. * **Real-time visibility**: Monitor progress, view live logs, and troubleshoot quickly. * **High throughput**: Use XL Ingest to parallelize large table loads by chunking source data, significantly reducing migration time. ## Choose the Right Flow Deployment Flow is available in two deployment options: * A Self-Managed Flow Application * Flow on Helios Both provide the same data migration and CDC features, but differ in how they are deployed and used. The following table outlines these two options: | Deployment Option | Self-Managed Flow Application | Flow on Helios | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Description** |
  • Installed and operated in your own environment.
  • Self-managed, installation required.
  • Provides control over where the application runs and how it integrates with your system.
|
  • Runs in the SingleStore Helios cloud deployment.
  • Fully managed by SingleStore, no installation required.
  • A Flow instance is provisioned directly from the UI, and usage is billed only while the instance is running.
| | **When to Use** |
  • When deployment placement or control over environment matters the most, such as replicating large volumes of data from an on-premises source to SingleStore. This is because Flow performs best for high-volume transfers when its services run close to the data source.
|
  • When your data sources and destinations are in the same cloud region, along with your Flow on Helios instance, ensuring excellent performance along with convenience and robustness.
  • When convenience and low operational overhead matter more than placement.
| For more details on the Self-Managed Flow application, refer to [Load Data with SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow.md). ## In this section * [Flow on Helios Sizing Guidelines](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-sizing-guidelines.md) * [Use Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/use-flow-on-helios.md) * [Scale a Flow Instance](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/scale-a-flow-instance.md) * [SingleStore Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest.md) * [SingleStore XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest.md) * [Data Type Mapping in SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/data-type-mapping-in-singlestore-flow.md) * [Monitor Flow on Helios with Alerts](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/monitor-flow-on-helios-with-alerts.md) * [Flow on Helios API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api.md) * [Flow on Helios FAQ](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-faq.md) *** Modified at: May 11, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/) (An index of the documentation is available at /llms.txt) --- # Data Type Mapping in SingleStore Flow The following mappings define how data types from source databases are converted to their corresponding SingleStore types during ingestion with Flow. ## In this section * [MySQL to SingleStore Data Type Mapping](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/data-type-mapping-in-singlestore-flow/mysql-to-singlestore-data-type-mapping.md) * [PostgreSQL to SingleStore Data Type Mapping](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/data-type-mapping-in-singlestore-flow/postgresql-to-singlestore-data-type-mapping.md) *** Modified at: December 9, 2025 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/data-type-mapping-in-singlestore-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/data-type-mapping-in-singlestore-flow/) (An index of the documentation is available at /llms.txt) --- # Flow on Helios API > **📝 Note**: This is a Preview feature. The Flow on Helios API allows you to create, configure, and manage Flow pipelines, enabling seamless integration with third-party applications and workflow automation. It provides an HTTP interface that primarily follows REST conventions, while also supporting RPC-style command endpoints for specific operations. Flow on Helios provides two APIs: [Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/flow-on-helios-ingest-api.md) API for data transfers up to 10 GB, and [XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/flow-on-helios-xl-ingest-api.md) API for larger tables. ## In this section * [Flow on Helios API Authentication](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/flow-on-helios-api-authentication.md) * [Flow on Helios Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/flow-on-helios-ingest-api.md) * [Flow on Helios XL Ingest API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/flow-on-helios-xl-ingest-api.md) *** Modified at: January 15, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-api/) (An index of the documentation is available at /llms.txt) --- # Flow on Helios FAQ **Do I need a license key to use Flow on Helios**? No, a license key is not required. You are only billed for the time your Flow instance is running. **How does billing for Flow work, and how do I pay for it?** Flow on Helios uses a pay-per-usage model. You pay only for the time your Flow instance is running. Charges are deducted from your Helios credits, so no separate payment is needed. **Where can I view the logs?** On the [Cloud Portal](http://portal.singlestore.com), go to **Ingestion** > **Load Data**, then select your Flow pipeline and click **Details**. When the dashboard opens, the **Logs** tab can be viewed on the right side of the toolbar. **Which IP addresses do I need to whitelist on the source database?** You must whitelist the outbound IP addresses of your SingleStore cluster in your source database’s network configuration. To find them, navigate to **Clusters**, open the **Actions** list (select the three dots) for your cluster, and then select **Access & Security**. Under **Firewall**, the IP addresses are listed in the **Outbound** tab. **Why am I getting the error: "Unable to connect to destination database"?** Please verify that your username and password are correct in the Destination Database configuration. To reset your SingleStore database password, navigate to **Clusters**, select **Connect > Your App** for your cluster, and then select **Reset password**. **If I reset the password for my SingleStore cluster, do I need to update it in all my Flow instances?** Yes, you are required to update the password in all your Flow pipelines with the new password. **I am unable to view the destination database while creating a Flow pipeline, even though my cluster is active and a database is already attached. Why is that? ** This issue may be due to firewall restrictions. Verify your firewall settings and add your current IP address to the list of allowed inbound IPs to ensure database access. **Can I connect to my source database using Flow via private links?** Yes. To connect, create an outbound private link in the Cloud Portal. Refer to [Configure Outbound Connections](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/#section-idm4587784992955233710635591742.md) for more information.  **How can I copy all the source tables to a SingleStore database of my choice?** On the Flow dashboard, go to the Destination Database configuration tab and select **Advanced Options**. Enter the database name in the **Database name and optional table name pattern** and **Database for staging tables** fields. **Can I define a schema for tables before extraction?** Yes. There are two ways to do this: Option 1:  1. Create all the tables using custom SQL queries. 2. Go to the Destination Database configuration tab, select **Advanced Options** and select **Truncate table instead of drop**. Option 2: 1. Select the tables you want to move to SingleStore and enable **Skip Initial Extract**.  2. Go to **Operations** and do a **Full Extract**. This creates the tables in the SingleStore database. 3. Once the tables are created, modify them to add shard or sort keys.  4. After modification, go to the Destination Database configuration tab, select **Advanced Options** and select **Truncate table instead of drop**. **How to copy just the schema of source before migration?** Select the tables you want to move to SingleStore and enable **Skip Initial Extract**. Go to **Operations** and do a **Full Extract**. This creates the tables in the SingleStore database. **When a connection fails, the error message says "Connection string is invalid. Unable to parse". How can I identify the issue?** This error typically indicates that you have spaces in any of the configuration fields or the hostname, port or database name is incorrect. Check these fields for formatting issues or incorrect values and try again. **My scheduled pipeline didn’t trigger - why?** This can happen if the scheduler is turned off or misconfigured, for example, set to run at 00h 00m 00s. In case of file-based replication on source (e.g., MySQL or Oracle Log Miner), Ingest may be waiting for the next log file to be created. **I have a database with 1TB of data. I want to migrate all the data to SingleStore and enable CDC for new transactions to my source table. How do I proceed?** For this use case, both Ingest and XL Ingest are needed. Follow these steps for the migration: 1. Identify tables greater than 5GB.  2. Select the tables from the list and select **Skip Initial Extract**. 3. Go to **Operations** and do a **Full Extract**. This creates the selected tables in SingleStore without any data.  4. Verify that the tables are created in SingleStore. 5. Select XL Ingest from the dropdown list on the top right of the dashboard.  6. Migrate the tables using XL Ingest. Refer to [SingleStore XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest.md) for more information.  7. Once the migration is done, go to Ingest and select all the tables. 8. Go to **Operations** and select **Sync New Tables**. This migrates smaller tables first and then starts the CDC for all the tables.  **I want to update my scheduler. How do I do it?** To update the scheduler, go to the Schedule tab, update the configuration, and click **Apply**. Note that updating the scheduler is only applicable to time-based replication (MySQL Continuous Log Miner, Oracle Log Miner, SQL Server) not for file-based replication.  **Error messages appear at the top of the screen and disappear quickly, making them hard to notice. How can I see them more easily?** You can go to the Logs tab and view all the errors for your instance. **I configured firewall rules to allow IP access, but I still cannot connect Flow to my source database. What could be the issue?** Ensure that the correct outbound IP addresses from your SingleStore cluster have been added to your source database's network configuration. Also, make sure the network allows connections on the port your database is using. **I have successfully established a connection, but I'm encountering authentication errors. Why is the connection failing after setup?** Flow does not automatically update database passwords if they are changed after the connection is created. If the username or password for your source or destination database has been modified, you may receive authentication errors. To resolve this, update the credentials in Flow and re-test the connection. **Why does Flow create a new database when I have specified a destination database during pipeline creation?** By default, Flow creates a new database with the same name as the source database name. To avoid this, go to the Destination Database configuration tab, select **Advanced Options **and specify the database name in the field **Schema for all tables**. **How can I monitor the progress of data ingestion without accessing logs?** You can track the ingestion progress from the Flow dashboard. The extraction progress bar provides real-time information on how much data has been transferred and how much is still pending. **How can I load multiple source tables into one target table in Flow?** Currently, Flow does not support loading multiple source tables into a single target table. **How can I add a prefix to the database names migrated from source to SingleStore?** Go to the Destination Database configuration tab, select **Advanced Options**, and add a prefix of your choice in the **Add Database Prefix** field. **I see `eff_dt` and `end_dt` column errors in the logs, how can I resolve this?** These errors typically occur when **Maintain History** was enabled during the initial extract but later disabled. This leads to a schema mismatch between the source and the target (SingleStore) database. To resolve this, you can choose one of the following options: * Delete the `eff_dt` and `end_dt` columns from your SingleStore database. * Drop the table from your SingleStore database, select **Redo Initial Extract** for the table, then go to **Operations** and select **Sync New Tables**. **Is the ENUM data type supported in Flow's data type casting?** No, Flow does not currently support the ENUM data type in data type casting. **How to load data into reference tables?** Loading data into reference tables using Flow requires using the DDL endpoint for destination connection. Using the DML endpoint may result in an error like: ``` LOAD DATA into reference table on a child aggregator is not permitted on child aggregators. Try the command again on the master aggregator.Loading data into reference tables using Flow requires using the DML endpoint for destination connection. Using the DDL endpoint may result in an error like: ``` To resolve this: 1. Identify the cluster group ID of your SingleStore Helios cluster. 2. In the Flow instance, update the hostname for the SingleStore Helios destination connection so that the `id` is replaced with the cluster group ID and the `dml` tag is changed to `ddl`. Keep the rest of the hostname unchanged. Example: 3. Test the connection again. It should now allow ingesting data into the reference table. ``` svc-78636867-cf31-4b41-4765-24a3997fd429-ddl.aws-ireland-2.svc.singlestore.com ``` **What happens when a new table is added to the source database, and how to start syncing it with Flow?** If a new table is created in the source database, Flow continues running without errors until that table is explicitly selected in the Flow dashboard. If the table is selected but no full extract has been performed and there are no changes on the source table yet, no errors are observed since there is nothing to extract or load. When changes occur on the source table before a full extract has been run, Flow reports errors such as "`No columns found for table …`" for that table. **Note**: **Sync Struct** detects schema changes only when the number of columns changes. If the number of columns remains the same (for example, a column rename without adding or dropping a column), **Sync Struct** does not detect the change automatically. To start syncing the new table: 1. Turn off the scheduler for the Flow instance. 2. Select the new table in the **Tables** configuration. 1. If the table is large (for example, larger than 5 GB), select **Skip Initial Extract** so that only the schema is created initially. 3. Go to **Operations** and select **Sync New Tables**. 1. If the table is marked as **Skip Initial Extract**, **Sync New Tables** creates an empty table on the destination. Use XL Ingest to load the data while the scheduler is off, and turn the scheduler back on so CDC can resume. 2. If the table is not marked as **Skip Initial Extract**, **Sync New Tables** runs a full extract and then starts CDC for the new table. **What is the recommended procedure for adding a new column to an existing table?** To add the new column safely: 1. Allow delta runs to complete so the table is fully synchronized. 2. Turn off the scheduler for the Flow instance. 3. Add the column on the source table. 4. On the **Operations** page, click **Sync Struct** to propagate the schema change to the destination. After **Sync Struct** completes successfully, CDC resumes with the updated schema. **How are column renames handled, and how to recover from related errors?** Column renames are not handled automatically by **Sync Struct**. If a column is renamed on the source without recreating the table in Flow, CDC fails with errors indicating a missing column, for example: * `Error(SOP615: Error details(PGT356): Unable to find column ... - possible structure change` To recover from this error and resume CDC: 1. Identify the table where one or more columns have been renamed. 2. Ensure the scheduler is turned off. 3. In the **Tables** configuration, select the table and enable **Redo Initial Extract**. 4. Go to **Operations** and run **Sync New Tables**. This: * Drops the existing destination table. * Recreates it with the updated schema (including the renamed column). * Reloads all data for that table from the source. To avoid a full reload, the following workaround can be used: * Turn off the scheduler. * Manually rename the column in the SingleStore database. * Roll back to the extract just before the error occurred. * Turn the scheduler back on. **Do schema changes in ETL reporting tables require Flow configuration changes?** Flow does not affect ETL tables (for example, `*_v2` tables), as these are loaded via stored procedures. If a column needs to be added to an ETL table (such as `outcome_v2`, `config_v2,` or `item_results_v2`), no changes are required in the Flow configuration. **Is it possible to add or modify a shard key column on an existing ETL table?** Shard keys cannot be changed on an existing table.  **How to perform a full resync of my entire environment (all clusters and Flow instances)?** A full environment resync reloads all data in the cluster, across all clusters and all Flow instances connected to that environment. A **Full Extract** drops and recreates the tables unless the **Truncate table instead of drop** option is selected. If **Truncate table instead of drop** is selected, the tables must be dropped manually. To reload an entire environment: 1. (Optional) Back up any required tables or databases. 2. Drop the existing tables or the entire database for the environment that needs to be reloaded.  **Note**: This step is only required if **Truncate table instead of drop** is enabled; otherwise, **Full Extract** drops tables automatically. 1. **Recommended**: Drop the existing Flow instance and create a new one. If this is not possible, reuse the same instance but treat it as a fresh setup. 2. Reconfigure the pipeline following the standard new customer setup documented in [Use Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/use-flow-on-helios/) (source configuration, destination configuration, tables, schedule, and initial full extract). **How to perform a full resync for a single customer? ** In Ingest, a single-customer resync reloads all tables for one specific customer (for example, one RDS instance or logical customer environment), without affecting other customers. To reload the data for a single customer: 1. (Optional) Back up the existing tables for that customer in SingleStore. 2. If **Truncate table instead of drop** is enabled for the pipeline, drop the customer’s tables from SingleStore manually; otherwise Flow drops and recreate them during the full extract. 3. In the **Tables** configuration, select all tables for that customer and enable **Redo Initial Extract**. 4. Go to **Operations** and run **Full Extract** to reload the data for those tables from the source. CDC resumes from the latest successful delta after the full extract completes. **Can I resync data for only a specific tenant or a subset of rows?** A tenant is a logical grouping of data - for example, a customer, test, or administrative environment. Tenants are mapped to clusters that share a cluster group (cluster ID). Flow resync operations are defined at the table or database level. Currently, Flow does not support resyncing only a subset of rows within a table. Resync is supported only for full tables or entire databases. **Workaround** Although Flow cannot resync only a subset of rows, you can approximate this behavior as follows: * Delete the affected rows from the table in SingleStore. * Identify the primary key values for those rows, and use XL Ingest to re-extract them, either by slicing or by adding the primary key to the `WHERE` clause. **How to resync only specific tables for a customer? ** Selective table resync reloads one or more chosen tables without affecting the rest of the pipeline, unlike a single-customer resync, which reloads all tables for that customer. To resync only selected tables: 1. (Optional) Back up the existing tables to be reloaded. 2. If **Truncate table instead of drop** is enabled for the pipeline, drop only the tables that need to be reloaded from SingleStore, otherwise Flow drops and recreates them during the full extract. 3. In the **Tables** configuration, select those tables and enable **Redo Initial Extract**. 4. (Optional) If only a subset of rows needs to be re-extracted during the full extract, add a `WHERE` clause filter for those tables.  **Note**: Enter only the filter condition without the `WHERE` keyword. 5. Go to **Operations** and run **Full Extract**. The `WHERE` clause applies only to the full extract; subsequent CDC runs continue to ingest all changes for those tables, regardless of the filter. *** Modified at: May 13, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-faq/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-faq/) (An index of the documentation is available at /llms.txt) --- # Flow on Helios Sizing Guidelines ## Overview Sizing guidelines help users make decisions on selecting the right Flow instance size on SingleStore Helios for their migration or data movement jobs. ## Size a Workload When selecting a Flow instance, evaluate your workload carefully. Consider the following key parameters: 1. **Data Size** * The total volume of data that needs to be migrated or streamed. * Includes both historical data and incremental changes. * Larger datasets generally require larger Flow instances to achieve migration in a reasonable timeframe. 2. **Number of Tables** * The count of tables involved in the job/pipeline. * A higher number of tables increases concurrency requirements and can benefit from larger Flow instances. 3. **Size of the Biggest Table** * The largest table in your workload may become a bottleneck. * If the largest table is very large (hundreds of GBs or TBs), it may dictate the minimum Flow size you need. 4. **Number of Threads on Source Database** * Determines the maximum parallelism you can achieve when reading from the source. * If your source can’t provide enough threads, increasing Flow instance size may not help. 5. **Estimated Time to Complete the Job** * The timeframe in which you want the migration to finish (e.g., overnight, within a maintenance window, or real-time). * Tighter SLAs require more compute and memory resources. ## Sizing Examples | Workload Type | Data Size | Number of Tables | Biggest Table Size | Threads Available | Completion Time Target | SuggestedFlowInstance | | --------------------------------- | ----------------------------- | ---------------- | ------------------------ | ----------------- | ------------------------ | --------------------- | | Small | \~50 GB | 10 | \~8 GB | 4 | 12 - 24 hours | f - 1 | | Medium | \~500 GB | 50 - 75 | \~100 GB | 8 - 12 | 6 - 8 hours | f - 3 | | Large | \~2 TB | 200+ | \~400 GB | 16 | \~6 hours | f - 6 | | Very Large/Enterprise | \~10 TB | 500+ | \~1 TB | 32+ | < 4 hours (tight SLA) | f - 8 | | Latency Sensitive/ Real-time Sync | \~300 GB/day (ongoing stream) | 20 - 30 | \~200 GB (growing daily) | 12 - 16 | Continuous / < 1 min lag | f - 4 | **Additional Notes** * **Thread Utilization on Source Database Server** Increasing the number of threads used on the source database server proportionally increases the load on system resources (CPU, memory, I/O). Proper sizing and monitoring are recommended to prevent performance degradation. * **Completion Time Target** The Completion Time Target represents an estimated duration derived from internal performance testing. Actual completion times may vary depending on the source database server's ability to transmit records at the required rate. * **Data Flow Behavior** During the initial data load, the system performs a bulk load operation in a sequential manner to ensure data consistency and reliability. * **Handling Large Tables** For very large tables, SingleStore recommends using XL Ingest to optimize throughput and reduce load time. *** Modified at: September 29, 2025 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-sizing-guidelines/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-sizing-guidelines/) (An index of the documentation is available at /llms.txt) --- # Monitor Flow on Helios with Alerts Flow alerting in SingleStore Helios monitors the health and activity of Flow instances associated with a cluster. Flow-specific trigger conditions are configured on the Alerts page for a cluster, where severity levels, thresholds, and email subscribers for each alert can be set. When Flow alerts are enabled for a cluster, the configured alert rules apply to all Flow instances mapped to that cluster. When a rule's threshold is met for any Flow instance, Helios sends email notifications to the cluster's alert subscribers. The Flow-specific trigger conditions are: * **Flow CPU Utilization:** Triggers an alert when CPU usage for at least one instance is above a configured threshold for a specified duration. * **Flow Memory Utilization**: Triggers an alert when memory usage for a Flow instance is above a configured threshold for a specified duration. * **Flow Inactivity**: Triggers an alert when a Flow instance has not moved any records for a configured period of time. * **Flow Scheduler Error**: Triggers an alert when the scheduler for a Flow instance enters an error state. For general alerting concepts (severity levels, thresholds, and subscribers), refer to [Alerts](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/alerts.md). ## Prerequisites * Flow on Helios version 4.3.6 or later. ## Configure Flow Alerts To configure Flow alert rules for a cluster: 1. In the Cloud Portal, navigate to **Monitoring** > **Alerts**. 2. At the top of the page, select the cluster group that contains your cluster. 3. In the **Alerts Configuration** table, locate the row for your cluster. 4. In the **Actions** column for that cluster, click the Edit icon. This opens the alert configuration, including Flow-related trigger conditions for that cluster. 5. In the **Edit Alert Rules** dialog, scroll to the Flow alert types: * Flow CPU Utilization * Flow Memory Utilization * Flow Inactivity * Flow Scheduler Error 6. For each Flow alert type that is required: 1. Enable it 2. Configure the severity levels (**Critical**, **Warn**, **Info**) with appropriate thresholds and durations 3. For Flow Scheduler Error, toggle it on (it functions as an on/off rule) 7. In the **Subscribers** section of the alerts configuration, add or update the list of email addresses that should receive alerts for this cluster.  8. Click **Save** to apply the alert configuration.  Once saved, the Flow alert rules are enforced for every Flow instance mapped to the cluster. When a configured condition is met, Helios sends email notifications according to the severity and the existing alerting behavior documented for clusters. *** Modified at: May 12, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/monitor-flow-on-helios-with-alerts/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/monitor-flow-on-helios-with-alerts/) (An index of the documentation is available at /llms.txt) --- # Scale a Flow Instance Flow on Helios instances can be resized after creation to match workload requirements and control credit consumption. Instance sizing is based on Flow instance size tiers (F1 - F3), each with a defined vCPU, memory, and credit rate per hour. When selecting a new size, consider the following: * **vCPU ratio to cluster**: Choose a Flow instance size so that Flow vCPUs are approximately one quarter of the vCPUs in the destination cluster (a 1 : 4 Flow-to-cluster vCPU ratio). This generally provides good performance balance between ingestion and query workloads. * **Workload characteristics**: If the size and characteristics of the data are already well understood (data volume, number of tables, largest table size, expected concurrency, and required completion time), adjust the instance size accordingly to better fit the workload. For detailed sizing guidance by workload type, including example scenarios and recommended sizes, refer to [Flow on Helios Sizing Guidelines](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/flow-on-helios-sizing-guidelines.md). ## Flow Instance Sizes The following table lists the available Flow instance sizes and their resources: | Size | vCPU | Memory | Credits per Hour (CR) | | ---- | ---- | ------ | --------------------- | | F1 | 2 | 4 GB | 0.123 CR | | F2 | 4 | 8 GB | 0.246 CR | | F3 | 8 | 16 GB | 0.492 CR | | F4 | 16 | 32 GB | 1.084 CR | | F5 | 32 | 64 GB | 2.069 CR | | F6 | 64 | 128 GB | 4.138 CR | | F7 | 96 | 384 GB | 9.643 CR | | F8 | 192 | 768 GB | 18.731 CR | Depending on environment and preview configuration, higher tiers (for example, F4 and above) may appear disabled in the dialog. In that case, only the enabled sizes are available for selection. ## Scale a Flow Instance in the Cloud Portal To change the size of an existing Flow on Helios instance: 1. In the Cloud Portal, navigate to **Load Data** > **Flow**. 2. In the **Actions** column for the instance to scale, click the three dots (…) and select **Scale**. 3. In the **Scale Flow Instance **dialog: 1. Review the recommendation text and confirm the target size based on the destination cluster vCPUs and workload characteristics. 2. Select the desired Flow instance size from the **Size** column. 3. Review the associated vCPU, Memory, and Credits Per Hour values. 4. **Save** the change in the dialog to apply the new instance size. After the change is applied, the Flow instance runs with the newly selected size, and subsequent usage is billed at the corresponding credit rate. For more information on how instance size affects cost, refer to [Helios Pricing](https://www.singlestore.com/cloud-pricing/). *** Modified at: July 6, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/scale-a-flow-instance/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/scale-a-flow-instance/) (An index of the documentation is available at /llms.txt) --- # SingleStore Ingest ## Overview SingleStore Ingest (“Ingest”) is real-time data replication software that replicates data from various sources to SingleStore. It is one of the primary components of SingleStore Flow. Ingest offers high performance, enabling real-time Change Data Capture from sources with zero load on the source systems. It captures changes and transfers them to the target system. It automates the creation of either an exact copy or a time-series copy of the data source in the target. It first performs a full initial load from the source, then incrementally merges changes to SingleStore. The entire process is fully automated. ## Supported Source Databases Ingest supports the following database sources: * Oracle * Microsoft SQL Server * MySQL * PostgreSQL * Snowflake * Any JDBC-complaint database (non-production preview, available for testing) * MongoDB (private preview) Contact your SingleStore account team or [SingleStore Sales](https://www.singlestore.com/contact/) if you want to move data from a source not listed above. ## Ingest Architecture Ingest replicates data from any supported source to a SingleStore destination database. It is a fully self-service, automated data replication tool. SingleStore Flow, of which Ingest is a part, offers several deployment strategies for its customers, including: * Standard deployment in an AWS environment * High Availability deployment in an AWS environment * Hybrid deployment using both on-premises and cloud infrastructure * Fully on-premises deployment Flow components can be deployed in Google Cloud and Microsoft Azure as well. AWS components and services are referenced here as an illustration of a common type of deployment. Ingest uses log-based Change Data Capture for data replication. The following is the technical architecture diagram that illustrates the standard setup in an AWS environment. The following diagram serves as the reference for all setup instructions. Estimated deployment time: Approximately 1 hour ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt3208fa2b1092e81e/6a7adb25594711dd234c267d/singlestore_flow_ingest_architecture-EgW798.png) ## Ingest / AWS Service Integration The following is the Ingest architecture which showcases integration with various optional AWS services in a standard deployment. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9341d4dac76d1021/6a7adbe342a1175ec6950983/singlestore_flow_ingest_aws_service_integration-uBfRFF.png) This architecture diagram illustrates a standard deployment that highlights the following features: * AWS services running alongside Ingest. * Recommended Flow architecture for a VPC in AWS. * Data flow between the source database, AWS, and SingleStore destination database, including security and monitoring features. * Security, including IAM, organized in a separate group and integrated with Ingest. ## Ingest High Availability Architecture The following High Availability architecture explains how Ingest is deployed in a multi-AZ setup. In the event of an instance or AZ failure, it automatically scales to another AZ without incurring any data loss. Estimated deployment time: Approximately 1 day ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9313e76f226c7dff/6a7adbe2c5d19301135db91d/singlestore_flow_ingest_ha_architecture-0KXTtm.png) ## Ingest Hybrid Architecture Ingest also offers a hybrid deployment model that combines on-premises services with those in the AWS Cloud. Ingest can be easily set up on a Windows server in an on-premises environment. The SingleStore destination endpoint resides in the AWS Cloud that creates a hybrid model. SingleStore recommends secure connectivity between on-premises and AWS services, which can be achieved using a VPN connection or AWS Direct Connect. Estimated deployment time: Approximately 2 hours to 1 day ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt84486e87b38a6464/6a7adb1d77b03469dc3f3fa1/singlestore_flow_ingest_hybrid_architecture-WbOrIR.png) ## In this section * [Source Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/source-database-setup.md) * [Destination Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/destination-database-setup.md) * [Configure Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/configure-ingest.md) * [Dashboard](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/dashboard.md) * [Configure Source Database](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/configure-source-database.md) * [Configure Destination Database](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/configure-destination-database.md) * [Tables](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/tables.md) * [Operations](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/operations.md) * [Settings](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/settings.md) * [Reports](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/reports.md) * [Logs](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/logs.md) * [Appendix](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/appendix.md) *** Modified at: February 26, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/) (An index of the documentation is available at /llms.txt) --- # SingleStore XL Ingest ## Overview SingleStore XL Ingest ("XL Ingest"), a component of SingleStore Flow, is companion software to [SingleStore Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest.md) ("Ingest"). When large tables (greater than 10GB) are included in Ingest, a full extract may not be feasible due to long processing times. This increases the likelihood of encountering issues, and if a problem occurs, the entire initial extract must be rerun. Therefore, XL Ingest is essential for working with large tables. XL Ingest handles the initial transfer of large tables by dividing them into smaller logical partitions. It then transfers multiple partitions from the source to the target in parallel. This ensures the transfer happens within a reasonable amount of time. Use XL Ingest to transfer the identified logical partitions of large tables in parallel to SingleStore for storage and processing. To transfer large tables from source database to SingleStore Helios, perform the steps outlined in [Select Tables](https://docs.singlestore.com/#section-idm234737541647291.md). Complete these steps before triggering an XL Ingest job to prevent data loss during the transition. **Note**: Pausing updates on the source database is not required during this process. Both Ingest and XL Ingest can operate independently and concurrently without disrupting the source data. ## Select Tables To ingest large data table(s) from source database to SingleStore, perform the following steps in Ingest before triggering a job in XL Ingest. 1. Navigate to **Dashboard** > **Tables** and select the gear icon. 2. Define a primary key (**Pkey**) and any necessary partitions for the table. 3. Enable **Skip Initial Extract** to bypass the initial extract and directly proceed with the delta load. 4. Select **Apply** to save the changes. 5. Navigate to the **Dashboard** tab. Under **Schedule**, disable the Ingest scheduler to ensure that all tables are moved to the destination at the same time. 6. Navigate to the **Operations** tab and initiate **Full Extract** to trigger the initial bulk load for all the selected tables, except for those tables marked as **Skip Initial Extract**. 7. Initiate **Sync New Tables** to trigger the initial bulk load for tables marked as **Redo Initial Extract** and newly-added tables in an ongoing replication. This captures the watermark for CDC and creates the table in the destination database (SingleStore). 8. Enable the ingest scheduler in Ingest after transferring tables using XL Ingest. **Note**: After marking tables with **Skip Initial Extract**, the next scheduled delta run automatically captures CDC for all tables, including those loaded with XL Ingest. XL Ingest prevents duplication during the CDC load by using a watermark to track changes. ## Split Table into Slices Large tables must be divided into notional slices based on the value of a single slice column, for example, primary key. For automatic slice determination, XL Ingest uses parameters like the number of slices needed and how many characters from the start of the slice column value must be used. **Note**: Date and time columns (for example `DATE`, `DATETIME`, `TIMESTAMP`) are not supported as slice columns in XL Ingest. Refer to [Configure Table Slicing](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest/configure-table-slicing.md) for more information on slicing. ## For Large Tables XL Ingest automatically determines the slices based on the specified parameters. For example, a slice column like `names` can be divided by the first 3 characters. Alternatively, you can manually enter the slice values instead of using auto-slice. ## For Partitioned Tables Counting records is not necessary as XL Ingest automatically determines the list of partitions to create, with each partition treated as a slice. ## For Smaller Tables Slicing may not be necessary. The entire table can be processed as a single slice. ## In this section * [Configure XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest/configure-xl-ingest.md) * [Configure Table Slicing](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest/configure-table-slicing.md) *** Modified at: May 28, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest/) (An index of the documentation is available at /llms.txt) --- # Use Flow on Helios Flow on Helios is a fully managed data migration and Change Data Capture (CDC) service within SingleStore Helios. It supports loading and synchronizing data from supported databases into SingleStore, without the need to install or manage any infrastructure. Flow is integrated into the [Cloud Portal](https://portal.singlestore.com/) as a part of the Load Data workflow. It includes two components:  * **Ingest**: Transfers schema and data for tables up to 10GB, and supports ongoing CDC. * **XL Ingest**: Handles larger tables by partitioning them and transferring partitions in parallel. ## Supported Source Databases The following databases are supported as sources: * MySQL * PostgreSQL * Oracle * Microsoft SQL Server * Snowflake These can be self-hosted, or hosted on: * AWS RDS * AWS Aurora * Google CloudSQL * Azure SQL **Note**: On-premise and self-hosted sources must be accessible over the internet and allow inbound connections from Helios IPs. For prerequisites and setup, refer to [Source Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/source-database-setup.md). ## Destination Database * SingleStore database hosted in Helios. To configure, refer to [Destination Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/destination-database-setup.md). ## Steps to Use Flow on Helios 1. On the Cloud Portal, navigate to **Ingestion** > **Load Data**. 2. **Select the Source Database** Choose from: * MySQL by Flow * PostgreSQL by Flow * Oracle by Flow * SQL Server by Flow * Snowflake by Flow 3. **Select the Destination Database** On the **Select Destination** page: * Choose the cluster and the destination database. **Note**: The cluster should be of the **Standard** type. * Under **Connection details**, choose an existing connection or create a new one, and enter the Connection name. **Note**: The Connection name must not start with a number, and cannot contain special characters, except for underscores. Spaces are allowed. 4. **Open the Flow Dashboard** The Dashboard provides a central interface to configure, manage, and monitor the overall status of an Ingest or XL Ingest instance. It displays information about the source, destination, tables, replication schedule, and a checklist. Click **Open Dashboard** to launch the dashboard. 5. **Select Ingestion Type** Choose **Ingest** from the dropdown on the top right. Refer to [SingleStore Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest.md) for more information. If the data to be ingested is larger than 10 GB, use XL Ingest by selecting XL Ingest from dropdown on the top right. Refer to [SingleStore XL Ingest ](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-xl-ingest.md)for more information. 6. **Complete the Checklist** Begin the data pipeline by following the steps in the **Checklist**: * **Setup Source**: Configure the source database. Refer to [Source Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/source-database-setup.md). * **Setup Destination**: Configure the destination database. Refer to [Destination Database Setup](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/destination-database-setup.md). * **Setup Tables**: Configure the tables to be ingested. Refer to [Tables](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/tables.md) for more information. * **Setup Schedule**: Define the replication schedule. 7. **Start the Pipeline** After completing the **Checklist**, start the pipeline from the **Operations** tab. The job begins based on the defined schedule. Refer to [Operations](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/operations.md) for more information. 8. **Monitor and Manage the Pipeline** Use the following tabs to track performance, troubleshoot issues, and adjust settings. * **Operations**: Provides options to manage and troubleshoot the data pipeline, and handle schema changes. Refer to [Operations](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/operations.md) for more information. * **Settings**: Contains configuration options for the Flow environment. Refer to [Settings](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/settings.md) for more information. * **Reports**: Used to generate reports and view table performance data, organized by day and month. Refer to [Reports](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/reports.md) for more information. * **Logs**: Monitors the progress of extract and load operations. Refer to [Logs](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/logs.md) for more information. To change the size of an existing Flow instance, refer to [Scale a Flow Instance](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/scale-a-flow-instance.md). *** Modified at: July 6, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow-on-helios/use-flow-on-helios/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios/use-flow-on-helios/) (An index of the documentation is available at /llms.txt) --- # Load Data with SingleStore Flow ## Overview SingleStore Flow (or, "Flow") is SingleStore's data migration and change data capture (CDC) solution which is comprised of two primary components: * **SingleStore Ingest**, also referred to as "Ingest," transfers the schema for all tables, moves table data for tables up to 10GB in size, and handles ongoing CDC for all tables into SingleStore. * **SingleStore XL Ingest**, also referred to as "XL Ingest," handles the initial transfer of large tables by breaking them into smaller logical partitions. It then copies multiple partitions from the source to the target in parallel. This helps transfer large tables efficiently, and is resilient to network outages and restarts of Flow, the source, or the destination. Flow supports the following source databases for transfer to SingleStore: * Oracle * Oracle RAC * Microsoft SQL Server * MySQL * PostgreSQL * Snowflake * Any JDBC-compliant database (non-production preview, available for testing) * MongoDB® (private preview) * Salesforce (public preview) * SingleStore (non-production preview) ## Why Use Flow Key benefits of using Flow: * **No-code setup**: Configure end-to-end migration via the Flow UI. * **Automatic schema handling**: Analyze the source database and automatically create the required schema in the target SingleStore database. * **Real-time visibility**: Monitor progress, view live logs, and troubleshoot quickly. * **High throughput**: Use XL Ingest to parallelize large table loads by chunking source data, significantly reducing migration time. ## Choose the Right Flow Deployment Flow is available in two deployment options: * A Self-Managed Flow Application * Flow on Helios Both provide the same data migration and CDC features, but differ in how they are deployed and used. The following table outlines these two options: | Deployment Option | Self-ManagedFlowApplication | FlowonHelios | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Description** |
  • Installed and operated in your own environment.
  • Self-managed, installation required.
  • Provides control over where the application runs and how it integrates with your system.
|
  • Runs in the SingleStore Helios cloud deployment.
  • Fully managed by SingleStore, no installation required.
  • A Flow instance is provisioned directly from the UI, and usage is billed only while the instance is running.
| | **When to Use** |
  • When deployment placement or control over the environment matters the most, such as replicating large volumes of data from an on-premises source to SingleStore. This is because Flow performs best for high-volume transfers when its services run close to the data source.
|
  • When your data sources and destinations are in the same cloud region, along with your Flow on Helios instance, ensuring excellent performance along with convenience and robustness.
  • When convenience and low operational overhead matter more than placement.
| For more details on Flow on Helios, refer to [Load Data with SingleStore Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios.md). ## In this section * [Free Trial Access to Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/free-trial-access-to-flow.md) * [Install SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/install-singlestore-flow.md) * [Rotate the Encryption Key](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/rotate-the-encryption-key.md) * [Upgrade SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/upgrade-singlestore-flow.md) * [Run Flow as a Linux Service](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/run-flow-as-a-linux-service.md) * [Secure Access to Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/secure-access-to-flow.md) * [SingleStore Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest.md) * [SingleStore XL Ingest](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-xl-ingest.md) * [Data Type Mapping in SingleStore Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow.md) * [SingleStore Flow API](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-api.md) * [SingleStore Ingest Operational FAQ](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq.md) * [SingleStore Flow Release Notes](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/singlestore-flow-release-notes.md) *** Modified at: May 6, 2026 Source: [/cloud/load-data/load-data-with-singlestore-flow/](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow/) (An index of the documentation is available at /llms.txt) --- # Manage Data The topics in this section provide information about data storage in SingleStore, backup and restore operations, and data exports. SingleStore supports unlimited storage databases. Unlimited storage databases use remote object stores, which are external to the SingleStore cluster. Database backups can be made as full or incremental backup and then restored. Backup targets can be local file systems, network file systems, or cloud storage services. In addition, you can create logical backups of one or more databases using the MySQL Dump utility and export them. ## In this section * [Database Storage](https://docs.singlestore.com/cloud/manage-data/database-storage.md) * [How Storage is Measured](https://docs.singlestore.com/cloud/manage-data/how-storage-is-measured.md) * [Benefits of Unlimited Storage](https://docs.singlestore.com/cloud/manage-data/benefits-of-unlimited-storage.md) * [Smart Disaster Recovery (DR): SmartDR](https://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr.md) * [Continuous Backups and Restore](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore.md) * [Back Up and Restore Data](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data.md) * [Export Data](https://docs.singlestore.com/cloud/manage-data/export-data.md) * [Moving Data](https://docs.singlestore.com/cloud/manage-data/moving-data.md) *** Modified at: March 17, 2026 Source: [/cloud/manage-data/](https://docs.singlestore.com/cloud/manage-data/) (An index of the documentation is available at /llms.txt) --- # Back Up and Restore Data > **📝 Note**: This process has been replaced with [Continuous Backups and Restore](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore.md) for AWS and GCP customers. It is currently applicable only for Azure. Backups are automatically taken at scheduled intervals by the system. These system-created backups are not directly accessible by users. To restore from these default system-created backups, you need to raise a [support](http://support.singlestore.com) ticket. > **📝 Note**: This topic applies to database backups made using the [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) command.Incremental backups cannot be made on unlimited storage databases.Unlimited storage databases can be restored (attached) to any point in time using PITR, which is not discussed in this topic. See [Database Storage](https://docs.singlestore.com/cloud/manage-data/database-storage.md) for information on PITR. > **⚠️ Warning**: SingleStore does not support restoring database backups from a newer version of SingleStore into an older version. Databases on a secondary cluster to which the primary database is being replicated cannot be backed up with the `BACKUP DATABASE` command. ## In this section * [Backup Data](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-data.md) * [Backup and Restore on S3, Azure or GCS](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-and-restore-on-s-3-azure-or-gcs.md) * [Items Included in and Excluded from a Backup](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/items-included-in-and-excluded-from-a-backup.md) * [Backup Information Schema Tables](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-information-schema-tables.md) * [Clearing Backup History](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/clearing-backup-history.md) * [Lock-free Backups](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/lock-free-backups.md) *** Modified at: August 6, 2025 Source: [/cloud/manage-data/back-up-and-restore-data/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/) (An index of the documentation is available at /llms.txt) --- # Backup and Restore on S3, Azure or GCS Backup and restore can specify an S3 bucket or Azure container or Google Cloud Storage (GCS). In this case, all the backup files across all the leaves (one per partition as described above) are put directly into the S3 bucket or Azure container or GCS. When restore is called, SingleStore will distribute the backup files appropriately and restore the data (similar to how restoring from an NFS drive works). When taking a database backup to an Azure bucket \[blob container] ensure that your service account has all required privileges, including the `blob feed` privilege. See [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) and [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) for syntax and examples. *** Modified at: September 22, 2023 Source: [/cloud/manage-data/back-up-and-restore-data/backup-and-restore-on-s-3-azure-or-gcs/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-and-restore-on-s-3-azure-or-gcs/) (An index of the documentation is available at /llms.txt) --- # Backup Data > **📝 Note**: This process has been replaced with [Continuous Backups and Restore](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore.md) for AWS and GCP customers. It is currently applicable only for Azure. Only full backups can be created. Incremental backups are not available. In SingleStore Helios backups cannot be created to local file systems. Backups are not run within the clusters themselves but on the Master Aggregator (MA) of the corresponding underlying cluster. Backups happen for clusters that have databases attached in R/W (read and write) mode. Clusters with databases attached in R/O (read only) mode do not get backed up. When you suspend a cluster you suspend any operations against it. Hence if you suspend a cluster around the time a backup is scheduled, then that day's backup gets skipped. In addition to the scheduled backups, you can also create your own backups through SQL from a MySQL client. You can back up into your AWS S3 bucket, your Azure Blob storage, or GCS and then restore as needed. ## Backup Schedule SingleStore Helios automatically backs up data on a default schedule at 2 am daily in the local time for the deployment region. The data is stored in AWS S3. ## Data Retention Period The retention period is seven days. You can always request a restore from a particular day in the last seven days. ## Altering the Scheduled Time for Your Backup You can raise a [Support](https://www.singlestore.com/support/) ticket to alter the scheduled time for your backup. In the support ticket you should provide the following information: * The name of the database for which you want to alter the scheduled backup time. * The name of the cluster on which the database is running. * The new scheduled backup time you want to be set for your database. Ensure you specify the time format clearly (12-hour or 24-hour) and include the time zone. The support team will provide an ETA when they receive the support ticket and subsequently update the ticket with the confirmation of the changes made. *** Modified at: May 11, 2026 Source: [/cloud/manage-data/back-up-and-restore-data/backup-data/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-data/) (An index of the documentation is available at /llms.txt) --- # Backup Information Schema Tables See the following reference links for definitions of each column in the information schema tables included in this guide: * [MV\_BACKUP\_HISTORY](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-history.md) * [MV\_BACKUP\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-status.md) *** Modified at: April 28, 2021 Source: [/cloud/manage-data/back-up-and-restore-data/backup-information-schema-tables/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/backup-information-schema-tables/) (An index of the documentation is available at /llms.txt) --- # Clearing Backup History You can remove all the entries in the `MV_BACKUP_HISTORY` table in the `information_schema` database by running `CLEAR BACKUP_HISTORY;`. `information_schema.MV_BACKUP_HISTORY` is populated only when the `BACKUP DATABASE` command is run to create a database backup. **Selectively Delete Rows from `information_schema.MV_BACKUP_HISTORY`** To clean up individual rows specific to a particular backup use: ```sql CLEAR BACKUP_HISTORY backup_id = ; ``` or ```sql CLEAR BACKUP_HISTORY ; ``` Success/failure are listed in the `STATUS` column of the `MV_BACKUP_HISTORY` table. *** Modified at: May 5, 2023 Source: [/cloud/manage-data/back-up-and-restore-data/clearing-backup-history/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/clearing-backup-history/) (An index of the documentation is available at /llms.txt) --- # Items Included in and Excluded from a Backup When you make a database backup, the following items are included in and excluded from the backup. | Included | Excluded | | ----------------- | ------------------- | | Tables | Resource Pools | | Views | System Variables | | Procedures | Users | | Pipelines | Privileges | | Functions | Roles | | Stored Procedures | Passwords | | | Pipeline Transforms | *** Modified at: February 28, 2023 Source: [/cloud/manage-data/back-up-and-restore-data/items-included-in-and-excluded-from-a-backup/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/items-included-in-and-excluded-from-a-backup/) (An index of the documentation is available at /llms.txt) --- # Lock-free Backups Lock-free backups do not block `INSERT`, `UPDATE`, and `DELETE` operations at any point during a backup. They do not need to lock and wait for write queries. Lock-free backups are enabled by default However, `ALTER DATABASE` command will not run while there is a backup job active on the relevant database. The backup takes the cluster operations lock for the entirety of the backup. ## Commands Blocked by the BACKUP Process * `ALTER DATABASE` * `DROP DATABASE` ## Commands Not Blocked by the BACKUP Process * `CREATE TABLE` (columnstore and rowstore) * `ALTER TABLE` (columnstore and rowstore) * `DROP TABLE` (columnstore and rowstore) * `DELETE TABLE` * `INSERT TABLE` * `TRUNCATE TABLE` (columnstore and rowstore) * `CREATE INDEX` * `DROP INDEX` (columnstore and rowstore) * `CREATE VIEW` * `DROP VIEW` * `ANALYZE TABLE` (columnstore and rowstore) * `OPTIMIZE COLUMNSTORE TABLE` * `SHOW TABLE STATUS` * `SHOW DATABASE STATUS` * `OPTIMIZE ROWSTORE TABLE` *** Modified at: July 11, 2025 Source: [/cloud/manage-data/back-up-and-restore-data/lock-free-backups/](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/lock-free-backups/) (An index of the documentation is available at /llms.txt) --- # Benefits of Unlimited Storage A major benefit of unlimited storage (An unlimited amount of storage space in the cloud that can be used to move data to seamlessly between memory, persistent cache, and storage.) is that it provides elastic scalability for storage in SingleStore self-hosted and cloud clusters. Even if you do not need the additional storage, this feature has several other benefits, including: * **It enables Point in time recovery (PITR)** in editions that support PITR. * **It can improve performance when rebalancing**. If you do rebalancing in regular storage it requires moving the data synchronously. But with unlimited storage, you only move the metadata (describing names and existence of files, not their contents) synchronously when rebalancing. Expanding your cluster is a size-of-metadata, not a size-of-data operation with unlimited storage. This is valuable if you have a large amount of data per leaf. * **It helps in getting more usable local disk space** since you don't need to store the full high-availability (HA) copy locally. You need HA row replicas for durability but you only need that for recently-changed data. It allows you to have more usable local disk space, have durability on new writes, and then lazily populate HA replica data from object storage to other nodes if a leaf fails. * **It enables excellent performance when your working set fits in the local disk**. Working set size is totally application-dependent. There is no precise definition of a working set, but a good working definition is: the smallest subset of data such that your workload is not more than 10% slower than if all the data is in the cache. Depending on your application, your working set might be 100% of the data, or it might be only 30% of the data. If your working set does not fit in cache then as in many computing systems, the performance will not be good. For example, if the working set does not fit in the RAM in a virtual memory system, it will thrash. Having large persistent cache device sizes that allow you to have your working set in persistent cache is the recommended solution to overcome this. Once the working set is stored on persistent cache, you can expect excellent performance. * **It gives additional fault tolerance** (a form of continuous online backup), in that if you lose a cluster altogether, the database data still exists on object storage, current up to the last two minutes or so of data. This does not protect against all faults, for example, dropping a database accidentally will drop the data. So some backups may still be needed, but potentially fewer. This also can save on backup storage space. Moreover, because PITR is available, you may no longer need to do incremental backups, saving time, effort, and space. *** Modified at: April 17, 2024 Source: [/cloud/manage-data/benefits-of-unlimited-storage/](https://docs.singlestore.com/cloud/manage-data/benefits-of-unlimited-storage/) (An index of the documentation is available at /llms.txt) --- # Continuous Backups and Restore > **📝 Note**: Continuous backups are available on AWS, GCP and Azure. Continuous backups asynchronously store all customer data in unlimited storage (bottomless storage) ensuring that every change (inserts, updates, and deletes) is durably persisted with 99.9999% reliability. With a default 7-day retention period, they allow you to restore your data to a point in time within this window using the always available point-in-time recovery (PITR) option. ## Benefits * Continuous backups do not interrupt active jobs and do not impact your workload. Since data is continuously backed up in the background, it does not consume additional CPU resources, unlike full backups. * You can restore your data whenever required, without the need to create Support tickets. * You can recover your data while your primary database is online. ## Enabling Continuous Backups Connect to your cluster and run the following query to check the value of the `bottomless_gc_retention_period_minutes` variable: ``` SELECT @@bottomless_gc_retention_period_minutes; ``` If the output value is 0 (zero) then Continuous Backups are disabled for the cluster. A non-zero value implies Continuous Backups are enabled. For example, a value of 10080 means backups are enabled with a retention period of 10080 minutes (7 days). ## Standard vs Enterprise Edition **Restoration Point** The Standard edition allows restoration to 00:00 UTC on any day within the retention window, whereas the Enterprise edition offers finer-grained restoration with one-second granularity to any point in time within the retention window. Both editions support self-service data restoration. **Data Retention Period** For both editions, the default retention period is 7 days and the minimum is 1 day. Standard edition’s retention period cannot be extended beyond 7 days. Enterprise edition has the option to extend this period up to 90 days or longer. Increasing the retention period results in higher storage costs due to the accumulation of data over time. Even if you delete your tables or records, your data is still retained for the duration of the retention period, potentially impacting your storage costs. Although the minimum retention period can be set to 1 day, SingleStore recommends setting the retention period to at least 3 days. Retention periods of less than 3 days significantly reduce the reaction time available to address any issues before data is permanently deleted. > **📝 Note**: To retain your data for extended time periods beyond what is available with continuous backups, you can use the [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) command to back up your database to your own object storage. ## Continuous Backups and PITR Continuous backups are the always-on history feed into bottomless storage happening in the background, while PITR is the feature that lets you use that available history to restore or branch a database at specific points in time. The restore granularity and available history are governed by your edition and configured retention period. ## Restoring Data from Continuous Backups Restoring data from continuous backups is a self-service online operation. To restore data: 1. Create a [branch](https://docs.singlestore.com/cloud/developer-resources/database-branching.md) of your database by specifying the relevant timestamp. 2. Use this new branch database to query and restore the data as needed. For example, consider a scenario where a bad update was executed on January 2nd. You can branch the database to a specific timestamp before the update was executed and recover the data. In the Standard edition, you can create a branch by specifying a specific date. The timestamp of the branch is automatically set to 00:00:00 and cannot be modified. ``` ATTACH DATABASE DB_X AS recover_DB_X AT TIME '2025-01-02'; ``` In the Enterprise edition, the data can be restored to a granularity of 1 second. You must ensure the timestamp is specified in UTC. ``` ATTACH DATABASE DB_X AS recover_DB_X AT TIME '2025-01-02 21:57:31'; ``` ## Durability Guarantee for Continuously Backed up Data SingleStore has built-in safeguards to ensure your data is never accidentally deleted. * **The Metadata Service (Inhouse service)**: safeguards your data by continuously monitoring and managing the metadata. Regular blob quality checks and checksum validations on metadata service are conducted to identify and address potential corruption. * **Robust GC (Garbage Collection) Process**: the GC process is designed to prevent accidental data deletion, even in the event of bugs. * **Cloud Provider’s versioning capability**: this serves as a last-resort failsafe for recovering data beyond your specified retention period. Leveraging the public cloud provider's versioning feature, data is retained for an additional 3 days.** It is important to note that this mechanism should not be considered an extension of your primary retention policy.** To recover data beyond your standard retention period, you must submit a support ticket. ## Types of Cloud Storage SingleStore Helios cloud storage has two primary categories: 1. **Active Storage**: This represents the storage you are currently using to support your specific applications. It includes your primary data, logs, and snapshots that are actively in use. 2. **Continuous Backups Storage**: This is the space consumed by your configured backup retention settings. The storage volume in this category will vary depending on your retention period, which is 7 days by default. ## How to Check Your Cloud Storage Usage You can calculate the total cloud storage usage by querying the information schema views, which summarize storage usage per storage ID in your cluster group. Here is a sample query to help you calculate: * Your total storage in megabytes, per storage ID. * The databases that share each storage ID, for easier interpretation of the results ```sql SELECT rs.STORAGE_ID, GROUP_CONCAT(DISTINCT pc.DATABASE_NAME) AS DATABASE_NAME, ROUND( (MAX(rs.BLOBS_BYTES)/1024/1024) + (MAX(rs.LOG_CHUNKS_BYTES)/1024/1024) + (MAX(rs.SNAPSHOTS_BYTES)/1024/1024), 2 ) AS TOTAL_STORAGE_MB FROM information_schema.MV_CLOUD_REMOTE_STATS rs INNER JOIN information_schema.MV_CLOUD_PER_COMPUTE_REMOTE_STATS pc ON rs.STORAGE_ID = pc.STORAGE_ID GROUP BY rs.STORAGE_ID; ``` ## Cloud Storage Calculations The total storage is calculated as the sum of your blob storage, log storage, and snapshots: Total Storage = BLOBS + LOGS + SNAPSHOTS `MV_CLOUD_REMOTE_STATS` reports storage at the per-storage level, which aligns with how usage is aggregated for billing. `MV_CLOUD_PER_COMPUTE_REMOTE_STATS` is a per-compute-session view; the same storage can appear in multiple compute sessions (for example, when a database is detached and re-attached). The query above uses this view only to list database names associated with each `STORAGE_ID` and does not rely on its retention columns. *** Modified at: July 10, 2026 Source: [/cloud/manage-data/continuous-backups-and-restore/](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore/) (An index of the documentation is available at /llms.txt) --- # Database Storage SingleStore Helios stores data remotely in an object store (unlimited storage database) or locally in a cluster, depending on the cloud provider you have chosen for the cluster deployment. AWS, Azure, and GCP clusters store data remotely in an object store. When you deploy a cluster using one of these cloud providers and create databases, the databases will automatically use unlimited storage for the cluster. ## Unlimited Storage Databases SingleStore automatically manages data across a three tiered storage architecture comprised of memory, local cache, and storage. When deploying SingleStore all of the storage tiers are automatically configured. This is referred to as an “Unlimited Storage” database. Unlimited Storage allows data loaded into SingleStore to be seamlessly moved between memory, persistent cache, and storage. The storage tier leverages the native object storage on AWS, Azure, or GCP and is transparent to the user and fully managed by SingleStore. Since unlimited storage databases are stored in object storage, their size is not limited by the size of the persistent cache, but only by available external object storage. Although the size of a database is effectively unlimited on public cloud objects stores, the following factors related to the local storage space limit the number of databases and tables you can create. * To avoid performance problems, there should be sufficient local storage space to hold the working set of all your databases, combined. * There is a per-database local storage (even if the database is empty) and memory overhead, and a per-table memory overhead for metadata. The amount of local storage space required per partition varies depending on the size of the blob cache which can grow and shrink based on the workload. In an unlimited storage database: * All columnstore data is stored externally in an object store. Recently accessed columnstore data objects are also cached locally in the compute cluster's persistent cache. * All rowstore data is stored locally in compute cluster memory and externally in object storage. * Data updates made on the cluster are flushed to the object store asynchronously. Typically, the object store will be no more than one minute behind the latest update on the cluster. * When you suspend a cluster in SingleStore Helios, the cluster is torn down completely, including the storage. During a resume, a new compute and storage is brought up. Therefore, the blob cache or all the cache on the disk is lost. In this scenario, even plan cache is lost because planned cache is stored on the disk. When you run the [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) command on AWS, Azure, and GCP clusters, SingleStore automatically creates an unlimited storage database. [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) and [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) commands can be used for database backup and restore respectively. When `RESTORE DATABASE` is run, data will be automatically restored into an unlimited storage database. ## Unlimited Storage Tiers **Memory:** SingleStore stores data in memory when using Rowstore, when caching data for Columnstore, and for operations which utilize the high performance characteristics of system memory. **Persistent Cache:** This tier is comprised of high performance block storage and serves Columnstore data and persists Rowstore data. For optimal performance a SingleStore deployment should be sized so that the working dataset (data to be queried) fits within the Persistent Cache. **Storage:** This is a durable and persistent layer stored within the cloud object storage. Data is regularly pushed to object storage providing a cool tier of data, and allowing for long-term retention beyond the lifetime of a deployment. It also serves to enable features such as Point-in-time-Recovery. The amount of data persisted in the storage tier may exceed the size of the active databases due to the retention of snapshots to be used for data recovery. ## Point-in-Time Recovery (PITR) > **📝 Note**: This feature is not available in all editions of SingleStore. . For more information, see [SingleStore Helios Editions](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md). > **📝 Note**: For the Enterprise edition, you can use either the `CREATE MILESTONE` with PITR or the `SYNC BOTTOMLESS DATABASE` command. PITR is a user-initiated operation that restores a unlimited storage database to a point in time. It is purely a recovery operation and does not require backups. It uses the unlimited storage feature and leverages the blobs stored in object store. It uses all data for the restore point that has been automatically flushed to the object store. The PITR window or timeline is the length of the retention period. All points in that timeline are accessible and you can do PITR back and forth within the timeline. For example, today at 9 AM, you can restore back to a point in time two days ago and then restore (roll) forward all the data as it was today at 9 AM, provided all these restore points are within the retention period timeline. When you invoke PITR, you can specify either a timestamp or a milestone. Creating milestones is a manual operation that you must do beforehand but they are not required to use PITR.  You can use the timestamp option instead.  Milestones are useful for cases where you do a risky operation, such as an upgrade or schema change, and you want to bookmark the point in time right before you do the operation, in case you have to roll it back and you do not want to guess what the timestamp needs to be. Attaching (restoring) an unlimited storage database can be faster than restoring an equivalent local storage database. This is because an attach of an unlimited storage database does not copy all data to the cluster, as is the case with the restore of a local storage database. Note that after an unlimited storage database is attached, queries may be slower for some time until remote data is cached locally in the cluster. To work with PITR, use the following commands: * [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) (Create an unlimited storage database) * [CREATE MILESTONE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-milestone.md) (Create a milestone, which is a marker to which you can restore) * [ATTACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database.md) (Attach an unlimited storage database and bring the database online) * [DETACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/detach-database.md) (Bring an unlimited storage database offline) For a basic walkthrough of creating milestones and restoring a database to milestone, see [Attach an Unlimited Storage Database Using Point-in-Time Recovery (PITR)](https://docs.singlestore.com/cloud/manage-data/database-storage/attach-an-unlimited-storage-database-using-point-in-time-recovery-pitr.md). A database can be restored to any point in time via the [Cloud Portal](https://portal.singlestore.com) or the `ATTACH DATABASE` command. PITR functionality is available in the Enterprise edition. To access this functionality, select **Clusters** and choose your cluster. Then, select **Databases** and choose your database. Finally, navigate to the Recovery tab. See [Disaster Recovery](https://docs.singlestore.com/cloud/security/disaster-recovery.md) for data retention and other related information. ## In this section * [Attach an Unlimited Storage Database Using Point-in-Time Recovery (PITR)](https://docs.singlestore.com/cloud/manage-data/database-storage/attach-an-unlimited-storage-database-using-point-in-time-recovery-pitr.md) * [Online Point-in-Time Recovery (PITR)](https://docs.singlestore.com/cloud/manage-data/database-storage/online-point-in-time-recovery-pitr.md) * [Rename an Unlimited Storage Database](https://docs.singlestore.com/cloud/manage-data/database-storage/rename-an-unlimited-storage-database.md) * [Configure the Retention Period for an Unlimited Storage Database](https://docs.singlestore.com/cloud/manage-data/database-storage/configure-the-retention-period-for-an-unlimited-storage-database.md) *** Modified at: May 11, 2026 Source: [/cloud/manage-data/database-storage/](https://docs.singlestore.com/cloud/manage-data/database-storage/) (An index of the documentation is available at /llms.txt) --- # Attach an Unlimited Storage Database Using Point-in-Time Recovery (PITR) > **📝 Note**: PITR is not available in all editions of SingleStore. For more information, see [SingleStore Helios Editions](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md). The following example demonstrates how to restore an unlimited storage database. The Setup section contains the prerequisite steps needed to create the database and milestones, prior to performing the restore. ## Setup 1\. Create the unlimited storage database `bottomless_db`. ```sql CREATE DATABASE bottomless_db; ``` 2\. Make some updates to `bottomless_db`. In this example, you update the database by creating a table and inserting some data: ```sql USE bottomless_db; CREATE TABLE t(a INT); INSERT INTO t(a) VALUES (10); INSERT INTO t(a) VALUES (20); ``` 3\. Create a milestone (a restore point): ```sql CREATE MILESTONE "after_second_insert" FOR bottomless_db; ``` 4\. Make more updates to `bottomless_db`: ```sql INSERT INTO t(a) VALUES (30); INSERT INTO t(a) VALUES (40); ``` Create a second milestone: ```sql CREATE MILESTONE "after_fourth_insert" FOR bottomless_db; ``` ## Perform the Restore Suppose the values `30` and `40` were inserted into `t` in error. You now want to restore the database to the milestone `after_second_insert`. Detach the database to bring it offline: ```sql DETACH DATABASE bottomless_db; ``` Then attach the database at the restore point: ```sql ATTACH DATABASE bottomless_db AT MILESTONE "after_second_insert"; ``` You can restore to any point in time for which all the data is available on an object store. It is not necessary to restore to a named milestone. A database cannot be restored to a PITR milestone taken before the database was dropped. This is because [DROP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-database.md) removes the PITR history. ## Daylight Saving Time Ambiguity in PITR When you do a PITR using the following timestamp format, ```sql ATTACH DATABASE x_db AT TIME 'YYYY-MM-DD HH:MM:SS'; ``` the engine uses the server timezone. This can create ambiguity when daylight saving time (DST) changes time by one hour. For example, if you input the time as 1:05AM, was that before or after DST? The time used by the engine and the user scripts may not match. The solution is to use Unix time stamps or alternatively, use milestones to avoid this problem. ## Online PITR The standard approach to PITR is offline because you must detach and then attach the database. As a result, the database cannot be accessed between those two operations. However, for some scenarios, you may not want to take the database offline, but you still want to do a PITR to a new database and manually copy some lost data back to the source database. *** Modified at: July 10, 2026 Source: [/cloud/manage-data/database-storage/attach-an-unlimited-storage-database-using-point-in-time-recovery-pitr/](https://docs.singlestore.com/cloud/manage-data/database-storage/attach-an-unlimited-storage-database-using-point-in-time-recovery-pitr/) (An index of the documentation is available at /llms.txt) --- # Configure the Retention Period for an Unlimited Storage Database Use the `Cluster Groups` path (`/v1/workspaceGroups/{workspaceGroupID}/storage/retentionPeriod` endpoint) in the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) to specify the retention period (in minutes) for an unlimited storage database. The retention period specifies how long the historical information used to enable [point-in-time recovery (PITR)](https://docs.singlestore.com/cloud/manage-data/database-storage/online-point-in-time-recovery-pitr.md) is retained. By default, the retention period is set to `10080` minutes. For example, the following API request sets the retention period to `7200` minutes for the cluster group with ID `058e2ed1-3000-4000-a000-512d334df903`: ```shell curl -X 'PATCH' \ 'https://api.singlestore.com/v1/workspaceGroups/058e2ed1-3000-4000-a000-512d334df903/storage/retentionPeriod' \ -H 'accept: */*' \ -H 'Content-Type: application/json' \ -d '{ "retentionPeriod": 7200 }' ``` Refer to [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. Once a milestone reaches the retention period, it is no longer available and does not appear in the `MV_MILESTONES` information schema view. You cannot attach an unlimited storage database at a milestone or time that is not within the retention period. For example, consider that the retention period is `1440` minutes (1 day) and you create a milestone for a database `dbExample` on June 1. On June 3, you will not be able to attach `dbExample` at the milestone created on June 1 or at a point in time that occurred on June 1. *** Modified at: May 11, 2026 Source: [/cloud/manage-data/database-storage/configure-the-retention-period-for-an-unlimited-storage-database/](https://docs.singlestore.com/cloud/manage-data/database-storage/configure-the-retention-period-for-an-unlimited-storage-database/) (An index of the documentation is available at /llms.txt) --- # Online Point-in-Time Recovery (PITR) > **📝 Note**: This process is now deprecated. Refer to the [Continuous Backups and Restore](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore.md) topic for the current backup and restore process. You can open a [Support](https://support.singlestore.com/) ticket, for an online PITR. The support ticket you open should include the following information: * The name of the database for which you want to do online PITR. * The name of the cluster on which it is running. * The name of the cluster on which you want it restored. * The name you want for the new restored database. The cluster must be in the same region. The support team will provide an ETA when they receive your ticket. Once your restored database is ready, the support team will update the support ticket with details. Support will copy an online database to a new database name, and the new database will have all the PITR history of the original. The database will be attached at the latest time available. The last few minutes of data may not be available in the copy. Then, you have to detach the new copy and re-attach it at whatever point in time you want. When you find the data you need, you can, for example, copy it back to the original database using `INSERT INTO … SELECT FROM… `commands. This allows you to recover accidentally removed or corrupted data into the original online database while keeping the original data online. For example, you accidentally deleted all the rows for one customer from a large Sales table. If you have thousands of customers, you may not want to take the whole database offline to recover these rows. Online PITR allows you to recover the rows while the database stays online. *** Modified at: June 4, 2026 Source: [/cloud/manage-data/database-storage/online-point-in-time-recovery-pitr/](https://docs.singlestore.com/cloud/manage-data/database-storage/online-point-in-time-recovery-pitr/) (An index of the documentation is available at /llms.txt) --- # Rename an Unlimited Storage Database To rename an unlimited storage database, run [DETACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/detach-database.md) followed by [ATTACH DATABASE ... AS ...](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database.md). The following example renames the database `example_db1` to `example_db2`. ```sql DETACH DATABASE example_db1; ATTACH DATABASE example_db1 AS example_db2; ``` *** Modified at: August 4, 2023 Source: [/cloud/manage-data/database-storage/rename-an-unlimited-storage-database/](https://docs.singlestore.com/cloud/manage-data/database-storage/rename-an-unlimited-storage-database/) (An index of the documentation is available at /llms.txt) --- # Export Data > **📝 Note**: If you are running SingleStore Helios, use the endpoint to run the commands discussed in this topic. This topic describes how to export data from SingleStore databases. ## Export a Single Database In SingleStore Helios, you can use the endpoint to run `mysqldump` that creates a .sql file containing the queries necessary to recreate a database. For example, if you have a database named “foo”, you can create a backup with the following command: ```shell mysqldump -h 127.0.0.1 -u root -P 3306 foo > foo.sql ``` ## Export All Databases In SingleStore Helios, use the endpoint to run the command `mysqldump`. For example: ```shell mysqldump -h 127.0.0.1 -u root -P 3306 --all-databases > full_backup.sql ``` *** Modified at: July 29, 2024 Source: [/cloud/manage-data/export-data/](https://docs.singlestore.com/cloud/manage-data/export-data/) (An index of the documentation is available at /llms.txt) --- # How Storage is Measured ## Overview SingleStore uses two key units for measuring data storage usage: | Unit | Description | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Total Data Storage Usage****(GB-h)** | Represents total data storage usage over a given period.Can be calculated by multiplying the data storage usage, in gigabytes (GB), by the duration, in hours, which yields total data storage usage in gigabyte-hours (GB-h).To illustrate, storing 10 GB for 3 hours = (10 GB) × (3 hours) = 30 GB-h | | **Average Data Storage Usage****(Avg GB)** | Represents the average data storage usage over a given period.Can be calculated by dividing the total data storage usage, in GB-h, by the duration, in hours, which yields the average data storage usage in gigabytes (GB).To illustrate, if a database’s total data storage usage is 720 GB-h over a 30-day month, the average data storage usage is:(720 GB-h) / (30 days × 24 hours per day) = 1 GB. | ## Storage Analysis ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte7bfe5ed28eed3b6/6a7adae442a1176cad95091f/helios_storage_bar_graphs-86Nmch.png) The graphs above display a day’s view of data storage usage, where each bar represents the average data storage usage throughout the day. For the cluster group, `Group 1 - Primary Region (AWS TIER 1: US WEST 2 (OREGON))`, the data storage graph shows: | Date | Average Data Storage Usage(Avg GB) | Hours per Day | Total Data Storage Usage(GB-h) | | -------- | ---------------------------------- | ------------- | ----------------------------------- | | June 4th | 2.22 GB | 24 | (2.22 GB) × (24 hours) = 53.28 GB-h | | June 5th | 5.45 GB | 24 | (5.45) × (24 hours) = 130.8 GB-h | | June 6th | 3.65 GB | 24 | (3.65) × (24 hours) = 87.6 GB-h | As these were the only three days in June in which this cluster stored data, the total data storage usage for the month of June is: 53.28 + 130.8 + 87.6 = 271.68 GB-h ## Billing Calculation For billing, storage is expressed as average data storage usage in gigabytes (Avg GB). To determine the average data storage usage for the month of June, divide the total data storage usage by the total number of hours in the month: * Total data storage usage in June: 271.68 GB-h * Total hours in June: 30 days × 24 hours = 720 hours (Total data storage usage in June) / (Total hours in June) (271.86 GB-h) / (720 hours) = 0.378, or 0.38 GB ## Verify with the Consumption Table The consumption table displays monthly metrics. The selected cluster reflects the average data storage usage of 0.38 GB, which is in line with the preceding calculation. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcb4769c6c3b2832e/6a7adaccf6825734cf58d957/helios_storage_consumption-WMnobH.png) *** Modified at: May 11, 2026 Source: [/cloud/manage-data/how-storage-is-measured/](https://docs.singlestore.com/cloud/manage-data/how-storage-is-measured/) (An index of the documentation is available at /llms.txt) --- # Moving Data This brief section describes a few methods to use if your data needs to be moved from one database to another. ## In this section * [Moving Data Between Databases](https://docs.singlestore.com/cloud/manage-data/moving-data/moving-data-between-databases.md) * [Moving Data Across Clusters](https://docs.singlestore.com/cloud/manage-data/moving-data/moving-data-across-clusters.md) *** Modified at: November 10, 2022 Source: [/cloud/manage-data/moving-data/](https://docs.singlestore.com/cloud/manage-data/moving-data/) (An index of the documentation is available at /llms.txt) --- # Moving Data Across Clusters You can only share data or attach databases within a single cluster group, not across groups. There is no direct method of moving tables or databases betweencluster groups currently available in SingleStore Helios. To move data between cluster groups, you can use one of the following methods: ## Move a Table or a Subset of Data (ETL via Object Storage) This method involves the following steps: 1. **Export from the source cluster group**: From a cluster in the source cluster group, write the table to S3/GCS/Azure Blob using `SELECT ... INTO`. For example: ```sql SELECT * FROM src_db.src_table INTO S3 's3://my-bucket/my-prefix/' CONFIG '{...}' CREDENTIALS '{...}'; ``` Variants exist for `INTO AZURE`, `INTO GCS`, etc. for details refer [SELECT](https://docs.singlestore.com/cloud/query-data/select.md) 2. **Re-create the schema in target cluster group**: In a cluster in the destination cluster group, create the destination table with matching DDL. 3. **Load into target cluster group**: In the destination cluster group, use a **Pipeline** (recommended) or `LOAD DATA` from that bucket to load into the new table. ```sql CREATE PIPELINE p_move_table AS LOAD DATA S3 's3://my-bucket/my-prefix/' CONFIG '{...}' CREDENTIALS '{...}' INTO TABLE dest_db.dest_table; START PIPELINE p_move_table; ``` For details on options available refer: [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) [Load Data with SingleStore Flow on Helios](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios.md) [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) ## Move an Entire Database If you want to move a whole database across cluster groups, you can: Run [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) from a cluster in the source cluster group to object storage. Run [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) from a cluster in the destination cluster group, pointing at that backup. *** Modified at: July 22, 2026 Source: [/cloud/manage-data/moving-data/moving-data-across-clusters/](https://docs.singlestore.com/cloud/manage-data/moving-data/moving-data-across-clusters/) (An index of the documentation is available at /llms.txt) --- # Moving Data Between Databases There are ways to move your data between databases depending on whether SingleStore Helios or SingleStore Self-Managed is where the data is being accessed. * [SELECT ... INTO](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select/#select-into-variable.md) , [CREATE PIPELINE AS LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md), then [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md). The following example uses S3, but this will work for HDFS, GCS, KAFKS, and AZURE as well. ```sql SELECT * FROM table_name_1 INTO S3 bucket/target CONFIG configuration_json CREDENTIALS credentials_json CREATE PIPELINE pipeline_name AS LOAD DATA S3 'bucket-name' | '' [CONFIG ''] CREDENTIALS '' INTO TABLE table_name_2; START PIPELINE pipeline_name; ``` *** Modified at: September 10, 2025 Source: [/cloud/manage-data/moving-data/moving-data-between-databases/](https://docs.singlestore.com/cloud/manage-data/moving-data/moving-data-between-databases/) (An index of the documentation is available at /llms.txt) --- # Smart Disaster Recovery (DR): SmartDR > **📝 Note**: This is a preview feature supported on AWS, GCP and Azure. Smart DR (A process that handles the continuous asynchronous replication of data between a primary and a secondary region. A primary region is the main geographic location where your database(s) currently reside and operate, while a secondary region is an additional, geographically separate location to which your database(s) are replicated for disaster recovery purposes.) creates and manages a continuous asynchronous replication of data between a primary and a geographically separate secondary region. This service allows you to access the secondary region with minimal downtime or data loss for critical applications. You can initiate Smart DR via the SingleStore Portal or API. The failover and failback processes are completely automated. Upon initiation, SingleStore activates your clusters in the secondary region, connects your databases to these clusters, and generates a connection string. You can then utilize this connection string to access your data or to configure your application. Smart DR replicates the exact topology from the primary to the secondary region and maintains all the users, permissions, and cluster configurations across the regions. ## Benefits The principal benefits of Smart DR are minimal ongoing costs and a low Recovery Point Objective (RPO of up to 10 minutes). Smart DR reduces your ongoing disaster recovery expenses by eliminating the need for active compute resources. You incur charges only for storage and data transfer. ## Use Case A primary use case for Smart DR is to guarantee business continuity in the face of a region outage. This could be due to a natural cause, technical failures, or human errors and actions. Depending on your business requirements, it may be essential to have both Multi-AZ High Availability (HA) and Smart DR. The distinction between the two is crucial because HA focuses on maintaining data availability for day-to-day activities despite minor disruptions, whereas disaster recovery is about recovering and restoring databases following a major regional outage. In conjunction with Smart DR, Point-in-Time Recovery (PITR), gives you the ability to go back in time and recover data in both the primary and secondary regions. ## Setting up Smart DR **Configuring Replication** 1. Go to **Clusters** from the left nav and select the cluster. 2. Click the three vertical dots against the selected cluster, and from the drop down list select **Configure SmartDR**. 3. Click **+Configure Replication**, choose the relevant database options and submit. The **Configure Smart DR** screen section displays 3 top level menu options: 1. **Clusters** displays the clusters name, region and the **Failover Role**. 2. **Databases** displays the replicated databases name and status. Clicking on the **Manage** button allows you to view the available databases and select them for replication. 3. **Settings** displays the **Primary Region** which is the region where your database(s) currently reside. and the **Secondary Region** to which you want to replicate the database(s). **Replication Type** which is **Storage only** by default. This implies the data is copied asynchronously between clusters and the secondary site does not require active compute nodes running. **Auto-replication** is disabled by default.  If enabled, new databases are replicated automatically. ## Pre-provisioning Pre-provisioning can be used to configure compute resources in a secondary region in advance. This enables you to: * Configure the private endpoints in the secondary region, before failover is initiated. * Test DR by failing over to the secondary region without disrupting your production environment. * Failover is faster because compute is already running. To pre-provision: 1. In the Portal, navigate to the `Replication` tab. 2. Click on `Enable Pre-Provisioning`. This starts the background process to configure the secondary region with the same topology of clusters as your primary region. During this process, your primary region continues to actively replicate data to the secondary region. To validate your failover capabilities and ensure business continuity, you can switch to the secondary region, attach your database to a cluster, and start querying the data from the secondary region. You can attach your application in the secondary region and verify that it can insert or update your database as expected, without impacting your production environment. This seamless testing is possible because when you attach your database to a cluster in the secondary region SingleStore Helios automatically creates a branch of your database. This branch reflects all the data up to the point of attachment. More importantly, this branch is independent of your primary region, ensuring that any updates or modifications made during the test failover do not impact your production environment. For more information on branching refer to [Database Branching](https://docs.singlestore.com/cloud/developer-resources/database-branching.md). ## Failover Once the database(s) replication is set up and synced up, you can fail over your application to the secondary region anytime there is a regional failure. To start the process: 1. Click the 3 vertical dots against the selected cluster, and from the drop down list select **Configure SmartDR**. 2. Click the **Failover** button on the left upper side of the **Configure Smart DR** screen. Check the *I confirm…* checkbox and click **Confirm** in the popup window. You can monitor the progress from the status bar. During the failover deployment, the system automatically performs the following tasks in the background: * Provisions the environment in the secondary region, maintaining the primary region's topology. * Provisions and configures all your clusters. * Attaches the databases to the cluster and provides a connection string. * Preserves user permissions, pipelines, firewall settings and other metadata from your primary region. The primary region cluster is automatically suspended as part of the failover task which cannot be resumed or terminated. **System-Managed Databases** After initiating the failover process, you may notice either through the UI or by running the SQL command `SHOW DATABASES`, there are two databases: one attached to the cluster and another with the same name but including a timestamp, in a detached state. The detached database, referred to as a *system-managed* database, is a continuation of your primary region's database. It's called "system-managed" because users cannot directly attach it to a cluster. SingleStore ensures synchronization between the primary region's database and this system-managed database in the secondary region in the background. During failover, SingleStore attaches a branch of this system-managed database to your cluster. In most failovers, the data in the branch database mirrors the detached database. However, in cases where the region connection is abruptly interrupted during data ingest, some data may not fully replicate during failover. In such scenarios, the system-managed database works behind the scenes to sync everything by continuously reconnecting to the primary database to pick up any missing rows. You can access the data in the system-managed database at any time by attaching it as a branch and recovering the missing rows. Importantly, database branches do not consume extra storage, allowing you to create as many branches as needed to read data from these system-managed databases without incurring additional storage costs. Also, SingleStore will display these system-managed databases in the secondary region only if the corresponding database is attached and active in the primary region during failover. ## Failback To initiate failback from the secondary region to the primary region: 1. Click the three vertical dots against the selected cluster, and in the drop down list select **Configure SmartDR**. 2. Click the **Failback** button on the left upper side of the **Configure Smart DR** screen. Check the *I confirm…* checkbox and click **Confirm** in the popup window. You can monitor the progress from the status bar. The system automatically performs the following tasks during failback: * Configures the primary region environment. * Attaches replicated databases to the cluster and provides the connection string. * Updates user permissions and other metadata with changes from the secondary region. Upon successful completion, the primary region becomes active, and the secondary region is no longer accessible. *** Modified at: July 6, 2026 Source: [/cloud/manage-data/smart-disaster-recovery-dr-smartdr/](https://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr/) (An index of the documentation is available at /llms.txt) --- # Query Data Once the data is loaded into SingleStore Helios, it can be queried for both analytical and transactional purposes. SingleStore Helios supports the following query-related features: * The [SELECT](https://docs.singlestore.com/cloud/query-data/select.md) command retrieves data from a database. From this basic SQL command, you can utilize more advanced query concepts such as [performing upserts](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/performing-upserts.md) and creating [common table expressions](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md). * The [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md), [DELETE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/delete.md), and [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) commands manipulate data within a database and can utilize querying elements such as `INSERT... SELECT`, `DELETE` with filters, join, and subqueries, or `UPDATE` with filters, joins, and subqueries. * The [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md), [EXPLAIN UI](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md), [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md), and [SHOW PROFILE UI](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) features can help you understand the performance of your queries. These features allow you to analyze query plans and gather execution statistics, which can aid in identifying bottlenecks and optimizing your database queries. The SingleStore [Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) feature provides query plan visualization through a graphical interface, which can help you tune database queries so that they run faster and/or use less resources. * SingleStore Helios supports [integration](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) with a wide range of third-party data analytics and business intelligence software such as Dremio, Looker, MicroStrategy, Power BI, Sisense, Tableau, and Tibco Spotfire. ## In this section * [SELECT](https://docs.singlestore.com/cloud/query-data/select.md) * [Basic Query Examples](https://docs.singlestore.com/cloud/query-data/basic-query-examples.md) * [Advanced Query Topics](https://docs.singlestore.com/cloud/query-data/advanced-query-topics.md) * [Query Plan Operations](https://docs.singlestore.com/cloud/query-data/query-plan-operations.md) * [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) * [Connect with Analytics and BI Tools](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools.md) *** Modified at: December 16, 2025 Source: [/cloud/query-data/](https://docs.singlestore.com/cloud/query-data/) (An index of the documentation is available at /llms.txt) --- # Advanced Query Topics The following topics give more information querying features are available beyond the basic `SELECT` statement. Then, we will explain some functionalities that will maintain the integrity of your data and how you can set some controls on how data is interpreted, compiled, or locked. Finally, this section gives a real-life example and takes you through the steps of creating a database to running analytics. ## In this section * [Performing Upserts](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/performing-upserts.md) * [WITH (Common Table Expressions)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md) * [Read Query Results in Parallel](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/read-query-results-in-parallel.md) * [Row Locking](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/row-locking.md) * [Two-Phase Transaction Commit](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/two-phase-transaction-commit.md) * [Flexible Parallelism](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/flexible-parallelism.md) * [Example: Building a Sample Stock Trade Database](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database.md) * [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md) *** Modified at: May 14, 2026 Source: [/cloud/query-data/advanced-query-topics/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/) (An index of the documentation is available at /llms.txt) --- # Code Generation A large contributor to SingleStore's superior query execution performance is its system of code generation (An industrial compiler to produce highly efficient machine code that enables low-level optimizations, which are not possible when executing queries via interpretation alone. By default, queries are interpreted first and then asynchronously compiled in the background for use in later executions. This speeds up query execution time for long and complex queries, while at the same time providing efficient query plans for later use.). In contrast to a traditional interpreter-based execution model for databases, SingleStore embeds an industrial compiler to produce highly efficient machine code that enables low-level optimizations, which are not possible when executing queries via interpretation alone. And now by default, queries are interpreted first and then asynchronously compiled in the background for use in later executions. This speeds up query execution time for long and complex queries, while at the same time providing efficient query plans for later use. As stated above, the first time SingleStore encounters a given query shape (Patterns or structures in query. Some query shapes are unsupported in SingleStore.), it will optimize and compile the query asynchronously for future invocations. This incurs minimal overhead which does not depend on the amount of data to be processed, but rather the complexity of the query. The process of code generation involves extracting parameters from the query then transforming the normalized query into a SingleStore-specific intermediate representation tailored to the system. Subsequent requests with the same shape can reuse this plan to complete queries both quickly and consistently. Code generation applies to all [Data Manipulation Language DML](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml.md) queries. In addition, SingleStore generates code during [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) and [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) statements. These [Data Definition Language DDL](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl.md) queries generate code in order to reduce the compilation time of subsequent DML queries against the table. ## The Interpreter Mode’s Effects on Code Generation The `interpreter_mode` setting controls whether SingleStore interprets or compiles query shapes, or does both. This section explains how this mode operates when SingleStore encounters a new query shape. ## `interpreter_mode` is set to `compile` or `llvm` In this mode, SingleStore compiles a query’s shape the first time it is encountered. As shown in the following example, the query `SELECT * FROM t WHERE col = 1;` takes longer to complete on the first run; this is a result of the compile overhead during the first step. Also note that the `WHERE` clauses are different in the second and third query, but both query shapes are identical. Therefore, SingleStore reuses the second query’s plan to run the third query. ```sql SELECT * FROM t WHERE col = 1; ``` ```output Empty set (0.13 sec) ``` ```sql SELECT * FROM t WHERE col = 1; ``` ```output Empty set (0.00 sec) ``` ```sql SELECT * FROM t WHERE col = 100000; ``` ```output Empty set (0.00 sec) ``` > **❗ Important**: Run the [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) command followed by the `SHOW PROFILE JSON` command to view detailed statistics on compilation time (displayed under `compile_time_stats` in the output). ## `interpreter_mode` is set to `compile_lite` In this mode, a query's shape is compiled, without any optimizations, the first time the query shape is encountered. This results in faster compilation than when `interpreter_mode` is set to `compile`. However, because `compile_lite` applies no optimizations, it will execute queries slower than `compile`. ## `interpreter_mode` is set to `compile_lite_first` In this mode, two parallel compilation threads are run: A foreground thread that compiles the query shape in `compile_lite` mode and a background thread that compiles the query shape in `compile` mode. When the background thread completes, the foreground thread switches to the background thread's compiled query plan. Generally, SingleStore recommends using `compile_lite_first` mode instead of `compile_lite` mode. An exception is when using `compile_lite_first` would consume too many CPU resources (as the same query shape is compiled twice). ## `interpreter_mode` is set to `interpret` or `mbc` In this mode, queries are interpreted and not compiled. ## `interpreter_mode` is set to `interpret_first` In this mode, SingleStore interprets and compiles a query shape in parallel, upon encountering the query shape for the first time. The query runs in interpreted mode until compilation of the query shape completes. In the following example, the first `SELECT * FROM t WHERE col = 1;` query runs faster than the first query in the previous example. ```sql SELECT * FROM t WHERE col = 1; ``` ```output Empty set (0.02 sec) ``` ```sql SELECT * FROM t WHERE col = 1; ``` ```output Empty set (0.00 sec) ``` ```sql SELECT * FROM t WHERE col = 100000; ``` ```output Empty set (0.00 sec) ``` As of MemSQL 6.8, `interpreter_mode` is set to `interpret_first` by default. ## Setting the Interpreter Mode `interpreter_mode` can be set: ## Using a Query Option To use the query option, add the `OPTION (interpreter_mode = )` clause to the end of a query. For example: ```sql SELECT * FROM t WHERE col = 1 OPTION (interpreter_mode = compile); ``` The `interpreter_mode` option is non-operational when a query containing the option is run on a leaf node. ## Understanding the Plancache When a query is first run, it activates code generation, gets optimized, and translated into lower-level machine language. On this first run of the query, the code generation for it will be recorded in the logs. This lower-level version of the query, with the wildcard @ replacing for parameters is stored on-disk in the plancache. You can see the plancache files in `/var/lib/memsql//plancache`/. Note the directories in this location, named for their IDs. In the ID dirs, you will find the .mo, .mpl, and .mu machine language files. If the data has undergone a lot of changes, then you might want the plancache to be updated with the new stats by running: [ANALYZE TABLE table\_name;](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md). This will collect new statistics and invalidate previously cached plans on tables that experienced significant changes, After code generation, the compiled query plans are saved for later use in a plancache. Each SingleStore Helios node (aggregator and leaf) has its own plans and plancache. Each user query (on the aggregator) is associated with a plan on that aggregator, and some number of plans (zero, one, or more) on the leaves. There are two layers of this plancache: the in-memory plancache and the on-disk plancache (located in the `@@global.plancachedir` directory). Plans are cached in the in-memory plancache until they expire or until that SingleStore Helios node restarts. A plan expires after the interval of time configured by the `plan_expiration_minutes` global variable has elapsed (the default value of this variable is 720, or 12 hours). When a plan expires from the in-memory plancache, it remains in the on-disk plancache and is loaded back into memory the next time the query is run. If the global variables `enable_disk_plan_expiration` and `disk_plan_expiration_minutes` are set, then a plan that has not been read from disk during the specified `disk_plan_expiration_minutes` limit will expire from the on-disk plancache and be removed from disk. To delete all plans from the in-memory and on-disk plancaches without waiting for garbage collection, refer to [Drop all Plans from the Plancache](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/managing-plancache-memory-and-disk-usage/#section-idm234875781961294.md) for more information. To see DML query plans that have been cached in the in-memory plancache, run the [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) command or query against the `information_schema.PLANCACHE` table. The `information_schema.PLANCACHE` and `information_schema.MV_PLANCACHE` views contain information about query plans that have been compiled and executed, cumulative query execution statistics about each plan, and plancache information for all nodes. Please note the details on how the `plan_id` and `activity_name` columns can change in the plancache views. The following examples are based on this query: `SELECT * FROM table_name ORDER BY column_name...` * When the plan expires from memory as set by the `plan_expiration_minutes` engine variable, if the query is re-run, note that the `plan_id` changes, the `activity_name` does not: ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 13 | Select_planc1_901e63e0a36b09c1 | +---------+--------------------------------+ ``` ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 33 | Select_planc1_901e63e0a36b09c1 | +---------+--------------------------------+ ``` * If plans are dropped from both memory and disk, then both the `plan_id` and `activity_name` changes. ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 33 | Select_planc1_901e63e0a36b09c1 | +---------+--------------------------------+ ``` ```sql DROP ALL FROM PLANCACHE; ``` ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 38 | Select_planc1_2950b7e9280f64bb | +---------+--------------------------------+ ``` * If the definition of the table queried changes or an engine variable that affects query plans changes, then the `plan_id` and `activity_name` columns both change in the plancache. The following example illustrates the change of both the `plan_id` and `activity_name` when a column is added to the table: ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 38 | Select_planc1_2950b7e9280f64bb | +---------+--------------------------------+ ``` ```sql ALTER TABLE planc1 ADD COLUMN d INT; ``` ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 39 | Select_planc1_4d6188772954c27c | +---------+--------------------------------+ ``` * The following example illustrates what happens when a plan-affecting engine variable is changed. In this example, auto profiling is enabled. The previous plan remains valid; however, there are two different plans for the same query. ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+--------------------------------+ | plan_id | activity_name | +---------+--------------------------------+ | 39 | Select_planc1_4d6188772954c27c | +---------+--------------------------------+ ``` ```sql SET GLOBAL enable_auto_profile = 1; ``` ```sql SELECT plan_id, activity_name FROM information_schema.PLANCACHE WHERE query_text LIKE 'SELECT * FROM planc1 ORDER BY%'; ``` ```output +---------+----------------------------------------+ | plan_id | activity_name | +---------+----------------------------------------+ | 39 | Select_planc1_4d6188772954c27c | | 45 | Select_profile_planc1_cb0dd9eed1c43d24 | +---------+----------------------------------------+ ``` ## Overriding Parameterization of Queries Parameterization is when the constants in a query are converted into parameters, so that queries with the same structure can reuse the query plan. Some query shapes defeat the purpose of parameterization, thereby causing large plancaches. To override parameterization, use the [NOPARAM](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/noparam.md) built-in function. ## Non-parametrized Function Parameters Following is a list of functions whose arguments are not parametrization and hence will create a new plan for each different value. | Function | Which argument is not parameterized | Explanation | | ------------------------------------------------------------------------------- | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `EXTRACT(unit FROM datetime)` | 1st argument (unit) | The date/time unit (for example,`DAY`,`MONTH`) is fixed into the plan; different units use different plans. | | `ROUND(num, decimals)` | 2nd argument (decimals) | The number of decimal places is baked into the plan, so`ROUND`(x, 2) and`ROUND`(x, 3) do not share a plan. | | `TRUNC(num, decimals)` | 2nd argument (decimals) | Same idea as`ROUND`: the decimals argument is not parameterized. | | `TRUNCATE(num, decimals)` | 2nd argument (decimals) | Same as above; changing decimals gives a different plan. | | `LPAD(str, len [, padstr])` | 2nd argument (len) | The target length is fixed into the plan; different lengths lead to different plans. | | `RPAD(str, len [, padstr])` | 2nd argument (len) | Same as`LPAD`for the len argument. | | `DATE_FORMAT(date, format [, ...])` | 2nd and later arguments (starting with format) | The format string (and any extra format arguments) are fixed; queries with different formats do not share a plan. | | `TIME_FORMAT(time, format [, ...])` | 2nd and later arguments | Same behavior as`DATE_FORMAT`. | | `TO_DATE(str, format [, ...])` | 2nd and later arguments (starting with format) | The format is not parameterized so the optimizer can specialize on it. | | `TO_NUMBER(str, format [, ...])` | 2nd argument (format) | The numeric parsing format is fixed into the plan. | | `FROM_UNIXTIME(unix_ts [, format, ...])` | 2nd and later arguments (starting with format) | The output format and related arguments are not parameterized. | | `GROUPING(expr)` | 1st argument (expr) | The grouping expression is treated as part of the plan shape, not as a parameter. | | `NOW([fsp])` | All arguments (normally the optional fsp) | The fractional-seconds precision (fsp) is fixed into the plan when present. | | `UTC_TIMESTAMP([fsp])` | All arguments | Same as`NOW`for the optional fsp. | | `UTC_TIME([fsp])` | All arguments | Same behavior as above. | | `CURTIME([fsp])` | All arguments | Same behavior as above. | | `GET_FORMAT(type, format)` | 1st argument (type) | The type (for example,`DATE`,`DATETIME)`is fixed into the plan. | | `JSON_MATCH_ANY(json_doc, path, pattern, ...); BSON_MATCH_ANY(...)` | 3rd and later arguments (patterns) | The pattern and any additional pattern arguments are not parameterized. | | `JSON_MATCH_ANY_EXISTS(json_doc, path, ...); BSON_MATCH_ANY_EXISTS(...)` | 2nd and later arguments (paths / patterns) | The path / pattern arguments are fixed into the plan. | | `JSON_EXTRACT(...)`family | 2nd and later arguments (paths), in some cases | The JSON path arguments are not parameterized when JSON key parameterization is disabled or inside a`JSON_MATCH_ANY`subtree. | | `BSON_EXTRACT_*`family | 2nd and later arguments (paths) | The BSON path arguments are not parameterized. | | `JSON_INCLUDE_MASK / JSON_EXCLUDE_MASK / BSON_INCLUDE_MASK / BSON_EXCLUDE_MASK` | 2nd and later arguments (mask / path) | The mask / path arguments are fixed, not parameterized. | | Vector sort functions (`VECTOR_SORT_I8 … VECTOR_SORT`) | 2nd argument | The second argument (for example, sort direction or related option) is not parameterized. | | `_MEMSQL_TABLE_ID_LOOKUP (internal)` | 1st argument | Internal helper; first argument is not parameterized. | | `INTERNAL_TEXT_TO_JSON(text, path) (internal)` | 2nd argument (path) | Internal helper; the JSON path is not parameterized. | ## Codegen Architecture Code generation produces a specialized programming language, the SingleStore (MemSQL) Plan Language. MPL is simple, tightly managed and designed specifically for SingleStore rather than for general purpose applications. SQL operator trees are converted directly to MPL abstract syntax trees, eliminating the need for expensive parsing and semantic resolution at query compilation time. In addition to the simplicity of the language, MPL benefits from exposing as primitives many relational operations such as “index seek” and scalar operations such as “`BIGINT` addition”. These operators are compiled statically along with the server and can easily be combined to form a query plan. In this way, much of the work of compilation can be performed ahead of time. ## SingleStore Bytecode When compiling a query, SingleStore first flattens MPL into a compact format known as SingleStore (MemSQL) Bytecode (MBC). Plans in this form can easily be serialized, interpreted or transformed to [LLVM](http://www.wikipedia.com/wiki/LLVM) bitcode for compilation to `x86_64` machine code. Interpreting the bytecode directly nearly eliminates the first-time cost of running a query, but can hamper execution performance: ```sql SELECT * FROM t WHERE j > 0; ``` ```output Empty set (0.05 sec) ``` ```sql SELECT * FROM t WHERE j > 0; ``` ```output Empty set (0.05 sec) ``` In this example, there is no additional latency on the first request because the query is interpreted. ```sql SELECT * FROM t WHERE j > 0; -- With code generation ``` ```output Empty set (0.08 sec) ``` ```sql SELECT * FROM t WHERE j > 0; -- Using the cached plan ``` ```output Empty set (0.02 sec) ``` Here, the additional latency on the first request is the result of code generation. However, the subsequent request is more than twice as fast as the interpreted execution above. This goes to show the power of SingleStore code generation. *** Modified at: July 10, 2026 Source: [/cloud/query-data/advanced-query-topics/code-generation/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation/) (An index of the documentation is available at /llms.txt) --- # Example: Building a Sample Stock Trade Database This tutorial shows you how to build a sample database containing millions of simulated stock trade data rows. You can use this database to test the features and functionality of SingleStore Helios. Some sample queries are also provided to show you how SingleStore Helios performs when doing concurrent reads while data is being written to disk. ## In this section * [Step 1: Create the database](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/step-1-create-the-database.md) * [Step 2: Load company profile data](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/step-2-load-company-profile-data.md) * [Step 3: Create data generator functionality](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/step-3-create-data-generator-functionality.md) * [Step 4: Generate trade data](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/step-4-generate-trade-data.md) * [Step 5: Run analytic queries](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/step-5-run-analytic-queries.md) * [Query 1](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/query-1.md) * [Query 2](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/query-2.md) * [Query 3](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/query-3.md) * [Query 4](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/query-4.md) *** Modified at: November 20, 2024 Source: [/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/example-building-a-sample-stock-trade-database/) (An index of the documentation is available at /llms.txt) --- # Flexible Parallelism SingleStore supports Flexible Parallelism (FP), which allows multiple cores on the same node to access the same database partition. FP is beneficial for analytic-style queries that process large amounts of data as it increases parallelism within a query. As cores are added to a SingleStore system, it is possible to reach a point where there are more cores than database partitions on each leaf node. Without FP, there is a limit of one core accessing each database partition and the maximum ratio of cores to partitions is 1:1. With FP, if additional cores are added to a database, multiple cores can be used to process a single partition. Thus, queries that process large amounts of data execute more quickly. Typically, analytic or hybrid transactional/analytic workloads gain benefits from FP, especially over time as you add more hardware and rebalance clusters. Strictly OLTP workloads typically do not benefit from FP. ## Types of Flexible Parallelism There are two types of FP in SingleStore: sub-partition parallelism and segment parallelism. Both types of FP enable multiple cores on a leaf to process different portions of a partition in parallel. ## Sub-Partition Parallelism Sub-partition parallelism is available on databases with [sub-partitions](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database/#section-idm4626710146376034307367484458.md). During query execution, the query threads on a leaf node are each assigned a set of sub-partitions, and the query threads scan their assigned sub-partitions in parallel. A common configuration is to specify one query thread per core, and in that configuration, each core scans a set of sub-partitions. Sub-partitions are divisions of database partitions. The engine variable `sub_to_physical_partition_ratio` variable controls the number of sub-partitions that are created per physical partition. When `sub_to_physical_partition_ratio` is set to a valid value greater than 0 (a power of 2 up to 64) before a database is created, sub-partitions are created and sub-partition parallelism is available to queries on columnstore tables in that database. Sub-partitions must be created at database creation time and cannot be created on existing databases. A database created without sub-partitions cannot use sub-partition parallelism. To add sub-partitions to an existing database, create a database with sub-partitions and copy the tables from the original database to the new one. One way to do this is with `CREATE TABLE LIKE` and `INSERT… SELECT`. The maximum sub-partition parallelism is `sub_to_physical_partition_ratio` \* number of physical partitions. ## Segment Parallelism For certain query shapes, the engine processes [segments](https://docs.singlestore.com/cloud/create-a-database/columnstore/managing-columnstore-segments.md) of columnstore tables in parallel. This functionality is called segment parallelism. In segment parallelism, there is no pre-computed split of work across threads; instead, when a thread finishes its work, that thread may be assigned to another segment to scan. Segment parallelism is controlled by the `query_parallelism_per_leaf_core` engine variable as described in the following sections. ## Sub-Partition vs. Segment Parallelism Segment parallelism differs from sub-partition parallelism in the granularity of the work done by the query threads and in the queries to which parallelism may be applied. In sub-partition parallelism, each query thread scans a pre-computed set of sub-partitions. Whereas, in segment parallelism, when a query thread finishes its work, that thread may be assigned to another segment to scan. Thus, segment parallelism allows threads that finish their work earlier to help other threads (at a segment granularity), while sub-partition level parallelism does not allow that behavior. Segment parallelism is preferred over sub-partition level parallelism, when possible, but segment parallelism is only applicable to simple query shapes that do not utilize the shard key. ## Flexible Parallelism Settings New clusters in SingleStore Helios are created with sub-partitions and can use sub-partition parallelism. The following settings are used by default: * One physical partition per leaf core at database creation time * 16 sub-partitions per physical partition * One thread per leaf core per query If you’re an existing customer who began using SingleStore Helios before May 2022, and wish to use sub-partition parallelism, contact Support. *** Modified at: July 6, 2026 Source: [/cloud/query-data/advanced-query-topics/flexible-parallelism/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/flexible-parallelism/) (An index of the documentation is available at /llms.txt) --- # Performing Upserts An upsert inserts a row `r` into a table if the table does not already contain an existing row with the same primary key or unique key as `r`. Otherwise, the upsert updates the existing row. An upsert can be performed by specifying the `ON DUPLICATE KEY UPDATE` clause of the `INSERT` statement. The following applies for an upsert: * If a row is to be inserted that would result in a duplicate value in a `PRIMARY KEY` or `UNIQUE KEY`, SingleStore Helios will instead perform an `UPDATE` of the old row. * An upsert can only be done against a `PRIMARY KEY` or `UNIQUE KEY` column. * `ON DUPLICATE KEY UPDATE` cannot update unique key columns in columnstore tables. For rowstore tables, `ON DUPLICATE KEY UPDATE` is allowed as long as the updated column isn't part of the table's explicitly defined shard key or primary key if there's no explicitly defined shard key. * An upsert conflict occurs when there are multiple rows that could potentially be updated, because there are multiple unique keys defined on the table. When such a conflict occurs, only one row is updated. There is no guarantee as to which one. * The affected row count will be 0 if no existing row is changed, 1 if a new row is inserted, and 2 if an existing row is updated. In Example 1 below, the `ON DUPLICATE KEY UPDATE` clause updates the existing row instead of inserting a new row to avoid duplicate entry in a `PRIMARY KEY` field, and the query result shows that 2 rows were affected. ## Examples The following examples use the `cust` table, which is defined as a columnstore table as follows: ```sql CREATE TABLE cust(name VARCHAR(32), id INT(11), orders INT(11), SORT KEY(id), UNIQUE KEY(id) USING HASH, SHARD KEY(id)); ``` ```sql DESC cust; ``` ```output +--------+-------------+------+------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------+-------------+------+------+---------+-------+ | name | varchar(32) | YES | | NULL | | | id | int(11) | YES | PRI | NULL | | | orders | int(11) | YES | | NULL | | +--------+-------------+------+------+---------+-------+ ``` Add three rows of sample data for the examples: ```sql INSERT INTO cust (name, id, orders) VALUES ("Chris",7214,2), ("Adam",3412,5), ("Elen",8301,4); ``` You could also define `cust` as a rowstore table, as follows, and the examples would return the same results. ```sql CREATE ROWSTORE TABLE cust(name VARCHAR(32), id INT(11) PRIMARY KEY, orders INT(11)); ``` ## Example 1: Basic Example of ON DUPLICATE KEY UPDATE The following example demonstrates the use of `INSERT` with `ON DUPLICATE KEY UPDATE`. ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 2 | | Elen | 8301 | 4 | | Adam | 3412 | 5 | +-------+------+--------+ ``` ```sql INSERT INTO cust (ID, ORDERS) VALUES (7214, 3) ON DUPLICATE KEY UPDATE ORDERS=3; ``` ```output Query OK, 2 rows affected ``` ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 3 | | Elen | 8301 | 4 | | Adam | 3412 | 5 | +-------+------+--------+ ``` ## Example 2: Using the VALUES() Function The `VALUES()` function can be used to refer to column values from the `INSERT` clause of the `INSERT ... ON DUPLICATE KEY UPDATE` statement. For example: ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 2 | | Adam | 3412 | 5 | | Elen | 8301 | 4 | +-------+------+--------+ ``` ```sql INSERT INTO cust (ID, ORDERS) VALUES (7214, 4) ON DUPLICATE KEY UPDATE ORDERS = VALUES(ORDERS) + ORDERS; ``` ```output Query OK, 2 rows affected ``` ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 6 | | Adam | 3412 | 5 | | Elen | 8301 | 4 | +-------+------+--------+ ``` Here, the `VALUES()` function refers to the value that would have been inserted, if there were no duplicate-key conflicts, and should refer to a column name of the destination table. ## Example 3: Using SELECT with ON DUPLICATE KEY UPDATE The following code block demonstrates the use of `SELECT` with `ON DUPLICATE KEY UPDATE`: ```sql SELECT * FROM cust_new; ``` ```output +------+------+--------+ | NAME | ID | ORDERS | +------+------+--------+ | Bill | 21 | 5 | | Gwen | 7214 | 3 | | Sam | 22 | 2 | +------+------+--------+ ``` The `cust_new` table is defined using the same table definition as `cust`, which is shown in the beginning of the examples. ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 6 | | Elen | 8301 | 4 | | Adam | 3412 | 5 | +-------+------+--------+ ``` ```sql INSERT INTO cust (NAME, ID, ORDERS) SELECT * FROM cust_new ON DUPLICATE KEY UPDATE NAME = VALUES(NAME), ORDERS = VALUES (ORDERS); ``` ```output Query OK, 4 rows affected Records: 3 Duplicates: 1 Warnings: 0 ``` ```sql SELECT * FROM cust; ``` ```output +------+------+--------+ | NAME | ID | ORDERS | +------+------+--------+ | Bill | 21 | 5 | | Gwen | 7214 | 3 | | Elen | 8301 | 4 | | Sam | 22 | 2 | | Adam | 3412 | 5 | +------+------+--------+ ``` Here, the `VALUES()` function refers to the value that would have been inserted, if there were no duplicate-key conflicts, and should refer to a column name of the destination table. ## Example 4: Updating Values through a Two-step Process The above examples are functionally equivalent to this common two-step operation. **Note:** While this works, it is much less performant than the single-steps above. ```sql SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Chris | 7214 | 6 | | Elen | 8301 | 4 | | Adam | 3412 | 5 | +-------+------+--------+ ``` ```sql SELECT * FROM cust_new; ``` ```output +------+------+--------+ | NAME | ID | ORDERS | +------+------+--------+ | Bill | 21 | 5 | | Gwen | 7214 | 3 | | Sam | 22 | 2 | +------+------+--------+ ``` The `cust_new` table is defined using the same table definition as `cust`, which is shown in the beginning of the examples. Copy all the unique values from the `cust_new` to `cust` table. ```sql INSERT IGNORE INTO cust SELECT * FROM cust_new; SELECT * FROM cust; ``` ```output +-------+------+--------+ | NAME | ID | ORDERS | +-------+------+--------+ | Bill | 21 | 5 | | Chris | 7214 | 6 | | Elen | 8301 | 4 | | Sam | 22 | 2 | | Adam | 3412 | 5 | +-------+------+--------+ ``` Now, perform an update to copy all the non-unique (duplicate) values. ```sql UPDATE cust JOIN cust_new ON ( cust_new.ID=cust.ID ) SET cust.NAME = cust_new.NAME, cust.ORDERS = cust_new.ORDERS; SELECT * FROM cust; ``` ```output +------+------+--------+ | NAME | ID | ORDERS | +------+------+--------+ | Bill | 21 | 5 | | Gwen | 7214 | 3 | | Elen | 8301 | 4 | | Sam | 22 | 2 | | Adam | 3412 | 5 | +------+------+--------+ ``` ## Example 5: Using ON DUPLICATE KEY UPDATE with a Pipeline The following example demonstrates an upsert on a Pipeline. Assume the directory `/order_files` has one file `orders1.txt`, which contains the following data: ``` Chris,7214,6 Elen,8301,4 Adam,3412,5 Rachel,9125,2 Susan,8301,7 George,3412,9 ``` Create a Pipeline with an `ON DUPLICATE KEY UPDATE` clause: ```sql CREATE PIPELINE p AS LOAD DATA FS '/order_files/orders*.txt' INTO TABLE cust FIELDS TERMINATED BY ',' ON DUPLICATE KEY UPDATE orders = 3; ``` Start the Pipeline: ```sql START PIPELINE p; ``` As the Pipeline ingests the data from `orders1.txt` into the `cust` table, it encounters the fifth and sixth records in the file, which contain the duplicate keys `8301` and `3412`. The second and third records containing those duplicate keys (which have already been imported into `cust`), are updated with `3`, because `orders = 3` is specified in the `ON DUPLICATE KEY UPDATE` clause. ```sql SELECT * FROM cust ORDER BY name; ``` ```output +--------+------+--------+ | name | id | orders | +--------+------+--------+ | Adam | 3412 | 3 | | Chris | 7214 | 6 | | Elen | 8301 | 3 | | Rachel | 9125 | 2 | +--------+------+--------+ ``` Example 5 works because the new value for `orders` is specified directly (`orders = 3`). If you try to update the value with an expression (such as a value from the input file), you will see an error like this: ``` Feature 'UPDATE value depends on tables other than that being updated.' is not supported by MemSQL. ``` In order to update existing values with a new value specified in the pipeline, use the `VALUES()` clause. So if the orders.txt file looks instead like this: ``` Chris,7214,6 Elen,8301,99 Adam,3412,99 Rachel,9125,2 Susan,8301,99 George,3412,9 ``` And we update the ON DUPLICATE KEY UPDATE statement: ```sql CREATE PIPELINE p2 AS LOAD DATA FS '/order_files/orders*.txt' INTO TABLE cust FIELDS TERMINATED BY ',' ON DUPLICATE KEY UPDATE orders = VALUES(orders); START PIPELINE p2; ``` The updated results: ``` SELECT * FROM cust ORDER BY name; ``` ```output +--------+------+--------+ | name | id | orders | +--------+------+--------+ | Adam | 3412 | 9 | | Chris | 7214 | 6 | | Elen | 8301 | 99 | | Rachel | 9125 | 2 | +--------+------+--------+ ``` ## Example 6: Using `ON DUPLICATE KEY UPDATE` with a Multi-Column Key This example uses the `product` table, which is defined as follows, with the multi-column unique key `(id_1,id_2)`. ```sql CREATE TABLE product(name VARCHAR(32), id_1 INT(11), id_2 INT(11), quantity INT(11), SORT KEY (id_1,id_2), UNIQUE KEY(id_1,id_2) USING HASH, SHARD KEY(id_1,id_2)); ``` Assume the table contains the following data: ``` +-------------+------+------+----------+ | name | id_1 | id_2 | quantity | +-------------+------+------+----------+ | blue pen | 2792 | 3 | 200 | | red pen | 2792 | 5 | 250 | | white paper | 4624 | 1 | 100 | | brown paper | 4624 | 8 | 175 | +-------------+------+------+----------+ ``` Attempt to insert a record with the quantity `325`. Because a record containing the key `(2792,5)` is already in the table, the existing record is updated with `325`: ```sql INSERT INTO product(name,id_1,id_2,quantity) VALUES ('red pen',2792,5,325) ON DUPLICATE KEY UPDATE quantity = 325; ``` Attempt to insert a record with the quantity `125`. Because there is no existing record containing the key `(4624,7)`, the insert succeeds. ```sql INSERT INTO product(name,id_1,id_2,quantity) VALUES ('yellow paper',4624,7,125) ON DUPLICATE KEY UPDATE quantity = 125; ``` ```output +--------------+------+------+----------+ | name | id_1 | id_2 | quantity | +--------------+------+------+----------+ | blue pen | 2792 | 3 | 200 | | red pen | 2792 | 5 | 325 | | white paper | 4624 | 1 | 100 | | yellow paper | 4624 | 7 | 125 | | brown paper | 4624 | 8 | 175 | +--------------+------+------+----------+ ``` ## Example 7: Using `ON DUPLICATE KEY UPDATE` with JSON data from Kafka The following example demonstrates the use of `ON DUPLICATE KEY UPDATE` with input data in the JSON format from Kafka. Define a table where the data will be inserted: ```sql CREATE TABLE users ( user_id INT PRIMARY KEY, user_name VARCHAR(100), user_email VARCHAR(100) ); ``` Sample JSON data input from Kafka can be: ``` { "user_id": 1, "user_name": "John Doe", "user_email": "johndoe@example.com" } ``` Create a pipeline to ingest this data from Kafka: ```sql CREATE PIPELINE user_pipeline AS LOAD DATA KAFKA 'kafka_server/my-topic' INTO TABLE users FORMAT JSON (user_id <- user_id, user_name <- user_name, user_email <- user_email) ON DUPLICATE KEY UPDATE user_name = VALUES(user_name), user_email = VALUES(user_email); ``` Run the pipeline USER\_PIPELINE: ```sql START PIPELINE user_pipeline; ``` View the contents of the table USERS: ``` SELECT * FROM users; ``` ```output +----------+-----------+--------------------+ | user_id | User_name | user_email | +----------+-----------+--------------------+ | 1 | John Doe. | johndoe@example.com| +----------+-----------+--------------------+ ``` *** Modified at: December 16, 2025 Source: [/cloud/query-data/advanced-query-topics/performing-upserts/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/performing-upserts/) (An index of the documentation is available at /llms.txt) --- # Read Query Results in Parallel In SingleStore, the results of a `SELECT` statement can be read in parallel by multiple readers. Each reader queries a unique partition, which contains a slice of the result set. Parallel query reads are intended for use by client applications that query data from SingleStore. These client applications need to connect to the master aggregator to run parallel query reads. A parallel query read can be coded in your client application logic using the following steps, in order. 1. Create a result table to store the result set whose data the application will query in parallel. ```sql CREATE [MATERIALIZED] RESULT TABLE AS SELECT ... FROM ; ``` The result table can be created in one of two read modes: * Single-read mode: Each reader can read the query results, from a specified partition, only once. * Multiple-read mode: Each reader can read the query results, from a specified partition, more than once. Also, multiple readers can read from the same partition. In this mode, creating the result table is slower and consumes more memory, as compared to single-read mode. To create the result table in multiple-read mode, specify the `MATERIALIZED` option. If the contents of `` change after the result table is created, the contents of the result table are not updated. > **📝 Note**: The connection that is used to create the result table (both non-materialized and materialized) must be open while a parallel read is in progress. Once the connection used to run `CREATE RESULT TABLE ...` is closed, the result table is dropped. 2. From each reader, query a partition in the result table that was created in the previous step. ```sql SELECT * FROM :: WHERE partition_id() = ; ``` There are some limits on the query shapes (Patterns or structures in query. Some query shapes are unsupported in SingleStore.) that are supported by the aggregator result table: * `SELECT` from an aggregator results table. It can only include `*` or a list of columns. There should be only exactly one `partition_id() = ` filter. It can include an optional `partition_row_id()` comparison operator filter (`=`, `<`, `>`, `<=`, `>=`, or `!=`). For example: ```sql SELECT * FROM :: t1_result_table WHERE partition_id() = 1 and partition_row_id() < 4; ``` * `INSERT INTO … SELECT` from an aggregator result table into a sharded table. It can only include `*` or a list of columns. The aggregator result table and the sharded table should have a matching shard key or an error will be generated. Filters are not allowed. For example: ```sql INSERT INTO t1 (colint, colchar, colst) SELECT * FROM :: t1_result_table; ``` > **📝 Note**: In single-read mode, each reader needs to run in a separate parallel process or thread. Also, in single-read mode, each reader must initiate its read query before any readers will receive data. To find the number of partitions in a database, run: ```sql SELECT num_partitions FROM information_schema.DISTRIBUTED_DATABASES WHERE database_name = ''; ``` 3. When readers no longer read from the result table, drop the result table. ```sql DROP RESULT TABLE ; ``` `DROP RESULT TABLE` can be run on the master aggregator or a child aggregator. *** Modified at: February 27, 2024 Source: [/cloud/query-data/advanced-query-topics/read-query-results-in-parallel/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/read-query-results-in-parallel/) (An index of the documentation is available at /llms.txt) --- # Row Locking SingleStore transactions use standard [2-phase locking](https://en.wikipedia.org/wiki/Two-phase_locking) for concurrency control to ensure serializability. Row locks are acquired as rows are written to and are held until the transaction that acquired them commits or rolls back. SingleStore takes row locks on write operations (for example, `DELETE`, `INSERT`, or `UPDATE`). These row locks resolve write conflicts in the user’s workload. They are not used to make the underlying data structure backing the index (skiplist or hashtable) thread safe. Row locks are not taken on read operations. There is no other locking mechanism in the SingleStore storage engine beyond these transaction locks. These locks are different from the traditional databases which also have locks inside the index backing data structure. Unlike SingleStore, the locks inside indexes in other systems are taken on both the read and write operations and are a source of contention in highly concurrent workloads. Columnstore tables either lock at the row level or the partition level, depending on the `columnstore_table_lock_threshold` value. See the [Locking in Columnstores](https://docs.singlestore.com/cloud/create-a-database/columnstore.md) topic for more information. SingleStore takes locks before changing the metadata for a blob file. (There is one blob file per column that stores 100K rows of compressed data). The file contents themselves are not updated, instead the bitmap metadata that tracks the deleted rows in the file is updated. ## Acquiring Row Locks Consider the following example table: ```sql CREATE TABLE tabs (c1 INT PRIMARY KEY, c2 INT, c3 INT, KEY(c2)); ``` There are two different ways queries acquire row locks in SingleStore: 1. The query is scanning a secondary key to run the write operation. For example, the query `DELETE FROM tabs WHERE c2 = 1` will seek and scan the secondary index on `c2` to run the `DELETE` operation. When the query finds a row where `c2 = 1`, it will seek into the primary key to acquire the row lock before marking the row as deleted. 2. The query is scanning the primary key to run the write operation. For example, the query `DELETE FROM tabs WHERE c1 = 1` will seek and scan the primary key on `c1` to run the `DELETE` operation. Since the query is already scanning the primary key, the index scan itself will acquire row locks to avoid an extra seek into the primary key (as is done when `DELETE` is scanning the secondary key). While scanning the primary key, the lock will be acquired before running any filters in the `WHERE` clause since the index scan operation itself cannot run those filters. The scan runs inside the storage layer, and the filters are run inside the query execution layer. This means even rows that will not be deleted are locked. For example, the query `DELETE FROM tabs WHERE c1 = 1 AND c3 = 1` locks rows where `c3` is not `1`. Although SingleStore releases the row lock if the filter doesn’t match (for example where `c3` is not `1`), for applications running concurrent `DELETE` queries, this operation is often not fast enough to avoid deadlocks. For example, the queries `DELETE FROM tabs WHERE c1 = 1 AND c3 = 1` and `DELETE FROM tabs WHERE c1 = 1 AND c3 = 2` will never deadlock if `c1` is not the primary key. The following example demonstrates how multi-table filters may lock rows that do not match the filters. Consider the following query, ```sql UPDATE stock JOIN product ON stock.qty = 10 AND stock.id = product.id SET ... ``` This query locks all the rows of the table `stock` where `stock.qty = 10`, including the rows where `stock.id` is not equal to `product.id`. Alternatively, use a single table filter to trim the number of rows locked. For columnstore tables, there is no difference in locking when filtering on key or non-key columns. For rowstore tables, filtering on key and non-key columns follow different code paths, which may affect the locks acquired. ## Extra Deadlocks in SingleStore Due to Sharding In a single box database, concurrent write operations on a table that scan the same index in a specific direction (either forward or reverse) cannot deadlock. For example, consider a transaction A that gets a lock on row(n) and is waiting for a lock on row(n+1). Now, consider another transaction B that gets a lock on row(n+1) and is waiting for a lock on row(n). If both the transactions scan the index in the same order, they will see the rows in the same order, and this deadlock ordering is impossible. In SingleStore, the index is partitioned up amongst shards that are spread across a cluster of leaf nodes. Even if the threads are scanning the index in the same order on each partition, they can still deadlock. For example, consider two `DELETE` commands that run at the same time, starting on two different partitions (say partitions 0 and 1). Let’s say, one of the two `DELETE` operations runs first on partition 0 and the second runs first on partition 1. Now, they may hit the lock acquisition deadlock in the example above. Single partition `DELETE` operations don’t have this issue. You can achieve a single partition `DELETE` operation by using a shard key filter in the `WHERE` clause. If the deadlocks are too frequent, see the [ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-1205-hy-000-lock-wait-timeout-exceeded-try-restarting-transaction.md) topic. *** Modified at: August 18, 2023 Source: [/cloud/query-data/advanced-query-topics/row-locking/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/row-locking/) (An index of the documentation is available at /llms.txt) --- # Two-Phase Transaction Commit SingleStore supports atomic, distributed transactions. Two-phase commit (2PC) is used in conjunction with locking to ensure data consistency across nodes in distributed transactions. Refer to [BEGIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/begin.md) for more information about using transactions in SingleStore. ## The 2PC Protocol The 2PC protocol ensures that all changes in a transaction are either committed or rolled back. Transactions in distributed database systems such as SingleStore modify data on multiple nodes. To maintain the integrity of a transaction, the database must make sure that one of two things happens: either all of the transaction's updates on all involved nodes are completed (the transaction commits) or none of the updates are completed (the transaction is rolled back). 2PC coordinates among the nodes to ensure this all-or-nothing behavior, also called transaction atomicity, is enforced. In 2PC, a node is designated as the coordinator and is responsible for coordinating transaction commit with the other nodes. The 2PC protocol operates in two phases, a prepare phase and a commit phase. In the prepare phase, the coordinator contacts all nodes involved in the update and sends them a prepare request, each node responds with a vote to confirm if they are (or are not) OK to commit. Once all the votes are received, if all nodes have indicated they are OK to commit, the coordinator sends out commit requests. The nodes each commit the transaction locally and send an acknowledgement (`ack`) back to the coordinator to indicate their local commit is complete. If one or more nodes have voted NO indicating they cannot commit, the coordinator will send rollback requests to all the nodes to rollback the entire transaction. Once the coordinator has received and recorded all the YES votes, the transaction is considered committed. If a node crashes or the connection between nodes is lost after this point, the logging process on all the nodes will ensure that all changes in the transaction are still recorded. ## 2PC Example Consider a `songs` table and the following query. Assume that there are millions of rows in the `songs` table which are distributed across many nodes. ```sql CREATE TABLE songs(Id INT, Name TEXT, Genre TEXT, Rating DOUBLE, PublishTimestamp DATETIME); ``` ```sql UPDATE songs SET Genre = 'Pop' WHERE Genre = 'Pop Music'; ``` This query updates all the rows in the `songs` table that have `Genre = 'Pop Music'` to `Genre = 'Pop'`. The rows with `Genre = 'Pop Music'`, which need to be updated, are distributed across multiple nodes, so this update involves multiple nodes. Since all the queries are run as transactions, this update must be atomic (all-or-nothing); that is either all rows with `Genre='Pop Music'` are updated or none of them are updated. Or, in other words, the updates in this query must be atomic, one of the ACID properties of transactions. The 2PC protocol helps ensure that atomicity. The figure below illustrates 2PC on the `songs` table distributed across two nodes. The coordinator first sends prepare requests to both the nodes. The nodes then vote either YES or NO. If both the nodes vote YES, then commit requests are sent to both nodes. The nodes commit the transactions locally, and respond with an `ack` to the coordinator. If any of the nodes vote NO then the coordinator sends a rollback request to all the nodes. ![Two phase Transaction Commit page](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4b9fd639ae0bc525/6a7adb4d1967ea16ad32506a/2PC_Figures-3Ts9hc.png) In SingleStore, the coordinator role is performed by the aggregator and Node 1, Node 2 are leaf nodes. All queries such as the above are run in transactions. The [BEGIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/begin.md), [COMMIT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/commit.md), and [ROLLBACK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/rollback.md) statements can be used to run multi-statement transactions, that is to run multiple updates within a transaction. The same 2PC process is used for those transactions. The above query uses an `UPDATE` statement, the discussion and behavior apply to all data updates including `INSERT` and `DELETE` DML commands. ## Troubleshooting 2PC If an attempt to kill a query (transaction) occurs while 2PC is in process, the query may end up in an unknown state and the system will send the following error message. Note that the request to terminate a query may come from a user request or from the SingleStore system. `OperationalError: 1997: Attempted to interrupt transaction execution, but the outcome is unknown because the transaction was already in its commit phase. Please check if it succeeded.` This error message means that a transaction (or a DML command) was in 2PC at the time the request to kill the transaction was received and so the system cannot determine if the request was received in time to kill the transaction or not. In this case, the user needs to check their data given their knowledge of what the update was doing, and determine if the transaction was committed or rolled back. That is, the transaction will either be rolled back or committed; however, the system cannot tell which action occurred. ## 2PC Impacts The 2PC feature is enabled by default on all SingleStore 8.5 versions and greater. The 2PC protocol causes a negligible performance impact. *** Modified at: July 10, 2026 Source: [/cloud/query-data/advanced-query-topics/two-phase-transaction-commit/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/two-phase-transaction-commit/) (An index of the documentation is available at /llms.txt) --- # WITH (Common Table Expressions) The `WITH` command specifies a temporary named result set, referred to as a Common Table Expression (CTE) (A named temporary result set that exists within the scope of a single statement and that can be referred to later within that statement, possibly multiple times.). ## Syntax Non-recursive (standard) CTE: ```sql WITH cte [, ...] cte: expression_name[(column_name, ...)] AS select_statement ``` Recursive CTE: ```sql WITH RECURSIVE cte [, ...] cte: expression_name [(column_name, ...)] AS ( UNION ALL ) select_statement; ``` ## Remarks * `WITH` clauses are supported as part of `SELECT`, `INSERT`, `UPDATE`, `DELETE`, and `REPLACE` statements for non-recursive CTEs. * Recursive CTEs must have the `materialize_ctes`[engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/non-sync-variables.md) enabled to work. If this engine variable is not enabled, an error is generated when attempting to execute a recursive CTE. ```sql ERROR 2730 UNKNOWN_ERR_CODE: Unsupported recursive common table expression query shape: the engine variable 'materialize_ctes' is currently set to OFF. In order to run the recursive common table expression, you need to set 'materialize_ctes' to 'AUTO' or 'ALL'. See the Materializing Common Table Expressions documentation for additional information. ``` * `WITH RECURSIVE` clauses defining recursive CTEs are only supported for `SELECT` statements for both the base case and the recursive case. * SingleStore supports materializing CTEs without recomputing them when the query contains `UNION/UNION ALL` and other `SET` operations. To enable the feature, set the engine variable `allow_materialize_cte_with_union` to `TRUE`. The default value for this engine variable is `FALSE` for version 8.1.19 and earlier. For more information about materializing CTEs, refer to [Materializing Common Table Expressions](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions/#section-id235694658605145.md). * By default, SingleStore Helios attempts to materialize CTEs. Refer to [Materializing Common Table Expressions](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions/#section-id235694658605145.md) for more information. * If a CTE defined in the `WITH` clause is not referenced in the `SELECT` statement, it does not affect the execution of the query. * The engine variable `internal_max_cte_depth` defines the maximum number of common table expressions (CTEs) that can be nested. The default value of this variable is `128`. * The engine variable `max_recursive_cte_iterations` sets the maximum number of times the recursive case is executed. ## Non-recursive vs recursive CTEs A non-recursive standard CTE does not reference itself within the statement it is executed in. Non-recursive CTEs are inline view (A SELECT statement embedded in the FROM or WITH clause of another SELECT statement that creates a temporary table that is operated on by the outer query.) definitions that appear at the start of a query. Also, a non-recursive CTE can be used to simplify a query by isolating subselects out of the main SQL statement. A recursive CTE references itself. They are useful when working with hierarchical data (A set of data items that are related to each other by hierarchical relationships. Hierarchical relationships exist where one item of data is the parent of another item(s).) because the CTE continues to execute until the query returns the desired portion of the hierarchy. A common example of hierarchical data is an organization chart where employees are listed under their direct managers. ![Graphic of an organization chart demonstrating hierarchical relationships.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt60fe7aca4df1299b/6a7adaae437e0f26b9dcfc33/Hierarchical_graphic-m1OeuH.png) This shows, in a graphical form, that a recursive CTE can retrieve from the database by recursively expanding the result set starting at the top with Maggie. The result set will still be in the form of a table. A recursive CTE contains two `SELECT` statements joined by a `UNION ALL` clause. The first `SELECT` statement is the base case which is executed once. The second `SELECT` statement is the recursive case where the CTE references itself, expanding repeatedly until the termination condition is met. The termination condition does not evaluate to TRUE or FALSE. The termination condition is not specific syntax; rather, the RCTE terminates when the recursive case does not return any new rows. The recursive CTE will use the data types from the columns in the base case if the columns in the recursive case have different data types. ## Limitations There are several limitations on which operations are allowed in the SELECT statements for the recursive case of recursive CTEs. Please note, this list is subject to change upon future releases. These operations are not allowed in the SELECT in the recursive case: * The base case in a recursive CTE must be a sharded table. * `SELECT DISTINCT` * `EXCEPT/MINUS` * `LEFT [OUTER] JOIN` * `RIGHT [OUTER] JOIN` * `INTERSECT` * `UNION` * `GROUP BY` * `ORDER BY` * `LIMIT` * [Aggregate functions](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions.md) like `COUNT`, `MAX`, etc. * [Window functions](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions.md) like `RANK`, `DENSE RANK`, etc. ## Troubleshooting timed-out recursive CTEs If the recursive CTE takes an inordinate amount of time to execute or times out, there can be an issue with the data’s structure or with the creation of the CTE itself. These issues prevent the recursive case of the CTE from terminating. Your data structure may not be truly hierarchical. Using the organization chart example, if Maggie also reported to John, then the data structure contains a cycle. If your data contains cycles, either update the data to remove cycles, or use some other method to make the expansion terminate, like explicitly checking expansion depth. Your recursive CTE query can be incorrectly coded and create an infinite loop. An infinite loop can occur when the SELECT clause in the recursive case outputs a value from the base case instead of the current iteration. This causes incorrect values to be used in the subsequent iteration’s JOIN operation. Both the query itself and the data must allow the recursive case to terminate. ## Known issue with recursive CTEs If a recursive CTE results in the error "'Table'.'doesn't exist'", the `materialize_ctes` setting may be disabled. This usually happens when upgrading from a pre-7.8 version where the `materialize_ctes` setting was not enabled by default. Refer to [WITH (Common Table Expressions)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md) for information on how to enable the functionality. ## Examples Non-recursive CTEs: ```sql WITH myCTE AS (select col from myTable) select col from myCTE; ``` ```sql WITH myCTE (colAlias) AS (select col from myTable) select colAlias from myCTE; ``` ```sql WITH orderCTE AS (select o_orderkey from orders), lineitemCTE AS (select l_orderkey from lineitem) select count(*) from orderCTE join lineitemCTE on o_orderkey = l_orderkey; ``` ```sql WITH foo AS (WITH bar AS (SELECT * FROM t) SELECT * FROM bar) SELECT * FROM foo; ``` ```sql WITH EmpSal(averageSal) AS (SELECT AVG(Salary) FROM Employee) SELECT EmpID, Name, Salary FROM Employee, EmpSal WHERE Employee.Salary > EmpSal.averageSal; ``` ```sql WITH ObjectCTE (Name, Id, Date) AS (SELECT objname, objectid, invoicedate FROM Inventory) INSERT INTO Itemlist(Name, objectid, createdDate) SELECT Name, Id, Date FROM ObjectCTE; SELECT * FROM Itemlist; ``` Recursive CTEs: Using the organization chart described above, this example illustrates how recursive CTEs simplify the retrieval of hierarchical data. ```sql CREATE TABLE cte_emp(id INT, mgr_id INT, name VARCHAR(30)); INSERT INTO cte_emp VALUES(1, NULL, "Maggie"), (2, 1, "John"), (3, 1, "Felix"), (4, 2, "Joan"), (5, 2, "Richard"), (6, 3, "Andy"), (7, 3, "Jill"); ``` ```sql WITH RECURSIVE org_chart (emp_id, mgr_id, name, level) AS ( SELECT cte_emp.id, cte_emp.mgr_id, cte_emp.name, 0 AS level FROM cte_emp WHERE cte_emp.mgr_id is NULL UNION ALL SELECT cte_emp.id, cte_emp.mgr_id, cte_emp.name, level+1 FROM cte_emp INNER JOIN org_chart ON cte_emp.mgr_id = org_chart.emp_id ) SELECT * FROM org_chart; ``` ```output +------+--------+---------+-------+ | id | mgr_id | name | level | +------+--------+---------+-------+ | 1 | NULL | Maggie | 0 | | 2 | 1 | John | 1 | | 5 | 2 | Richard | 2 | | 4 | 2 | Joan | 2 | | 7 | 3 | Jill | 2 | | 3 | 1 | Felix | 1 | | 6 | 3 | Andy | 2 | +------+--------+---------+-------+ ``` Another example is a graph of connected nodes. A recursive CTE can retrieve the nodes and their relationships to one other. The `id` column is the node id and `pr` is the parent node to the node in the `id` column. ```sql CREATE TABLE cte_g (id INT, pr INT); INSERT INTO cte_g VALUES (0, null), (1, 0), (2, 0),(3, 1), (4, 1), (5, 2), (6, 2); ``` ```sql WITH RECURSIVE depths AS ( SELECT 0 depth, * FROM cte_g WHERE id = 0 UNION ALL SELECT depth + 1, cte_g.id, cte_g.pr FROM cte_g JOIN depths ON cte_g.pr = depths.id ) SELECT depth, id FROM depths; ``` ```output +-------+------+ | depth | id | +-------+------+ | 0 | 0 | | 1 | 1 | | 2 | 3 | | 2 | 4 | | 1 | 2 | | 2 | 6 | | 2 | 5 | +-------+------+ ``` The following examples show the recursive CTE will use the data types from the columns in the base case if the columns in the recursive case have different data types. ```sql CREATE TABLE cte_g (id INT, pr INT, SHARD KEY(id)); INSERT INTO cte_g VALUES (0, null), (1, 0), (2, 0),(3, 1), (4, 1), (5, 2), (6, 2); ``` ```sql WITH RECURSIVE depths AS ( SELECT 0 depth, id, pr FROM cte_g WHERE pr IS NULL UNION ALL SELECT CAST((depth+1) AS DECIMAL(10, 2)), cte_g.id, cte_g.pr FROM cte_g, depths WHERE cte_g.pr = depths.id ) SELECT depth, id FROM depths; ``` ```output +-------+------+ | depth | id | +-------+------+ | 0 | 0 | | 2 | 5 | | 1 | 2 | | 2 | 4 | | 1 | 1 | | 2 | 3 | | 2 | 6 | +-------+------+ ``` ```sql WITH RECURSIVE routes (id, path) AS ( SELECT id, cast ( 1 as char(30) ) FROM cte_g WHERE pr IS NULL UNION ALL SELECT cte_g.id, CONCAT ( routes.path, '-->', cte_g.id ) FROM cte_g JOIN routes ON routes.id = cte_g.pr ) SELECT * FROM routes; ``` ```output +------+------------------------+ | id | cast ( 1 as char(30) ) | +------+------------------------+ | 0 | 1 | | 5 | 1-->2-->5 | | 2 | 1-->2 | | 4 | 1-->1-->4 | | 1 | 1-->1 | | 3 | 1-->1-->3 | | 6 | 1-->2-->6 | +------+------------------------+ ``` ## Materializing Common Table Expressions ## Overview By using the variable `materialize_ctes`, common table expressions (CTEs) are materialized when queries with CTEs are run. The materialized results can then be referenced multiple times by the parent query. The CTE queries are cached, which results in optimized performance. This is particularly useful for running queries that contain an expensive CTE that produces a relatively small number of rows and is used two or more times. The materializing CTEs functionality is enabled by default. Enable or disable this functionality by setting the `materialize_ctes` engine variable, or setting it per session via the `MATERIALIZE_CTES` session variable: ```sql SET MATERIALIZE_CTES = { 'ALL' | 'AUTO' | 'OFF' }; ``` The following settings are available: * `ALL`: Each CTE that appears more than once in the query is materialized into an internal result table. * `AUTO`: This is the default value. A suite of optimizations are enabled. Currently, only the [Predicate Pushdown](https://docs.singlestore.com/#UUID-c15a59d5-111d-dab1-46a0-fd1183c14080.md) optimization is available. * `OFF`: CTEs are inlined, which is similar to [VIEW definitions](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md). If users can't or don't want to use engine variables, the `materialize_ctes` engine variable can be set on a per query basis by adding an `OPTION` expression at the end of the query: ```sql SELECT * FROM foo OPTION(materialize_ctes="OFF"); SELECT * FROM foo OPTION(materialize_ctes="AUTO"); SELECT * FROM foo OPTION(materialize_ctes="ALL"); ``` Materialization can also be turned off per CTE: ```sql WITH foo as (select with(materialize=off) * from titanic), bar as (select * from titanic) select * from foo, bar; ``` ## Remarks * Materialized CTEs appear in the `EXPLAIN` as `m_foo_i`, where `foo` is the name of the CTE in the original query, and `i` is an integer value generated during name normalization. * The results table that is created from a materialized CTE is analogous to an intermediate results table that is created as part of a distributed join query. * Plans for queries with CTEs are cacheable regardless of the setting of the `materialize_ctes` variable. * Some `SELECT` statements may not be materializable. For example, selects with limits, selects with aggregates and no group by, and selects without `FROM` clauses cannot be materialized. All CTEs whose selects are non-materializable will get inlined with the parent query. ## Predicate Pushdown To avoid materializing unused rows, a rudimentary form of predicate pushdown in CTEs is supported. This optimization is enabled only when the `materialize_ctes` session variable is set to `AUTO`. An example of how a query is internally rewritten by SingleStore Helios is shown below. ```sql WITH foo AS (SELECT * FROM t) SELECT * FROM foo, foo as bar WHERE foo.a = 1 AND bar.b = 2; ``` is transformed to: ```sql WITH foo AS (SELECT * FROM t WHERE t.a = 1 OR t.b = 2) SELECT * FROM foo, foo as bar WHERE foo.a = 1 AND bar.b = 2; ``` Unlike normal predicate pushdown, the original top-level predicates remain. *** Modified at: August 4, 2026 Source: [/cloud/query-data/advanced-query-topics/with-common-table-expressions/](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions/) (An index of the documentation is available at /llms.txt) --- # Basic Query Examples This section explains how to run queries on SingleStore via a series of examples. For the sake of simplicity, the sample data used in the examples below is very small (10 rows or less per table) compared to typical workloads. The queries in the examples below consist of several database operations like index scans, full-table scans, joins, and aggregations. First, connect to SingleStore (see the [Connect to SingleStore](https://docs.singlestore.com/cloud/connect-to-singlestore.md) section for details on how to connect). Then run the following queries to set up the data. You can also try this [notebook](https://www.singlestore.com/spaces/basic-query-examples/) in the portal using SingleStore Spaces ```sql -- Create the database CREATE DATABASE memsql_example; USE memsql_example; -- Create 3 tables: departments, employees, and salaries CREATE TABLE departments ( id int, name varchar(255), PRIMARY KEY (id) ); CREATE TABLE employees ( id int, deptId int, managerId int, name varchar(255), hireDate date, state char(2), PRIMARY KEY (id) ); CREATE TABLE salaries ( employeeId int, salary int, PRIMARY KEY (employeeId) ); -- Populate each table with data INSERT INTO departments (id, name) VALUES (1, 'Marketing'), (2, 'Finance'), (3, 'Sales'), (4, 'Customer Service'); INSERT INTO employees (id, deptId, managerId, name, hireDate, state) VALUES (1, 2, NULL, "Karly Steele", "2011-08-25", "NY"), (2, 1, 1, "Rhona Nichols", "2008-09-11", "TX"), (3, 4, 2, "Hedda Kent", "2005-10-27", "TX"), (4, 2, 1, "Orli Strong", "2001-07-01", "NY"), (5, 1, 1, "Leonard Haynes", "2011-05-30", "MS"), (6, 1, 5, "Colette Payne", "2002-10-22", "MS"), (7, 3, 4, "Cooper Hatfield", "2010-08-19", "NY"), (8, 2, 4, "Timothy Battle", "2001-01-21", "NY"), (9, 3, 1, "Doris Munoz", "2008-10-22", "NY"), (10, 4, 2, "Alea Wiggins", "2007-08-21", "TX"); INSERT INTO salaries (employeeId, salary) VALUES (1, 885219), (2, 451519), (3, 288905), (4, 904312), (5, 919124), (6, 101538), (7, 355077), (8, 900436), (9, 41557), (10, 556263); ``` Now let’s run a simple query to ask how many rows are in the employees table. ```sql SELECT COUNT(*) from employees; ``` ```output +----------+ | COUNT(*) | +----------+ | 10 | +----------+ 1 row in set (0.61 sec) ``` Note that the first time you run the query, it will take a significant amount of time (about a second) - this is because it is compiling the query. The plan is then saved so that the next time you run the query, it will be much faster. See the [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md) section for more details. Here is another simple query that lists the ID and name of each employee: ```sql SELECT id, name FROM employees ORDER BY id; ``` ```output +----+-----------------+ | id | name | +----+-----------------+ | 1 | Karly Steele | | 2 | Rhona Nichols | | 3 | Hedda Kent | | 4 | Orli Strong | | 5 | Leonard Haynes | | 6 | Colette Payne | | 7 | Cooper Hatfield | | 8 | Timothy Battle | | 9 | Doris Munoz | | 10 | Alea Wiggins | +----+-----------------+ 10 rows in set (0.73 sec) ``` > **📝 Note**: Queries without an `ORDER BY` clause will not have a guaranteed result set order, even if there is a primary key on the table. SQL’s `WHERE` clause can be used to filter results. Here is a query that lists only employees that work in Texas: ```sql SELECT id, name FROM employees WHERE state = 'TX' ORDER BY id; ``` ```output +----+---------------+ | id | name | +----+---------------+ | 2 | Rhona Nichols | | 3 | Hedda Kent | | 10 | Alea Wiggins | +----+---------------+ 3 rows in set (0.74 sec) ``` Note that if you change the parameters in the query -- e.g. changing TX to NY -- SingleStore can reuse the same query plan, so the query does not need to be recompiled: ```sql SELECT id, name FROM employees WHERE state = 'NY' ORDER BY id; ``` ```output +----+-----------------+ | id | name | +----+-----------------+ | 1 | Karly Steele | | 4 | Orli Strong | | 7 | Cooper Hatfield | | 8 | Timothy Battle | | 9 | Doris Munoz | +----+-----------------+ 5 rows in set (0.00 sec) ``` Below are several more queries demonstrating common SQL operations. Employees hired before 2002: ```sql SELECT id, name, hireDate FROM employees WHERE hireDate < '2002-01-01' ORDER BY id; ``` ```output +----+----------------+------------+ | id | name | hireDate | +----+----------------+------------+ | 4 | Orli Strong | 2001-07-01 | | 8 | Timothy Battle | 2001-01-21 | +----+----------------+------------+ 2 rows in set (0.77 sec) ``` List employees and their departments: ```sql SELECT e.name, d.name department FROM employees e, departments d WHERE e.deptId = d.id ORDER BY name; ``` ```output +-----------------+------------------+ | name | department | +-----------------+------------------+ | Alea Wiggins | Customer Service | | Colette Payne | Marketing | | Cooper Hatfield | Sales | | Doris Munoz | Sales | | Hedda Kent | Customer Service | | Karly Steele | Finance | | Leonard Haynes | Marketing | | Orli Strong | Finance | | Rhona Nichols | Marketing | | Timothy Battle | Finance | +-----------------+------------------+ 10 rows in set (0.93 sec) ``` Number of employees in each state: ```sql SELECT state, COUNT(*) from employees group by state ORDER BY state; ``` ```output +-------+----------+ | state | COUNT(*) | +-------+----------+ | MS | 2 | | NY | 5 | | TX | 3 | +-------+----------+ 3 rows in set (0.82 sec) ``` Highest salary amongst all employees: ```sql SELECT MAX(salary) FROM salaries; ``` ```output +-------------+ | MAX(salary) | +-------------+ | 919124 | +-------------+ 1 row in set (0.58 sec) ``` Employee with the highest salary: ```sql SELECT e.name, s.salary FROM employees e, salaries s WHERE e.id = s.employeeId and s.salary = (SELECT MAX(salary) FROM salaries); ``` ```output +----------------+--------+ | name | salary | +----------------+--------+ | Leonard Haynes | 919124 | +----------------+--------+ 1 row in set (0.98 sec) ``` Average salary of employees in each state: ```sql SELECT e.state, AVG(salary) FROM employees e JOIN salaries s on e.id = s.employeeId GROUP BY e.state ORDER BY e.state; ``` ```output +-------+-------------+ | state | AVG(salary) | +-------+-------------+ | MS | 510331.0000 | | NY | 617320.2000 | | TX | 432229.0000 | +-------+-------------+ 3 rows in set (1.72 sec) ``` List of managers: ```sql SELECT name FROM employees WHERE id IN (SELECT managerId FROM employees) ORDER BY name; ``` ```output +----------------+ | name | +----------------+ | Karly Steele | | Leonard Haynes | | Orli Strong | | Rhona Nichols | +----------------+ 4 rows in set (1.40 sec) ``` List of non-managers: ```sql SELECT name FROM employees WHERE id NOT IN (SELECT managerId FROM employees) ORDER BY name; ``` ```output +-----------------+ | name | +-----------------+ | Alea Wiggins | | Colette Payne | | Cooper Hatfield | | Doris Munoz | | Hedda Kent | | Timothy Battle | +-----------------+ 6 rows in set (1.43 sec) ``` Number of employees reporting to each manager: ```sql SELECT m.name, COUNT(*) count FROM employees m JOIN employees e ON m.id = e.managerId GROUP BY m.id ORDER BY count DESC; ``` ```output +----------------+-------+ | name | count | +----------------+-------+ | Karly Steele | 4 | | Orli Strong | 2 | | Rhona Nichols | 2 | | Leonard Haynes | 1 | +----------------+-------+ 4 rows in set (0.95 sec) ``` Number of employees reporting to each employee: ```sql SELECT m.name, COUNT(e.id) count FROM employees m LEFT JOIN employees e ON m.id = e.managerId GROUP BY m.id ORDER BY count desc; ``` ```output +-----------------+-------+ | name | count | +-----------------+-------+ | Karly Steele | 4 | | Rhona Nichols | 2 | | Orli Strong | 2 | | Leonard Haynes | 1 | | Doris Munoz | 0 | | Alea Wiggins | 0 | | Cooper Hatfield | 0 | | Hedda Kent | 0 | | Timothy Battle | 0 | | Colette Payne | 0 | +-----------------+-------+ 10 rows in set (0.84 sec) ``` Manager of each employee: ```sql SELECT e.name employee_name, m.name manager_name FROM employees e LEFT JOIN employees m ON e.managerId = m.id ORDER BY manager_name; ``` ```output +-----------------+----------------+ | employee_name | manager_name | +-----------------+----------------+ | Karly Steele | NULL | | Doris Munoz | Karly Steele | | Rhona Nichols | Karly Steele | | Orli Strong | Karly Steele | | Leonard Haynes | Karly Steele | | Colette Payne | Leonard Haynes | | Timothy Battle | Orli Strong | | Cooper Hatfield | Orli Strong | | Alea Wiggins | Rhona Nichols | | Hedda Kent | Rhona Nichols | +-----------------+----------------+ 10 rows in set (1.04 sec) ``` Total salary of the employees reporting to each manager: ```sql SELECT m.name, sum(salary) FROM employees m JOIN employees e ON m.id = e.managerId JOIN salaries s ON s.employeeId = e.id GROUP BY m.id ORDER BY SUM(salary) DESC; ``` ```output +----------------+-------------+ | name | SUM(salary) | +----------------+-------------+ | Karly Steele | 2316512 | | Orli Strong | 1255513 | | Rhona Nichols | 845168 | | Leonard Haynes | 101538 | +----------------+-------------+ 4 rows in set (1.86 sec) ``` Employees in the finance department earning more than their manager: ```sql SELECT e.name employee_name, se.salary employee_salary, m.name manager_name, sm.salary manager_salary FROM employees e JOIN salaries se ON e.id = se.employeeId JOIN employees m ON m.id = e.managerId JOIN salaries sm ON sm.employeeId = m.id JOIN departments d ON d.id = e.deptId WHERE d.name = 'Finance' AND sm.salary < se.salary ORDER BY employee_salary, manager_salary; ``` ```output +---------------+-----------------+--------------+----------------+ | employee_name | employee_salary | manager_name | manager_salary | +---------------+-----------------+--------------+----------------+ | Orli Strong | 904312 | Karly Steele | 885219 | +---------------+-----------------+--------------+----------------+ 1 row in set (1.46 sec) ``` For documentation on all the types of queries supported by SingleStore, see [SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference.md). *** Modified at: July 10, 2025 Source: [/cloud/query-data/basic-query-examples/](https://docs.singlestore.com/cloud/query-data/basic-query-examples/) (An index of the documentation is available at /llms.txt) --- # Connect with Analytics and BI Tools SingleStore clusters can be integrated with the following third-party analytics and BI tools: * [Alteryx Designer](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer.md) * [Cognos Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics.md) * [Denodo](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo.md) * [Dremio](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio.md) * [GoldenGate](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate.md) * [GoodData](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata.md) * [Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker.md) * [MicroStrategy](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy.md) * [Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi.md) * [Sisense](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense.md) * [Superset](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset.md) * [Tableau](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau.md) * [ThoughtSpot](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot.md) * [Tibco Spotfire](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire.md) * [Zoho Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics.md) * [Other Tools](https://docs.singlestore.com/cloud/load-data/integrate-with-singlestore-helios.md) ## In this section * [Connect with Alteryx Designer](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer.md) * [Connect with IBM Cognos Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics.md) * [Connect with Denodo](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo.md) * [Connect with Dremio](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio.md) * [Connect with Oracle GoldenGate](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate.md) * [Connect with GoodData](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata.md) * [Connect with Looker](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker.md) * [Connect with MicroStrategy](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy.md) * [Connect with Power BI](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi.md) * [Connect with Sisense](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense.md) * [Connect with Superset](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset.md) * [Connect with Tableau](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau.md) * [Connect with ThoughtSpot](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot.md) * [Connect with Tibco Spotfire](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire.md) * [Connect with Zoho Analytics](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics.md) *** Modified at: April 7, 2026 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/) (An index of the documentation is available at /llms.txt) --- # Connect with Alteryx Designer You can connect to your SingleStore database from Alteryx Designer and perform analytics operations. Alteryx supports both read and write operations for the SingleStore data source. For standard workflow processing, Alteryx supports the following tools for the SingleStore data source: * [Input Data](https://help.alteryx.com/current/en/designer/tools/in-out-tools/input-data-tool.html) * [Output Data](https://help.alteryx.com/current/en/designer/tools/in-out-tools/output-data-tool.html) Alteryx supports the following authentication methods for the SingleStore data source: * No authentication * Username/password * Generic OAuth ## Prerequisites * Download and install [the SingleStore ODBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) version 1.1.5+ * [Download and install](https://help.alteryx.com/current/en/license-and-activate/install/download-and-install-a-product.html) Alteryx Designer ## Connect to SingleStore using a DSN and DCM To connect to SingleStore from Alteryx using a DSN, you must [create a Data Source Name (DSN)](https://help.alteryx.com/current/en/designer/data-sources/odbc-and-oledb-database-connections.html#idp399997) for your ODBC connection. Once the DSN is created, perform the following tasks: 1. On the **Alteryx Designer** window, drag either the Input Data or Output Data tool from the tool palette to the workflow canvas area. 2. Select the **Use Data Connection Manager (DCM)** check box. 3. In the **Data Connections** window, select **Data Sources > Singlestore ODBC**. If Alteryx is unable to detect the SingleStore ODBC driver, a pop-up indicates that the driver must be installed. 4. On the **Connection Manager** window, select **New** to add a new data source. 5. Select **ODBC** from the list. This option is selected by default. 6. Enter a Data Source Name (DSN) for the new data source, for example, **Singlestore Server DSN**. 7. Select the **ODBC DSN** field to see a list of available DSNs, and select a DSN to use for the data source. 8. Select **Save**. 9. To link a credential, select **Connect Credential** and **select an Authentication Method** from the list. 10. Select **Connect** to use the Data Source and Credential in an Input Data or Output Data tool. ## Configure DSN-less Connection Using DCM To connect to SingleStore from Alteryx using a DSN-less connection, 1. On the **Alteryx Designer** window, drag either the Input Data or Output Data tool from the tool palette to the workflow canvas area. 2. Select the **Use Data Connection Manager (DCM)** check box. 3. In the **Data Connections** window, select **Data Sources > SingleStore Quick Connect**. If Alteryx is unable to detect the SingleStore ODBC driver, a pop-up indicates that the driver must be installed. 4. On the **Connection Manager** window, select **New** to add a new data source. 5. Select **Singlestore ODBC DSN-less** from the list. This option is selected by default. 6. Enter the following connection configuration information: * Data Source Name for the connection, for example, **Singlestore DSN-less** * Hostname or IP address of the SingleStore deployment Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * Name of the SingleStore database to connect with * Port of the SingleStore database 7. (Optional) Select the **Driver** field to change the selected driver. 8. Select **Save**. 9. To link a credential, select **Connect Credential** and select an **Authentication Method** from the list. 10. Select **Connect** to use the Data Source and Credential in an Input Data or Output Data tool. ## Limitations Alteryx does not support the following for the SingleStore data source: * Spatial data types * Table names containing `'` (single quote) * Writing to `DOUBLE` type columns * `BIT`, `BINARY`, `VARBINARY`, `TINYBLOB`, `BLOB`, `MEDIUMBLOB`, and `LONGBLOB` SingleStore data types *** Modified at: September 26, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-alteryx-designer/) (An index of the documentation is available at /llms.txt) --- # Connect with Denodo Denodo is a data virtualization platform that allows you to integrate your SingleStore data in real-time into a unified view and perform analytics and BI operations. To connect with Denodo, you need to configure the SingleStore JDBC driver as a database adapter in Denodo. Subsequently, add your SingleStore Helios database as a JDBC data source using this adapter. ## Prerequisites [Download](https://github.com/memsql/S2-JDBC-Connector/releases/latest) the `.jar` file for the latest version of [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). ## Connect Denodo to SingleStore Helios **Configure the SingleStore JDBC driver as a database adapter in Denodo:** 1. On the Denodo dashboard, select **File > Extensions management**. 2. On the **Extension management** dialog, on the **Libraries** tab, from the **Type** list, select **JDBC Drivers**. 3. On the **Import resources** dialog, from the **Resource type** list, select **jdbc\_other**. 4. In the **Custom version** field, enter a name for the driver. For convenience, you can use the **singlestore-jdbc-client-\** format. For example, **singlestore-jdbc-client-1.1.5**. This name is visible in the dialog where you can configure JDBC data sources. 5. Select **+ Add**. Add the **.jar** file downloaded earlier to the **Load File** dialog. 6. Select **Save**. The SingleStore JDBC driver is now configured as an adapter in Denodo. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt71f76c9e4659b742/6a7adb3c448e4e6caa5b5457/denodo1-TEcgQw.png) **Add your SingleStore Helios database as a JDBC data source**: 1. Right-click on a database in the **Server Explorer**, and select **New > Data Source > JDBC**. 2. On the **Configuration** tab on the dialog that opens, enter or select the configuration options for your SingleStore Helios database and JDBC adapter. * **Name**: Name for the new data source. For example, **SingleStore**. * **Database adapter**: Select the name you provided for the driver. For example, **singlestore-jdbc-client-1.1.5**. * **Database class**: `com.singlestore.jdbc.Driver`. * **Database URI**: Specify the connection string for your database in the `jdbc:singlestore://:/` format. For example, **jdbc:singlestore://svchost:3306/dbTest**. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Authentication**: Specify the authentication method for your connection. 3. Select **Test connection** to test connectivity to your database. Select **Save**. You can now use Denodo to explore your SingleStore Helios databases. ## References * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) * [Importing JDBC Sources](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/creating_data_sources_and_base_views/jdbc_sources/jdbc_sources#importing-jdbc-sources) * [Creating Data Sources and Base Views](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/creating_data_sources_and_base_views/creating_data_sources_and_base_views) * [Importing a JDBC Driver for a Database that is not Listed](https://community.denodo.com/docs/html/browse/8.0/en/vdp/administration/importing_extensions/importing_extensions#importing-a-jdbc-driver-for-a-database-that-is-not-listed) *** Modified at: September 26, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-denodo/) (An index of the documentation is available at /llms.txt) --- # Connect with Dremio Dremio is an open source project that enables business analysts and data scientists to explore and analyze any data at any time, regardless of its location, size, or structure. Dremio combines a scale-out architecture with columnar execution and acceleration to achieve interactive performance on any data volume. You can connect Dremio to SingleStore using the [SingleStore connector for Dremio](https://github.com/memsql/dremio-singlestore). The following certification matrix shows the recommended version of Dremio: | Certification Matrix | Versions | | -------------------- | -------- | | Dremio | 17.0.0+ | ## Prerequisites Before connecting Dremio to SingleStore Helios, perform the following tasks: 1. [Download and install](https://www.dremio.com/deploy/) Dremio. 2. Download and install the latest version of [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md). 3. Ensure the SingleStore cluster is running on the server. 4. [Download](https://github.com/memsql/dremio-singlestore/releases/latest) the SingleStore connector for Dremio. ## Configure JARS and Run Dremio Log in to a linux machine, and perform the following tasks: 1. Download the `.jar` plugin from [GitHub](https://github.com/memsql/dremio-singlestore/releases/latest), and place this `.jar` plugin in the **$DREMIO\_HOME/jars** directory in Dremio. 2. Download [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) `.jar` file from [GitHub](https://github.com/memsql/S2-JDBC-Connector/releases/latest), and place this `.jar` file in the **$DREMIO\_HOME/jars/3rdparty** directory. 3. Start (or restart) the `dremio` service: ```shell sudo /opt/dremio/bin/dremio start ``` 4. Open a browser, and connect to Dremio running on the linux server: **http\://\:9047/** ## Connect Dremio to SingleStore 1. Open a the following URL in a browser to connect to Dremio running on the linux server: **http\://\:9047/** 2. On the Welcome to Dremio screen, enter the **Username** and **Password** to log into Dremio. 3. On the left navigation pane, select **Sources > Add Source**. 4. On the **Add Data Source** dialog, from the list, select **SingleStore**. 5. In the **New SingleStore Source** window, enter the following information: * **Name**: Name of the connection. * **Host**: IP address or hostname of the SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Database**: Name of the database to connect with. * **Port**: Default is `3306`. * **Username** and **Password**: Username and password of the database user. 6. Select **Save**. Your new SingleStore Helios connection is now available and ready to use. ## In this section * [Data Reflection and Query Acceleration](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio/data-reflection-and-query-acceleration.md) * [Best practices guide](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio/best-practices-guide.md) *** Modified at: September 26, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-dremio/) (An index of the documentation is available at /llms.txt) --- # Connect with GoodData GoodData is a comprehensive analytics platform that enables you to explore, visualize, and derive insights from your SingleStore data. It offers a cloud-based, scalable solution for building and managing dashboards, reports, and data-driven applications. To integrate your SingleStore Helios database into GoodData, create a new [data source](https://www.gooddata.com/docs/cloud/connect-data/concepts/data-source/) using the GoodData UI or the [Entity API](https://www.gooddata.com/docs/cloud/api-and-sdk/api/#API-EntityAPI). ## Prerequisites * An active SingleStore Helios deployment. * Add the [GoodData Cloud IP address](https://www.gooddata.com/docs/cloud/security-and-compliance/cloud-security-settings/) to the IP Allowlist (firewall) of your SingleStore Helios deployment (go to **[Cloud Portal](https://portal.singlestore.com/) > your\_cluster > Firewall > IP Allowlist > Edit**). * SingleStore recommends using a dedicated SingleStore database user with the [required privileges](https://docs.singlestore.com/#section-idm23449831585298.md) for integration with GoodData. Run the following command to create a user: ```sql CREATE USER IDENTIFIED BY ''; ``` Refer to [CREATE USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) for more information. ## Create a SingleStore Data Source in GoodData The SingleStore data source can be configured in GoodData using any of the following methods: * Using the GoodData UI * Using the GoodData Entity API ## Using the GoodData UI To connect to SingleStore from GoodData: 1. On the GoodData homepage, select **Data sources > Connect data**. 2. On the **Connect data** dialog, select **SingleStore** from the list of supported databases. 3. Enter a name for the data source, for example, **SingleStore**. 4. Enter or select the following information in the **Enter your SingleStore credentials** section: * **Connection URL**: * Enter the hostname or IP address of your SingleStore Helios deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * Enter the port of your SingleStore Helios deployment. Default is `3306`. * **SSL mode**: Select the SSL mode from the list. * **Username**: Enter the username of the SingleStore database user. * **Password**: Enter the password for the SingleStore database user. * **Database name**: Enter the name of the SingleStore database to connect with. 5. Select **Connect**. 6. From the **Set cache preference** list, select the cache preference. 7. Select **Save**. The SingleStore data source is configured and ready to use. ## Using the GoodData Entity API To configure a data source via the GoodData Entity API, make an API call to the `../api/v1/entities/dataSources` endpoint. Here's an example: ```Shell curl $HOST_URL/api/v1/entities/dataSources \ -H "Content-Type: application/vnd.gooddata.api+json" \ -H "Accept: application/vnd.gooddata.api+json" \ -H "Authorization: Bearer $API_TOKEN" \ -X POST \ -d '{ "data": { "type": "dataSource", "id": "", "attributes": { "name": "", "url": "jdbc:mysql://:/", "schema": "", "type": "SINGLESTORE", "username": "", "password": "" }}}' | jq . ``` On a successful API call, the server returns a response similar to the following: ``` { "data": { "type": "dataSource", "id": "", "attributes": { "name": "", "url": "jdbc:mysql://:/", "schema": "", "type": "SINGLESTORE", "username": "" } }, "links": { "self": "$HOST_URL/api/v1/entities/dataSources/" } } ``` For more information, refer to [GoodData API](https://www.gooddata.com/docs/cloud/api-and-sdk/api/api_reference_all/) documentation. ## Remarks * The SingleStore data source only supports Basic Authentication. * The following table lists the minimum permissions required for the SingleStore database user to perform the respective task: | Task | Minimum Permission Required | | ---------------- | --------------------------- | | Data Rendering | `SELECT` | | Caching database | `CREATE`,`DROP` | Refer to [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) for more information. * For a list of SingleStore features that are not supported by GoodData, refer to [Unsupported Features and Limitations](https://www.gooddata.com/docs/cloud/connect-data/create-data-sources/singlestore/#CreateaSingleStoreDataSource-UnsupportedFeaturesandLimitations). *** Modified at: May 11, 2026 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-gooddata/) (An index of the documentation is available at /llms.txt) --- # Connect with IBM Cognos Analytics You can load your SingleStore database in IBM Cognos Analytics and perform analytics and BI operations. You need to add a [data server connection](https://www.ibm.com/docs/en/cognos-analytics/11.1.0?topic=sources-data-servers) to SingleStore in Cognos Analytics and load the metadata for one (or more) schema(s). Once you have added a data server, you can create data modules, reports, dashboards, create a new table using SQL etc. See [IBM Cognos Analytics documentation](https://www.ibm.com/docs/en/cognos-analytics/11.1.0). ## Prerequisites * Access to IBM Cognos Analytics. * Hands-on knowledge of Cognos Analytics application (see [IBM Cognos Analytics documentation](https://www.ibm.com/docs/en/cognos-analytics/11.2.0)). ## Add a Data Server Connection to SingleStore in Cognos Analytics 1. Sign in to [Cognos Analytics](https://www.ibm.com/products/cognos-analytics), and select **Manage** from the (collapsed) menu. 2. Select **Data server connections**, and then select the **Add data server** button. 3. On the **Create data server connection** dialog, change the connection name from the default to something else, for example, **SingleStore-test**. 4. From the **Connection type** list, select **SingleStoreDB**. Select **Next**. ![Create data server.connection for IBM Cognos graphic.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7f4e5b4eacca6c7c/6a7adba1c33f4ff38dd4ef16/cognos-1-O1f3o4.png) 5. Under **Connection**, in the **JDBC URL** box, enter the connection parameters for your cluster in the `jdbc:singlestore://:/database` format. * On the [Cloud Portal](https://portal.singlestore.com/), select **Clusters > *\* > Connect > Connect Directly > BI Tools**. On the **BI Tools** tab, select **Others** from the list. Use these connection parameters. On the [Cloud Portal](https://portal.singlestore.com/), select **Clusters > *\* > Firewall**. Under **IP Allowlist**, select **Allow from any IP address**. ![Create data server connection for IBM Cognos graphic.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6e58b25a0a691cc0/6a7adadf889c37d789057c7e/cognos-2-MVNIWy.png) 6. Under **Authentication**, from the **Method** list, select **Use signon**. 7. Select the **Add Signon** button. 8. On the **User ID and password** dialog, enter the login credentials for your cluster. Select **Done** to save the credentials. ![Provide user ID and password for IBM Cognos graphic.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc00573f5addf8583/6a7adba0d402d99ed3eadd29/cognos-4-cloud-yZVXPS.png) To test connectivity to your cluster, select the **Test connection** button. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8c6e157289f57722/6a7adac95947112edc4c2671/cognos-5-ie5YML.png) 9. Select **Next**, and then select **Create** to save the connection configuration. 10. On the **Data server connections** screen, select the recently created connection. Select the vertical three-dot menu next to the connection, and select **Assets**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9ff7fdba4107e96f/6a7adbb2de23155c93fcc50c/cognos-6-CPr70Q.png) 11. On the **Assets** screen, select the vertical three-dot menu next to the asset (database) you want to import, and then select **Load metadata** to load metadata for your schema. Use **Load options** to modify **Load metadata** settings. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltfb7d3f389d85fc72/6a7adb8f96b5a6b28086fde6/cognos-7-1WDBGd.png) ## Create a New Data Module To connect to a SingleStore database, 1. On the Cognos Analytics home page, select **New > Data module** (from the collapsed menu). 2. On the left navigation menu, select data servers and schemas. 3. Select the required server and database from the list. Select **OK**. For more information, see [Creating a data module](https://www.ibm.com/docs/en/cognos-analytics/11.1.0?topic=sources-adding-tables-data-module). Once the database is uploaded, you can view the tables in the selected database on the **Grid** tab. You can join the tables in the database on the **Relationships** tab. To create custom tables, use the **Custom tables** tab. ## Troubleshooting **Issue: Cognos Analytics does not support zero values in date.** **Solution:**SingleStore supports zero values in dates (e.g. 0000-00-00), but Cognos Analytics does not. To solve this issue, you can, 1. Assign `NULL` values instead of zeroes. 2. Use extreme values for these dates, for example 1970-01-01 or 9999-12-31. If you do not have access to the data, use the **SQL View** option and change zero values to `NULL` values. **Issue: Cognos Analytics may not support cluster specific constructs when using custom SQL syntax.** **Solution:** Use the Pass-Through SQL type to run queries in cluster. Pass-through SQL allows you to run queries without any restrictions that the data source imposes on the subqueries. See [Supported SQL types](https://www.ibm.com/docs/en/cognos-analytics/11.1.0?topic=analytics-supported-sql-types) for more information. > **⚠️ Warning**: Pass-through SQL type allows a user to perform any SQL operation on the database. Hence, the database administrator must ensure that a database user has necessary permissions configured. For example, read-only permission ensures that a user does not accidentally delete data. *** Modified at: May 11, 2026 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-ibm-cognos-analytics/) (An index of the documentation is available at /llms.txt) --- # Connect with Looker You can configure Looker to connect with SingleStore Helios. Supported version: | Looker | 7.6.29 and later | | ------ | ---------------- | ## Prerequisites * Access to Looker portal. * A SingleStore Helios cluster in `Active` state. ## Connect Looker to SingleStore Helios To connect Looker to SingleStore Helios, perform the following steps: 1. Log in to the Looker portal. 2. On the Looker home page, under **Database**, select **Admin > Connections**. 3. On the **Connections** page, select **Add Connection** and enter the following details: * **Name**: The name of the connection. * **Dialect**: SingleStore 7+. * **Host**: The hostname of your cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Default is `3306`. * **Database**: The name of the database. * The SingleStore database username and password. 4. Select **Test these settings** to test the connection settings. Once you are successfully connected, select **Add connection**. ## References * [Configure SingleStore connector](https://cloud.google.com/integration-connectors/docs/connectors/singlestore/configure) * [Create and edit projects](https://cloud.google.com/looker/docs/create-projects) * [Looker instructions](https://cloud.google.com/looker/docs/db-config-mysql-mariadb-singlestore) *** Modified at: September 29, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-looker/) (An index of the documentation is available at /llms.txt) --- # Connect with MicroStrategy MicroStrategy is a business intelligence software, which offers a wide range of data analytics capabilities. As a suite of applications, it offers data discovery, advanced analytics, data visualizations, embedded BI, and banded reports and statements. This topic shows you how to integrate MicroStrategy Desktop and MicroStrategy 2019 with SingleStore Helios. ## MicroStrategy Desktop ## Prerequisites Prior to configuring MicroStrategy Desktop to connect to SingleStore Helios, you will need the following software installed. * MicroStrategy Desktop version 11.1.1 * Either the [MySQL 64-bit ODBC driver (version 8.0.15)](https://dev.mysql.com/get/Downloads/Connector-ODBC/8.0/mysql-connector-odbc-8.0.15-winx64.msi) or the [MariaDB 64-bit ODBC driver (version 3.1.0)](https://downloads.mariadb.com/Connectors/odbc/connector-odbc-3.1.0/mariadb-connector-odbc-3.1.0-win64.msi). Click on either link to download the install file for the driver. After the download completes, run the install. ## Creating a Data Source Name (DSN) to Connect to SingleStore Helios 1. Open the **ODBC Data Source Administrator**. To find this program on Windows 7, select the **Control Panel** from the Start Menu. Then click **Administrative Tools**, followed by **Data Sources (ODBC)**. To find this program on Windows 10, type **ODBC Data Sources** in the search bar on the Start Menu. 2. Click the **System DSN** tab. 3. Click **Add…** on the right. Select either the MySQL ODBC 8.0 Driver or the MariaDB ODBC 3.1 Driver and click **Finish**. 4. A window appears, where you will configure the settings needed to create the data source. Specify a **Data Source Name**. In the next section, you will configure MicroStrategy to connect to SingleStore Helios via this data source. For **TCP/IP Server**, specify the IP address or server name where the SingleStore Helios master aggregator node resides. Specify the **User** and **Password** to use to connect to SingleStore Helios. 5. If using the MySQL driver, click **Details >>**. For **Authentication** type `mysql_native_password`. Then click **Details <<**. 6. Click **Test** to test the connection. The connection should be successful. 7. Click **OK**. The data source appears on the **System DSN** tab. ## Connecting MicroStrategy to SingleStore Helios via your ODBC Data Source 1. Open MicroStrategy Desktop. Click the **New Dossier** button. 2. Click on the **New Data** button, which is located on the lower part of the right pane. 3. For the **Data Source**, select either **MySQL** or **MariaDB**. 4. The **Select Import Options** screen appears. Choose **Select Tables** and click **Next**. 5. On the **Import from Table - Select** screen, click the **`+`** button, which is located to the right of **Data Sources**. 6. The **Data Source** window appears. Select **DSN Data Sources**. For **DSN**, select the data source name that you created in [Creating a Data Source Name (DSN) to Connect to SingleStore Helios](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy/#UUID-658f0648-7064-7a52-faff-9caba6e70571.md) in the section above. After you select the **DSN**, specify a **Data Source Name**, which is how you will refer to the data source when using it in MicroStrategy. Click **Save**. 7. On the **Import from Table - Select** screen, the data source you created in the previous window appears in the **Data Sources** Section on the left. Click on this data source. Its tables appear in the **Available Tables** pane on the right. You can now explore the data in these tables. ## MicroStrategy 2019 When integrating MicroStrategy 2019 with SingleStore Helios, the recommended versions are shown below: | Certification Matrix | Versions | | ------------------------- | -------- | | MicroStrategy Enterprise | 2019 | | SingleStore Helios | 7.1 | | MySQL ODBC Unicode driver | 8.0 | ## MicroStrategy three-tier architecture A MicroStrategy system is built around a multi-tier structure. This section provides instructions on how to set up a three-tier architecture. The diagram below shows a four-tier system including the use of MicroStrategy Developer. The instructions below will not cover how to set up or use MicroStrategy Developer as part of a MicroStrategy-SingleStore Helios solution. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9445cac445dc8be4/6a7adad2889c3776b5057c77/16083b577c2228-DHGYpI.gif) **Tier 1** The first tier consists of two databases: the data warehouse, which contains the information that your users analyze; and the MicroStrategy metadata, which contains information about your MicroStrategy projects. This topic shows how to use a MySQL server for storing the metadata and how to use a SingleStore Helios cluster as a data warehouse. **Tier 2** The second tier consists of MicroStrategy Intelligence Server also called I-server, which runs your reports, dossiers, and documents against the data warehouse. **Tier 3** The third tier is MicroStrategy Web, which delivers the reports to a client. ## Install Tiger Virtual Network Computing (VNC) TigerVNC is a high-performance, platform-neutral implementation of VNC (Virtual Network Computing), a client/server application that allows users to launch and interact with graphical applications on remote machines. You will use this for all MicroStrategy interactions in GUI mode. The following instructions describe how to perform and start a VNC server on the machine that will run MicroStrategy 2019. In this guide, the OS for that machine is RHEL 7.x. 1. First, follow the instructions from [RedHat’s documentation](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-tigervnc) to install TigerVNC. 2. As the root user, uncomment the last two lines in `/etc/sysconfig/vncservers` and save the file. 3. Run `echo "kernel.sem=250 32000 100 2048" >> /etc/sysctl.conf`. 4. Reboot the machine. 5. Create the `mstruser` user. Set a password as follows: ```shell su – mstruser ``` ```shell vncpasswd ``` Set the password. 6. Start the vncserver as root. ```shell /etc/rc.d/init.d/vncserver start ``` Check the output messages and confirm the server is listening on port 5901 or 5902. Also check the `vnc.log` to confirm the port. After you have a VNC server set up, you must set up the TigerVNC client application on a client machine. 1. On your client machine, download and install the [latest stable version of TigerVNC](https://github.com/TigerVNC/tigervnc/releases). 2. Open a terminal window and connect to the VNC server. ```shell ssh -i "" -L 5901:localhost:5901 mstruser@ ``` 3. Start the Tiger client application and set the VNC server address to localhost:5901. 4. Click **Connect**. 5. Enter the password you set above. If successful, a Tiger VNC window will open with a built-in terminal window. > **📝 Note**: The Tiger VNC client will render only if the VNC terminal is active on the server. ## Install MicroStrategy 2019 on Red Hat 7.x 1. On your server, download MicroStrategy 2019 Intelligent Enterprise from [MicroStrategy](https://www.microstrategy.com). The filename will be similar to MicroStrategy\_11.1\_Linux.tar.gz. **Note** You will need a license from MicroStrategy to use MicroStrategy 2019 Intelligent Enterprise. 2. Create the directory `/var/lib/mstr`. Make sure sufficient space is available. 3. Extract the tar file mentioned above into `/var/lib/mstr`. 4. Change directories into `/var/lib/mstr/Installations/QueryReportingAnalysis_Linux`. ```shell cd /var/lib/mstr/Installations/QueryReportingAnalysis_Linux ``` **Note**: Please ensure that all the directories belong to the `mstruser`. 5. Run `./setup.sh`. This script will load the GUI window. **Note**: The user configuring and starting VNC must be same as user installing MicroStrategy, which in this case is `mstruser`. 6. Follow the prompts on the screen for the installation. 7. Specify `/var/lib/mstr/MicroStrategy` as the location for all the software components. 8. Disable SELinux and then reboot the machine. 9. Install Perl (if not already present). ```shell sudo yum install perl ``` 10. Install the following Linux packages on the Red Hat machine as shown below: ```shell sudo yum install libXcursor libXdamage libXext \ libXfixes libXi libXrender libXtst nss nss-util \ nspr cups-libs dbus-libs expat libXScrnSaver libXrandr \ alsa-lib pango cairo atk at-spi2-atk gtk3 gdk-pixbuf2 libgcc -y ``` 11. Install only the necessary components. (MicroStrategy will automatically indicate if there are dependencies.) 12. Ensure that the installation was successful by checking the onscreen messages and reviewing `./MicroStrategy/log/install.log`. ## Install MySQL Driver for MicroStrategy 1. On your MicroStrategy Intelligence server, follow instructions from [here](https://tecadmin.net/install-mysql-on-centos-redhat-and-fedora/) to install the MySQL 5.7 suite (MySQL Community Server and MySQL Service). **Note**: MicroStrategy 2019 does not work with MySQL 5.2 ODBC driver or MySQL 8.0 (mysql80-community-release-el7-1.noarch.rpm). Also, check `/usr/lib64/` for the file `libmyodbc8w.so`. This must be on your Intelligence Enterprise server before continuing. 2. Run `/home/mstruser/MicroStrategy/bin/mstrodbcadx` and configure an ODBC connection. 3. Click **Add** and enter the details on the next screen. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4ee78c45a0869627/6a7adafe4c4bfb6963cc1f14/16083b577ca650-mc3dI7.png) ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta6f2420638a168da/6a7adbc16c6eac2124f08d3a/16083b577d1b01-rZ2bqz.png) 4. Enter the username and password for the MySQL server and click **Test**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltf5b1b65f805e901a/6a7adb0f227b1c555e58ce31/16083b577d9511-9rogsJ.png) ## Configure MicroStrategy Intelligence Server on Red Hat 1. Run `/home/mstruser/MicroStrategy/bin/mstrlicmgr` and check the first screen (shown below). The license status should be **Active**; otherwise, click on the **License Administration** tab and follow the screen prompts to request an activation code. MicroStrategy will send the code to your registered email. Using the same screen enter the Activation code to activate the installation. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltecd1b70541134317/6a7ada59c2cc092a1723db43/16083b577e1521-P1skjU.png) 2. Run `/home/mstruser/MicroStrategy/bin/mstrcfgwiz`. 3. Choose the first option “Create Metadata,….”. 4. Uncheck **History and Statistics** option and click **Next**. 5. Select **MYSQL8** in the next screen for DSN and provide your user name and password (same as ODBC connection). Click **Next**. 6. On “Metadata Repository Connection Information”, click **Apply**. **Note**: MicroStrategy metadata will be created in the metadata2019 database in MySQL Server. 7. Click **Configure Intelligence Server**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt98eca0fa51b674f7/6a7ada58bdcff0143ec377ca/16083b577e896f-5Lo5o3.png) On the next screen the DSN information will be pre-populated. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4e153202757f9f58/6a7adbd06c6eac1e7ff08d3c/16083b577f084e-PIZw1n.png) 8. On the next screen do not enter any password, just click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8028d027a957fc3d/6a7ada70d402d99a9deadcb0/16083b578035d9-Vckge7.png) 9. On the next screen enter `MSTR2019_IServer` as the Name and click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltf42e596c7e214839/6a7adbca42a117cdba950979/16083b5780ad0b-bRQhC8.png) 10. On the next screen leave the port numbers as the default selected and click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8ef863595344d905/6a7ada7473d9bd0cfc292476/16083b5781195c-2j7lgm.png) 11. On the next screen no need to check “Configure SSL”, so click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt65f427ad7bb76d31/6a7ada6a9f5251906965e8cd/16083b57818c46-XbkLIX.png) 12. On the next screen check the “Enable…” checkbox for statistics and provide the same credentials of MySQL as you did before. Click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5f416d9834e776b5/6a7adbccfc63ab8be564187d/16083b5781f8f8-E96gdn.png) ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt27ee364b6f957526/6a7ada71c7a40cb39130ea68/16083b578277bb-po1pkD.png) 13. On the next screen, click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6101b30d900babe3/6a7ada7577b03470093f3f7a/16083b5783043f-UzXejK.png) 14. On the next screen, click **Apply**. MicroStrategy will run the scripts and confirm completion. Click **Finish**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcfe2dd390a781d64/6a7adbd126618c8423843021/16083b57838393-6W2lKt.png) ## Install JDK on Red Hat MicroStrategy Web Interface requires Java to be installed, so you must download and install the JDK on the RedHat server that is running MicroStrategy 2019 Intelligent Enterprise. 1. Navigate to the [Oracle JDK download page](https://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase6-419409.html). 2. Download jdk-6u45-linux-x64-rpm.bin. **Note**: You may need an Oracle account to download the JDK. 3. Login to the Red Hat server which has MicroStrategy installed already and run `sudo su -` to assume root privileges. 4. Copy the file `jdk-6u45-linux-x64-rpm.bin` to the `/var/lib/mstr` directory and give the file execute permissions. ```shell chmod 744 ``` 5. Execute the file as `./jdk-6u45-linux-x64-rpm.bin`. It will unpack all components under `/opt/sun/javadb`. 6. Update `/home/mstruser/.bash_profile` with the following: ```shell export JAVA_HOME=/opt/sun/javadb export PATH=$JAVA_HOME/bin:$PATH ``` 7. Verify that Java was installed correctly by running `java -version` and `javac -version`. Version 1.6.0\_45 should be listed. ## Install Tomcat 7 on Red Hat Now you will need to install Tomcat 7 on your RedHat server running the MicroStrategy Intelligence server software. 1. Login to the Intelligence server. 2. Install Tomcat 7. ```shell sudo yum install tomcat ``` **Note**: All important files will be installed in the `/usr/share/tomcat` directory. 3. Edit the `/usr/share/tomcat/conf/tomcat.conf` file and change the value for `JAVA_OPTS` as follows: ```shell JAVA_OPTS="-Xms1024m -Xmx2048m" ``` 4. Start Tomcat. ```shell sudo systemctl start tomcat ``` ## Start MicroStrategy Intelligence Server Start the Intelligence Server by running `/home/mstruser/MicroStrategy/bin/mstrsvcmgr` and then click **Start**. The Intelligence Server will start and the status will change to **Running**. > **📝 Note**: Tiger VNC window may hang when the MicroStrategy Intelligence Server is running and hence make that terminal unusable.An alternate way of starting Intelligence server is from the command line by running the following from the `MicroStrategy/bin` directory:```shell > sudo ./mstrctl -s IntelligenceServer start > > ``` ## Configure MicroStrategy Project Source Once the MicroStrategy metadata is built and the Intelligence Server has been configured, it is necessary to create one or more Project Sources. A Project Source could be defined as a direct or indirect access point to the metadata. It is used by MicroStrategy Developer and other client interfaces to build, interact, and manage the contents of the MicroStrategy metadata. You need to have a machine running MicroStrategy Developer for creating the project source. Follow these steps for creating a project source: 1. Go to the **Tools** > **Project Source Manager**. 2. Provide a Project Source name. 3. Click **Add** and then select **Server** for the connection mode. 4. Provide the Intelligence Server IP address, or server name, and a port number if applicable. 5. To open the web interface, open the following URL: ```shell https://:8080/ MicroStrategy/servlet/mstrWebAdmin ``` 6. Use `Administrator` as the login username with no password. ## Accessing SingleStore Helios database through MicroStrategy web interface Login into the web interface and follow the steps below to access the SingleStore Helios database: 1. Click on the MicroStrategy Web Home on the Administrator Page: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt02f496fdc09d9927/6a7ada61c2cc09d25b23db47/16083b5783f429-ofOvUw.png) 2. You will be able to see the source project created above. Click on the project source. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt289fd014de3d1b79/6a7ada7af682572a8758d938/16083b57846824-gny8xA.png) 3. Enter the user name and password. A successful login will take you to the Home page of your project source. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt33fef84e7fb3e494/6a7adb945947116fba4c268e/16083b5784ea06-igESOV.png) 4. Select **Add External Data** under the Analyze section. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0bece5a1e1c762a9/6a7adad577b0341bbf3f3f92/16083b57857613-Km5mJg.png) 5. On the Connect to your Data page, select **Databases**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt376e88edc3985088/6a7ada62d402d93c35eadca4/16083b5785eae8-nt30jz.png) 6. Select **Select Tables** option and then click **Next**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt87b45352f04ea004/6a7adbb84c4bfb0f4dcc1f55/16083b5786635a-28PIhu.png) You will be able to see the existing SingleStore Helios data sources along with the databases. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt38728485132cb7a1/6a7adb95366687462a27e3b3/16083b5786de5a-bMwz1o.png) 7. Select the required database and then drag drop the required tables on to the canvas. Click **Finish**. 8. Based upon your requirement, select the data access mode: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt0cd959e7d3c6b073/6a7adbc4f68257825958d991/16083b57875e86-KhuNli.png) 9. Save the connection and you are ready to create dossiers, documents, and much more. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt172fcc6252cc1b17/6a7ada63889c374eb4057c5e/16083b5787de2c-1U6Pu7.png) *** Modified at: April 6, 2023 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-microstrategy/) (An index of the documentation is available at /llms.txt) --- # Connect with Oracle GoldenGate You can load your data from various sources into SingleStore using Oracle GoldenGate and analyze the data in real time. Add a connection to your SingleStore Helios cluster in GoldenGate, and use it in the Replicat that loads data in the target SingleStore database. ## Prerequisites * Access to Oracle GoldenGate. * [Download](https://www.oracle.com/in/middleware/technologies/goldengate-downloads.html) and [install](https://docs.oracle.com/en/middleware/goldengate/core/21.3/coredoc/install-oracle-goldengate.html) Oracle GoldenGate Microservices for MySQL v21.3.x. * [Install](https://docs.oracle.com/en/middleware/goldengate/core/21.3/coredoc/install-installing-patches-ma.html#GUID-EAC33ABB-4278-415A-84D2-F4DFD2DEC360) the following patch: **Patch 35271106: Oracle GoldenGate 21.10.0.0.1 Microservices for MySQL-compatIble DBs**. * Hands-on knowledge of the GoldenGate application/UI (refer to [GoldenGate documentation](https://docs.oracle.com/en/middleware/goldengate/core/21.3/index.html)). ## Connect to SingleStore Helios from Oracle GoldenGate To connect your SingleStore databases with Oracle GoldenGate, perform the following tasks: 1. Create a source **Extract**, select **Administration Service > Overview**. Under **Extracts**, select the **+** (plus) icon to add a new extract. The trails from this extract will be used by a SingleStore Replicat. Any database supported by Oracle GoldenGate as a source can support a SingleStore Replicat. 2. Select **Administration Service > Configuration > Database > Credentials > +** (plus icon) to create/configure a connection to the SingleStore Helios database. 3. On the **Credentials** page, enter or select the following information: * **Credential Alias**: Enter an alias for this credential configuration. * **Database Server**: Enter the IP address or hostname of the SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: The default port is `3306`. * **User ID**: Enter the username of the SingleStore user. The user specified in the **Credential Alias** must have the required privileges to create a table in the SingleStore database. * **Password**: Enter the password for the SingleStore user. * **Verify Password**: Re-enter the **Password**. * Use defaults for other configuration items. 4. Select **Submit**.The **Credential Alias** is created. 5. From the **Credentials** section, use the **Credential Alias** created earlier to connect to the target database. 6. If you want to create a Parallel Replicat, skip to step 8. A checkpoint table is automatically created for a Parallel Replicat when it starts. Create a checkpoint table to be used by the Replicat. Select the **+** (plus) icon next to **Checkpoint**. 7. Enter a name for the table in the **Checkpoint Table** box, and select **Submit**. The checkpoint table is added. 8. Select **Administration Service > Overview**. To create a new Replicat, select the **+** (plus) icon next to **Replicats**. 9. From the **Replicat Type** list, select a Replicat type, and then select **Next**. 10. On the **Basic Information** page, select or enter the following information: * **Process Name**: Enter a name for the process. * **Intent**: Select the Intent from the list. * **Credential Alias**: Select the Credential Alias created earlier from the list. * **Trail Name**: Name of the Extract created in step 1. * Use defaults for other configuration items. 11. Select **Next**. 12. On the **Parameter File** page, update the parameter setting to include the required source schema tables that will be mapped to the target schema tables in SingleStore. 13. Select **Create and Run**, and confirm that the new Replicat is running. To confirm that the replication process is running, perform some DML operations on your database, and then perform the following tasks: * Under **Extract**, select **Action > Details**. On the **Statistics** tab, you can see the number of operations performed. * Select the **Statistics** tab for the Replicat (**Replicat > Action > Details**). The updates are visible in the **Table Statistics** section. ## Remarks * You cannot replicate your data to views. Refer to [CREATE VIEW](https://docs.singlestore.com/managed-service/en/reference/sql-reference/data-definition-language-ddl/create-view.html) for more information. * Updates to columns in the shard key are not supported. * Updates to the primary key on tables without explicit shard keys are not supported, because SingleStore assigns the primary key column as the shard key in these cases. * The `LIMITROWS` option for the Replicat is not supported for SingleStore tables without either a primary key or a unique key that is spread across multiple partitions. * SingleStore does not support cross-database transactions. Therefore, Replicat can only map to a single database or schema. This includes mappings for checkpoints and heartbeat tables, which must all be created in the same database. Refer to [Oracle GoldenGate Integration for SingleStore Helios](https://www.singlestore.com/blog/oracle-goldengate-integration-for-singlestoredb/) for related information. *** Modified at: September 26, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-oracle-goldengate/) (An index of the documentation is available at /llms.txt) --- # Connect with Power BI You can create interactive dashboards and analysis reports from your SingleStore Helios cluster's data in Power BI. ## Prerequisites Before connecting Power BI to SingleStore Helios, perform the following tasks. 1. Download and install the latest version of [Power BI Desktop](https://powerbi.microsoft.com/en-us/desktop/). > **📝 Note**: The SingleStore Direct Query connector only supports 64-bit versions of Power BI. 2. Download and install the latest version of [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md). If you are using an older version of Power BI and the connector, see [Upgrade the SingleStore Direct Query Connector](https://docs.singlestore.com/#section-idm323347929938178.md). ## Upgrade the SingleStore Direct Query Connector > **📝 Note**: You may need to update the Power BI reports after upgrading the connector. The SingleStore Direct Query connector is included in the latest version of Power BI. To upgrade the connector to the latest version: 1. If you are using the SingleStore Power BI Bundle, uninstall it. Ensure that the connector files are deleted from the machine. By default, the connector files are located in the `%USERPROFILE%\Documents\Power BI Desktop\Custom Connectors` folder. 2. Download and install the latest version of [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) on the machines with Power BI Desktop and gateway. > **📝 Note**: If the same machine is running Power BI Desktop and acting as the gateway, download and install the SingleStore ODBC driver on this machine. 3. Update Power BI Desktop to version 2.106.883.0 or later. ## License SINGLESTORE, INC. ("SINGLESTORE") AGREES TO GRANT YOU AND YOUR COMPANY ACCESS TO THIS OPEN SOURCE SOFTWARE CONNECTOR AND PROVIDE SUPPORT & MAINTENANCE OR OTHER SERVICES ONLY IF (A) YOU AND YOUR COMPANY REPRESENT AND WARRANT THAT YOU, ON BEHALF OF YOUR COMPANY, HAVE THE AUTHORITY TO LEGALLY BIND YOUR COMPANY AND (B) YOU, ON BEHALF OF YOUR COMPANY ACCEPT AND AGREE TO BE BOUND BY ALL OF THE OPEN SOURCE TERMS AND CONDITIONS APPLICABLE TO THIS OPEN SOURCE CONNECTOR AS SET FORTH BELOW (THIS “AGREEMENT”), WHICH SHALL BE DEFINITIVELY EVIDENCED BY ANY ONE OF THE FOLLOWING MEANS: YOU, ON BEHALF OF YOUR COMPANY, CLICKING THE “DOWNLOAD, “ACCEPTANCE” OR “CONTINUE” BUTTON, AS APPLICABLE OR COMPANY’S INSTALLATION, ACCESS OR USE OF THE OPEN SOURCE CONNECTOR AND SHALL BE EFFECTIVE ON THE EARLIER OF THE DATE ON WHICH THE DOWNLOAD, ACCESS, COPY OR INSTALL OF THE CONNECTOR OR USE ANY SERVICES (INCLUDING ANY UPDATES OR UPGRADES) PROVIDED BY SINGLESTORE. APPLICABLE OPEN SOURCE LICENSE: GNU LESSER GENERAL PUBLIC LICENSE Version 2.1 IF YOU OR YOUR COMPANY DO NOT AGREE TO THESE TERMS AND CONDITIONS, DO NOT CHECK THE ACCEPTANCE BOX, AND DO NOT DOWNLOAD, ACCESS, COPY, INSTALL OR USE THE SOFTWARE OR THE SERVICES. ## In this section * [Data Connectivity Modes](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/data-connectivity-modes.md) * [Connect Power BI Desktop to SingleStore](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/connect-power-bi-desktop-to-singlestore.md) * [Connect Power BI Service to SingleStore via Power BI Gateway](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/connect-power-bi-service-to-singlestore-via-power-bi-gateway.md) *** Modified at: February 6, 2023 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-power-bi/) (An index of the documentation is available at /llms.txt) --- # Connect with Sisense You can connect Sisense to SingleStore Helios. Sisense supports two types of data models: **ElastiCube Model** and **Live Model**. * When you use the **ElastiCube Model**, you import data from data sources, and then merge, manipulate, and query the data. If the data in the data sources changes, you will need to refresh the data sources to see the changes. * When you use the **Live Model**, you work directly with data sources. You do not need to refresh the data sources if the data changes. ## Prerequisites Download and install Sisense. Refer to [Sisense Documentation](https://documentation.sisense.com/) for more information. ## Connect Sisense to SingleStore using ElastiCube Model 1. Open **Sisense**, and select the **Data** tab from the top menu. 2. Select **+ElastiCube**. Specify a name for your new ElastiCube Model, and select **Save**. 3. A screen appears that has a row labeled **Data >>** with the name of the ElastiCube Model you specified in the previous step. Select the **+Data** button on the right. Then, select the **SingleStore** icon from the available connectors. 4. A screen appears where you specify your SingleStore Helios connection details. For the **Location**, specify the IP address or hostname of your cluster. Then, specify the username and password used to connect to your cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. Select **Next**. 5. Select the database that you want to use. Then, select the tables or views in this database that you want to work with. Select **Done**. Your SingleStore Helios cluster is now configured and can be accessed from Sisense using ElastiCube Model. ## Connect Sisense to SingleStore using Live Model 1. Open **Sisense**, and select the **Data** tab from the top menu. 2. Select **+Live**. Specify a name for your new Live Model, and select **Save**. 3. A screen appears that has a row labeled **Data >>** with the name of the Live Model you specified in the previous step. Select the **+Data** button on the right. Then, select the **SingleStore** icon from the available connectors. 4. A screen appears where you specify your SingleStore Helios connection details. For the **Location**, specify the IP address or hostname of your cluster. Then, specify the username and password used to connect to your cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. Select **Next**. 5. Select the database that you want to use. Then, select the tables or views in this database that you want to work with. Select **Done**. Your SingleStore Helios cluster has now been configured and can be accessed from Sisense using Live Model. *** Modified at: February 6, 2023 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-sisense/) (An index of the documentation is available at /llms.txt) --- # Connect with Superset Apache Superset is an open-source business intelligence (BI) and data visualization tool. It enables you to run queries and build interactive dashboards on your SingleStore databases, supporting both SQL-based workflows and visual dataset exploration for the SingleStore database. To connect with SingleStore, specify the connection URL using the [SingleStore SQLAlchemy dialect](https://github.com/singlestore-labs/sqlalchemy-singlestoredb) (`singlestoredb://`) in the following format: `singlestoredb://:@:/` where, * `hostname`: IP address or hostname of the SingleStore cluster. * `port`: Port of the SingleStore cluster. Default is `3306`. * `username`: Username of the SingleStore database user with which to connect. * `password`: Password for the SingleStore database user. ## Prerequisites Install the `sqlalchemy-singlestoredb` package. ## Connect Superset to SingleStore To connect your SingleStore database, configure a database connection in Superset: 1. Log in to Superset. 2. From the **Settings** list, select **Database Connections**. 3. On the **Databases** page, select the **+ Database** button. 4. On the **Connect a database** dialog, from the **Supported databases** list, select **SingleStore**. 5. On the **Basic** tab, enter the following: * **Display Name**: Enter **SingleStore**. * **SQLAlchemy URI**: Enter the connection URL in the following format: `singlestoredb://:@:/` * Select **Test Connection** to verify connectivity to your SingleStore database. 6. Select **Connect**. The SingleStore database is now added as a database connection in Superset. Either create a dataset or query with SQL to explore your SingleStore database. Refer to [Superset Documentation](https://superset.apache.org/user-docs/) for more information. *** Modified at: April 7, 2026 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-superset/) (An index of the documentation is available at /llms.txt) --- # Connect with Tableau You can use SingleStore Helios and Tableau together to ingest your data and perform real-time analytics. You can connect with your SingleStore Helios cluster from Tableau using the following drivers/connectors: * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) (Tableau Desktop) * [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) (Tableau Server) * MySQL ODBC Driver (Tableau Prep) For most use cases, SingleStore recommends using the SingleStore ODBC/JDBC driver (as applicable). You can connect to your SingleStore Helios cluster from the following tools/products offered by Tableau: * [Tableau Desktop](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau.md) * [Tableau Prep](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-prep.md) * [Tableau Server](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-server.md) SingleStore recommends having hands-on experience on using Tableau. For more information, refer to [Tableau Help Documentation](https://www.tableau.com/support/help). ## In this section * [Connect with Tableau Desktop](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau.md) * [Connect with Tableau Prep](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-prep.md) * [Connect with Tableau Server](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/connect-with-tableau-server.md) *** Modified at: September 26, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tableau/) (An index of the documentation is available at /llms.txt) --- # Connect with ThoughtSpot ThoughtSpot is a modern analytics and business intelligence (BI) platform that enables users to run search-driven and AI-powered data analytics. You can integrate your SingleStore databases with ThoughtSpot and generate Answers and Liveboards using live queries. Refer to [SingleStore Overview - ThoughtSpot Documentation](https://docs.thoughtspot.com/cloud/latest/connections-singlestore) for related information. ## Create a Connection to SingleStore in ThoughtSpot To connect to SingleStore from ThoughtSpot: 1. Log in to your ThoughtSpot account. 2. On the Home page, select **Create New > Connections**. Alternatively, select **Data > Connections > Others**. 3. Enter a name and description for the connection. 4. From the list of connections, select **SingleStore**, and then select **Continue**. 5. On the **Create Connection** dialog, enter the connection configuration of your SingleStore Helios deployment. 1. **Host**: Hostname or IP address of your SingleStore deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. 2. **Port**: Port of the SingleStore deployment. 3. **User**: Username of the SingleStore database user. 4. **Password**: Password for the SingleStore database user. 5. **Database**: (Optional) Name of the SingleStore database to connect with. Enter **Database** and the database name in the **Key** and **Value** fields, respectively. Refer to [SingleStore Connection Reference](https://docs.thoughtspot.com/cloud/latest/connections-singlestore-reference) for more supported attributes. 6. (Optional) Configure other settings as required. 6. Select **Continue**. 7. Select the tables and the columns from each (selected) table to add to the connection. 8. Select **Create Connection > Create**. The SingleStore connection is added to the **Data > Connections** page. Select the name of the connection to view the tables and columns added to the connection. After adding the connection, you can start searching your SingleStore database using **Search now**. ## Manage a Connection to SingleStore ## Edit a Connection To edit (or update) an existing SingleStore connection: 1. On the Home page, select **Data > Connections**. 2. Select the connection to edit, and then select **Edit connection**. 3. Update the connection details as required, including connection name, configuration, and the selected tables and columns. 4. After making the required changes, select **Update**. 5. Select **Confirm** to update the connection. The selected SingleStore connection is now updated. ## Validate Connection Metadata You can validate a connection to detect any changes in the metadata. Upon validation, ThoughtSpot generates a JSON file with metadata changes. Use ThoughtSpot Modeling Language (TML) to apply the update to the connection. To validate a SingleStore connection: 1. On the Home page, select **Data > Connections**. 2. Select a connection to validate. 3. Select the ellipsis (three dots), and then select **Validate**. If ThoughtSpot detects any changes in the connection's metadata, a message is displayed with the option to download the metadata changes. Select **Download metadata change log** to download the JSON with metadata changes. ## Share a Connection Users with `can manage data` and `admin` privileges in ThoughtSpot can share a connection with other ThoughtSpot users or groups with `can manage data` privilege. A user or group with access to a connection can add, remove, and modify tables in the connection. To share a SingleStore connection with another ThoughtSpot user or group: 1. On the Home page, select **Data > Connections**. 2. Select the connection(s) to share, and then select **Share**. 3. In the **Share connection** dialog, enter the ID of the user or group with which to share the connection. Optionally, search the user or group. 4. Select **Done**. The selected connection is now accessible to the user or group with the entered ID. ## Revoke Access to a Connection To revoke access for a user or group: 1. On the Home page, select **Data > Connections**. 2. Select a connection, and then select **Share**. 3. In the **Share connection** dialog, select **Remove access** for a user or group to revoke access. 4. Select **Done**. Access is now revoked for the selected user(s) or group(s). ## Remap a Connection To remap the existing tables or columns, update the connection parameters in the source mapping YAML. Check the dependencies before and after remapping. To remap a SingleStore connection: 1. On the Home page, select **Data > Connections**. 2. Select a connection to remap. 3. Select the ellipsis (three dots), and then select **Remapping**. 4. Select **Download** to download the source mapping. 5. Update and save the required changes in the mapping file. 6. On the **Remapping** page, select **Browse your files** and upload the updated mapping file. The mapping of the selected connection is now updated. ## Delete a Table from a Connection Before deleting a table, ThoughtSpot checks for dependencies and displays a list of dependent objects. You cannot delete a table with dependent objects; update or delete the dependent objects before deleting a table from a connection. To delete a table from a SingleStore connection: 1. On the Home page, select **Data > Connections**. 2. Select a connection. 3. Select the table to delete, and then select **Delete**. 4. (Optional) If the selected table has dependencies, you must delete or remove all the dependencies. Otherwise, the delete operation is blocked. The dependencies are listed in the **Cannot Delete** dialog. To delete the dependencies: 1. Select a dependency (linked dependent object) from the **Cannot Delete** dialog. 2. Modify or delete the object. 5. Select **Delete** to confirm the deletion. The selected table is now deleted from the connection. ## Remarks * The selected tables and columns are added as links in ThoughtSpot. Because ThoughtSpot does not cache the linked data and it queries the linked data, rendering the search results may take a while. * Tables with dependencies or dependent objects, such as Models, Liveboards, or Answers, cannot be deleted from a connection. ## Passthrough Functions ThoughtSpot Cloud supports passthrough functions for SingleStore. Passthrough functions enable you to send custom SQL queries directly to SingleStore. ThoughtSpot does not interpret the expressions sent via passthrough functions. Refer to [Passthrough functions for SingleStore](https://docs.thoughtspot.com/cloud/latest/connections-singlestore-passthrough) for information on passthrough functions, including supported passthrough functions, row-level security (RLS) in passthrough functions, and limitations. *** Modified at: October 10, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-thoughtspot/) (An index of the documentation is available at /llms.txt) --- # Connect with Tibco Spotfire Tibco Spotfire is a complete analytics solution. You can use Tibco Spotfire to combine your SingleStore data in a single analysis and get a holistic view of the same with an interactive visualization. Spotfire software makes businesses smart, delivers AI-driven analytics, and makes it easier to plot interactive data on maps. It speeds up data analysis across an organization for faster, confident, and accurate decision-making. The following certification matrix shows the recommended Spotfire version and the SingleStore driver: | Spotfire Analytics 12.0.11 LTS and later | SingleStoreODBC Driver | | ---------------------------------------- | ---------------------- | ## Prerequisites * Download and install [Tibco Spotfire](https://www.spotfire.com/downloads) (for Windows). Create an account and register with a username and password. * Download and install the latest version of [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md). ## Connect Tibco Spotfire to SingleStore 1. Open the Tibco Spotfire desktop application. A sign in screen is displayed. Select **Tibco Login**. 2. On the web authentication page, log in with the credentials used while registering to download Tibco Spotfire. 3. After logging in, you are redirected to the desktop application and the homepage of Tibco Spotfire is displayed. 4. In Spotfire Analyst, open the **Files and data** flyout, and select **Connect to**. Select **SingleStore**, and then select **New Connection**. 5. In the connection dialog, enter the following: * Server IP address / name Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * Database System ( select **SingleStore**) * Authentication Method as Database Authentication * Username to access the database * Password to access the database 6. Select **Connect**. After a successful connection, the **Database** drop down is populated with all the existing databases. Select a database, and select **OK**. 7. In the connection view window, select the tables you want to import. 8. The **Add data to analysis** popup is displayed. Select **Add as new data table > Import > OK**. You are now ready to explore your SingleStore data using Tibco Spotfire. ## Troubleshooting * **Issue**: Corrupted non-english characters in data loaded through Data Connection. **Resolution**: Use the Unicode version of the SingleStore ODBC driver for proper handling of non-english characters in the data. * **Issue**: Fetch performance issue or "Connection reset by peer" error with multiple concurrent connections. **Resolution**: Use the latest version of the SingleStore ODBC driver instead of MySQL ODBC drivers. * **Issue**: Data inconsistency due to Prefetch limit of 100000 and Auto-Paging behavior. **Resolution**: The default Prefetch limit was removed in Spotfire version 12.0.11 LTS, which is the minimum recommended version to use with SingleStore. *** Modified at: December 18, 2025 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-tibco-spotfire/) (An index of the documentation is available at /llms.txt) --- # Connect with Zoho Analytics You can import data from tables, views, stored procedures, etc. from your SingleStore Helios databases in Zoho Analytics and perform analytical operations. You can also use custom queries to import your data. To sync your databases in Zoho Analytics, you can either set up a periodic synchronization schedule or synchronize data on-demand. Refer to [Import data from SingleStore](https://www.zoho.com/analytics/help/cloud-database/singlestore.html) for related information. You can load your SingleStore data into Zoho Analytics, as: * [Import SingleStore Helios databases into Zoho Analytics](https://docs.singlestore.com/#section-idm4544915270022433718928210803.md) * [Connect using Zoho Analytics Live](https://docs.singlestore.com/#section-idm4506154351604833718935536661.md) ## Prerequisites * Access to Zoho Analytics * An active SingleStore Helios cluster * Add the IP address for Zoho Analytics to the IP Allowlist (go to **Cloud Portal > your\_cluster > Firewall > IP Allowlist > Edit**). ## Import SingleStore Helios databases into Zoho Analytics ## Import from a New Data Source To import your SingleStore Helios databases into Zoho Analytics from a new data source: 1. On the Zoho Analytics home page, select **Import Your Data > Databases > SingleStore**. 2. On the **Connect to Cloud Database** page, enter or select the following information, and then select **Next**. * **Cloud Service Name**: Select **SingleStore** from the list. * **Endpoint**: Enter the hostname or IP address of your SingleStore Helios cluster. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * **Port**: Use the default value, `3306`. * **Username**: Enter the username of the SingleStore Helios database user that is used to access the database. * **Password**: Enter the password for the SingleStore Helios database user. * **Database Name**: Enter the name of the SingleStore Helios database to import. * **Use SSL**: Select this checkbox if the SingleStore Helios cluster is configured to accept only SSL-enabled connections. * **Connection Type**: Select **Data Import**. 3. On the **Select Data to Import** page, select the data to import, and then select **Next**. Select one of the available options, and perform the required actions. * **Single Table** 1. Select the table (or view) to import from the **Select Table** list, and select **Next**. 2. On the **Import Settings and Preview** page, specify a name for the workspace and table to create in Zoho Analytics. You can also select the columns to import and modify the column names and data types. Continue from step 4. * **Multiple Tables** 1. Select the table(s) (or views) to import from the list, and select **Next**. 2. On the **Import Settings** page, specify a name for the workspace to create in Zoho Analytics. You can also rename the selected tables. Continue from step 4. * **Custom Query**: Enter the query in the box, and select **Next**. Here's a sample query: ```sql SELECT OrderID, Quantity FROM dbTest.Stock LIMIT 1000; ``` Continue from step 4. * **Stored Procedure**: Specify the stored procedure in the box, and select **Next**. Here's a sample: ``` CALL{ stored_procedure_name } () ``` 4. On the **Import Settings** page, from the **On Import Errors** list, select the action to take in the event of import errors. Select **Next**. Here are the available options: * Set Empty Value for the Column * Skip Corresponding Rows * Don't Import the Data 5. On the **Schedule this Import Periodically** page, configure the import schedule. You can also configure the import behavior using the **How do you want to Import** list. 6. Select **Create**. The **Cloud Database Import Initiated** page notifies you that the import process has started. You may not access the tables before the import task is complete. Select **OK**. You can now create reports and dashboards using the imported data. ## Import from an Existing Data Source To import more tables (or views) into Zoho Analytics from an existing data source: 1. Open your Zoho Analytics workspace configured to import data from a SingleStore Helios database. 2. Select **+ Create > Create New Table > New Table / Import Data > Cloud Databases**. 3. On the **Connect to Cloud Database** page, select the existing **SingleStore** connection for your SingleStore Helios cluster. 4. Select **Next**, and then continue from step 3 of the [Import from a New Data Source](https://docs.singlestore.com/#section-idm4570299554608033718928761305.md) process. ## Connect using Zoho Analytics Live To connect with Zoho Analytics Live: 1. On the Zoho Analytics home page, select **Import Your Data > Cloud Databases**. 2. On the **Connect to Cloud Database** page, enter or select the connection configuration information. Refer to [Import from a New Data Source - Step 2](https://docs.singlestore.com/#section-idm4570299554608033718928761305.md) for related information. 3. Under **Connection Type**, select **Live Connect**, and then select **Next**. 4. On the **Connect Live to Cloud Database** page, select the tables, and then select **Create**. For each selected table from the SingleStore Helios database, a table is created in Zoho Analytics. You can rename the tables in Zoho Analytics, if needed. When you access a table, the data is loaded in real-time from the SingleStore Helios database. ## Update Connection Settings To configure your existing connection's settings: 1. Open your Zoho Analytics workspace. 2. On the left navigation pane, select **Data Sources**. 3. From the list of data sources, select the **SingleStore** data source to update. 4. On the **Data Sources** page, under **SingleStore**, select **Edit Connection**. 5. On the **Cloud Database - Edit Connection** dialog, update the connection settings, and select **Save**. The connection settings are updated to the specified values. ## FAQs * **Can I import data from Views created in a SingleStore Helios database?** Yes, you can import your tables as well as views created in a SingleStore Helios database into your Zoho Analytics workspace. * **Are the foreign keys defined between tables in the SingleStore Helios database available/linked in Zoho Analytics on import?** Yes, when you import multiple tables from a SingleStore Helios database, the foreign keys defined between them are linked in Zoho Analytics as well. The foreign keys are created as Lookup columns in Zoho Analytics. Although, if you import data from a single table at a time (using the Single Table option), the foreign keys are not linked in Zoho Analytics. In such cases, you can manually link the tables using Lookup columns. Refer to [Joining Tables](https://www.zoho.com/analytics/help/table/joining-tables.html) for more information. *** Modified at: July 24, 2026 Source: [/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics/](https://docs.singlestore.com/cloud/query-data/connect-with-analytics-and-bi-tools/connect-with-zoho-analytics/) (An index of the documentation is available at /llms.txt) --- # Query Plan Operations This topic describes the operations that a query plan may use. These operations are displayed when you use [Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md), run [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md), or run [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) to show a query plan. The examples in this topic use two tables : `t`, a rowstore table with a primary key, and `ct` a columnstore table. These tables are both in database `db1`. For more information about interpreting these operators to increase performance, see the [Query Tuning Guide](https://docs.singlestore.com/cloud/query-data/query-tuning.md). ```sql CREATE ROWSTORE TABLE t(id INT PRIMARY KEY, a INT, b INT, KEY(a)); CREATE TABLE ct(a INT, b INT, SORT KEY(a), SHARD KEY(a)); ``` ## Table access methods * `Project` - outputs a subset of columns of the input (for example, a `SELECT`statement that calls out specific columns from a table) in a particular order, and optionally computes new columns that are expressions of existing ones (for example, `SELECT column_a / column_b AS column_c FROM table_name`). * `TableScan` - scans every row in a table using an index * `IndexSeek` - navigates to a particular row using an index * `IndexRangeScan` - scans a range of rows using an index * `ColumnStoreScan` - scans a columnstore table * `OrderedColumnStoreScan` - scans a table using the columnstore sort key in key order ```sql EXPLAIN SELECT * FROM t WHERE t.a = 5; ``` ```output +---------------------------------------------+ | EXPLAIN | +---------------------------------------------+ | Project [t.id, t.a, t.b] | | Gather partitions:all | | Project [t.id, t.a, t.b] | | IndexRangeScan db.t, KEY a (a) scan:[a = 5] | +---------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM ct; ``` ```output +---------------------------------------------------------------+ | EXPLAIN | +---------------------------------------------------------------+ | Project [ct.a, ct.b] | | Gather partitions:all | | Project [ct.a, ct.b] | | ColumnStoreScan db1.ct, KEY a (a) USING CLUSTERED COLUMNSTORE | +---------------------------------------------------------------+ ``` ## Filter Methods * `Filter` - reads a stream of input rows and outputs only those rows that match a specified condition * `BloomFilter` - filters rows based on matching them against a join condition from a corresponding `HashJoin` The following examples show how `Bloomfilter` can appear in a query plan alone, and how it can appear as part of `ColumnstoreFilter` when used on a columnstore table, respectively. ```sql EXPLAIN SELECT * FROM rowstore_table_a straight_join (SELECT WITH (no_merge_this_select=true) * FROM columnstore_table_a) t WITH (bloom_filter=true) ON t.column_b = rowstore_table_a.column_b; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Top limit:[@@SESSION.`sql_select_limit`] | | Gather partitions:all est_rows:1 alias:remote_0 | | Project [r0.column_a, r0.column_b, r1.column_a AS column_a_1, r1.column_b AS column_b_2] est_rows:1 est_select_cost:4 | | Top limit:[?] | | HashJoin | | |---HashTableProbe [r1.column_b = r0.column_b] | | | HashTableBuild alias:r1 | | | Repartition [columnstore_table_a.column_a, columnstore_table_a.column_b] AS r1 shard_key:[column_b] est_rows:1 | | ColumnStoreScan database_name.columnstore_table_a, KEY column_a (column_a) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:1 est_filtered:1 | | TableScan r0 storage:list stream:yes table_type:sharded est_table_rows:1 est_filtered:1 | | Repartition [rowstore_table_a.column_a, rowstore_table_a.column_b] AS r0 shard_key:[column_b] est_rows:1 | | TableScan database_name.rowstore_table_a table_type:sharded_rowstore est_table_rows:1 est_filtered:1 | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM columnstore_table_b straight_join (SELECT WITH (no_merge_this_select=true) * FROM columnstore_table_a) t WITH (bloom_filter=true) ON t.column_b = columnstore_table_b.column_b; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Top limit:[@@SESSION.`sql_select_limit`] | | Gather partitions:all est_rows:1 alias:remote_0 | | Project [r0.column_a, r0.column_b, r1.column_a AS column_a_1, r1.column_b AS column_b_2] est_rows:1 est_select_cost:4 | | Top limit:[?] | | HashJoin | | |---HashTableProbe [r1.column_b = r0.column_b] | | | HashTableBuild alias:r1 | | | Repartition [columnstore_table_a.column_a, columnstore_table_a.column_b] AS r1 shard_key:[column_b] est_rows:1 | | | ColumnStoreScan database_name.columnstore_table_a, KEY column_a (column_a) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:1 est_filtered:1 | | TableScan r0 storage:list stream:yes table_type:sharded est_table_rows:1 est_filtered:1 | | Repartition [columnstore_table_b.column_a, columnstore_table_b.column_b] AS r0 shard_key:[column_b] est_rows:1 | | ColumnStoreScan database_name.columnstore_table_b, KEY column_a (column_a) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:1 est_filtered:1 | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` ## `ColumnstoreFilter` Table Access Method: Applies a Filter to a Columnstore Table The following example demonstrates the ColumnstoreFilter query operation, using the table `articles`: ```sql CREATE TABLE articles ( id INT UNSIGNED, year int UNSIGNED, title VARCHAR(200), body TEXT, SHARD KEY(id), SORT KEY (id), KEY (id) USING HASH, KEY (title) USING HASH, KEY (year) USING HASH); ``` The `EXPLAIN` statement shows the ColumnStoreFilter operation with `index`, because a hash index is used to apply the filter. ```sql EXPLAIN SELECT * FROM articles WHERE title = 'Interesting title here'; ``` ```output +------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 | | Project [articles.id, articles.year, articles.title, articles.body] | | ColumnStoreFilter [articles.title = 'Interesting title here' index] | | ColumnStoreScan d.articles, KEY id_2 (id) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore | +------------------------------------------------------------------------------------------------------+ ``` ## GROUP BY and Aggregations * `Aggregate` - computes an aggregate * `HashGroupBy` - uses a hash table to compute group by results * `StreamingGroupBy` - leverages the fact that the underlying operation produces rows in order to compute group by results. The advantage of `StreamingGroupBy` is that it only uses a constant amount of memory * `ShuffleGroupBy` - occurs when a `GROUP BY` clause operates on a set of columns that do not include the shard key. First, a local `GROUP BY` is performed per host. Then, the data is repartitioned and `GROUP BY` is completed. * `Distinct` - removes duplicate rows ```sql EXPLAIN SELECT SUM(id) FROM t; ``` ```output +-----------------------------------------+ | EXPLAIN | +-----------------------------------------+ | Project [`sum(id)`] | | Aggregate [SUM(`sum(id)`) AS `sum(id)`] | | Gather partitions:all | | Project [`sum(id)`] | | Aggregate [SUM(t.id) AS `sum(id)`] | | TableScan db1.t, PRIMARY KEY (id) | +-----------------------------------------+ ``` ```sql EXPLAIN SELECT SUM(id) FROM t GROUP BY a+1; ``` ```output +------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------+ | Project [`sum(id)`] | | HashGroupBy [SUM(`sum(id)`) AS `sum(id)`] groups:[t.a + 1] | | Gather partitions:all | | Project [`sum(id)`, t.a + 1 AS op, t.a, 1 AS op_1] | | HashGroupBy [SUM(t.id) AS `sum(id)`] groups:[t.a + 1] | | TableScan db1.t, PRIMARY KEY (id) | +------------------------------------------------------------+ ``` ## Distributed data movement * `Gather` - collects all the results from the leaf nodes to the aggregator node. When a query can be routed to a single partition it has the attribute `partitions:single`. If `Gather` collects data from all the partitions the attribute is set to `partitions:all`. If the shard key matches an `IN` list predicate, then the attribute is set to `partitions:inlist`. The query will only be sent to partitions that match the values in the `IN` list. Queries that have `partitions:single` are called *single partition queries*. An advantage of single partition queries is that they can scale to much higher concurrency and throughput because they only need to execute on a single partition. * `GatherMerge` - collects ordered streams of rows from the leaf nodes and merges them to output an ordered stream. ```sql EXPLAIN SELECT * FROM t WHERE id = 1; ``` ```output +-------------------------------------------------+ | EXPLAIN | +-------------------------------------------------+ | Gather partitions:single | | Project [t.id, t.a, t.b] | | IndexSeek db1.t, PRIMARY KEY (id) scan:[id = 1] | +-------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM t WHERE id > 1; ``` ```output +------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------+ | Project [t.id, t.a, t.b] | | Gather partitions:all | | Project [t.id, t.a, t.b] | | IndexRangeScan db1.t, PRIMARY KEY (id) scan:[id > 1] | +------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM t WHERE id IN (2,3,4); ``` ```output +-----------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------+ | Gather partitions:inlist alias:remote_0 | | Project [t.id, t.a, t.b] | | IndexSeek demo.t, PRIMARY KEY (id) scan:[id IN (...)] table_type:sharded_rowstore | +-----------------------------------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM t ORDER BY id; ``` ```output +-----------------------------------+ | EXPLAIN | +-----------------------------------+ | Project [t.id, t.a, t.b] | | GatherMerge [t.id] partitions:all | | Project [t.id, t.a, t.b] | | TableScan db.t, PRIMARY KEY (id) | +-----------------------------------+ ``` * `Repartition` - redistributes a dataset to hash-partition it on a particular key. * `Broadcast` - broadcasts a dataset to every node in a cluster. * `Branch` - a `Branch` operator represents shared computations. A shared result table is now computed once, shown once, and shared across different branches in the plan. For instance, in a broadcast `LEFT JOIN` plan. > **📝 Note**: For broadcast `LEFT JOIN`, a `Branch` operator is added to more accurately represent shared computations. A shared result table is now computed once, shown once, and shared across different branches in the plan.```sql > EXPLAIN SELECT t.*, ct.* FROM t LEFT JOIN ct ON t.a = ct.a; > > ``````output > > +--------------------------------------------------------------------------------------------------------------------------------------------+ > | EXPLAIN | > +--------------------------------------------------------------------------------------------------------------------------------------------+ > | Gather partitions:all est_rows:30 alias:remote_0 | > | Project [SUBQ_VWW_1.a, SUBQ_VWW_1.b, SUBQ_VWW_1.a_1, SUBQ_VWW_1.b_2] est_rows:30 est_select_cost:32 | > | TableScan 1tmp AS SUBQ_VWW_1 storage:list stream:yes est_table_rows:30 est_filtered:30 | > | UnionAll est_rows:30 | > | |---Project [r4.a, r4.b, r4.a_1, r4.b_2] est_rows:1 | > | | Filter [$0 = 8] | > | | HashGroupBy [COUNT(*) AS $0] groups:[r4.i0] | > | | TableScan r4 storage:list stream:no table_type:sharded est_table_rows:30 est_filtered:30 | > | | Project [r3.i0, r3.a, r3.b, r3.a_1, r3.b_2] alias:r4 est_rows:30 | > | | TableScan r3 storage:list stream:yes table_type:sharded est_table_rows:30 est_filtered:30 | > | | Repartition AS r3 shard_key:[i0] est_rows:30 | > | | Branch [SUBQ_VWW_0.ConstIntCol IS NULL] position:[2/2] | > | Project [r2.a, r2.b, r2.a_1, r2.b_2] est_rows:30 | > | TableScan r2 storage:list stream:yes table_type:sharded est_table_rows:30 est_filtered:30 | > | Project [r1.i0, r1.a, r1.b, SUBQ_VWW_0.a_1, SUBQ_VWW_0.b_2] alias:r2 est_rows:30 | > | Branch [SUBQ_VWW_0.ConstIntCol IS NOT NULL] position:[1/2] | > | HashJoin type:right | > | |---HashTableProbe [r1.a = SUBQ_VWW_0.a_1] | > | | HashTableBuild alias:r1 | > | | Project [r0.a, r0.b, i0] alias:r1 hash_key:[a] est_rows:1 | > | | Window [ROW_NUMBER() OVER () AS i0] | > | | TableScan r0 storage:list stream:yes table_type:reference est_table_rows:1 est_filtered:1 | > | | Broadcast [t.a, t.b] AS r0 distribution:tree est_rows:1 | > | | ColumnStoreScan test1.t, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:1 est_filtered:1 | > | TableScan 0tmp AS SUBQ_VWW_0 storage:list stream:yes est_table_rows:3,072 est_filtered:3,072 | > | Project [ct.a AS a_1, ct.b AS b_2, 0 AS ConstIntCol] est_rows:3,072 | > | ColumnStoreScan test1.ct, KEY a (a) USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:3,072 est_filtered:3,072 | > +--------------------------------------------------------------------------------------------------------------------------------------------+ > 27 rows in set (0.01 sec) > ``` ```sql EXPLAIN SELECT * FROM t,ct WHERE t.id = ct.b; ``` ```output +-----------------------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------------------+ | Project [t.id, t.a, t.b, r0.a_1, r0.b_2] | | Gather partitions:all est_rows:1 | | Project [t.id, t.a, t.b, r0.a_1, r0.b_2] est_rows:1 est_select_cost:3 | | NestedLoopJoin | | |---IndexSeek db1.t, PRIMARY KEY (id) scan:[id = r0.b_2] est_table_rows:1 est_filtered:1 | | TableScan r0 storage:list stream:no | | Repartition [ct.a AS a_1, ct.b AS b_2] AS r0 shard_key:[b_2] est_rows:1 | | ColumnStoreScan db1.ct, KEY a (a) USING CLUSTERED COLUMNSTORE est_table_rows:1 est_filtered:1 | +-----------------------------------------------------------------------------------------------+ ``` * `ChoosePlan` indicates that SingleStore will choose one of the listed plans at runtime based on cost estimates. `estimate` illustrates the statistics that are being estimated, but note that these SQL statements are not actually estimated. Instead, SingleStore uses index information to estimate these statistics. ```sql EXPLAIN SELECT * FROM t WHERE id > 5 AND a > 5; ``` ```output +-----------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------+ | Project [t.id, t.a, t.b] | | Gather partitions:all | | Project [t.id, t.a, t.b] | | ChoosePlan | | | :estimate | | | SELECT COUNT(*) AS cost FROM db1.t WHERE t.id > 5 | | | SELECT COUNT(*) AS cost FROM db1.t WHERE t.a > 5 | | |---Filter [t.a > 5] | | | IndexRangeScan db1.t, PRIMARY KEY (id) scan:[id > 5] | | +---Filter [t.id > 5] | | IndexRangeScan db1.t, KEY a (a) scan:[a > 5] | +-----------------------------------------------------------+ ``` ## Other SQL Methods * `UnionAll` - combines the result set of two or more `SELECT` statements. It does not remove duplicate rows but simply return all rows. * `Window` - window operation to facilitate computing values over specified subsets of rows in a window partition. A set of rows in a window partition is referred to as a window frame. ```sql EXPLAIN EXTENDED SELECT * FROM t1 UNION ALL SELECT * FROM t2; ``` ```output +-----------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------+ | UnionAll | | |---Project [t2.c1] | | | TableScan db1.t2 table_type:sharded_rowstore | | +---Project [t1.c1] | | TableScan db1.t1 table_type:sharded_rowstore | +-----------------------------------------------------------+ ``` ## Ingest Methods * `SegmentMerge` - the operation to merge smaller segments into big ones while preserving the global ordering already in place. It’s done by sorting segments by min value or the sort key and merging them and often with other conditions as well. * `SegmentSort` - sort the rows within each newly written segment, and compute the min/max sort key values within the segment. ```sql SHOW CREATE TABLE t; ``` ```output +-------+----------------------------------------------------------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------------------------+ | t | CREATE TABLE 't' ( | | | 'a' int(11) DEFAULT NULL, | | | 'id' bigint(11) NOT NULL AUTO_INCREMENT, | | | SORT KEY 'id' ('id') | | | , SHARD KEY () | | | ) AUTO_INCREMENT=1048577 AUTOSTATS_CARDINALITY_MODE=OFF AUTOSTATS_HISTOGRAM_MODE=OFF | | | AUTOSTATS_SAMPLING=OFF SQL_MODE='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER' | +-------+----------------------------------------------------------------------------------------+ ``` ```sql SHOW CREATE TABLE s; ``` ```output +-------+----------------------------------------------------------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------------------------+ | s | CREATE TABLE 's' ( | | | 'a' bigint(20) DEFAULT NULL, | | | 'id' bigint(20) DEFAULT NULL, | | | SORT KEY 'id' ('id') | | | , SHARD KEY () | | | ) AUTOSTATS_CARDINALITY_MODE=OFF AUTOSTATS_HISTOGRAM_MODE=OFF AUTOSTATS_SAMPLING=OFF | | | SQL_MODE='STRICT_ALL_TABLES,NO_AUTO_CREATE_USER' | +-------+----------------------------------------------------------------------------------------+ ``` ```sql EXPLAIN INSERT INTO s SELECT * FROM t; ``` ```output +------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------------------------------+ | Gather partitions:all est_rows:1,049,121 alias:remote_0 parallelism_level:partition | | SegmentMerge | | InsertInto xdb.s [t.a, t.id] local:yes est_rows:1,049,121 | | ColumnStoreScan xdb.t, SORT KEY id (id) table_type:sharded_columnstore est_table_rows:1,049,121 est_filtered:1,049,121 | +------------------------------------------------------------------------------------------------------------------------+ ``` ## Joins Join methods: * `HashTableBuild` - builds the hash table used to execute the HashJoin. * `HashTableProbe` - takes every row from the left leg of a HashJoin and scans the hash table being built by the HashTableBuild operation to return the match rows based on the join keys. The following are the three types of joins that the optimizer can perform. They are listed in order of least complex to the most complex algorithm to complete. 1. `NestedLoopJoin` - performs a nested loop join: for every row on the outer side of the join SingleStore scans into the inner table to find all the matching rows. The complexity of the `NestedLoopJoin` operation is on the order of the number of rows in the outer table times the number of rows on the inner table. If there are millions of rows in each table, this operation is not efficient. It is best to add an index and or shard keys to the tables for that use case. If no index or shard key exists in either table, the optimizer performs keyless sharding or sorting on the table outer table before the `NestedLoopJoin` operation. This is displayed by an index seek operation in the `EXPLAIN` output. This is the fallback join that the optimizer will perform if all other joins fail. 2. `HashJoin` - performs a hash join: SingleStorebuilds a hash table from one of the joined tables. For every row in the table stated to the left of a `JOIN` syntax, the hash table is scanned. If there is a match, the rows are joined. The hash table must fit in memory, so SingleStore attempts to construct the hash table utilizing the table with fewer rows. If there isn't enough memory to create the hash table, then a `NestedLoopJoin` is performed instead. 3. `MergeJoin` - performs a merge join: SingleStore scans both inner and outer sides of the join at the same time and merges matching rows. Both tables must have a sort key(s) and shard key(s) on the column that is being joined. If none of these conditions exist, the other two joins are considered. Since values to be joined are in sort order, both tables are scanned at the same time. The optimizer needs to perform only one scan. The shard key requirement makes the join local ensuring we only consider matches on each partition. This is why the `MergeJoin` is the most performant. > **📝 Note**: `MergeJoin` is only supported for inner joins, not for outer joins. ```sql EXPLAIN SELECT * FROM t t1, t t2 WHERE t1.id = t2.a; ``` ```output +------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------+ | Project [t1.id, t1.a, t1.b, r0.id_1, r0.a_2, r0.b_3] | | Gather partitions:all est_rows:1 | | Project [t1.id, t1.a, t1.b, r0.id_1, r0.a_2, r0.b_3] est_rows:1 est_select_cost:3 | | NestedLoopJoin | | |---IndexSeek db1.t AS t1, PRIMARY KEY (id) scan:[id = r0.a_2] est_table_rows:1 est_filtered:1 | | TableScan r0 storage:list stream:no | | Repartition [t2.id AS id_1, t2.a AS a_2, t2.b AS b_3] AS r0 shard_key:[a_2] est_rows:1 | | TableScan db1.t AS t2, PRIMARY KEY (id) est_table_rows:1 est_filtered:1 | +------------------------------------------------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM t, ct WHERE t.b = ct.b; ``` ```output +-----------------------------------------------------------------------------------------------+ | EXPLAIN | +-----------------------------------------------------------------------------------------------+ | Project [r1.id, r1.a, r1.b, ct.a AS a_1, ct.b AS b_2] | | Gather partitions:all est_rows:1 | | Project [r1.id, r1.a, r1.b, ct.a AS a_1, ct.b AS b_2] est_rows:1 est_select_cost:4 | | HashJoin [r1.b = ct.b] | | |---Broadcast [t.id, t.a, t.b] AS r1 est_rows:1 | | | TableScan db1.t, PRIMARY KEY (id) est_table_rows:1 est_filtered:1 | | ColumnStoreScan db1.ct, KEY a (a) USING CLUSTERED COLUMNSTORE est_table_rows:1 est_filtered:1 | +-----------------------------------------------------------------------------------------------+ ``` ```sql EXPLAIN SELECT * FROM ct t1, ct t2 WHERE t1.a = t2.a; ``` ```output +--------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------+ | Project [t1.a, t1.b, t2.a, t2.b] | | Gather partitions:all | | Project [t1.a, t1.b, t2.a, t2.b] | | MergeJoin condition:[t2.a = t1.a] | | |---OrderedColumnStoreScan db1.ct AS t2, KEY a (a) USING CLUSTERED COLUMNSTORE | | +---OrderedColumnStoreScan db1.ct AS t1, KEY a (a) USING CLUSTERED COLUMNSTORE | +--------------------------------------------------------------------------------+ ``` ## Handling Parameter-Dependent Query Plan Issues A parameter-dependent query plan issue can occur when the query optimizer generates a query execution plan optimized for a specific parameter value/set of values. Because parameter values can change, a cached plan is no longer optimal for parameter values that are used in consecutive executions. These types of plans can cause query performance problems. Workarounds that can reduce parameter-dependent query plan performance issues are: * Using both the `with(row_count=xxx)` and `with(selectivity=x.x)` query hints. The hints can be used with the right table after the `JOIN`: ```sql JOIN [AS alias] with(row_count=xxx, selectivity=x.x) ON ``` These hints override the statistics the query optimizer uses. `with(row_count=xxx)` treats the table as having xxx number of rows. `with(selectivity=x.x)` sets an estimate of the fraction of the table's rows that are not filtered. * Using the [NOPARAM](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/noparam.md) function. This function disables the parameterization of constants before a query plan is compiled. Please be aware when disabling parameterization, separate query plans are created for the different parameter values thereby causing an increase in compiling time and resources. * Adding comments to the query. Comments are not parameterized, so using them forces the optimizer to generate different plans. Please note, the client can remove the comments in a query, thereby preventing the effectiveness of this workaround. For example: ```sql SELECT /*1*/ 1; ``` The server could receive the following command: ```sql SELECT 1; ``` Again, please be aware of the increase in compiling time and resources needed when separate query plans are created. *** Modified at: August 7, 2026 Source: [/cloud/query-data/query-plan-operations/](https://docs.singlestore.com/cloud/query-data/query-plan-operations/) (An index of the documentation is available at /llms.txt) --- # Query Tuning ## Things to Consider Before Query Tuning Before creating your database, it's important to consider the workload that will be placed upon it. The three variables that can be optimized to increase performance are concurrency, throughput, and latency. *Concurrency* is the number of queries that are executing a on the database the same time. If your workload is primarily lightweight inserts and updates on your table concurrency will like be low. However, if a large number of those operations occur simultaneously, it can negatively impact overall performance. *Throughput* is the number of queries that are executing per unit of time. It is a function of concurrency/unit of time. If your workload is primarily lightweight inserts and updates on a table, perhaps those queries can be encapsulated within stored procedures to prevent repeated compilation. *Latency* is the query execution time. This variable can be optimized by following query performance best practices, such as by choosing the best database schema, table type, and keys for your business needs and workloads. Issues that are beyond the database engine's control can also cause latency, such as poor network connectivity. Lastly, your cluster size will determine the throughput and performance of databases in the cluster. A larger size will typically result in better query performance, lower latencies, and a higher number of concurrent queries.  You can utilize [Resource Pool Commands](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands.md) and [Query Performance, Workload Management, and Statistics](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics.md) as tools to monitor and improve cluster and query performance. ## Run ANALYZE The `ANALYZE` command collects data statistics on a table to facilitate accurate query optimization. This is especially important for optimizing complex analytical queries. You should `ANALYZE` your tables after inserting, updating, or deleting large numbers of rows (30% of your table row count is a good rule of thumb). See the [Statistics and Sampling](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling.md) topic for more information. ## Look for Hot Queries You can do query analysis on hot running queries by using [SingleStore Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) or by running the [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) command. For example, you can use a query like the following to see the select queries with the highest average execution time. Of course, you can modify the query to filter by database, look for specific types of queries, find queries using the most total time or the highest memory use, etc. ```sql SELECT Database_Name, Query_Text, Commits, Execution_Time, Execution_Time/Commits AS avg_time_ms, Average_Memory_Use FROM information_schema.plancache WHERE Query_Text LIKE 'select%' ORDER BY avg_time_ms DESC; ``` ## Impact of Hash Index on a Column having a Unique Index or Sort Key or Shard Key A unique key requires an index implicitly and is influenced by a hash index unless it is created as an unenforced unique key. The primary utility of a sort key is in segment elimination, so if you read a maximum of 1 million rows to find a single row based on a sort key matching lookup, a hash index will find the row and read 4096 rows; however, this depends on the cardinality of the column(s) in the key versus the table size. If there are a million rows in each partition that match the sort key, then the hash index is not useful. Shard keys allow you to be selective in checking which partition may own a value. Since spreading different values of a variable (x) around partitions decreases the per-partition cardinality, it has some impact but it's not a replacement for other keys. ## Check if Queries are Using Indexes One important query performance consideration is adding appropriate [indexes](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md) for your queries. You can use [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) to see whether queries are using indexes. The following examples show how to identify cases where an index can greatly improve query performance. ## Indexes for Filters Consider the following rowstore table: ```sql CREATE ROWSTORE TABLE qtune_1 (a INT, b INT); ``` Suppose we are running queries like: ```sql SELECT * FROM qtune_1 WHERE a = 3; ``` `EXPLAIN` shows that running the query against the current table schema requires a full `TableScan` - scanning all the rows of `t`, which is unnecessarily expensive if a small fraction of the values in `t` equal 3. ```sql EXPLAIN SELECT * FROM qtune_1 WHERE a = 3; ``` ```output +------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:partition | | Project [qtune_1.a, qtune_1.b] | | Filter [qtune_1.a = 3] | | TableScan test1.qtune_1 table_type:sharded_rowstore | +------------------------------------------------------------------+ ``` If an index is added, the query will instead use an `IndexRangeScan` on the key `a`: ```sql ALTER TABLE qtune_1 ADD INDEX (a); EXPLAIN SELECT * FROM qtune_1 WHERE a = 3; ``` ```output +----------------------------------------------------------------------------------+ | EXPLAIN | +----------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:partition | | Project [qtune_1.a, qtune_1.b] | | IndexRangeScan test1.qtune_1, KEY a (a) scan:[a = 3] table_type:sharded_rowstore | +----------------------------------------------------------------------------------+ ``` A query that filters on both `a` and `b` is unable to take advantage of the filtering on `b` to reduce the rows that need to be scanned. As demonstrated below, the scan uses `a = 3` only. ```sql EXPLAIN SELECT * FROM qtune_1 WHERE a = 3 AND b = 4; ``` ```output +----------------------------------------------------------------------------------+ | EXPLAIN | +----------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:partition | | Project [qtune_1.a, qtune_1.b] | | Filter [qtune_1.b = 4] | | IndexRangeScan test1.qtune_1, KEY a (a) scan:[a = 3] table_type:sharded_rowstore | +----------------------------------------------------------------------------------+ ``` Adding an index on `b` allows the query to scan more selectively: ```sql ALTER TABLE qtune_1 ADD INDEX (b); EXPLAIN SELECT * FROM qtune_1 WHERE a = 3 AND b = 4; ``` ```output +-------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:partition | | Project [qtune_1.a, qtune_1.b] | | IndexRangeScan test1.qtune_1, KEY a_2 (a, b) scan:[a = 3 AND b = 4] table_type:sharded_rowstore | +-------------------------------------------------------------------------------------------------+ ``` For columnstore tables, it's a best practice to set a sort key the table to take advantage of segment elimination. Segment elimination occurs when filtering on the sort key. The minimum/maximum value metadata for each segment is used at query execution time to determine whether a segment can match a filter. If not, the segment is skipped entirely and no data is examined. Consider a columnstore table with an explicit sort key: ```sql CREATE TABLE qtune_2 (a int, b int, SORT KEY(a)); ``` Suppose the following queries are run: ```sql SELECT * FROM qtune_2 WHERE a = 3; ``` `EXPLAIN` shows that running the query against this table schema requires a `ColumnStoreScan` - scanning the segments (column `a`) which contain values equal to `3`. All segments not containing the value equal to `3` are not scanned. Furthermore, since segments are ordered by min and max values, the engine can easily locate the segment that contains `3`: ```sql EXPLAIN SELECT * FROM qtune_2 WHERE a = 3; ``` ```output +------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------+ | Gather partitions:all alias:remote_0 parallelism_level:segment | | Project [qtune_2.a, qtune_2.b] | | ColumnStoreFilter [qtune_2.a = 3] | | ColumnStoreScan test1.qtune_2, SORT KEY a (a) table_type:sharded_columnstore | +------------------------------------------------------------------------------+ ``` ## Indexes for `GROUP BY` and `ORDER BY` Another class of cases where indexes can improve query performance is `GROUP BY` and `ORDER BY` . Consider this rowstore table: ```sql CREATE ROWSTORE TABLE qtune_3 (a int, b int); ``` Executing the following query requires a `HashGroupBy`. SingleStore builds a hash table with an entry for each group of `a`: ```sql EXPLAIN SELECT a, SUM(b) FROM qtune_3 GROUP BY a; ``` ```output +-------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------+ | Project [remote_0.a, `SUM(b)`] est_rows:1 | | HashGroupBy [SUM(remote_0.`SUM(b)`) AS `SUM(b)`] groups:[remote_0.a] | | Gather partitions:all est_rows:1 alias:remote_0 parallelism_level:partition | | Project [qtune_3.a, `SUM(b)`] est_rows:1 | | HashGroupBy [SUM(qtune_3.b) AS `SUM(b)`] groups:[qtune_3.a] | | TableScan test1.qtune_3 table_type:sharded_rowstore est_table_rows:1 est_filtered:1 | +-------------------------------------------------------------------------------------+ ``` However, with an index on `a`, SingleStore can execute the query with a `StreamingGroupBy` operation because by scanning the index on `a`, it can process all elements of a group consecutively. ```sql ALTER TABLE qtune_3 ADD INDEX(a); EXPLAIN SELECT a, SUM(b) FROM qtune_3 GROUP BY a; ``` ```output +------------------------------------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------------------------------------+ | Project [remote_0.a] est_rows:1 | | HashGroupBy [] groups:[remote_0.a] | | Gather partitions:all est_rows:1 alias:remote_0 parallelism_level:partition | | Project [qtune_3.a] est_rows:1 | | StreamingGroupBy [] groups:[qtune_3.a] | | TableScan test1.qtune_3, KEY a (a) table_type:sharded_rowstore est_table_rows:1 est_filtered:1 | +------------------------------------------------------------------------------------------------+ ``` For a columnstore table, the column(s) in the `GROUP BY` clause must match the `SORT KEY` for a `StreamingGroupBy` to be considered. An `ORDER BY` on a rowstore table without an index, SingleStore needs to sort: ```sql CREATE ROWSTORE TABLE qtune_4 (a INT, b INT); ``` ```sql EXPLAIN SELECT * FROM qtune_4 ORDER BY b; ``` ```output +------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------+ | Project [remote_0.a, remote_0.b] | | Sort [remote_0.b] | | Gather partitions:all alias:remote_0 parallelism_level:partition | | Project [qtune_4.a, qtune_4.b] | | Sort [qtune_4.b] | | TableScan test1.qtune_4 table_type:sharded_rowstore | +------------------------------------------------------------------+ ``` With an index, SingleStore can eliminate the need to sort: ```sql ALTER TABLE qtune_4 ADD INDEX (b); EXPLAIN SELECT * FROM qtune_4 ORDER BY b; ``` ```output +----------------------------------------+ | EXPLAIN | +----------------------------------------+ | Project [qtune_4.a, qtune_4.b] | | GatherMerge [qtune_4.b] partitions:all | | Project [qtune_4.a, qtune_4.b] | | TableScan test1.qtune_4, KEY b (b) | +----------------------------------------+ ``` As mentioned above, it's a best practice to set a sort key on columnstore tables to take advantage of segment elimination on queries with `GROUP BY` and `ORDER BY` clauses. ## Fanout vs Single-Partition Queries SingleStore’s distributed architecture takes advantage of CPUs from many servers to execute your queries. This provide for extremely fast performance on aggregation queries that scan millions of rows. However, for transactional queries that select relatively few rows. It is best for each query to only involve a single partition. When a query has equality filters that completely match the shard key of the table, SingleStore can optimize it to only require execution on a single partition. Using this table as an example: ```sql CREATE TABLE urls ( domain varchar(128), path varchar(8192), time_sec int, status_code binary(3), ... shard key (domain, path, time_sec) ); ``` The following query only involves a single partition because it has equality filters on all columns of the shard key, so the rows which match can only be on a single partition. Using `EXPLAIN`, `Gather partitions:single` indicates SingleStore is using a single-partition plan. ```sql EXPLAIN SELECT status_code FROM urls WHERE domain = 'youtube.com' AND path = '/watch?v=euh_uqxwk58' AND time_sec = 1; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------------------------------------------------+ | Gather partitions:single | | Project [urls.status_code] | | IndexRangeScan test2.urls, SHARD KEY domain (domain, path, time_sec) scan:[domain = "youtube.com" AND path = "/watch?v=euh_uqxwk58" AND time_sec = 1] | +-------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The following query which does not filter on `time_sec`, does not match a single partition. Therefore, the query requires selecting from all partitions, as indicated by `Gather partitions:all`. ```sql EXPLAIN SELECT status_code FROM urls WHERE domain = 'youtube.com'AND path = '/watch?v=euh_uqxwk58'; ``` ```output +--------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------------------+ | Project [urls.status_code] | | Gather partitions:all | | Project [urls.status_code] | | IndexRangeScan test2.urls, SHARD KEY domain (domain, path, time_sec) scan:[domain = "youtube.com" AND path = "/watch?v=euh_uqxwk58"] | +-------------------------------------------------------------------------------------------------------------------------------------- ``` To fix this, the table `urls` could be sharded on `domain`. This would make it easier to write queries that route to a single partition. However, some domains will have more pages than other domains and this could lead to data skew. Choosing a shard key is often a balancing act: we want the least restrictive shard key possible while also ensuring that we do not have data skew. A good compromise in this case would be to shard on `(domain,path)`. For more on choosing a performant shard key, see the related step in the [Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md) guide. If these types of performance issues are common, SingleStore recommends reviewing your table data structures overall using the full guide. ## Distributed Joins SingleStore’s query execution architecture allows you to run arbitrary SQL queries on any table regardless of data distribution. However, you can often improve performance by optimizing your schema to minimize data movement during query execution. ## Collocating Joins Consider the following tables: ```sql CREATE TABLE lineitem( l_orderkey INT NOT NULL, l_linenumber INT NOT NULL, ... PRIMARY KEY(l_orderkey, l_linenumber) ); ``` ```sql CREATE TABLE orders( o_orderkey INT NOT NULL, ... PRIMARY KEY(o_orderkey) ); ``` When `lineitem` and `orders` are joined with the current schema, a distributed join is performed and data is repartitioned from the `lineitem` table. ```sql EXPLAIN SELECT COUNT(*) FROM lineitem JOIN orders ON o_orderkey = l_orderkey; ``` ```output +---------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +---------------------------------------------------------------------------------------------------------------------------------+ | Project [`COUNT(*)`] | | Aggregate [SUM(`COUNT(*)`) AS `COUNT(*)`] | | Gather partitions:all est_rows:1 | | Project [`COUNT(*)`] est_rows:1 est_select_cost:1762812 | | Aggregate [COUNT(*) AS `COUNT(*)`] | | NestedLoopJoin | | |---IndexSeek test.orders, PRIMARY KEY (o_orderkey) scan:[o_orderkey = r0.l_orderkey] est_table_rows:565020 est_filtered:565020 | | TableScan r0 storage:list stream:no | | Repartition [lineitem.l_orderkey] AS r0 shard_key:[l_orderkey] est_rows:587604 | | TableScan test.lineitem, PRIMARY KEY (l_orderkey, l_linenumber) est_table_rows:587604 est_filtered:587604 | +---------------------------------------------------------------------------------------------------------------------------------+ ``` The performance of this query can be improved by adding an explicit shard key to the `lineitem`table on `l_orderkey`. Now we can perform a local join between `lineitem` and `orders`. ``` +--------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------+ | Project [`COUNT(*)`] | | Aggregate [SUM(`COUNT(*)`) AS `COUNT(*)`] | | Gather partitions:all | | Project [`COUNT(*)`] | | Aggregate [COUNT(*) AS `COUNT(*)`] | | ChoosePlan | | | :estimate | | | SELECT COUNT(*) AS cost FROM test.lineitem | | | SELECT COUNT(*) AS cost FROM test.orders | | |---NestedLoopJoin | | | |---IndexSeek test.orders, PRIMARY KEY (o_orderkey) scan:[o_orderkey = lineitem.l_orderkey] | | | TableScan test.lineitem, PRIMARY KEY (l_orderkey, l_linenumber) | | +---NestedLoopJoin | | |---IndexRangeScan test.lineitem, PRIMARY KEY (l_orderkey, l_linenumber) scan:[l_orderkey = orders.o_orderkey] | | TableScan test.orders, PRIMARY KEY (o_orderkey) | +--------------------------------------------------------------------------------------------------------------------+ ``` ## Reference Table Joins Reference tables are replicated to each aggregator and leaf in the cluster, therefore their use case is best for data that is small and slowly changing. The rate at which this data changes depends on the application, but a rule of thumb is 1 million rows updated once a day. Consider the following schema: ```sql CREATE TABLE customer( c_custkey INT NOT NULL, c_nationkey INT NOT NULL, ... PRIMARY KEY(c_custkey), key(c_nationkey) ); ``` ```sql CREATE TABLE nation( n_nationkey INT NOT NULL, ... PRIMARY KEY(n_nationkey) ); ``` With the current schema, when we join the `customer` and `nation` tables together on `nationkey`, the `nation` table must be broadcast each time the query is run. ```sql EXPLAIN SELECT COUNT(*) FROM customer JOIN nation ON n_nationkey = c_nationkey; ``` ```output +-------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------------------------------------------+ | Project [`Count(*)`] | | Aggregate [SUM(`Count(*)`) AS `Count(*)`] | | Gather partitions:all est_rows:1 | | Project [`Count(*)`] est_rows:1 est_select_cost:1860408 | | Aggregate [COUNT(*) AS `Count(*)`] | | NestedLoopJoin | | |---IndexRangeScan test.customer, KEY c_nationkey (c_nationkey) scan:[c_nationkey = r1.n_nationkey] est_table_rows:1856808 est_filtered:1856808 | | TableScan r1 storage:list stream:no | | Broadcast [nation.n_nationkey] AS r1 est_rows:300 | | TableScan test.nation, PRIMARY KEY (n_nationkey) est_table_rows:300 est_filtered:300 | +-------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The broadcast can be avoided by making `nation` a [reference](https://docs.singlestore.com/cloud/create-a-database/other-schema-concepts.md) table as it is relatively small and changes rarely. While broadcasting such a small table will likely have a negligible effect on single-query latency, repeatedly doing so can have an outsize effect on concurrent workloads. ```sql CREATE REFERENCE TABLE nation( n_nationkey INT NOT NULL, ... PRIMARY KEY(n_nationkey) ); ``` ```sql EXPLAIN SELECT COUNT(*) FROM customer JOIN nation ON n_nationkey = c_nationkey; ``` ```output +-------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +-------------------------------------------------------------------------------------------------------------+ | Project [`Count(*)`] | | Aggregate [SUM(`Count(*)`) AS `Count(*)`] | | Gather partitions:all | | Project [`Count(*)`] | | Aggregate [COUNT(*) AS `Count(*)`] | | ChoosePlan | | | :estimate | | | SELECT COUNT(*) AS cost FROM test.customer | | | SELECT COUNT(*) AS cost FROM test.nation | | |---NestedLoopJoin | | | |---IndexSeek test.nation, PRIMARY KEY (n_nationkey) scan:[n_nationkey = customer.c_nationkey] | | | TableScan test.customer, PRIMARY KEY (c_custkey) | | +---NestedLoopJoin | | |---IndexRangeScan test.customer, KEY c_nationkey (c_nationkey) scan:[c_nationkey = nation.n_nationkey] | | TableScan test.nation, PRIMARY KEY (n_nationkey) | +-------------------------------------------------------------------------------------------------------------+ ``` ## Joins on the Aggregator Consider the following schema and row counts: ```sql CREATE TABLE customer( c_custkey INT NOT NULL, c_acctbal DECIMAL(15,2) NOT NULL, PRIMARY KEY(c_custkey) ); ``` ```sql CREATE TABLE orders( o_orderkey INT NOT NULL, o_custkey INT NOT NULL, o_orderstatus varchar(20) NOT NULL, PRIMARY KEY(o_orderkey), key(o_custkey) ); ``` ``` SELECT COUNT(*) FROM orders; ``` ```output +----------+ | COUNT(*) | +----------+ | 429786 | +----------+ ``` ```sql SELECT COUNT(*) FROM orders WHERE o_orderstatus = 'open'; ``` ```output +----------+ | COUNT(*) | +----------+ | 1000 | +----------+ ``` ```sql SELECT COUNT(*) FROM orders WHERE o_orderstatus = 'open'; ``` ```output +----------+ | COUNT(*) | +----------+ | 1000 | +----------+ ``` ```sql SELECT COUNT(*) FROM customer; ``` ```output +----------+ | COUNT(*) | +----------+ | 1014726 | +----------+ ``` ```sql SELECT COUNT(*) FROM customer WHERE c_acctbal > 100.0; ``` ```output +----------+ | COUNT(*) | +----------+ | 988 | +----------+ ``` Note that while `customer` and `orders` are fairly large, when a query is filtered on open orders and account balances greater than 100, relatively few rows match. As a result, when `orders` and `customer` are joined using these filters, the join can be performed on an aggregator. This is shown by `EXPLAIN` as having a separate `Gather` operator for `orders` and `customer` and a `HashJoin` operator above the `Gather` . ```sql EXPLAIN SELECT o_orderkey FROM customer JOIN orders WHERE c_acctbal > 100.0 AND o_orderstatus = 'open' AND o_custkey = c_custkey; ``` ```output +------------------------------------------------------------------+ | EXPLAIN | +------------------------------------------------------------------+ | Project [orders.o_orderkey] | | HashJoin [orders.o_custkey = customer.c_custkey] | | |---TempTable | | | Gather partitions:all | | | Project [orders_0.o_orderkey, orders_0.o_custkey] | | | Filter [orders_0.o_orderstatus = "open"] | | | TableScan test3.orders AS orders_0, PRIMARY KEY (o_orderkey) | | TableScan 0tmp AS customer storage:list stream:yes | | TempTable | | Gather partitions:all | | Project [customer_0.c_custkey] | | Filter [customer_0.c_acctbal > 100.0] | | TableScan test3.customer AS customer_0, PRIMARY KEY (c_custkey) | +------------------------------------------------------------------+ ``` By default, SingleStore will perform this optimization when each `Gather` pulls less than 120,000 rows from the leaves. This threshold can be changed via the `max_subselect_aggregator_rowcount` variable. The optimization on this query can also be disabled manually via the `leaf_pushdown` hint. The `leaf_pushdown` hint forces the optimizer to perform as much work as possible on the leaf nodes. ```sql EXPLAIN SELECT WITH(LEAF_PUSHDOWN=TRUE) o_orderkey FROM customer JOIN orders WHERE c_acctbal > 100.0 AND o_orderstatus = 'open' AND o_custkey = c_custkey; ``` ```output +--------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------------+ | Project [r0.o_orderkey] | | Gather partitions:all est_rows:11 | | Project [r0.o_orderkey] est_rows:11 est_select_cost:1955 | | Filter [customer.c_acctbal > 100.0] | | NestedLoopJoin | | |---IndexSeek test3.customer, PRIMARY KEY (c_custkey) scan:[c_custkey = r0.o_custkey] est_table_rows:1013436 est_filtered:1092 | | TableScan r0 storage:list stream:no | | Repartition [orders.o_orderkey, orders.o_custkey] AS r0 shard_key:[o_custkey] est_rows:972 | | Filter [orders.o_orderstatus = "open"] | | TableScan test3.orders, PRIMARY KEY (o_orderkey) est_table_rows:92628 est_filtered:972 | +--------------------------------------------------------------------------------------------------------------------------------+ ``` ```sql SET max_subselect_aggregator_rowcount=500; ``` ```sql EXPLAIN SELECT o_orderkey FROM customer JOIN orders WHERE c_acctbal > 100.0 AND o_orderstatus = 'open' AND o_custkey = c_custkey; ``` ```output +--------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------------+ | Project [r0.o_orderkey] | | Gather partitions:all est_rows:11 | | Project [r0.o_orderkey] est_rows:11 est_select_cost:1955 | | Filter [customer.c_acctbal > 100.0] | | NestedLoopJoin | | |---IndexSeek test3.customer, PRIMARY KEY (c_custkey) scan:[c_custkey = r0.o_custkey] est_table_rows:1013436 est_filtered:1092 | | TableScan r0 storage:list stream:no | | Repartition [orders.o_orderkey, orders.o_custkey] AS r0 shard_key:[o_custkey] est_rows:972 | | Filter [orders.o_orderstatus = "open"] | | TableScan test3.orders, PRIMARY KEY (o_orderkey) est_table_rows:92628 est_filtered:972 | +--------------------------------------------------------------------------------------------------------------------------------+ ``` ## Feedback Reoptimization This feature allows you to auto-tune a query’s performance issues caused by a poor plan created using inaccurate estimates. It reoptimizes the query by feeding in the statistics collected from the query's first run back into the optimizer. It helps resolve performance issues without adding any hints or query rewrites. The Feedback Reoptimization process is as follows. Run the respective commands in sequence to auto-tune a query: 1. Profile a query. ``` PROFILE ; ``` 2. Run the reoptimization command which recompiles the query with the stats from the latest profile. ``` REOPTIMIZE; ``` 3. Save the query so that it replaces the existing plan in the plan cache. ``` REOPTIMIZE COMMIT; ``` A query needs a profile before it can be reoptimized. A profile collects statistics such as join sizes and filter selectivity, which help to make decisions around join order, data movement, and the hash build. The `REOPTIMIZE` command causes the optimizer to run again for the profiled query, using the runtime information instead of the estimated value. You can run the `SHOW PROFILE JSON` command after the `REOPTIMIZE` command to view the latest profile information. To use the new plan, run the `REOPTIMIZE COMMIT` command which creates a new plan entry for the original query text. If plan pinning is enabled, the command will also pin the reoptimized plan to the original query text. A query can be reoptimized many times. Each time `REOPTIMIZE` is run, a new plan is generated, and the system learns more about how the plans perform and can make more informed decisions for future plans. To reoptimize a query again, re-run the `REOPTIMIZE` command . To run a specific reoptimized plan, add the reoptimize level to the command, e.g. `REOPTIMIZE 1` will run the first reoptimized plan generated and `REOPTIMIZE 2` will run the second reoptimized plan generated. Once you have generated a plan you want to use for your original query, you can commit that plan via `REOPTIMIZE COMMIT ;`. After doing so, the next time you run your original query, the system will use the reoptimized plan you committed. ## Related Topics * Training: [Query Tuning](https://training.singlestore.com/learn/course/internal/view/elearning/634/query-tuning) * Training: [Performance Benchmarking](https://training.singlestore.com/learn/course/internal/view/elearning/618/performance-benchmarking) ## In this section * [Feedback Reoptimization](https://docs.singlestore.com/cloud/query-data/query-tuning/feedback-reoptimization.md) * [Join Memory Reduction Optimization](https://docs.singlestore.com/cloud/query-data/query-tuning/join-memory-reduction-optimization.md) * [Query Performance Tools](https://docs.singlestore.com/cloud/query-data/query-tuning/query-performance-tools.md) * [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) * [SingleStore Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) * [Statistics and Sampling](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling.md) * [Testing Your Queries and Performance](https://docs.singlestore.com/cloud/query-data/query-tuning/testing-your-queries-and-performance.md) * [Troubleshooting Poorly Performing Queries](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md) * [Workload Profiling](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling.md) *** Modified at: July 18, 2025 Source: [/cloud/query-data/query-tuning/](https://docs.singlestore.com/cloud/query-data/query-tuning/) (An index of the documentation is available at /llms.txt) --- # Feedback Reoptimization Feedback Reoptimization (FR) is an automatic mechanism that optimizes query plans based on execution statistics rather than on optimizer estimates. FR is designed for situations where optimizer estimates based on [automatically gathered statistics](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-concepts.md) are not sufficient to create an optimal plan and execution statistics are required to create a better plan. FR proactively detects suboptimal plans and triggers reoptimization, while also providing a precise, auditable way for users to explicitly mark a plan for reoptimization. FR ensures safety under concurrency, provides concrete means for user control, and surfaces comprehensive runtime and historical feedback for observability. FR learns from history and over time is expected to improve query plans. However, FR does not guarantee a better plan in a single FR run. FR works iteratively and learns from both good and bad plans. It accumulates learning on successive query executions, and gradually converges on a plan. Allow FR to iterate until it converges. In certain situations, FR may not be able to produce improved plans. For example, FR cannot address parameter sensitivity. For such scenarios, consider disabling FR. FR uses the following concepts. * **Feedback Reoptimization (FR)**: An automatic mechanism that optimizes query plans based on execution statistics rather than on optimizer estimates. * **Automatic FR Marking:** A sub-feature of FR. The engine automatically detects suboptimal plans and marks such plans for FR. * **Explicit FR Marking:** Manually mark a plan for FR using the [REOPTIMIZE MARK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize-mark.md) command. * **Explicit Reoptimization:** Manually reoptimize a plan by running the [REOPTIMIZE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize.md) command. ## Enable Feedback Reoptimization and Automatic Marking Enable FR and automatic FR marking with the following commands. ```sql SET SESSION enable_auto_profile = ON; SET SESSION auto_profile_type = SMART; SET SESSION enable_automatic_feedback_reoptimization = ON; SET SESSION optimizer_feedback_reoptimization_auto_marking = AUTO; ``` Setting `enable_auto_profile` to `ON` and `auto_profile_type` to `SMART` enables the profiling and collection of execution statistics for FR. Setting `enable_automatic_feedback_reoptimization` to `ON` enables feedback reoptimization itself. Setting `optimizer_feedback_reoptimization_auto_marking` to `ON` specifies that plans will be automatically marked at the end of the execution if they have the potential to be improved by FR. FR occurs at the next user execution for marked plans. Setting this variable to `AUTO` enables automatic marking, but FR will not occur on automatically marked plans. Disable FR and automatic FR marking with the following commands. ```sql SET SESSION enable_automatic_feedback_reoptimization = OFF; SET SESSION optimizer_feedback_reoptimization_auto_marking = OFF; ``` ## Feedback Reoptimization Process FR occurs: * Automatically when automatic FR marking is enabled. * Manually when plans are manually marked for reoptimization. When automatic FR marking is enabled, after each query execution, FR analyzes runtime statistics and compares estimated versus actual statistics for row counts, CPU usage, memory, and network usage. FR uses metrics including: amount of misestimation, query execution time, existence of a compiled plan, and performance of previous FR plans to determine if a plan qualifies for FR. Qualified plans are marked for FR. Alternatively, a user can manually mark a plan for FR. ## Plan Replacement and Policy Once a plan is marked for FR, plan replacement works as follows. * FR generates a new plan, collects metrics for the new plan (CPU time, memory, network usage), and then compares the metrics for the new plan with the metrics for the original plan. * If the new plan improves over the old plan based on an evaluation of CPU time, memory, and network usage, FR adopts the new plan; if not, FR drops the new plan and retains the original plan. ## Convergence FR works iteratively, improving plans with each successive FR attempt. To avoid infinite cycling, total FR attempts per plan are capped. Ongoing or new queries use the currently valid plan instance, and are unaffected by in-process FR. ## Automatic Marking Requirements Automatic marking occurs under the following conditions. * FR automatically marks queries that run using auto-profiling. FR does not automatically mark queries when the explicit `PROFILE` command is used. * FR requires consistent CPU measurements to validate whether the corrected plan improves performance. Thus FR waits until the second execution of the query, when the query plan is fully compiled, to consider whether to mark the query. * FR has a CPU time threshold (500ms) for automatic marking. Queries that fall below this threshold are not automatically marked. The following are required for automatic marking to occur. **Engine Variables** The engine variables must be set as follows. ```sql SET SESSION enable_auto_profile = ON; SET SESSION auto_profile_type = SMART; SET SESSION optimizer_feedback_reoptimization_auto_marking = ON; (or AUTO) ``` **Execution** * The most recent execution must use a fully-compiled plan. * The most recent execution must use auto-profiling; it must not use an explicit `PROFILE SELECT...` command. FR uses heuristics based on the number and size of misestimations and CPU time to determine if a plan qualifies for automatic marking. ## Monitor Feedback Reoptimization Monitor FR systemwide using the [SHOW FEEDBACK REOPTIMIZATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-feedback-reoptimization-status.md) command or per-query by viewing `OPTIMIZER_NOTES` from the [PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/plancache.md). ## Systemwide Monitoring The following command displays system-wide FR information. ```sql SHOW FEEDBACK REOPTIMIZATION STATUS; ``` The output displays up-to-date atomic counters, including: * The number of feedback-reoptimized queries currently running. * The number of plans marked for FR. * The total number of plans in the plancache. * Configured threshold values. * System-level stats from startup including total automatic and explicit feedback reoptimizations and number of skips of FR. ## Per-Query Monitoring The `OPTIMIZER_NOTES` in the [PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/plancache.md) view provide information about FR including: * If, when, and how many times a plan was feedback reoptimized. * If a plan was feedback reoptimized, which metrics improved with that reoptimization. * Historical observed improvement and variance per feedback-reoptimization step. * Largest estimation mismatches to support diagnosis and audit. Run this command to see the `OPTIMIZER_NOTES`. ```sql SELECT OPTIMIZER_NOTES FROM information_schema.PLANCACHE WHERE ...; ``` ## Manually Mark a Plan for Feedback Reoptimization Manually mark a plan that has been profiled for reoptimization using the plan's `plan_id`. Run the following series of commands to manually mark a plan for FR. * Run [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) to list plans and obtain the `ID` of the plan to be reoptimized. ```sql SHOW PLANCACHE; ``` * Run [REOPTIMIZE MARK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize-mark.md) to mark the plan for FR. ```sql REOPTIMIZE MARK ; ``` After a plan has been marked, the next execution of the query triggers FR. ## Examples ## Example 1: Automatic Marking - FR Flow FR happens automatically without user intervention. For automatic FR, the flow is as follows. 1. The user runs a complex query.  2. The system automatically profiles and detects a significant mismatch between the estimates used to create the plan and the actual runtime statistics. 3. The plan is marked for FR. 4. On the next execution of the plan, FR runs automatically; the statistics are compared, FR swaps in the new plan if it detects significant improvement in at least one metric, and other metrics do not worsen significantly. 5. Run [SHOW FEEDBACK REOPTIMIZATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-feedback-reoptimization-status.md) or use [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) to obtain diagnostic detail. ## Example 2: Manual Marking - FR Flow Manually trigger FR using the following steps. 1. Run [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) to identify target `plan_id`. 2. Run [REOPTIMIZE MARK \](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize-mark.md) to mark the plan for FR. 3. On the next execution of the plan, FR runs automatically; the statistics are compared, and the FR plan is swapped in if significant improvement is detected in at least one metric, and other metrics do not worsen significantly. 4. Run [SHOW FEEDBACK REOPTIMIZATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-feedback-reoptimization-status.md) or use [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) to obtain diagnostic detail. *** Modified at: May 29, 2026 Source: [/cloud/query-data/query-tuning/feedback-reoptimization/](https://docs.singlestore.com/cloud/query-data/query-tuning/feedback-reoptimization/) (An index of the documentation is available at /llms.txt) --- # Join Memory Reduction Optimization The join memory reduction optimization is a new query optimization feature designed to help prevent memory overflow (spilling) during hash joins on large datasets. When enabled, it automatically rewrites qualifying joins into smaller, manageable pieces—reducing memory pressure and the risk of out-of-memory (OOM) errors. This feature is especially useful when joining large tables on high-cardinality columns, where traditional hash joins are likely to exceed memory limits. The join memory reduction optimization is built into the query optimizer. Queries with a hash join on a column with a cardinality exceeding the specified threshold are split into several smaller join queries. The results of the smaller join queries are recombined using `UNION ALL`. > **📝 Note**: The join memory reduction optimization introduces overhead, which can negatively impact performance. SingleStore recommends using the join memory reduction optimization only when necessary, for example, to prevent out-of-memory (OOM) errors. ## Configure Join Memory Reduction Optimization The join memory reduction optimization splits large hash joins into smaller joins to avoid memory overflow. This optimization is configured by specifying: * The cardinality threshold at which the optimization will be triggered. * The size of the smaller joins (groups) into which the larger hash join should be split. The two engine variables used to configure these two values are: | `cardinality_threshold_for_join_split_rewrite` | Specifies the cardinality threshold for the join memory reduction optimization. If all relevant join columns have a cardinality above this threshold, the`JOIN`query will be split into smaller join queries.A value of`UINT_MAX`(the default) indicates that the optimization is disabled. | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `target_group_size_for_join_split_rewrite` | Specifies the maximum size of the input (in rows) to the smaller join queries, also called the group size. | To enable the join memory reduction optimization, set the `cardinality_threshold_for_join_split_rewrite` engine variable to a value other than `UINT_MAX`. To disable the join memory reduction optimization, set the `cardinality_threshold_for_join_split_rewrite` engine variable to`UINT_MAX`(`4294967295`). ## Select the Engine Variable Values ## `cardinality_threshold_for_join_split_rewrite` * When both join columns have cardinality greater than `cardinality_threshold_for_join_split_rewrite`, the hash join will be rewritten into smaller joins. * Choose the value of this variable based on the cardinality threshold beyond which the original query is expected to encounter OOM errors. * Proper selection ensures that the optimization is only triggered when truly needed, preventing unnecessary performance degradation. Use the following command to set this variable's value. ```sql SET SESSION cardinality_threshold_for_join_split_rewrite = ``` ## `target_group_size_for_join_split_rewrite` * The goal for the value of this variable is to strike a balance between minimizing the number of splits and ensuring that each split can be efficiently processed without memory issues. * SingleStore recommends using the following formula to choose a value for `target_group_size_for_join_split_rewrite`. * `target_group_size_for_join_split_rewrite` = cardinality of the joined column with the lowest cardinality / `N` * `N` is the number of splits * Choose `N` to be as small as possible (e.g., 2) while ensuring that the `target_group_size_for_join_split_rewrite` remains manageable within the available memory. * The `target_group_size_for_join_split_rewrite` must be set to a value above 10,000,000 to avoid unnecessary overhead. Use the following command to set this variable's value. ```sql SET SESSION target_group_size_for_join_split_rewrite = ``` ## Remarks The following apply to the join memory reduction optimization: * The tables must be columnstore tables. * The tables must have column statistics available. * A histogram on the high cardinality column must exist. * The join condition on the high cardinality column must be an equality condition on two columns, for example: `t1.a = t2.b`. * The join condition on the high cardinality column cannot include an inequality. For example, if the join condition is `t1.a < t2.b` , the join will not be rewritten. * The join condition on the high cardinality column cannot include a constant value or expressions. For example, if the join condition is `t1.a = 1`,  the join will not be rewritten. * The joined columns must be of numeric types (integer or double). ## Example Consider two tables, `t1` and `t2`, both with an `ID` column of high cardinality. Assume the cardinalities of the ID columns are as follows: * `t1.ID` has distinct values (1-20M) thus has cardinality 20,000,000 * `t2.ID` has distinct values (1-40M) thus has cardinality 40,000,000 The following query joins tables `t1` and `t2` on the high-cardinality `ID` columns. This join is a candidate for the join memory reduction optimization. ```sql SELECT * FROM t2 JOIN t1 on t2.ID = t1.ID; ``` To enable join memory reduction optimization for this query, use the following commands. ```sql SET SESSION cardinality_threshold_for_join_split_rewrite = 12000000; SET SESSION target_group_size_for_join_split_rewrite = 10000000; ``` Setting `cardinality_threshold_for_join_split_rewrite = 12000000` indicates that if a join column has a cardinality greater than `12,000,000`, the original query should be rewritten into smaller joins. Both `t1.ID` and `t2.ID` have a cardinality greater than `12,000,000`, so the engine will apply the join memory reduction optimization to this query. Note that both columns have cardinality greater than `12,000,000`, so the threshold is met. Setting `target_group_size_for_join_split_rewrite = 10000000` enables the join memory reduction optimization and specifies the size of the inputs to the smaller joins (also called the groups). With these settings, the previous query will be rewritten as follows: ```sql (SELECT * FROM t2 JOIN t1 ON t2.ID = t1.ID AND t1.ID<10000000 ) UNION ALL (SELECT * FROM t2 JOIN t1 ON t2.ID = t1.ID AND t1.ID >= 1000000 ); ``` The single join is rewritten into two smaller joins which are combined with a `UNION ALL`. The smaller joins are created by adding a filter on the `t1.ID` column. The `t1.ID` column has 20,000,000 distinct values (a cardinality of 20,000,000). This column is used to break the query into two groups of size `target_group_size_for_join_split_rewrite` using the filters `t1.ID < 10000000` and `t1.ID >= 10000001`. The column with the smaller cardinality, `t1.ID` in this example, is used to create the groups. *** Modified at: March 19, 2026 Source: [/cloud/query-data/query-tuning/join-memory-reduction-optimization/](https://docs.singlestore.com/cloud/query-data/query-tuning/join-memory-reduction-optimization/) (An index of the documentation is available at /llms.txt) --- # Query History ## Overview SingleStore supports high-frequency event-tracing of DML-related activity. As of SingleStore 9.0, this support extends to DDL-related activity. This feature offers a robust infrastructure and method for tracing events that are expected to occur frequently. Query event tracing is designed to help identify which queries take the longest or that fail so they can be isolated and optimized. This feature is at the core of [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history/query-history-examples.md) which displays these query trace events over time. For query execution reporting, the `Query_completion` trace monitors query execution details and can be viewed in near real-time. Query event traces are configurable, where a query execution may only be traced if it takes longer than, for example, 1,000 milliseconds (1 second) to run, where the query text may be set to either be included or omitted. | **Event Type** | `Query_completion` | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Engine Variable** | `trace_events_queue_size`Set the value of this engine variable to`0`to turn off query event tracing | | **Parameter** | **Value(s)** | | **Capture query text?** | `Query_text = on``Query_text = off` | | **Record queries**that take*x*amount of time to complete, in milliseconds | `Duration_threshold_ms`=*x*While`Duration_threshold_ms`can be set to any millisecond value, setting its value to`0`will negatively impact the usefulness of this feature and impair thecluster'sperformance. | | **Trace all queries that have failed** | `Trace_all_failed_executions= on``Trace_all_failed_executions= off` | ## Access Query History in the Cloud Portal In the Cloud portal, open your cluster and navigate to the **Query History** page from the monitoring section of the cluster. This page shows query completion events over time and allows you to filter by user, time range, and other attributes. ## Enhanced Query Tracing [Enhanced query tracing](https://docs.singlestore.com/#section-idm234834767845771.md) allows users to trace the high-level execution metrics collected by [MV\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities.md) and [MV\_FINISHED\_TASKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-finished-tasks.md) along with `Query_completion` traces. Enhanced query tracing is provided as a configuration option for `Query_completion` traces. ## Prerequisites * A cluster running SingleStore version 8.5 or later for Query History. Version 8.9 or later is required for enhanced query tracing. * Version 9.0 or later is required for capturing DDL activity in Query History and `MV_TRACE_EVENTS`. * Access to Query History in the Cloud portal ## The MV\_TRACE\_EVENTS Information Schema The `MV_TRACE_EVENTS` [information schema](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/mv-trace-events.md) is a ring buffer with a size defined by the value of the `trace_events_queue_size` engine variable. When this limit is reached, the oldest events are overwritten as the buffer wraps around. While `MV_TRACE_EVENTS` can be used to view the most recent events that meet a specified criteria, the information displayed is only temporary. As such, the Query History feature in the Cloud portal is recommended for viewing query trace events over time. ## View MV\_TRACE\_EVENTS The following examples can be used to record those queries that take longer than 1,000 milliseconds, or one second, to run, which can be used to create a collection of all of the expensive events that have occurred over a specified period of time. The results can provide insight into which workloads are the most demanding on a cluster, and can be used for workload and application tuning and debugging. ## Successful Queries Run the following to display those SQL statements that have been executed successfully. Note that the following steps are provided to demonstrate how the query events are traced, and only provide a glimpse into which queries are captured. To see the list of successful queries over time, you may filter for “successful queries” in Query History. 1. Note that the `trace_events_queue_size` value is s set to 16,000,000 bytes by default and can be updated with a new value via the following SQL statement. ```sql SET GLOBAL trace_events_queue_size = 16000000; ``` 2. Enable the `Query_completion` event type. ```sql CREATE EVENT TRACE Query_completion WITH (Query_text = on, Duration_threshold_ms = 1000); ``` 3. Display the query trace events. ```sql SELECT * FROM information_schema.MV_TRACE_EVENTS; ``` ```output NODE_ID,NODE_IP,NODE_TYPE,NODE_START_EPOCH_S,EVENT_ID,EVENT_TYPE,TIME,DETAILS 1,127.0.0.1,MA,1698096978,1,Query_completion,2023-10-25 16:39:51.747556,"{""activity_name"":""Select_MV_PROCESS_PERMISSIONS__et_al_453a6b42e2c4cc91"",""connection_id"":44,""context_database"":"""",""duration_ms"":76,""plan_id"":55,""query_text"":""select * from information_schema.mv_trace_events"",""resource_pool_name"":""default_pool"",""row_count"":86,""start_time"":""2023-10-25 16:39:51.671264"",""success"":1,""user_name"":""root""}" 1,127.0.0.1,MA,1698096978,2,Query_completion,2023-10-25 16:40:17.988318,"{""activity_name"":""Select_MV_PROCESS_PERMISSIONS__et_al_453a6b42e2c4cc91"",""connection_id"":44,""context_database"":"""",""duration_ms"":2,""plan_id"":55,""query_text"":""select * from information_schema.mv_trace_events"",""resource_pool_name"":""default_pool"",""row_count"":89,""start_time"":""2023-10-25 16:40:17.986134"",""success"":1,""user_name"":""root""}" ``` The payload that describes an event is stored in the `DETAILS` JSON field that contains the following fields: ```json { "activity_name": string, "connection_id": integer, "context_database": string, "duration_ms": bigint, "plan_id": integer, "query_category": string, "query_text": string, (on by default; can be sizeable as comments are included) "resource_pool_name": string, (empty if no resource pool) "row_count": integer, "start_time": string (datetime(6)), "success": bool, "user_name": string (empty string if the user has been dropped) [, "error_code": string, ] (field only present if success = false) [, "error_message": string ] (field only present if success = false) } ``` The `duration_ms` field represents the total elapsed wall-clock time from when a query starts until it completes. It is not expected to equal the sum of the individual execution metrics (for example, `cpuTimeMs`, `lockTimeMs`, `networkTimeMs`). These metrics are aggregated across the parallel execution on the leaf nodes and are provided to help identify bottlenecks, not as per-node durations that add up to `duration_ms`. To investigate a slowdown, compare `duration_ms` and these metrics between different runs of the same query. The `query_category` field indicates whether the query is a DML or DDL operation. In version 9.0 and later, both DML and DDL activity is captured. **Note**: Currently, when a multi-insert query is traced, a `plan_id` of `-1` is reflected without an associated `activity_name`. **Note**: For read queries, `row_count` refers to the total number of rows sent back to the client. For write queries, `row_count` refers to the total number of rows affected across the cluster. The following shows the `DETAILS` column JSON output. The `query_text` contains literals (sanitized via the [SECRET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/secret.md) function), and includes comments. ```json { "activity_name":"Select_MV_PROCESS_PERMISSIONS__et_al_453a6b42e2c4cc91", "connection_id":44, "context_database":"", "duration_ms":76, "plan_id":55, "query_category": "DML", "query_text":"select * from information_schema.mv_trace_events", "resource_pool_name":"default_pool", "row_count": 86, "start_time":"2023-10-25 16:39:51.671264", "success":1, "user_name":"root" } ``` ``` { "activity_name": "DropTable_t__et_al_1234567890abcdef", "connection_id": 17, "context_database": "db", "duration_ms": 42, "plan_id": 3, "query_category": "DDL", "query_text": "DROP TABLE t", "resource_pool_name": "default_pool", "row_count": 0, "start_time": "2025-04-02 10:15:23.481920", "success": 1, "user_name": "root" } ``` ## Failed Queries Run the following to display those SQL statements that have failed to execute and the associated reason. Note that the following steps are provided to demonstrate how the query events are traced, and they only provide a glimpse into which queries are captured. To see the list of failed queries over time, you may filter for “failed queries” in Query History. 1. Note that the `trace_events_queue_size` value is s set to 16,000,000 bytes by default and can be updated with a new value via the following SQL statement. ```sql SET GLOBAL trace_events_queue_size = 16000000; ``` 2. Enable the Query\_completion event type to capture failed queries. ```sql CREATE EVENT TRACE Query_completion WITH (Query_text = on, Duration_threshold_ms = 1000, Trace_all_failed_executions= on); ``` 3. Run the following to display the query events. ```sql SELECT * FROM information_schema.MV_TRACE_EVENTS; ``` ```output NODE_ID,NODE_IP,NODE_TYPE,NODE_START_EPOCH_S,EVENT_ID,EVENT_TYPE,TIME,DETAILS 1,127.0.0.1,MA,1698096978,1,Query_completion,2023-10-25 16:39:51.747556,"{""activity_name"":""Select_MV_PROCESS_PERMISSIONS__et_al_453a6b42e2c4cc91"",""connection_id"":44,""context_database"":"""",""duration_ms"":76,""plan_id"":55,""query_text"":""select * from information_schema.mv_trace_events"",""resource_pool_name"":""default_pool"",""row_count"":86,""start_time"":""2023-10-25 16:39:51.671264"",""success"":1,""user_name"":""root""}" 1,127.0.0.1,MA,1698096978,2,Query_completion,2023-10-25 16:40:17.988318,"{""activity_name"":""Select_MV_PROCESS_PERMISSIONS__et_al_453a6b42e2c4cc91"",""connection_id"":44,""context_database"":"""",""duration_ms"":2,""plan_id"":55,""query_text"":""select * from information_schema.mv_trace_events"",""resource_pool_name"":""default_pool"",""row_count"":86,""start_time"":""2023-10-25 16:40:17.986134"",""success"":1,""user_name"":""root""}" ``` ## Time-Bound Queries Run the following to display those SQL statements that are time-bound. Note that the following steps are provided to demonstrate how the query events are traced, and only provide a glimpse into which queries are captured. To see the list of time-bound queries over time, you may filter for “time-bound queries” in Query History. 1. Note that the `trace_events_queue_size` value is s set to 16,000,000 bytes by default and can be updated with a new value via the following SQL statement. ```sql SET GLOBAL trace_events_queue_size = 16000000; ``` 2. Run the following to display just those query events that were created since a specified time `@t`. ```sql SET @t = NOW(6); /* Or use a desired time instead of the current time */ ``` 3. Run the following to display the query events. ```sql SELECT * FROM information_schema.MV_TRACE_EVENTS WHERE TIME > @t; ``` ## Enhanced Query Tracing With enhanced query tracing, users can trace high-level execution metrics collected by [MV\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities.md) and [MV\_FINISHED\_TASKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-finished-tasks.md). Enhanced query tracing is controlled via a configuration option for `Query_completion` called `Enhanced_stats_collection`. There are three configuration options: `lite`, `full`, and `off`. The feature is `off` by default. For the remainder of this document, "enhanced query tracing" refers to `Query_completion` tracing with `Enhanced_stats_collection` set to `lite` or `full`. ## Usage If an existing `Query_completion` event is present on the cluster, drop that event before enabling enhanced query tracing. ```sql DROP EVENT TRACE Query_completion; ``` Create an event trace with `Enhanced_stats_collection` set to `off`, `lite`, or `full`. ```sql CREATE EVENT TRACE Query_completion WITH (Enhanced_stats_collection = {off | lite | full}, ...); ``` The new configuration will update asynchronously across all nodes in the cluster. To view the configuration, query [MV\_TRACE\_EVENTS\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/mv-trace-events-status.md) and examine the JSON string in the `CONFIGURATION` column. ## Explanation of Modes ## Off High-level query execution stats from `MV_ACTIVITIES` or `MV_FINISHED_TASKS` are not collected. ## Lite Collects distributed high-level execution metrics of the query's execution and stores the aggregated metrics together with the `Query_completion` event. ```json { "activity_name": "Update_t__et_al_290900f5eb4d75d2", "connection_id": 8, "context_database": "db", "duration_ms": 33, "enhanced_stats": { "aggregated_stats": { "avgMemoryBytes": 263055, "blobCacheMissBytes": 0, "blobCacheWaitTimeMs": 0, "cpuTimeMs": 54, "cpuWaitTimeMs": 0, "diskLogicalReadBytes": 40530, "diskLogicalWriteBytes": 28784, "diskPhysicalReadBytes": 0, "diskPhysicalWriteBytes": 212992, "diskSpillingBytes": 0, "diskTimeMs": 0, "extractorTimeMs": 0, "lockTimeMs": 0, "logBufferLargeTxTimeMs": 0, "logBufferWaitMs": 0, "logBufferWriteBytes": 77009, "logFlushWaitMs": 0, "memoryMBSecs": 0.032894, "memoryMajorFaults": 0, "networkRecvBytes": 2961, "networkSendBytes": 4448, "networkTimeMs": 40, "rgQueuedTimeMs": 0, "rowLockTimeMs": 0, "transformTimeMs": 0, "wmQueuedTimeMs": 0 } }, "plan_id": 27, "query_category", "DML", "query_text": "update t set b = b+1", "resource_pool_name": "default_pool", "row_count": 4, "start_time": "2025-03-05 16:39:01.411371", "success": 1, "user_name": "root" } ``` ## Full Collects the same stats as `lite`. Instead of only storing aggregated stats, also stores stats aggregated per-node. ```json { "activity_name": "Select_t1__et_al_54e9e5098b9ef23f", "connection_id": 8, "context_database": "db", "duration_ms": 244, "enhanced_stats": { "aggregated_stats": { "avgMemoryBytes": 18025, "blobCacheMissBytes": 0, "blobCacheWaitTimeMs": 0, "cpuTimeMs": 169, "cpuWaitTimeMs": null, "diskLogicalReadBytes": 748154, "diskLogicalWriteBytes": 818, "diskPhysicalReadBytes": null, "diskPhysicalWriteBytes": null, "diskSpillingBytes": 0, "diskTimeMs": null, "extractorTimeMs": 0, "lockTimeMs": 2, "logBufferLargeTxTimeMs": 0, "logBufferWaitMs": 0, "logBufferWriteBytes": 0, "logFlushWaitMs": 0, "memoryMBSecs": 0.01931, "memoryMajorFaults": null, "networkRecvBytes": 229227, "networkSendBytes": 6752, "networkTimeMs": 162, "rgQueuedTimeMs": 0, "rowLockTimeMs": 0, "transformTimeMs": 0, "wmQueuedTimeMs": 0 }, "distributed_stats": [ { "info": { "node_id": 1 }, "stats": { "avgMemoryBytes": 14246, "blobCacheMissBytes": 0, "blobCacheWaitTimeMs": 0, "cpuTimeMs": 101, "cpuWaitTimeMs": null, "diskLogicalReadBytes": 55624, "diskLogicalWriteBytes": 142, "diskPhysicalReadBytes": null, "diskPhysicalWriteBytes": null, "diskSpillingBytes": 0, "diskTimeMs": null, "extractorTimeMs": 0, "lockTimeMs": 0, "logBufferLargeTxTimeMs": 0, "logBufferWaitMs": 0, "logBufferWriteBytes": 0, "logFlushWaitMs": 0, "memoryMBSecs": 0.003326, "memoryMajorFaults": null, "networkRecvBytes": 226569, "networkSendBytes": 3629, "networkTimeMs": 142, "rgQueuedTimeMs": 0, "rowLockTimeMs": 0, "transformTimeMs": 0, "wmQueuedTimeMs": 0 } }, { "info": { "node_id": 2 }, "stats": { "avgMemoryBytes": 19078, "blobCacheMissBytes": 0, "blobCacheWaitTimeMs": 0, "cpuTimeMs": 68, "cpuWaitTimeMs": null, "diskLogicalReadBytes": 692530, "diskLogicalWriteBytes": 676, "diskPhysicalReadBytes": null, "diskPhysicalWriteBytes": null, "diskSpillingBytes": 0, "diskTimeMs": null, "extractorTimeMs": 0, "lockTimeMs": 2, "logBufferLargeTxTimeMs": 0, "logBufferWaitMs": 0, "logBufferWriteBytes": 0, "logFlushWaitMs": 0, "memoryMBSecs": 0.015985, "memoryMajorFaults": null, "networkRecvBytes": 2658, "networkSendBytes": 3123, "networkTimeMs": 20, "rgQueuedTimeMs": 0, "rowLockTimeMs": 0, "transformTimeMs": 0, "wmQueuedTimeMs": 0 } } ] }, "plan_id": 2, "query_category": "DML", "query_text": "select a, b, data from t1 join t2 on a = id", "resource_pool_name": "default_pool", "row_count": 3, "start_time": "2025-01-29 14:45:32.790434", "success": 1, "user_name": "root" } ``` ## Remarks * Some of the statistics collected are considered "advanced statistics" and are only collected when `read_advanced_counters` is set to `ON`. Refer to [Management Statistics Reference](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/management-statistics-reference.md) for details. When `read_advanced_counters` is set to `OFF`, the values for these statistics appear as null. * In rare cases, when errors are encountered in query execution, or when certain activities are killed as part of normal query processing, some execution stats are not traced. * The execution metrics collected by enhanced query tracing may have noticeable differences and discrepancies from the same metrics collected by [Historical Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md). * Runtime metrics are collected at a slightly different point in time in query execution for enhanced query tracing than they are for workload monitoring / profiling. In the majority of cases, the difference should not be noticeable. * For write queries with two-phase commit (2PC) enabled, enhanced query tracing includes stats collected by any internal `PREPARE` queries. In contrast, workload monitoring / profiling does not necessarily associate the internal `PREPARE` query with the user query. * Queries that create remote result tables on leaf nodes may have differences in metrics due to the difference in timing of the collection of metrics from the result tables between enhanced query tracing and workload monitoring / profiling. * Queries that are retried during execution (e.g. during online failover) will have all of their attempts' stats collected and aggregated for enhanced query tracing as a single query execution (i.e. in one `Query_completion` event). In contrast, for workload monitoring / profiling, every failed attempt increments the `failure_count`, and each retry is considered a separate execution. ## In this section * [Query History Examples](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history/query-history-examples.md) *** Modified at: July 20, 2026 Source: [/cloud/query-data/query-tuning/query-history/](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history/) (An index of the documentation is available at /llms.txt) --- # Query Performance Tools ## Run ANALYZE The `ANALYZE` command collects data statistics on a table to facilitate accurate query optimization. This is especially important for optimizing complex analytical queries. You should `ANALYZE` your tables after inserting, updating, or deleting large numbers of rows (30% of your table row count is a good rule of thumb). See the [Statistics and Sampling](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling.md) topic for more information. ## EXPLAIN Returns detailed information on how the query is compiled. See [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) for more details. ## PROFILE Provides detailed resources usage metrics about a query. See [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) for more details. ## Auto Profiling Prior to the 8.1 release, `PROFILE` could only be utilized on a per-query basis. The command was appended to the beginning of the query to be profiled. Once the query finished executing, the `SHOW PROFILE` command was run to obtain the profile results. This profiling method is called an explicit profile since the `PROFILE` command is explicitly written with the query. ```sql PROFILE ; SHOW PROFILE; ``` Now SingleStore offers different automatic profiling modes (called "auto profiling" hereafter): `SMART,` `FULL` and `LITE`. Auto profiling and explicit profiling share the same query plans, but the difference between them is the level of statistics collected. Auto profiling is on by default. `SMART` enables selective profiling instead of profiling every eligible query and is the default setting. It profiles a query only when the optimizer determines that runtime statistics from that query may help improve future plan optimization, especially for Feedback Re-optimization. It skips profiling for queries unlikely to benefit, thereby reducing overhead. `FULL` auto profiling collects statistics for every run of the query with an existing query plan. Execution time and network time are not collected. The results provide row counts for each operation, memory usage, and network traffic. The row count information can be used to hint at where the overall execution is processing the most rows; thereby giving direction on how to troubleshoot or optimize a query. `FULL` auto profiling may cause additional network latency between 1-5 milliseconds on mostly low latency queries. `FULL` auto profiling can cause a small variable overhead on the CPU. `FULL` auto profiling works best for queries where adding this latency won't degrade the performance of the overall workload; such as analytical queries that execute in the order of seconds not milliseconds. `LITE` auto profiling collects statistics on the first run of a new query plan and when a query plan is loaded from disk to memory. There is no information collected on execution time, network time, memory usage, and network traffic. The results do provide row counts on each operation as with `FULL` auto profiling. There is negligible memory overhead for high [throughput](https://docs.singlestore.com/cloud/query-data/query-tuning/#section-idm4654689443204832656783025904.md) and low [latency](https://docs.singlestore.com/cloud/query-data/query-tuning/#section-idm4654689443204832656783025904.md) queries for using the `LITE` auto profiling setting. `LITE` auto profiling works best on transaction processing workloads with mostly inserts and updates. However, memory overhead can increase proportionally to the in-memory plancache size. You can set the `auto_profile_type` engine variable to `SMART,` `FULL` or `LITE` as follows: ```sql SET GLOBAL auto_profile_type = SMART | FULL | LITE; ``` > **📝 Note**: If you choose to utilize auto profiling, if you decide to turn it off (`enable_auto_profile = OFF;`), your plans will be recompiled. This can cause a serious slowdown in a heavy online workload, so you may wish to do this during slack time or scheduled downtime. You can toggle between `LITE` and `FULL` auto profiling instead (`SET GLOBAL auto_profile_type = FULL | LITE;`). ## Auto Profiling Use Cases ## Capturing Live vs Historical Profiles The profiles of both explicitly- and auto-profiled queries are obtained using the same method. Live profiles can be obtained from actively running queries. Historical profiles are associated with query plans. Live profiles (also called a partial profile) are available if the query is profiled while currently executing. Information about the query's execution can be retrieved by running the `SHOW PROCESSLIST` command. Using the `ID` from the `SHOW PROCESSLIST` results as the `process_id`, the live profile can be obtained by running the following command in a different session. The user must have `PROCESS` permission. ```sql SHOW PROFILE [JSON] PROCESS [ON NODE ]; ``` > **📝 Note**: Please note, when you run `SHOW PROFILE PROCESS` on a running query, the query's execution will be interrupted as the engine gathers the currently collected statistics to be returned to the client. It's important to avoid running `SHOW PROFILE PROCESS` at high frequency, as it can degrade the performance of the query. To obtain a comprehensive profile after the session has completed the execution of the profiled query, you can simply run `SHOW PROFILE [JSON]` (without the `PROCESS` and `process_id` arguments) from the same session that just finished executing the query. This applies to both explicitly profiled queries and auto-profiled queries that have collected statistics. Historical profiles are obtained by adding a `PLAN` and `plan_id` argument to the `SHOW PROFILE` command. The `plan_id` is obtained by running the `SHOW PLANCACHE` command. If there are a lot of query plans to filter through, it is easier to query the `information_schema.PLANCACHE` view with the appropriate filters to narrow the search for the correct plan. ```sql SHOW PROFILE [JSON] PLAN [ON NODE ]; ``` Profile statistics from previous executions are stored independently of individual users and sessions. Therefore, in a scenario where two sessions are concurrently profiling the same query plan, the profile statistics of the session that finishes execution last will be displayed when running `SHOW PROFILE PLAN`. This is where running `SHOW PROFILE` on the session that finished first becomes useful, as the profile statistics from that session's query execution remain available. > **📝 Note**: Please note, when executing the same query from different clients, two plans are generated with a different `plan_id` in the plancache. There is no way to manually drop historical query profiles without removing the associated query plan from the in-memory plancache. However, they are automatically dropped when the associated query plan expires from the in-memory plancache. ## Profiling Use Cases in Production Environments The following use cases assume the auto profile type is set to `LITE` except for the last use case. * Investigating why a query inexplicably slows down: Despite the absence of operators' individual execution times and network times in the initial run, the plan's profile stats still include the total execution time, row counts, and memory usage. As a result, if the query plan starts running slower than anticipated, the user can save the initial run's stats and perform an explicit profile execution. The explicit profile will show which parts of the query are most expensive and how the query execution has changed by comparing row counts, memory usages, and network traffic. * Investigating why a workload or a procedure inexplicably slows down: In this situation, if the exact issue cannot be identified through workload profiling and existing monitoring tools, the `auto_profile_type` can be set to `FULL` to collect basic stats from the procedures and or sessions that require profiling. Additionally, during intervals between query executions, the user can periodically send `SHOW PROFILE [JSON] PLAN` queries to persist individual profiles. If more detailed stats are needed for specific query plans, the explicit profile approach can be utilized. * Checking whether a long-running query is progressing or should be terminated: The user can capture and save a live profile of the query assuming the query is collecting statistics. After waiting for a few minutes, the user can capture another live profile and compare the two profiles, specifically examining the row counts. If the row counts have changed, it indicates that the query is making progress, but if they haven't, it is likely that the query is stuck at some point. Or, if the row counts are only changing for one operator, then that operator may be the source of the slowness; refactoring the query or changing table and index structures may improve that. * Determining which query plans are more likely to be resource-intensive: In this situation, the user may anticipate high demand from their application and consequently expect increased resource usage during a specific time window. To proactively prepare, the user can analyze the profile stats of existing plans to estimate which plans are likely to be more resource-demanding on the cluster. * Obtaining actual profiling information from users so the support team can troubleshoot query performance issues: This situation is especially helpful when a potentially degraded query is part of a workload and it's difficult to profile the individual queries. Also, some query degradation presents itself intermittently. Setting auto profiling can capture the unexpected performance decline. ## Profiling Use Cases in Query/Workload Tuning * Understanding what the impact would be if new DML queries are added to a workload: The user can switch the `auto_profile_type` to `FULL`, execute the workload, and then switch the `auto_profile_type` back to `LITE`. Then, the user can obtain profile statistics on the query plans that were executed as part of the workload. If specific query plans require detailed information such as individual execution times and network times, the user can explicitly profile them as well. * Performance testing a stored procedure: Typically, [workload profiling](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling.md) is the primary approach for analyzing the performance of an entire workload. Query profiling is intended to complement workload profiling, so if there is a noticeable change in the workload profiling results, query profiles of both new and existing DML queries can be captured and compared. It is important to note that the profiles of existing DML queries may undergo significant changes due to the introduction of new DML queries, which can potentially alter row counts. Leveraging auto profiles can assist in capturing these changes effectively. ## Visual Explain Visual Explain is a feature of SingleStore that allows customers to see a query plan visualized via a graphical interface. This is useful for tuning database queries to reduce run-time or resource usage. See [SingleStore Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) for more details. ## Query Performance Tool Results For The Same Query Using a join between the `people` and `age` tables shown in the shard key and sort key topics, this is what we get when we submit the same query to EXPLAIN, PROFILE, and VISUAL EXPLAIN. EXPLAIN: ```sql EXPLAIN SELECT people.user, people.first, people.last, age.age FROM people JOIN age ON people.id = age.age_id ORDER BY age.age; ``` ```output +----------------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +----------------------------------------------------------------------------------------------------------------------------------------------------+ | GatherMerge [remote_0.age] partitions:all est_rows:8 alias:remote_0 | | Project [people.user, people.first, people.last, age.age] est_rows:8 est_select_cost:16 | | TopSort limit:[?] [age.age] | | HashJoin | | |---HashTableProbe [people.id = age.age_id] | | | HashTableBuild alias:age | | | Project [age_0.age, age_0.age_id] est_rows:8 | | | ColumnStoreScan demo.age AS age_0, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:8 est_filtered:8 | | ColumnStoreScan demo.people, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:8 est_filtered:8 | +----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` PROFILE: ```sql PROFILE SELECT people.user, people.first, people.last, age.age FROM people JOIN age ON people.id = age.age_id ORDER BY age.age; ``` ```output +-------------+-----------+------------+-----+ | user | first | last | age | +-------------+-----------+------------+------ | cjohnson | carrie | johnson | 10 | | dyarbrough | dennis | yarbrough | 13 | | pquincy | penelope | quincy | 21 | | cachebe | chioma | achebe | 29 | | jmalik | jahid | malik | 35 | | kharris | keisha | harris | 43 | | tling | tai | ling | 67 | | sstevens | samuel | stevens | 72 | +-------------+-----------+------------+-----+ { "plan_warnings": { }, "execution_warnings": { }, "mpl_path":"\/var\/lib\/memsql\/instance\/plancache\/a91\/Select_profile_age__et_al_a9111431d10dfe90ceed8b3f2b8e9c1d9706242c27af5f7ee4e97d7d3954a29a_6400bba1c33851b2", "profile":[ { "executor":"GatherMerge", "keyId":4295163904, "order":[ "remote_0.age" ], "partitions":"all", "est_rows":"8", "est_rows_source":"JOIN", "query":"SELECT STRAIGHT_JOIN `people`.`user` AS `user`, `people`.`first` AS `first`, `people`.`last` AS `last`, `age`.`age` AS `age` FROM (`demo_0`.`people` as `people` STRAIGHT_JOIN ( SELECT WITH(NO_MERGE_THIS_SELECT=1) `age_0`.`age` AS `age`, `age_0`.`age_id` AS `age_id` FROM `demo_0`.`age` as `age_0` ) AS `age`WITH (gen_min_max = TRUE)) WHERE (`people`.`id` = `age`.`age_id`) ORDER BY 4 \/*!90623 OPTION(NO_QUERY_REWRITE=1, INTERPRETER_MODE=INTERPRET_FIRST, CLIENT_FOUND_ROWS=1)*\/", "alias":"remote_0", "actual_total_time":{ "value":0 }, "start_time":{ "value":0 }, "end_time":{ "value":1 }, "actual_row_count":{ "value":8, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "inputs":[ { "executor":"Project", "keyId":196736, "out":[ { "alias":"", "projection":"people.user" }, { "alias":"", "projection":"people.first" }, { "alias":"", "projection":"people.last" }, { "alias":"", "projection":"age.age" } ], "est_rows":"8", "est_rows_source":"JOIN", "est_select_cost":"16", "subselects":[], "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "network_traffic":{ "value":202, "avg":101.000000, "stddev":18.000000, "max":119, "maxPartition":0 }, "network_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "inputs":[ { "executor":"Sort", "keyId":196744, "order":[ "age.age" ], "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "memory_usage":{ "value":785424, "avg":392712.000000, "stddev":0.000000, "max":392712, "maxPartition":0 }, "inputs":[ { "executor":"HashJoin", "keyId":327681, "type":"inner", "subselects":[], "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "inputs":[ { "executor":"HashTableProbe", "keyId":327825, "condition":[ "people.id = age.age_id" ], "inputs":[ { "executor":"HashTableBuild", "keyId":327824, "alias":"age", "actual_row_count":{ "value":8, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "memory_usage":{ "value":262144, "avg":131072.000000, "stddev":0.000000, "max":131072, "maxPartition":0 }, "inputs":[ { "executor":"Project", "keyId":327808, "out":[ { "alias":"", "projection":"age_0.age" }, { "alias":"", "projection":"age_0.age_id" } ], "est_rows":"8", "est_rows_source":"JOIN", "subselects":[], "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "inputs":[ { "executor":"ColumnStoreScan", "keyId":4295360512, "db":"demo", "table":"age", "alias":"age_0", "index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "storage":"columnar", "table_type":"sharded_columnstore", "columnstore_in_memory_scan_type":"TableScan", "columnstore_in_memory_scan_index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "est_table_rows":"8", "est_filtered":"8", "est_filtered_source":"DEFAULT", "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "memory_usage":{ "value":262144, "avg":131072.000000, "stddev":0.000000, "max":131072, "maxPartition":0 }, "segments_scanned":{ "value":2, "avg":1.000000, "stddev":0.000000, "max":1, "maxPartition":0 }, "segments_skipped":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_fully_contained":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_filter_encoded_data":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "blob_fetch_network_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_in_blob_cache":{ "value":2, "avg":1.000000, "stddev":0.000000, "max":1, "maxPartition":0 }, "inputs":[] } ] } ] } ] }, { "executor":"ColumnStoreScan", "keyId":4295229440, "db":"demo", "table":"people", "alias":"people", "index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "storage":"columnar", "table_type":"sharded_columnstore", "columnstore_in_memory_scan_type":"TableScan", "columnstore_in_memory_scan_index":"KEY __UNORDERED () USING CLUSTERED COLUMNSTORE", "est_table_rows":"8", "est_filtered":"8", "est_filtered_source":"DEFAULT", "actual_row_count":{ "value":8, "avg":4.000000, "stddev":1.000000, "max":5, "maxPartition":0 }, "actual_total_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "start_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "memory_usage":{ "value":262144, "avg":131072.000000, "stddev":0.000000, "max":131072, "maxPartition":0 }, "segments_scanned":{ "value":2, "avg":1.000000, "stddev":0.000000, "max":1, "maxPartition":0 }, "segments_skipped":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_fully_contained":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_filter_encoded_data":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "blob_fetch_network_time":{ "value":0, "avg":0.000000, "stddev":0.000000, "max":0, "maxPartition":0 }, "segments_in_blob_cache":{ "value":2, "avg":1.000000, "stddev":0.000000, "max":1, "maxPartition":0 }, "inputs":[] } ] } ] } ] } ] } ], "version":"4", "info":{ "memsql_version":"7.5.8", "memsql_version_hash":"12c73130aa6881ec53d57b44a654e4bada1a07c5", "num_online_leaves":"2", "num_online_aggs":"1", "context_database":"demo" }, "query_info":{ "query_text":"PROFILE SELECT people.user, people.first, people.last, age.age\n FROM people\n JOIN age ON people.id = age.age_id\n ORDER BY age.age", "total_runtime_ms":"2", "text_profile":"GatherMerge [remote_0.age] partitions:all est_rows:8 alias:remote_0 exec_time: 0ms start_time: 00:00:00.000 end_time: 00:00:00.001 actual_rows: 8\nProject [people.user, people.first, people.last, age.age] est_rows:8 est_select_cost:16 actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000 network_traffic: 0.202000 KB network_time: 0ms\nSort [age.age] actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000 memory_usage: 785.424011 KB\nHashJoin actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000\n|---HashTableProbe [people.id = age.age_id]\n| HashTableBuild alias:age actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000 memory_usage: 262.144012 KB\n| Project [age_0.age, age_0.age_id] est_rows:8 actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000\n| ColumnStoreScan demo.age AS age_0, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:8 est_filtered:8 actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000 memory_usage: 262.144012 KB segments_scanned: 2 segments_skipped: 0 segments_fully_contained: 0 blob_fetch_network_time: 0ms segments_in_blob_cache: 2\nColumnStoreScan demo.people, KEY __UNORDERED () USING CLUSTERED COLUMNSTORE table_type:sharded_columnstore est_table_rows:8 est_filtered:8 actual_rows: 8 exec_time: 0ms start_time: 00:00:00.000 memory_usage: 262.144012 KB segments_scanned: 2 segments_skipped: 0 segments_fully_contained: 0 blob_fetch_network_time: 0ms segments_in_blob_cache: 2\nCompile Total Time: 0ms\n", "compile_time_stats":{ "mbc_emission":"0", "create_mbc_context":"0", "optimizer_query_rewrites":"0", "optimizer_stats_analyze":"0", "optimizer_stats_other":"0", "optimizer_setting_up_subselect":"0", "optimizer_distributed_optimizations":"0", "optimizer_enumerate_temporary_tables":"0", "optimizer_singlebox_optimizations_agg":"0", "optimizer_stats_autostats":"0", "generating_query_mpl":"0", "generating_user_function_mpl":"0", "unknown":"0", "total":"0" } } } ``` VISUAL EXPLAIN: ```sql SELECT people.user, people.first, people.last, age.age FROM people JOIN age ON people.id = age.age_id ORDER BY age.age; ``` ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt9e363d27f5ac9f4e/6a7ada525947110c064c2649/Visual_Explain_QP_-XrFj7x.png) *** Modified at: June 26, 2026 Source: [/cloud/query-data/query-tuning/query-performance-tools/](https://docs.singlestore.com/cloud/query-data/query-tuning/query-performance-tools/) (An index of the documentation is available at /llms.txt) --- # SingleStore Visual Explain ## Overview SingleStore Visual Explain (or simply “Visual Explain”) is a feature that allows query plans to be visualized graphically. This is useful for tuning database queries to reduce execution times and/or resource usage. Visual Explain can be accessed from the Cloud Portal directly from the SQL Editor or by clicking the **Visual Explain** link. It is also offered as a free [stand-alone service](https://explain.labs.singlestore.com). To visualize a query plan, you may: * **Use the SQL Editor**: Highlight the desired query in the SQL Editor, click the **Visual Explain** drop-down, and select either [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) or [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md). * **Use JSON output**: The `SHOW PROFILE JSON` and `EXPLAIN JSON` SQL statements can display JSON output that can be pasted or uploaded into Visual Explain to show the query plan. * **Use a URL**: The `SHOW PROFILE UI` and `EXPLAIN UI` SQL statements can display a URL that will open Visual Explain and show the query plan. Use the [provided example](https://docs.singlestore.com/#section-idm4635610284734433781854947158.md) to learn how to visualize a query plan with Visual Explain. ## Visual Explain Metrics From the upper left-hand corner of Visual Explain, you may select the **Actual** or **Estimated** options for a given query plan. At the center of Visual Explain is a tree composed of clickable elements, where each element is a single query plan operation. The tree is read from the bottom to the top, so the element at the top of the tree returns the result to the SQL client. Each element includes the following information: * The query plan operation performed * The percent of total query execution time for each query plan operation * The number of rows processed by each query plan operation This number could be greater than the number of rows in a given table if the query includes a statement, such as a nested loop join, that requires the engine to read the table multiple times. * The associated table on which each query plan operation is performed (where applicable) The following metrics are also shown on the right of Visual Explain. * If an element in the graphic has not been clicked, Visual Explain displays a summary of all query plan operations. * **Summary** * Total execution time * Total number of operations * Total network traffic * **Details** * Each query plan operation * Time spent on each query plan operation * If an element in the graphic has been clicked, Visual Explain displays a summary of its query operations. ## Export a Query Plan To export a query plan for use *with* Visual Explain, either: * Run a `PROFILE` SQL statement followed by `SHOW PROFILE JSON` or run an  `EXPLAIN JSON` SQL statement. Both of these options display JSON output that will show the query plan when pasted or uploaded into Visual Explain. * Run a `PROFILE` SQL statement followed by `SHOW PROFILE UI` or run an  `EXPLAIN UI` SQL statement. Both of these options display a URL that, when opened in a browser, will show the query plan in Visual Explain. Use the [provided example](https://docs.singlestore.com/#section-idm4635610284734433781854947158.md) to learn how to visualize a query plan with Visual Explain. To export a query plan *from* Visual Explain, click the vertical dots menu in the upper right-hand corner of Visual Explain and select **Export**. ## Import a Query Plan To import an initial query plan from a JSON output file that was created from either the `PROFILE` or `EXPLAIN` SQL statements, you may either: 1. Click the **Upload File** button and select the desired JSON output file to upload. 2. Click the **Paste JSON** button and paste the JSON output of a query plan into the provided field. To replace a query plan with another: 1. Click the vertical dots menu in the upper right-hand corner of Visual Explain. 2. Select either: 1. The **Import Explain/Profile** option and select the desired JSON output file to upload. 2. The **Paste JSON Explain/Profile** option and paste the JSON output into the provided field. Use the [provided example](https://docs.singlestore.com/#section-idm4635610284734433781854947158.md) to learn how. ## Example The following example demonstrates how to use Visual Explain to profile a query and improve its performance. ## Create the Dataset 1. In the SQL Editor or a SQL client, create the following rowstore table in the database of your choice (such as a test database) and fill it with mock order data. ```sql CREATE ROWSTORE TABLE if not exists expedited_orders( order_id int, order_date datetime, key(order_id), shard(order_id)); INSERT INTO expedited_orders VALUES(1, now()); ``` 2. Re-run the following SQL statement until 524,288 rows of data have been inserted into the table. ```sql INSERT INTO expedited_orders SELECT order_id+(SELECT max(order_id) FROM expedited_orders), now() FROM expedited_orders; ``` ## Obtain an Order Date 1. In the SQL Editor or a SQL client, run the following query to obtain a sample `order_date`. ```sql SELECT order_date, count(*) FROM expedited_orders GROUP BY 1 HAVING count(*) = 1; ``` 2. Run the following query using an `order_date` from the previous query result. **Note**: The remainder of these steps will use `2023-07-11 15:56:08` for the `order_date`. Use an `order_date` from your own database to ensure that this example will work. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ## Visualize and Optimize the Query * *Use the SQL Editor* 1. In the SQL Editor, highlight this query, click the **Visual Explain** drop-down in the upper right-hand corner, and select **Profile**. Visualizing this query reveals that SingleStore performed a full table scan (indicated by TableScan) in this query. As a consequence, each row was scanned to determine which order has an `order_date` of `2023-07-11 15:56:08`. Note that the execution time for this query is 410 milliseconds, which includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8219355af6a74c62/6a7adba51967ea8121325077/visual_explain_example_1-KmH2tp.png)To determine if adding an index will improve performance:1. In the SQL Editor, add an index on the `order_date` column. ```sql CREATE INDEX date_index ON expedited_orders (order_date); ``` 2. Re-run the `SELECT * FROM expedited_orders` query. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 3. Highlight this query, click the **Visual Explain** drop-down in the upper right-hand corner, and select **Profile**. Visualizing this query reveals that SingleStore used an index range scan (indicated by IndexRangeScan) instead of a full table scan, thereby limiting the scan to only those rows that should contain the specified `order_date`. Note that the execution time for this query is down to 221 milliseconds, which also includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8111329d0baca10f/6a7ada50f682576b7858d924/visual_explain_example_2-cSm7dQ.png)To see how quickly this query will run without the query compilation time:1. In the SQL Editor, re-run the `SELECT * FROM expedited_orders` query. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 2. Highlight this query, click the **Visual Explain** drop-down in the upper right-hand corner, and select **Profile**. Visualizing this query reveals that the execution time has been reduced to 26 milliseconds. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltff2afdb7308cc173/6a7ada51c5d19348f95db8a2/visual_explain_example_3-Bn97xn.png) * *Paste JSON Output* 1. In a SQL client, run the following `PROFILE`-related queries to obtain the JSON output. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` 2. Click the Visual Explain link in the left nav, click the **Paste JSON** button, paste the JSON output into the provided field, and click the **Show Explain** button. Visualizing this query reveals that SingleStore performed a full table scan (indicated by TableScan) in this query. As a consequence, each row was scanned to determine which order has an `order_date` of `2023-07-11 15:56:08`. Note that the execution time for this query is 410 milliseconds, which includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8219355af6a74c62/6a7adba51967ea8121325077/visual_explain_example_1-KmH2tp.png)To determine if adding an index will improve performance:1. Add an index on the `order_date` column. ```sql CREATE INDEX date_index ON expedited_orders (order_date); ``` 2. Re-run the `SELECT * FROM expedited_orders` query. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 3. Re-run the `PROFILE`-related queries to obtain the JSON output. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` 4. Copy the JSON output to the clipboard. 5. In Visual Explain, click the vertical dots menu in the upper right-hand corner, click the **Paste JSON Explain/Profile** option, paste the JSON output into the provided field, and click the **Show Explain** button. Visualizing this query reveals that SingleStore used an index range scan (indicated by IndexRangeScan) instead of a full table scan, thereby limiting the scan to only those rows that should contain the specified `order_date`. Note that the execution time for this query is down to 221 milliseconds, which also includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8111329d0baca10f/6a7ada50f682576b7858d924/visual_explain_example_2-cSm7dQ.png)To see how quickly this query will run without the query compilation time:1. Re-run the `SELECT * FROM expedited_orders` query. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 2. Re-run the `PROFILE`-related queries to obtain the JSON output. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` 3. Copy the JSON output to the clipboard. 4. In Visual Explain, click the vertical dots menu in the upper right-hand corner, click the **Paste JSON Explain/Profile** option, paste the JSON output into the provided field, and click the **Show Explain** button. Visualizing this query reveals that the execution time has been reduced to 26 milliseconds. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltff2afdb7308cc173/6a7ada51c5d19348f95db8a2/visual_explain_example_3-Bn97xn.png) * *Upload/Import a JSON Output File* 1. In a SQL client, run the following `PROFILE`-related queries to obtain a JSON output file. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` Save the output to a local file. 2. Click on the Visual Explain link in the left nav, click the **Upload File** button, and select this JSON output file to upload. Visualizing this query reveals that SingleStore performed a full table scan (indicated by TableScan) in this query. As a consequence, each row was scanned to determine which order has an `order_date` of `2023-07-11 15:56:08`. Note that the execution time for this query is 410 milliseconds, which includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8219355af6a74c62/6a7adba51967ea8121325077/visual_explain_example_1-KmH2tp.png)To determine if adding an index will improve performance:1. Add an index on the `order_date` column. ```sql CREATE INDEX date_index ON expedited_orders (order_date); ``` 2. Re-run the `SELECT * FROM expedited_orders` query. ``` SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 3. Re-run the `PROFILE`-related queries to obtain a JSON output file. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` Save the output to a local file. 4. In Visual Explain, click the vertical dots menu in the upper right-hand corner, click the **Import Explain/Profile** button, and select this JSON output file to upload. Visualizing this query reveals that SingleStore used an index range scan (indicated by IndexRangeScan) instead of a full table scan, thereby limiting the scan to only those rows that should contain the specified `order_date`. Note that the execution time for this query is down to 221 milliseconds, which also includes the query compilation time. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8111329d0baca10f/6a7ada50f682576b7858d924/visual_explain_example_2-cSm7dQ.png)To see how quickly this query will run without the query compilation time:1. Re-run the `SELECT * FROM expedited_orders` query. ```sql SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` 2. Re-run the `PROFILE`-related queries to obtain a JSON output file. ```sql PROFILE SELECT * FROM expedited_orders WHERE order_date = "2023-07-11 15:56:08"; ``` ```sql SHOW PROFILE JSON; ``` Save the output to a local file. 3. In Visual Explain, click the vertical dots menu in the upper right-hand corner, click the **Import Explain/Profile** button, and select this JSON output file to upload. Visualizing this query reveals that the execution time has been reduced to 26 milliseconds. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltff2afdb7308cc173/6a7ada51c5d19348f95db8a2/visual_explain_example_3-Bn97xn.png) ## Summary By employing Visual Explain to help optimize this example, the execution time for the `SELECT * FROM expedited_orders query` has been reduced: * From an initial 410 milliseconds for a full table scan * To 221 milliseconds after introducing an index, which includes the query compilation time * To only 26 milliseconds after the query had been compiled For additional examples, refer to the final lab in the SingleStore [Query Tuning](https://training.singlestore.com/learn/course/internal/view/elearning/634/query-tuning) training course. ## Related Topics * [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) and [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) * [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) * [Query Plan Operations](https://docs.singlestore.com/cloud/query-data/query-plan-operations.md) *** Modified at: May 22, 2026 Source: [/cloud/query-data/query-tuning/singlestore-visual-explain/](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain/) (An index of the documentation is available at /llms.txt) --- # Statistics and Sampling In order to create good query plans, the query optimizer needs information about the size and distribution of table data. The optimizer uses this information to estimate the number of rows produced by different parts of a query plan as it searches to find the best overall plan. This section explains which statistical methods should be used to gather your database's table data. It also details how to set up those methods on your tables. ## In this section * [Statistics and Sampling Concepts](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-concepts.md) * [Statistics and Sampling Tasks](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-tasks.md) *** Modified at: January 11, 2022 Source: [/cloud/query-data/query-tuning/statistics-and-sampling/](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/) (An index of the documentation is available at /llms.txt) --- # Testing Your Queries and Performance ## Running Queries Hopefully at this point you have your data in SingleStore. You can check this by running some basic `SELECT` statements within the SQL Editor. Next, you may want to try out some queries that you were running with your last database. Generally, you’ll want to run queries twice to get a true understanding of the runtime as the first run must create and cache the query plan. ## Visual Explain One great feature of SingleStore Helios is Visual Explain. If you encounter a situation in which your query is taking longer than expected, highlight the desired query in the SQL Editor and click the Visual Explain icon to the left of the Run button. The icon resembles a tree. After clicking, you may then choose between [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) and [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md). Refer to [SingleStore Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) to learn how to navigate the visual explanation or profiling of your query. Once you identify a bottleneck, you should be able to make changes either to your schema or to your query itself in order to improve speed. You can manually (non-visually) run `EXPLAIN` or `PROFILE` from any client; see the links above for details on the commands. ## Benchmarking If you’re moving over from any of the databases we’ve talked about so far, you are probably interested in improving performance. At SingleStore, we’ve developed an easy-to-use tool for benchmarking called dbbench. You can check that out [here](https://github.com/memsql/dbbench). Once you’ve installed the packages to your host machine, you can walk through this [tutorial](https://github.com/memsql/dbbench/blob/master/TUTORIAL.md). All you will have to do is change the host from 127.0.0.1 to the endpoint listed for your cluster in the Cloud Portal (it should look something like: `svc-xxx-dml.aws-virginia-1.db.memsql.com`). The port will remain `3306`, as listed. You can then move onto testing based on your application workload. *** Modified at: May 22, 2026 Source: [/cloud/query-data/query-tuning/testing-your-queries-and-performance/](https://docs.singlestore.com/cloud/query-data/query-tuning/testing-your-queries-and-performance/) (An index of the documentation is available at /llms.txt) --- # Troubleshooting Poorly Performing Queries ## Diagnostic Tools SingleStore has several tools for troubleshooting query performance. The [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history/query-history-examples.md) dashboard helps identify resource-intensive queries and workloads. It is based on query event tracing, which can trace individual queries and obtain performance and utilization metrics. Query History can also be used to view statistics for queries that are expected to run frequently. The [Historical Workload Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md) dashboards present a holistic view across all queries. These dashboards provide native monitoring which allows users to quickly understand their application workloads and debug performance-related issues. [Enhanced query tracing](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) allows you to trace the high-level execution metrics collected by `MV_ACTIVITIES` and `MV_FINISHED_TASKS` along with `Query_completion` traces. [Active Workload Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/active-workload-monitoring.md) provides a snapshot of metrics during a specified reporting period. Metrics include CPU utilization, number of query executions, and memory consumed. It allows you to debug a workload in real time as queries are running. Historical Workload Monitoring and Query History allow you to go back to a specific time and debug whereas Active Workload Monitoring is on-demand looking at the current queries. ## 1. Identifying Expensive Queries 1. In the Cloud Portal, navigate to the **Active Workload Monitoring** page. It helps you analyze query behavior during a specific period of time. It does this by comparing metrics at two points in time — the start and end of the recording window. 2. If you have monitoring data, toggle the time selector to the applicable time range for the workload or query of interest. By default, the last hour is shown. 3. If one has been identified, switch the database selector to the suspect database. ![All active activities are listed with name, activity category, database, elapsed time, and more in the Workload Monitoring page.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2336da4b316bf194/6a7adb693ce8e21a2ccea13c/16083b5794f43e-7Fup0Q.png) 4. Sort on `Elapsed Time` and `Run Count` by selecting the field name. Hover over the `Time Spent` bar and review which queries are consuming the most amount of time. 5. Sort on CPU, Memory, and Network to identify other resource-intensive queries. 6. Distinguish query shapes that may be resource-intensive due to having a high `run count` compared to others. For example, is a query using resources because of how many times it has been run, or is it using a large amount of resources across a low number of runs? 7. Note the `activity_name` of queries that have high elapsed time, run count, and/or that are using a large amount of a given resource. 8. Select the **Nodes** tab to identify which node these queries are running. The **Nodes** tab can also help to confirm why a specific node is running slowly. ## Next Steps * If you have identified a long-running query that is using a large amount of resources or has an unusual run count, continue to [Step 2](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). * If you have not identified any useful information, skip to [Step 3](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 2. Investigating Expensive Queries 1. If a query has a high run count and is using a large amount of resources (on average), it is important to identify why the query is running at such high concurrency. * Is this normal for your application? * Can you change how often it is run? 2. If you have identified a query with a low run count that uses a large amount of resources, or a frequent query that is a normal occurrence, copy the query from the Cloud Portal. To get the query text from what is displayed, query the information\_schema table [MV\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-queries.md). ```sql SELECT * FROM information_schema.MV_QUERIES where activity_name = ; ``` 3. Check the `PLAN_WARNINGS` column for any hints about potential optimizations. If there are no hints, continue to the next step. 4. If the query in question is an `INSERT ... SELECT` statement, copy only the `SELECT` clause of the query. 5. Navigate to the SQL Editor in the Cloud Portal for your database, and paste in the query. 6. Run the query in the SQL Editor by selecting the **Run** button in the top right-hand corner of the page. After the first run is complete, highlight the query and select the **Visual Explain** icon to the left of the **Run** button to display the query plan in a graphical interface. Here, you will be given a choice between `EXPLAIN` and `PROFILE`, which display the query plan and further details for each query operation, respectively. SingleStore recommends running both of these options. ![A visual explain page displaying the query plan of the executed query in graphical interface.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt7aafc543864094b9/6a7adb7105b7b570ca18002d/16083b579590ed-DsvKb9.png) 7. Check if there is anything unusual about the query plan. You may find a [query plan operation](https://docs.singlestore.com/cloud/query-data/query-plan-operations.md) that takes a large amount of time. If so, investigate that particular operator. You may also find that each component is similarly time-consuming. In either case, here are some questions to consider: * Are you reshuffling or repartitioning a large amount of data? 1. Common indicators of this include the `Project` operator combined with large `SELECT` costs or `Repartitioning` being present in the query plan. 2. If so, can you add or change a shard key? * Are you doing a full scan of a table? 1. Look for the `TableScan` operator to indicate this information. 2. If so, can you add/change an index or sort key? * Is the query performing slowly or using more resources because of a schema change? * Is the query performing slowly because the data working set increased? * Are [autostats](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-concepts.md) enabled for the tables in the query? 1. Run [SHOW CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-table.md) 2. Look for `AUTOSTATS_ENABLED=FALSE`, `AUTOSTATS_CARDINALITY_MODE=OFF`, or `AUTOSTATS_HISTOGRAM_MODE` in the `CREATE TABLE` statement to determine this. * Do you need to invalidate and refresh your current query plan? 1. Autostats do not invalidate the plan automatically. To invalidate the plan, run [ANALYZE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md) on the table manually. 2. Alternatively, to drop the current plan, run [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md) and re-run the query to see if it performs better. 8. Refer to [Optimizing Table Data Structures](https://docs.singlestore.com/cloud/create-a-database/optimizing-table-data-structures.md) for additional details, as well as [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) for further guidance on how to optimize specific query shapes and tables. ## Next Steps * If you did not identify any further useful information here: continue to [Step 3](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 3. Are any queries waiting on something? Use the following steps to determine whether your queries are being blocked or waiting for another reason. 1. Check the Active Queries page in the Cloud Portal, or the `PROCESSLIST`. * On the active queries page in the Cloud Portal, check whether there are many queued queries by sorting on the `status` column. * The default `workload_management_max_queue_depth` before queries fail is 100. ![An Active Queries page showing the active processes.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt8f0a75d654768b8d/6a7adb65227b1c7a5b58ce47/16083b57960a3f-zmB6Fr.png) * If you are using the database client, query [MV\_PROCESSLIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-processlist.md). `SELECT * FROM information_schema.MV_PROCESSLIST WHERE state = 'queued';` 2. Are there queries lingering or in an unkillable state? * Check the `PROCESSLIST` or the Active Queries page by using the above. 1. Sort by `time`. If there is a long runtime (depending on a normal run time for your queries, which could be minutes, hours, or days), it is likely the query is hung. 2. Identify those queries and run [KILL QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/kill-connection-and-kill-query.md) on them. This can be done in the Cloud Portal by checking the boxes shown above, or from the client. 3. You can also run [KILLALL QUERIES](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/killall-queries.md) to remove a large number of queries at once. **Note: This will kill all queries, whether hanging or running.** 4. If you see queries with both a long runtime and a state of `KILLED` on a given node, there may be queries stuck in an unkillable state. You will file a support ticket in this case. * Are queries queued by the workload manager? Check this by querying [MV\_WORKLOAD\_MANAGEMENT\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-workload-management-status.md) and looking at `Queued Queries (from global queue)` & `Queued Queries (from local queue)` within the `Stat` and `Value` columns. `SELECT * FROM information_schema.WORKLOAD_MANAGEMENT_STATUS;` 3. Are queries blocked? * Check this by querying [information\_schema.MV\_BLOCKED\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-blocked-queries.md). `SELECT * FROM information_schema.MV_BLOCKED_QUERIES;` * Which query is blocked? Check the `QUERY_TEXT` column. * What is blocking it and where is it running? Check the `BLOCKING_NODE_ID` and `BLOCKING_QUERY_TEXT` columns. * Check the blocking reason by looking at the `BLOCKING_TYPE` * Kill the blocking query by using the `ID` column, which is the same as the `ID` in `information_schema.PROCESSLIST`, with `KILL QUERY `. 4. If your queries are failing: * Is the queue depth too large? Query [MV\_PROCESSLIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-processlist.md) to check this. `SELECT COUNT(*) FROM information_schema.MV_PROCESSLIST WHERE state = 'queued';` * Are these queries getting killed by the resource governor? Check resource the status of resource pools by querying [RESOURCE\_POOLS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pools.md). `SELECT * FROM information_schema.RESOURCE_POOLS;` Note, resource pools are available to Enterprise customers. * Examine the failure errors from failed queries. ## Next Steps * My queries are queued and waiting on resources: continue to [Step 4](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). * There are no queued, blocked, or hung queries: continue to [Step 4](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 4. Are key resources being overused? The next step in troubleshooting your cluster is looking at key resources to see if they are being overused. 1. For a quick sanity check, navigate to the Cloud Portal Dashboard to look at resource usage against their respective totals available by hovering over each. ![Workspace resource usage showing vCPU, memory, and Cache usage.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt06ce6259293bb815/6a7adb4c96b5a6fe6486fdd5/Workspaces-qZ1D6B.png) 2. Use the respective `mv_sysinfo` tables to look at CPU and memory usage. 3. Sort on CPU Usage, Memory Usage, and Disk Usage. Are there resources being overused? 4. If you see one or more resources approaching their limit (e.g., memory, disk, or CPU limits), consider the following: * Have you added a new workload to your cluster that is using more resources? * Have you loaded more data (on disk, or in memory)? * Have you increased concurrency? * Does your workload require you to scale up the size of your deployment? * Is there an expensive workload or query that is causing a spike and could be optimized? ## Next Steps * If you identify one or more key resources being overused: return to [Step 1](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md), to identify whether there are specific queries causing this. * If the cause is not specific queries, or there are no resource usage issues: continue to [Step 5](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 5. Are there other ongoing operational activities? The next step in troubleshooting your cluster is checking other ongoing operational activities that may be affecting your cluster performance. 1. Navigate to the Active Queries page in the Cloud Portal, or query [MV\_PROCESSLIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-processlist.md) ```sql SELECT * FROM information_schema.mv_processlist; ``` 2. Search (ctrl-F) in the Active Queries page, or look in the results of your query to `MV_PROCESSLIST` for any of the following: `ALTER, CREATE, DROP, ADD|REMOVE LEAF/AGGREGATOR, BACKUP/RESTORE, REBALANCE, RESTORE REDUNDANCY`. 3. For either the Active Queries page, or `MV_PROCESSLIST`, check the `status` column to see if any of these activities are long running, or waiting on other operations. * If an operation is `waiting on metadata lock`, the query is likely waiting on a cluster wide operation such as a backup. ## Next Steps * If no other ongoing operational activities were identified: continue to [Step 6](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 6. Checking Node, Partition, and Overall Database Health The next step in troubleshooting your cluster is checking its overall health. 1. If nodes are offline these will be fixed automatically. This need not be investigated. 2. Go to the Databases page, and examine the suspect database. * Click on the database, check the Partitions tab, and make sure everything is online and attached. 3. If there are offline or detached partitions, file a support ticket. ## Next Steps * If all nodes, partitions, and databases are healthy: continue to [Step 7](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md). ## 7. Checking Partition Number and Data Skew This step checks whether there are enough partitions and if data is properly balanced across your cluster. 1. Check the total core-to-partition ratio per leaf for the database that is having trouble: * Run [SHOW PARTITIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-partitions.md) on the database in question and compare that with the total number of cores in your cluster. * Is this ratio at least 1:1? If you have high concurrency, consider increasing the concurrency from 2, 3, or 4 to 1 core to partition ratio. 2. Check that your data is redundant by running [EXPLAIN RESTORE REDUNDANCY](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/explain-restore-redundancy.md). 3. Data Skew: refer to the [Detecting and Resolving Data Skew](https://docs.singlestore.com/cloud/create-a-database/detecting-and-resolving-data-skew.md). ## Additional Things to Consider **SingleStore Queries Typically Run Faster the Second Time They Are Executed** Traditional relational database management systems interpret SQL queries the same way interpreters for languages like Python and Ruby run programs. The first time a SingleStore server encounters a given query shape, it will optimize and compile the query for future invocations. This incurs overhead which does not depend on the amount of data to be processed, but rather the complexity of the query. The process of code generation involves extracting parameters from the query and then transforming the normalized query into a SingleStore specific intermediate representation tailored to the system. Subsequent requests with the same shape can reuse this plan to complete both quickly and consistently. **Balancing Expensive Single Partition Queries Against Data Skew** In general, queries that operate on a single partition are great for overall performance because only a portion of the cluster's resources is being utilized. This leaves the resources dedicated to the other partitions free to run other queries. It also cuts out some of the overhead on the aggregator for coordinating a cluster-wide, distributed query gathering and merging data from all partitions. However, it is important to select a shard key that promotes lightweight single-partition queries. For example, if you have a `PRODUCTS` table for a university bookstore: ```sql CREATE TABLE IF NOT EXISTS PRODUCTS (prod_id int, prod_name varchar(20), category varchar(10), color varchar(10) ); ``` The shard key could be set on the `category` column if the primary workload against the table is querying inventory and filtering on `category`. If this fictional university bookstore sells mostly t-shirts and sweatshirts (`category` = 'apparel'), the most queried data would be located in one partition. So the data skew caused by selecting `category` as the shard key causes an expensive single partition workload. Query processing is distributed at 1 CPU core per partition. In our example, we have data skew on one partition and a single partition query frequently runs against it; so the query will run single-threaded. This increases latency or the query can be queued. Meanwhile, the other resources in the cluster are not being utilized; negating both the performance achievements you were aiming for and the main benefit of using a distributed database. In this example, `prod_id` would be the best choice as the shard key. Or creating and sharding on an auto-increment `id` column since it is the best practice to NOT shard on a varchar() data type to use. Or the table can be created as a columnstore and `category` can be set as the cluster columnstore key, so filtering on the category will benefit from segment elimination. Since the "apparel" values will be located in the same location on disk, the query will perform better. *** Modified at: June 12, 2026 Source: [/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries/](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries/) (An index of the documentation is available at /llms.txt) --- # Workload Profiling This topic contains reference information about workload profiling management views. ## In this section * [Concepts and Terminology](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/concepts-and-terminology.md) * [MV\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities.md) * [MV\_ACTIVITIES\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-cumulative.md) * [MV\_ACTIVITIES\_EXTENDED](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-extended.md) * [MV\_ACTIVITIES\_EXTENDED\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-extended-cumulative.md) * [MV\_QUERY\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-query-activities.md) * [MV\_QUERY\_ACTIVITIES\_EXTENDED\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-query-activities-extended-cumulative.md) * [MV\_TASKS](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-tasks.md) * [Management Statistics Reference](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/management-statistics-reference.md) *** Modified at: September 30, 2022 Source: [/cloud/query-data/query-tuning/workload-profiling/](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/) (An index of the documentation is available at /llms.txt) --- # SELECT The `SELECT` statement is used to retrieve data from a table. See the [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) page for more details such as syntax, remarks, and examples. The `SELECT` clause specifies the columns to be retrieved from the table(s). All columns can be specified. Distinct (not repeated) values for a column can also be specified. The `SELECT` page gives details of the syntax for these options. The following clauses make up the basic parts of a `SELECT` statement. The clauses are optional except for the `FROM` clause in most cases. ## The FROM Clause The `FROM` clause specifies which table(s) data will be queried from. If multiple tables are queried, the subsequent table(s) that follow the first table will use a `JOIN` clause. See [JOIN and Subqueries](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select/#join-and-subqueries.md) for details about join syntax. ## The WHERE Clause The `WHERE` clause specifies the conditions that must be met for a row to be included in the result set. ## The GROUP BY and HAVING Clauses The `GROUP BY` clause groups rows together that have the same values for columns listed in the `SELECT` clause. The GROUP BY clause must be used for columns not aggregated. See the [Aggregate Functions](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions.md) section for more information on how to use the `GROUP BY` clause in queries that contain aggregate functions. The `HAVING` clause is used to filter the results of queries that use a `GROUP BY` clause. The `HAVING` clause can refer to grouped and ungrouped columns. ## The ORDER BY Clause The `ORDER BY` clause sorts the result set of a query based on one or more columns. Either ascending (`ASC`) or descending (`DESC`) can be specified. By default, if you don't specify `ASC` or `DESC`, the sort will be in ascending order. ## The LIMIT Clause The `LIMIT` clause is used to restrict the number of rows returned in the result set to the number specified. ## Related Information [SQL Functions List](https://docs.singlestore.com/cloud/reference/sql-reference/sql-functions-list.md) - this is a list of all the SQL functions SingleStore Helios current used. SQL Functions can be used in `SELECT` statements. *** Modified at: June 22, 2023 Source: [/cloud/query-data/select/](https://docs.singlestore.com/cloud/query-data/select/) (An index of the documentation is available at /llms.txt) --- # Reference Reference material for configuring and using SingleStore, including: * Configuration reference, system limits, engine variables, toolbox commands, etc. * Operator and Tools reference * SQL command reference * Fusion SQL reference * Information Schema reference * Troubleshooting * SingleStore Data API * `Management` API * [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md) ## In this section * [Configuration Reference](https://docs.singlestore.com/cloud/reference/configuration-reference.md) * [Information Schema Reference](https://docs.singlestore.com/cloud/reference/information-schema-reference.md) * [SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference.md) * [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) * [Troubleshooting Reference](https://docs.singlestore.com/cloud/reference/troubleshooting-reference.md) * [System Databases](https://docs.singlestore.com/cloud/reference/system-databases.md) * [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) * [Management API Overview](https://docs.singlestore.com/cloud/reference/management-api.md) * [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) * [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md) *** Modified at: April 1, 2026 Source: [/cloud/reference/](https://docs.singlestore.com/cloud/reference/) (An index of the documentation is available at /llms.txt) --- # Code Engine - Powered by Wasm The Code Engine feature in SingleStore supports creating functions (UDFs, UDAFs, and TVFs) using code compiled to WebAssembly (Wasm). The Code Engine uses the `wasmtime` runtime to compile and run WebAssembly code. Refer to [Wasmtime](https://docs.wasmtime.dev/) for more information. This feature supports any language that can compile to the Wasm core specification, which allows you to create UDFs/UDAFs/TVFs in a language of your choice using existing code libraries and run them in a sandboxed environment for enhanced security. Each Wasm function instance runs in its own in-process sandbox. It uses a linear memory model and provides hard memory protection boundaries. By default, each Wasm function sandbox is allocated `16MB` of memory. You can use the `GROW TO` clause to modify the memory allocated to a Wasm function sandbox. Once a Wasm function is added to SingleStore, it becomes a part of the database. Hence, it is also included in the database backup and restore operations. ## Why use Wasm-based Functions SingleStore recommends using Wasm-based functions for integrating complex logic and data structures or using existing C/C++ or Rust code. Key benefits to using Wasm: * **Security**: Run your code in a sandboxed environment for enhanced security. * **Code reusability**: Leverage existing code base without rewriting the logic in SQL. * **Code integration**: Integrate complex logic and data structures that cannot be expressed using PSQL. If your logic and data structures can be implemented in [Procedural SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions.md) (PSQL) with reasonable effort in a way that performs well, use PSQL. Refer to [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md) for related information. ## Use Cases The following is a list of common real-world applications that have been implemented using Wasm-based functions: * [Sentiment Analysis](https://github.com/singlestore-labs/singlestoredb-extension-sentiment): Perform sentiment analysis using Valence Aware Dictionary and Sentiment Reasoner (VADER). * [Bloom Filters](https://github.com/singlestore-labs/singlestoredb-extension-bloom-filters): Generate Bloom filters from a column of string values and then run the Bloom filter on another column. * [JSON Flatten](https://github.com/singlestore-labs/singlestoredb-extension-json-flatten): Perform flattening operations on JSON objects. * [HyperLogLog (HLL) Sketches](https://github.com/singlestore-labs/singlestoredb-extension-hll-sketch): Generate HLL sketches using different operations and algorithms and return diagnostic information. * [Theta Sketches](https://github.com/singlestore-labs/singlestoredb-extension-theta-sketch): Generate Theta sketches using different operations and algorithms and return diagnostic information. ## Build Wasm-based Functions You can build Wasm extensions for SingleStore on Mac, Windows, or Linux using [Docker](https://docs.docker.com/get-docker/) development containers. Refer to [SingleStore Wasm Toolkit](https://singlestore-labs.github.io/singlestore-wasm-toolkit/html/Tutorial-Overview.html) for a tutorial on creating Wasm functions, importing them in SingleStore, and evaluating them in queries. For more examples, refer to [Create Wasm UDFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs/#section-idm4543865861515233102331638184.md) or [Create Wasm TVFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-tvfs/#section-idm4623033066499233187115419357.md). Refer to [Create Wasm-based UDAFs](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate/#section-idm4573709425052833819733285792.md) for an example on creating Wasm-based user-defined aggregate functions (UDAFs). ## Memory Management Wasm modules are serialized to the plancache. Refer to [Managing Plancache Memory and Disk Usage](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/managing-plancache-memory-and-disk-usage/#section-idm234457677444929.md) for information on how Wasm-modules are evicted from memory. ## Runtime Constraints The Wasm UDF/TVF must not require runtime capabilities that the database does not provide out of the box. For example, a Wasm UDF/TVF can not make system calls, open files, open sockets, send network messages, create processes, or create threads. SingleStore does not support these operations out of the box to protect the security and integrity of its service process. ## Related Topics | [Wasm Application Binary Interfaces (ABIs)](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/wasm-application-binary-interfaces-abis.md) | Information on supported Wasm ABIs.SingleStoresupports both the Basic and Canonical ABIs. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | [Configure Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/configure-wasm.md) | Configure Wasm extensibility using engine variables. | | [Create Wasm UDFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs.md) | Create Wasm-based user-defined functions (UDFs). | | [Create Wasm TVFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-tvfs.md) | Create Wasm-based user-defined table-valued functions (TVFs). | | [Create Wasm UDAFs](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate/#section-idm4508754721451233639501215694.md) | Create Wasm-based user-defined aggregate functions (UDAFs). | | [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md) | Understand when to use PSQL, Wasm-based, or external functions. | | [Procedural Extensions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions.md) | Information on creating Procedural SQL (PSQL) functions or procedures. | | [Extensions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/extensions.md) | Combine Wasm binaries and WIT interfaces into a packaged archive and create, manage, or deploy these resources using a script. | ## FAQs * **Can I use a Wasm binary outside of the SingleStore database once it is imported into the database?** No. Once a Wasm binary is imported into a SingleStore database, you cannot use it for applications outside the database. However, you can make a copy of the binary and use it instead. * **What happens if there is a mismatch between the WIT file and the generated code?** A mismatch may cause a Wasm trap or unexpected results. The exact behavior depends on how the Wasm program interprets the arguments. * **Can I reverse engineer a Wasm binary once it is imported into the database?** No. Once a Wasm binary is imported into the database, it is not possible to see the binary data, it can only be executed. Hence, you can only reverse engineer the Wasm binary behaviorally and not by disassembling the binary. * **What happens when a Wasm function crashes or runs out of memory?** If a Wasm function crashes or runs out of memory, it causes the query to fail and returns an error. If possible, the error message may include a stack into the Wasm function. ## In this section * [Wasm Application Binary Interfaces (ABIs)](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/wasm-application-binary-interfaces-abis.md) * [Configure Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/configure-wasm.md) * [Wasm Data Type Coercions](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/wasm-data-type-coercions.md) * [Create Wasm UDFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs.md) * [Create Wasm TVFs](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-tvfs.md) * [Improve Wasm Function Performance](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/improve-wasm-function-performance.md) * [Troubleshooting](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/troubleshooting.md) *** Modified at: June 13, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/) (An index of the documentation is available at /llms.txt) --- # Configure Wasm Use the following global variables to configure the Code Engine for Wasm extensibility: | Variable Name | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable_wasm` | Specifies if a user can create or call Wasm UDFs/TVFs. | | `wasm_max_compiled_module_size` | Specifies the maximum size (in bytes) that a compiled Wasm module can use. | | `wasm_max_linear_memory_size` | Specifies the maximum linear memory (in bytes) that an individual Wasm module can use. This will further constrain UDF/TVF-specific`GROW TO`values. | | `wasm_max_raw_module_size` | Specifies the maximum size (in bytes) of Wasm modules that may be loaded. The size is defined as the size of raw, uncompiled data passed in the`CREATE FUNCTION`statement. | Refer to [Wasm Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for more information, including default values. *** Modified at: July 16, 2026 Source: [/cloud/reference/code-engine-powered-by-wasm/configure-wasm/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/configure-wasm/) (An index of the documentation is available at /llms.txt) --- # Create Wasm TVFs You can create Wasm TVFs using the `CREATE FUNCTION` statement. > **📝 Note**: SingleStore supports only the [Canonical ABI](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) with Wasm TVFs because the return type is `TABLE`. For information on creating PSQL user-defined table-valued functions, refer to [CREATE FUNCTION (TVF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md). ## Syntax ```sql CREATE [OR REPLACE] FUNCTION function_name { | } (USING EXPORT '"' '"') explicitTypedPred: AS WASM (ABI CANONICAL ) (GROW TO ) FROM implicitTypedPred: RETURNS TABLE AS WASM (ABI CANONICAL ) (GROW TO ) FROM (WITH WIT FROM ) function_name: typeSpec: '(' ? ')' RETURNS TABLE '(' ')' numBytes: [0-9]+ paramList: (',' )* param: varName: contentSrc: BASE64 '"' '"' | HTTP '"' '"' ? ? | LINK | S3 ? | AZURE ? | GCS ? linkName: connStr: base64: url: cred: CREDENTIALS '"' '"' credSpec: config: CONFIG '"' '"' configSpec: s3_config: S3 { '' | '' } [CONFIG ''] CREDENTIALS '' azure_config: AZURE { '' | '' | '' } CREDENTIALS '' [CONFIG ''] gcs_config: GCS { '' | '' } CREDENTIALS '' [CONFIG ''] ``` Here's an expansion of the syntax for an explicitly-typed Wasm TVF: ```sql CREATE FUNCTION '(' ')' RETURNS TABLE '(' ')' AS WASM (ABI CANONICAL) (GROW TO ) (USING EXPORT ‘"’ ‘"’) ``` Here's an expansion of the syntax for an implicitly-typed Wasm TVF using Canonical ABI: ```sql CREATE FUNCTION RETURNS TABLE AS WASM (ABI CANONICAL) (GROW TO ) (WITH WIT ) (USING EXPORT ‘"’ ‘"’) ``` ## Remarks * The Wasm function must return an `ARRAY` of `RECORDS`, specified as `list` in the WIT specification. * You can call a Wasm TVF in the `FROM` clause of a `SELECT` statement. * The implicitly-typed and explicitly-typed Wasm TVFs default to the Canonical ABI if the `ABI` clause is omitted in the `CREATE FUNCTION` statement. * The WIT document specification must be specified. * The `GROW TO` clause is optional for the Canonical ABI. The default size is `16 MB` (256 Wasm pages). ## Wasm TVF Sources SingleStore supports Wasm TVFs created from the following sources in the `CREATE FUNCTION` statement: * **Base-64 encoded string**: You can pass the contents of a Wasm module directly to the `CREATE FUNCTION` statement as an inline base-64 encoded string, see Examples. The maximum size of the `CREATE FUNCTION` statement can be modified using the `max_allowed_packet` global variable. * **Remote URL**: To download the content of a Wasm module from a remote endpoint, you can specify the URL of the endpoint using the `HTTP` clause. Here's an example: ```sql CREATE FUNCTION wasm_udf AS WASM FROM HTTP 'https://url/to/udf.wasm' ... ``` Optionally, you can specify the `CONFIG` and `CREDENTIALS` clauses for the URL. The `LINK` object is not supported with `HTTP`, and the credential configuration must be provided inline in the `CREATE FUNCTION` command. * **Cloud Provider**: You can download the content of a Wasm module from the following cloud providers: S3, Azure, and GCS. You also can specify the URL and connection credentials using the `LINK` clause, see Examples. For examples on using the `CONFIG ` clause, see [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md). > **📝 Note**: While not officially supported by SingleStore, you may use the `pushwasm` tool to upload the `.wit` and `.wasm` files to your cluster and create Wasm UDFs, UDAFs, or TVFs. For more information, refer to [pushwasm](https://github.com/singlestore-labs/pushwasm). ## Wasm UDF/TVF Naming Convention You can specify a Wasm UDF/TVF name that is different from the WIT specification in the `CREATE FUNCTION` statement. When specifying a custom name for the Wasm UDF/TVF, you may use any name that is valid for a database UDF/TVF. This will be the Wasm function's alias in the database. Additionally, specify the real name of the Wasm function, as declared in the WIT file, in the `USING EXPORT` clause. This name is case-sensitive and can contain hyphens. For example, consider the following WIT specification: ```wit run: func(pass1: f32, pass2: f32, pass3: f32) -> f32 ``` The following example specifies a custom name calc\_score: ```sql CREATE FUNCTION calc_score AS WASM FROM LINK rec 'module/run.wasm' WITH WIT FROM LINK rec 'module/run.wit' USING EXPORT "run"; ``` You can use distinct aliases to create multiple UDFs/TVFs from the same Wasm function. If the `USING EXPORT` clause is omitted from the `CREATE FUNCTION` statement: * When using a WIT file, the name of the Wasm UDF/TVF **must** match the name of the corresponding Wasm function as declared in the WIT IDL, after substituting hyphens with underscores. * When you are not using a WIT file, the name of the Wasm UDF/TVF **must** match the name of the function exported from the module. This is usually the same function name declared in the source language. If the `USING EXPORT` clause is omitted and the Wasm module has functions with the same name but different case, the `CREATE FUNCTION` statement returns an error. You cannot import functions with matching case-insensitive names from a Wasm module. ## Examples ## Example 1 The following example creates a Wasm TVF (in Rust) that splits a string and returns the two sub-strings. ## Setup VS Code 1. Install the **Dev Containers** extension in VS Code. 2. Open the **Command Palette**, and select **Dev Containers: Open Folder in Container**. 3. Navigate to the directory where you want to build your project, and select **Open**. 4. On the **Select Features** dialog, from the list, select the **Rust** container. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta1d0529adb1d5481/6a7ada53f682571a3458d928/wasm-rust-container-IjzMPJ.png) 5. Use defaults for other configuration items. 6. After VS Code completes building the project, open a terminal. Install the `cargo-wasi` subcommand to build the code for `wasm32-wasi` target. Refer to [cargo wasi](https://github.com/bytecodealliance/cargo-wasi) for more information. ```shell cargo install cargo-wasi ``` Proceed with the example. ## Create a Wasm TVF 1. Initialize a Cargo package in this directory using the following command: ```shell cargo init --vcs none --lib ``` 2. Update the **Cargo.toml** file with the following configuration: ```TOML [package] name = "split" version = "0.1.0" edition = "2018" [dependencies] wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen.git", rev = "60e3c5b41e616fee239304d92128e117dd9be0a7" } [lib] crate-type = ["cdylib"] ``` 3. Create a **.wit** file with the WIT specification, for example **split.wit**. ```wit record subphrase { str: string, idx: s32 } split-str: func(phrase: string, delim: string) -> list ``` 4. Add the following code to the **src/lib.rs** file: ```Rust wit_bindgen_rust::export!("split.wit"); struct Split; use crate::split::Subphrase; impl split::Split for Split { fn split_str(phrase: String, delim: String) -> Vec { phrase .split(&delim) .scan(0, |idx, s| { let current = Subphrase { str: s.to_string(), idx: *idx as i32 }; *idx += (s.len() + delim.len()) as i32; Some(current) }) .collect() } } ``` 5. Compile the program into a Wasm module using the following command: ```shell cargo wasi build --lib ``` The **split.wasm** file is created in the **target/wasm32-wasi/debug/** directory. > **📝 Note**: When your code is debugged and ready for production, add the `--release` flag to the `cargo build` command instead to enable further optimizations and remove the debug info. For example,```shell > cargo build --lib --release > ```The resulting **split.wasm** file is created in the **target/wasm32-wasi/release/** directory. 6. Upload the **split.wasm** and **split.wit** files to your cloud storage account, say an S3 bucket named **wasm-modules**. ## Load the Wasm TVF Execute the following command in the SingleStore command line: ```sql CREATE DATABASE wasm_tutorial; USE wasm_tutorial; CREATE FUNCTION split_str RETURNS TABLE AS WASM FROM S3 'wasm-modules/split.wasm' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}' WITH WIT FROM S3 'wasm-modules/split.wit' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}'; ``` This command imports the **split.wit** and **split.wasm** files created earlier in this example. The Wasm TVF is now ready to use. For example, ```sql SELECT * FROM split_str('wasm_rocks_the_house', '_'); ``` ```output +-------+-----+ | str | idx | +-------+-----+ | wasm | 0 | | rocks | 5 | | the | 11 | | house | 15 | +-------+-----+ ``` ## Example 2 You can use a Wasm TVF in the `FROM` clause of a query, i.e., you can specify the values from a table on the left of the Wasm TVF in the `FROM` clause as arguments to the Wasm TVF. Use a comma or the `JOIN` keyword before the Wasm TVF name in the `FROM` clause. This feature performs a an implicit lateral join between the expression on the left of the Wasm TVF and the results of the Wasm TVF. The following example code uses the `split_str()` Wasm TVF from Example 1 and the **strg** table: ```sql SELECT * FROM strg; ``` ```output +------------+ | Name | +------------+ | anna_moore | | adam_west | | sam_duke | +------------+ ``` ```sql SELECT Name, str, idx FROM strg JOIN split_str(Name, '_'); ``` ```output +------------+-------+-----+ | Name | str | idx | +------------+-------+-----+ | anna_moore | anna | 0 | | anna_moore | moore | 5 | | sam_duke | sam | 0 | | sam_duke | duke | 4 | | adam_west | adam | 0 | | adam_west | west | 5 | +------------+-------+-----+ ``` For more examples on using Wasm TVFs, refer to [SingleStore Wasm Toolkit](https://github.com/singlestore-labs/singlestore-wasm-toolkit). *** Modified at: January 25, 2024 Source: [/cloud/reference/code-engine-powered-by-wasm/create-wasm-tvfs/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-tvfs/) (An index of the documentation is available at /llms.txt) --- # Create Wasm UDFs You can create Wasm UDFs using the `CREATE FUNCTION` statement. SingleStore also supports Wasm-based user-defined aggregate functions (UDAFs). Refer to [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md) for information on creating Wasm-based UDAFs. For information on creating PSQL user-defined scalar-valued functions, refer to [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md). ## Syntax ```sql CREATE [OR REPLACE] FUNCTION function_name { | } (USING EXPORT '"' '"') explicitTypedPred: AS WASM (ABI ) (GROW TO ) FROM implicitTypedPred: AS WASM (ABI ) (GROW TO ) FROM (WITH WIT FROM ) function_name: typeSpec: '(' ? ')' RETURNS abiSpec: BASIC | CANONICAL numBytes: [0-9]+ paramList: (',' )* param: result: TABLE () | varName: contentSrc: BASE64 '"' '"' | HTTP '"' '"' ? ? | LINK | S3 ? | AZURE ? | GCS ? linkName: connStr: base64: url: cred: CREDENTIALS '"' '"' credSpec: config: CONFIG '"' '"' configSpec: s3_config: S3 { '' | '' } [CONFIG ''] CREDENTIALS '' azure_config: AZURE { '' | '' | '' } CREDENTIALS '' [CONFIG ''] gcs_config: GCS { '' | '' } CREDENTIALS '' [CONFIG ''] ``` Here's an expansion of the syntax for an explicitly-typed Wasm UDF: ```sql CREATE FUNCTION AS WASM (ABI ) (GROW TO ) (USING EXPORT ‘"’ ‘"’) ``` Here's an expansion of the syntax for an implicitly-typed Wasm UDF using Canonical ABI: ```sql CREATE FUNCTION AS WASM (ABI CANONICAL) (GROW TO ) (WITH WIT ) (USING EXPORT ‘"’ ‘"’) ``` Here's an expansion of the syntax for an implicitly-typed Wasm UDF using Basic ABI: ```sql CREATE FUNCTION AS WASM ABI BASIC (GROW TO ) (USING EXPORT ‘"’ ‘"’) ``` ## Remarks * When using the Canonical ABI, an implicitly-typed Wasm UDF automatically infers the data type information from the WIT specification. * When using the Basic ABI, an implicitly-typed Wasm UDF infers the data type information from the function's signature. * The explicitly-typed Wasm UDF allows you to specify your own data types and maps them to the corresponding type supported by the current ABI. See [Wasm Data Type Coercions](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/#section-idm454386583893923310231399493.md) for more information. > **⚠️ Warning**: Using inappropriate data types may cause the UDF to return unpredictable results or an error. * The implicitly-typed Wasm UDF defaults to the Canonical ABI if the ABI clause is not specified in the `CREATE FUNCTION` statement. * The explicitly-typed Wasm UDF does not support WIT document specification. * The Basic ABI does not support WIT document specification. * The implicitly-typed Wasm UDF does not support the specification of parameters or return types in a UDF. * The WIT document specification must be specified in an implicitly-typed Wasm UDF that uses Canonical ABI. * The `GROW TO` clause is optional for both the Canonical ABI and Basic ABI. The default size is `16 MB` (256 Wasm pages). ## Wasm UDF Sources SingleStore supports Wasm UDFs created from the following sources in the `CREATE FUNCTION` statement: * **Base-64 encoded string**: You can pass the contents of a Wasm module directly to the `CREATE FUNCTION` statement as an inline base-64 encoded string, see Examples. The maximum size of the `CREATE FUNCTION` statement can be modified using the `max_allowed_packet` global variable. * **Remote URL**: To download the content of a Wasm module from a remote endpoint, you can specify the URL of the endpoint using the `HTTP` clause. Here's an example: ```sql CREATE FUNCTION wasm_udf AS WASM FROM HTTP 'https://url/to/udf.wasm' ... ``` Optionally, you can specify the `CONFIG` and `CREDENTIALS` clauses for the URL. The `LINK` object is not supported with `HTTP`, and the credential configuration must be provided inline in the `CREATE FUNCTION` command. * **Cloud Provider**: You can download the content of a Wasm module from the following cloud providers: S3, Azure, and GCS. You also can specify the URL and connection credentials using the `LINK` clause, see Examples. For examples on using the `CONFIG ` clause, see [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md). > **📝 Note**: While not officially supported by SingleStore, you may use the `pushwasm` tool to upload the `.wit` and `.wasm` files to your cluster and create Wasm UDFs, UDAFs, or TVFs. For more information, refer to [pushwasm](https://github.com/singlestore-labs/pushwasm). ## Wasm UDF/TVF Naming Convention You can specify a Wasm UDF/TVF name that is different from the WIT specification in the `CREATE FUNCTION` statement. When specifying a custom name for the Wasm UDF/TVF, you may use any name that is valid for a database UDF/TVF. This will be the Wasm function's alias in the database. Additionally, specify the real name of the Wasm function, as declared in the WIT file, in the `USING EXPORT` clause. This name is case-sensitive and can contain hyphens. For example, consider the following WIT specification: ```wit run: func(pass1: f32, pass2: f32, pass3: f32) -> f32 ``` The following example specifies a custom name calc\_score: ```sql CREATE FUNCTION calc_score AS WASM FROM LINK rec 'module/run.wasm' WITH WIT FROM LINK rec 'module/run.wit' USING EXPORT "run"; ``` You can use distinct aliases to create multiple UDFs/TVFs from the same Wasm function. If the `USING EXPORT` clause is omitted from the `CREATE FUNCTION` statement: * When using a WIT file, the name of the Wasm UDF/TVF **must** match the name of the corresponding Wasm function as declared in the WIT IDL, after substituting hyphens with underscores. * When you are not using a WIT file, the name of the Wasm UDF/TVF **must** match the name of the function exported from the module. This is usually the same function name declared in the source language. If the `USING EXPORT` clause is omitted and the Wasm module has functions with the same name but different case, the `CREATE FUNCTION` statement returns an error. You cannot import functions with matching case-insensitive names from a Wasm module. ## Examples ## Example 1 - Create a Wasm UDF Using Rust The following example creates a Wasm UDF that computes the expression **x^y** in Rust and builds the project in Visual Studio Code using the `wasm32-wasi` target.. This example imports the Wasm module from an S3 bucket ## Setup VS Code 1. Install the **Dev Containers** extension in VS Code. 2. Open the **Command Palette**, and select **Dev Containers: Open Folder in Container**. 3. Navigate to the directory where you want to build your project, and select **Open**. 4. On the **Select Features** dialog, from the list, select the **Rust** container. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta1d0529adb1d5481/6a7ada53f682571a3458d928/wasm-rust-container-IjzMPJ.png) 5. Use defaults for other configuration items. 6. After VS Code completes building the project, open a terminal. Install the `cargo-wasi` subcommand to build the code for `wasm32-wasi` target. Refer to [cargo wasi](https://github.com/bytecodealliance/cargo-wasi) for more information. ```shell cargo install cargo-wasi ``` Proceed with the example. ## Create a Wasm UDF 1. Initialize a Cargo package in this directory using the following command: ```shell cargo init --vcs none --lib ``` 2. Update the **Cargo.toml** file with the following configuration: ```TOML [package] name = "powerof" version = "0.1.0" edition = "2018" [dependencies] wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen.git", rev = "60e3c5b41e616fee239304d92128e117dd9be0a7" } [lib] crate-type = ["cdylib"] ``` 3. Create a **.wit** file with the WIT specification, for example **powerof.wit**. ```wit power-of: func(base: s32, exp: s32) -> s32 ``` This specifies that the `power-of` function accepts two 32-bit integers and returns a single 32-bit integer. 4. Add the following code to the **src/lib.rs** file: ```Rust wit_bindgen_rust::export!("powerof.wit"); struct Powerof; impl powerof::Powerof for Powerof { fn power_of(base: i32, exp: i32) -> i32 { let mut res = 1; for _i in 0..exp { res *= base; } res } } ``` 5. Compile the program into a Wasm module using the following command: ```shell cargo wasi build --lib ``` The **powerof.wasm** file is created in the **target/wasm32-wasi/debug/** directory. > **📝 Note**: When your code is debugged and ready for production, add the `--release` flag to the `cargo build` command instead to enable further optimizations and remove the debug info. For example,```shell > cargo wasi build --lib --release > ```The resulting **powerof.wasm** file is created in the **target/wasm32-wasi/release/** directory. 6. Upload the **powerof.wasm** and **powerof.wit** files to your cloud storage account, say an S3 bucket named **wasm-modules**. ## Load the Wasm UDF Execute the following command in the SingleStore command-line: ```sql CREATE DATABASE wasm_test; USE wasm_test; CREATE FUNCTION power_of AS WASM FROM S3 'wasm-modules/powerof.wasm' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}' WITH WIT FROM S3 'wasm-modules/powerof.wit' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}'; ``` This command imports the **powerof.wit** and **powerof.wasm** files created earlier in this example. The Wasm UDF is now ready to use. For example, ```sql SELECT power_of(2,4) AS Result; ``` ```output +-----------+ | Result | +-----------+ | 16 | +-----------+ ``` ## Example 2 - Create a Wasm UDF Using C The following example creates a Wasm UDF that computes the expression **x^y** using the C language. See *Setup VS Code* in the previous example to build this example in VS Code and include the required dependencies, for example `wit-bindgen`. 1. Create a **.wit** file with the WIT specification, for example **power.wit**. ```wit power-of: func(base: s32, exp: s32) -> s32 ``` This specifies that the power-of function accepts two 32-bit integers as arguments and returns a single 32-bit integer. 2. Run `wit-bindgen` on your IDL to generate Wasm bindings. ```shell wit-bindgen c --export power.wit ``` 3. Add the following code to the source file (**power.c**): ```c int32_t power_of(int32_t base, int32_t exp) { int32_t res = 1; for (int32_t i = 0; i < exp; ++i) { res *= base; } return res; } ``` 4. Compile the program into a Wasm module using the following command: ```shell clang \ --target=wasm32-unknown-wasi \ -mexec-model=reactor \ -s \ -I. \ -o power.wasm \ power.c ``` This command generates the **power.wasm** file that contains the Wasm module. 5. Upload the **power.wasm** and **power.wit** files to your cloud storage account, say an S3 bucket named **wasm-modules**. 6. Run the following command in SingleStore command-line to load the Wasm module: ```sql CREATE FUNCTION power_of AS WASM FROM S3 'wasm-modules/power.wasm' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}' WITH WIT FROM S3 'wasm-modules/power.wit' CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}'; ``` The Wasm module is now ready to use. For example, ```sql SELECT power_of(4,2) AS Result; ``` ```output +-----------+ | Result | +-----------+ | 16 | +-----------+ ``` ## Example 3 - Load Wasm Using a Base-64 encoded String The following example shows how to load a Wasm module created in C, which is encoded to a base-64 encoded string: 1. Add the following code to the source file, say **mult.c**: ```c int mult(int a,int b) { return a * b; } ``` 2. Compile the program into a Wasm module using the following command: ```shell clang \ --target=wasm32-unknown-wasi \ -mexec-model=reactor \ -Wl,--export-all \ -o mult.wasm mult.c ``` 3. To encode a Wasm file to a base-64 encoded string, run the following command (in Linux): ```shell base64 -w 0 mult.wasm ``` ```output AGFzbQEAAAABCgJgAABgAn9/AX8DBAMAAAEEBQFwAQEBBQMBAAIGKwd/AUGAiAQLfwBBgAgLfwBBgAgLfwBBgAgLfwBBgIgEC38AQQALfwBBAQsHqAELBm1lbW9yeQIAEV9fd2FzbV9jYWxsX2N0b3JzAAALX2luaXRpYWxpemUAARlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQAEbXVsdAACDF9fZHNvX2hhbmRsZQMBCl9fZGF0YV9lbmQDAg1fX2dsb2JhbF9iYXNlAwMLX19oZWFwX2Jhc2UDBA1fX21lbW9yeV9iYXNlAwUMX190YWJsZV9iYXNlAwYKSwMCAAsIABCAgICAAAs9AQZ/I4CAgIAAIQJBECEDIAIgA2shBCAEIAA2AgwgBCABNgIIIAQoAgwhBSAEKAIIIQYgBSAGbCEHIAcPCw== ``` Copy the base-64 encoded from the standard output (`stdout`). 4. Run the following command to load the Wasm module: ```sql CREATE FUNCTION mult AS WASM ABI BASIC FROM BASE64 'AGFzbQEAAAABCgJgAABgAn9/AX8DBAMAAAEEBQFwAQEBBQMBAAIGKwd/AUGAiAQLfwBBgAgLfwBBgAgLfwBBgAgLfwBBgIgEC38AQQALfwBBAQsHqAELBm1lbW9yeQIAEV9fd2FzbV9jYWxsX2N0b3JzAAALX2luaXRpYWxpemUAARlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQAEbXVsdAACDF9fZHNvX2hhbmRsZQMBCl9fZGF0YV9lbmQDAg1fX2dsb2JhbF9iYXNlAwMLX19oZWFwX2Jhc2UDBA1fX21lbW9yeV9iYXNlAwUMX190YWJsZV9iYXNlAwYKSwMCAAsIABCAgICAAAs9AQZ/I4CAgIAAIQJBECEDIAIgA2shBCAEIAA2AgwgBCABNgIIIAQoAgwhBSAEKAIIIQYgBSAGbCEHIAcPCw=='; ``` The Wasm module is now ready to use. For example, ```sql SELECT mult(4,2) AS Result; ``` ```output +-----------+ | Result | +-----------+ | 8 | +-----------+ ``` ## Example 4 - Load Wasm Using LINK The following example shows how to store the credentials in a `LINK` object and use it in the `CREATE FUNCTION` statement: ```sql CREATE LINK accessLink AS S3 CREDENTIALS '{ "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]" }' CONFIG '{"region": "us-east-1"}'; CREATE FUNCTION power_of AS WASM FROM LINK accessLink 'wasm-modules/power.wasm' WITH WIT FROM LINK accessLink 'wasm-modules/power.wit'; ``` ## Example References For more examples, refer to [SingleStore Wasm Toolkit](https://github.com/singlestore-labs/singlestore-wasm-toolkit). ## Troubleshooting * If a Wasm UDF returns a backtrace indicating an out-of-memory exception, increase the amount of linear memory available. To increase the amount of linear memory available: * Re-create the UDF with a higher value for `GROW TO`, and/or * Increase the value of `wasm_max_linear_memory_size` global variable. * To list all the Wasm UDFs for the current database, use the `SHOW FUNCTIONS` command. For Wasm UDFs, the `Function Type` is `User Defined Function` and the `Runtime Type` field is `Wasm`. * To view the SQL command used to create a Wasm UDF, use the `SHOW CREATE FUNCTION` command. Since the body of the Wasm module is binary and large, it is omitted in the output. For example, ```sql CREATE OR REPLACE FUNCTION mywasm(input int(11) NOT NULL) RETURNS int(11) NOT NULL AS WASM ABI CANONICAL GROW TO 16777216; ``` * To view detailed resource metrics on a command using Wasm UDFs, use the `PROFILE` command. * SingleStore provides a Rust crate that can act as a debugger. For more information see [Wasm Remote Debugger Service](https://github.com/singlestore-labs/singlestore-wasm-toolkit/tree/main/crates/debugger). * If a Wasm code generates an exception, use the `SHOW WASM BACKTRACE` command to view the complete backtrace from the error message. For example, ```sql SHOW WASM BACKTRACE \G ``` ```output * 1. row * Backtrace: wasm trap: wasm `unreachable` instruction executed wasm backtrace: 0: 0xe00d - !abort 1: 0x7aaa - std::sys::wasi::abort_internal::hb387f3c4e8f42ceb at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys/wasi/mod.rs:85:14 2: 0x9be4 - std::process::abort::hdfad6720ede82433 at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/process.rs:2059:5 3: 0xa0b9 - rust_oom at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/alloc.rs:341:5 -------- output truncated -------- ``` *** Modified at: June 13, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs/) (An index of the documentation is available at /llms.txt) --- # Improve Wasm Function Performance Here are some examples of ways to improve the performance of your Wasm code: * Compile the Wasm function in "release" mode and not in debug mode. To compile in release mode, * In Rust, pass the `--release` flag to cargo, and * In C/C++, pass the `-O3` option to the compiler. * If your program can benefit by building data structures in advance, you may improve performance by maintaining a cache in your Wasm program. For example, you can cache reusable data between function calls. The following example caches compiled regular expressions so it does not have to compile the regular expression argument again for every row passed to the function: ```rust wit_bindgen_rust::export!("s2regex.wit"); use regex::Regex; use std::cell::RefCell; use std::collections::HashMap; struct S2regex; thread_local! { static COMPILED_RGXS: RefCell> = RefCell::new(HashMap::new()); } impl s2regex::S2regex for S2regex { fn capture(input: String, pattern: String) -> String { COMPILED_RGXS.with(|c| { let mut map = c.borrow_mut(); let re = map .entry(pattern) .or_insert_with_key(|pattern| Regex::new(pattern).unwrap()); re.captures(&input) .and_then(|c| c.get(1)) .map(|c| c.as_str()) .unwrap_or_default() .to_string() }) } } ``` The following query has to compile the regular expression only for the first row: ```sql SELECT * FROM tbl WHERE s2regex(tbl.col, '') = "val"; ``` *** Modified at: March 25, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/improve-wasm-function-performance/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/improve-wasm-function-performance/) (An index of the documentation is available at /llms.txt) --- # Troubleshooting **How to find bug(s) in the Wasm code when it is called from SQL?** Debug the code outside of the database. You may use a test harness, or single-step through the code in a debugger. *** Modified at: March 25, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/troubleshooting/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/troubleshooting/) (An index of the documentation is available at /llms.txt) --- # Wasm Application Binary Interfaces (ABIs) When a Wasm function is created, you can specify one of the following Application Binary Interfaces (ABIs): Basic or Canonical. The default ABI type is **Canonical**, unless specified explicitly. * **Basic**: This is the bare-bones Wasm ABI. It supports only 32-bit and 64-bit integers and floating point numbers. * **Canonical**: This ABI type is defined by the [Canonical ABI](https://github.com/WebAssembly/interface-types/blob/40f157ad429772c2b6a8b66ce7b4df01e83ae76d/proposals/interface-types/CanonicalABI.md) specification (a part of the [Interface Types](https://github.com/WebAssembly/interface-types) proposal). It is a superset of Basic ABI, and it allows usage of structured and complex interface types. SingleStore supports both the Basic and Canonical Wasm Application Binary Interfaces (ABIs). * Using the Basic ABI, SingleStore supports any language that can compile to Wasm code, if the UDF accepts and returns only numeric data types. * Using the Canonical ABI, you can pass and return complex data types from UDF/TVF. SingleStore currently supports the following languages for Canonical ABI: * C/C++ * Rust > **📝 Note**: The user must ensure that the specified ABI type matches the Wasm module implementation. A mismatch may lead to unexpected results at runtime. ## Canonical ABI The canonical ABI is a proposed part of [WASI](https://github.com/WebAssembly/WASI). To use the canonical ABI, you must specify additional metadata in the form of a [WIT IDL](https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md) string. The `wit-bindgen` tool, which generates bindings for the Canonical ABI, declares function and structure names as hyphenated strings. Using hyphenated names for UDFs/TVFs requires them to be enclosed in backticks (`` ` ``), Hence, all function, parameter, and record field names loaded from the Wasm modules are renamed to use underscores instead of hyphens. For example, consider the following Wasm UDF specified by the WIT definition: ```WIT user-function-name: func() -> string; ``` To call this Wasm UDF, we use the following syntax: ```sql SELECT user_function_name(); ``` *** Modified at: August 4, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/wasm-application-binary-interfaces-abis/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/wasm-application-binary-interfaces-abis/) (An index of the documentation is available at /llms.txt) --- # Wasm Data Type Coercions The following table describes how database types are coerced to and from Wasm ABI types when using explicitly-typed UDFs/TVFs. Empty cells indicate that automatic coercion is not available currently, and you must cast that SingleStore type to another data type for which data type conversion is available. | **SingleStoreData Type** | **Wasm Basic ABI Type** | **Wasm Canonical ABI Type** | | ------------------------ | ----------------------- | --------------------------- | | ARRAY | | list<...> | | BIGINT | i64 | i64 | | BINARY | u32 | u32 | | BINARY(...) | | list\ | | BIT | | | | BLOB | | list\ | | BOOL | i32 | u8 | | CHAR | i32 | char | | CHAR(...) | | string | | DATE | | | | DATETIME | | | | DATETIME(6) | | | | DECIMAL | | | | DOUBLE | f64 | float64 | | ENUM | | | | FLOAT | f32 | float32 | | GEOGRAPHY | | string (WKT format) | | GEOGRAPHYPOINT | | string (WKT format) | | INT | i32 | i32 | | JSON | | string | | LONGBLOB | | list\ | | LONGTEXT | | string | | MEDIUMBLOB | | list\ | | MEDIUMINT | i32 | i32 | | MEDIUMTEXT | | string | | RECORD | | record | | SET | | | | SMALLINT | i32 | i16 | | TEXT | | string | | TIME | | | | TIME(6) | | | | TIMESTAMP | i64 | i64 | | TIMESTAMP(6) | i64 | i64 | | TINYBLOB | | list\ | | TINYINT | i32 | i8 | | TINYTEXT | | string | | VARBINARY | | list\ | | VARCHAR | | string | | YEAR | i32 | i32 | ## Remarks * Wasm does not support Base-10 numeric or date-time data types. You must specify/cast these data types to ABI-supported data types. * The `GEOGRAPHY` and `GEOGRAPHYPOINT` data types are converted to strings using the [WKT representation](https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry). * SingleStore recommends casting the `ENUM` data type to strings. * Coercions to/from canonical ABI string types require that the data be encoded using multi-byte UTF-8 (`utf8mb3` or `utf8mb4` collations). *** Modified at: March 25, 2025 Source: [/cloud/reference/code-engine-powered-by-wasm/wasm-data-type-coercions/](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/wasm-data-type-coercions/) (An index of the documentation is available at /llms.txt) --- # Configuration Reference ## Overview > **📝 Note**: In SingleStore Helios, you cannot set global variables or sync variables. This section lists the [cluster, database, and table limits](https://docs.singlestore.com/cloud/reference/configuration-reference/system-limits.md) for your SingleStore Helios instance. ## In this section * [AVX2 Instruction Set Verification](https://docs.singlestore.com/cloud/reference/configuration-reference/avx-2-instruction-set-verification.md) * [Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables.md) * [System Limits](https://docs.singlestore.com/cloud/reference/configuration-reference/system-limits.md) *** Modified at: September 29, 2023 Source: [/cloud/reference/configuration-reference/](https://docs.singlestore.com/cloud/reference/configuration-reference/) (An index of the documentation is available at /llms.txt) --- # AVX2 Instruction Set Verification SingleStore Helios uses AVX2 on Intel and Intel-compatible processors. *** Modified at: April 12, 2024 Source: [/cloud/reference/configuration-reference/avx-2-instruction-set-verification/](https://docs.singlestore.com/cloud/reference/configuration-reference/avx-2-instruction-set-verification/) (An index of the documentation is available at /llms.txt) --- # Engine Variables Engine variables are configuration settings that control various aspects of the database's behavior. Engine variables fine-tune the performance and behavior of SingleStore to best suit specific workload requirements. SingleStore defines engine variables as either sync or non-sync. Sync variables are cluster-wide variables that you set on the master aggregator. When you set the variable, the value is propagated across the cluster to all relevant nodes. The value is persisted in the internal cluster database. Non-sync variables are set locally, independently on each node. The value does not persist in the engine. Engine variables can be set globally or per session in scope. Global engine variables affect all connections to the database. If a global engine variable value is changed, then the change affects all existing and subsequent connections. Session engine variables affect the individual client connections they are set on. Session engine variables can also be set globally. After setting a session variable globally via the `SET GLOBAL` command, client connection pools should be flushed to make the new value effective throughout a cluster. Engine variables that can be set per session and globally are denoted in the “Description” column of the [list of variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) table by the verbiage “This is a session variable that can also be set globally”. Otherwise, the engine variable can only be set globally. | | **Sync** | **Non-Sync** | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Global** | Sync global variables are cluster-wide engine variables that are set on the master aggregator and their values are propagated across the cluster to all relevant nodes. Their values are effective for the current connection to the node and any other existing or subsequent connections. The values persist across restarts. | Non-sync global engine variables are set locally and independently on each node. Their values are effective for the current connection to the node and any other existing or subsequent connections. The values do not persist across restarts. | | **Session** | Sync session variables are cluster-wide engine variables that are set on the master aggregator and their values are propagated across the cluster to all relevant nodes if set with`SET GLOBAL`. If set with`SET`(or`SET SESSION`), the values will only change the copy if the variable is local to the connection. Their values are specific to each session and do not affect other connections. | Non-sync session engine variables are set locally and independently on each node. Their values are specific to each session and do not affect other connections. | The following sections discuss these variables and how to set them. ## In this section * [Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/sync-variables.md) * [Non-Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/non-sync-variables.md) * [Reading Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/reading-variables.md) * [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) * [Assigning Expressions to Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/assigning-expressions-to-variables.md) * [Reserved Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/reserved-engine-variables.md) *** Modified at: February 18, 2025 Source: [/cloud/reference/configuration-reference/engine-variables/](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/) (An index of the documentation is available at /llms.txt) --- # System Limits This topic specifies maximum sizes and numbers of various SingleStore Helios components. Use the following links to navigate to a section below: ## Cluster-Level Limits | Components | Limit | | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Maximumclusters in a group | 5 | | Maximum number ofclusters to which a database can be attached in R/W mode | 1 | | Maximumclustersize | S-384 (vCPU: 3072; Memory: 24576 GB)Refer to the[pricing](https://www.singlestore.com/cloud-pricing/#compute)page for more details. | | Number of databases percluster | No fixed limit. Recommended limit: 20.It depends on the size of each database. Databases are sized differently based on the number of partitions, amount of data, metadata and number of tables, etc.Refer to[SingleStore Helios Sizing Guidelines](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/singlestore-helios-sizing-guidelines.md)for more details. | | Maximum number of concurrent queries | This varies according to the deployment size. | | Maximum simultaneous client connections | This varies according to the deployment size. | ## Database-Level Limits | Item | Max Spec (enforced) | Max Spec (recommended) | Comments | | -------------------------------------- | ------------------- | ---------------------- | ------------------------------------------------ | | Number of tables in a database | | | Each table costs some per-table memory overhead. | | Number of partitions in acluster | None | | | | Number of database partitions per node | 2414 | | | ## Table-Level Limits | Item | Max Spec (enforced) | Max Spec (recommended) | Comments | | -------------------------------------------------------------------- | -------------------------------------------------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Number of columns per table | 4096 | | | | Row size in bytes | 64KB \* | | Limit does not apply to variable-length strings such as VARCHAR, VARBINARY, TEXT, etc. | | Length of names in characters (applies to table names, column names) | 256 | | | | Number of indexes per table | 63 | | | | Number of columns per index | 32 | | | | Number of tables in a join for each SELECT | 253 | | | | Number of entries in a`SELECT`projection (target) list | 12800 | | | | Number of nested sub-select queries using the`UNION`statement | 337 | 40 | Beyond 40 the`UNION`statements are converted to`TABLE`function. | | Number of parameters per query | 1,048,576 | | Configurable via`parametrizer_query_max_params`. Refer to[List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md)for more information. | | Function Arguments | Non-external functions and stored procedures: 256External functions: 4,096 | | | \* The number of fields that will fit in the 64KB maximum row size is determined by the field types and how rows are laid out in the in-memory rowstore format. In some cases, writes (inserts and updates) to columnstore tables are implemented by storing the newly written rows in a rowstore format before flushing them to the column-oriented format. The row size limit applies to fixed-width data which includes a 4-byte nullability flag for nullable fields. The row size limit does not include the body of variable-width fields, but only the in-row portion, which includes an 8-byte memory location pointer and an optional 4-byte nullability flag. The sum of the in-row data size for all columns cannot exceed 64KB." ## Naming Constraints | Item | Constraints | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | General |
  • Name cannot be blank.
  • Name cannot end with a space.
| | Database names |
  • Database names can only contain alphanumeric characters and underscores, and cannot start with a digit.
  • Maximum length of 64 characters.
| | Table names |
  • Table names can contain both alphanumeric and non-alphanumeric characters, but cannot contain the / character.
  • Maximum length of 256 characters. **
| | Other database objects (stored procedures, views, user-defined functions, user-defined aggregate functions, and table-valued functions) | Maximum length of 256 characters. \*\* | | User names |
  • Non-JWT users: Maximum length of 32 characters.
  • JWT users: Maximum length of 320 characters.
| | Role names | No length limit. Cannot contain commas or spaces. | | Group names | No length limit. Cannot contain commas or spaces. | \*\* Column aliases are truncated during projection (`SELECT`) to 256 bytes (not characters) which impacts names that use multi-byte characters. The full names are stored in the database and can be queried. Information schema tables can be used to view the full names. *** Modified at: June 29, 2026 Source: [/cloud/reference/configuration-reference/system-limits/](https://docs.singlestore.com/cloud/reference/configuration-reference/system-limits/) (An index of the documentation is available at /llms.txt) --- # Data API SingleStore provides the Data API to execute SQL statements against your database over an HTTP connection. The Data API can be used to integrate SingleStore with serverless architecture, develop custom applications, and build seamless integrations with applications. SingleStore's Data API is authenticated to the server using the [HTTP Basic and Bearer Authentication](https://docs.singlestore.com/cloud/reference/data-api/data-api-authentication.md) standards, which must be used in conjunction with SSL (HTTPS) for secure communication over the network. The Data API uses standard HTTP methods and response codes and supports URI versioning. It also uses JSON-encoded responses with support for conventional [data type conversions](https://docs.singlestore.com/cloud/reference/data-api/data-api-data-type-conversion.md). The Data API is accessible via a built-in HTTP server. The Data API supports all SQL statements that can be run in a direct database connection with SingleStore Helios. To learn about the supported Data API endpoints, refer to [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md). The Data API can support concurrent requests up to the value of the `max_connection_threads` engine variable. Refer to [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for more information. SingleStore does not recommend updating or managing session-related entities using the Data API, for example, temporary tables, session variables, etc. Temporary tables created using the Data API are not automatically dropped when a connection is closed. Hence, you must explicitly drop any temporary tables created using the Data API before closing a connection (from within the same connection). To drop any temporary tables that exist after the connection is closed, kill the connection associated with that temporary table using the following commands: ```sql -- Obtain the connection ID for the temporary table -- SHOW TEMPORARY TABLES; -- Use the connection ID to kill the connection -- KILL CONNECTION ; ``` ## FAQs * **Is this a REST API?** No, the Data API is a JSON-based SQL API. * **Is there a limit on the request body size?** The request body size must not exceed 1MB. * **Is there a limit on the response size?** The response is streamed not buffered. There are no limits explicitly set on the response size. * **How many concurrent requests are supported by the Data API?** The Data API can process up to 192 requests in parallel per aggregator, which is the default value for the `max_connection_threads` engine variable. To support additional requests, you can increase the value of this engine variable up to the maximum value supported by the database engine. * **Does it support API pagination?** By itself, the Data API does not implement any logic to handle pagination. However, users can build pagination into the queries they send to the API. ## In this section * [Data API Authentication](https://docs.singlestore.com/cloud/reference/data-api/data-api-authentication.md) * [Access the Data API](https://docs.singlestore.com/cloud/reference/data-api/access-the-data-api.md) * [Enable the Data API](https://docs.singlestore.com/cloud/reference/data-api/enable-the-data-api.md) * [Data API Data Type Conversion](https://docs.singlestore.com/cloud/reference/data-api/data-api-data-type-conversion.md) * [Data API Error Handling](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling.md) * [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md) *** Modified at: August 3, 2023 Source: [/cloud/reference/data-api/](https://docs.singlestore.com/cloud/reference/data-api/) (An index of the documentation is available at /llms.txt) --- # Access the Data API Data API can be accessed using Basic and Bearer Authentication methods at `protocol://hostname:port/api/version/` where: * `protocol` is HTTPS/HTTP; protocol must be HTTPS when SSL is enabled. * `hostname` represents the IP address or hostname of the SingleStore Helios deployment. Refer to [SingleStore Helios Endpoints](https://docs.singlestore.com/cloud/connect-to-singlestore/singlestore-helios-endpoints.md) to determine the endpoint (`host:port`) of your deployment. * `port` is the secure WebSocket proxy port configured. * `version` is the Data API version; currently v1 and v2 are supported. To access SingleStore Data API via Basic Authentication, you need to provide your SingleStore account credentials (username and password) in each HTTP request. The credentials must be passed as a Base-64 encoded `username:password` string in the Authorization header in the HTTP client. For example, the Authorization header of `demo:Afu4XjzB1ns` appears as follows. ```HTTP Header Authorization: Basic ZGVtbzpBZnU0WGp6QjFucw== ``` Using an HTTP client such as cURL, you can either pass your credentials as parameters in the `--user` or the `-u` option or add them before the `hostname` in the HTTP request, as shown below. ```shell curl -u "username:password" https://hostname:port/api/v2/exec ``` ```shell curl https://username:password@hostname:port/api/v2/exec ``` cURL encodes the `username:password` string using Base-64 and inserts it in the Authorization header before sending the HTTP request to the server. Here's an example of using Bearer Authentication to access the Data API via cURL: ```shell curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.C1um3oUNP4178sTG9WahtfZxN0-GqnYTPsbx-zhH9Qk" https://hostname:port/api/v2/exec ``` You can use API development tools and libraries to send Data API requests and handle responses. You can also test-run the Data API at the Linux command line. For more information on how to use the Data API, refer to [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md). *** Modified at: September 26, 2025 Source: [/cloud/reference/data-api/access-the-data-api/](https://docs.singlestore.com/cloud/reference/data-api/access-the-data-api/) (An index of the documentation is available at /llms.txt) --- # Data API Authentication SingleStore's Data API uses Basic and Bearer Authentication standards. You can also use JWTs for password-less access to the database with Bearer Authentication. To authenticate via JWTs, specify the JWT in the Bearer Authorization header. For successful authentication, the JWT must be signed using a key listed in the JWKS that is fetched from the `jwks_endpoint` in the engine. See [Authenticate via JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md) for more information. To enable JWT-based authentication on SingleStore Helios, * [Configure the JWKS](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt/#section-idm4592226252089633070902698635.md) endpoint. Set the `jwks_endpoint` variable on the database server. * Make a HTTP request to the `/api/v2/jwks_setup` endpoint using the `POST` method. A user agent can authenticate with the server by sending its credentials in an Authorization request header. The Authorization header contains the authentication method (Basic or Bearer) followed by a space and then the authentication information constructed from a Base-64 encoded string `username:password|JWT`. ```HTTP Header Authorization: [Basic | Bearer] ``` For example, the Basic Authorization header for the username `demo` and password `Afu4XjzB1ns` appears as follows, where `ZGVtbzpBZnU0WGp6QjFucw==` is the Base-64 encoding of the `demo:Afu4XjzB1ns` string. ```HTTP Header Authorization: Basic ZGVtbzpBZnU0WGp6QjFucw== ``` If the server requires the user agent to authenticate itself after receiving an unauthenticated request, it will respond with a *401 Unauthorized* status and the *WWW-Authenticate* header. > **📝 Note**: When using a third-party SQL client or development tool, you must first [add a database user](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) to log into a SingleStore Helios database. > **⚠️ Warning**: As the Basic and Bearer Authentication methods transfer the username and password (or JWTs) over the network in clear text, it must be used in conjunction with HTTPS/SSL for added security. The production usage of SingleStore's Data API should only take place with SSL (HTTPS) enabled to ensure that the authentication information is secure. *** Modified at: September 16, 2025 Source: [/cloud/reference/data-api/data-api-authentication/](https://docs.singlestore.com/cloud/reference/data-api/data-api-authentication/) (An index of the documentation is available at /llms.txt) --- # Data API Data Type Conversion SingleStore’s Data API returns JSON objects as responses. The following tables map the [SingleStore data types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) to the corresponding JSON data types. ## Data Type Mapping for `/api/v2/` | Category | SingleStore Data Type | JSON Data Type | Notes | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Boolean |
  • BOOL
  • TINYINT
| Number | A`TINYINT`value of`0`is considered`false`, and non-zero values are considered`true`. SingleStore returns`false`as`0`and`true`as`1`. | | Numeric |
  • TINYINT
  • SMALLINT
  • MEDIUMINT
  • INT
  • BIGINT
  • FLOAT
  • DOUBLE
| Number | JSON number data type supports both integers and floating point values, and they are returned as-is from SingleStore. For example,
  • 24
  • 25567
  • 3.14
  • 2345.45632
  • 1e20
  • 1.34e-10
The API supports 64-bit unsigned integer data type. | | JSON |
  • JSON
| JSON | JSON type values are directly added to the row tuple without any type conversion, which eliminates the need for additional parsing in later steps. For example,
  • { "hello": "world" }
  • [ 1, 2, null, 3 ]
  • "data"
  • 1.4
JSON null values and SQL null values cannot be differentiated when using the Data API. | | String and Geospatial |
  • DECIMAL
  • GEOGRAPHY
  • GEOGRAPHYPOINT
  • TEXT
  • CHAR
  • VARCHAR
  • TINYTEXT
  • MEDIUMTEXT
| String | JSON strings are enclosed within double quotes. For example,
  • "567.2389354"
  • "POINT(-74.04451396 40.68924403)"
  • "POLYGON((1 1,2 1,2 2, 1 2, 1 1))"
  • "Connectors"
| | Varbinary |
  • VARBINARY
  • LONGBLOB
  • MEDIUMBLOB
  • TINYBLOB
  • BLOB
| Base64-encoded string | SingleStore`VARBINARY`values are converted to base-64 encoded strings as JSON values. Hence, you may need additional parsing before using these values. The following examples show SingleStore values and their equivalent JSON strings:
  • A VARBINARY(200) type value of 'databases' is converted to "ZGF0YWJhc2Vz".
  • A TINYBLOB type value of 'binary' is converted to "YmluYXJ5"
  • A BLOB type value of 'blobs' is converted to "YmxvYnM"
To return binary column values as string, you can cast it. For example,`SELECT CAST(binary_col_name AS CHAR)...` | | Time and Date |
  • YEAR
  • TIME
  • DATE
  • DATETIME
  • TIMESTAMP
  • DATETIME(6)
  • TIMESTAMP(6)
| String | Date and time values are converted to JSON strings, and they are no longer parsed through the Golang time module. For example,
  • "2022"
  • "12:04:00"
  • "2006-01-02"
  • "2006-01-02 15:04:05"
  • "2006-01-02 15:04:59.735214"
`DATETIME`/`TIMESTAMP`values with`0`microseconds precision follow SingleStore semantics. | > **❗ Important**: If a column is nullable, make sure to prepare to handle null values. ## Data Type Mapping for `/api/v1/` | Category | SingleStore Data Type | JSON Data Type | Notes | | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Boolean |
  • BOOL
  • TINYINT
| Number | A`TINYINT`value of`0`is considered`false`, and non-zero values are considered`true`. SingleStore returns`false`as`0`and`true`as`1`. | | Numeric |
  • TINYINT
  • SMALLINT
  • MEDIUMINT
  • INT
  • BIGINT
  • FLOAT
  • DOUBLE
| Number | JSON number data type supports both integers and floating point values. For example,
  • 24
  • 25567
  • 3.14
  • 2345.45632
  • 1e20
  • 1.34e-10
| | JSON |
  • JSON
| JSON | JSON type values are directly added to the row tuple without any type conversion, which eliminates the need for additional parsing in later steps. For example,
  • { "hello": "world" }
  • [ 1, 2, null, 3 ]
  • "data"
  • 1.4
JSON null values and SQL null values cannot be differentiated when using the Data API. | | String and Geospatial |
  • DECIMAL
  • GEOGRAPHY
  • GEOGRAPHYPOINT
  • TEXT
  • CHAR
  • VARCHAR
  • TINYTEXT
  • MEDIUMTEXT
| String | JSON strings are enclosed within double quotes. For example,
  • "567.2389354"
  • "POINT(-74.04451396 40.68924403)"
  • "POLYGON((1 1,2 1,2 2, 1 2, 1 1))"
  • "Connectors"
| | Varbinary |
  • VARBINARY
  • LONGBLOB
  • MEDIUMBLOB
  • TINYBLOB
  • BLOB
| Base64-encoded string | SingleStore`VARBINARY`values are converted to base-64 encoded strings as JSON values. Hence, you may need additional parsing before using these values. The following examples show SingleStore values and their equivalent JSON strings:
  • A VARBINARY(200) type value of 'databases' is converted to "ZGF0YWJhc2Vz".
  • A TINYBLOB type value of 'binary' is converted to "YmluYXJ5"
  • A BLOB type value of 'blobs' is converted to "YmxvYnM"
To return binary column values as string, you can cast it. For example,`SELECT CAST(binary_col_name AS CHAR)...` | | Time and Date |
  • YEAR
| Number | `YEAR`type values are converted to JSON number type. For example, 2001. | |
  • TIME
  • DATE
  • DATETIME
  • TIMESTAMP
  • DATETIME(6)
  • TIMESTAMP(6)
| String | Date and time values (except for`YEAR`) are converted to JSON strings. For example,
  • "12:04:00"
  • "2006-01-02"
  • "2006-01-02 15:04:05"
  • "2006-01-02 15:04:59.735214"
| | > **❗ Important**: - If a column is nullable, make sure to prepare to handle null values. > - Invalid `DATETIME` and `TIMESTAMP` values cannot be used over the Data API, although SingleStore allows them without the HTTP proxy. > - `UNSIGNED` numbers must be specified as `NOT NULL`. The Data API returns an error if it receives a value larger than the maximum `SIGNED` value and if the result type is `NOT NULL`. ## Prerequisite to Upgrade from Data API v1 to v2 To upgrade from Data API v1 to v2: * Change `/api/v1/` in the API request URL to `/api/v2/`. * Note the data type conversion differences between `/api/v1` and `/api/v2`. For example, the `YEAR` data type is converted to a JSON string in `/api/v2` and a JSON number in `/api/v1`. *** Modified at: June 10, 2022 Source: [/cloud/reference/data-api/data-api-data-type-conversion/](https://docs.singlestore.com/cloud/reference/data-api/data-api-data-type-conversion/) (An index of the documentation is available at /llms.txt) --- # Data API Endpoint Reference SingleStore currently supports the following API endpoints: | API Endpoint URL | Description | | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [/api/v2/exec](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/exec.md) | Executes a SQL statement without returning result sets; typically used for executing DDL and DML statements for which result sets are not expected, such as`CREATE TABLE`and`INSERT`statements. | | [/api/v2/jwks\_setup](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/jwks-setup.md) | Triggers update of the JWKS used by the Data API. Once the JWKS is updated and validated, the Data API will use the corresponding`jwks_endpoint`to authenticate requests to the API. | | [/ping](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/ping.md) | Verifies that the HTTP service is running and connectable. | | [/api/v2/query/rows](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/query-rows.md) | Executes a SQL statement and returns result sets; typically used for the`SELECT`statement for which result sets are expected. The result sets contain column names mapped to row values in a single field. | | [/api/v2/query/tuples](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/query-tuples.md) | Executes a SQL statement and returns result sets along with the schema; typically used for the`SELECT`statement for which result sets are expected. The result sets contain rows and columns in separate fields with the schema displayed for each column. | | [/api/v2/spec](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/spec.md) | Returns the OpenAPI 3.0 specification for this service. | > **📝 Note**: The difference between `/query/rows` and `/query/tuples` endpoints lies in how the query results are presented in the JSON format. In addition, the `/query/tuples` endpoint offers better performance for queries with large result sets.The HTTP request to the `/ping` endpoint does not require the SingleStore account credentials passed in an Authorization header. ## In this section * [exec](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/exec.md) * [jwks\_setup](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/jwks-setup.md) * [ping](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/ping.md) * [query/rows](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/query-rows.md) * [query/tuples](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/query-tuples.md) * [spec](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/spec.md) *** Modified at: February 7, 2023 Source: [/cloud/reference/data-api/data-api-endpoint-reference/](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/) (An index of the documentation is available at /llms.txt) --- # Data API Error Handling On successful processing of SingleStore's Data API requests, the server returns the *200 OK* HTTP status line. On Data API request failure, the server returns a status code for the error along with a message describing the issue. The error messages are returned as bare strings in the response body. The following table provides a list of HTTP response status codes that are used for handling errors in SingleStore’s Data API requests. | HTTP Status Code | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 400 Bad Request | A general error code used for client request issues for reasons, such as:
  • Request body is not JSON
  • Request body does not match the input schema
  • SQL is malformed
  • Context database does not exist
| | 401 Unauthorized |
  • Invalid authentication credentials
  • Authentication failed
When using JWTs to authenticate, this error code may indicate the following issues:
  • The specified JWT is invalid
  • Invalid JWT signature, i.e., the JWT signature cannot be verified using JWKS fetched from the engine
| | 403 Forbidden | Authorization error from SingleStore | | 404 Not Found | Path not found | | 405 Method Not Allowed | Incorrect HTTP method used for the endpoint specified | | 408 Request Timeout | Timeout while waiting for the client to provide a complete request with the headers and the JSON body | | 500 Internal Server Error | A general error code used to handle many different error conditions and are generally considered unexpected. For example, you might receive this error if some of your data partitions are offline during a query or if the HTTP service is unable to communicate with the underlying SingleStore node. | | 504 Gateway Timeout | Timeout while communicating with the underlying SingleStore node | *** Modified at: January 25, 2023 Source: [/cloud/reference/data-api/data-api-error-handling/](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling/) (An index of the documentation is available at /llms.txt) --- # Enable the Data API SingleStore’s Data API is accessible via a built-in HTTP server. The Data API is enabled by default. When the Data API is enabled, it will be hosted on a port that is different from the SingleStore Helios port. ## Test the Data API Call the `GET /ping` API request to verify that the HTTP service is running and connectable. To perform this action using cURL, make a `GET` request to the `/ping` endpoint with no parameters: ```shell curl https://svc-XXXX-dml.aws-ireland-1.svc.singlestore.com/ping ``` If the Data API service is available and is able to respond to requests, you will receive the following response along with a *200 OK* status code, indicating a successful HTTP response. ```shell ping ``` If an error occurs, you will receive one of the [HTTP response status codes](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling.md), along with the error description in the response body. For more information, refer to the [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md). The `GET /ping` request can only verify whether the HTTP service is running and connectable. To verify that the database can receive queries, use the `/exec` and `/query` endpoints. The following cURL command makes a `POST` API request to the `/query/rows` endpoint. (Replace `root:password` in the HTTP request with your SingleStore `username:password` string.) ```shell curl -H "Content-Type: application/json" --data '{"sql": "select 1+1"}' --user 'root:password' https://svc-XXXX-dml.aws-ireland-1.svc.singlestore.com/api/v2/query/rows ``` If the API call is successful, you should see the following results along with a *200 OK* status code. ```JSON {"results":[{"rows":[{"1+1":2}]}]} ``` If an error occurs, you will receive one of the [HTTP response status codes](https://docs.singlestore.com/cloud/reference/data-api/data-api-error-handling.md), along with the error description in the response body. For more information, refer to the [Data API Endpoint Reference](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference.md). If you format the JSON output with a formatting tool, the results appear as follows. (Recommended JSON formatting tools: [jq](https://stedolan.github.io/jq/), [JSONLint](https://jsonlint.com/)) ```JSON { "results": [ { "rows": [ { "1+1": 2 } ] } ] } ``` **Note**: The `GET /ping` request should be used in automated health checks. To verify specific health metrics, use the `/exec` and `/query` endpoints. *** Modified at: July 18, 2024 Source: [/cloud/reference/data-api/enable-the-data-api/](https://docs.singlestore.com/cloud/reference/data-api/enable-the-data-api/) (An index of the documentation is available at /llms.txt) --- # Fusion SQL > **📝 Note**: This is a Preview feature. SingleStore Fusion SQL commands enable you to interface with the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) from [SingleStore Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md). Fusion SQL commands allow you to perform the following operations: * Create or delete cluster groups, request information on available cluster groups, list the cluster groups * Create, suspend, resume, or delete clusters, request information on clusters, list the clusters * Create, update, or delete a folder in a Stage, request information on a folder * Upload, download, modify, or delete a file in a Stage, personal space, or shared space, show information on files * Run jobs using notebooks, show job parameters, runtimes, and executions, show jobs, delete jobs, wait on jobs * List the regions available for a user * Show Fusion SQL commands, help, and grammar ## Enable Fusion SQL Fusion SQL commands are enabled by default in notebooks on the [Cloud Portal](https://portal.singlestore.com). To use the Fusion SQL commands outside of the notebook environment using the [SingleStore Python SDK](https://singlestoredb-python.labs.singlestore.com/api.html#management-api), enable the `SINGLESTOREDB_FUSION_ENABLED` environment variable. ``` SINGLESTOREDB_FUSION_ENABLED=1 ``` ## Authentication When used in notebooks, authentication is handled via signed JWTs. Refer to [Authenticate via JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md) for more information. ## Run Fusion SQL Commands Fusion SQL commands are run in the SQL and Python cells of a [SingleStore Notebook](https://docs.singlestore.com/cloud/container-services/notebooks/using-notebooks.md) on the [Cloud Portal](https://portal.singlestore.com). These commands are handled by the handler classes and are not sent to the database server. You can also run Fusion SQL commands outside of SingleStore notebooks using the Management objects in the [SingleStore Python SDK](https://singlestoredb-python.labs.singlestore.com/api.html#management-api). Authenticate the connection using an API key. Refer to [Generate an API Key](https://docs.singlestore.com/cloud/reference/management-api/#section-idm4495199953840032756423936306.md) for information on generating API keys on the [Cloud Portal](https://portal.singlestore.com). Explore the following notebooks for getting started: * [Getting Started with Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/): Demonstrates how to create, suspend, and resume SingleStore Helios deployments using Fusion SQL commands. * [Managing Stage files with Fusion SQL](https://www.singlestore.com/spaces/managing-stage-files-with-fusion-sql/): Shows how to create and delete files and folders in a Stage using Fusion SQL commands. ## Supported Fusion SQL Commands Run [SHOW FUSION COMMANDS](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-commands.md) to list all the available Fusion commands. Use the `LIKE` clause to return a list of commands that match the specified pattern, such as: ```sql SHOW FUSION COMMANDS LIKE '%WORKSPACE%'; ``` You can also view the grammar and help for a Fusion SQL command using the `SHOW FUSION GRAMMAR FOR` and `SHOW FUSION HELP FOR` commands, respectively. ```sql SHOW FUSION GRAMMAR FOR ; SHOW FUSION HELP FOR ; ``` ## In this section * [CREATE STAGE FOLDER](https://docs.singlestore.com/cloud/reference/fusion-sql/create-stage-folder.md) * [CREATE WORKSPACE GROUP](https://docs.singlestore.com/cloud/reference/fusion-sql/create-workspace-group.md) * [CREATE WORKSPACE](https://docs.singlestore.com/cloud/reference/fusion-sql/create-workspace.md) * [DOWNLOAD PERSONAL FILE](https://docs.singlestore.com/cloud/reference/fusion-sql/download-personal-file.md) * [DOWNLOAD SHARED FILE](https://docs.singlestore.com/cloud/reference/fusion-sql/download-shared-file.md) * [DOWNLOAD STAGE FILE](https://docs.singlestore.com/cloud/reference/fusion-sql/download-stage-file.md) * [DROP JOBS](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-jobs.md) * [DROP PERSONAL FILE](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-personal-file.md) * [DROP SHARED FILE](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-shared-file.md) * [DROP STAGE {FILE | FOLDER}](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-stage-file-folder.md) * [DROP WORKSPACE GROUP](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-workspace-group.md) * [DROP WORKSPACE](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-workspace.md) * [RESUME WORKSPACE](https://docs.singlestore.com/cloud/reference/fusion-sql/resume-workspace.md) * [RUN JOB USING NOTEBOOK](https://docs.singlestore.com/cloud/reference/fusion-sql/run-job-using-notebook.md) * [SHOW FUSION COMMANDS](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-commands.md) * [SHOW FUSION GRAMMAR FOR](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-grammar-for.md) * [SHOW FUSION HELP FOR](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-help-for.md) * [SHOW JOB EXECUTIONS FOR](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-executions-for.md) * [SHOW JOB PARAMETERS FOR](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-parameters-for.md) * [SHOW JOB RUNTIMES](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-runtimes.md) * [SHOW JOBS](https://docs.singlestore.com/cloud/reference/fusion-sql/show-jobs.md) * [SHOW REGIONS](https://docs.singlestore.com/cloud/reference/fusion-sql/show-regions.md) * [SHOW PERSONAL FILES](https://docs.singlestore.com/cloud/reference/fusion-sql/show-personal-files.md) * [SHOW SHARED FILES](https://docs.singlestore.com/cloud/reference/fusion-sql/show-shared-files.md) * [SHOW STAGE FILES](https://docs.singlestore.com/cloud/reference/fusion-sql/show-stage-files.md) * [SHOW WORKSPACE GROUPS](https://docs.singlestore.com/cloud/reference/fusion-sql/show-workspace-groups.md) * [SHOW WORKSPACES](https://docs.singlestore.com/cloud/reference/fusion-sql/show-workspaces.md) * [SUSPEND WORKSPACE](https://docs.singlestore.com/cloud/reference/fusion-sql/suspend-workspace.md) * [UPLOAD FILE TO STAGE](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-file-to-stage.md) * [UPLOAD PERSONAL FILE TO](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-personal-file-to.md) * [UPLOAD SHARED FILE TO](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-shared-file-to.md) * [USE WORKSPACE](https://docs.singlestore.com/cloud/reference/fusion-sql/use-workspace.md) * [WAIT ON JOBS](https://docs.singlestore.com/cloud/reference/fusion-sql/wait-on-jobs.md) *** Modified at: November 21, 2025 Source: [/cloud/reference/fusion-sql/](https://docs.singlestore.com/cloud/reference/fusion-sql/) (An index of the documentation is available at /llms.txt) --- # CREATE STAGE FOLDER The `CREATE STAGE FOLDER` command creates a folder at the specified path within a [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md). ## Syntax ``` CREATE STAGE FOLDER '' [ { IN GROUP { ID '' | '' } | IN { ID '' | '' } } ] [ OVERWRITE ] ``` ## Arguments * `stage_path`: The path in the Stage space where the folder is created. It must end with a trailing `/`. * `deployment_ID`, `deployment_name`: The ID or name of the deployment (cluster group) in which the Stage is attached. ## Remarks * If the `OVERWRITE` clause is specified, any existing folder with the same name at the specified path is overwritten. * The `IN` and `IN GROUP` clauses specify the ID or the name of the deployment in which the Stage is attached. ## Example The following command creates a folder in the Stage attached to the deployment named **s2-group**: ```sql CREATE STAGE FOLDER `/data/` IN 's2-group'; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/create-stage-folder/](https://docs.singlestore.com/cloud/reference/fusion-sql/create-stage-folder/) (An index of the documentation is available at /llms.txt) --- # CREATE WORKSPACE GROUP The `CREATE WORKSPACE GROUP` command creates a new cluster group. ## Syntax ``` CREATE WORKSPACE GROUP [ IF NOT EXISTS ] '' IN REGION { ID '' | '' } [ WITH PASSWORD '' ] [ EXPIRES AT '' ] [ WITH FIREWALL RANGES '',... ] [ WITH BACKUP BUCKET KMS KEY ID '' ] [ WITH DATA BUCKET KMS KEY ID '' ] [ WITH SMART DR ] [ ALLOW ALL TRAFFIC ] [ WITH UPDATE WINDOW ':' ] ``` ## Arguments * `group_name`: Name of the cluster group. * `region_ID`, `region_name`: The ID or name of the region in which the cluster group is created. * `password`: The admin password for the cluster group. The password must contain the following: * At least 8 characters * At least one uppercase character * At least one lowercase character * At least one number or special character * `expiry_time`: The timestamp or the time interval at which the cluster group automatically expires (terminates). * `ip_range`: A list of allowed IP addresses or CIDR block. * `backup_key_ID`: The KMS key ID associated with the backup bucket. * `data_key_ID`: The KMS key ID associated with the data bucket. * `day:hour`: The day of the week (0-6 starting on a Sunday) and the hours of the day (0-23 UTC) when the updates are applied to the cluster group. ## Remarks * Specify the `IF NOT EXISTS` clause to create a new cluster group only when a cluster group with the specified name does not exist already. If a cluster group with the specified name already exists and the `IF NOT EXISTS` clause is not specified, the `CREATE WORKSPACE GROUP` command returns an error. * If the `WITH BACKUP BUCKET KMS KEY ID` clause is specified, it enables Customer-Managed Encryption Keys (CMEK) encryption for the backup bucket of the cluster group. This feature is only supported in cluster groups deployed in AWS. * If the `WITH DATA BUCKET KMS KEY ID` clause is specified, it enables Customer-Managed Encryption Keys (CMEK) encryption for the data bucket and Amazon Elastic Block Store (EBS) volumes of the cluster group. This feature is only supported in cluster groups deployed in AWS. * Specify the `WITH SMART DR` clause to enable [Smart Disaster Recovery (DR): SmartDR](https://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr.md) for the cluster group. * Specify the `ALLOW ALL TRAFFIC` clause to allow access to the cluster group from any IP address. ## Examples * The following command creates a cluster group in the specified region: ```sql CREATE WORKSPACE GROUP 's2-group' IN REGION 'US East 2 (Ohio)'; ``` * The following command creates a cluster group with additional configurations: ```sql CREATE WORKSPACE GROUP IF NOT EXISTS 's2-group' IN REGION ID '93b61160-0000-1000-9000-977b8e2e3ee5' WITH FIREWALL RANGES '0.0.0.0/0' WITH SMART DR; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/create-workspace-group/](https://docs.singlestore.com/cloud/reference/fusion-sql/create-workspace-group/) (An index of the documentation is available at /llms.txt) --- # CREATE WORKSPACE The `CREATE WORKSPACE` command creates a new cluster in the specified cluster group. Refer to [Compute Clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters.md) for more information. ## Syntax ``` CREATE WORKSPACE [ IF NOT EXISTS ] '' [ IN GROUP { ID '' | '' } ] WITH SIZE '' [ AUTO SUSPEND AFTER { SECONDS | MINUTES | HOURS | DAYS } WITH TYPE { IDLE | SCHEDULED | DISABLED } ] [ ENABLE KAI ] [ WITH CACHE CONFIG ] [ WAIT ON ACTIVE ] ``` ## Arguments * `cluster_name`: Name of the cluster. * `group_ID`, `group_name`: ID or name of the deployment (cluster group) in which the cluster is created. * `cluster_size`: Size of the cluster in the cluster size notation, for example, "S-1". * `suspend_time`: Time in the specified units after which the cluster is automatically suspended, depending on the specified suspend type. * `multiplier`: Specifies the multiplier for the persistent cache associated with the cluster. ## Remarks * Specify the `IF NOT EXISTS` clause to create a new cluster only when a cluster with the specified name does not exist already. If a cluster with the specified name already exists and the `IF NOT EXISTS` clause is not specified, the `CREATE WORKSPACE` command returns an error. * If the `WITH CACHE CONFIG` clause is specified, it enables the cache configuration multiplier. It can have one of the following values: `1`, `2`, or `4`. * If the `WAIT ON ACTIVE` clause is specified, query execution is paused for this cluster until it reaches the `ACTIVE` state. * Specify `ENABLE KAI` to enable [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md) (the MongoDB® API) for the cluster. ## Example The following command creates a new cluster named **test** in a cluster group named **s2-group**: ```sql CREATE WORKSPACE 'test' IN GROUP 's2-group' WITH SIZE 'S-1' WAIT ON ACTIVE; ``` *** Modified at: July 24, 2026 Source: [/cloud/reference/fusion-sql/create-workspace/](https://docs.singlestore.com/cloud/reference/fusion-sql/create-workspace/) (An index of the documentation is available at /llms.txt) --- # DOWNLOAD PERSONAL FILE The `DOWNLOAD PERSONAL FILE` command downloads a file from the personal space ([Cloud Portal](https://portal.singlestore.com) > Editor > Personal). ## Syntax ``` DOWNLOAD PERSONAL FILE '' [ TO '' ] [ OVERWRITE ] [ ENCODING '' ] ``` ## Arguments * `filename`: The name of the file in the personal space. * `local_path`: Specifies the download path in the local directory. * `encoding`: The encoding applied to the downloaded file. ## Remarks * If the `OVERWRITE` clause is specified, any existing file with the same name at the specified download path is overwritten. * By default, files are downloaded with the binary encoding. Specify the `ENCODING` clause and omit the `TO` clause, to view the contents of the file on the standard output. * If the `TO ` clause is not specified, the file is displayed on the standard output. ## Examples * The following command displays the contents of a file named **stats.csv** in the personal space on the standard output: ```sql DOWNLOAD PERSONAL FILE 'stats.csv' ENCODING 'utf8'; ``` * The following command downloads the file to a specific location in the local storage: ```sql DOWNLOAD PERSONAL FILE 'stats.csv' TO '/tmp/stats.csv' OVERWRITE ENCODING 'utf8'; ``` If the **/tmp/stats.csv** file exists in the local storage, it is overwritten. *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/download-personal-file/](https://docs.singlestore.com/cloud/reference/fusion-sql/download-personal-file/) (An index of the documentation is available at /llms.txt) --- # DOWNLOAD SHARED FILE The `DOWNLOAD SHARED FILE` command downloads a file from the shared space ([Cloud Portal](https://portal.singlestore.com) > Editor > Shared). ## Syntax ``` DOWNLOAD SHARED FILE '' [ TO '' ] [ OVERWRITE ] [ ENCODING '' ] ``` ## Arguments * `filename`: The name of the file in the shared space. * `local_path`: Specifies the download path in the local directory. * `encoding`: The encoding applied to the downloaded file. ## Remarks * If the `OVERWRITE` clause is specified, any existing file with the same name at the specified download path is overwritten. * By default, the files are downloaded with the binary encoding. Specify the `ENCODING` clause and omit the `TO` clause, to view the contents of the file on the standard output. * If the `TO ` clause is not specified, the file is displayed on the standard output. ## Examples * The following command displays the contents of a file named **stats.csv** in the shared space, on the standard output: ```sql DOWNLOAD SHARED FILE 'stats.csv' ENCODING 'utf8'; ``` * The following command downloads the file to a specific location in the local storage: ```sql DOWNLOAD SHARED FILE 'stats.csv' TO '/tmp/stats.csv' OVERWRITE ENCODING 'utf8'; ``` If the **/tmp/stats.csv** file exists in the local storage, it is overwritten. *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/download-shared-file/](https://docs.singlestore.com/cloud/reference/fusion-sql/download-shared-file/) (An index of the documentation is available at /llms.txt) --- # DOWNLOAD STAGE FILE The `DOWNLOAD STAGE FILE` command downloads a file at the specified path within a [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md). ## Syntax ``` DOWNLOAD STAGE FILE '' [ { IN GROUP { ID '' | '' } | IN { ID '' | '' } } ] [ TO '' ] [ OVERWRITE ] [ ENCODING '' ] ``` ## Arguments * `stage_path`: The path to the file in the Stage. * `deployment_ID`, `deployment_name`: The ID or name of the deployment (cluster group) in which the Stage is attached. * `local_path`: Specifies the download path in the local directory. * `encoding`: The encoding applied to the downloaded file. ## Remarks * If the `OVERWRITE` clause is specified, any existing file with the same name at the specified download path is overwritten. * The `IN` and `IN GROUP` clauses specify the ID or the name of the deployment in which the Stage is attached. * By default, files are downloaded with the binary encoding. Specify the `ENCODING` clause and omit the `TO` clause, to view the contents of the file on the standard output. * If the `TO ` clause is not specified, the file is displayed on the standard output. ## Examples * The following command displays the contents of a file in the Stage, on the standard output: ```sql DOWNLOAD STAGE FILE '/data/stats.csv' IN 's2-group' ENCODING 'utf8'; ``` * The following command downloads the file to a specific location in the local storage: ```sql DOWNLOAD STAGE FILE '/data/stats.csv' IN 's2-group' TO '/tmp/stats.csv' OVERWRITE ENCODING 'utf8'; ``` If the **/tmp/stats.csv** file exists in the local storage, it is overwritten. *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/download-stage-file/](https://docs.singlestore.com/cloud/reference/fusion-sql/download-stage-file/) (An index of the documentation is available at /llms.txt) --- # DROP JOBS The `DROP JOBS` command drops (deletes) the jobs with the specified ID. ## Syntax ``` DROP JOBS '' ``` ## Argument * `job_ids`: A comma-separated list of IDs of jobs to delete. ## Example The following command drops the jobs with IDs **62c230ea-0000-1000-9000-c08a7a3fd6c0** and **5915e2d8-0000-1000-9000-bb81033c63bc**: ```sql DROP JOBS '62c230ea-0000-1000-9000-c08a7a3fd6c0', '5915e2d8-0000-1000-9000-bb81033c63bc'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/drop-jobs/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-jobs/) (An index of the documentation is available at /llms.txt) --- # DROP PERSONAL FILE The `DROP PERSONAL FILE` command deletes a file in the personal space ([Cloud Portal](https://portal.singlestore.com) > Editor > Personal). ## Syntax ``` DROP PERSONAL FILE '' ``` ## Argument * `filename`: The name of the file to delete in the personal space. ## Example The following command deletes a file in the personal space: ```sql DROP PERSONAL FILE 'stats.csv'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/drop-personal-file/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-personal-file/) (An index of the documentation is available at /llms.txt) --- # DROP SHARED FILE The `DROP SHARED FILE` command deletes a file in the shared space ([Cloud Portal](https://portal.singlestore.com) > Editor > Shared). ## Syntax ``` DROP SHARED FILE '' ``` ## Argument * `filename`: The name of the file to delete in the shared space. ## Example The following command deletes a file in the shared space: ```sql DROP SHARED FILE 'stats.csv'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/drop-shared-file/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-shared-file/) (An index of the documentation is available at /llms.txt) --- # DROP STAGE {FILE | FOLDER} The `DROP STAGE {FILE | FOLDER}` command deletes a file or folder at the specified path within a [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md). ## Syntax ``` DROP STAGE {FILE | FOLDER} '' [ { IN GROUP { ID '' | '' } | IN { ID '' | '' } } ] [RECURSIVE] ``` ## Arguments * `stage_path`: The path to the file or folder in the Stage. Folder paths must end with a trailing `/`. * `deployment_ID`, `deployment_name`: The ID or name of the deployment (cluster group) in which the Stage is attached. ## Remarks * The `IN` and `IN GROUP` clauses specify the ID or the name of the deployment in which the Stage is attached. * To delete a folder, use the `FOLDER '' ... [RECURSIVE]` clause. The path must end with a trailing `/`. * The `RECURSIVE` clause is only supported in the `DROP STAGE FOLDER` command. * If the `RECURSIVE` clause is specified, the specified folder is deleted recursively, i.e., the contents of the specified folder are deleted before the folder itself is deleted. * To delete a file, use the `FILE ...` clause. ## Examples * The following command deletes a file in the Stage attached to a deployment named **s2-group**: ```sql DROP STAGE FILE '/data/stats.csv' IN 's2-group'; ``` * The following command deletes a folder in the Stage attached to a deployment named **s2-group**: ```sql DROP STAGE FOLDER '/data/' IN 's2-group' RECURSIVE; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/drop-stage-file-folder/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-stage-file-folder/) (An index of the documentation is available at /llms.txt) --- # DROP WORKSPACE GROUP The `DROP WORKSPACE GROUP` command deletes the specified cluster group. ## Syntax ``` DROP WORKSPACE GROUP [ IF EXISTS ] { ID '' | '' } [ WAIT ON TERMINATED ] [ FORCE ] ``` ## Argument * `group_ID`, `group_name`: ID or name of the cluster group to delete. ## Remarks * Specify the `IF EXISTS` clause to attempt the delete operation only if a cluster group with the specified ID or name exists. * Specify the `WAIT ON TERMINATED` clause to pause query execution until the cluster group is in the `TERMINATED` state. * If the `FORCE` clause is specified, the cluster group is deleted (terminated) even if it contains clusters. ## Example The following command deletes a cluster group named **s2-group**: ```sql DROP WORKSPACE GROUP 's2-group' FORCE; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/drop-workspace-group/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-workspace-group/) (An index of the documentation is available at /llms.txt) --- # DROP WORKSPACE The `DROP WORKSPACE` command deletes the specified cluster. ## Syntax ``` DROP WORKSPACE [ IF EXISTS ] { ID '' | '' } [ IN GROUP { ID '' | '' } ] [ WAIT ON TERMINATED ] ``` ## Arguments * `cluster_ID`, `cluster_name`: ID or name of the cluster to delete. * `group_ID`, `group_name`: ID or name of the cluster group that contains the cluster. ## Remarks * Specify the `IF EXISTS` clause to attempt the delete operation only if a cluster with the specified ID or name exists. * The `IN GROUP` clause specifies the ID or name of the cluster group that contains the cluster to delete. * Specify the `WAIT ON TERMINATED` clause to pause query execution until the cluster is in the `TERMINATED` state. * All the databases attached to a cluster are detached when the cluster is deleted (terminated). ## Example The following command deletes a cluster named **examples** in a cluster named **s2-group**: ```sql DROP WORKSPACE IF EXISTS 'examples' IN GROUP 's2-group'; ``` *** Modified at: July 24, 2026 Source: [/cloud/reference/fusion-sql/drop-workspace/](https://docs.singlestore.com/cloud/reference/fusion-sql/drop-workspace/) (An index of the documentation is available at /llms.txt) --- # RESUME WORKSPACE The `RESUME WORKSPACE` command resumes a suspended cluster. Refer to [Manage Clusters](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-clusters.md) for related information. ## Syntax ``` RESUME WORKSPACE { ID '' | '' } [ IN GROUP { ID '' | '' } ] [ DISABLE AUTO SUSPEND ] [ WAIT ON RESUMED ] ``` ## Arguments * `cluster_ID`, `cluster_name`: ID or name of the cluster to resume. * `group_ID`, `group_name`: ID or name of the cluster group that contains the cluster. ## Remarks * The `IN GROUP` clause specifies the ID or name of the cluster group that contains the cluster to resume. * Specify the `DISABLE AUTO SUSPEND` clause to disable auto-suspend for the specified cluster upon resume. * Specify the `WAIT ON RESUMED` clause to pause query execution until the cluster is in the `RESUMED` state. ## Example The following command resumes a cluster named **examples** in a cluster group named **s2-group**: ```sql RESUME WORKSPACE 'examples' IN GROUP 's2-group'; ``` *** Modified at: July 24, 2026 Source: [/cloud/reference/fusion-sql/resume-workspace/](https://docs.singlestore.com/cloud/reference/fusion-sql/resume-workspace/) (An index of the documentation is available at /llms.txt) --- # RUN JOB USING NOTEBOOK The `RUN JOB USING NOTEBOOK` command creates a notebook job that immediately runs the specified notebook in a [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) using the specified runtime and parameters. ## Syntax ``` RUN JOB USING NOTEBOOK '' [ WITH RUNTIME '' ] [ WITH PARAMETERS ] ``` ## Arguments * `notebook_path`: Path in the Stage where the notebook file is stored. * `runtime_name`: Name of the runtime the scheduled job is run with. * `param_json`: The parameters passed to the job in JSON format, for example, `{"": "", ... }`. It only supports string, integer, float, and boolean type values. ## Remarks * The scheduled notebook job is run immediately after the `RUN JOB USING NOTEBOOK` command is run. ## Example The following command creates a scheduled job that runs a notebook named **example\_notebook.ipynb** using the `notebooks-cpu-small` runtime: ```sql RUN JOB USING NOTEBOOK 'example_notebook.ipynb' WITH RUNTIME 'notebooks-cpu-small' WITH PARAMETERS { "strParam": "string", "intParam": 1, "floatParam": 1.0, "boolParam": true } ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/run-job-using-notebook/](https://docs.singlestore.com/cloud/reference/fusion-sql/run-job-using-notebook/) (An index of the documentation is available at /llms.txt) --- # SHOW FUSION COMMANDS The `SHOW FUSION COMMANDS` command lists all the supported Fusion SQL commands. ## Syntax ``` SHOW FUSION COMMANDS [ LIKE '' ] ``` ## Argument * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns Fusion SQL commands that match the specified pattern. Uses `%` as the wildcard character. ## Example The following command lists Fusion SQL commands that start with `SHOW`: ```sql SHOW FUSION COMMANDS LIKE 'SHOW%'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-fusion-commands/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-commands/) (An index of the documentation is available at /llms.txt) --- # SHOW FUSION GRAMMAR FOR The `SHOW FUSION GRAMMAR FOR` command returns the grammar (syntax) for the specified Fusion SQL command. ## Syntax ``` SHOW FUSION GRAMMAR FOR '' ``` ## Argument * `command`: A Fusion SQL command. ## Example The following command displays the grammar for the `CREATE WORKSPACE` Fusion SQL command: ```sql SHOW FUSION GRAMMAR FOR 'CREATE WORKSPACE'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-fusion-grammar-for/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-grammar-for/) (An index of the documentation is available at /llms.txt) --- # SHOW FUSION HELP FOR The `SHOW FUSION HELP FOR` command displays the documentation for the specified Fusion SQL command. ## Syntax ``` SHOW FUSION HELP FOR '' ``` ## Argument * `command`: A Fusion SQL command. ## Example The following command displays the documentation for the `CREATE WORKSPACE` Fusion SQL command: ```sql SHOW FUSION HELP FOR 'CREATE WORKSPACE'; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-fusion-help-for/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-fusion-help-for/) (An index of the documentation is available at /llms.txt) --- # SHOW JOB EXECUTIONS FOR The `SHOW JOB EXECUTIONS FOR` command shows the runs (executions) for the job with the specified ID within the specified range. ## Syntax ``` SHOW JOB EXECUTIONS FOR '' FROM TO [ EXTENDED ] ``` ## Arguments * `job_ID`: ID of the job. * `limit`: An integer that specifies the execution number to start from or end at. ## Remarks * Specify `EXTENDED` to return more information on the job runs. * Specify the execution number to start from and end at using the `FROM` and `TO` clauses, respectively. ## Example The following command shows extended information on the executions for the job with ID **62c230ea-0000-1000-9000-c08a7a3fd6c0** from execution number `1` to `10`. ```sql SHOW JOB EXECUTIONS FOR '62c230ea-0000-1000-9000-c08a7a3fd6c0' FROM 1 TO 10 EXTENDED; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-job-executions-for/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-executions-for/) (An index of the documentation is available at /llms.txt) --- # SHOW JOB PARAMETERS FOR The `SHOW JOB PARAMETERS FOR` command returns the parameters specified for the job with the specified ID. ## Syntax ``` SHOW JOB PARAMETERS FOR '' ``` ## Argument * `job_ID`: ID of the job. ## Example The following command shows the parameters specified for the job with ID `058e2ed1-3000-4000-a000-512d334df903`: ```sql SHOW JOB PARAMETERS FOR '058e2ed1-3000-4000-a000-512d334df903'; ``` ```output Name Value Type endpoint svc-xxxx-dml.aws-region.svc.singlestore.com string port 3306 integer username s2user string enableSSL True boolean ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-job-parameters-for/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-parameters-for/) (An index of the documentation is available at /llms.txt) --- # SHOW JOB RUNTIMES The `SHOW JOB RUNTIMES` command returns the runtimes available for jobs. ## Syntax ``` SHOW JOB RUNTIMES ``` ## Example The following command returns the runtime available for jobs: ```sql SHOW JOB RUNTIMES; ``` ```output Name Description notebooks-cpu-small Best for basic computations, small datasets notebooks-cpu-medium Best for moderate computations, medium-sized datasets notebooks-gpu-t4 Best for machine learning and AI workloads ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-job-runtimes/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-job-runtimes/) (An index of the documentation is available at /llms.txt) --- # SHOW JOBS The `SHOW JOBS` command returns information on the job(s) with the specified ID(s). ## Syntax ``` SHOW JOBS '', ... [ EXTENDED ] [ LIKE '' ``` ## Arguments * `job_ID`: A comma-separated list of job ID(s). * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns jobs with names that match the specified pattern. Uses `%` as the wildcard character. **Note**: The `LIKE` clause filters by job names, not job IDs. ## Remarks * Specify `EXTENDED` to return more information on the specified job(s). ## Example The following command shows extended information on the job with ID `058e2ed1-3000-4000-a000-512d334df903`: ```sql SHOW JOBS '058e2ed1-3000-4000-a000-512d334df903' EXTENDED; ``` ```output JobID Name CreatedAt EnqueuedBy CompletedExecutions NotebookPath DatabaseName TargetID TargetType Description TerminatedAt CreateSnapshot MaxDurationInMins ExecutionIntervalInMins Mode StartAt ResumeTarget 058e2ed1-3000-4000-a000-512d334df903 jobExample 2025-08-28 12:24:10.864610 609e4c83-6000-4000-8000-24ab62c2097c 0 Basic Query Examples .ipynb None None None Sample test Job None 0 120 None Once 2025-09-26 11:30:00 None ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-jobs/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-jobs/) (An index of the documentation is available at /llms.txt) --- # SHOW PERSONAL FILES The `SHOW PERSONAL FILES` command returns a list of files in the personal space ([Cloud Portal](https://portal.singlestore.com) > Editor > Personal). ## Syntax ``` SHOW PERSONAL FILES [ LIKE '' ] [ ORDER BY '' [ ASC | DESC ],... ] [ LIMIT ] [ RECURSIVE ] [ EXTENDED ] ``` ## Arguments * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns files with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. * Specify the `RECURSIVE` clause to list the files recursively. * Specify the `EXTENDED` clause to return more information on the files. ## Example The following command lists the files in the personal space with additional information: ```sql SHOW PERSONAL FILES EXTENDED; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-personal-files/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-personal-files/) (An index of the documentation is available at /llms.txt) --- # SHOW REGIONS The `SHOW REGIONS` command returns a list of all the regions for the user. ## Syntax ``` SHOW REGIONS [ LIKE '' ] [ ORDER BY '' [ ASC | DESC ], ... ] [ LIMIT ] ``` ## Arguments * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns regions with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. ## Example The following command returns all the regions in the US and sorts the result in ascending order by `Name`. ```sql SHOW REGIONS LIKE 'US%' ORDER BY Name; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-regions/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-regions/) (An index of the documentation is available at /llms.txt) --- # SHOW SHARED FILES The `SHOW SHARED FILES` command returns a list of files in the shared space ([Cloud Portal](https://portal.singlestore.com) > Editor > Shared). ## Syntax ``` SHOW SHARED FILES [ LIKE '' ] [ ORDER BY '' [ ASC | DESC ],... ] [ LIMIT ] [ RECURSIVE ] [ EXTENDED ] ``` ## Arguments * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns files with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. * Specify the `RECURSIVE` clause to list the files recursively. * Specify the `EXTENDED` clause to return more information on the files. ## Example The following command lists the files in the shared space with additional information: ```sql SHOW SHARED FILES EXTENDED; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-shared-files/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-shared-files/) (An index of the documentation is available at /llms.txt) --- # SHOW STAGE FILES The `SHOW STAGE FILES` command returns a list of files in the Stage. ## Syntax ``` SHOW STAGE FILES [ { IN GROUP { ID '' | '' } | IN { ID '' | '' } } ] [ AT '' ] [ LIKE '' ] [ ORDER BY '' [ ASC | DESC ],... ] [ LIMIT ] [ RECURSIVE ] [ EXTENDED ] ``` ## Arguments * `deployment_ID`, `deployment_name`: The ID or name of the deployment in which the Stage is attached. * `path`: A path in the Stage. * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns files with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. * The `IN` and `IN GROUP` clauses specify the ID or name of the deployment (cluster group) in which the Stage is attached. * Specify the `RECURSIVE` clause to list the files recursively. * Specify the `EXTENDED` clause to return more information on the files. ## Example The following command lists the files at the specified path with additional information: ```sql SHOW STAGE FILES IN 's2-group' AT "/data/" EXTENDED; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/show-stage-files/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-stage-files/) (An index of the documentation is available at /llms.txt) --- # SHOW WORKSPACE GROUPS The `SHOW WORKSPACE GROUPS` command returns information on cluster groups. ## Syntax ``` SHOW WORKSPACE GROUPS [ LIKE '' ] [ EXTENDED ] [ ORDER BY '' [ ASC | DESC ],... ] [ LIMIT ] ``` ## Arguments * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns cluster groups with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. * Specify the `EXTENDED` clause to return more information on the files. ## Example The following command returns information on cluster groups that match the specified pattern: ```sql SHOW WORKSPACE GROUPS LIKE 'Marketing%' EXTENDED ORDER BY Name; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/show-workspace-groups/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-workspace-groups/) (An index of the documentation is available at /llms.txt) --- # SHOW WORKSPACES The `SHOW WORKSPACES` command returns information on clusters. ## Syntax ``` SHOW WORKSPACES [ IN GROUP { ID '' | '' } ] [ LIKE '' ] [ EXTENDED ] [ ORDER BY '' [ ASC | DESC ],... ] [ LIMIT ] ``` ## Arguments * `group_ID`, `group_name`: The ID or name of the deployment (cluster group) that contains the cluster. * `pattern`: Specifies a pattern similar to the SQL `LIKE` clause and returns clusters with names that match the specified pattern. Uses `%` as the wildcard character. * `key`: Key used to sort the result set. * `limit`: An integer value. ## Remarks * The `LIMIT` clause limits the number of results to the specified number. * By default, the result is sorted in the ascending order. * Specify the `EXTENDED` clause to return more information on the files. ## Example The following command returns information on all the clusters in the cluster group named **s2-group** and sorts the result by `Name` in the ascending order: ```sql SHOW WORKSPACES IN GROUP 's2-group' EXTENDED ORDER BY Name; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/show-workspaces/](https://docs.singlestore.com/cloud/reference/fusion-sql/show-workspaces/) (An index of the documentation is available at /llms.txt) --- # SUSPEND WORKSPACE The `SUSPEND WORKSPACE` command suspends a cluster. Refer to [Manage Clusters](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-clusters.md) for related information. ## Syntax ``` SUSPEND WORKSPACE { ID '' | '' } [ IN GROUP { ID '' | '' } ] [ WAIT ON SUSPENDED ] ``` ## Arguments * `cluster_ID`, `cluster_name`: ID or name of the cluster to suspend. * `group_ID`, `group_name`: ID or name of the cluster group that contains the cluster. ## Remarks * The `IN GROUP` clause specifies the ID or name of the cluster group that contains the cluster to resume. * Specify the `WAIT ON SUSPENDED` clause to pause query execution until the cluster is in the `SUSPENDED` state. ## Example The following command suspends a cluster named **examples** in a cluster group named **s2-group**: ```sql SUSPEND WORKSPACE 'examples' IN GROUP 's2-group'; ``` *** Modified at: July 24, 2026 Source: [/cloud/reference/fusion-sql/suspend-workspace/](https://docs.singlestore.com/cloud/reference/fusion-sql/suspend-workspace/) (An index of the documentation is available at /llms.txt) --- # UPLOAD FILE TO STAGE The `UPLOAD FILE TO STAGE` command uploads a file at the specified path within a [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md). ## Syntax ``` UPLOAD FILE TO STAGE '' [ { IN GROUP { ID '' | '' } | IN { ID '' | '' } } ] FROM '' [ OVERWRITE ] ``` ## Arguments * `stage_path`: The path in the Stage where the file is uploaded. It must end with a trailing `/`. * `deployment_ID`, `deployment_name`: The ID or name of the deployment (cluster group) in which the Stage is attached. * `local_path`: Specifies the path to the file in the local directory. ## Remarks * If the `OVERWRITE` clause is specified, any existing file with the same name at the specified path is overwritten. * The `IN` and `IN GROUP` clauses specify the ID or the name of the deployment in which the Stage is attached. ## Example The following command uploads a file to the Stage attached to the deployment named **s2-group** and overwrites any existing files with the same name at the specified path: ```sql UPLOAD FILE TO STAGE '/data/stats.csv' IN 's2-group' FROM '/tmp/stats.csv' OVERWRITE; ``` *** Modified at: July 23, 2026 Source: [/cloud/reference/fusion-sql/upload-file-to-stage/](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-file-to-stage/) (An index of the documentation is available at /llms.txt) --- # UPLOAD PERSONAL FILE TO The `UPLOAD PERSONAL FILE TO` command uploads a file to the personal space ([Cloud Portal](https://portal.singlestore.com) > Editor > Personal). ## Syntax ``` UPLOAD PERSONAL FILE TO '' FROM '' [ OVERWRITE ] ``` ## Arguments * ``: Name of the file to upload. * `local_path`: Specifies the path to the file in the local directory. ## Remarks * When the `OVERWRITE` clause is specified, any existing file with the same name in the personal space is overwritten. ## Example The following command uploads a file to the personal space and overwrites any existing files with the same name: ```sql UPLOAD PERSONAL FILE TO 'stats.csv' FROM '/tmp/stats.csv' OVERWRITE; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/upload-personal-file-to/](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-personal-file-to/) (An index of the documentation is available at /llms.txt) --- # UPLOAD SHARED FILE TO The `UPLOAD SHARED FILE TO` command uploads a file to the shared space ([Cloud Portal](https://portal.singlestore.com) > Editor > Shared). ## Syntax ``` UPLOAD SHARED FILE TO '' FROM '' [ OVERWRITE ] ``` ## Arguments * ``: Name of the file to upload. * `local_path`: Specifies the path to the file in the local directory. ## Remarks * When the `OVERWRITE` clause is specified, any existing file with the same name in the shared space is overwritten. ## Example The following command uploads a file to the shared space and overwrites any existing files with the same name: ```sql UPLOAD SHARED FILE TO 'stats.csv' FROM '/tmp/stats.csv' OVERWRITE; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/upload-shared-file-to/](https://docs.singlestore.com/cloud/reference/fusion-sql/upload-shared-file-to/) (An index of the documentation is available at /llms.txt) --- # USE WORKSPACE The `USE WORKSPACE` command changes the current cluster and database in the notebook. ## Syntax ``` USE WORKSPACE { ID '' | '' | @@CURRENT } [ IN GROUP { ID '' | '' } ] [ WITH DATABASE 'database_name' ] ``` ## Arguments * `cluster_ID`, `cluster_name`: ID or name of the cluster to use. * `group_id`, `group_name`: ID or name of the cluster group that contains the cluster. * `database_name`: Name of the SingleStore database to connect with. ## Remarks * The `WITH DATABASE` clause specifies the default database for the current session. * Specify `@@CURRENT` to connect to a database in the current cluster. For example: ```sql USE WORKSPACE @@CURRENT WITH DATABASE dbTest; ``` * SingleStore recommends using this command only in interactive notebooks. It may not function as intended in scheduled notebooks. ## Example The following command sets the current cluster and database to **examples** and **dbTest**, respectively. ```sql USE WORKSPACE 'examples' WITH DATABASE 'dbTest'; ``` *** Modified at: July 24, 2026 Source: [/cloud/reference/fusion-sql/use-workspace/](https://docs.singlestore.com/cloud/reference/fusion-sql/use-workspace/) (An index of the documentation is available at /llms.txt) --- # WAIT ON JOBS The `WAIT ON JOBS` command waits for the job(s) with the specified ID(s) to complete. ## Syntax ``` WAIT ON JOBS '',... [ WITH TIMEOUT { SECONDS | MINUTES | HOURS } ] ``` ## Arguments * `job_ID`: A comma-separated list of IDs of the jobs. * `timeout`: The number of seconds, minutes, or hours to wait for the job to complete. ## Example The following command waits on a job with the specified ID to complete with a timeout of `60` seconds. ```sql WAIT ON JOBS '058e2ed1-3000-4000-a000-512d334df903' WITH TIMEOUT 60 SECONDS; ``` *** Modified at: July 3, 2026 Source: [/cloud/reference/fusion-sql/wait-on-jobs/](https://docs.singlestore.com/cloud/reference/fusion-sql/wait-on-jobs/) (An index of the documentation is available at /llms.txt) --- # Information Schema Reference Views in the information schema (Holds the information or metadata for all the databases in a cluster.) surface metadata about clusters and all the components within them. SingleStore Helios supports the same `information_schema` views as MySQL, as well as a variety of additional views that are specific to SingleStore. The MySQL views are often useful, but it is important to note that some are included mostly for compatibility purposes, meaning that values for some columns may always be `NULL`. Many of the information schema views have an `mv_` version, which is a view that aggregates over `lmv_` views that are generated independently on each leaf. ## In this section * [Information Schema View List](https://docs.singlestore.com/cloud/reference/information-schema-reference/information-schema-view-list.md) * [Accessing the Information Schema](https://docs.singlestore.com/cloud/reference/information-schema-reference/accessing-the-information-schema.md) * [Examples](https://docs.singlestore.com/cloud/reference/information-schema-reference/examples.md) * [Backup and Restore](https://docs.singlestore.com/cloud/reference/information-schema-reference/backup-and-restore.md) * [Cluster Component](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component.md) * [Workload Profiling](https://docs.singlestore.com/cloud/reference/information-schema-reference/workload-profiling.md) * [Data Ingest](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest.md) * [Management](https://docs.singlestore.com/cloud/reference/information-schema-reference/management.md) * [Query Performance, Workload Management, and Statistics](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics.md) * [Replication Management](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management.md) * [Resource and User Management](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management.md) * [Segment, Column, and Merge Status](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status.md) * [Database Branching](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching.md) * [Trace Events](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events.md) * [Unlimited Storage](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage.md) *** Modified at: September 30, 2025 Source: [/cloud/reference/information-schema-reference/](https://docs.singlestore.com/cloud/reference/information-schema-reference/) (An index of the documentation is available at /llms.txt) --- # Accessing the Information Schema A majority of views in the information schema have *row-level security*. Information Schema views are different from other types of base tables that can either be accessed in full or not depending on permissions. They are views into underlying data about a cluster and the objects within it. Users have permissions to see information schema metadata (Information about a database's schema, access to the database, storage, built-in programs, and or other information about the data such as date created, file size, and etc.) about the database objects they have permissions to access. For example, if a user has permissions to access table t1 but not table t2, they will see rows containing metadata about t1 in information schema views but not t2. Alternatively, it is possible to grant the SHOW METADATA permission, which grants a user broad access to metadata about all database objects, without requiring read or write permissions to those database objects. A few of the information schema views require an additional permission to access in full. To access the Pipelines Views, users may have the `CREATE PIPELINE`, `SHOW METADATA`, `SHOW PIPELINE`, or the `ALTER PIPELINE` permissions. Similarly, to access the `LOAD_DATA_ERRORS` table, users may have the `INSERT`, or `SHOW METADATA` permissions. For example, accessing MV\_QUERIES requires the PROCESS permission. For more information, see the [Permissions Matrix](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/permissions-matrix.md). Remember that SHOW METADATA must be set at the cluster level. For example, use: ```sql GRANT SHOW METADATA ON *.* to testuser; ``` instead of: ```sql GRANT SHOW METADATA ON information_schema.processlist to testuser; ``` ```output ERROR 1044 ER_DBACCESS_DENIED_ERROR: Access denied for user 'admin'@'%' to database 'information_schema' ``` Because information schema views are simply views into the internal data structure which holds the information being surfaced in a given view, most information schema views do not have an impact on memory. For the same reason, the best way to secure this data is by managing user access. To access the `information_schema`, users can either `SELECT` data from specific view(s) with a SQL query, or use one of the [SHOW Commands](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands.md) which will surface a subset of data from the `information_schema` depending on the command. The same permissions apply to any information schema data whether it is being accessed via a query or a `SHOW` command. *** Modified at: July 8, 2025 Source: [/cloud/reference/information-schema-reference/accessing-the-information-schema/](https://docs.singlestore.com/cloud/reference/information-schema-reference/accessing-the-information-schema/) (An index of the documentation is available at /llms.txt) --- # Backup and Restore These information schema views contain information about backup, restore, and recovery operations. The views included here are available as both MV (management view) and LMV (local view per node). ## In this section * [MV\_RECOVERY\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/backup-and-restore/mv-recovery-status.md) *** Modified at: August 30, 2022 Source: [/cloud/reference/information-schema-reference/backup-and-restore/](https://docs.singlestore.com/cloud/reference/information-schema-reference/backup-and-restore/) (An index of the documentation is available at /llms.txt) --- # Cluster Component This section includes information schema views that fall into two categories: those that contain information about the status of cluster components and distribution of data across them, and others that contain information for the purpose of MySQL compatibility. ## In this section * [AGGREGATORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/aggregators.md) * [CHARACTER\_SETS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/character-sets.md) * [COLLATIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/collations.md) * [COLLATION\_CHARACTER\_SET\_APPLICABILITY](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/collation-character-set-applicability.md) * [COLUMNS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/columns.md) * [DISTRIBUTED\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-databases.md) * [DISTRIBUTED\_DATABASES\_ON\_SITES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-databases-on-sites.md) * [DISTRIBUTED\_PARTITIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-partitions.md) * [LEAVES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/leaves.md) * [MV\_CACHED\_BLOBS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-cached-blobs.md) * [MV\_CLUSTER\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-cluster-status.md) * [MV\_DISTRIBUTED\_DATABASES\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-distributed-databases-status.md) * [MV\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-events.md) * [MV\_HOSTS\_INFORMATION](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-hosts-information.md) * [MV\_TRANSACTION\_TABLE](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-transaction-table.md) * [PARAMETERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/parameters.md) * [PROJECTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/projections.md) * [ROUTINES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/routines.md) * [SCHEMATA](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/schemata.md) * [SITES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/sites.md) * [TABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/tables.md) * [TABLE\_CONSTRAINTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/table-constraints.md) * [VIEWS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/views.md) *** Modified at: December 12, 2023 Source: [/cloud/reference/information-schema-reference/cluster-component/](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/) (An index of the documentation is available at /llms.txt) --- # Data Ingest These information schema views contain exhaustive information about `LOAD DATA` and pipeline errors, configuration, and behavior. The information in these views is especially useful for debugging errors that may occur during the extraction, transformation, or loading phases of a pipeline’s lifecycle or while executing the `LOAD DATA` command. ## In this section * [LOAD\_DATA\_ERRORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/load-data-errors.md) * [PIPELINES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines.md) * [PIPELINES\_BATCHES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches.md) * [PIPELINES\_BATCHES\_METADATA](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches-metadata.md) * [PIPELINES\_BATCHES\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches-summary.md) * [PIPELINES\_CURSORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-cursors.md) * [PIPELINES\_ERRORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-errors.md) * [PIPELINES\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-files.md) * [PIPELINES\_ICEBERG\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-iceberg-files.md) * [PIPELINES\_OFFSETS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-offsets.md) *** Modified at: February 18, 2026 Source: [/cloud/reference/information-schema-reference/data-ingest/](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/) (An index of the documentation is available at /llms.txt) --- # Database Branching This topic contains reference information about Database Branching management views. ## In this section * [MV\_CLOUD\_ATTACHED\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-attached-databases.md) * [MV\_CLOUD\_DUPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-duplication-status.md) * [MV\_CLOUD\_PER\_COMPUTE\_REMOTE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-per-compute-remote-stats.md) * [MV\_CLOUD\_REMOTE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-remote-stats.md) *** Modified at: March 13, 2024 Source: [/cloud/reference/information-schema-reference/database-branching/](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/) (An index of the documentation is available at /llms.txt) --- # Examples ## Finding Expensive Queries For example, to find the five most expensive queries, run: ```sql SELECT EXECUTION_TIME/COMMITS, QUERY_TEXT from information_schema.PLANCACHE ORDER BY EXECUTION_TIME/COMMITS DESC LIMIT 5; ``` ## Finding Specific Queries In SingleStore Helios, the `PROCESSLIST` view also has a `PLAN_ID` column that can be used to look up the query currently running on a given connection in `PLANCACHE`. For example: ```sql SELECT QUERY_TEXT FROM information_schema.PROCESSLIST as pl JOIN information_schema.PLANCACHE as pc ON pl.PLAN_ID = pc.PLAN_ID; ``` This is the most reliable way to find the query executing on a given connection. The `PROCESSLIST.INFO` column is not always set by SingleStore Helios. > **📝 Note**: If you are dealing with sensitive data and want to limit access to query information, but not the full `PROCESSLIST` view, this can be controlled via the variable, `show_query_parameters`. By default, this variable is set to `ON`, so query parameters will be visible in the output of `SELECT` from `information_schema.processlist`. If set to `OFF`, parameters will be hidden. This can be changed only at startup, not during runtime; it must be set in the memsql.cnf file, followed by a system restart. ## Finding the Number of Queries Running on All Databases You can use the `processlist` to get connections on the database level. `SELECT * FROM INFORMATION_SCHEMA.MV_PROCESSLIST` shows all the queries running on the cluster. If you filter or group by the field `db`, you can get the number of connections by database. For example: ```sql SELECT db, COUNT(*) FROM information_schema.mv_processlist WHERE db IS NOT NULL GROUP BY 1; ``` ```output +--------------------+----------+ | db | COUNT(*) | +--------------------+----------+ | information_schema | 2 | | metrics | 2 | +--------------------+----------+ ``` *** Modified at: September 17, 2024 Source: [/cloud/reference/information-schema-reference/examples/](https://docs.singlestore.com/cloud/reference/information-schema-reference/examples/) (An index of the documentation is available at /llms.txt) --- # Information Schema View List Here is an alphabetical list of all the `information_schema` views that SingleStore supports. Data retention value listed as "N/A" means the view returns applicable data in the current state of the cluster. There are some additional, unused tables that are included purely for backward compatibility with MySQL, listed [here](https://docs.singlestore.com/#section-idm4576386560755233723333816108.md). | Table Name | Table Description | Data Retention | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [ADVANCED\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/advanced-histograms.md) | This view contains information about histograms that have been collected by automatic statistics or using`ANALYZE`. | N/A | | [AGGREGATE\_FUNCTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/aggregate-functions.md) | This view contains information about user defined aggregate functions (UDAF). | N/A | | [AGGREGATORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/aggregators.md) | This view contains information about aggregators. It is useful for understanding the number and attributes of the aggregators for a givencluster. | N/A | | [AVAILABILITY\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/availability-groups.md) | This view contains information about[availability groups](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/high-availability.md). | N/A | | [CHARACTER\_SETS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/character-sets.md) | This view stores information about available character sets. | N/A | | [COLLATION\_CHARACTER\_SET\_APPLICABILITY](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/collation-character-set-applicability.md) | This view stores information about the available collations for each character set. | N/A | | [COLLATIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/collations.md) | This view stores information about collations for each character set. | N/A | | [COLUMN\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/column-statistics.md) | This view contains information about each column in a givencluster. Each row displays information about a given column on a given database partition. | N/A | | [COLUMNAR\_SEGMENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/columnar-segments.md) | This view stores information about each`columnar_segment`([column segment](https://docs.singlestore.com/cloud/create-a-database/columnstore.md)). | N/A | | [COLUMNS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/columns.md) | This view stores information about each column. | N/A | | [CORRELATED\_COLUMN\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/correlated-column-statistics.md) | This view contains information about correlated columns. | The column correlation data is based on metadata rows that are persisted within the user database. The lifetime of these rows is tied to the referenced user objects -- when those objects are dropped or deleted, the rows referencing those objects are dropped from the correlation view. | | [CURRENT\_USER\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/current-user-groups.md) | This view contains information about the group(s) the current user belongs to. | N/A | | [CURRENT\_USER\_ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/current-user-roles.md) | This view contains information about the role(s) of the current user. | N/A | | [DISTRIBUTED\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-databases.md) | This view contains information about databases. | N/A | | [DISTRIBUTED\_DATABASES\_ON\_SITES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-databases-on-sites.md) | This view contains information about how databases are distributed acrossclusters. | N/A | | [DISTRIBUTED\_PARTITIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/distributed-partitions.md) | This view contains information about partitions. | N/A | | [GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/groups.md) | This view contains information about user groups in the database. | N/A | | [GROUP\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/group-privileges.md) | This view contains information about privileges as they relate to the roles and groups they are associated with. | N/A | | [GROUPS\_ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/groups-roles.md) | This view contains information about groups and the roles that are part of them. | N/A | | [IND\_CS\_PARTITION\_ROW\_SEGMENT\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/ind-cs-partition-row-segment-groups.md) | This view store information about groups of row segments that are sorted together. | N/A | | [INDEX\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/index-statistics.md) | This view contains statistics about indexes. Each row displays information about a given index on a given database partition. | N/A | | [JSON\_COLUMN\_SCHEMA](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/json-column-schema.md) | This view contains information about the schema inferred for JSON columns in columnstore tables. | N/A | | [KEY\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/key-column-usage.md) | This view contains information about columns that are part of unique keys (constraints) and displays one row per column in a unique constraint. | N/A | | [LEAVES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/leaves.md) | This view contains information about leaves. | N/A | | [LINKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/links.md) | This view contains information about connection links. | N/A | | [LMV\_COLUMNSTORE\_ACTIVE\_MERGE\_PLANS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/lmv-columnstore-active-merge-plans.md) | This view contains information about merge plans that are running on the current node. | N/A | | [LMV\_ASYNC\_COMPILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-async-compiles.md) | This view shows the queries being compiled. | N/A | | [LMV\_CONNECTION\_ATTRIBUTES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-connection-attributes.md) | This view shows connection attributes (key value pairs) for the node. | Set when the connection is established and remain until the connection is closed. | | [LMV\_GLOBAL\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/lmv-global-counters.md) | This view shows cumulative global metrics for the current node.For example, one metric provides information about the amount of work performed by the merger on the current node. | N/A | | [LMV\_JOIN\_OPTIMIZATION\_RESULTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-join-optimization-results.md) | This view contains information about the results of join optimization operations. This table displays rows for each query plan and each table joined in the query plan. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [LMV\_LOCAL\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-local-databases.md) | This view shows all databases in a cluster and their current states. | N/A | | [LMV\_PROFILE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-profile-stats.md) | This view contains statistics collected with[PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md). The table shows only the`PROFILE`from the last query instance of a given activity. | N/A | | [LMV\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/lmv-replication-status.md) | This view provides the replication status of every replication link on a single node. This view can be accessed from any node in thecluster. | N/A | | [LMV\_TABLE\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/lmv-table-counters.md) | This view shows cumulative per-table metrics for the current node.For example, one metric provides information about the amount of work performed by the merger on a specific table on the current node. | N/A | | [LOAD\_DATA\_ERRORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/load-data-errors.md) | This table is where the errors are logged when the`LOAD DATA`command is used with the`ERRORS HANDLE`clause. | N/A | | [MV\_ACTIVE\_DISTRIBUTED\_TRANSACTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-active-distributed-transactions.md) | This view shows internal information on distributed transactions that are currently active in the system. | N/A | | [MV\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities.md) | This view provides a high-level summary profile of all tasks which ran*recently*on any node in thecluster. | N/A | | [MV\_ACTIVITIES\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-cumulative.md) | This view provides a high-level summary profile of completed and currently running tasks on all nodes of thecluster. | For query activity, it is tied to in-memory plancache expiration (default setting is 720 minutes). For database activity, it is tied to the lifetime of the database.However, this is reset on node restart. | | [MV\_ACTIVITIES\_EXTENDED](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-extended.md) | This view provides a detailed profile of all tasks which ran recently on any node in thecluster. | N/A | | [MV\_ACTIVITIES\_EXTENDED\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-extended-cumulative.md) | This view provides a detailed profile of completed and currently running tasks on all nodes of thecluster. | For query activity, it is tied to in-memory plancache expiration (default setting is 720 minutes). For database activity, it is tied to the lifetime of the database.However, this is reset on node restart. | | [MV\_AGGREGATED\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-aggregated-column-usage.md) | This view provides a summary of`mv_query_column_usage`, aggregating over the columns over all queries run. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_AGGREGATED\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-aggregated-replication-status.md) | This view provides the aggregated replication status of each database in thecluster. | N/A | | [MV\_BACKUP\_HISTORY](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-history.md) | This view stores information about backups that have been made. | N/A | | [MV\_BACKUP\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-status.md) | This view stores information about the status of ongoing backups. | Stores entries until[CLEAR BACKUP\_HISTORY](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data/clearing-backup-history.md)command is executed | | [MV\_BLOCKED\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-blocked-queries.md) | This view shows which queries are waiting, and what they are waiting for. It can be used to help debug unexpected waiting issues. | N/A | | [MV\_BOTTOMLESS\_API\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-api-events.md) | This view gives details on every remote API call that is made from the engine. | Stores the 115,122 most recent events on each node since the last restart | | [MV\_BOTTOMLESS\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-databases.md) | This view provides basic information about all the databases on an unlimited storage cluster. It includes databases currently attached on the cluster, as well as any detached databases. | N/A | | [MV\_BOTTOMLESS\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-status.md) | This view gives an overview of all the unlimited storage databases in acluster, includes the reference databases and the partition databases. | N/A | | [MV\_BOTTOMLESS\_STATUS\_EXTENDED](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-status-extended.md) | This view provides more diagnostic information compared to`MV_BOTTOMLESS_STATUS`, such as more detailed LSN, upload statistics, API statistics, and logchunk/snapshot information. | N/A | | [MV\_BOTTOMLESS\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-summary.md) | This view is an at-a-glance rollup table the shows what is happening at the moment with the unlimited storagecluster. | N/A | | [MV\_BT](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-bt.md) | This view shows backtrace information for all all nodes in thecluster. This is primarily used by SingleStore Customer Support. | N/A | | [MV\_CACHED\_BLOBS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-cached-blobs.md) | This view stores information about currently cached blobs. | N/A | | [MV\_CLOUD\_ATTACHED\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-attached-databases.md) | This view shows information about all the attached databases and branches. | N/A | | [MV\_CLOUD\_BRANCHES](https://docs.singlestore.com/cloud/reference/information-schema-reference/workload-profiling/mv-cloud-branches.md) | This view displays all databases and branches, and their properties. | N/A | | [MV\_CLOUD\_DUPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-duplication-status.md) | This view provides Information about the status of the current duplication process. | N/A | | [MV\_CLOUD\_PER\_COMPUTE\_REMOTE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-per-compute-remote-stats.md) | This view provides information about how much storage usage is associated to each compute session. | N/A | | [MV\_CLOUD\_REMOTE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/database-branching/mv-cloud-remote-stats.md) | This view provides information about the total storage usage per storage\_ID. | N/A | | [MV\_CLUSTER\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-cluster-status.md) | This view stores information about partitions within acluster. | N/A | | [MV\_COLLECTED\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-collected-row-change-counts.md) | This view displays the best-effort counter for the number of changes that have been observed for the table partitions over time, and are persisted to disk. | N/A | | [MV\_COLUMNAR\_SEGMENT\_INDEX](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnar-segment-index.md) | This view stores information about indexes used to sort columnstore segments. | N/A | | [MV\_COLUMNSTORE\_ACTIVE\_MERGE\_PLANS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-active-merge-plans.md) | This view contains information about merge plans that are running. | N/A | | [MV\_COLUMNSTORE\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-files.md) | This view stores information about columnstore files. | N/A | | [MV\_COLUMNSTORE\_MERGE\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-merge-status.md) | This view stores information about the status of[background\_merger](https://docs.singlestore.com/cloud/create-a-database/columnstore.md)processes for columnstore tables. | N/A | | [MV\_CONNECTION\_ATTRIBUTES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-connection-attributes.md) | This view shows connection attributes (key value pairs) for all nodes in thecluster. | Set when the connection is established and remain until the connection is closed. | | [MV\_DATA\_DISK\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-data-disk-usage.md) | Displays the breakdown of disk usage by database. | N/A | | [MV\_DISK\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-disk-usage.md) | Displays the breakdown of disk usage bySingleStore. | N/A | | [MV\_DISTRIBUTED\_DATABASES\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-distributed-databases-status.md) | This view contains information about the status of databases. | N/A | | [MV\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-events.md) | This view contains information about events and is useful for monitoring events acrossclusters over time. | Stores the 1,028 most recent events | | [MV\_FINISHED\_TASKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-finished-tasks.md) | This view provides a detailed profile of completed tasks on all nodes of thecluster. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_FTS\_MERGED\_INDEXES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-fts-merged-indexes.md) | This view shows full-text indexes that span multiple JLucene segments. The view only includes merged indexes, that is indexes with more than one segment. | N/A | | [MV\_FULLTEXT\_INDEXES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-fulltext-indexes.md) | This view contains segment-level information about full-text indexes for all tables and all databases in thecluster. | N/A | | [MV\_GLOBAL\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-counters.md) | This view shows cumulative global per-node metrics.For example, one metric provides information about the amount of work performed by the merger on a specific node. | N/A | | [MV\_GLOBAL\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-status.md) | This view contains information on the global server status variables that are set on thecluster’s nodes. | N/A | | [MV\_GLOBAL\_VARIABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-variables.md) | This view contains information about the global variables that are set on thecluster’s nodes. | N/A | | [MV\_HOSTS\_INFORMATION](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-hosts-information.md) | This view contains information about hosts. | N/A | | [MV\_MILESTONES](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-milestones.md) | This view displays the list of milestones that have been created in the engine. This is the list of points that can be used to perform PITR. | N/A | | [MV\_NODES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-nodes.md) | This view describes allSingleStore Heliosnodes in thecluster. | N/A | | [MV\_PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-plancache.md) | This view shows plancache information for all nodes in thecluster. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_PROCESSLIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-processlist.md) | This view contains information about currently running threads on aclusterfor all nodes. | N/A | | [MV\_PROSPECTIVE\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-prospective-histograms.md) | This view contains information about columns which were used in queries where a histogram could have been utilized, but did not have histograms. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-queries.md) | This view describes queries seen bySingleStore Helios. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_QUERY\_ACTIVITIES](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-query-activities.md) | This view provides a high-level summary profile of all the queries that ran recently in thecluster. | N/A | | [MV\_QUERY\_ACTIVITIES\_EXTENDED\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-query-activities-extended-cumulative.md) | This view provides a detailed profile of completed and currently running queries on all nodes of thecluster. It holds a row per query. | Tied to in-memory plancache expiration (default setting is 720 minutes) and reset on node restart. | | [MV\_QUERY\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-query-column-usage.md) | This view provides information about the usage of columns by queries in the workload. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_QUERY\_PROSPECTIVE\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-query-prospective-histograms.md) | This view contains information about columns which were used in queries where a histogram could have been utilized, but did not have histograms. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [MV\_REBALANCE\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-rebalance-status.md) | This view contains information about the status of`REBALANCE`operations. | N/A | | [MV\_RECOVERY\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/backup-and-restore/mv-recovery-status.md) | This view includes information about the status of the current recovery process. | N/A | | [MV\_­REPLICATION\_­MANAGEMENT\_­ORPHANS\_­STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-orphans-state.md) | This view shows the state of orphan partitions (orphans are partition databases that exist on a node but are either detached or not referenced in metadata). | N/A | | [MV\_­REPLICATION\_­MANAGEMENT\_­PARTITIONS\_­STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-partitions-state.md) | This view shows the state of partition instances relevant to the node, which are listed in cluster metadata. | N/A | | [MV\_­REPLICATION\_­MANAGEMENT\_­REF\_DB\_­STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-ref-db-state.md) | This view shows the state of the reference database replica on the node. | N/A | | [MV\_­REPLICATION\_­MANAGEMENT\_­STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-state.md) | This view shows the aggregated state of replication management. | N/A | | [MV\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-status.md) | This view provides the replication status of each replication link in all nodes of thecluster. | N/A | | [MV\_RESOURCE\_POOL\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/mv-resource-pool-status.md) | This view contains information about the status of resource pools aggregating this information across all nodes in the cluster. | N/A | | [MV\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-row-change-counts.md) | As this view is primarily for internal use, SingleStore recommends using[MV\_COLLECTED\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-collected-row-change-counts.md)instead. | N/A | | [MV\_SYSINFO\_CPU](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-cpu.md) | This view describes the total CPU usage breakdown ofSingleStore Heliosand other processes per node. | N/A | | [MV\_SYSINFO\_CPULIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-cpulist.md) | This view describes the total CPU usage breakdown ofSingleStore Heliosand other processes per core. | N/A | | [MV\_SYSINFO\_DISK](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-disk.md) | This view describes the total disk usage breakdown ofSingleStore Heliosand other processes. | N/A | | [MV\_SYSINFO\_MEM](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-mem.md) | This view describes the total memory usage breakdown ofSingleStore Heliosand other processes. | N/A | | [MV\_SYSINFO\_NET](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-net.md) | This view describes the total network usage by interface. | N/A | | [MV\_TABLE\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-table-counters.md) | This view shows cumulative per-table, per-node metrics.For example, one metric provides information about the amount of work performed by the merger for a specific table on a specific node. | N/A | | [MV\_TASKS](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-tasks.md) | This view provides a detailed profile of all currently running tasks on all nodes of thecluster. | N/A | | [MV\_TRANSACTION\_TABLE](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/mv-transaction-table.md) | Contains distributed transaction metadata. | A row is created when a distributed transaction prepares on the node to which LMV\_TRANSACTION\_TABLE belongs. The row is deleted when all the nodes that participated in that distributed transaction are aware of that distributed transaction's outcome. | | [MV\_VERSION\_HISTORY](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-version-history.md)(and lmv\_version\_history\_extended) | This view shows an aggregate view of every upgrade theclusterhas gone through. | This information is historical and remains unless intentionally deleted. | | [MV\_WORKLOAD\_MANAGEMENT\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-workload-management-status.md) | This view contains information about the workload per aggregator. The statistics available in the`STAT`and`VALUE`columns are those displayed by`SHOW WORKLOAD MANAGEMENT STATUS`. | N/A | | [OPTIMIZER\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/optimizer-statistics.md) | This view contains[column statistics](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-tasks.md)calculated on columns when a table is processed by autostats or the`ANALYZE`command. | N/A | | [PARAMETERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/parameters.md) | This view contains information about parameters. | N/A | | [PIPELINES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines.md) | This view stores high-level information about any pipelines that have been created in thecluster. Each row represents a single pipeline. | N/A | | [PIPELINES\_BATCHES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches.md) | This view contains detailed, low-level information about individual batches as they’re loaded into the database. | Stores the value set by the engine variable`pipelines_batches_metadata_to_keep`(default setting is 1000 batches) | | [PIPELINES BATCHES METADATA](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches-metadata.md) | This view contains metadata about individual batches as they are loaded into the database. It includes information that provides batch-level tracking details. | N/A | | | | | | [PIPELINES\_BATCHES\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-batches-summary.md) | This view contains high-level information about individual batches as they are loaded into the database. It contains one row per batch. | N/A | | [PIPELINES\_CURSORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-cursors.md) | This view contains information about a pipeline’s offset ranges. Each row represents a single data source partition. | N/A | | [PIPELINES\_ERRORS](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-errors.md) | This view contains detailed information about errors that occurred during extraction, transformation, or loading. Each row represents a single error event. | Stores the value set by the engine variable`pipelines_errors_retention_minutes`(default setting is 1440 minutes) | | [PIPELINES\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-files.md) | This view stores information about files that have been extracted from a file system-like data source, such as Amazon S3. Each row represents a single file. | N/A | | [PIPELINES\_ICEBERG\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines-iceberg-files.md) | This view stores information about Iceberg data files ingested using pipelines with[`FORMAT ICEBERG`](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md). | N/A | | [PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/plancache.md) | This view contains information about all query statements thatSingleStorehas compiled and executed, as well as cumulative query execution statistics associated with each plan. | Tied to in-memory plancache expiration (default setting is 720 minutes) | | [PROJECTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/projections.md) | This view contains information about projections. | N/A | | [RANGE\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/range-statistics.md) | This view is deprecated and no longer used. This table previously contained information about legacy histograms that have been collected using`ANALYZE`. Current histograms are shown in the`ADVANCED_HISTOGRAMS`table instead. | N/A | | [RESOURCE\_POOLS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pools.md) | This view contains information about resource pools. | N/A | | [RESOURCE\_POOL\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pool-privileges.md) | This view contains information about resource pool grants and privileges. | N/A | | [ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/roles.md) | This view is a list of all roles and their properties related to audit permissions or security compliance. | N/A | | [ROLE\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/role-privileges.md) | This view contains information about roles as they relate to privileges in acluster. | N/A | | [ROUTINES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/routines.md) | This view contains information about routines in acluster. | N/A | | [SCHEMA\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/schema-privileges.md) | This view contains information about the privileges in a givencluster. | N/A | | [SCHEMATA](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/schemata.md) | This view contains information about schemas (databases). | N/A | | [SITES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/sites.md) | This view contains information about sites, and is available withSingleStore Helios. | N/A | | [STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/statistics.md) | This view contains statistics about table indexes. It displays one row per column per index. | N/A | | [TABLE\_CONSTRAINTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/table-constraints.md) | This view contains information about table constraints. | N/A | | [TABLE\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/table-privileges.md) | This view contains information about tables and their related privileges in a givencluster. | N/A | | [TABLE\_STATISTICS / INTERNAL\_TABLE\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/table-statistics-and-internal-table-statistics.md) | This view contains information about table statistics. It contains one row per table, per partition.`INTERNAL_TABLE_STATICS`(an internal use-only table added in v7.8) contains the same columns, and shows information forSingleStoreinternal metadata tables. | N/A | | [TABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/tables.md) | This view contains information about the tables visible to the current node. | N/A | | [USER\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/user-privileges.md) | This view contains information about the users and their related privileges in a givencluster. | N/A | | [USER\_VARIABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/user-variables.md) | This view contains information about user defined variables. See more about how to define these in[User-Defined Variables](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables.md). | N/A | | [USERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/users.md) | This view contains information about users for a givencluster. | N/A | | [USERS\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/users-groups.md) | This view contains information about how groups and users are related for a givencluster. | N/A | | [VECTOR\_INDEX](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/vector-index.md) | This view contains information about the in-memory and on-disk sizes of[vector indexes](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md). | N/a | | [VIEWS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/views.md) | This view contains information about the views in acluster. | N/A | ## Unused Tables Unlisted and unused tables included only for backward compatibility. | Table Name | | ------------------------ | | COLUMN\_PRIVILEGES | | EVENTS | | FILES | | PLUGINS | | PROFILING | | REFERENTIAL\_CONSTRAINTS | | TABLESPACES | | TRIGGERS | *** Modified at: July 30, 2026 Source: [/cloud/reference/information-schema-reference/information-schema-view-list/](https://docs.singlestore.com/cloud/reference/information-schema-reference/information-schema-view-list/) (An index of the documentation is available at /llms.txt) --- # Management This topic contains reference information about each management view. ## In this section * [LMV\_GLOBAL\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/lmv-global-counters.md) * [LMV\_TABLE\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/lmv-table-counters.md) * [MV\_AGGREGATED\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-aggregated-column-usage.md) * [MV\_AGGREGATED\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-aggregated-replication-status.md) * [MV\_BLOCKED\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-blocked-queries.md) * [MV\_COLLECTED\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-collected-row-change-counts.md) * [MV\_DATA\_DISK\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-data-disk-usage.md) * [MV\_DISK\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-disk-usage.md) * [MV\_FINISHED\_TASKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-finished-tasks.md) * [MV\_GLOBAL\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-counters.md) * [MV\_GLOBAL\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-status.md) * [MV\_GLOBAL\_VARIABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-global-variables.md) * [MV\_NODES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-nodes.md) * [MV\_QUERIES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-queries.md) * [MV\_QUERY\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-query-column-usage.md) * [MV\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-row-change-counts.md) * [MV\_SYSINFO\_CPU](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-cpu.md) * [MV\_SYSINFO\_CPULIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-cpulist.md) * [MV\_SYSINFO\_DISK](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-disk.md) * [MV\_SYSINFO\_MEM](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-mem.md) * [MV\_SYSINFO\_NET](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-sysinfo-net.md) * [MV\_TABLE\_COUNTERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-table-counters.md) * [MV\_VERSION\_HISTORY](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-version-history.md) * [MV\_CONSENSUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-consensus.md) * [MV\_CONSENSUS\_NODES](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/mv-consensus-nodes.md) *** Modified at: February 18, 2026 Source: [/cloud/reference/information-schema-reference/management/](https://docs.singlestore.com/cloud/reference/information-schema-reference/management/) (An index of the documentation is available at /llms.txt) --- # Query Performance, Workload Management, and Statistics This section includes information schema views that contain statistics and other data which are useful for monitoring and improving query performance, as well as managing the workload of a cluster. ## In this section * [ADVANCED\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/advanced-histograms.md) * [AGGREGATE\_FUNCTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/aggregate-functions.md) * [COLUMN\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/column-statistics.md) * [CORRELATED\_COLUMN\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/correlated-column-statistics.md) * [INDEX\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/index-statistics.md) * [LMV\_ASYNC\_COMPILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-async-compiles.md) * [LMV\_CONNECTION\_ATTRIBUTES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-connection-attributes.md) * [LMV\_JOIN\_OPTIMIZATION\_RESULTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-join-optimization-results.md) * [LMV\_LOCAL\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-local-databases.md) * [LMV\_PROFILE\_STATS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/lmv-profile-stats.md) * [MV\_ACTIVE\_DISTRIBUTED\_TRANSACTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-active-distributed-transactions.md) * [MV\_BACKUP\_HISTORY](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-history.md) * [MV\_BACKUP\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-backup-status.md) * [MV\_BT](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-bt.md) * [MV\_CONNECTION\_ATTRIBUTES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-connection-attributes.md) * [MV\_FTS\_MERGED\_INDEXES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-fts-merged-indexes.md) * [MV\_FULLTEXT\_INDEXES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-fulltext-indexes.md) * [MV\_PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-plancache.md) * [MV\_PROCESSLIST](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-processlist.md) * [MV\_PROSPECTIVE\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-prospective-histograms.md) * [MV\_QUERY\_PROSPECTIVE\_HISTOGRAMS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-query-prospective-histograms.md) * [MV\_WORKLOAD\_MANAGEMENT\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/mv-workload-management-status.md) * [OPTIMIZER\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/optimizer-statistics.md) * [PLANCACHE](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/plancache.md) * [RANGE\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/range-statistics.md) * [STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/statistics.md) * [TABLE\_STATISTICS and INTERNAL\_TABLE\_STATISTICS](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/table-statistics-and-internal-table-statistics.md) * [USER\_VARIABLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/user-variables.md) * [VECTOR\_INDEX](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/vector-index.md) *** Modified at: March 9, 2026 Source: [/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/) (An index of the documentation is available at /llms.txt) --- # Replication Management These information schema views show information that can be helpful for understanding, monitoring, and debugging replication management. The views included here are available as both MV (management view) and LMV (local view per node) views. > **📝 Note**: These views show a very low-level view of how replication management is working. The data is intended for use with debugging and as such, there may be redundancies or extraneous information. The schema should not be relied upon to be "stable" at this point in time. Views, columns, etc. will change. ## In this section * [LMV\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/lmv-replication-status.md) * [MV\_REPLICATION\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-status.md) * [MV\_REPLICATION\_MANAGEMENT\_ORPHANS\_STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-orphans-state.md) * [MV\_REPLICATION\_MANAGEMENT\_PARTITIONS\_STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-partitions-state.md) * [MV\_REPLICATION\_MANAGEMENT\_REF\_DB\_STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-ref-db-state.md) * [MV\_REPLICATION\_MANAGEMENT\_STATE](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/mv-replication-management-state.md) *** Modified at: September 29, 2022 Source: [/cloud/reference/information-schema-reference/replication-management/](https://docs.singlestore.com/cloud/reference/information-schema-reference/replication-management/) (An index of the documentation is available at /llms.txt) --- # Resource and User Management This section includes information schema views that contain information which is useful for monitoring and managing the users, permissions, and resources of a cluster. ## In this section * [AVAILABILITY\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/availability-groups.md) * [CURRENT\_USER\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/current-user-groups.md) * [CURRENT\_USER\_ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/current-user-roles.md) * [GROUP\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/group-privileges.md) * [ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/roles.md) * [GROUPS\_ROLES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/groups-roles.md) * [MV\_RESOURCE\_POOL\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/mv-resource-pool-status.md) * [RESOURCE\_POOLS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pools.md) * [RESOURCE\_POOL\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pool-privileges.md) * [ROLE\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/role-privileges.md) * [SCHEMA\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/schema-privileges.md) * [TABLE\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/table-privileges.md) * [USER\_PRIVILEGES](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/user-privileges.md) * [USERS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/users.md) * [GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/groups.md) * [USERS\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/users-groups.md) * [LINKS](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/links.md) *** Modified at: January 12, 2026 Source: [/cloud/reference/information-schema-reference/resource-and-user-management/](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/) (An index of the documentation is available at /llms.txt) --- # Segment, Column, and Merge Status This section includes information schema views that contain information about the way data is stored and organized within a cluster. ## In this section * [COLUMNAR\_SEGMENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/columnar-segments.md) * [IND\_CS\_PARTITION\_ROW\_SEGMENT\_GROUPS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/ind-cs-partition-row-segment-groups.md) * [JSON\_COLUMN\_SCHEMA](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/json-column-schema.md) * [KEY\_COLUMN\_USAGE](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/key-column-usage.md) * [LMV\_COLUMNSTORE\_ACTIVE\_MERGE\_PLANS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/lmv-columnstore-active-merge-plans.md) * [MV\_COLUMNAR\_SEGMENT\_INDEX](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnar-segment-index.md) * [MV\_COLUMNSTORE\_ACTIVE\_MERGE\_PLANS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-active-merge-plans.md) * [MV\_COLUMNSTORE\_FILES](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-files.md) * [MV\_COLUMNSTORE\_MERGE\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-columnstore-merge-status.md) * [MV\_REBALANCE\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-rebalance-status.md) * [MV\_ROW\_CHANGE\_COUNTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/mv-row-change-counts.md) *** Modified at: July 3, 2025 Source: [/cloud/reference/information-schema-reference/segment-column-and-merge-status/](https://docs.singlestore.com/cloud/reference/information-schema-reference/segment-column-and-merge-status/) (An index of the documentation is available at /llms.txt) --- # Trace Events The trace event views provide information specifically for query trace events, which is used by the [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) feature. ## In this section * [MV\_TRACE\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/mv-trace-events.md) * [MV\_TRACE\_EVENTS\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/mv-trace-events-status.md) * [LMV\_TRACE\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/lmv-trace-events.md) *** Modified at: January 3, 2024 Source: [/cloud/reference/information-schema-reference/trace-events/](https://docs.singlestore.com/cloud/reference/information-schema-reference/trace-events/) (An index of the documentation is available at /llms.txt) --- # Unlimited Storage The `MV_BOTTOMLESS_*` views provide information specifically for [unlimited storage databases](https://docs.singlestore.com/cloud/manage-data/benefits-of-unlimited-storage.md). ## In this section * [MV\_BOTTOMLESS\_API\_EVENTS](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-api-events.md) * [MV\_BOTTOMLESS\_API\_EVENTS\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-api-events-summary.md) * [MV\_BOTTOMLESS\_DATABASES](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-databases.md) * [MV\_BOTTOMLESS\_STATUS](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-status.md) * [MV\_BOTTOMLESS\_STATUS\_EXTENDED](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-status-extended.md) * [MV\_BOTTOMLESS\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-summary.md) * [MV\_BOTTOMLESS\_SUMMARY\_PER\_NODE](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-summary-per-node.md) * [MV\_MILESTONES](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-milestones.md) *** Modified at: May 27, 2026 Source: [/cloud/reference/information-schema-reference/unlimited-storage/](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/) (An index of the documentation is available at /llms.txt) --- # Workload Profiling This topic contains reference information about workload profiling management views. ## In this section * [MV\_CLOUD\_BRANCHES](https://docs.singlestore.com/cloud/reference/information-schema-reference/workload-profiling/mv-cloud-branches.md) *** Modified at: April 15, 2025 Source: [/cloud/reference/information-schema-reference/workload-profiling/](https://docs.singlestore.com/cloud/reference/information-schema-reference/workload-profiling/) (An index of the documentation is available at /llms.txt) --- # Management API Overview The SingleStore `Management` API provides a REST-based interface for provisioning, configuring, operating, and managing cloud resources within SingleStore Helios. The API is designed to support infrastructure automation, lifecycle management, and seamless integration with DevOps, CI/CD, and infrastructure-as-code workflows, enabling you to automate deployment pipelines, dynamically scale environments, enforce governance policies, manage operational workflows, and more. This RESTful API allows you to perform the following operations: * **Cluster groups**: Create, update, or delete cluster groups, request information on available cluster groups, list the cluster groups * **Clusters**: Create, update, suspend, resume, or delete clusters, request information on clusters, list the clusters, get the outbound allowlist for a cluster * **Projects**: List the projects accessible to a user * **Private connections**: Create, update, or delete private connections, request information on private connections * **Flow**: Create or terminate a Flow instance, request information on a Flow instance, list the Flow instances * **Invitations**: Invite a user, revoke an invitation, list the open invitations, request information on an invitation * **Teams**: Create, update, or delete a team for the current organisation, request information on a team, list the teams * **Users**: Add an existing user to the current organization, delete a user, request information on a user, list the users * **Roles**: Create, update, or delete a custom role, request information on a role, list the roles * **Jobs**: Create, update, or delete a job, request information on jobs, list the jobs * **Stage**: * Create, update, or delete a folder in a Stage, request information on a folder * Upload, download, modify, or delete a file in a Stage * **Files**: Create, upload, update, download, or delete a file in the personal, shared, or model space, list the files, request information on a file * **Secrets**: Create, update, or delete a secret, request information on a secret, list the secrets * **Audit logs**: Request Control Plane audit log entries * **Organizations**: Request information on current organization * **Billing**: Get raw compute and storage usage information for your organization * **Shared Edition** (Shared Tier): * Create or delete a Shared Edition deployment (starter cluster), request information on a starter cluster, list the starter clusters * Create, update, or delete a user in a Shared Edition deployment * **Cloud workload identity and delegated entities**: Request information on the cloud workload identity of a cluster, add or remove delegated entities, list the configured delegated entities * **Regions**: List the regions for a user The `Management` API supports [HTTP Bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/), which must be used in conjunction with SSL (HTTPS) for secure communication over the network. API calls made over HTTP will fail. For more information, refer to [Authorization](https://docs.singlestore.com/cloud/reference/management-api.md). You can also interface with the `Management` API from [SingleStore Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) using the [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) commands. ## Authorization The `Management` API authenticates requests using a unique, hex-encoded alphanumeric key. You can generate this API key on the [Cloud Portal](https://portal.singlestore.com/). When a user is a member of multiple organizations, they can have multiple API keys, one for each organization. The API key defines the operations that a user can perform. ## Generate an API Key Follow these instructions to generate API keys: 1. On the Cloud Portal, in the navigation pane, select your organization. 2. In the navigation pane, select **Configuration > API keys > Active Keys > Create API key**. 3. Specify a name and expiration date for the API key, and select **Create**. > **❗ Important**: The API key is displayed **only once**. Be sure to copy and securely store the API key. ## Revoke an API Key To revoke an API key, navigate to the **API keys** section under **Configuration** on the Cloud Portal. On the **Active Keys** tab, select the delete icon under the **Actions** column for the API key, and select **Revoke**. API keys that are not manually revoked expire on the expiration date defined while creating the API key. A revoked API key cannot be reinstated. ## Control Access to the API SingleStore recommends using **IP Allowlist** to secure inbound access to clusters using the `Management` API. To allow an IP address, login to the [Cloud Portal](https://portal.singlestore.com/). Under your organization, select **Configuration > ****API Keys > Security > Edit > Only allow access from specified IP addresses**. Under **IP Allowlist**, add the IP address to allow. ![A dialog box for editing IP Address Access.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt46f0c53e5c2c3ea9/6a7adb4f9f5251717765e912/ip_allowlist-uMoBeU.png) By default, IP Allowlist is set to **Allow access from anywhere**, which allows access to anyone with the API key. ## Access the Management API SingleStore’s `Management` API supports [HTTP Bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/). To access the API, you must provide your API key in the Authorization header in each HTTP request. An example Authorization header is as follows. ``` Authorization: Bearer 28790ecd46256d9024e7b21a737493b2ed2872e6883808a87843d1ce9839d0d3 ``` While making API calls through an HTTP client (such as cURL), you can pass the API key using the `-H` parameter: ```shell curl -X 'GET' \ 'api_endpoint_url' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' ``` ## Clusters A Cluster is an independent logical deployment of compute resources which can be used to run a workload in lieu of a cluster. A database can be shared across multiple clusters through R/W (read/write) and R/O (read only) attachments. Contact [SingleStore Support](https://support.singlestore.com) for access to the clusters and the API. Each cluster is a part of a cluster group, which allows configuration and management of clusters. An organization can have multiple cluster groups, and each cluster group can have up to five clusters. ## Connect to a Cluster Once a cluster becomes active, use the cluster endpoint to establish a SQL connection in conjunction with the admin user and password (password for the cluster group). Here is a sample connection: ```shell singlestore -h svc-xxxx-1-dml.aws-cancentral-1.svc.singlestore.com -u admin --password='QUNL@S|(})jZX1VDOp%*EMzm' ``` ## Attach a Database After connecting to a cluster endpoint, execute the `CREATE DATABASE` command to create the specified database and attach it to the cluster in the read-write mode. Refer to [Attach a Database to a Cluster](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database/#section-idm4562675609292833157466852139.md) for more information. ## Detach a Database Refer to [Detach a Database from a Cluster](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/detach-database/#section-idm4562675561145633157492898422.md) for more information. ## Manage the API using Golang Client The [singlestore-go](https://github.com/singlestore-labs/singlestore-go) repository is a collection of SingleStore Golang SDKs. The modules in the `management` package in the Golang SDK can be used to interact with the `Management` API. To import the SDK, add the following line to your `*.go` file: ```go import "github.com/singlestore-labs/singlestore-go/management" ``` For more information, see [Management API Golang SDK documentation](https://pkg.go.dev/github.com/singlestore-labs/singlestore-go/management). Here’s a [sample project](https://github.com/singlestore-labs/regions-lister) that uses the `Management` API Golang client. ## References * [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) * [Secrets](https://docs.singlestore.com/cloud/developer-resources/secrets.md) * [Connect with Private Services and Networks](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections.md) ## In this section * [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) *** Modified at: May 26, 2026 Source: [/cloud/reference/management-api/](https://docs.singlestore.com/cloud/reference/management-api/) (An index of the documentation is available at /llms.txt) --- # Management API Reference * **components** * **parameters** * **cloudPrincipalID** * *description: ID of the cloud principal* * *in: path* * *name: cloudPrincipalID* * **schema** * *format: uuid* * *type: string* * **cloudfunctionID** * *description: ID of the cloud function* * *example: 7b68b78d-a000-4000-9000-0864fd331524* * *in: path* * *name: cloudfunctionID* * **schema** * *format: uuid* * *type: string* * **connectionID** * *description: ID of the private connection* * *in: path* * *name: connectionID* * **schema** * *format: uuid* * *type: string* * **fields** * *description: Comma-separated values list that correspond to the filtered fields for returned entities* * *in: query* * *name: fields* * **schema** * *type: string* * **flowID** * *description: ID of the SingleStore Flow instance* * *in: path* * *name: flowID* * **schema** * *format: uuid* * *type: string* * **invitationID** * *description: ID of the user invitation* * *in: path* * *name: invitationID* * **schema** * *format: uuid* * *type: string* * **jobID** * *description: ID of the job* * *example: 7b68b78d-a000-4000-9000-0864fd331524* * *in: path* * *name: jobID* * **schema** * *format: uuid* * *type: string* * **lastKnownUpdatedAt** * *description: The timestamp used for concurrency control. The resource will be updated only if the current last updated timestamp matches the given value.* * *example: 2026-01-18T09:56:56Z* * *in: query* * *name: lastKnownUpdatedAt* * **schema** * *format: date-time* * *type: string* * **limit** * *description: Maximum number of items to return in a paginated query* * *in: query* * *name: limit* * **schema** * *type: integer* * **offsetID** * *description: ID of the last item from the previous page, used to continue pagination.* * *in: query* * *name: offsetID* * **schema** * *format: uuid* * *type: string* * **organizationID** * *description: ID of the organization* * *in: path* * *name: organizationID* * **schema** * *format: uuid* * *type: string* * **projectID** * *description: ID of the project* * *in: path* * *name: projectID* * **schema** * *format: uuid* * *type: string* * **resourceType** * *description: Resource type* * **examples** * **cluster** * *value: cluster* * **organization** * *value: organization* * **secret** * *value: secret* * **team** * *value: team* * **workspace** * *value: workspace* * *in: path* * *name: resourceType* * **schema** * *type: string* * **role** * *description: The role name* * *in: path* * *name: role* * **schema** * *type: string* * **secretID** * *description: ID of the secret* * *in: path* * *name: secretID* * **schema** * *format: uuid* * *type: string* * **serviceAccountID** * *description: ID of the service account* * *in: path* * *name: serviceAccountID* * **schema** * *format: uuid* * *type: string* * **teamID** * *description: ID of the team* * *in: path* * *name: teamID* * **schema** * *format: uuid* * *type: string* * **updateNotebookSnapshot** * *description: Indicates whether to update the notebook snapshot after updating the cloud function.* * *in: query* * *name: updateNotebookSnapshot* * **schema** * *type: boolean* * **userID** * *description: ID of the user* * *in: path* * *name: userID* * **schema** * *format: uuid* * *type: string* * **workspaceGroupID** * *description: ID of the workspace group* * *in: path* * *name: workspaceGroupID* * **schema** * *format: uuid* * *type: string* * **workspaceID** * *description: ID of the workspace* * *in: path* * *name: workspaceID* * **schema** * *format: uuid* * *type: string* * **schemas** * **AppToken** * **properties** * **expiresAt** * *description: The time at which this JWT token will become invalid* * *example: 11-22-2024T15:04:05* * *type: string* * **jwt** * *description: The JWT token that can be used to interact with a singlestore native App* * *example: eyJhbGciOiJI...* * *type: string* * **required** * *0: jwt* * *1: expiresAt* * *type: object* * **AuditLog** * *description: Represents an audit log entry* * **properties** * **apiKeyID** * *description: The API key ID used to perform this action, if applicable* * *format: uuid* * *type: string* * **attributes** * *description: additional keys and values that are specific to the audit log type* * *type: object* * **auditID** * *example: 17290909-0000-1000-9000-e30410f1b05a* * *format: uuid* * *type: string* * **clusterID** * *description: The database cluster tied to this event* * *format: uuid* * *type: string* * **createdAt** * *description: The timestamp of when the audit log entry was created in RFC3339Nano format* * *type: string* * **error** * *description: text error message, if any relating to this entry* * *example: * * *type: string* * **firstName** * *description: The first name of a redacted user* * *type: string* * **labels** * *description: A list of audit keywords* * **example** * *0: featureflag* * **items** * *type: string* * *type: array* * **lastName** * *description: The last name of a redacted user* * *type: string* * **orgID** * *description: The organization tied to this event* * *format: uuid* * *type: string* * **projectID** * *description: This property is deprecated. This is an internal project ID, not related to other project related fields.* * *format: uuid* * *type: string* * **reason** * *description: A human-readable description of what happened* * *example: feature flag (ssoLogin) enabled for all orgs* * *type: string* * **sessionID** * *description: the authorization session id tied to this event* * *format: uuid* * *type: string* * **source** * *description: The audit log entry source* * **enum** * *0: Portal* * *1: Admin* * *2: SystemJob* * *example: Portal* * *type: string* * **teamID** * *description: ID of the team related to this event* * *format: uuid* * *type: string* * **type** * *description: The audit log entry type* * *example: Login* * *type: string* * **userEmail** * *description: The user email address* * *example: user\@email.com* * *type: string* * **userID** * *description: The user ID* * *example: 17290909-0000-1000-9000-e30410f1b05a* * *type: string* * **userType** * *description: the type of user that triggered the audit log entry* * **enum** * *0: Unspecified* * *1: System* * *2: Employee* * *3: Customer* * *4: Automation* * *5: Studio* * *6: CIAutomation* * *7: SharedTier* * *8: SNIProxy* * *9: NimbusGateway* * *10: Cluster* * *example: Customer* * *type: string* * **workspaceID** * *description: The workspace id tied to this event* * *format: uuid* * *type: string* * *type: object* * **AutoScale** * *description: Specifies the \[autoscale]\(https\://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/workspace-scaling/#autoscaling) setting for the workspace.* * **properties** * **changedAt** * *description: Timestamp of last auto-scale configuration change* * *format: date-time* * *type: string* * **lastAutoScaledAt** * *description: Timestamp of last auto-scaling event* * *format: date-time* * *type: string* * **maxScaleFactor** * *description: The maximum scale factor allowed for the workspace. It can have the following values: 1, 2, or 4. On creation, if not specified, autoscaling is disabled. To disable autoscaling on update, set to 1.* * *type: number* * **sensitivity** * *default: NORMAL* * *description: Specifies the sensitivity of the autoscale operation to changes in the workload. It can have the following values: \`LOW\`, \`NORMAL\`, or \`HIGH\`. By default, the sensitivity is set to \`NORMAL\`.* * **enum** * *0: LOW* * *1: NORMAL* * *2: HIGH* * *example: LOW* * *type: string* * *type: object* * **BillingUsage** * *description: Represents the information related to billing usage* * **properties** * **metric** * *description: The metric type* * **enum** * *0: compute-hour* * *type: string* * **usage** * *description: Usage information* * **items** * **properties** * **endTime** * *description: The end time of the interval* * *example: 2024-06-01 09:00:00 +0000 UTC* * *type: string* * **ownerId** * *description: ID of the owner of the resource* * *example: 78af2f46-0000-1000-9000-3f6f5365d878* * *type: string* * **resourceId** * *description: ID of the resource* * *example: 78af2f46-0000-1000-9000-3f6f5365d878* * *type: string* * **resourceName** * *description: Name of the resource* * *example: test-workspace* * *type: string* * **resourceType** * *description: The resource type* * *example: Workspace* * *type: string* * **startTime** * *description: The start time of the interval* * *example: 2024-06-01 08:00:00 +0000 UTC* * *type: string* * **value** * *description: Resource usage value* * *example: 30.00* * *type: string* * *type: object* * *type: array* * *type: object* * **CloudFunctionCreate** * **allOf** * **0** * *$ref: #/components/schemas/CloudFunctionFields* * **1** * **properties** * **isAgent** * *description: Whether the cloud function runs as an agent* * *type: boolean* * *type: object* * *description: Represents information specified while creating a cloud function* * **required** * *0: name* * **CloudFunctionFields** * *description: Common fields for cloud function create and update operations* * **properties** * **databaseName** * *description: Name of the database to connect to* * *example: my\_database* * *type: string* * **description** * *description: Description of the cloud function* * *example: My custom cloud function* * *type: string* * **idleTimeoutSeconds** * *description: Idle timeout in seconds* * *type: integer* * **image** * *description: Container image to use (mutually exclusive with \`notebookPath\`)* * *example: my-registry/my-image:latest* * *type: string* * **name** * *description: Name of the cloud function* * *example: my-cloud-function* * *type: string* * **notebookPath** * *description: Path to the notebook file (mutually exclusive with \`image\`)* * *example: my-notebook.ipynb* * *type: string* * **poolName** * *description: Nova pool name for execution* * *example: my-pool* * *type: string* * **targetID** * *description: Target resource ID (required if \`targetType\` is specified)* * *example: 7b68b78d-a000-4000-9000-0864fd331524* * *format: uuid* * *type: string* * **targetType** * *description: Type of the target resource (required if \`targetID\` is specified). It can have one of the following values: \`WORKSPACE\`, \`CLUSTER\`, or \`VIRTUAL\_WORKSPACE\`.* * **enum** * *0: WORKSPACE* * *1: CLUSTER* * *2: VIRTUAL\_WORKSPACE* * *example: WORKSPACE* * *type: string* * *type: object* * **CloudFunctionUpdate** * **allOf** * **0** * *$ref: #/components/schemas/CloudFunctionFields* * *description: Represents information specified while updating a cloud function* * **CloudPrincipal** * **allOf** * **0** * *$ref: #/components/schemas/CloudPrincipalIDResponse* * **1** * *$ref: #/components/schemas/CloudPrincipalUpdate* * **2** * **properties** * **cloudPrincipal** * *description: Customer cloud identity bound to this principal.* * *example: arn:aws:iam::123456789012:role/IngestionRole* * *type: string* * **createdBy** * *description: Identifier of the user that created the cloud principal.* * *format: uuid* * *type: string* * **email** * *description: The generated fake email address for the cloud principal.* * *example: cloud-principal-abc123\@service.singlestore.com* * *type: string* * **required** * *0: cloudPrincipal* * *1: email* * *2: name* * *3: description* * *type: object* * *description: Information about a SingleStoreDB Cloud principal.* * **CloudPrincipalCreate** * **allOf** * **0** * *$ref: #/components/schemas/CloudPrincipalUpdate* * **1** * **properties** * **cloudPrincipal** * *description: Customer cloud identity to bind to this principal.* * *example: arn:aws:iam::123456789012:role/IngestionRole* * *type: string* * **required** * *0: name* * *1: cloudPrincipal* * *type: object* * *description: Information specified when creating a cloud principal.* * **CloudPrincipalIDResponse** * *description: Represents a cloud principal identifier response payload.* * **properties** * **cloudPrincipalID** * *description: Cloud principal identifier.* * *format: uuid* * *type: string* * **required** * *0: cloudPrincipalID* * *type: object* * **CloudPrincipalUpdate** * *description: Information specified when updating a cloud principal.* * **properties** * **description** * *description: Cloud principal description.* * *example: Cloud principal used by ingestion jobs.* * *type: string* * **name** * *description: Cloud principal name.* * *example: ingestion-role-renamed* * *type: string* * *type: object* * **CloudProvider** * *description: Cloud provider* * **enum** * *0: AWS* * *1: GCP* * *2: Azure* * *type: string* * **CloudWorkloadIdentity** * **properties** * **identity** * *description: The cloud workload identity bound to the workspace group that can be used to access cloud resources. The identity format is determined by the cloud provider where the workspace group is hosted. For example, on AWS it's an IAM role ARN.* * *example: arn:aws:iam::123456789012:role/sp5e57fc943d4245dda93431b40f883073* * *type: string* * **required** * *0: identity* * *type: object* * **ControlAccessAction** * **properties** * **grants** * **items** * *$ref: #/components/schemas/ControlAccessRole* * *type: array* * **revokes** * **items** * *$ref: #/components/schemas/ControlAccessRole* * *type: array* * **required** * *0: grants* * *1: revokes* * *type: object* * **ControlAccessRole** * **properties** * **role** * *type: string* * **teams** * **items** * *format: uuid* * *type: string* * *type: array* * **users** * **items** * *format: uuid* * *type: string* * *type: array* * **required** * *0: role* * *1: teams* * *2: users* * *type: object* * **DelegatedEntitiesAppend** * **properties** * **entities** * *description: List of customer-provided entities (such as IAM role ARNs) to add to the workspace group's delegated entities.* * **example** * *0: arn:aws:iam::987654321098:role/singlestore-iceberg-readonly* * *1: arn:aws:iam::987654321098:role/singlestore-s3-pipelines* * **items** * *type: string* * *type: array* * **required** * *0: entities* * *type: object* * **DelegatedEntitiesRemove** * **properties** * **entities** * *description: List of customer-provided entities (such as IAM role ARNs) to remove from the workspace group's delegated entities. If not provided in the request body, entities can be specified via the 'entities' query parameter.* * **example** * *0: arn:aws:iam::987654321098:role/singlestore-iceberg-readonly* * **items** * *type: string* * *type: array* * *type: object* * **DelegatedEntity** * **properties** * **createdAt** * *description: Creation timestamp* * *format: date-time* * *type: string* * **entity** * *description: The customer-provided entity that the workspace group's cloud workload identity can assume. The entity format is determined by the cloud provider where the workspace group is hosted. For example, on AWS it's a customer-owned IAM role ARN.* * *example: arn:aws:iam::987654321098:role/CustomerRole* * *type: string* * **required** * *0: entity* * *1: createdAt* * *type: object* * **EgressClusterIdentityCreate** * *description: Represents the information specified to create an export cluster identity* * **properties** * **catalogInfo** * *description: Catalog information for creating the export cluster identity* * *type: object* * **storageInfo** * *description: Storage information for creating the export cluster identity* * *type: object* * **required** * *0: storageInfo* * *1: catalogInfo* * *type: object* * **Execution** * **properties** * **executionID** * *description: The ID of execution* * *format: uuid* * *type: string* * **executionNumber** * *description: Number of the execution* * *type: number* * **finishedAt** * *description: Finish time of the execution* * *format: date-time* * *type: string* * **jobID** * *description: ID of the job that this execution belongs to* * *format: uuid* * *type: string* * **scheduledStartTime** * *description: Scheduled start time of the execution* * *format: date-time* * *type: string* * **snapshotNotebookPath** * *description: Path to the notebook which captures the result of this execution* * *type: string* * **startedAt** * *description: Actual start time of the execution* * *format: date-time* * *type: string* * **status** * **enum** * *0: Unknown* * *1: Scheduled* * *2: Running* * *3: Completed* * *4: Failed* * *5: Error* * *6: Canceled* * *example: Scheduled* * *type: string* * **required** * *0: executionID* * *1: jobID* * *2: status* * *3: scheduledStartTime* * *4: executionNumber* * *type: object* * **ExecutionsMetadata** * **properties** * **endExecutionNumber** * *type: number* * **startExecutionNumber** * *type: number* * **required** * *0: startExecutionNumber* * *1: endExecutionNumber* * *type: object* * **ExecutionsResult** * **properties** * **executions** * **items** * *$ref: #/components/schemas/Execution* * *type: array* * **executionsMetadata** * *$ref: #/components/schemas/ExecutionsMetadata* * **required** * *0: executions* * *1: executionsMetadata* * *type: object* * **FileLocationSchema** * *description: The file location can be one of 'personal', 'shared' or 'models'* * **enum** * *0: personal* * *1: shared* * *2: models* * *example: personal* * *type: string* * **FileObjectMetadata** * *description: Represents the metadata corresponding to a file object* * **properties** * **content** * **oneOf** * **0** * *example: Hello World!* * *type: string* * **1** * **example** * **items** * *$ref: #/components/schemas/FileObjectMetadata* * *type: array* * **created** * *example: 2023-09-08T17:35:43Z* * *type: string* * **format** * *description: Format of the response* * **enum** * *0: json* * *type: string* * **last\_modified** * *example: 2023-09-08T17:35:43Z* * *type: string* * **mimetype** * *type: string* * **name** * *description: Name of the file object* * *example: file.json* * *type: string* * **path** * *description: Path of the file object* * *example: parent\_folder/sample\_folder/file.json* * *type: string* * **size** * *type: integer* * **type** * *description: Object type* * **enum** * *0: * * *1: json* * *2: directory* * *type: string* * **writable** * *type: boolean* * *type: object* * **Flow** * *description: Represents information related to a SingleStore Flow instance* * **properties** * **createdAt** * *description: Timestamp of when the Flow instance was created* * *format: date-time* * *type: string* * **databaseName** * *description: Name of the SingleStore database associated with the Flow instance* * *type: string* * **deletedAt** * *description: (If included in the output) The timestamp of when the Flow instance was terminated* * *format: date-time* * *type: string* * **endpoint** * *description: Endpoint of the Flow instance* * *type: string* * **flowID** * *description: ID of the Flow instance* * *format: uuid* * *type: string* * **name** * *description: Name of the Flow instance* * *type: string* * **size** * *description: Size of the Flow instance (in Flow size notation), such as "F1"* * *type: string* * **status** * *description: Status of the Flow instance, such as whether it is Running or Deleted* * *type: string* * **userName** * *description: Name of the SingleStore database user associated with the Flow instance* * *type: string* * **workspaceID** * *description: ID of the workspace associated with the Flow instance* * *format: uuid* * *type: string* * **required** * *0: name* * *1: flowID* * *2: createdAt* * *type: object* * **FlowCreate** * *description: Represents the information specified while creating a SingleStore Flow instance* * **properties** * **databaseName** * *description: Name of the SingleStore database to connect to* * *example: database\_1* * *type: string* * **name** * *description: Name of the Flow instance* * *example: flow-instance-1* * *type: string* * **size** * *description: Size of the Flow instance (in Flow size notation), such as "F1"* * *example: F2* * *type: string* * **userName** * *description: Username of the SingleStore database user with which to connect to the SingleStore database* * *example: admin* * *type: string* * **workspaceID** * *description: The ID of the workspace to associate the Flow instance with* * *example: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *format: uuid* * *type: string* * **required** * *0: name* * *1: workspaceID* * *2: userName* * *3: databaseName* * *type: object* * **IdentityRole** * **properties** * **resourceID** * *description: The resource id* * *format: uuid* * *type: string* * **resourceType** * *description: The resource type* * *type: string* * **role** * *description: The role can access the resource* * *type: string* * **required** * *0: role* * *1: resourceType* * *2: resourceID* * *type: object* * **Job** * **properties** * **completedExecutionsCount** * *description: Count of completed executions for the job* * *type: integer* * **createdAt** * *description: Creation time of the job* * *format: date-time* * *type: string* * **description** * *description: Description of the job* * *example: Sample Description* * *type: string* * **enqueuedBy** * *description: ID of the user who created the job* * *format: uuid* * *type: string* * **executionConfig** * *$ref: #/components/schemas/JobExecutionConfig* * **jobID** * *description: ID of the job* * *format: uuid* * *type: string* * **jobMetadata** * *description: Array containing information about the max/avg execution duration and the number of executions with a particular status for the job* * **items** * *$ref: #/components/schemas/JobMetadata* * *type: array* * **name** * *description: Name of the job* * *example: Sample Job* * *type: string* * **schedule** * *$ref: #/components/schemas/JobSchedule* * **targetConfig** * *$ref: #/components/schemas/JobTargetConfig* * **terminatedAt** * *description: Termination time of the job* * *format: date-time* * *type: string* * **required** * *0: jobID* * *1: enqueuedBy* * *2: executionConfig* * *3: schedule* * *4: createdAt* * *5: completedExecutionsCount* * *6: jobMetadata* * *type: object* * **JobCreate** * *description: Represents the information specified when creating a job.* * **properties** * **description** * *description: Description of the job* * *example: Job Description* * *type: string* * **executionConfig** * **properties** * **createSnapshot** * *description: Indicates whether the job will save snapshots* * *type: boolean* * **notebookPath** * *description: Path to the shared notebook file that contains the code that needs to be run on a schedule as part of this job* * *example: sample\_notebook.ipynb* * *type: string* * **runtimeName** * *description: Name of the runtime* * *example: notebooks-cpu-small* * *type: string* * **required** * *0: notebookPath* * *1: createSnapshot* * *type: object* * **name** * *description: Name of the job* * *example: Job Name* * *type: string* * **parameters** * *description: Array containing the parameters for the job* * **items** * *$ref: #/components/schemas/JobParameter* * *type: array* * **schedule** * *$ref: #/components/schemas/JobSchedule* * **targetConfig** * *$ref: #/components/schemas/JobTargetConfig* * **required** * *0: name* * *1: executionConfig* * *2: schedule* * *type: object* * **JobExecutionConfig** * **properties** * **createSnapshot** * *description: Indicates whether the job will save snapshots* * *type: boolean* * **maxAllowedExecutionDurationInMinutes** * *description: Maximum allowed execution duration for the job in minutes* * *type: integer* * **notebookPath** * *description: Path to the shared notebook file that contains the code that needs to be run on a schedule as part of this job* * *example: sample\_notebook.ipynb* * *type: string* * **required** * *0: notebookPath* * *1: createSnapshot* * *2: maxAllowedExecutionDurationInMinutes* * *type: object* * **JobMetadata** * **properties** * **avgDurationInSeconds** * *description: Average duration of executions (in seconds) with the corresponding status* * *format: float* * *type: number* * **count** * *description: The number of executions with the corresponding status* * *type: integer* * **maxDurationInSeconds** * *description: Maximum duration of executions (in seconds) with the corresponding status* * *format: float* * *type: number* * **status** * **enum** * *0: Unknown* * *1: Scheduled* * *2: Running* * *3: Completed* * *4: Failed* * *5: Error* * *6: Canceled* * *example: Completed* * *type: string* * **required** * *0: status* * *1: count* * *type: object* * **JobParameter** * **properties** * **name** * *description: Name of the parameter* * *example: foo* * *type: string* * **type** * *description: Type of the parameter* * **enum** * *0: string* * *1: integer* * *2: float* * *3: boolean* * *example: integer* * *type: string* * **value** * *description: Value of the parameter* * *example: 10* * *type: string* * **required** * *0: name* * *1: value* * *2: type* * *type: object* * **JobSchedule** * **properties** * **executionIntervalInMinutes** * *description: The time interval between executions of this job. This is required for a Recurring job.* * *type: integer* * **mode** * **enum** * *0: Recurring* * *1: Once* * *type: string* * **startAt** * *description: The time at which the first execution of this job should begin.* * *format: date-time* * *type: string* * **required** * *0: mode* * *type: object* * **JobTargetConfig** * **properties** * **databaseName** * *description: Name of the database* * *example: sample\_db* * *type: string* * **resumeTarget** * *description: Indicates whether executions of this job will resume the target associated with the job, if it is suspended* * *type: boolean* * **targetID** * *description: The target ID for the job, this could be the ID of Workspace, Cluster or VirtualWorkspace (in case of shared-tier).* * *format: uuid* * *type: string* * **targetType** * **enum** * *0: Workspace* * *1: Cluster* * *2: VirtualWorkspace* * *type: string* * **required** * *0: targetType* * *1: targetID* * *2: resumeTarget* * *type: object* * **Model** * **properties** * **connectionURL** * *description: Model base url* * *type: string* * **modelName** * *description: Name or ID of the model* * *type: string* * **name** * *description: Name of the Model* * *type: string* * **serviceID** * *description: Model ID* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **required** * *0: serviceID* * *1: name* * *2: connectionURL* * *3: modelName* * *type: object* * **ModelStart** * **properties** * **hostingPlatform** * *description: Platform hosting the model* * **enum** * *0: Nova* * *1: Amazon* * *2: Azure* * *example: Nova* * *type: string* * **modelID** * *description: ID of the model* * *type: string* * **modelName** * *description: Name of the model* * *type: string* * **rawConnectionURL** * *description: Direct connection URL to the model service* * *type: string* * **version** * *description: Version of the model* * *type: string* * **required** * *0: modelName* * *1: modelID* * *2: version* * *3: hostingPlatform* * *4: rawConnectionURL* * *type: object* * **NotebookCloudFunction** * *description: A cloud function resource* * **properties** * **createdAt** * *description: Timestamp of when the cloud function was created* * *example: 2025-08-08T13:41:45.835577Z* * *format: date-time* * *type: string* * **createdBy** * *description: ID of the user that created the cloud function* * *example: 41e2268a-b000-4000-9000-4ed98936a401* * *format: uuid* * *type: string* * **description** * *description: Description of the cloud function* * *example: My custom cloud function* * *type: string* * **endpoint** * *description: The endpoint URL for invoking the cloud function* * *example: https\://app.singlestore.com/functions/7b68b78d-a000-4000-9000-0864fd331524/* * *type: string* * **lastUpdatedAt** * *description: Timestamp of when the cloud function was last updated* * *example: 2025-08-09T10:20:30.123456Z* * *format: date-time* * *type: string* * **name** * *description: Name of the cloud function* * *example: my-cloud-function* * *type: string* * **serviceID** * *description: Unique identifier for the cloud function* * *example: 7b68b78d-a000-4000-9000-0864fd331524* * *format: uuid* * *type: string* * **status** * *description: Current status of the cloud function* * **enum** * *0: Initializing* * *1: Active* * *2: Failed* * *3: Deleted* * *4: Idle* * *example: Initializing* * *type: string* * **required** * *0: serviceID* * *1: name* * *2: status* * *3: createdBy* * *4: createdAt* * *type: object* * **NotebookCloudFunctionPaginationMetadata** * *description: Pagination metadata for the cloud functions list* * **properties** * **count** * *description: Number of cloud functions in the current page* * *type: integer* * **hasNextPage** * *description: Indicates if there is a next page* * *type: boolean* * **hasPreviousPage** * *description: Indicates if there is a previous page* * *type: boolean* * **totalCount** * *description: Total number of cloud functions* * *type: integer* * **required** * *0: count* * *1: hasPreviousPage* * *2: hasNextPage* * *3: totalCount* * *type: object* * **Organization** * *description: Represents information related to an organization* * **properties** * **firewallRanges** * *description: The list of allowed IP addresses which can access the Management API* * **items** * *type: string* * *type: array* * **name** * *description: Name of the organization* * *type: string* * **orgID** * *description: ID of the organization* * *format: uuid* * *type: string* * **required** * *0: orgID* * *type: object* * **PermissionDefinition** * **properties** * **description** * *description: Permission description* * *type: string* * **permission** * *description: Permission name* * *type: string* * **resourceType** * *description: Resource type for the permission* * *type: string* * **required** * *0: permission* * *1: resourceType* * *type: object* * **PermissionsForResourceType** * **properties** * **permissions** * *description: The permissions of the resource type on the role* * **items** * *type: string* * *type: array* * **resourceType** * *description: Resource type* * *type: string* * **required** * *0: resourceType* * *1: permissions* * *type: object* * **PermissionsSpecification** * **properties** * **permissions** * *description: The permissions of the resource type on the role* * **items** * *type: string* * *type: array* * **resourceType** * *description: Resource type* * *type: string* * **required** * *0: resourceType* * *1: permissions* * *type: object* * **PrivateConnection** * *description: Represents information related to a private link connection* * **properties** * **activeAt** * *description: The timestamp of when the private connection became active* * *example: 2023-09-18T09:56:56Z* * *type: string* * **allowList** * *description: The private connection allow list. This is the account ID for AWS, subscription ID for Azure, and the project name GCP* * *example: my-allow-list* * *type: string* * **allowedPrivateLinkIDs** * **items** * *description: Allowed Private Link ID* * *example: vpce-01a376f2625261f22* * *type: string* * *type: array* * **createdAt** * *description: The timestamp of when the private connection was created* * *example: 2023-09-18T09:56:56Z* * *type: string* * **deletedAt** * *description: The timestamp of when the private connection was deleted* * *example: 2023-09-18T09:56:56Z* * *type: string* * **endpoint** * *description: The service endpoint* * *example: com.amazonaws.vpce.us-east-1.vpce-svc-08e20961f648f9d* * *type: string* * **outboundAllowList** * *description: The account ID which must be allowed for outbound connections* * *example: arn:aws:iam:xxxxxxxxx:root* * *type: string* * **privateConnectionID** * *description: The ID of the private connection* * *example: 8900372e-0000-1000-9000-4c47638040af* * *format: uuid* * *type: string* * **serviceName** * *description: The name of the private connection service* * *example: My service* * *type: string* * **sqlPort** * *description: The SQL port* * *type: number* * **status** * *description: The status of the private connection* * **enum** * *0: PENDING* * *1: ACTIVE* * *2: DELETED* * *example: ACTIVE* * *type: string* * **type** * *description: The private connection type* * **enum** * *0: INBOUND* * *1: OUTBOUND* * *example: INBOUND* * *type: string* * **updatedAt** * *description: The timestamp of when the private connection was last updated* * *example: 2023-09-18T09:56:56Z* * *type: string* * **websocketsPort** * *description: The websockets port* * *type: number* * **workspaceGroupID** * *description: The ID of the workspace group containing the private connection* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *format: uuid* * *type: string* * **workspaceID** * *description: The ID of the workspace to connect with* * *example: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *format: uuid* * *type: string* * **required** * *0: privateConnectionID* * *1: workspaceGroupID* * *type: object* * **PrivateConnectionCreate** * *description: Represents the information specified when creating a private connection* * **properties** * **allowList** * *description: The private connection allow list* * *example: my-allow-list* * *type: string* * **kaiEndpointID** * *description: VPC Endpoint ID for AWS* * *example: vpce-12345678901234567* * *type: string* * **serviceName** * *description: The name of the private connection service* * *example: My private link* * *type: string* * **sqlPort** * *description: The SQL port* * *type: number* * **type** * *description: The private connection type* * **enum** * *0: INBOUND* * *1: OUTBOUND* * *example: INBOUND* * *type: string* * **websocketsPort** * *description: The websockets port* * *type: number* * **workspaceGroupID** * *description: The ID of the workspace group containing the private connection* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *format: uuid* * *type: string* * **workspaceID** * *description: The ID of the workspace to connect with* * *format: uuid* * *type: string* * **required** * *0: workspaceGroupID* * *type: object* * **PrivateConnectionKaiInfo** * *description: Represents the information for creating private connection to SingleStore Kai* * **properties** * **serviceName** * *description: VPC Endpoint Service Name for AWS* * *example: com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdefg* * *type: string* * *type: object* * **PrivateConnectionOutboundAllowList** * *description: Represents information related to a private connection outbound allow list* * **properties** * **outboundAllowList** * *description: The account ID allowed for outbound connections* * *example: arn:aws:iam:xxxxxxxxx:root* * *type: string* * *type: object* * **PrivateConnectionUpdate** * *description: Represents the information specfied when updating a private connection* * **properties** * **allowList** * *description: The private connection allow list* * *example: my-allow-list* * *type: string* * *type: object* * **Project** * **allOf** * **0** * *$ref: #/components/schemas/ProjectIDResponse* * **1** * *$ref: #/components/schemas/ProjectCreate* * **2** * **properties** * **createdAt** * *description: The timestamp of when the project was created* * *format: date-time* * *type: string* * **required** * *0: createdAt* * *type: object* * *description: Full project representation* * **ProjectCreate** * **allOf** * **0** * *$ref: #/components/schemas/ProjectUpdate* * **1** * **properties** * **edition** * *$ref: #/components/schemas/ProjectEdition* * **required** * *0: edition* * *type: object* * *description: Request body for creating a project* * **ProjectEdition** * *description: Edition of the project* * **enum** * *0: STANDARD* * *1: ENTERPRISE* * *2: PREMIUM* * *3: DEDICATED* * *4: SHARED* * *type: string* * **ProjectIDResponse** * *description: Represents the project identifier response payload* * **properties** * **projectID** * *format: uuid* * *type: string* * **required** * *0: projectID* * *type: object* * **ProjectUpdate** * *description: Request body for updating a project* * **properties** * **name** * *description: Name of the project* * *type: string* * **required** * *0: name* * *type: object* * **Region** * *description: Represents information related to a region in which a workspace group is created* * **properties** * **provider** * *$ref: #/components/schemas/CloudProvider* * **region** * *description: Name of the region* * *type: string* * **regionID** * *description: ID of the region* * *format: uuid* * *type: string* * **required** * *0: regionID* * *1: region* * *2: provider* * *type: object* * **RegionV2** * *description: Represents information related to a region in which a workspace group is created* * **properties** * **provider** * *$ref: #/components/schemas/CloudProvider* * **region** * *description: Name of the region* * *type: string* * **regionName** * *description: The region code name* * *type: string* * **required** * *0: region* * *1: provider* * *2: regionName* * *type: object* * **ReplicatedDatabase** * *description: Represents information related to a database's replication status* * **properties** * **databaseName** * *description: Name of the database* * *type: string* * **duplicationState** * *description: Duplication state of the database* * **enum** * *0: Pending* * *1: Active* * *2: Inactive* * *3: Error* * *type: string* * **region** * *description: Name of the region* * *type: string* * **required** * *0: databaseName* * *1: region* * *2: duplicationState* * *type: object* * **ResourceRole** * **properties** * **description** * *description: the description about the role* * *type: string* * **grants** * **items** * *$ref: #/components/schemas/ResourceRoleGrant* * *type: array* * **role** * *description: the role can access the resource* * *type: string* * **required** * *0: role* * *1: description* * *2: grants* * *type: object* * **ResourceRoleGrant** * **properties** * **createdAt** * *description: Creation timestamp* * *format: date-time* * *type: string* * **createdBy** * *$ref: #/components/schemas/UserInfo* * **identity** * **oneOf** * **0** * *$ref: #/components/schemas/UserInfo* * **1** * *$ref: #/components/schemas/TeamInfo* * **inheritedFrom** * **properties** * **resourceID** * *description: ID of the resource* * *format: uuid* * *type: string* * **resourceType** * *type: string* * **role** * *type: string* * **required** * *0: role* * *1: resourceType* * *2: resourceID* * *type: object* * **required** * *0: identity* * *1: createdAt* * *type: object* * **RetentionPeriodParams** * *description: Represents the information to be specified when updating the retention period of a workspace group.* * **properties** * **retentionPeriod** * *description: Retention period in minutes* * *type: integer* * **required** * *0: retentionPeriod* * *type: object* * **RoleCreate** * **allOf** * **0** * *$ref: #/components/schemas/RoleUpdate* * **1** * **properties** * **role** * *description: The name for the custom role* * *type: string* * **required** * *0: role* * *type: object* * **RoleCreateV2** * **allOf** * **0** * *$ref: #/components/schemas/RoleReplaceV2* * **1** * **properties** * **role** * *description: The name for the custom role* * *type: string* * **required** * *0: role* * *type: object* * **RoleDefinition** * **properties** * **createdAt** * *description: Creation timestamp.* * *format: date-time* * *type: string* * **createdBy** * *$ref: #/components/schemas/UserInfo* * **description** * *description: A description of the role* * *type: string* * **inherits** * *description: The roles that current role will inherit from* * **items** * *$ref: #/components/schemas/TypedRole* * *type: array* * **isCustom** * *description: Indicate the role is a custom role* * *type: boolean* * **permissions** * *description: The permissions that the role has* * **items** * *type: string* * *type: array* * **resourceType** * *description: The resource type for the role* * *type: string* * **role** * *description: The role name* * *type: string* * **updatedAt** * *description: Timestamp of most recent state change.* * *format: date-time* * *type: string* * **updatedBy** * *$ref: #/components/schemas/UserInfo* * **required** * *0: role* * *1: resourceType* * *2: permissions* * *3: isCustom* * *4: inherits* * *type: object* * **RoleDefinitionV2** * **properties** * **createdAt** * *description: Creation timestamp.* * *format: date-time* * *type: string* * **createdBy** * *$ref: #/components/schemas/UserInfo* * **description** * *description: A description of the role* * *type: string* * **permissions** * *description: Permissions for the role definition* * **items** * *$ref: #/components/schemas/PermissionsForResourceType* * *type: array* * **role** * *description: The role name* * *type: string* * **updatedAt** * *description: Timestamp of most recent state change.* * *format: date-time* * *type: string* * **updatedBy** * *$ref: #/components/schemas/UserInfo* * **required** * *0: role* * *1: permissions* * *type: object* * **RoleReplaceV2** * **properties** * **description** * *description: A description of the role* * *type: string* * **permissions** * *description: All permissions defined for the role to be replaced* * **items** * *$ref: #/components/schemas/PermissionsSpecification* * *type: array* * **role** * *description: Optional role name and must match the role path parameter when provided* * *type: string* * **required** * *0: permissions* * *type: object* * **RoleUpdate** * **properties** * **description** * *description: A description of the role* * *type: string* * **inherits** * *description: The roles that the current role will inherit from* * **items** * *$ref: #/components/schemas/TypedRole* * *type: array* * **permissions** * *description: The permissions are seperated by the resourceType, like you can put 'View Storage' DR for 'Cluster' resourceType but not 'Organization'* * **items** * *type: string* * *type: array* * **required** * *0: permissions* * *1: inherits* * *type: object* * **RuntimesResult** * **properties** * **description** * *description: The description of the runtime* * *example: 1750m vCPU | 15Gi* * *type: string* * **name** * *description: The name of the runtime* * *example: notebooks-cpu-small* * *type: string* * **required** * *0: name* * *1: description* * *type: object* * **Secret** * *description: Represents information related to SingleStore Helios secrets.* * **properties** * **createdAt** * *description: The timestamp of when the secret was created.* * *example: 2023-09-18T09:56:56Z* * *type: string* * **createdBy** * *description: The ID of the user that created the secret.* * *example: 17290909-0000-1000-9000-e30410f1b05a* * *format: uuid* * *type: string* * **deletedAt** * *description: The timestamp of when the secret was deleted.* * *example: 2023-09-18T09:56:56Z* * *type: string* * **deletedBy** * *description: The ID of the user that deleted the secret.* * *example: 17290909-0000-1000-9000-e30410f1b05a* * *format: uuid* * *type: string* * **lastUpdatedAt** * *description: The timestamp of when the secret was last updated.* * *example: 2023-09-18T09:56:56Z* * *type: string* * **lastUpdatedBy** * *description: The ID of the user that last updated the secret.* * *example: 17290909-0000-1000-9000-e30410f1b05a* * *format: uuid* * *type: string* * **name** * *description: The name of the secret. It can only contain alphanumeric characters or underscores.* * *example: my\_secret* * *type: string* * **secretID** * *description: The ID of the secret.* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **value** * *description: The value of the secret.* * *example: my\_secret\_value* * *type: string* * **required** * *0: secretID* * *1: name* * *2: createdAt* * *3: createdBy* * *4: lastUpdatedAt* * *5: lastUpdatedBy* * *type: object* * **SecretCreate** * *description: Represents the information specified when creating a secret.* * **properties** * **name** * *description: The name of the secret. It can only contain alphanumeric characters or underscores.* * *example: my\_secret* * *type: string* * **value** * *description: The value of the secret.* * *example: my\_secret\_value* * *type: string* * **required** * *0: name* * *1: value* * *type: object* * **SecretUpdate** * *description: Represents the information specified when updating a secret.* * **properties** * **value** * *description: The value of the secret.* * *example: my\_secret\_value* * *type: string* * *type: object* * **ServiceAccount** * **allOf** * **0** * *$ref: #/components/schemas/ServiceAccountIDResponse* * **1** * *$ref: #/components/schemas/ServiceAccountUpdate* * **2** * **properties** * **createdBy** * *description: Identifier of the user that created the service account.* * *format: uuid* * *type: string* * **email** * *description: The generated fake email address for the service account.* * *example: service-account-abc123\@service.singlestore.com* * *type: string* * **required** * *0: email* * *1: name* * *2: description* * *type: object* * *description: Information about a SingleStoreDB Cloud service account.* * **ServiceAccountCreate** * **allOf** * **0** * *$ref: #/components/schemas/ServiceAccountUpdate* * **1** * **required** * *0: name* * *type: object* * *description: Information specified when creating a service account.* * **ServiceAccountIDResponse** * *description: Represents a service account identifier response payload.* * **properties** * **serviceAccountID** * *description: Service account identifier.* * *format: uuid* * *type: string* * **required** * *0: serviceAccountID* * *type: object* * **ServiceAccountUpdate** * *description: Information specified when updating a service account.* * **properties** * **description** * *description: Service account description.* * *example: Service account used by ingestion jobs.* * *type: string* * **name** * *description: Service account name.* * *example: ingestion-service-renamed* * *type: string* * *type: object* * **SharedTierCreateUser** * *description: Represents the information specified when creating a user in a shared tier workspace* * **properties** * **password** * *description: The starter workspace user password to connect the new user to the database.* * *example: My-password1!* * *type: string* * **userName** * *description: The starter workspace user name to connect the new user to the database* * *example: new-user* * *type: string* * **required** * *0: userName* * *type: object* * **SharedTierCreateVirtualCluster** * **properties** * **databaseName** * *description: Name of the database* * *example: database\_60c6b* * *type: string* * **name** * *description: Name of the starter cluster* * *example: starter-cluster-1* * *type: string* * **projectID** * *description: Assigns the starter cluster to a project* * *example: 2d7bd375-6303-46c4-903a-8a3d5b4922c0* * *format: uuid* * *type: string* * **provider** * *description: Name of the provider* * **enum** * *0: AWS* * *1: GCP* * *2: AZURE* * *type: string* * **regionName** * *description: The region code* * *example: us-east-1* * *type: string* * **required** * *0: name* * *1: databaseName* * *2: provider* * *3: regionName* * *type: object* * **SharedTierCreateVirtualWorkspace** * **properties** * **databaseName** * *description: Name of the database* * *example: database\_60c6b* * *type: string* * **name** * *description: Name of the shared workspace* * *example: shared-workspace-1* * *type: string* * **projectID** * *description: Assigns the shared workspace to a project* * *example: 2d7bd375-6303-46c4-903a-8a3d5b4922c0* * *format: uuid* * *type: string* * **provider** * *description: Name of the provider* * **enum** * *0: AWS* * *1: GCP* * *2: AZURE* * *type: string* * **regionName** * *description: The region code* * *example: us-east-1* * *type: string* * **required** * *0: name* * *1: databaseName* * *2: provider* * *3: regionName* * *type: object* * **SharedTierMigrateStatusV2** * *description: Represents the information related to the migration of a Shared Edition deployment (starter cluster) to a cluster group* * **properties** * **clusterID** * *description: ID of the new cluster* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **migratedScheduledJobs** * *description: Indicates whether the migration operation included scheduled jobs.* * *type: boolean* * **migrationStatus** * *description: Status of the migration operation* * **enum** * *0: PENDING* * *1: SUCCEEDED* * *2: FAILED* * *example: SUCCEEDED* * *type: string* * **required** * *0: migrationStatus* * *1: clusterID* * *2: migratedScheduledJobs* * *type: object* * **SharedTierUpdateUser** * *description: Represents the information specified when updating a user in a shared tier workspace* * **properties** * **password** * *description: The virtual workspace user password to connect the new user to the database.* * *example: My-password1!* * *type: string* * *type: object* * **SharedTierVirtualCluster** * **properties** * **databaseName** * *description: Name of the database* * *example: database\_60c6b* * *type: string* * **endpoint** * *description: The application endpoint of the SingleStore database* * *example: svc-3482219c-a389-4079-b18b-d50662524e8a-shared-dml.aws-virginia-6.svc.singlestore.com:3333* * *type: string* * **mysqlDmlPort** * *description: The MySQL DML port* * *type: integer* * **name** * *description: Name of the starter cluster* * *example: starter-cluster-1* * *type: string* * **projectID** * *description: ID of the project to which the virtual cluster is assigned.* * *example: 2d7bd375-0000-1000-9000-8a3d5b4922c0* * *format: uuid* * *type: string* * **virtualClusterID** * *description: ID of the starter cluster* * *format: uuid* * *type: string* * **websocketPort** * *description: The websockets port* * *type: integer* * *type: object* * **SharedTierVirtualWorkspace** * **properties** * **databaseName** * *description: Name of the database* * *example: database\_60c6b* * *type: string* * **endpoint** * *description: The application endpoint of the SingleStore database* * *example: svc-3482219c-a389-4079-b18b-d50662524e8a-shared-dml.aws-virginia-6.svc.singlestore.com:3333* * *type: string* * **mysqlDmlPort** * *description: The MySQL DML port* * *type: integer* * **name** * *description: Name of the starter workspace* * *example: starter-workspace-1* * *type: string* * **projectID** * *description: ID of the project to which the virtual workspace is assigned.* * *example: 2d7bd375-0000-1000-9000-8a3d5b4922c0* * *format: uuid* * *type: string* * **virtualWorkspaceID** * *description: ID of the starter workspace* * *format: uuid* * *type: string* * **websocketPort** * *description: The websockets port* * *type: integer* * *type: object* * **SimulateUsageRequest** * *description: Request body for simulating billing usage* * **example** * *endBefore: 2024-01-31T23:59:59Z* * **resourceConfigurations** * **0** * *edition: Standard* * *endBefore: 2024-01-31T23:59:59Z* * *id: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *size: S-1* * *startAt: 2024-01-01T00:00:00Z* * *status: Active* * *startAt: 2024-01-01T00:00:00Z* * **properties** * **endBefore** * *description: The end time of the simulation period (must be after startAt)* * *example: 2024-06-02T08:00:00Z* * *format: date-time* * *type: string* * **resourceConfigurations** * *description: Array of resource configurations to simulate* * **items** * *$ref: #/components/schemas/SimulationResourceConfig* * *type: array* * **startAt** * *description: The start time of the simulation period* * *example: 2024-06-01T08:00:00Z* * *format: date-time* * *type: string* * **required** * *0: startAt* * *1: endBefore* * *2: resourceConfigurations* * *type: object* * **SimulateUsageResponse** * *description: Response containing simulated usage data* * **properties** * **resourceUsage** * *description: Array of simulated resource usage entries* * **items** * *$ref: #/components/schemas/SimulatedResourceUsage* * *type: array* * *type: object* * **SimulatedResourceUsage** * *description: Simulated usage data for a single resource* * **example** * *endBefore: 2024-01-31T23:59:59Z* * *id: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *metric: ComputeHour* * *name: test-workspace* * *startAt: 2024-01-01T00:00:00Z* * *type: Workspace* * **properties** * **amount** * *description: The computed usage amount* * *format: double* * *type: number* * **endBefore** * *description: The end time of the usage period (optional)* * *format: date-time* * *type: string* * **id** * *description: The ID of the resource* * *format: uuid* * *type: string* * **metric** * *description: The metric used to measure usage (optional)* * **enum** * *0: ComputeHour* * *type: string* * **name** * *description: The name of the resource (optional)* * *type: string* * **parentID** * *description: The ID of the parent resource (optional)* * *format: uuid* * *type: string* * **startAt** * *description: The start time of the usage period (optional)* * *format: date-time* * *type: string* * **type** * *description: The type of resource (optional)* * **enum** * *0: Cluster* * *1: Workspace* * *2: WorkspaceGroup* * *type: string* * **required** * *0: id* * *1: amount* * *type: object* * **SimulationResourceConfig** * *description: Configuration for a simulated resource* * **properties** * **cacheMultiplier** * *description: The cache multiplier (optional)* * **enum** * *0: X1* * *1: X2* * *2: X4* * *type: string* * **cellID** * *description: The ID of the cell (optional)* * *format: uuid* * *type: string* * **edition** * *description: The edition of the resource (optional)* * **enum** * *0: Standard* * *1: Enterprise* * *2: Dedicated* * *type: string* * **endBefore** * *description: The end time for this resource configuration* * *format: date-time* * *type: string* * **id** * *description: The ID of the resource* * *format: uuid* * *type: string* * **multiAZ** * *description: Whether the resource uses multiple availability zones (optional)* * *type: boolean* * **scaleFactor** * *description: The scale factor (optional)* * **enum** * *0: X1* * *1: X2* * *2: X4* * *type: string* * **size** * *description: The size of the resource, e.g., S-00, S-1, S-2 (optional)* * *type: string* * **startAt** * *description: The start time for this resource configuration* * *format: date-time* * *type: string* * **status** * *description: The status of the resource (optional)* * **enum** * *0: Active* * *1: Paused* * *2: Terminated* * *type: string* * **required** * *0: id* * *1: startAt* * *2: endBefore* * *type: object* * **StorageDRSetup** * *description: Represents the information specified to setup Storage DR* * **properties** * **autoReplication** * *description: If set to true, all existing and future databases created will be automatically replicated to the secondary region. If set to false, turns off auto replication. * * *type: boolean* * **databaseNames** * *description: List of database names (can be an empty list if setting up Auto-Replication).* * **items** * *example: x\_db* * *type: string* * *type: array* * **regionID** * *description: Region ID of the secondary region* * *format: uuid* * *type: string* * **required** * *0: regionID* * *1: databaseNames* * *type: object* * **StorageDRStatus** * *description: Represents Storage DR status information* * **properties** * **compute** * *description: Represents information related to a workspace group's latest storage DR operation* * **properties** * **completedAttachments** * *description: The number of database attachments that have been setup* * *type: integer* * **completedWorkspaces** * *description: The number of workspaces that have been setup* * *type: integer* * **secondaryWorkspaceGroupIDs** * *description: The IDs of the secondary (standby) workspace groups in this DR group* * **items** * *format: uuid* * *type: string* * *type: array* * **storageDRState** * *description: Status of Storage DR operation* * **enum** * *0: Active* * *1: Completed* * *2: Failed* * *3: Expired* * *4: Canceled* * *type: string* * **storageDRType** * *description: Name of Storage DR operation* * **enum** * *0: Failover* * *1: Failback* * *2: PreProvisionStart* * *3: PreProvisionStop* * *type: string* * **totalAttachments** * *description: The total number of database attachments to setup* * *type: integer* * **totalWorkspaces** * *description: The total number of workspaces to setup* * *type: integer* * **required** * *0: storageDRType* * *1: storageDRState* * *2: totalWorkspaceBuildTasks* * *3: totalDatabaseAttachTasks* * *4: completedWorkspaceBuildTasks* * *5: completedDatabaseAttachTasks* * *type: object* * **storage** * **items** * *$ref: #/components/schemas/ReplicatedDatabase* * *type: array* * **required** * *0: compute* * *1: storage* * *type: object* * **TableEgressDrop** * *description: Represents the information specified to drop exporting data from a table to Apache Iceberg format* * **properties** * **egressID** * *description: The ID of the export operation* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *type: string* * **required** * *0: egressID* * *type: object* * **TableEgressResume** * *description: Represents the information specified to resume exporting data from a table to Apache Iceberg format* * **properties** * **egressID** * *description: The ID of the export operation* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *type: string* * **required** * *0: egressID* * *type: object* * **TableEgressStart** * *description: Represents the information specified to start exporting data from a table to Apache Iceberg format* * **properties** * **catalogInfo** * *description: Catalog information for egressing data* * *type: object* * **columnNames** * **items** * *description: The names of the columns to egress* * *example: column1* * *type: string* * *type: array* * **databaseName** * *description: The name of the database* * *example: sample\_db* * *type: string* * **incremental** * *description: Whether the export process is incremental or not* * *type: boolean* * **partitionSpec** * *description: Partition specification of the table* * *type: object* * **properties** * *description: Properties of the table* * *type: object* * **refreshInterval** * *description: The incremental export process refresh internal in seconds* * *type: integer* * **sortOrderSpec** * *description: Sort order specification of the table* * *type: object* * **storageID** * *description: The SingleStore identifier that uniquely identifies the database associated with the export process* * *type: string* * **storageInfo** * *description: Storage information for egressing data* * *type: object* * **tableName** * *description: The name of the table* * *example: sample\_table* * *type: string* * **required** * *0: databaseName* * *1: tableName* * *2: columnNames* * *3: storageInfo* * *4: catalogInfo* * *type: object* * **TableEgressStatus** * *description: Represents the status of an export operation* * **properties** * **status** * *description: The status of the export operation* * *example: Running* * *type: string* * **statusMsg** * *description: The status message of the export operation* * *type: string* * **required** * *0: status* * *type: object* * **TableEgressStatusParams** * *description: Represents the status of exporting data from a table to Apache Iceberg format* * **properties** * **egressID** * *description: The ID of the export operation* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *type: string* * **required** * *0: egressID* * *type: object* * **TableEgressSuspend** * *description: Represents the information specified to suspend exporting data from a table to Apache Iceberg format* * **properties** * **egressID** * *description: The ID of the export operation* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *type: string* * **required** * *0: egressID* * *type: object* * **Team** * *description: A SingleStoreDB Cloud team.* * **example** * *createdAt: 2023-09-18T09:56:56Z* * *description: All analysts in the abc.com organization* * **memberTeams** * **0** * *description: * * *name: APAC Analysts* * *teamID: 3a6fd5c9-616c-4dea-b41c-92eb1d442889* * **memberUsers** * **0** * *email: jane.doe\@abc.com* * *firstName: Jane* * *lastName: Doe* * *userID: 121e74be-6b93-44ac-b240-9be663f680de* * **1** * *email: jack.smith\@abc.com* * *firstName: Jack* * *lastName: Smith* * *userID: a69181f0-8d64-4b31-a516-ce1b6483edd4* * *name: Analysts* * *teamID: c74bb6a6-0000-1000-9000-1d874fa277b0* * **properties** * **createdAt** * *description: Team creation timestamp.* * *example: 2023-09-18T09:56:56Z* * *type: string* * **description** * *description: Team description.* * *example: All analysts in the abc.com organization* * *type: string* * **memberTeams** * **items** * *$ref: #/components/schemas/TeamInfo* * *type: array* * **memberUsers** * **items** * *$ref: #/components/schemas/UserInfo* * *type: array* * **name** * *description: Team name.* * *example: Analysts* * *type: string* * **teamID** * *description: Team identifier.* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **required** * *0: teamID* * *1: name* * *2: description* * *type: object* * **TeamCreate** * *description: Information specified when creating a team.* * **properties** * **description** * *description: Optional team description.* * *example: All analysts in the abc.com organization* * *type: string* * **name** * *description: Team name.* * *example: Analysts* * *type: string* * **required** * *0: name* * *type: object* * **TeamInfo** * *description: Summary information about a SingleStoreDB Cloud team.* * **properties** * **description** * *description: Team description.* * *example: All analysts in the abc.com organization* * *type: string* * **name** * *description: Team name.* * *example: Analysts* * *type: string* * **teamID** * *description: Team identifier.* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **required** * *0: teamID* * *1: name* * *2: description* * *type: object* * **TeamUpdate** * *description: Information specified when updating a team.* * **properties** * **addMemberTeamIDs** * *description: List of identifiers of teams to be added as members.* * **items** * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * *type: array* * **addMemberUserEmails** * *description: List of emails of users to be added as members.* * **items** * *example: jane.doe\@abc.com* * *type: string* * *type: array* * **addMemberUserIDs** * *description: List of identifiers of users to be added as members.* * **items** * *example: 121e74be-6b93-44ac-b240-9be663f680de* * *format: uuid* * *type: string* * *type: array* * **description** * *description: Team description.* * *example: All analysts in the abc.com organization* * *type: string* * **name** * *description: Team name.* * *example: Analysts* * *type: string* * **removeMemberTeamIDs** * *description: List of identifiers of teams to be removed from members.* * **items** * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * *type: array* * **removeMemberUserEmails** * *description: List of emails of users to be removed from members.* * **items** * *example: jane.doe\@abc.com* * *type: string* * *type: array* * **removeMemberUserIDs** * *description: List of identifiers of users to be removed from members.* * **items** * *example: 121e74be-6b93-44ac-b240-9be663f680de* * *format: uuid* * *type: string* * *type: array* * *type: object* * **TypedRole** * **properties** * **resourceType** * *description: Resource type of the role* * *type: string* * **role** * *description: The role name* * *type: string* * **required** * *0: role* * *1: resourceType* * *type: object* * **UpdateWindow** * *description: Represents information related to an update window* * **properties** * **day** * *description: Day of week (0-6), starting on Sunday* * *type: number* * **hour** * *description: Hour of day - 0 to 23 (UTC)* * *type: number* * **required** * *0: hour* * *1: day* * *type: object* * **User** * **allOf** * **0** * *$ref: #/components/schemas/UserInfo* * **1** * *description: Information about a SingleStoreDB Cloud user.* * **properties** * **cloudPrincipal** * *description: Customer cloud identity bound to a cloud principal user.* * *example: arn:aws:iam::123456789012:role/IngestionRole* * *type: string* * **createdBy** * *description: Identifier of the user that created a service account or cloud principal user.* * *format: uuid* * *type: string* * **description** * *description: Description for service account and cloud principal users.* * *example: Service account used by ingestion jobs.* * *type: string* * **userKind** * *description: User kind.* * **enum** * *0: Standard* * *1: ServiceAccount* * *2: CloudPrincipal* * *example: Standard* * *type: string* * **required** * *0: userKind* * *type: object* * **UserInfo** * *description: Summary information about a SingleStoreDB Cloud user.* * **properties** * **email** * *description: The email address of the user.* * *example: jane.doe\@abc.com* * *type: string* * **firstName** * *description: First name of the user.* * *example: Jane* * *type: string* * **lastName** * *description: Last name of the user.* * *example: Doe* * *type: string* * **userID** * *description: User identifier.* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **required** * *0: userID* * *1: email* * *2: firstName* * *3: lastName* * *type: object* * **UserInvitation** * *description: Information specified when creating a user invitation.* * **properties** * **actedAt** * *description: Timestamp of most recent state change.* * *format: date-time* * *type: string* * **createdAt** * *description: Creation timestamp.* * *format: date-time* * *type: string* * **email** * *description: User email address.* * *example: alice\@abc.com* * *type: string* * **invitationID** * *description: Invitation identifier.* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **message** * *description: Welcome message* * *example: Welcome to SingleStore\@abc.com* * *type: string* * **state** * *description: Invitation state. Pending, Accepted, Refused, or Revoked.* * **enum** * *0: Pending* * *1: Accepted* * *2: Refused* * *3: Revoked* * *example: Pending* * *type: string* * **teamIDs** * *description: List of user teams.* * **items** * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * *type: array* * *type: object* * **UserInvitationCreate** * *description: Information specified when creating a user invitation.* * **properties** * **email** * *description: User email address.* * *example: alice\@abc.com* * *format: email* * *type: string* * **teamIDs** * *description: List of user teams.* * **items** * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * *type: array* * **required** * *0: email* * *type: object* * **V2SimulateUsageResponse** * *description: Response containing simulated usage data* * **properties** * **resourceUsage** * *description: Array of simulated resource usage entries* * **items** * *$ref: #/components/schemas/V2SimulatedResourceUsage* * *type: array* * *type: object* * **V2SimulatedResourceUsage** * *description: Simulated usage data for a single resource* * **example** * *endBefore: 2024-01-31T23:59:59Z* * *id: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *metric: ComputeHour* * *name: test-cluster* * *startAt: 2024-01-01T00:00:00Z* * *type: Cluster* * **properties** * **amount** * *description: The computed usage amount* * *format: double* * *type: number* * **endBefore** * *description: The end time of the usage period (optional)* * *format: date-time* * *type: string* * **id** * *description: The ID of the resource* * *format: uuid* * *type: string* * **metric** * *description: The metric used to measure usage (optional)* * **enum** * *0: ComputeHour* * *type: string* * **name** * *description: The name of the resource (optional)* * *type: string* * **parentID** * *description: The ID of the parent resource (optional)* * *format: uuid* * *type: string* * **startAt** * *description: The start time of the usage period (optional)* * *format: date-time* * *type: string* * **type** * *description: The type of resource (optional)* * **enum** * *0: Cluster* * *type: string* * **required** * *0: id* * *1: amount* * *type: object* * **Workspace** * *description: Represents information related to a workspace* * **properties** * **autoScale** * *$ref: #/components/schemas/AutoScale* * **autoSuspend** * *description: (If included in the output) Represents the current auto suspend settings enabled for this workspace. If autoSuspend has an empty value, then the auto suspend settings are disabled* * **properties** * **idleAfterSeconds** * *description: (If included in the output) The duration (in seconds) the workspace must be inactive until it automatically suspends* * *type: number* * **idleChangedAt** * *description: (If included in the output) The timestamp when idleAfterSeconds was last changed* * *type: string* * **scheduledAfterSeconds** * *description: (If included in the output) The scheduled duration (in seconds) after which the workspace must be suspended* * *type: number* * **scheduledChangedAt** * *description: (If included in the output) The timestamp when scheduledSuspendAt was last changed* * *type: string* * **scheduledSuspendAt** * *description: (If included in the output) The timestamp when the workspace will be suspended* * *type: string* * **suspendType** * *description: The type of auto suspend currently enabled* * **enum** * *0: IDLE* * *1: SCHEDULED* * *type: string* * **suspendTypeChangedAt** * *description: (If included in the output) The timestamp when suspendType was last changed* * *type: string* * **required** * *0: suspendType* * *type: object* * **cacheConfig** * *description: Specifies the multiplier for the persistent cache associated with the workspace. It can have one of the following values: 1, 2, or 4. * * *type: number* * **createdAt** * *description: The timestamp of when the workspace was created* * *type: string* * **deploymentType** * *description: Specifies the deployment type for the workspace. It can have one of the following values: \`PRODUCTION\` or \`NON-PRODUCTION\`. If the value wasn't changed on creation, then the default will be \`PRODUCTION\`. If set to \`NON-PRODUCTION\`, the upgrades are only applied to the non-production workspaces. * * **enum** * *0: PRODUCTION* * *1: NON-PRODUCTION* * *type: string* * **endpoint** * *description: Endpoint to connect to the workspace* * *type: string* * **kaiEnabled** * *description: Specifies if the workspace is kai enabled or not.* * *type: boolean* * **lastResumedAt** * *description: (If included in the output) The timestamp of when the workspace was last resumed* * *type: string* * **name** * *description: Name of the workspace* * *type: string* * **resumeAttachments** * *description: (If included in the output) The result of database attachments after the workspace was resumed* * **items** * *description: Represents information related to database attachments* * **properties** * **attachment** * *description: The type of attachment* * **enum** * *0: READWRITE* * *1: READONLY* * *type: string* * **database** * *description: Name of the database* * *type: string* * **error** * *description: The error if the attachment was not successful* * *type: string* * **success** * *description: Whether the attachment was successful or not* * *type: boolean* * **required** * *0: database* * *1: attachment* * *2: success* * *type: object* * *type: array* * **scaleFactor** * *description: (If included in the output) The scale factor specified for the workspace. The scale factor can be 1, 2 or 4. * * *type: number* * **scalingProgress** * *description: (If included in the output) The current progress percentage for the scaling workspace* * *type: number* * **size** * *description: Size of the workspace (in workspace size notation), such as "S-00" or "S-1" * * *type: string* * **state** * *description: State of the workspace* * **enum** * *0: ACTIVE* * *1: PENDING* * *2: SUSPENDED* * *3: FAILED* * *4: TERMINATED* * *type: string* * **terminatedAt** * *description: (If included in the output) The timestamp of when the workspace was terminated* * *type: string* * **workspaceGroupID** * *description: ID of the workspace group containing the workspace* * *format: uuid* * *type: string* * **workspaceID** * *description: ID of the workspace* * *format: uuid* * *type: string* * **required** * *0: name* * *1: workspaceID* * *2: workspaceGroupID* * *3: size* * *4: state* * *5: createdAt* * *type: object* * **WorkspaceCreate** * *description: Represents the information specified while creating a workspace* * **properties** * **autoScale** * *$ref: #/components/schemas/AutoScale* * **autoSuspend** * *description: Auto suspend settings for the workspace. If this field is not provided, no settings will be enabled.* * **properties** * **suspendAfterSeconds** * *description: When to suspend the workspace, according to the suspend type chosen* * *type: number* * **suspendType** * *description: The auto suspend mode for the workspace can have the values \`IDLE\`, \`SCHEDULED\`, or \`DISABLED\` (to create the workspace with no auto suspend settings).* * **enum** * *0: IDLE* * *1: SCHEDULED* * *2: DISABLED* * *example: IDLE* * *type: string* * *type: object* * **cacheConfig** * *description: Specifies the multiplier for the persistent cache associated with the workspace. If specified, it enables the cache configuration multiplier. It can have one of the following values: 1, 2, or 4. * * *type: number* * **enableKai** * *description: Whether to create a SingleStore Kai-enabled workspace* * *type: boolean* * **name** * *description: Name of the workspace* * *example: demo-workspace* * *type: string* * **scaleFactor** * *description: Specifies the scale factor for scaling the workspace base size. When specified, the compute resources are scaled in proportion to the specified scale factor, while the disk remains unaffected. Consequently, the credit consumption is also increased proportionately. It can have one of the following values: 1, 2, or 4. For more information, refer to \[Workspace Scaling]\(https\://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/workspace-scaling/). * * *type: number* * **size** * *description: Size of the workspace (in workspace size notation), such as "S-1". The default value is "S-00". * * *example: S-2* * *type: string* * **workspaceGroupID** * *description: ID of the workspace group in which the workspace is created* * *example: c74bb6a6-0000-1000-9000-1d874fa277b0* * *format: uuid* * *type: string* * **required** * *0: name* * *1: workspaceGroupID* * *type: object* * **WorkspaceGroup** * *description: Represents information related to a workspace group* * **properties** * **allowAllTraffic** * *description: Whether or not all traffic is allowed to the workspace group* * *type: boolean* * **createdAt** * *description: The timestamp of when the workspace was created* * *type: string* * **deploymentType** * *description: Deployment type of the workspace group* * **enum** * *0: PRODUCTION* * *1: NON-PRODUCTION* * *type: string* * **expiresAt** * *description: The timestamp of when the workspace group will expire. At expiration, the workspace group is terminated and all the data is lost.* * *type: string* * **firewallRanges** * *description: The list of allowed inbound IP addresses. An empty list indicates that no inbound requests are allowed.* * **items** * *type: string* * *type: array* * **highAvailabilityTwoZones** * *description: Whether deployment across two Availability Zones is enabled.* * *type: boolean* * **name** * *description: Name of the workspace group* * *type: string* * **optInPreviewFeature** * *description: Whether 'Opt-in to Preview Features & Updates' is enabled* * *type: boolean* * **outboundAllowList** * *description: The account ID which must be allowed for outbound connections. This is only applicable to AWS provider.* * *type: string* * **projectID** * *description: ID of the project to which the workspace group is assigned.* * *format: uuid* * *type: string* * **projectName** * *description: Name of the project to which the workspace group is assigned.* * *type: string* * **provider** * *$ref: #/components/schemas/CloudProvider* * **regionID** * *description: ID of the region* * *format: uuid* * *type: string* * **regionName** * *description: The region code name* * *type: string* * **smartDRStatus** * *description: The status of Smart Disaster Recovery (SmartDR) for the workspace group. For more information, refer to \[the documentation]\(https\://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr/).* * **enum** * *0: ACTIVE* * *1: STANDBY* * *type: string* * **state** * *description: State of the workspace group* * **enum** * *0: ACTIVE* * *1: PENDING* * *2: FAILED* * *3: TERMINATED* * *type: string* * **terminatedAt** * *description: (If included in the output) The timestamp of when the workspace group was terminated* * *type: string* * **updateWindow** * *$ref: #/components/schemas/UpdateWindow* * **workspaceGroupID** * *description: ID of the workspace group* * *format: uuid* * *type: string* * **required** * *0: name* * *1: state* * *2: workspaceGroupID* * *3: createdAt* * *4: regionID* * *5: provider* * *6: regionName* * *type: object* * **WorkspaceGroupCreate** * *description: Represents the information specified while creating a workspace group* * **properties** * **adminPassword** * *description: The admin password for the workspace group. The password must contain: \* At least 14 characters \* At least one uppercase character \* At least one lowercase character \* At least one numeric character \* At least one special character \* At most two consecutive sequential characters \* At most three consecutive identical characters If a password is not specified while creating a workspace group or if an invalid password is provided, a valid password is generated and returned in the response object. * * *example: * * *type: string* * **allowAllTraffic** * *description: If enabled, allows all traffic to the workspace group.* * *type: boolean* * **deploymentType** * *description: The deployment type that will be applied to all the workspaces within the workspace group. The default value is \`PRODUCTION\`* * **enum** * *0: PRODUCTION* * *1: NON-PRODUCTION* * *example: PRODUCTION* * *type: string* * **expiresAt** * *description: The timestamp of when the workspace group will expire. If the expiration time is not specified, the workspace group will have no expiration time. At expiration, the workspace group is terminated and all the data is lost. Expiration time can be specified as a timestamp or duration. For example, \* "2023-09-02T15:04:05Z07:00" \* "2023-09-02T15:04:05-0700" \* "2023-09-02T15:04:05" \* "2023-09-02" \* "3h30m" * * *example: * * *type: string* * **firewallRanges** * *description: A list of allowed CIDR ranges. An empty list indicates that no inbound requests are allowed. Required * * **example** * *0: 192.168.0.1/32* * *1: 192.168.0.81/12* * **items** * *type: string* * *type: array* * **highAvailabilityTwoZones** * *description: Enables deployment across two Availability Zones.* * *type: boolean* * **name** * *description: Name of the workspace group* * *example: demo-workspace-group* * *type: string* * **optInPreviewFeature** * *description: If enabled, the deployment gets the latest features and updates immediately. Suitable only for \`NON-PRODUCTION\` deployments and cannot be changed after creation.* * *type: boolean* * **projectID** * *description: Assigns the workspace group to a project, which specifies the edition of the workspace group.* * *example: 2d7bd375-0000-1000-9000-8a3d5b4922c0* * *format: uuid* * *type: string* * **provider** * *$ref: #/components/schemas/CloudProvider* * **regionID** * *description: ID of the region where the new workspace group is created* * *example: 7e7ffd27-0000-1000-9000-e72828a81ac7* * *format: uuid* * *type: string* * **regionName** * *description: The region code name* * *example: us-west1* * *type: string* * **updateWindow** * *$ref: #/components/schemas/UpdateWindow* * **required** * *0: name* * *1: firewallRanges* * *type: object* * **WorkspaceGroupUpdate** * *description: Represents the information specified while updating a workspace group* * **properties** * **adminPassword** * *description: The admin password for the workspace group. The password must contain: \* At least 14 characters \* At least one uppercase character \* At least one lowercase character \* At least one numeric character \* At least one special character \* At most two consecutive sequential characters \* At most three consecutive identical characters * * *example: * * *type: string* * **allowAllTraffic** * *description: Whether to allow all traffic to the workspace group* * *type: boolean* * **deploymentType** * *description: The deployment type that will be applied to all the workspaces within the workspace group* * **enum** * *0: PRODUCTION* * *1: NON-PRODUCTION* * *example: PRODUCTION* * *type: string* * **expiresAt** * *description: The timestamp of when the workspace group will expire. If the expiration time is not specified, the workspace group will have no expiration time. At expiration, the workspace group is terminated and all the data is lost. Expiration time can be specified as a timestamp or duration. For example, \* "2023-09-02T15:04:05Z07:00" \* "2023-09-02T15:04:05-0700" \* "2023-09-02T15:04:05" \* "2023-09-02" \* "3h30m" * * *example: * * *type: string* * **firewallRanges** * *description: A list of allowed CIDR ranges. An empty list indicates that no inbound requests are allowed.* * **example** * *0: 192.168.0.1/32* * *1: 192.168.0.81/12* * **items** * *type: string* * *type: array* * **name** * *description: Name of the workspace group* * *example: new-workspace-group-name* * *type: string* * **updateWindow** * *$ref: #/components/schemas/UpdateWindow* * *type: object* * **WorkspaceResume** * *description: Represents additional information specified when resuming a workspace* * **properties** * **disableAutoSuspend** * *description: Whether to disable auto suspend or keep the existing auto suspend settings. By default, \`disableAutoSuspend\` is set to \`false\`, and the existing auto suspend settings are preserved.* * *type: boolean* * *type: object* * **WorkspaceUpdate** * *description: Represents the information specified while updating a workspace* * **properties** * **autoScale** * *$ref: #/components/schemas/AutoScale* * **autoSuspend** * *description: Specifies the auto suspend mode for the workspace. It can have one of the following values: \`IDLE\`, \`SCHEDULED\`, or \`DISABLED\`.* * **properties** * **suspendAfterSeconds** * *description: The time interval (in seconds) after which the workspace is suspended, depending on the auto suspend mode.* * *type: number* * **suspendType** * *description: The type of auto suspend mode. Set to \`DISABLED\` to disable auto suspend.* * **enum** * *0: IDLE* * *1: SCHEDULED* * *2: DISABLED* * *example: IDLE* * *type: string* * *type: object* * **cacheConfig** * *description: Specifies the multiplier for the persistent cache associated with the workspace. If specified, it enables the cache configuration multiplier. It can have one of the following values: 1, 2, or 4. * * *type: number* * **deploymentType** * *description: The deployment type that will be applied to all the workspaces within the group* * **enum** * *0: PRODUCTION* * *1: NON-PRODUCTION* * *example: PRODUCTION* * *type: string* * **enableKai** * *description: Whether to enable SingleStore Kai in this workspace* * **enum** * *type: boolean* * **scaleFactor** * *description: Specifies the scale factor for scaling the workspace base size. When specified, the compute resources are scaled in proportion to the specified scale factor, while the disk remains unaffected. Consequently, the credit consumption is also increased proportionately. It can have one of the following values: 1, 2, or 4. For more information, refer to \[Workspace Scaling]\(https\://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/workspace-scaling/). * * *type: number* * **size** * *description: Size of the workspace (in workspace size notation), such as "S-1". The default value is "S-00". * * *example: S-2* * *type: string* * *type: object* * **securitySchemes** * **ApiKeyAuth** * *bearerFormat: API key* * *description: For information on authenticating your API requests, refer to \[\`Management\` API Overview]\(https\://docs.singlestore.com/managed-service/en/reference/management-api.html). * * *scheme: bearer* * *type: http* * **info** * **contact** * *name: SingleStore Support* * *url: https\://www\.singlestore.com/support/* * *description: The \`Management\` API can be used to create and manage workspaces, workspace groups, private connections, etc. SingleStore recommends reading the \[\`Management\` API Overview]\(https\://docs.singlestore.com/managed-service/en/reference/management-api.html) before getting started with the API reference. All the URLs referenced in this API documentation use the \`https\://api.singlestore.com\` service endpoint as their base. * * *title: SingleStore Management API* * *version: 1.2.166* * *openapi: 3.0.0* * **paths** * **/v1/auditLogs** * **get** * *description: Returns a list of all of the audit log entries for the user organization or audit logs for user redaction* * **parameters** * **0** * *description: The audit log type* * *in: query* * *name: type* * **schema** * *type: string* * **1** * *description: The audit log source* * *in: query* * *name: source* * **schema** * *type: string* * **2** * *description: The start date (inclusive) for the query in RFC3339 format (2019-10-12T07:20:50.52Z)* * *in: query* * *name: startDate* * **schema** * *type: string* * **3** * *description: The end date (inclusive) for the query in RFC3339 format (2019-10-12T07:20:50.52Z)* * *in: query* * *name: endDate* * **schema** * *type: string* * **4** * *description: The maximum number of rows to return* * *in: query* * *name: limit* * **schema** * *type: number* * **5** * *description: The value of the nextToken returned from a previous query. This will guarantee the next results are new since the last query* * *in: query* * *name: nextToken* * **schema** * *type: string* * **6** * *description: A first name to match against* * *in: query* * *name: firstName* * **schema** * *type: string* * **7** * *description: A last name to match against* * *in: query* * *name: lastName* * **schema** * *type: string* * **8** * *description: An email address to match against* * *in: query* * *name: email* * **schema** * *format: email* * *type: string* * **9** * *description: A user ID to match against* * *in: query* * *name: userID* * **schema** * *format: uuid* * *type: string* * **10** * *description: Filter by the API key ID used to perform the action* * *in: query* * *name: apiKeyID* * **schema** * *format: uuid* * *type: string* * **11** * *description: A workspace group ID to filter by* * *in: query* * *name: workspaceGroupID* * **schema** * *format: uuid* * *type: string* * **12** * *description: A project ID to filter by* * *in: query* * *name: projectID* * **schema** * *format: uuid* * *type: string* * **13** * *description: A team ID to filter by* * *in: query* * *name: teamID* * **schema** * *format: uuid* * *type: string* * **14** * *description: A workspace ID to filter by* * *in: query* * *name: workspaceID* * **schema** * *format: uuid* * *type: string* * **15** * *description: If true, sort audit logs by createdAt in descending order* * *in: query* * *name: sortByDescending* * **schema** * *type: boolean* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **auditLogs** * **items** * *$ref: #/components/schemas/AuditLog* * *type: array* * **nextToken** * *description: The nextToken value can be used in a subsequent query to guarantee any log entries are new since this query* * *type: string* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all of the audit logs for the user organization or audit logs for user redaction* * **tags** * *0: AuditLogs* * **/v1/billing/usage** * *description: The \`usage\` endpoint under billing can report compute and storage usage.* * **get** * *description: Returns the compute and storage usage of an organization in a specified timeframe. The results can be grouped by hour, day, and month using the \`aggregateType\` parameter. The usage entries in the result set also include information on the resource ID, type, name, and usage. Note that the usage periods are rounded to the nearest hour/day/month that includes the requested range. For example, if the start time is "2023-07-30T18:30:00Z" and the end time is "2023-07-31T18:30:00Z", the hourly usage entries would be returned for the period from "2023-07-30T18:00:00Z" to "2023-07-31T19:00:00Z". Same goes for daily and monthly usage entries. * * **parameters** * **0** * *description: The metric type. It can have the following values: \* \`ComputeCredit\`: Returns the compute usage. \* \`StorageAvgByte\`: Returns the average bytes of storage usage. By default, usage for both metrics is returned. * * *in: query* * *name: metric* * **schema** * **enum** * *0: ComputeCredit* * *1: StorageAvgByte* * *type: string* * **1** * *description: The start time for the usage interval in UTC ISO8601 format. For example, "2023-07-30T18:30:00Z". * * *in: query* * *name: startTime* * **schema** * *type: string* * **2** * *description: The end time for the usage interval in UTC ISO8601 format. For example, "2023-07-30T18:30:00Z". * * *in: query* * *name: endTime* * **schema** * *type: string* * **3** * *description: The interval used to aggregate the usage. It can have the following values: \`hour\`, \`day\`, and \`month\`. By default, the results are grouped by hour. * * *in: query* * *name: aggregateBy* * **schema** * **enum** * *0: hour* * *1: day* * *2: month* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **billingUsage** * **items** * *$ref: #/components/schemas/BillingUsage* * *type: array* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Returns the compute and storage usage of an organization* * **tags** * *0: Billing* * **/v1/billing/usage/simulate** * *description: Allows a user to simulate billing usage given different resource configurations over time.* * **post** * *description: Simulates billing usage for resources based on provided configurations. This endpoint allows you to estimate compute usage costs by specifying resource configurations and time ranges. You must provide a time range (startAt and endBefore). An array of resource configurations may optionally be provided to simulate state changes of resources over time. Each resource configuration defines the characteristics of a resource during a specific time period. If no configurations are provided, a simulation will be run using the existing resources in the project for the time range based on their current configurations. * * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SimulateUsageRequest* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SimulateUsageResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Returns an array of simulated compute usage for the given resource configurations.* * **tags** * *0: Billing* * **/v1/cloudPrincipals** * *description: The \`cloudPrincipals\` endpoint is used to manage cloud principals.* * **post** * *description: Creates a cloud principal in the current organization. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudPrincipalCreate* * *description: Cloud principal creation parameters.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudPrincipal* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **409** * *description: Cloud principal already exists in this organization* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a cloud principal in the current organization.* * **tags** * *0: Cloud Principals* * **/v1/cloudPrincipals/{cloudPrincipalID}** * **delete** * *description: Deletes the specified cloud principal from the current organization. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/cloudPrincipalID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudPrincipalIDResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A cloud principal with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a cloud principal.* * **tags** * *0: Cloud Principals* * **patch** * *description: Updates the name and/or description for the specified cloud principal. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/cloudPrincipalID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudPrincipalUpdate* * *description: Cloud principal update parameters.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudPrincipal* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A cloud principal with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates a cloud principal.* * **tags** * *0: Cloud Principals* * **/v1/cloudfunctions** * **get** * *description: Lists all the cloud functions in the current organization.* * **parameters** * **0** * *$ref: #/components/parameters/limit* * **1** * *$ref: #/components/parameters/offsetID* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **cloudFunctions** * **items** * *$ref: #/components/schemas/NotebookCloudFunction* * *type: array* * **metadata** * *$ref: #/components/schemas/NotebookCloudFunctionPaginationMetadata* * *type: object* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Lists the cloud functions* * **tags** * *0: CloudFunctions* * **post** * *description: Creates a new cloud function. Upon successful completion of the request, the ID of the cloud function is returned in the API response.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudFunctionCreate* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **cloudFunctionID** * *example: 7b68b78d-a000-4000-9000-0864fd331524* * *type: string* * *type: object* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new cloud function* * **tags** * *0: CloudFunctions* * **/v1/cloudfunctions/{cloudfunctionID}** * **delete** * *description: Deletes the cloud function with the specified ID. You must specify the ID of the cloud function in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/cloudfunctionID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/NotebookCloudFunction* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **404** * *description: Not found* * **500** * *description: Internal server error* * *summary: Deletes a cloud function* * **tags** * *0: CloudFunctions* * **get** * *description: Returns information on the cloud function with the specified ID. You must specify the cloud function ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/cloudfunctionID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/NotebookCloudFunction* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **404** * *description: Not found* * **500** * *description: Internal server error* * *summary: Gets information about a cloud function* * **tags** * *0: CloudFunctions* * **patch** * *description: Updates the cloud function with the specified ID. You must specify the cloud function ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/cloudfunctionID* * **1** * *$ref: #/components/parameters/lastKnownUpdatedAt* * **2** * *$ref: #/components/parameters/updateNotebookSnapshot* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudFunctionUpdate* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/NotebookCloudFunction* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **404** * *description: Not found* * **409** * *description: Conflict error* * **500** * *description: Internal server error* * *summary: Updates a cloud function* * **tags** * *0: CloudFunctions* * **/v1/cloudfunctions/{cloudfunctionID}/token** * **get** * *description: Returns a token that can be used to interact with cloud function applications.* * **parameters** * **0** * *$ref: #/components/parameters/cloudfunctionID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/AppToken* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Gets a cloud function token* * **tags** * *0: CloudFunctions* * **/v1/files/fs/{location}** * **get** * *description: Returns a list of all files from personal, shared or models space. You must specify the \`location\`. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: Location of the files* * *in: path* * *name: location* * **schema** * *$ref: #/components/schemas/FileLocationSchema* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/FileObjectMetadata* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all files from personal, shared or models space.* * **tags** * *0: Files* * **/v1/files/fs/{location}/{path}** * **delete** * *description: Deletes the file at the specified path in personal, shared or models space. You must specify the \`location\` and file \`path\` in the API call. * * **parameters** * **0** * *description: Location of the files* * *in: path* * *name: location* * **schema** * *$ref: #/components/schemas/FileLocationSchema* * **1** * *description: Path in personal, shared or models space to a file to delete* * *example: filename.ipynb* * *in: path* * *name: path* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *description: filename.ipynb* * *type: string* * **path** * *description: /* * *type: string* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a file from personal, shared or models space.* * **tags** * *0: Files* * **get** * *description: Returns the information about a file or redirects to the file download URL. You must specify the \`location\` and the file \`path\` in the API call. * * **parameters** * **0** * *description: Location of the files* * *in: path* * *name: location* * **schema** * *$ref: #/components/schemas/FileLocationSchema* * **1** * *description: Path in personal, shared or models space to a file* * *example: filename.ipynb* * *in: path* * *name: path* * **schema** * *type: string* * **2** * *description: If enabled, the API request returns only metadata for the specified file instead of downloading it.* * *in: query* * *name: metadata* * **schema** * *type: boolean* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/FileObjectMetadata* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a file or downloads a file from personal, shared or models space.* * **tags** * *0: Files* * **patch** * *description: Modifies the path of the existing file in personal, shared or models space to the new path specified in the request body. You must specify the \`location\` and file \`path\` in the API call. * * **parameters** * **0** * *description: Location of the files* * *in: path* * *name: location* * **schema** * *$ref: #/components/schemas/FileLocationSchema* * **1** * *description: Path in personal, shared or models space to modify* * *example: filename.ipynb* * *in: path* * *name: path* * **schema** * *type: string* * **requestBody** * **content** * **application/json** * **example** * *newPath: new\_filename.ipynb* * **schema** * **properties** * **newPath** * *description: New path of the file* * *type: string* * *type: object* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *description: filename.ipynb* * *type: string* * **path** * *description: /* * *type: string* * *type: object* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Modifies a file in personal, shared or models space.* * **tags** * *0: Files* * **put** * *description: This endpoint allows you to create or upload a new file at a specified path within personal, shared or models space. The behavior depends on the contents of the request body and the format of the path: \- \*\*File Upload:\*\* If the request body contains a file, the file is uploaded to the specified path within personal, shared or models space. You must specify the \`location\` and the file \`path\` in the API call. * * **parameters** * **0** * *description: Location of the files* * *in: path* * *name: location* * **schema** * *$ref: #/components/schemas/FileLocationSchema* * **1** * *description: Path in personal, shared or models space* * *example: filename.ipynb* * *in: path* * *name: path* * **schema** * *type: string* * **requestBody** * **content** * **multipart/form-data** * **schema** * **properties** * **file** * *description: File to upload* * *format: binary* * *type: string* * **required** * *0: file* * *type: object* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *example: filename.ipynb* * *type: string* * **path** * *example: /* * *type: string* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new file or uploads a file to personal, shared or models space.* * **tags** * *0: Files* * **/v1/flow** * *description: Manages SingleStore Flow instances* * **get** * *description: Returns a list of all of the SingleStore Flow instances accessible to the user. Use the \`includeTerminated\` parameter to get a list of terminated Flow instances. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: To include any terminated Flow instances, set to \`true\`* * *in: query* * *name: includeTerminated* * **schema** * *type: boolean* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *createdAt: 2026-01-18T09:56:56Z* * *flowID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *name: flow-instance-1* * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785c* * **schema** * **items** * *$ref: #/components/schemas/Flow* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the Flow instances the user can access* * **tags** * *0: Flow* * **post** * *description: Creates a new SingleStore Flow instance for the current user. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/FlowCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *flowID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **flowID** * *format: uuid* * *type: string* * **required** * *0: flowID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new Flow instance* * **tags** * *0: Flow* * **/v1/flow/{flowID}** * **delete** * *description: Terminates a SingleStore Flow instance with the specified ID. You must specify the Flow instance ID in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/flowID* * **responses** * **200** * **content** * **application/json** * **example** * *flowID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **flowID** * *format: uuid* * *type: string* * **required** * *0: flowID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A Flow instance with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Terminates a Flow instance* * **tags** * *0: Flow* * **get** * *description: Returns information about the SingleStore Flow instance with the specified ID, in JSON format. You must specify the Flow instance ID in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/flowID* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2026-01-18T09:56:56Z* * *flowID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *name: flow-instance-1* * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785c* * **schema** * *$ref: #/components/schemas/Flow* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A Flow instance with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a Flow instance* * **tags** * *0: Flow* * **/v1/invitations** * **get** * *description: Returns a list of open invitations for the current organization. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: Show only invitations with emails matching this value or regular expression.* * *in: query* * *name: email* * **schema** * *type: string* * **1** * *description: Show only invitations with state matching this value.* * *in: query* * *name: state* * **schema** * *type: string* * **2** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/UserInvitation* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists open invitations for the current organization.* * **tags** * *0: Invitations* * **post** * *description: Creates and sends an invitation to join the organization to an email address. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/UserInvitationCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/UserInvitation* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **409** * *description: User is already a member* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Invites a user to the current organization* * **tags** * *0: Invitations* * **/v1/invitations/{invitationID}** * **delete** * *description: Revokes the specified invitation. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/invitationID* * **responses** * **200** * **content** * **application/json** * **example** * *invitationID: 03891d70-ea86-403e-8463-113917fd0466* * **schema** * **properties** * **invitationID** * *format: uuid* * *type: string* * **required** * *0: invitationID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A invitation with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Revokes an invitation* * **tags** * *0: Invitations* * **get** * *description: Returns information for the specified invitation. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/invitationID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2023-09-18T09:56:56Z* * *email: alice\@abc.com* * *invitationID: c74bb6a6-0000-1000-9000-1d874fa277b1* * *message: Welcome to SingleStore\@abc.com* * *state: Pending* * **teamIDs** * **schema** * *$ref: #/components/schemas/UserInvitation* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: An invitation with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about an invitation.* * **tags** * *0: Invitations* * **/v1/jobs** * **post** * *description: This creates a job. A job sets a scheduled execution of a notebook and can cause notebooks to be run on a schedule or one-time only This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/JobCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Job* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new job* * **tags** * *0: Jobs* * **/v1/jobs/{jobID}** * **delete** * *description: This deletes a job and all execution snapshots associated with a job. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/jobID* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Deletes a job* * **tags** * *0: Jobs* * **get** * *description: Returns the metadata on target, schedule and notebook for a job specified by a jobid. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/jobID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Job* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Gets information about the specified job* * **tags** * *0: Jobs* * **/v1/jobs/{jobID}/executions** * **get** * *description: Returns the information of the executions of a specified job. Specify which executions you want with the parameters. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/jobID* * **1** * *description: Start execution number.* * *in: query* * *name: start* * **schema** * *type: number* * **2** * *description: End execution number.* * *in: query* * *name: end* * **schema** * *type: number* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ExecutionsResult* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Get executions for a job* * **tags** * *0: Jobs* * **/v1/jobs/{jobID}/parameters** * **get** * *description: Returns the array of the parameters of a specified job. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/jobID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/JobParameter* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Get parameters for a job* * **tags** * *0: Jobs* * **/v1/jobs/runtimes** * **get** * *description: Returns the list of available runtimes for jobs. This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/RuntimesResult* * *type: array* * *description: OK* * **400** * *description: Bad request* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Get the list of available job runtimes* * **tags** * *0: Jobs* * **/v1/organizations/{organizationID}/accessControls** * **get** * *description: Return all the roles on the resource type, which include all grants of the given resource* * **parameters** * **0** * *$ref: #/components/parameters/organizationID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/ResourceRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get RBAC access information about the organization* * **tags** * *0: Organizations* * **patch** * *description: Grant or revoke grants on roles of the resource* * **parameters** * **0** * *$ref: #/components/parameters/organizationID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ControlAccessAction* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Update the RBAC grants on the organization* * **tags** * *0: Organizations* * **/v1/organizations/current** * **get** * *description: Returns information about the current authorized user's organization.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Organization* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about the current organization* * **tags** * *0: Organizations* * **/v1/privateConnections** * *description: The \`privateConnections\` endpoint can be used to create and manage private connections to SingleStoreDB Cloud workspaces.* * **post** * *description: Creates a new private connection. Upon successful completion of the request, a private connection is scheduled for creation. To query the private connection status, use the endpoints for the workspace group and workspace (if provided). * * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/PrivateConnectionCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *privateConnectionID: b07fff9f-0000-1000-9000-b1974e06b736* * **schema** * **properties** * **privateConnectionID** * *format: uuid* * *type: string* * **required** * *0: privateConnectionID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new private connection* * **tags** * *0: Private Connections* * **/v1/privateConnections/{connectionID}** * **delete** * *description: Deletes a private connection for the specified connection ID. Upon successful completion, a private connection is scheduled for deletion.* * **parameters** * **0** * *$ref: #/components/parameters/connectionID* * **responses** * **200** * **content** * **application/json** * **example** * *privateConnectionID: b07fff9f-0000-1000-9000-b1974e06b736* * **schema** * **properties** * **privateConnectionID** * *format: uuid* * *type: string* * **required** * *0: privateConnectionID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A private connection with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a private connection* * **tags** * *0: Private Connections* * **get** * *description: Returns private connection information for the specified connection ID, in JSON format. You must specify the connection ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/connectionID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/PrivateConnection* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A private connection with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a private connection* * **tags** * *0: Private Connections* * **patch** * *description: Updates a private connection. You must specify the connection ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/connectionID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/PrivateConnectionUpdate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *privateConnectionID: b07fff9f-0000-1000-9000-b1974e06b736* * **schema** * **properties** * **privateConnectionID** * *format: uuid* * *type: string* * **required** * *0: privateConnectionID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A private connection with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates a private connection* * **tags** * *0: Private Connections* * **/v1/projects** * *description: The projects endpoint can be used to manage projects.* * **get** * *description: Returns a list of all of the projects accessible to the user. This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *createdAt: 2025-10-24T09:56:56Z* * *edition: STANDARD* * *name: demo-project* * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * **schema** * **items** * *$ref: #/components/schemas/Project* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the projects the user can access* * **tags** * *0: Projects* * **post** * *description: Creates a new project for the current user. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ProjectCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * **schema** * *$ref: #/components/schemas/ProjectIDResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new project* * **tags** * *0: Projects* * **/v1/projects/{projectID}** * **delete** * *description: Deletes a project with the specified project ID. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/projectID* * **responses** * **200** * **content** * **application/json** * **example** * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * **schema** * *$ref: #/components/schemas/ProjectIDResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A project with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a project* * **tags** * *0: Projects* * **get** * *description: Returns information for the specified project ID in JSON format. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/projectID* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2025-10-24T09:56:56Z* * *edition: STANDARD* * *name: demo-project* * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * **schema** * *$ref: #/components/schemas/Project* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A project with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a project* * **tags** * *0: Projects* * **patch** * *description: Updates the specified project. You can only update its name. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/projectID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ProjectUpdate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * **schema** * *$ref: #/components/schemas/ProjectIDResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A project with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates a project* * **tags** * *0: Projects* * **/v1/regions** * **get** * *description: \*\*Note\*\*: This endpoint is deprecated. Use \`/v2/regions\` instead. Returns a list of valid regions for the user that support workspaces, including the region ID and provider for each region.* * **parameters** * **0** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *provider: GCP* * *region: US West 1 (Oregon)* * *regionID: 1c1de314-0000-1000-9000-5047ff90842e* * **1** * *provider: AWS* * *region: Europe Central 1 (Frankfurt)* * *regionID: 7e7ffd27-0000-1000-9000-e72828a81ac7* * **2** * *provider: AWS* * *region: Asia Pacific Southeast 2 (Sydney)* * *regionID: c74bb6a6-0000-1000-9000-1d874fa277b7* * **schema** * **items** * *$ref: #/components/schemas/Region* * *type: array* * *description: OK* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all of the regions for the user that support workspaces* * **tags** * *0: Regions* * **/v1/regions/sharedtier** * **get** * *description: Returns a list of valid regions that support Shared Edition deployments (starter workspaces) for the user, including the region code and provider for each region.* * **parameters** * **0** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *provider: GCP* * *region: US West 1 (Oregon)* * *regionName: us-west1* * **1** * *provider: AWS* * *region: Europe Central 1 (Frankfurt)* * *regionName: eu-central-1* * **2** * *provider: AWS* * *region: Asia Pacific Southeast 2 (Sydney)* * *regionName: ap-southeast-2* * **schema** * **items** * *$ref: #/components/schemas/RegionV2* * *type: array* * *description: OK* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all the regions that support Shared Edition deployments (starter workspaces) for the user* * **tags** * *0: Regions* * **/v1/roles/{resourceType}** * **get** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/RoleDefinition* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all roles for the given resource type, including custom roles* * **tags** * *0: Roles* * **post** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleCreate* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinition* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new role for the given resource type* * **tags** * *0: Roles* * **/v1/roles/{resourceType}/{role}** * **delete** * *description: Deletes the specified role. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **1** * *$ref: #/components/parameters/role* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Resource not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a role.* * **tags** * *0: Roles* * **get** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **1** * *$ref: #/components/parameters/role* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinition* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets a given role from a given resource type* * **tags** * *0: Roles* * **put** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **1** * *$ref: #/components/parameters/role* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleUpdate* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinition* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Resource not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Update the name, permission, resource type, and inherits for the custom role.* * **tags** * *0: Roles* * **/v1/secrets** * *description: The \`secrets\` endpoint can be used to create and manage secrets.* * **get** * *description: Returns a list of all of the secrets accessible to the user. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: Name of the secret.* * *in: query* * *name: name* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/Secret* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the secrets the user can access.* * **tags** * *0: Secrets* * **post** * *description: Creates a new secret, which can be used to store sensitive information such as passwords, API keys, and other credentials. You must specify a name and value for the secret in the API request body. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SecretCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Secret* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **409** * *description: Secret with specified name already exists* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new secret* * **tags** * *0: Secrets* * **/v1/secrets/{secretID}** * **delete** * *description: Deletes a secret with the specified ID. You must specify the secret ID in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/secretID* * **responses** * **200** * **content** * **application/json** * **example** * *secretID: 03891d70-0000-1000-9000-113917fd0355* * **schema** * **properties** * **secretID** * *format: uuid* * *type: string* * **required** * *0: secretID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A secret with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a secret* * **tags** * *0: Secrets* * **get** * *description: Returns information for the specified secret ID, in JSON format. You must specify the secret ID in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/secretID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Secret* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A secret with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a secret* * **tags** * *0: Secrets* * **patch** * *description: Updates the name or value for the specified secret ID. You must specify the secret ID in the API call and at least one parameter (name or value) in the request body. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/secretID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SecretUpdate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Secret* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A secret with the specified ID was not found* * **409** * *description: Secret with specified name already exists* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates the name or value of a secret* * **tags** * *0: Secrets* * **/v1/secrets/{secretID}/accessControls** * **get** * *description: Return all the roles on the resource type, which include all grants of the given resource* * **parameters** * **0** * *$ref: #/components/parameters/secretID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/ResourceRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get RBAC access information about the secret* * **tags** * *0: Secrets* * **patch** * *description: Grant or revoke grants on roles of the resource* * **parameters** * **0** * *$ref: #/components/parameters/secretID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ControlAccessAction* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Update the RBAC grants on the secret* * **tags** * *0: Secrets* * **/v1/serviceAccounts** * *description: The \`serviceAccounts\` endpoint is used to manage service accounts.* * **post** * *description: Creates a service account in the current organization. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ServiceAccountCreate* * *description: Service account creation parameters.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ServiceAccount* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **409** * *description: Service account with the specified name already exists* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a service account in the current organization.* * **tags** * *0: Service Accounts* * **/v1/serviceAccounts/{serviceAccountID}** * **delete** * *description: Deletes the specified service account from the current organization. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/serviceAccountID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ServiceAccountIDResponse* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A service account with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a service account.* * **tags** * *0: Service Accounts* * **patch** * *description: Updates the name and/or description for the specified service account. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/serviceAccountID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ServiceAccountUpdate* * *description: Service account update parameters.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ServiceAccount* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A service account with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates a service account.* * **tags** * *0: Service Accounts* * **/v1/sharedtier/virtualWorkspaces** * **get** * *description: Returns a list of all Shared Edition deployments (starter workspaces) for the current user. This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/SharedTierVirtualWorkspace* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: List all Shared Edition deployments (starter workspaces)* * **tags** * *0: SharedTier* * **post** * *description: Creates a new Shared Edition deployment (starter workspace) for the current user. You must specify a name, region ID, and firewall range(s) for the starter workspace in the API request body. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierCreateVirtualWorkspace* * **responses** * **200** * **content** * **application/json** * **example** * *virtualWorkspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **virtualWorkspaceID** * *format: uuid* * *type: string* * **required** * *0: virtualWorkspaceID* * *type: object* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **/v1/sharedtier/virtualWorkspaces/{virtualWorkspaceID}** * **delete** * *description: Deletes a Shared Edition deployment (starter workspace) for the specified ID. You must specify the ID of the starter workspace in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualWorkspaceID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Deletes a Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **get** * *description: Returns information about a Shared Edition deployment (starter workspace) with the specified ID. You must specify the ID of the starter workspace in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualWorkspaceID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierVirtualWorkspace* * *description: OK* * **401** * *description: Unauthorized* * **404** * *description: Not found* * **500** * *description: Internal server error* * *summary: Returns information about a Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **/v1/sharedtier/virtualWorkspaces/{virtualWorkspaceID}/users** * **post** * *description: Creates a new user in a Shared Edition deployment (starter workspace) with the specified ID. You must specify the ID of the starter workspace in the API call. You can specify a password for the user in the request body. If a password is not specified, a password is automatically generated and returned in the API response. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualWorkspaceID* * **schema** * *format: uuid* * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierCreateUser* * **responses** * **200** * **content** * **application/json** * **schema** * **example** * *password: Auto-generated-password1!* * *userID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **properties** * **password** * *type: string* * **userID** * *format: uuid* * *type: string* * **required** * *0: userID* * *1: password* * *type: object* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new user in a Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **/v1/sharedtier/virtualWorkspaces/{virtualWorkspaceID}/users/{userID}** * **delete** * *description: Deletes a user with the specified user ID from a Shared Edition deployment (starter workspace). You must specify the ID of the starter workspace and the user in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualWorkspaceID* * **schema** * *format: uuid* * *type: string* * **1** * *in: path* * *name: userID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Deletes a user from a Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **patch** * *description: Updates a user in a Shared Edition deployment (starter workspace) with the specified user ID. You must specify the ID of the starter workspace and the user in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualWorkspaceID* * **schema** * *format: uuid* * *type: string* * **1** * *in: path* * *name: userID* * **schema** * *format: uuid* * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierUpdateUser* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Updates a user in a Shared Edition deployment (starter workspace)* * **tags** * *0: SharedTier* * **/v1/stage/{deploymentID}/fs** * **get** * *description: Returns a list of all files and folders located in the Stage's root folder of the specified deployment (either a workspace group or starter workspace). You must specify the \`deploymentID\`. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: ID of the deployment* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *in: path* * *name: deploymentID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/FileObjectMetadata* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists files and folders in the root folder.* * **tags** * *0: Stage* * **/v1/stage/{deploymentID}/fs/{path}** * **delete** * *description: Deletes the file or folder at the specified path in a Stage. This endpoint supports both URL-encoded and regular URL paths. Here are some examples of valid requests: \- Delete a folder: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_folder/ \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_folder%2F \- Delete a file: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_file.txt \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_file.txt You must specify the \`deploymentID\` and the folder/file path in the API call. \## Note: To delete a folder, ensure that the path includes a trailing slash. * * **parameters** * **0** * *description: ID of the deployment* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *in: path* * *name: deploymentID* * **schema** * *format: uuid* * *type: string* * **1** * *description: Path in the Stage to a file or folder to delete* * **examples** * **file** * *value: parent\_folder/sample\_folder/file.json* * **folder** * *value: parent\_folder/sample\_folder/* * *in: path* * *name: path* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *description: sample\_file.txt* * *type: string* * **path** * *description: parent\_folder/sample\_file.txt* * *type: string* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a file or folder* * **tags** * *0: Stage* * **get** * *description: If the specified path is a folder, the API returns the list of files and folders inside this folder in the Stage. If the specified path is a file, the API call is redirected to a download URL. This endpoint supports both URL-encoded and regular URL paths. Here are some examples of valid requests: \- Get a folder: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_folder/ \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_folder%2F \- Get a file: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_file.txt \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_file.txt You must specify the \`deploymentID\` and the folder/file path in the API call. \## Note: To Get a folder, ensure that the path includes a trailing slash. * * **parameters** * **0** * *description: ID of the deployment* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *in: path* * *name: deploymentID* * **schema** * *format: uuid* * *type: string* * **1** * *description: Path in the Stage to a file or folder* * **examples** * **file** * *value: parent\_folder/sample\_folder/filename.json* * **folder** * *value: parent\_folder/sample\_folder/* * *in: path* * *name: path* * **schema** * *type: string* * **2** * *description: If enabled, the API request returns only metadata for the specified file instead of downloading it. This parameter is ignored if the specified path is a folder.* * *in: query* * *name: metadata* * **schema** * *type: boolean* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/FileObjectMetadata* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a folder or downloads a file* * **tags** * *0: Stage* * **patch** * *description: Modifies the path of the existing file or folder in a Stage to the new path specified in the request body. This endpoint supports both URL-encoded and regular URL paths, so both of the following examples are valid requests: \- Modify a folder: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_folder/ \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_folder%2F \- Modify a file: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_file.txt \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_file.txt You must specify the \`deploymentID\` and the folder/file path in the API call. \## Note: To modify the path of a folder, ensure that it includes a trailing slash. * * **parameters** * **0** * *description: ID of the deployment* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *in: path* * *name: deploymentID* * **schema** * *format: uuid* * *type: string* * **1** * *description: Path in the Stage to modify* * **examples** * **file** * *value: parent\_folder/sample\_folder/file.json* * **folder** * *value: parent\_folder/sample\_folder/* * *in: path* * *name: path* * **schema** * *type: string* * **requestBody** * **content** * **application/json** * **examples** * **file** * **value** * *newPath: parent\_folder/sample\_folder/file.json* * **folder** * **value** * *newPath: parent\_folder/sample\_folder/* * **schema** * **properties** * **newPath** * *description: New path of the file or folder* * *type: string* * *type: object* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *description: sample\_file.txt* * *type: string* * **path** * *description: parent\_folder/sample\_file.txt* * *type: string* * *type: object* * *description: OK* * **307** * *description: Temporary Redirect* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Not Found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Modifies a file or folder in a Stage* * **tags** * *0: Stage* * **put** * *description: This endpoint allows you to create a new folder or upload a new file at a specified path within Stage. The behavior depends on the contents of the request body and the format of the path: \- \*\*File Upload:\*\* If the request body contains a file, the file is uploaded to the specified path within the Stage. \- \*\*Folder Creation:\*\* If the request body does not contain a file and the path ends with a trailing slash (\`/\`), a new folder is created at the specified path. \- \*\*Empty File Creation:\*\* If the request body does not contain a file and the path does not end with a trailing slash, an empty file is created at the specified path. \*\*Please ensure that the path is correctly formatted according to the desired action. Misformatted paths may lead to unexpected behavior.\*\* This endpoint supports both URL-encoded and regular URL paths. Here are some examples of valid requests: \- Create a folder: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_folder/ \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_folder%2F \- Upload a file: \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder/sample\_folder/file.json \- /v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/parent\_folder%2Fsample\_folder%2Ffile.json You must specify the \`deploymentID\` and the folder/file path in the API call. * * **parameters** * **0** * *description: ID of the deployment* * *example: 68af2f46-0000-1000-9000-3f6f5365d878* * *in: path* * *name: deploymentID* * **schema** * *format: uuid* * *type: string* * **1** * *description: Path in the Stage* * **examples** * **file** * *value: parent\_folder/sample\_folder/file.json* * **folder** * *value: parent\_folder/sample\_folder/* * *in: path* * *name: path* * **schema** * *type: string* * **requestBody** * **content** * **multipart/form-data** * **schema** * **properties** * **file** * *description: File to upload* * *format: binary* * *type: string* * *type: object* * **responses** * **200** * **content** * **application/json** * **schema** * **properties** * **name** * *example: file.json* * *type: string* * **path** * *example: /parent\_folder/sample\_folder* * *type: string* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new folder or uploads a file* * **tags** * *0: Stage* * **/v1/teams** * *description: The \`teams\` endpoint is used to manage teams.* * **get** * *description: Returns a list of teams in the current organization. * * **parameters** * **0** * *description: Show only teams with names matching this value or regular expression.* * *in: query* * *name: name* * **schema** * *type: string* * **1** * *description: Show only teams with descriptions matching this value or regular expression.* * *in: query* * *name: description* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/Team* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the teams defined for the current organization. Teams may be filtered by name or description. An empty list is returned if no teams match the filter.* * **tags** * *0: Teams* * **post** * *description: Creates a new team. You must specify a name in the API request body.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/TeamCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/Team* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **409** * *description: Team with specified name already exists* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new team* * **tags** * *0: Teams* * **/v1/teams/{teamID}** * **delete** * *description: Deletes the specified team. * * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **responses** * **200** * **content** * **application/json** * **example** * *teamID: 03891d70-ea86-403e-8463-113917fd0355* * **schema** * **properties** * **teamID** * *format: uuid* * *type: string* * **required** * *0: teamID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A team with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a team* * **tags** * *0: Teams* * **get** * *description: Returns information for the specified team. * * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2023-09-18T09:56:56Z"* * *description: All analysts in the abc.com organization* * *name: Analysts* * *teamID: c74bb6a6-0000-1000-9000-1d874fa277b0* * **schema** * *$ref: #/components/schemas/Team* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A team with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a team.* * **tags** * *0: Teams* * **patch** * *description: Updates the name, description, and/or members for the specified team. * * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/TeamUpdate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *teamID: 03891d70-ea86-403e-8463-113917fd0355* * **schema** * **properties** * **teamID** * *format: uuid* * *type: string* * **required** * *0: teamID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A team with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates the name, description, and/or members of a team* * **tags** * *0: Teams* * **/v1/teams/{teamID}/accessControls** * **get** * *description: Return all the roles on the resource type, which include all grants of the given resource* * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/ResourceRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get RBAC access information about the team* * **tags** * *0: Teams* * **patch** * *description: Grant or revoke grants on roles of the resource* * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ControlAccessAction* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Update the RBAC grants on the team* * **tags** * *0: Teams* * **/v1/teams/{teamID}/identityRoles** * **get** * *description: Return granted roles to the team* * **parameters** * **0** * *$ref: #/components/parameters/teamID* * **1** * *description: Show only granted roles by resource type* * *in: query* * *name: resourceType* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/IdentityRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get granted roles to the team* * **tags** * *0: Teams* * **/v1/users** * *description: The \`users\` endpoint is used to manage users.* * **get** * *description: Returns a list of users in the current organization. This route is currently not supported by BYOC.* * **parameters** * **0** * *description: Show only users with an email address matching this value or regular expression.* * *in: query* * *name: email* * **schema** * *type: string* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/User* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the users in the current organization.* * **tags** * *0: Users* * **post** * *description: Adds a user already known to the portal to the current organization. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * **properties** * **email** * *description: The email address of the user to add to the organization.* * *format: email* * *type: string* * **required** * *0: email* * *type: object* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: User with the specified email address not found* * **409** * *description: User is already a member* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Adds an existing user to the current organization* * **tags** * *0: Users* * **/v1/users/{userID}** * **delete** * *description: Removes the specified user from the organization. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/userID* * **responses** * **200** * **content** * **application/json** * **example** * *userID: 03891d70-ea86-403e-8463-113917fd0466* * **schema** * **properties** * **userID** * *format: uuid* * *type: string* * **required** * *0: userID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A user with the specified ID was not found or is not a member of the organization* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Removes a user from the organization* * **tags** * *0: Users* * **get** * *description: Returns information for the specified user. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/userID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * *email: alice\@abc.com* * *firstName: Alice* * *lastName: Jones* * *userID: c74bb6a6-0000-1000-9000-1d874fa277b1* * *userKind: Standard* * **schema** * *$ref: #/components/schemas/User* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A user with the specified ID was not found or is not a member of the organization* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a user.* * **tags** * *0: Users* * **/v1/users/{userID}/identityRoles** * **get** * *description: Return granted roles to the user This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/userID* * **1** * *description: Show only granted roles by resource type* * *in: query* * *name: resourceType* * **schema** * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/IdentityRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get granted roles to the user* * **tags** * *0: Users* * **/v1/users/current** * **get** * *description: Returns information about the current user (i.e., the user making the API call). This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * *email: alice\@abc.com* * *firstName: Alice* * *lastName: Jones* * *userID: c74bb6a6-0000-1000-9000-1d874fa277b1* * *userKind: Standard* * **schema** * *$ref: #/components/schemas/User* * *description: OK* * *summary: Gets information about the current user* * **tags** * *0: Users* * **/v1/workspaceGroups** * *description: The \`workspaceGroups\` endpoint can be used to manage workspaces.* * **get** * *description: Returns a list of all of the workspace groups accessible to the user. Use the \`includeTerminated\` parameter to get a list of terminated workspace groups. * * **parameters** * **0** * *description: To include any terminated workspace groups, set to \`true\`* * *in: query* * *name: includeTerminated* * **schema** * *type: boolean* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *createdAt: 2023-09-18T09:56:56Z* * *name: demo-workspace-group* * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * *projectName: demo-project* * *provider: AWS* * *regionID: 7d6b3a5a-0000-1000-9000-d454af4c785c* * *regionName: eu-central-1* * *state: ACTIVE* * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **items** * *$ref: #/components/schemas/WorkspaceGroup* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the workspace groups the user can access* * **tags** * *0: Workspace Groups* * **post** * *description: Creates a new workspace group for the current user. You must specify the name, region ID, and firewall ranges for the workspace group in the API request body. You may use the admin user password to connect with any workspace of the group. The admin user password can be specified in the request body. If the admin user password is not specified in the API request, a password is generated and returned in the response object. * * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/WorkspaceGroupCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **adminPassword** * *type: string* * **workspaceGroupID** * *format: uuid* * *type: string* * **required** * *0: workspaceGroupID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new workspace group* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}** * **delete** * *description: Terminates a workspace group with the specified workspace group ID. You must specify the workspace group ID in the API call. By default, you may only terminate empty workspace groups (a workspace group without workspaces). To terminate a workspace group with active workspaces, use the \`force\` parameter. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **1** * *description: To terminate a workspace group even if it has active workspaces, set to \`true\`* * *in: query* * *name: force* * **schema** * *type: boolean* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceGroupID** * *format: uuid* * *type: string* * **required** * *0: workspaceGroupID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace group with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Terminates a workspace group* * **tags** * *0: Workspace Groups* * **get** * *description: Returns information for the specified workspace group ID, in JSON format. You must specify the workspace group ID in the API call. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2023-09-18T09:56:56Z* * *name: demo-workspace-group* * *projectID: 70ab4e59-0000-1000-9000-76a5b06ed9e3* * *projectName: demo-project* * *provider: AWS* * *regionID: 7d6b3a5a-0000-1000-9000-d454af4c785c* * *regionName: eu-central-1* * *state: ACTIVE* * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * *$ref: #/components/schemas/WorkspaceGroup* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace group with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a workspace group* * **tags** * *0: Workspace Groups* * **patch** * *description: Updates workspace group information for the specified workspace group, including the name, admin password, and firewall ranges. Specify the workspace group's new parameters in the request body. You must specify the workspace group ID in the API call. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/WorkspaceGroupUpdate* * *description: Here's a sample of JSON data sent to the API in the request body.* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceGroupID** * *format: uuid* * *type: string* * **required** * *0: workspaceGroupID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace group with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates a workspace group* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/accessControls** * **get** * *description: Return all the roles on the resource type, which include all grants of the given resource* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/ResourceRole* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get RBAC access information about the workspace group* * **tags** * *0: Workspace Groups* * **patch** * *description: Grant or revoke grants on roles of the resource* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/ControlAccessAction* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Update the RBAC grants on the workspace group* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/delegatedEntities** * **delete** * *description: Removes customer-provided entities from the list of entities that the workspace group's cloud workload identity can assume. Entities to remove can be specified in the request body, via query parameters, or both (values will be merged and deduplicated).* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **1** * *description: Entities to remove. Supports comma-separated values or multiple parameters.* * *in: query* * *name: entities* * **schema** * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/DelegatedEntitiesRemove* * *description: Optional request body to specify entities to remove.* * **responses** * **204** * *description: No Content* * **207** * **content** * **application/json** * **example** * **deleted** * *0: arn:aws:iam::123456789012:role/singlestore-iceberg-readonly* * **failed** * *0: arn:aws:iam::123456789012:role/singlestore-s3-pipelines-users-bucket-readonly* * **schema** * **properties** * **deleted** * **items** * *type: string* * *type: array* * **failed** * **items** * *type: string* * *type: array* * **required** * *0: deleted* * *1: failed* * *type: object* * *description: Multi-Status* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace group type or organization* * **409** * *description: Identity not configured for workspace group* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Removes entities that the workspace group's cloud workload identity can assume* * **tags** * *0: Workspace Groups* * *1: Delegated Entities* * **get** * *description: Returns the list of customer-provided entities, such as IAM roles, that the workspace group's cloud workload identity can assume to access customer resources.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/DelegatedEntity* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace group type or organization* * **409** * *description: Identity not configured for workspace group* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets the list of entities that the workspace group's cloud workload identity can assume* * **tags** * *0: Workspace Groups* * *1: Delegated Entities* * **post** * *description: Adds customer-provided entities, such as IAM roles, to the list of entities that the workspace group's cloud workload identity can assume.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **requestBody** * **content** * **application/json** * **example** * **entities** * *0: arn:aws:iam::987654321098:role/singlestore-iceberg-readonly* * *1: arn:aws:iam::987654321098:role/singlestore-s3-pipelines* * **schema** * *$ref: #/components/schemas/DelegatedEntitiesAppend* * *description: Specify the entities to add in the request body.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/DelegatedEntity* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace group type or organization* * **409** * *description: Identity not configured for workspace group* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Adds entities that the workspace group's cloud workload identity can assume* * **tags** * *0: Workspace Groups* * *1: Delegated Entities* * **/v1/workspaceGroups/{workspaceGroupID}/identity** * **get** * *description: Returns the cloud workload identity information associated with the workspace group. The identity format is determined by the cloud provider where the workspace group is hosted. For example, on AWS it's an IAM role ARN.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudWorkloadIdentity* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get the workspace group's cloud workload identity information* * **tags** * *0: Workspace Groups* * *1: Cloud Workload Identity* * **/v1/workspaceGroups/{workspaceGroupID}/privateConnections** * **get** * *description: Returns private connection information for the specified workspace group ID, in JSON format. You must specify the workspace group ID in the API call. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/PrivateConnection* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace group with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets private connection information for a workspace group* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/failback** * **patch** * *description: You must specify the workspace group ID of the group in the standby (secondary) region from which you are triggering the failback. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Starts failback to the primary region* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/failover** * **patch** * *description: You must specify the workspace group ID of the group in the inactive (primary) region from which you are triggering the failover. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Starts failover to the secondary region* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/regions** * **get** * *description: Returns a list of regions with regions IDs in JSON format. You must specify the workspace group ID of the group you are setting up for disaster recovery. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/Region* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about which regions you can setup as a disaster recovery backup* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/setup** * **post** * *description: You must specify the workspace group ID of the group you are setting up for disaster recovery and the region ID of your secondary region. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/StorageDRSetup* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Sets up Storage DR for the workspace group. Backup region and selected databases to be replicated are provided as part of the request.* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/startPreProvision** * **patch** * *description: You must specify the workspace group ID of the group in the primary region from which you are requesting to start pre-provision. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Start pre-provisioning from primary region* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/status** * **get** * *description: Returns the replication status of each database and the status of the latest Storage DR operation (Failover, Failback, etc.). You must specify the workspace group ID of the group that you are requesting status information for. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/StorageDRStatus* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about the storage DR status of the workspace group* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/DR/stopPreProvision** * **patch** * *description: You must specify the workspace group ID of the group in the primary region from which you are requesting to stop pre-provision. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Stops pre-provisioning from primary region* * **tags** * *0: Workspace Groups* * **/v1/workspaceGroups/{workspaceGroupID}/storage/retentionPeriod** * **patch** * *description: Updates the continuous backup retention period for all existing and future databases on a workspace group. This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceGroupID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RetentionPeriodParams* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates the retention period for continuous backups* * **tags** * *0: Workspace Groups* * **/v1/workspaces** * *description: The \`workspaces\` endpoint can be used to manage workspaces.* * **get** * *description: Returns a list of all of the workspaces accessible to the user in the specified workspace group. You must specify the workspace group ID in the API call to list the workspaces in the group. Use the \`includeTerminated\` parameter to list the terminated workspaces. * * **parameters** * **0** * *description: ID of the workspace group* * *in: query* * *name: workspaceGroupID* * **schema** * *format: uuid* * *type: string* * **1** * *description: To include any terminated workspaces, set to \`true\`* * *in: query* * *name: includeTerminated* * **schema** * *type: boolean* * **2** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *createdAt: 2023-09-18T09:56:56Z* * *endpoint: 192.168.224.3:30342* * *name: demo-workspace* * *size: S-00* * *state: ACTIVE* * *workspaceGroupID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **items** * *$ref: #/components/schemas/Workspace* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace group with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists the workspaces the user can access* * **tags** * *0: Workspaces* * **post** * *description: Creates a new workspace for the current user in the specified workspace group. You must specify the name of the workspace and the workspace group ID in the request body. Once a workspace is created, you can neither change its name nor its workspace group. See \[Workspaces]\(https\://docs.singlestore.com/managed-service/en/reference/management-api.html#workspaces-729524) for more information. This API call does not take any request parameters.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/WorkspaceCreate* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceID** * *format: uuid* * *type: string* * **required** * *0: workspaceID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new workspace* * **tags** * *0: Workspaces* * **/v1/workspaces/{workspaceID}** * **delete** * *description: Terminates a workspace with the specified workspace ID. You must specify the workspace ID in the API call. All the databases attached to the workspace are detached when the workspace is terminated. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceID** * *format: uuid* * *type: string* * **required** * *0: workspaceID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Terminates a workspace* * **tags** * *0: Workspaces* * **get** * *description: Returns workspace information for the specified workspace ID, in JSON format. You must specify the workspace ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **1** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * *createdAt: 2023-09-18T09:56:56Z* * *endpoint: 192.168.224.3:30342* * *name: demo-workspace* * *size: S-00* * *state: ACTIVE* * *workspaceGroupID: 7d6a4a5a-0000-1000-9000-d454af4c785b* * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * *$ref: #/components/schemas/Workspace* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information about a workspace* * **tags** * *0: Workspaces* * **patch** * *description: Updates workspace information for the specified workspace, including the size. Specify the workspace's new parameters in the request body. You must specify the workspace ID in the API call. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/WorkspaceUpdate* * *description: Here's a sample of JSON data sent to the API in the request body.* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceID** * *format: uuid* * *type: string* * **required** * *0: workspaceID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **409** * *description: Request conflict* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Updates information about a workspace* * **tags** * *0: Workspaces* * **/v1/workspaces/{workspaceID}/delegatedEntities** * **delete** * *description: Performs the same operation as the workspace group delegated entities DELETE endpoint. Removing entities through this workspace endpoint will update the delegated entities configuration for the entire workspace group. Entities to remove can be specified in the request body, via query parameters, or both (values will be merged and deduplicated).* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **1** * *description: Entities to remove. Supports comma-separated values or multiple parameters.* * *in: query* * *name: entities* * **schema** * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/DelegatedEntitiesRemove* * *description: Optional request body to specify entities to remove.* * **responses** * **204** * *description: No Content* * **207** * **content** * **application/json** * **example** * **deleted** * *0: arn:aws:iam::123456789012:role/singlestore-iceberg-readonly* * **failed** * *0: arn:aws:iam::123456789012:role/singlestore-s3-pipelines-users-bucket-readonly* * **schema** * **properties** * **deleted** * **items** * *type: string* * *type: array* * **failed** * **items** * *type: string* * *type: array* * **required** * *0: deleted* * *1: failed* * *type: object* * *description: Multi-Status* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace type or organization* * **409** * *description: Identity not configured for workspace* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Removes entities that the workspace's cloud workload identity can assume* * **tags** * *0: Workspaces* * *1: Delegated Entities* * **get** * *description: Returns the same list of delegated entities as the workspace group delegated entities endpoint. The workspace inherits and operates on the delegated entities configuration from its parent workspace group.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/DelegatedEntity* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace type or organization* * **409** * *description: Identity not configured for workspace* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets the list of entities that the workspace's cloud workload identity can assume* * **tags** * *0: Workspaces* * *1: Delegated Entities* * **post** * *description: Performs the same operation as the workspace group delegated entities POST endpoint. Adding entities through this workspace endpoint will update the delegated entities configuration for the entire workspace group.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **requestBody** * **content** * **application/json** * **example** * **entities** * *0: arn:aws:iam::987654321098:role/singlestore-iceberg-readonly* * *1: arn:aws:iam::987654321098:role/singlestore-s3-pipelines* * **schema** * *$ref: #/components/schemas/DelegatedEntitiesAppend* * *description: Specify the entities to add in the request body.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/DelegatedEntity* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **403** * *description: Operation not supported for workspace type or organization* * **409** * *description: Identity not configured for workspace* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Adds entities that the workspace's cloud workload identity can assume* * **tags** * *0: Workspaces* * *1: Delegated Entities* * **/v1/workspaces/{workspaceID}/identity** * **get** * *description: Returns the same cloud workload identity information as the workspace group's cloud workload identity endpoint. The workspace inherits the identity configuration from its parent workspace group.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/CloudWorkloadIdentity* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Get the workspace's cloud workload identity information* * **tags** * *0: Workspaces* * *1: Cloud Workload Identity* * **/v1/workspaces/{workspaceID}/privateConnections** * **get** * *description: Returns private connection information for the specified workspace ID, in JSON format. You must specify the workspace ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **1** * *$ref: #/components/parameters/fields* * **2** * *description: ID of the workspace group* * *in: query* * *name: workspaceGroupID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/PrivateConnection* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets private connection information for a workspace* * **tags** * *0: Workspaces* * **/v1/workspaces/{workspaceID}/privateConnections/kai** * **get** * *description: Returns the information needed to create private connection to SingleStore Kai. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: workspaceID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/PrivateConnectionKaiInfo* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A private connection with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets information to create private connection to SingleStore Kai for a workspace* * **tags** * *0: Private Connections* * *1: Workspaces* * **/v1/workspaces/{workspaceID}/privateConnections/outboundAllowList** * **get** * *description: Returns the account ID which must be allowed for outbound connections.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/PrivateConnectionOutboundAllowList* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets the outbound allow list for a workspace* * **tags** * *0: Workspaces* * **/v1/workspaces/{workspaceID}/resume** * **post** * *description: Resumes a workspace with the specified workspace ID. You must specify the workspace ID in the API call. * * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/WorkspaceResume* * *description: Here's a sample of JSON data sent in the request body to the API.* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceID** * *format: uuid* * *type: string* * **required** * *0: workspaceID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Resumes a workspace* * **tags** * *0: Workspaces* * **/v1/workspaces/{workspaceID}/suspend** * **post** * *description: Suspends a workspace with the specified workspace ID. You must specify the workspace ID in the API call.* * **parameters** * **0** * *$ref: #/components/parameters/workspaceID* * **responses** * **200** * **content** * **application/json** * **example** * *workspaceID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **workspaceID** * *format: uuid* * *type: string* * **required** * *0: workspaceID* * *type: object* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: A workspace with the specified ID was not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Suspends a workspace* * **tags** * *0: Workspaces* * **/v2/authorization/permissions** * **get** * *description: This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/PermissionDefinition* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all permissions* * **tags** * *0: Authorization* * **/v2/authorization/permissions/{resourceType}** * **get** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/resourceType* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/PermissionDefinition* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists permissions for the given resource type* * **tags** * *0: Authorization* * **/v2/authorization/roles** * **get** * *description: This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/RoleDefinitionV2* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all roles* * **tags** * *0: Authorization* * **post** * *description: This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleCreateV2* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinitionV2* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Creates a new role* * **tags** * *0: Authorization* * **/v2/authorization/roles/{role}** * **delete** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/role* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Resource not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Deletes a role* * **tags** * *0: Authorization* * **get** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/role* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinitionV2* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Resource not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets a role* * **tags** * *0: Authorization* * **put** * *description: This route is currently not supported by BYOC.* * **parameters** * **0** * *$ref: #/components/parameters/role* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleReplaceV2* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/RoleDefinitionV2* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **404** * *description: Resource not found* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Replaces a role* * **tags** * *0: Authorization* * **/v2/organizations/{organizationID}/workspaceGroups/{workspaceGroupID}/metrics** * **get** * *description: Returns metrics for the specified workspace group ID, in \[OpenMetrics]\(https\://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md) compatible format. You must specify the organization ID and workspace group ID in the API call. To fetch the organization id use the /v1/organizations/current endpoint.* * **parameters** * **0** * *$ref: #/components/parameters/organizationID* * **1** * *$ref: #/components/parameters/workspaceGroupID* * **responses** * **200** * **content** * **text/plain** * *example: # TYPE singlestoredb\_cloud\_threads\_running gauge \# singlestoredb\_cloud\_threads\_running{extractor="monitoring-customer-prd/memsql-exporter",node="node-3337afc7-443e-4126-b784-413903527186-aggregator-0",role="CA",workspace\_group\_id="3337afc7-443e-4126-b784-413903527186",workspace\_name="singlestore-central"} 1* * **schema** * *type: string* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Gets metrics for a workspace group* * **tags** * *0: Metrics* * **/v2/regions** * **get** * *description: Returns a list of valid regions for the user, including the region code and provider for each region.* * **parameters** * **0** * *$ref: #/components/parameters/fields* * **responses** * **200** * **content** * **application/json** * **example** * **0** * *provider: GCP* * *region: US West 1 (Oregon)* * *regionName: us-west1* * **1** * *provider: AWS* * *region: Europe Central 1 (Frankfurt)* * *regionName: eu-central-1* * **2** * *provider: AWS* * *region: Asia Pacific Southeast 2 (Sydney)* * *regionName: ap-southeast-2* * **schema** * **items** * *$ref: #/components/schemas/RegionV2* * *type: array* * *description: OK* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: Lists all of the regions* * **tags** * *0: Regions* * **/v2/sharedtier/virtualClusters** * **get** * *description: Returns a list of all Shared Edition deployments (starter clusters) for the current user. This route is currently not supported by BYOC.* * **responses** * **200** * **content** * **application/json** * **schema** * **items** * *$ref: #/components/schemas/SharedTierVirtualCluster* * *type: array* * *description: OK* * **400** * *description: Invalid request* * **401** * *description: Unauthorized* * **429** * *description: Too many requests* * **500** * *description: Internal server error* * *summary: List all Shared Edition deployments (starter clusters)* * **tags** * *0: SharedTier* * **post** * *description: Creates a new Shared Edition deployment (starter cluster) for the current user. You must specify a name, region ID, and firewall range(s) for the starter cluster in the API request body. This route is currently not supported by BYOC.* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierCreateVirtualCluster* * **responses** * **200** * **content** * **application/json** * **example** * *virtualClusterID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **schema** * **properties** * **virtualClusterID** * *format: uuid* * *type: string* * **required** * *0: virtualClusterID* * *type: object* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **/v2/sharedtier/virtualClusters/{virtualClusterID}** * **delete** * *description: Deletes a Shared Edition deployment (starter cluster) for the specified ID. You must specify the ID of the starter cluster in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualClusterID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Deletes a Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **get** * *description: Returns information about a Shared Edition deployment (starter cluster) with the specified ID. You must specify the ID of the starter cluster in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualClusterID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierVirtualCluster* * *description: OK* * **401** * *description: Unauthorized* * **404** * *description: Not found* * **500** * *description: Internal server error* * *summary: Returns information about a Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **/v2/sharedtier/virtualClusters/{virtualClusterID}/users** * **post** * *description: Creates a new user in a Shared Edition deployment (starter cluster) with the specified ID. You must specify the ID of the starter cluster in the API call. You can specify a password for the user in the request body. If a password is not specified, a password is automatically generated and returned in the API response. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualClusterID* * **schema** * *format: uuid* * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierCreateUser* * **responses** * **200** * **content** * **application/json** * **schema** * **example** * *password: Auto-generated-password1!* * *userID: 7d6b3a5a-0000-1000-9000-d454af4c785b* * **properties** * **password** * *type: string* * **userID** * *format: uuid* * *type: string* * **required** * *0: userID* * *1: password* * *type: object* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Creates a new user in a Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **/v2/sharedtier/virtualClusters/{virtualClusterID}/users/{userID}** * **delete** * *description: Deletes a user with the specified user ID from a Shared Edition deployment (starter cluster). You must specify the ID of the starter cluster and the user in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualClusterID* * **schema** * *format: uuid* * *type: string* * **1** * *in: path* * *name: userID* * **schema** * *format: uuid* * *type: string* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Deletes a user from a Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **patch** * *description: Updates a user in a Shared Edition deployment (starter cluster) with the specified user ID. You must specify the ID of the starter cluster and the user in the API call. This route is currently not supported by BYOC.* * **parameters** * **0** * *in: path* * *name: virtualClusterID* * **schema** * *format: uuid* * *type: string* * **1** * *in: path* * *name: userID* * **schema** * *format: uuid* * *type: string* * **requestBody** * **content** * **application/json** * **schema** * *$ref: #/components/schemas/SharedTierUpdateUser* * **responses** * **200** * **content** * **application/json** * **schema** * *type: boolean* * *description: OK* * **401** * *description: Unauthorized* * **500** * *description: Internal server error* * *summary: Updates a user in a Shared Edition deployment (starter cluster)* * **tags** * *0: SharedTier* * **security** * **0** * **ApiKeyAuth** * **servers** * **0** * *url: https\://api.singlestore.com* * **tags** * **0** * *description: The authorization API is available only for organizations where the 'RBACRoleV2' feature flag is enabled. * * *name: Authorization* *** Modified at: July 9, 2026 Source: [/cloud/reference/management-api/reference/](https://docs.singlestore.com/cloud/reference/management-api/reference/) (An index of the documentation is available at /llms.txt) --- # SingleStore Kai Use the SingleStore Kai ("the API") to connect to a SingleStore Helios cluster from MongoDB® clients and tools. You can run the MongoDB® queries natively in your SingleStore Helios cluster using the API without a code change. Additionally, you can use the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) and [Data API](https://docs.singlestore.com/cloud/reference/data-api.md) (SQL endpoint) in conjunction with the API. SingleStore Kai natively supports the BSON data type for storage. Refer to [Kai Data Storage Model](https://docs.singlestore.com/cloud/reference/singlestore-kai/kai-data-storage-model.md) for more information. > **📝 Note**: You can only run the MongoDB® commands in your SingleStore Helios database using the SingleStore Kai-enabled endpoint. **Disclaimer**: SingleStore is not a MongoDB® partner. SingleStore Kai™ is an API that offers compatibility and enhanced performance for applications built on MongoDB®. ## Why Use the API for MongoDB® SingleStore Kai offers an innovative, fast, and easy API to perform exceptionally fast analytics on JSON data for your MongoDB® applications without losing transactional performance. Key benefits to using SingleStore Kai: * **Drive faster analytics** on JSON data for your MongoDB® applications. * **Zero code changes and zero data transformations** required for supported commands and operations. * **Best of both worlds (NoSQL + SQL)**, lets developers utilize both the MongoDB® API and a SQL API to power their applications. * **Scale-out** SingleStore without any additional cost and simply by defining a shard key. * **Vector support** lets users provide semantics on vectors using `dot_product`, `euclidean_distance`, and many other functions available through the API. ## How the API Works The API uses a proxy service (provided by SingleStore) to run the MongoDB® commands and aggregation pipelines. The proxy service receives the MongoDB® commands, generates the SQL equivalent of the command, and runs the SQL in the SingleStore environment. It then reads the resulting rowsets and returns the results as a collection of BSON (Binary JSON) documents back to the MongoDB® client application. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2b1e5a9c8cdb77dc/6a7adac35967e5ae1d5d1e0f/mongodb-arch-updated-VswJGM.png) ## Availability The SingleStore Kai feature is available in all the regions where SingleStore Helios can be deployed. ## In this section * [Getting Started with SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai.md) * [Kai Data Storage Model](https://docs.singlestore.com/cloud/reference/singlestore-kai/kai-data-storage-model.md) * [Migrate from MongoDB® to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/migrate-from-mongodb-to-singlestore.md) * [Replicate MongoDB® Collections to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore.md) * [SingleStore Extension Commands](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands.md) * [Supported MongoDB® Commands, Data Types, and Operators](https://docs.singlestore.com/cloud/reference/singlestore-kai/supported-mongodb-commands.md) * [BSON Data Type Mapping](https://docs.singlestore.com/cloud/reference/singlestore-kai/data-type-mapping.md) * [MongoDB® to SQL Mapping](https://docs.singlestore.com/cloud/reference/singlestore-kai/mongodb-to-sql-mapping.md) * [Best Practices](https://docs.singlestore.com/cloud/reference/singlestore-kai/best-practices.md) * [Error Handling](https://docs.singlestore.com/cloud/reference/singlestore-kai/error-handling.md) * [Limitations](https://docs.singlestore.com/cloud/reference/singlestore-kai/limitations.md) *** Modified at: June 26, 2026 Source: [/cloud/reference/singlestore-kai/](https://docs.singlestore.com/cloud/reference/singlestore-kai/) (An index of the documentation is available at /llms.txt) --- # Best Practices Here are some best practices to manage performance tradeoffs between transactions and analytics using SingleStore Kai. ## Create Indexes To improve query performance, SingleStore recommends creating indexes on the collections. Refer to [Understanding Keys and Indexes in SingleStore](https://docs.singlestore.com/cloud/create-a-database/understanding-keys-and-indexes-in-singlestore.md) for more information. ## Faster Transactions (But Lower Resilience) SingleStore enables `retryWrites` by default, which provides higher resilience against network errors. Disabling the `retryWrites` setting will improve latency for write operations, but the application might need to manually handle errors due to transient network conditions. ```shell "mongodb://:@svc-XXXX.svc.singlestore.com:27017/?retryWrites=false&authMechanism=PLAIN&tls=true&loadBalanced=true" ``` SingleStore recommends leaving `retryWrites` enabled for ingesting data. *** Modified at: February 1, 2024 Source: [/cloud/reference/singlestore-kai/best-practices/](https://docs.singlestore.com/cloud/reference/singlestore-kai/best-practices/) (An index of the documentation is available at /llms.txt) --- # BSON Data Type Mapping SingleStore Kai natively supports the BSON data type for storage. > **📝 Note**: The BSON type and BSON-specific functions must only be used for directly querying tables created by SingleStore Kai using SQL. ## BSON Type Conversion * For all BLOB/string to BSON casting, SingleStore interprets the source as raw BSON bytes. * For JSON to BSON casting, SingleStore converts the JSON data according to the [MongoDB® Extended JSON V2](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/) format (Relaxed or Canonical). * BSON can be casted to JSON using the `:>JSON` notation. This conversion is performed in the MongoDB® Extended JSON V2 Relaxed format. The following table maps SingleStore data types to the corresponding BSON data types : | **SingleStoreData Type** | **BSON Data Type** | | ------------------------ | ------------------------------------------------ | | BIGINT | INT64 | | BINARY | BINARY (subtype 0) | | BIT | INT32 or INT64 | | BLOB | BINARY | | CHAR | STRING | | DATE | DATETIME | | DATETIME | DATETIME | | DATETIME(6) | DATETIME | | DECIMAL | INT32, INT64, DOUBLE, or DECIMAL (based on size) | | DOUBLE | DOUBLE | | ENUM | STRING | | FLOAT | DOUBLE | | GEOGRAPHY | STRING | | GEOGRAPHYPOINT | STRING | | INT | INT32 | | JSON | BSON | | LONGBLOB | BINARY | | LONGTEXT | STRING | | MEDIUMBLOB | BINARY | | MEDIUMINT | INT32 | | MEDIUMTEXT | STRING | | SMALLINT | INT32 | | TEXT | STRING | | TIME | STRING | | TIMESTAMP | DATETIME | | TIMESTAMP6 | DATETIME | | TINYBLOB | BINARY | | TINYINT | BOOLEAN | | TINYTEXT | STRING | | VARBINARY | BINARY | | VARCHAR | STRING | | VECTOR(F32) | BSON Array of DOUBLE type elements | | VECTOR(F64) | BSON Array of DOUBLE type elements | | VECTOR(I8) | BSON Array of INT32 type elements | | VECTOR(I16) | BSON Array of INT32 type elements | | VECTOR(I32) | BSON Array of INT32 type elements | | VECTOR(I64) | BSON Array of INT64 type elements | | YEAR | INT32 | *** Modified at: April 25, 2025 Source: [/cloud/reference/singlestore-kai/data-type-mapping/](https://docs.singlestore.com/cloud/reference/singlestore-kai/data-type-mapping/) (An index of the documentation is available at /llms.txt) --- # Error Handling ## Troubleshoot Connectivity to the API Endpoint To resolve connectivity issues to the SingleStore Kai ("the API") endpoint, perform the following steps. Move on to the next step only if the current step is working or the associated issue is resolved. 1. **Get the Connection String**: On the [Cloud Portal](https://portal.singlestore.com/), select **Clusters**. For your cluster, select **Connect > CLI Client**. If the API is enabled for your cluster, the connection string is visible under the **MongoDB Client** tab. The connection string is in `mongodb://:@:/` format. Here's a sample connection string: ``` mongodb://:@svc-XXX-mongo.YYY.svc.singlestore.com:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true ``` If the connection string is not present, ensure that the API endpoint is enabled for your cluster. You may need to create a new cluster and enable the API for this cluster. 2. **Ensure that the API endpoint's DNS can be resolved**: Copy the hostname of your API endpoint and run the following command to ensure that the client can access the service: ```shell nslookup svc-XXX-mongo.YYY.svc.singlestore.com ``` ```output Non-authoritative answer: Name: svc-XXX-mongo.YYY.svc.singlestore.com Address: xx.xxx.xxx.xx Name: svc-XXX-mongo.YYY-mongo.aws-virginia-6.svc.singlestore.com Address: xx.xx.xxx.xx ``` If this command returns an error, the client is unable to find (or resolve) the DNS of the API endpoint. Run this command from a different machine on a different network (for example, an EC2 instance). If the command works on other machines, then the issue is specific to the initial (original) client machine's environment. 3. **Establish a TCP connection**: Create a TCP connection to the API using the following command: ```shell telnet svc-XXX-mongo.YYY.svc.singlestore.com 27017 ``` ```output Trying xx.xxx.xxx.xx... Connected to svc-XXX-mongo.YYY.svc.singlestore.com. Escape character is '^]'. ``` If the connection is accepted, the output is similar to the one displayed above. If the output is different, a firewall rule may be blocking the connection. Run this command from a different machine on a different network (for example, from an EC2 instance). If the command works on other machines, then the issue is specific to the initial (original) client machine's environment. 4. **Establish a TLS connection, and perform initial handshake**: Use the `openssl s_client` to establish a TLS connection to the API endpoint and send a handshake message. Run the following command with the hostname of the API endpoint: ```shell (echo -e "\x24\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\xdd\x07\x00\x00\x00\x00\x00\x00\x00\x0f\x00\x00\x00\x08\x69\x73\x6d\x61\x73\x74\x65\x72\x00\x00" ; cat) | openssl s_client -connect svc-XXX-mongo.YYY.svc.singlestore.com:27017 ``` ```output read R BLOCK... ??ismastermsg... ``` If the output is different from the one displayed above, the machine is unable to create a TLS connection to the API endpoint. Run this command from a different machine on a different network (for example, from an EC2 instance). If the command works on other machines, then the issue is specific to the initial (original) client machine's environment. You may need to install a root certificate, if shown in the certificate chain output from `openssl`. 5. **Connect to the API endpoint without authentication**: Remove the ``, ``, and `authMechanism` parameter from the connection string, and ping the API endpoint, using `mongosh` (MongoDB® Shell) as follows: ```shell mongosh --eval "db.runCommand({ping:1})" "mongodb://svc-XXX-mongo.YYY.svc.singlestore.com:27017/?tls=true&loadBalanced=true" ``` ```output Current Mongosh Log ID: 64586253e71f86ba3f7fa0c1 ... ------ The server generated these startup warnings when booting 2023-05-08T02:45:42Z: Welcome to SingleStore with MongoDB compatibility! 2023-05-08T02:45:42Z: https://www.singlestore.com ------ { ok: 1 } ... ``` If the output does not contain a welcome message from SingleStore and an `ok` message, the machine is unable to ping the API endpoint. Run this command from a different host machine on a different network (for example, from an EC2 instance). If the command works on other machines, then the issue is specific to the initial (original) client machine's environment. 6. **Connect to the API endpoint with authentication**: Connect to the API endpoint using a known simple client, such as `mongosh` (MongoDB® Shell) using the original connection string copied from the [Cloud Portal](https://portal.singlestore.com). Replace the placeholders in the connection string with the username and password. Additionally, you can execute a simple SQL command, such as `SHOW DATABASES`. Here's an example: ```shell mongosh --eval "show databases" "mongodb://admin:pa55w0rd@svc-XXX-mongo.YYY.svc.singlestore.com:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true" ``` ```output Current Mongosh Log ID: 64586253e71f86ba3f7fa0c1 ... ------ The server generated these startup warnings when booting 2023-05-08T02:45:42Z: Welcome to SingleStore with MongoDB compatibility! 2023-05-08T02:45:42Z: https://www.singlestore.com ------ information_schema 0.00 B dbTest 0.00 B ``` The output contains a welcome message from SingleStore and a list of databases attached to the cluster. 7. **Connect to the API endpoint from the original client**: If all the previous steps have been successfully completed, try connecting to the API endpoint with the original client (the client application that you faced connectivity issues with, before starting the troubleshooting process). If possible, try connecting from the same machine on which you were able to successfully connect using `mongosh`. If you are still facing connectivity issues, contact [SingleStore Support](https://support.singlestore.com). ## Troubleshoot Specific Errors ## `MongoNetworkError: getaddrinfo ENOTFOUND svc-XXX-mongo.YYY.svc.singlestore.com` **Issue**: The client is unable to resolve the DNS for the API endpoint.  **Solution**: Ensure that you are using the correct connection string as shown in the Cloud Portal. ## `MongoNetworkError: Client network socket disconnected before secure TLS connection was established` **Issue**: The client is not allowed through the firewall rules for the cluster. **Solution**: Update the firewall rules to allow access to the client. ## `MongoNetworkError: ...ssl3_get_record:wrong version number:...` **Issue**: The client tried to connect to the DDL or DML endpoint. **Solution**: Ensure that you are using the correct connection string as shown in the Cloud Portal. ## `MongoServerError: Authentication failed` **Issue**: Incorrect username or password. **Solution**: Use the correct username or password in the connection string. *** Modified at: May 11, 2026 Source: [/cloud/reference/singlestore-kai/error-handling/](https://docs.singlestore.com/cloud/reference/singlestore-kai/error-handling/) (An index of the documentation is available at /llms.txt) --- # Getting Started with SingleStore Kai You can run MongoDB® queries on your SingleStore Kai ("the API") enabled clusters from MongoDB® clients/tools after performing these minimal steps: * Ensure that the [prerequisites](https://docs.singlestore.com/#section-idm4594681483993633674384689413.md) are met. * [Enable the API](https://docs.singlestore.com/#section-idm4503126944668833674386321804.md) endpoint for your cluster. * [Access the API](https://docs.singlestore.com/#section-idm4570325445928033674393546742.md) using supported tools/applications. **Disclaimer**: SingleStore is not a MongoDB® partner. SingleStore Kai™ is an API that offers compatibility and enhanced performance for applications built on MongoDB®. ## Prerequisites * A cluster group running SingleStore version 8.5 or later. * Run the following command to check the version: ```sql SELECT @@memsql_version; ``` * A SingleStore Kai-enabled cluster. Refer to [Enable the API](https://docs.singlestore.com/#section-idm4503126944668833674386321804.md) for more information. * Any MongoDB® client that uses the MongoDB® wire protocol or standard MongoDB® driver that supports the `loadBalanced` mode. ## Enable the API To enable the API for your SingleStore Helios cluster, perform the following tasks: 1. On the [Cloud Portal](https://portal.singlestore.com/), select **Create New > Cluster**. 2. On the **Create New Cluster** page, enter or select the configuration settings as applicable. 3. Under **Settings**, enable the **MongoDB® Compatible Endpoint API** toggle switch. 4. Select **Create Cluster**. The API is now enabled for your cluster. You can only enable or disable the API endpoint while creating a cluster. **Note**: The following global compatibility configuration variables are automatically set when the API is enabled for a cluster: * `json_compatibility_level`: `'8.0'` * `data_conversion_compatibility_level`: `'8.0'` * `regexp_format`: `'advanced'` If any of these variables cannot be set to the values specified above for a cluster, the API rejects any user authentication requests. ## Access the API A SingleStore Kai-enabled cluster has two different endpoints: * `mongodb://` * `mysql://` For example: * ``` mongodb://:@svc-XXXX.svc.singlestore.com:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true ``` * ``` mysql://svc-XXXX.svc.singlestore.com:3306 ``` Connect to the `mongodb://` endpoint to run MongoDB® queries. This endpoint supports SASL/PLAIN authentication mechanism. Clients/applications must connect to the API in load-balanced mode. If a client/application connects to the API without specifying `loadBalanced=true` in the connection string, the API rejects the authentication request with an explanatory error message. You can also connect to your MongoDB® instance via AWS PrivateLink. Refer to [Connect to MongoDB® using AWS PrivateLink](https://docs.singlestore.com/cloud/connect-to-singlestore/private-connections/connect-to-singlestore-helios-using-aws-privatelink/connect-to-mongodb-using-aws-privatelink.md) for more information. ## Manage Users and Permissions To access a cluster's `mongodb://` endpoint, a user must have the `EXECUTE` permission to the `cluster` database. You can add users and grant permissions only through the SQL endpoint, using SQL commands. The following example shows how to grant the `EXECUTE` permission to a user: ```sql GRANT EXECUTE ON cluster.* TO joe; ``` Refer to [Manage Database Users](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) for more information. ## Kai Shell The Kai Shell feature allows you to run MongoDB® queries on the Cloud Portal on a SingleStore Kai-enabled cluster. You can use any of the [supported commands, data types, and operators](https://docs.singlestore.com/cloud/reference/singlestore-kai/supported-mongodb-commands.md) in this shell similar to any other MongoDB® client. ## Access the Kai Shell To access the Kai Shell on the Cloud Portal, on the left navigation pane, select **Editor > Open Kai Shell**. Select the SingleStore Kai-enabled cluster you want to run your MongoDB® queries against from the dropdown list at the top. The shell automatically connects to the selected cluster, eliminating the need for a connection string. Here's an example: 1. Create a SingleStore Kai-enabled cluster on the Cloud Portal. Once the cluster is deployed, proceed to the next step. 2. On the left navigation pane, select **Editor >** **Open Kai Shell**. 3. From the dropdown list at the top, select your cluster. You can now run MongoDB® commands on your cluster using this shell. For example, run the following commands to insert a document in a collection named **exampleCollection**, and then view the contents of this collection: ```MongoDB use dbTest db.exampleCollection.insert({ _id: 1, message: "Hello there!"}) db.exampleCollection.find() ``` ```output [ { _id: 1, message: 'Hello there!' } ] ``` Refer to [Examples](https://docs.singlestore.com/#section-idm4606719460784033674417574169.md) for CRUD examples. ## Supported Tools You can connect to your SingleStore Kai-enabled cluster's `mongodb://` endpoint using the following tools: * [Kai Shell](https://docs.singlestore.com/#section-idm4599119925816033938378931294.md) (on the Cloud Portal) * [MongoDB® Shell (`mongosh`)](https://docs.singlestore.com/#N1684375786108.md) * [MongoDB® Database Tools](https://www.mongodb.com/docs/database-tools/) * Any MongoDB® client that uses the MongoDB® wire protocol or standard MongoDB® driver. Here are a few examples: * [MongoDB® for Visual Studio Code extension](https://docs.singlestore.com/#section-idm4580972749387233674416881795.md) * Mongoose.js * Prisma * Spring Boot You can also connect to SingleStore Kai in load-balanced mode via various application development tools using the following MongoDB® drivers: | **Language/Tool** | **Driver** | **Version** | | ----------------- | --------------------------------------------------------------------- | ----------- | | C | [mongo-c-driver](https://github.com/mongodb/mongo-c-driver) | 1.19.0 | | C++ | [mongo-c++-driver](https://github.com/mongodb/mongo-cxx-driver) | 3.7.0 | | C# / .NET | [mongo-csharp-driver](https://github.com/mongodb/mongo-csharp-driver) | 2.13.0 | | Go | [mongo-go-driver](https://github.com/mongodb/mongo-go-driver) | 1.6.2 | | Java | [mongo-java-driver](https://github.com/mongodb/mongo-java-driver) | 4.3.0 | | Kotlin | [mongo-java-driver](https://github.com/mongodb/mongo-java-driver) | 4.3.0 | | PHP | [mongo-php-driver](https://github.com/mongodb/mongo-php-driver) | 1.11.0 | | Python | [mongo-python-driver](https://github.com/mongodb/mongo-python-driver) | 3.12.0 | | Ruby | [mongo-ruby-driver](https://github.com/mongodb/mongo-ruby-driver) | 2.16.0 | | Rust | [mongo-rust-driver](https://github.com/mongodb/mongo-rust-driver) | 2.1.0 | | Scala | [mongo-java-driver](https://github.com/mongodb/mongo-java-driver) | 4.3.0 | | Swift | [mongo-swift-driver](https://github.com/mongodb/mongo-swift-driver) | 1.2.0 | ## Connect with MongoDB® Shell `mongosh` To connect using the `mongosh` shell, use the following command: ```shell mongosh "mongodb://@:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true" ``` For example: ```shell mongosh "mongodb://admin@svc-XXXX.svc.singlestore.com:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true" ``` At the next prompt, enter the password to access the API endpoint. ## Connect with MongoDB® for Visual Studio Code Extension To connect to the API endpoint from Visual Studio Code, perform the following tasks: 1. Install the **MongoDB® for VS Code** extension. 2. On the navigation pane, select the **MongoDB** icon. 3. Under **Connections**, select **Add Connection**. 4. Under **Connect with Connection String**, select **Connect**. 5. Enter the connection string in the following format: ``` mongodb://:@:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true ``` Once the connection is successful, a **Connected to: \:27017** message is displayed along with a **MongoDB connection successful** popup message. ## Examples The following examples perform CRUD operations using the API endpoint. **Create a Database** The following command creates a database named **dbExample**. ```MongoDB use dbExample show dbs ``` ```output mongo 0.00 B dbExample 0.00 B information_schema 0.00 B ``` **Create a Collection and Insert Documents** Run the following command to create a collection named **exampleCollection** and add multiple documents to this collection: ```mongodb db.exampleCollection.insertMany( [ { _id: 1, Code: "xv1f", Qty: 45 }, { _id: 2, Code: "nm3w", Qty: 30 }, { _id: 3, Code: "qoma", Qty: 20 }, { _id: 4, Code: "hr3k", Qty: 15 } ] ) ``` ```output { acknowledged: true, insertedIds: { '0': 1, '1': 2, '2': 3, '3': 4 } } ``` **Read Values from a Collection** The following command returns all the documents stored in **exampleCollection**: ```mongodb db.exampleCollection.find() ``` ```output [ { _id: 1, Code: 'xv1f', Qty: 45 }, { _id: 2, Code: 'nm3w', Qty: 30 }, { _id: 3, Code: 'qoma', Qty: 20 }, { _id: 4, Code: 'hr3k', Qty: 15 } ] ``` **Update a Collection** The following example updates the first document in **exampleCollection** where **Code** equals **"xv1f"**: ```mongodb db.exampleCollection.updateOne( { Code: "xv1f" }, { $set: { Qty: 40 } } ) ``` ```output { acknowledged: true, insertedId: null, matchedCount: 1, modifiedCount: 1, upsertedCount: 0 } ``` ```mongodb db.exampleCollection.find() ``` ```output [ { _id: 1, Code: 'xv1f', Qty: 40 }, { _id: 2, Code: 'nm3w', Qty: 30 }, { _id: 3, Code: 'qoma', Qty: 20 }, { _id: 4, Code: 'hr3k', Qty: 15 } ] ``` The following example updates all the documents in **exampleCollection** where **Qty** is less than or equal to **25**. ```mongodb db.exampleCollection.updateMany( { Qty: {$lte: 25 } }, { $set: { Comment: "Restock required" }, $inc: { Qty: -10 } } ) ``` ```output { acknowledged: true, insertedId: null, matchedCount: 2, modifiedCount: 2, upsertedCount: 0 } ``` ```mongodb db.exampleCollection.find() ``` ```output [ { _id: 1, Code: 'xv1f', Qty: 40 }, { _id: 2, Code: 'nm3w', Qty: 30 }, { _id: 3, Code: 'qoma', Qty: 10, Comment: 'Restock required' }, { _id: 4, Code: 'hr3k', Qty: 5, Comment: 'Restock required' } ] ``` **Delete a Document** The following command deletes all the documents from **exampleCollection** that match the specified condition: ```mongodb db.exampleCollection.deleteMany( { Comment: "Restock required", Qty: { $lt: 10 } } ) ``` ```output { acknowledged: true, deletedCount: 1 } ``` *** Modified at: May 19, 2026 Source: [/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai/](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai/) (An index of the documentation is available at /llms.txt) --- # Kai Data Storage Model By default, tables created using SingleStore Kai store data in the BSON format. The `_id` field of the document is added to the `_id` column, and the rest of the fields in the document are added to the `_more` column of the table in SingleStore. Tables created using SingleStore Kai have the following columns: | Column Name | Data Type | Description | | ----------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `_id` | BSON NOT NULL | The`_id`field required in allMongoDB®documents. | | `_more` | BSON NOT NULL | Contains all the fields in the document, except the`_id`field. | | `$_id` | PERSISTED LONGBLOB | Used to implementSingleStoreunique indexes onMongoDB®collections, similar to BSON unique indexes. This column contains the`_id`field normalized using the`BSON_NORMALIZE`or`BSON_NORMALIZE_NO_ARRAY`function, which transform the BSON data into a byte stream that preserves the BSON comparison when compared byte-by-byte. | For example, run the following command on a Kai-enabled cluster to create a collection named `exampleCollection`: ```MongoDB db.exampleCollection.insertMany( [ { _id: 1, Code: "xv1f", Qty: 45 }, { _id: 2, Code: "nm3w", Qty: 30 }, { _id: 3, Code: "qoma", Qty: 20 }, { _id: 4, Code: "hr3k", Qty: 15 } ] ) ``` This `exampleCollection` is stored in SingleStore as: ```MongoDB db.runCommand({sql:"SHOW CREATE TABLE dbTest.exampleCollection"}).cursor.firstBatch[0]["Create Table"] ``` ```output CREATE TABLE `exampleCollection` ( `_id` bson NOT NULL, `_more` bson NOT NULL COMMENT 'KAI_MORE' , `$_id` as BSON_NORMALIZE_NO_ARRAY(`_id`) PERSISTED longblob COMMENT 'KAI_AUTO' , SHARD KEY `__SHARDKEY` (`$_id`), UNIQUE KEY `__PRIMARY` (`$_id`) USING HASH, SORT KEY `__UNORDERED` () ) COMMENT 'KAI_CID:2QZxxxx1xxxxxxxx' AUTOSTATS_CARDINALITY_MODE=INCREMENTAL AUTOSTATS_HISTOGRAM_MODE=CREATE AUTOSTATS_SAMPLING=ON SQL_MODE='STRICT_ALL_TABLES' ``` The `COMMENT` clauses shown in this `CREATE TABLE` SQL statement, for example, `COMMENT 'KAI_AUTO'`, `COMMENT 'KAI_MORE'`, or `COMMENT 'KAI_CID:2QZxxxx1xxxxxxxx'`, are reserved for internal use. Using these `COMMENT` clauses in a `CREATE TABLE` statement manually may lead to unexpected behavior. When an index is created using the Kai API, SingleStore generates a computed column with the comment `KAI_AUTO` to indicate that this column was created using the Kai API and stores this information in the table metadata. This information is used during query generation for MongoDB® queries. SingleStore recommends creating MongoDB® collections and indexes using the SingleStore Kai endpoint (`mongodb://`) if you intend to query these collections using the Kai API. ## View the BSON Data To view the BSON data, SingleStore recommends the following: * Use the [Kai Shell](https://docs.singlestore.com/cloud/reference/singlestore-kai/getting-started-with-singlestore-kai/#section-idm4599119925816033938378931294.md) or other supported MongoDB® tools, such as MongoDB® Compass. * Cast the columns to JSON using the following SQL command: ```sql SELECT _id :> JSON , _more :> JSON FROM ; ``` ## Example To view the documents in a collection named `exampleCollection` using the Kai Shell, run the following command: ```mongodb db.exampleCollection.find() ``` ```output [ { _id: 4, Code: 'hr3k', Qty: 15 }, { _id: 3, Code: 'qoma', Qty: 20 }, { _id: 1, Code: 'xv1f', Qty: 45 }, { _id: 2, Code: 'nm3w', Qty: 30 } ] ``` To view the documents in `exampleCollection` using SQL commands, run the following command in the SQL command-line: ```sql SELECT _id :> JSON AS _id, _more :> JSON AS _more FROM exampleCollection; ``` ```output +-----+--------------------------+ | _id | _more | +-----+--------------------------+ | 4 | {"Code":"hr3k","Qty":15} | | 3 | {"Code":"qoma","Qty":20} | | 1 | {"Code":"xv1f","Qty":45} | | 2 | {"Code":"nm3w","Qty":30} | +-----+--------------------------+ ``` *** Modified at: April 14, 2025 Source: [/cloud/reference/singlestore-kai/kai-data-storage-model/](https://docs.singlestore.com/cloud/reference/singlestore-kai/kai-data-storage-model/) (An index of the documentation is available at /llms.txt) --- # Limitations * [System Limits](https://docs.singlestore.com/cloud/reference/configuration-reference/system-limits.md) apply to SingleStore Kai. *** Modified at: May 10, 2024 Source: [/cloud/reference/singlestore-kai/limitations/](https://docs.singlestore.com/cloud/reference/singlestore-kai/limitations/) (An index of the documentation is available at /llms.txt) --- # Migrate from MongoDB® to SingleStore To migrate your collection from MongoDB® to SingleStore, you can perform the following tasks: * Replicate your MongoDB® collections using Change Data Capture (CDC). * Back up your collections using `mongodump`, and then restore the backup into your SingleStore Helios cluster using `mongorestore`. * Export your collection using `mongoexport`, and then import the exported data into your SingleStore Helios cluster using `mongoimport`. ## Replicate MongoDB® Collections to SingleStore SingleStore allows you to perform a one-time data replication or ongoing CDC (Change Data Capture) from MongoDB® collections into SingleStore. Refer to [Replicate MongoDB® Collections to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore.md) for more information. ## Use `mongodump` and `mongorestore` ## Back Up Data using `mongodump` You can back up your collections in the BSON format using the `mongodump` command-line tool. Run the `mongodump` command from the system command-line. Refer to [mongodump](https://www.mongodb.com/docs/database-tools/mongodump/) for more information. Here's a sample command to create a backup: ```shell mongodump --uri="mongodb://:@:27017" \ --db= --collection= \ --out= ``` For MongoDB® Atlas instances, use the `mongodb+srv://` scheme. For example: ```shell mongodump --uri="mongodb+srv://:@" \ --db= --collection= \ --out= ``` ## Restore the Backup using `mongorestore` To restore the backup of your MongoDB® collections created using `mongodump` into your SingleStore Helios cluster, use the `mongorestore` command-line tool. Run the `mongorestore` command from the system command-line. Refer to [mongorestore](https://www.mongodb.com/docs/database-tools/mongorestore/) for more information. > **📝 Note**: Do not restore and build the indexes specified in the backup. Use the `--noIndexRestore` option to ignore the indexes in the backup. To create indexes in SingleStore, use the [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) command instead. Here's a sample command to restore your MongoDB® backup into SingleStore: ```shell mongorestore \ --uri 'mongodb://:@:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true' \ --noIndexRestore --ssl ``` ## Use `mongoexport` and `mongoimport` ## Export Data using `mongoexport` You can export your collection in either JSON or CSV format using the `mongoexport` command-line tool. Run the `mongoexport` command from the system command-line. Refer to [mongoexport](https://www.mongodb.com/docs/database-tools/mongoexport/) for more information. Run the following command to export collections from a MongoDB® endpoint: ```shell mongoexport --uri='mongodb://:@:27017' \ --db= --collection= \ --out= ``` Run the following command to export collections from a MongoDB® Atlas endpoint: ```shell mongoexport --uri mongodb+srv://@/ \ --collection= --out= ``` ## Import Data using `mongoimport` To import the data exported using `mongoexport` into your SingleStore Helios cluster, use the `mongoimport` command-line tool. `mongoimport` supports Extended JSON, CSV, and TSV formats that have been exported via `mongoexport`. Run the `mongoimport` command from the system command-line. Refer to [mongoimport](https://www.mongodb.com/docs/database-tools/mongoimport/) for more information. ```shell mongoimport \ --uri 'mongodb://:@:27017/?authMechanism=PLAIN&tls=true&loadBalanced=true' \ --db --collection \ --file ``` If your file contains arrays, add `--jsonArray` after the database name. *** Modified at: March 13, 2024 Source: [/cloud/reference/singlestore-kai/migrate-from-mongodb-to-singlestore/](https://docs.singlestore.com/cloud/reference/singlestore-kai/migrate-from-mongodb-to-singlestore/) (An index of the documentation is available at /llms.txt) --- # MongoDB® to SQL Mapping Here are a few conceptual mappings between MongoDB® and SQL. **Databases** Similar to MongoDB®, SingleStore databases serve as a lightweight container for scoping permissions and organizing collections, and they are implicitly created with a default configuration when collections are created. **Collections to Tables** In MongoDB®, collections are containers for documents. They are created either explicitly with the `createCollection` command or implicitly when a write occurs targeting a collection. In SingleStore, collections are represented as SQL tables. If a collection is created implicitly, a table is automatically created. **Documents to Rows** Whenever a BSON document is inserted through SingleStore Kai ("the API"), the document itself is mapped to a row in a target table. All top-level fields in the document are mapped to columns in the target table. Any top-level fields that do not match any columns are grouped together and inserted into a `_more` column. When a document is retrieved through the API, values from all the columns except the `_more` column are added to the document first. The contents of the `_more` column are included afterwards. If a field in the `_more` column has the same name as another column, the value from the existing column takes precedence over the field in the `_more` column. **BSON** Similar to MongoDB®, SingleStore Kai natively supports the BSON data type for storage. ## MongoDB® Terminologies and Concepts Mappings Table The following table maps MongoDB® terminologies and concepts to their SQL equivalents: | **MongoDB®Terms/Concepts** | **SQL Terms/Concepts** | | --------------------------------------- | --------------------------------------------------------- | | Database | Database | | Collection | Table | | View | View | | Document | Row | | Top-level field | Column | | Additional fields not matching a column | `_more`column | | Index | Index | | Primary key: the`_id`field | Primary key: Any unique column (including the`_id`column) | | `$lookup`or embedded documents | `JOIN` | | `$match` | `WHERE`,`HAVING` | | `$group` | `GROUP BY` | | `$sort` | `ORDER BY` | | `$limit` | `LIMIT` | | `$sum` | `SUM()` | | `$sum`,`$sortByCount` | `COUNT()` | ## `CREATE TABLE` and `ALTER TABLE` Statement Examples The following table shows `CREATE TABLE` and `ALTER TABLE` statements written in SQL and their MongoDB® equivalents: | **MongoDB®Statements** | **SQL Statements** | | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `db.createCollection("Stock")` | `CREATE TABLE Stock ( ID INT NOT NULL AUTO_INCREMENT, Code Varchar(30), Qty INT );` | | `db.createCollection("Stock", {rowStore:true} )` | `// Create Rowstore table CREATE ROWSTORE TABLE Stock ();` | | `//Create a new field db.Stock.updateMany( { }, { $set: { Status: "Y" } })` | `//Create a new column ALTER TABLE Stock ADD COLUMN Status CHAR(1); UPDATE Stock Set Status = 'Y';` | | `//Remove a field db.Stock.updateMany( { }, { $unset: { "Status": 1 } } )` | `//Remove a column ALTER TABLE DROP COLUMN Stock;` | | `db.Stock.drop()` | `DROP TABLE Stock;` | | `db.txs.createIndex( {"inventory.Quantity_Inventory": 1})` | ``ALTER TABLE inventory ADD COLUMN Quantity_Inventory AS JSON_EXTRACT_STRING(`_more`,Quantity_Inventory) PERSISTED INT;`` | ## `SELECT` Statement Examples The following table shows `SELECT` statements written in SQL and their MongoDB® equivalents: | **MongoDB®Statements** | **SQL Statements** | | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | `db.Stock.find()` | `SELECT * FROM Stock;` | | `db.Stock.find( { Status = "Y" }, { Code: 1, Qty: 1, _id: 0 } )` | `SELECT Code, Qty FROM Stock WHERE Status = "Y";` | | `db.Stock.find( { Status: "Y", Qty: { $gt: 25 } } )` | `SELECT * FROM Stock WHERE Status = "Y" AND Qty > 25;` | | `db.Stock.find( { $or: [{Status: "Y"}, {Qty: { $gt: 25}} ] } )` | `SELECT * FROM Stock WHERE Status = "Y" OR Qty > 25;` | | `db.Stock.find({ Stock: /^i/ }).sort({ _id: 1}) // Or db.Stock.find({ Stock: { $regex: /^i/ }}).sort({ _id: 1})` | `SELECT * FROM Stock WHERE Code LIKE "i%" ORDER BY ID ASC;` | | `db.Stock.find().sort({ id: -1 })` | `SELECT * FROM Stock ORDER BY ID DESC;` | | `db.Stock.find().limit(5).skip(2)` | `SELECT * FROM Stock LIMIT 5 SKIP 2;` | | `db.Stock.find( { Status: "Y" }).explain()` | `EXPLAIN SELECT * FROM Stock WHERE Code = "Y";` | ## `INSERT` Statement Examples The following table shows `INSERT` statements written in SQL and their MongoDB® equivalents: | **MongoDB®Statements** | **SQL Statements** | | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | `db.Stock.inserOne( {Stock: "at6h", Qty: 35 })` | `INSERT INTO Stock (Code, Qty) VALUES ( "at6h", 35);` | | `db.Stock.insertMany([ { Stock: "ik7b", Qty: 40 }, { Stock: "ph9x", Qty: 25 }, { Stock: "at6h", Qty: 20 }])` | `INSERT INTO Stock (Code, Qty) VALUES ("ik7b", 40), ("ph9x", 25), ("at6h", 20);` | ## `UPDATE` Statement Example The following table shows `UPDATE` statements written in SQL and their MongoDB® equivalents: | **MongoDB®Statements** | **SQL Statements** | | ------------------------------------------------------------------------------- | ----------------------------------------------- | | `db.Stock.updateMany( { Qty: { $gt: 40 } }, { $set: { Status: "Y" } })` | `UPDATE Stock SET Status = "Y" WHERE Qty > 40;` | ## `DELETE` Statement Examples The following table shows `DELETE` statements written in SQL and their MongoDB® equivalents: | **MongoDB®Statements** | **SQL Statements** | | ------------------------------------------- | --------------------------------------- | | `db.Stock.deleteMany( { Status: "Y" })` | `DELETE FROM Stock WHERE Status = "Y";` | | `db.Stock.deleteMany({})` | `DELETE FROM Stock;` | ## Aggregate Function Examples The following table shows aggregate functions written in MongoDB® and their SQL equivalents: | **MongoDB®Statements** | **SQL Statements** | | -------------------------------------------------------------------------------------------------- | ----------------------------- | | `db.Stock.aggregate([ { $group: { _id: null, count: { sum: 1 }}} ])` | `SELECT COUNT(*) FROM Stock;` | | `db.Stock.aggregate([ { $group: { _id: null, total: { $sum: "$Qty" }}} ])` | `SELECT SUM(Qty) FROM Stock;` | *** Modified at: August 9, 2024 Source: [/cloud/reference/singlestore-kai/mongodb-to-sql-mapping/](https://docs.singlestore.com/cloud/reference/singlestore-kai/mongodb-to-sql-mapping/) (An index of the documentation is available at /llms.txt) --- # Replicate MongoDB® Collections to SingleStore Change Data Capture (CDC) pipelines enable you to ingest historical data and sync the continuous changes to data as they happen on the source MongoDB® database. SingleStore allows you to replicate your existing MongoDB® collections to your SingleStore Helios database using Change Data Capture (CDC). The CDC pipeline ingests data in the BSON format. You can perform the replication using any of the following methods: * [Replicate MongoDB® Collections using MongoDB® Wrapper Commands](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-mongodb-wrapper-commands.md) through the SingleStore Kai endpoint * [Replicate MongoDB® Collections using SQL](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-sql.md) To replicate data from MongoDB® using CDC via AWS PrivateLink, refer to [Replicate MongoDB® Collections via AWS PrivateLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-via-aws-privatelink.md). ## Prerequisites ## Source Database Requirements The replication feature uses MongoDB® [change streams](https://www.mongodb.com/docs/manual/changeStreams/), and it works only with replica sets. A MongoDB® user must have the following roles (privileges) to replicate the collections in SingleStore: * Read the `admin` database for the operation log (oplog) * Read the `config` database in the configuration server * `listDatabases` privilege * `clusterMonitor` role * Cluster-wide `find` and `changeStream` privileges * Write permission to the `singlestore` database. The `singlestore` database is automatically created during the replication process. You can provide the privileges or assign roles to the MongoDB® user using the MongoDB® Atlas UI or MongoDB® commands. For example: * Using the UI for MongoDB® Atlas: 1. On the MongoDB® Cloud dashboard, select **Projects > \**. 2. In the left navigation pane, under **Security**, select **Database Access**. 3. On the **Database Users page**, select **Edit** for the MongoDB® user used to connect to the MongoDB® instance. 4. On the **Edit User** dialog, under **Database User Privileges**, select **Specific Privileges > Add Specific Privilege**. 5. Add the following privileges: * `readAnyDatabase` * `read` for the `config` database * `readWrite` for the `singlestore` database ![Provide the readWrite privilege for the singlestore database to the user.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blted17cafda8869051/6a7adb02889c373053057c80/mongo-singlestore-db-privilege-C0hiD6.png) 6. Select **Update User**. * Using MongoDB® commands for self-managed deployments: The following commands create a role and then assign the role to a user. ```MongoDB db.adminCommand({ createRole: 'cdcRole', privileges: [ {'resource': {'cluster': true}, 'actions': ['find', 'changeStream']} ], roles: [ {'role': 'read', 'db': 'admin'}, {'role': 'read', 'db': 'local'}, {'role': 'read', 'db': 'config'}, {'role': 'readWrite', 'db': 'singlestore'} ]}) db.adminCommand({ createUser: 'cdcUser', pwd: 's2mongoCDC12', roles: ['cdcRole'] }) ``` Refer to [Database Commands](https://www.mongodb.com/docs/manual/reference/command/) for more information. ## Allow Access to the MongoDB® Instance To allow SingleStore to connect to your MongoDB® instance (MongoDB® Atlas or self-managed deployment), allow incoming traffic from the outbound IP addresses of your SingleStore Helios cluster group. > **📝 Note**: Allow all the outbound IP addresses. 1. Copy the outbound IP address of your SingleStore Helios cluster. 1. On the [Cloud Portal](https://portal.singlestore.com), select **Clusters**. Select the three dots under the **Actions** column for your cluster, and then select **Access & Security** from the list. 2. In the **Firewall** section, select **Outbound**. Copy all the IPs listed under **IPs for Outbound Connections**. 2. Allow incoming traffic to the MongoDB® instance from the SingleStore Helios outbound IP addresses. Follow the instructions for your deployment: * For MongoDB® Atlas deployments, add the outbound IPs to the MongoDB® **IP Access List**. Refer to [Configure IP Access List Entries](https://www.mongodb.com/docs/atlas/security/ip-access-list/) for related information. * For MongoDB® self-managed deployments, allow incoming traffic from the IP addresses. Refer to [Network and Configuration Hardening](https://www.mongodb.com/docs/manual/core/security-hardening/#network-hardening) for more information. ## Limitations * SingleStore does not support data replication using CDC pipelines from MongoDB® Atlas standalone or serverless instances. Replication is only supported for replica set deployments. * Because MongoDB® time series collections do not support change streams, they cannot be replicated in SingleStore using CDC. Refer to [Time Series Collection Limitations](https://www.mongodb.com/docs/manual/core/timeseries/timeseries-limitations/) for more information. ## In this section * [Replicate MongoDB® Collections using MongoDB® Wrapper Commands](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-mongodb-wrapper-commands.md) * [Replicate MongoDB® Collections using SQL](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-sql.md) * [Replicate MongoDB® Collections via AWS PrivateLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-via-aws-privatelink.md) *** Modified at: March 9, 2026 Source: [/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/) (An index of the documentation is available at /llms.txt) --- # SingleStore Extension Commands SingleStore extension commands are custom commands unique to SingleStore Kai. You can run the following extension commands using a SingleStore Kai-enabled endpoint. | Function | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | [createCollection](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createcollection.md) | Creates a new collection. | | [createIndexes](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createindexes.md) | Creates indexes onMongoDB®collections. | | [createLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createlink.md) | Creates a new link used to connect with remoteMongoDB®instances. | | [createView](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createview.md) | Creates a non-materialized view. | | [dotProduct](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/dotproduct.md) | Returns the scalar product of two vector inputs. | | [dropLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/droplink.md) | Deletes a link created using the`createLink`command. | | [euclideanDistance](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/euclideandistance.md) | Returns the scalar Euclidean distance between two vector inputs. | | [getDefaultCollectionOptions](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/getdefaultcollectionoptions.md) | Returns the defaults that are used when a collection is created implicitly by a write command. | | [merge](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/merge.md) | Writes the result of an aggregation pipeline to a specified collection. | | [setDefaultCollectionOptions](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/setdefaultcollectionoptions.md) | Sets the defaults that are used when a collection is created implicitly by a write command. | | [setProfilingLevel](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/setprofilinglevel.md) | Sets the database profiling level for queries run using theSingleStore Kai(`mongodb://`) endpoint. | | [sql](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/sql.md) | Runs the specified SQL statement. | | [vectorSearch](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/vectorsearch.md) | Performs an approximate nearest neighbor (ANN) vector search. | ## In this section * [createCollection](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createcollection.md) * [createIndexes](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createindexes.md) * [createLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createlink.md) * [createView](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/createview.md) * [dotProduct](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/dotproduct.md) * [dropLink](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/droplink.md) * [euclideanDistance](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/euclideandistance.md) * [getDefaultCollectionOptions](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/getdefaultcollectionoptions.md) * [merge](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/merge.md) * [setDefaultCollectionOptions](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/setdefaultcollectionoptions.md) * [setProfilingLevel](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/setprofilinglevel.md) * [sql](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/sql.md) * [vectorSearch](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/vectorsearch.md) *** Modified at: March 25, 2025 Source: [/cloud/reference/singlestore-kai/singlestore-extension-commands/](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands/) (An index of the documentation is available at /llms.txt) --- # Supported MongoDB® Commands, Data Types, and Operators The following tables reflect which MongoDB® commands, data types, and operators are supported by SingleStore Kai. ## Query and Projection Operators ## Array Operators | Operator | Supported | | ---------- | --------- | | $all | ✔ | | $elemMatch | ✔ | | $size | ✔ | ## Bitwise Operators | Operator | Supported | | ------------- | --------- | | $bitsAllSet | ✔ | | $bitsAnySet | ✔ | | $bitsAllClear | ✔ | | $bitsAnyClear | ✔ | ## Comment Operator | Operator | Supported | | -------- | --------- | | $comment | ✔ | ## Comparison Operators | Operator | Supported | | -------- | --------- | | $eq | ✔ | | $gt | ✔ | | $gte | ✔ | | $lt | ✔ | | $lte | ✔ | | $ne | ✔ | | $in | ✔ | | $nin | ✔ | ## Element Operators | Operator | Supported | | -------- | --------- | | $exists | ✔ | | $type | ✔ | ## Evaluation Query Operators | Operator | Supported | | ----------- | --------- | | $expr | ✔ | | $jsonSchema | \[empty] | | $mod | ✔ | | $regex | ✔ | | $text | ✔ | | $where | \[empty] | ## Logical Operators | Operator | Supported | | -------- | --------- | | $or | ✔ | | $and | ✔ | | $not | ✔ | | $nor | ✔ | ## Projection Operators | Operator | Supported | | ---------- | --------- | | $ | ✔ | | $elemMatch | ✔ | | $meta | \[empty] | | $slice | ✔ | ## Update Operators ## Update Array Operators | Operator | Supported | | ----------------- | --------- | | $ | ✔ | | $\[] | ✔ | | $\[\] | ✔ | | $addToSet | ✔ | | $pop | ✔ | | $pullAll | ✔ | | $pull | ✔ | | $push | ✔ | | $pushAll | \[empty] | ## Update Bitwise Operators | Operator | Supported | | -------- | --------- | | $bit | ✔ | ## Update Field Operators | Operator | Supported | | ------------ | --------- | | $inc | ✔ | | $mul | ✔ | | $rename | ✔ | | $setOnInsert | ✔ | | $set | ✔ | | $unset | ✔ | | $min | ✔ | | $max | ✔ | | $currentDate | ✔ | ## Update Modifiers | Operator | Supported | | --------- | --------- | | $each | ✔ | | $slice | ✔ | | $sort | ✔ | | $position | ✔ | ## Aggregation Pipeline Operators ## Aggregation Accumulator Expressions | Operator | Supported | | ------------- | --------- | | $sum | ✔ | | $integral | ✔ | | $shift | ✔ | | $avg | ✔ | | $first | ✔ | | $last | ✔ | | $max | ✔ | | $min | ✔ | | $push | ✔ | | $addToSet | ✔ | | $stdDevPop | ✔ | | $stdDevSamp | ✔ | | $accumulator | \[empty] | | $firstN | ✔ | | $bottom | ✔ | | $bottomN | ✔ | | $lastN | ✔ | | $maxN | \[empty] | | $mergeObjects | ✔ | | $top | ✔ | | $topN | ✔ | ## Arithmetic Operators | Operator | Supported | | --------- | --------- | | $abs | ✔ | | $add | ✔ | | $ceil | ✔ | | $divide | ✔ | | $exp | ✔ | | $floor | ✔ | | $ln | ✔ | | $log | ✔ | | $log10 | ✔ | | $mod | ✔ | | $multiply | ✔ | | $pow | ✔ | | $sqrt | ✔ | | $subtract | ✔ | | $rand | ✔ | | $trunc | ✔ | | $round | ✔ | ## Array Operators | Operator | Supported | | -------------- | --------- | | $arrayElemAt | ✔ | | $arrayToObject | ✔ | | $concatArrays | ✔ | | $filter | ✔ | | $indexOfArray | ✔ | | $isArray | ✔ | | $objectToArray | ✔ | | $range | ✔ | | $reverseArray | ✔ | | $reduce | ✔ | | $size | ✔ | | $slice | ✔ | | $zip | ✔ | | $in | ✔ | | $count | ✔ | | $first | ✔ | | $firstN | ✔ | | $sortArray | \[empty] | | $last | ✔ | | $lastN | ✔ | | $map | ✔ | | $maxN | \[empty] | | $minN | \[empty] | ## Boolean Operators | Operator | Supported | | -------- | --------- | | $and | ✔ | | $or | ✔ | | $not | ✔ | ## Comparison Operators | Operator | Supported | | ---------- | --------- | | $cmp | ✔ | | $eq | ✔ | | $gt | ✔ | | $gte | ✔ | | $lt | ✔ | | $lte | ✔ | | $ne | ✔ | | $in | ✔ | | $nin | ✔ | | $sortArray | \[empty] | ## Conditional Expression Operators | Operator | Supported | | -------- | --------- | | $cond | ✔ | | $ifNull | ✔ | | $switch | ✔ | ## Custom Aggregation Expression Operators | Operator | Supported | | ------------ | --------- | | $accumulator | \[empty] | | $function | \[empty] | ## Data Size Operators | Operator | Supported | | ----------- | --------- | | $binarySize | ✔ | | $bsonSize | ✔ | ## Data Type Operators | Operator | Supported | | --------- | --------- | | $type | ✔ | | $isNumber | ✔ | ## Date Operators | Operator | Supported | | --------------- | --------- | | $dayOfWeek | ✔ | | $dayOfMonth | ✔ | | $dayOfYear | ✔ | | $year | ✔ | | $month | ✔ | | $week | ✔ | | $hour | ✔ | | $minute | ✔ | | $second | ✔ | | $millisecond | ✔ | | $dateToString | ✔ | | $isoDayOfWeek | ✔ | | $isoWeek | ✔ | | $dateFromParts | ✔ | | $dateToParts | ✔ | | $dateFromString | ✔ | | $isoWeekYear | ✔ | | $dateAdd | ✔ | | $dateDiff | ✔ | | $dateSubtract | ✔ | | $dateTrunc | ✔ | ## Literal Operator | Operator | Supported | | -------- | --------- | | $literal | ✔ | ## Object Expression Operators | Operator | Supported | | -------------- | --------- | | $mergeObjects | ✔ | | $objectToArray | ✔ | | $setField | ✔ | | $unsetField | ✔ | ## Natural Operator | Operator | Supported | | -------- | --------- | | $natural | \[empty] | ## Set Operators | Operator | Supported | | ---------------- | --------- | | $setEquals | ✔ | | $setIntersection | ✔ | | $setUnion | ✔ | | $setDifference | ✔ | | $setIsSubset | ✔ | | $anyElementTrue | ✔ | | $allElementsTrue | ✔ | ## Stage Operators | Operator | Supported | | ------------------ | --------- | | $collStats | ✔ | | $project | ✔ | | $match | ✔ | | $redact | \[empty] | | $limit | ✔ | | $skip | ✔ | | $unwind | ✔ | | $group | ✔ | | $sample | ✔ | | $sort | ✔ | | $geoNear | \[empty] | | $lookup | ✔ | | $merge | ✔ | | $out | ✔ | | $indexStats | ✔ | | $setWindowFields | ✔ | | $facet | ✔ | | $bucket | \[empty] | | $bucketAuto | \[empty] | | $sortByCount | ✔ | | $addFields | ✔ | | $replaceRoot | ✔ | | $count | ✔ | | $currentOp | \[empty] | | $listLocalSessions | \[empty] | | $listSessions | \[empty] | | $graphLookup | ✔ | | $changeStream | ✔ | | $densify | \[empty] | | $documents | ✔ | | $fill | \[empty] | | $unionWith | ✔ | | $planCacheStats | \[empty] | | $unset | ✔ | | $replaceWith | ✔ | | $search | \[empty] | | $searchMeta | \[empty] | | $vectorSearch | ✔ | ## String Operators | Operator | Supported | | ------------- | --------- | | $concat | ✔ | | $indexOfBytes | ✔ | | $indexOfCP | ✔ | | $trim | ✔ | | $ltrim | ✔ | | $rtrim | ✔ | | $regexFind | \[empty] | | $regexFindAll | \[empty] | | $regexMatch | ✔ | | $replaceOne | ✔ | | $replaceAll | ✔ | | $split | ✔ | | $strcasecmp | ✔ | | $strLenBytes | ✔ | | $strLenCP | ✔ | | $substr | ✔ | | $substrBytes | ✔ | | $substrCP | ✔ | | $toString | ✔ | | $toLower | ✔ | | $toUpper | ✔ | | $trim | ✔ | ## System Variables | Variable | Supported | | -------------- | --------- | | $$CURRENT | ✔ | | $$CLUSTERTIME | \[empty] | | $$DESCEND | \[empty] | | $$KEEP | \[empty] | | $$NOW | \[empty] | | $$PRUNE | \[empty] | | $$REMOVE | ✔ | | $$ROOT | ✔ | | $$SEARCH\_META | \[empty] | ## Text Search Operator | Operator | Supported | | -------- | --------- | | $meta | \[empty] | ## Timestamp Expression Operators | Operator | Supported | | ------------ | --------- | | $tsIncrement | ✔ | | $tsSecond | ✔ | ## Trigonometry Expression Operators | Operator | Supported | | ----------------- | --------- | | $sin | ✔ | | $cos | ✔ | | $tan | ✔ | | $asin | ✔ | | $acos | ✔ | | $atan | ✔ | | $atan2 | \[empty] | | $asinh | \[empty] | | $acosh | \[empty] | | $atanh | \[empty] | | $sinh | \[empty] | | $cosh | \[empty] | | $tanh | \[empty] | | $degreesToRadians | ✔ | | $radiansToDegrees | ✔ | ## Type Conversion Operators | Operator | Supported | | ----------- | --------- | | $convert | ✔ | | $toBool | ✔ | | $toDate | ✔ | | $toDecimal | ✔ | | $toDouble | ✔ | | $toInt | ✔ | | $toLong | ✔ | | $toObjectId | ✔ | | $toString | ✔ | ## Variable Operators | Operator | Supported | | -------- | --------- | | $map | ✔ | | $let | ✔ | ## Miscellaneous Operators | Operator | Supported | | --------------- | --------- | | $denseRank | ✔ | | $documentNumber | ✔ | | $getField | ✔ | | $rand | ✔ | | $rank | ✔ | | $sampleRate | ✔ | ## Data Types | Data Type | Supported | | ----------------------- | --------- | | Double | ✔ | | String | ✔ | | Object | ✔ | | Array | ✔ | | Binary Data | ✔ | | ObjectId | ✔ | | Boolean | ✔ | | Date | ✔ | | Null | ✔ | | 32-bit Integer (int) | ✔ | | Timestamp | ✔ | | 64-bit Integer (long) | ✔ | | MinKey | ✔ | | MaxKey | ✔ | | Decimal128 | ✔ | | Regular Expression | ✔ | | JavaScript | ✔ | | JavaScript (with scope) | ✔ | | Undefined | ✔ | | Symbol | ✔ | | DBPointer | ✔ | ## Indexes and Index Properties ## Indexes | Property | Supported | | ------------------ | --------- | | Single Field Index | ✔ | | Compound Index | ✔ | | Multikey Index | ✔ | | Text Index | \[empty] | | Wildcard Indexes | \[empty] | | 2dsphere | \[empty] | | 2d Index | \[empty] | | GeoHaystackIndexes | \[empty] | | Hashed Index | \[empty] | ## Index Properties | Property | Supported | | ---------------- | --------- | | TTL | \[empty] | | Unique | ✔ | | Partial | \[empty] | | Case Insensitive | \[empty] | | Hidden | \[empty] | | Sparse | \[empty] | | Background | \[empty] | ## Cursor Methods | Method | Supported | | ---------------------------- | --------- | | cursor.batchSize() | ✔ | | cursor.close() | ✔ | | cursor.isClosed() | ✔ | | cursor.collation() | \[empty] | | cursor.comment() | ✔ | | cursor.count() | ✔ | | cursor.explain() | ✔ | | cursor.forEach() | ✔ | | cursor.hasNext() | ✔ | | cursor.hint() | \[empty] | | cursor.isExhausted() | ✔ | | cursor.itcount() | ✔ | | cursor.limit() | ✔ | | cursor.map() | ✔ | | cursor.maxScan() | \[empty] | | cursor.maxTimeMS() | \[empty] | | cursor.max() | \[empty] | | cursor.min() | \[empty] | | cursor.next() | ✔ | | cursor.noCursorTimeout() | \[empty] | | cursor.objsLeftInBatch() | ✔ | | cursor.pretty() | ✔ | | cursor.readConcern() | ✔ | | cursor.readPref() | ✔ | | cursor.returnKey() | \[empty] | | cursor.showRecordId() | \[empty] | | cursor.size() | ✔ | | cursor.skip() | ✔ | | cursor.sort() | ✔ | | cursor.tailable() | \[empty] | | cursor.toArray() | ✔ | | cursor.addOption() | \[empty] | | cursor.allowDiskUse() | \[empty] | | cursor.allowPartialResults() | \[empty] | | cursor.maxAwaitTimeMS() | \[empty] | | cursor.tryNext() | \[empty] | ## Database Commands ## Administrative Commands | Command | Supported | | ------------------------------ | --------- | | Capped Collections | | | cloneCollectionAsCapped | | | collMod | | | collMod: expireAfterSeconds | | | convertToCapped | | | copydb | | | create | ✔ | | createView | | | createIndexes | ✔ | | currentOp | ✔ | | drop | ✔ | | dropDatabase | ✔ | | dropIndexes | ✔ | | filemd5 | | | killCursors | ✔ | | killOp | | | listCollections | ✔ | | listDatabases | ✔ | | listIndexes | ✔ | | reIndex | | | renameCollection | | | fsync | | | fsyncUnlock | | | getDefaultRWConcern | | | getAudtifConfig | | | getClusterParameter | | | getParameter | | | logRotate | | | rotateCertificates | | | setFeatureCompatibilityVersion | | | setIndexCommitQuorum | | | setClusterParameter | | | setParameter | | | setDefaultRWConcern | | | shutdown | | ## Aggregation Commands | Command | Supported | | --------- | --------- | | aggregate | ✔ | | count | ✔ | | distinct | ✔ | | mapReduce | | ## Authentication Commands | Command | Supported | | ------------ | --------- | | authenticate | ✔ | | getnonce | | | logout | | ## Diagnostic Commands | Command | Supported | | ----------------------- | --------- | | buildInfo | ✔ | | collStats | ✔ | | connPoolStats | | | connectionStatus | ✔ | | dataSize | | | dbHash | | | dbStats | ✔ | | explain | ✔ | | explain: executionStats | ✔ | | features | | | getCmdLineOpts | | | getLog | | | hostInfo | | | listCommands | | | ping | ✔ | | profile | | | serverStatus | | | shardConnPoolStats | ✔ | | top | | | validate | | | validateDBMetadata | | | whatsmyuri | | | driverOIDTest | | | \_isSelf | | | lockinfo | | | netstat | | ## Query and Write Operations | Operation | Supported | | ---------------------- | --------- | | delete | ✔ | | find | ✔ | | findAndModify | ✔ | | getLastError | | | getMore | ✔ | | getPrevError | | | insert | ✔ | | parallelCollectionScan | | | resetError | | | update | ✔ | | Change streams | | | GridFS | ✔ | ## Role Management Commands | Command | Supported | | ------------------------ | --------- | | createRole | | | dropRole | | | dropAllRolesFromDatabase | | | grantPrivilegesToRole | | | grantRolesToRole | | | invalidateUserCache | | | revokePrivilegesFromRole | | | revokeRolesFromRole | | | updateRole | | | rolesInfo | | ## Sharding Commands | Command | Supported | | ---------------------------- | --------- | | abortReshardCollection | | | addShard | | | addShardToZone | | | balancerCollectionStatus | | | balancerStart | | | balancerStatus | | | balancerStop | | | checkShardingIndex | | | clearJumboFlag | | | cleanupOrphaned | | | cleanupReshardCollection | | | commitReshardCollection | | | configureCollectionBalancing | | | enableSharding | | | flushRouterConfig | | | getShardMap | | | getShardVersion | | | isdbgrid | | | listShards | | | medianKey | | | moveChunk | | | movePrimary | | | moveRange | | | mergeChunks | | | refineCollectionShardKey | | | removeShard | | | removeShardFromZone | | | reshardCollection | | | setAllowMigrations | | | setShardVersion | | | shardCollection | ✔ | | shardingState | | | split | | | splitVector | | | unsetSharding | | | updateZoneKeyRange | | ## Session Commands | Command | Supported | | ------------------------ | --------- | | abortTransaction | ✔ | | commitTransaction | ✔ | | endSessions | ✔ | | killAllSessions | \[empty] | | killAllSessionsByPattern | \[empty] | | killSessions | \[empty] | | refreshSessions | \[empty] | | startSession | \[empty] | ## User Management Commands | Command | Supported | | ------------------------ | --------- | | createUser | \[empty] | | dropAllUsersFromDatabase | \[empty] | | dropUser | \[empty] | | grantRolesToUser | \[empty] | | revokeRolesFromUser | \[empty] | | updateUser | \[empty] | | userInfo | \[empty] | *** Modified at: January 30, 2026 Source: [/cloud/reference/singlestore-kai/supported-mongodb-commands/](https://docs.singlestore.com/cloud/reference/singlestore-kai/supported-mongodb-commands/) (An index of the documentation is available at /llms.txt) --- # SQL Reference ## In this section * [SQL Command Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/sql-command-syntax.md) * [SQL Commands List](https://docs.singlestore.com/cloud/reference/sql-reference/sql-commands-list.md) * [SQL Functions List](https://docs.singlestore.com/cloud/reference/sql-reference/sql-functions-list.md) * [Comparison Operators and Functions](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions.md) * [Database Object Case-Sensitivity](https://docs.singlestore.com/cloud/reference/sql-reference/database-object-case-sensitivity.md) * [Data Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types.md) * [Character Encoding](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding.md) * [Operations that Take Either a Database or a Cluster Level Lock](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock.md) * [User-Defined Variables](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables.md) * [Restricted Keywords](https://docs.singlestore.com/cloud/reference/sql-reference/restricted-keywords.md) * [Aggregate Functions](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions.md) * [BSON Functions](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions.md) * [Cluster Management Commands](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands.md) * [Code Generation Functions](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions.md) * [Conditional Functions](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions.md) * [Data Definition Language DDL](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl.md) * [Data Manipulation Language DML](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml.md) * [Date and Time Functions](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions.md) * [Distinct Count Estimation Functions](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions.md) * [Full-Text Search Functions](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions.md) * [Geospatial Functions](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions.md) * [Identifier Generation Functions](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions.md) * [Information Functions](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions.md) * [JSON Functions](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions.md) * [Numeric Functions](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions.md) * [Operational Commands](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands.md) * [Pipelines Commands](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands.md) * [Procedural SQL Reference](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference.md) * [Regular Expression Functions](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions.md) * [Resource Pool Commands](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands.md) * [Security Management Commands](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands.md) * [SHOW Commands](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands.md) * [String Functions](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions.md) * [Time Series Functions](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions.md) * [Trace Event Commands](https://docs.singlestore.com/cloud/reference/sql-reference/trace-event-commands.md) * [Vector Functions](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions.md) * [Window Functions](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions.md) *** Modified at: July 23, 2025 Source: [/cloud/reference/sql-reference/](https://docs.singlestore.com/cloud/reference/sql-reference/) (An index of the documentation is available at /llms.txt) --- # Aggregate Functions ## In this section * [ANY\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/any-value.md) * [APPROX\_PERCENTILE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/approx-percentile.md) * [AVG](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/avg.md) * [BIT\_AND](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-and.md) * [BIT\_OR](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-or.md) * [BIT\_XOR](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-xor.md) * [COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/count.md) * [GROUP\_CONCAT](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/group-concat.md) * [MAX](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/max.md) * [MEDIAN](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/median.md) * [MIN](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/min.md) * [MOD](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/mod.md) * [STD](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/std.md) * [STDDEV](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev.md) * [STDDEV\_POP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev-pop.md) * [STDDEV\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev-samp.md) * [SUM](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/sum.md) * [VARIANCE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/variance.md) * [VAR\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/var-samp.md) *** Modified at: March 18, 2024 Source: [/cloud/reference/sql-reference/aggregate-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/) (An index of the documentation is available at /llms.txt) --- # BSON Functions ## In this section * [BSON\_ARRAY\_CONTAINS\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-contains-bson.md) * [BSON\_ARRAY\_PUSH](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-push.md) * [BSON\_ARRAY\_SLICE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-slice.md) * [BSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-build-array.md) * [BSON\_BUILD\_OBJECT](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-build-object.md) * [BSON\_COMPARE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-compare.md) * [BSON\_EXTRACT\_ALL\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-all-bson.md) * [BSON\_EXTRACT\_BIGINT](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bigint.md) * [BSON\_EXTRACT\_BINARY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-binary.md) * [BSON\_EXTRACT\_BOOL](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bool.md) * [BSON\_EXTRACT\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bson.md) * [BSON\_EXTRACT\_DATETIME](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-datetime.md) * [BSON\_EXTRACT\_DOUBLE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-double.md) * [BSON\_EXTRACT\_STRING](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-string.md) * [BSON\_GET\_TYPE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-get-type.md) * [BSON\_INCLUDE\_MASK and BSON\_EXCLUDE\_MASK](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-include-mask.md) * [BSON\_INPUT\_STREAM\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-input-stream-to-array.md) * [BSON\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-length.md) * [BSON\_MATCH\_ANY and BSON\_MATCH\_ANY\_EXISTS](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-match-any-and-bson-match-any-exists.md) * [BSON\_MERGE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-merge.md) * [BSON\_NORMALIZE\_\*](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-normalize.md) * [BSON\_SET\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-set-bson.md) * [BSON\_UNWIND](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-unwind.md) * [BSON\_UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-update.md) * [IS\_BSON\_NULL](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/is-bson-null.md) * [Multi-Value Hash Index (BSON)](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/multi-value-hash-index-bson.md) *** Modified at: July 9, 2025 Source: [/cloud/reference/sql-reference/bson-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/) (An index of the documentation is available at /llms.txt) --- # Character Encoding ## Working with Character Sets and Collations A character set is a collection of symbols and their encodings. A collation defines the rules for comparing and sorting the characters in a character set. SingleStore supports a variety of character sets and each character set can have multiple collations. By default, the character set and collation are set to `utf8mb4` and `utf8mb4_bin`, respectively, across the cluster. You can override the default values, as explained in [Specifying Character Set and Collation for Clusters](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/specifying-character-set-and-collation-for-clusters.md). ## Unicode Support SingleStore supports the Unicode standard that includes the characters in the Basic Multilingual Plane (BMP) and the supplementary characters that lie outside the BMP. The first 65536 Unicode characters in the BMP, whose code points range from U+0000 to U+FFFF, are encoded in variable length from 1 to 3 bytes per character. The supplementary characters, whose code points range from U+10000 to U+10FFFF, are encoded in lengths of 4 bytes per character. With the 4-byte character encoding (`utf8mb4`), SingleStore supports all the characters in the BMP and supplementary characters that lie outside the BMP, including the private use area (PUA) which can contain the encoding of pictographic symbols (emojis) and ancient scripts, such as Egyptian hieroglyphs. ## In this section * [Character Sets Supported](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/character-sets-supported.md) * [Collations Supported](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/collations-supported.md) * [Specifying Character Set and Collation for Clusters](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/specifying-character-set-and-collation-for-clusters.md) * [Character Set and Collation Override](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/character-set-and-collation-override.md) * [Migrating and Troubleshooting Collations](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/migrating-and-troubleshooting-collations.md) * [Implicit Collation in Special Cases](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/special-cases.md) *** Modified at: July 13, 2026 Source: [/cloud/reference/sql-reference/character-encoding/](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/) (An index of the documentation is available at /llms.txt) --- # Cluster Management Commands ## Node Requirements for SingleStore Helios Commands SingleStore Helios commands must be run on the appropriate type(s) of node in a SingleStore Helios cluster. When `sync_permissions` and `enable_query_forwarding` are set to `ON`, all Data Definition Language (DDL) commands can be forwarded from child aggregator to master. This list of requirements reflects this behavior. * X = recommended to run on this type of node * P = possible (but not recommended) to run on this type of node * blank = cannot run on this type of node | SingleStore HeliosCommand | Master Aggregator | Child Aggregator | Leaf | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ---------------- | ---- | | [ANALYZE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md) | X | X | | | [ALTER DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-database.md) | X | X | | | [ALTER RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/alter-resource-pool.md) | X | X | | | [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) | X | X | P | | [ALTER VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-view.md) | X | X | P | | [ALTER USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/alter-user.md) | X | X | | | [ALTER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/alter-pipeline.md) | X | X | | | [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) | X | X | | | [BEGIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/begin.md) | X | X | P | | [COMMIT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/commit.md) | X | X | P | | [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md) | X | X | | | [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) | X | X | P | | [CREATE FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) | X | X | | | [CREATE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-group.md) | X | X | | | [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) | X | X | P | | [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) | X | X | | | [CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-procedure.md) | X | X | | | [CREATE RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/create-resource-pool.md) | X | X | | | [CREATE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-role.md) | X | X | | | [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) | X | X | P | | [CREATE USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) | X | X | | | [CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md) | X | X | | | [DELETE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/delete.md) | X | X | P | | [DESCRIBE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/describe.md) | X | X | X | | [DROP AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-aggregate.md) | X | X | | | [DROP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-database.md) | X | X | P | | [DROP FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-function.md) | X | X | | | [DROP GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-group.md) | X | X | | | [DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-index.md) | X | X | P | | [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) | X | X | | | [DROP PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-procedure.md) | X | X | | | [DROP RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/drop-resource-pool.md) | X | X | | | [DROP ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-role.md) | X | X | | | [DROP TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-table.md) | X | X | P | | [DROP USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-user.md) | X | X | X | | [DROP VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-view.md) | X | X | P | | [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md) | X | X | P | | [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) | X | X | X | | [FLUSH TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/flush-tables.md) | X | | P | | [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) | X | X | X | | [GRANT GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-group.md) | X | X | X | | [GRANT ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-role.md) | X | X | X | | [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) | X | X | P | | [KILL CONNECTION](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/kill-connection-and-kill-query.md) | X | X | P | | [KILLALL QUERIES](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/killall-queries.md) | X | X | P | | [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) | X | X | P | | [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) | X | X | P | | [PROFILE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/profile-pipeline.md) | X | X | | | [REPAIR DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/repair-database.md) | X | X | X | | [REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/replace.md) | X | X | P | | [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) | X | X | | | [ROLLBACK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/rollback.md) | X | X | P | | [REVOKE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke.md) | X | X | | | [REVOKE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-group.md) | X | X | | | [REVOKE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-role.md) | X | X | | | [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) | X | X | P | | [SHOW COLUMNS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-columns.md) | X | X | | | [SHOW CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-table.md) | X | X | P | | [SHOW DATABASES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-databases.md) | X | X | P | | [SHOW GRANTS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-grants.md) | X | X | X | | [SHOW INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) | X | X | P | | [SHOW INDEXES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) | X | X | P | | [SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) | X | X | | | [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) | X | X | X | | [SHOW PROCESSLIST](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-processlist.md) | X | X | X | | [SHOW REPLICATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/show-replication-status.md) | X | X | X | | [SHOW SCHEMAS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-schemas.md) | X | X | P | | [SHOW GLOBAL STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-global-status.md) | X | X | X | | [SHOW GLOBAL VARIABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-variables.md) | X | X | X | | [SHOW RESOURCE POOLS](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/show-resource-pools.md) | X | X | P | | [SHOW SESSION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-session-status.md) | X | X | X | | [SHOW SESSION VARIABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-variables.md) | X | X | X | | [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md) | X | X | X | | [SHOW TABLE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-table-status.md) | X | X | X | | [SHOW TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-tables.md) | X | X | X | | [SHOW VARIABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-variables.md) | X | X | X | | [SHOW WARNINGS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-warnings.md) | X | X | X | | [SHOW ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-errors.md) | X | X | X | | [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) | X | X | | | [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) | X | X | | | [TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/truncate.md) | X | X | | | [UNLOCK TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/unlock-tables.md) | X | | | | [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) | X | X | P | | [USE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/use.md) | X | X | P | ## Leaf States Each leaf is in one of the following states: | State | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Unknown` | In this state, a leaf is not part of thecluster. | | `Online` | This is the default, healthy state of a leaf. In the`online`state, the leaf is an active member of the distributed system and is either currently serving or ready to serve data to the aggregators. | | `Offline` | The master aggregator periodically sends a heartbeat (ping) to all the nodes in aclusterto determine if they are responsive and online. A leaf enters the`offline`state if the master aggregator cannot reach it. The heartbeat frequency is based on typical network latencies and node responsiveness. It is set to a default of 150ms which has been empirically determined.If theclusteris in redundancy 2 any partitions on the`offline`node will be failed over to the leaves pair. The master aggregator continues to ping`offline`nodes to detect when they should be moved into the`attaching`state. | | `Recovering` | Not online, and not available for read or write queries. A leaf in`recovering`state has been restarted and is replaying data back into memory. | | `Detached` | In this state, the leaf is detached from thecluster. | | `Attaching` | A leaf transitions from`offline`to`attaching`when it is once again reachable by heartbeats. | The following diagram summarizes the leaf states and the transitions between them. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt4f2b0cf749e87065/6a7adb4e3ce8e2aba5cea134/16083b57986b1b-2Mg2on.png) ## In this section * [EXPLAIN RESTORE REDUNDANCY](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/explain-restore-redundancy.md) * [FLUSH EXTRACTOR POOLS](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/flush-extractor-pools.md) * [FLUSH HOSTS](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/flush-hosts.md) * [SHOW AGGREGATORS](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/show-aggregators.md) *** Modified at: November 18, 2025 Source: [/cloud/reference/sql-reference/cluster-management-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/cluster-management-commands/) (An index of the documentation is available at /llms.txt) --- # Code Generation Functions This page contains a list of all code generation commands in SingleStore. ## In this section * [NOPARAM](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/noparam.md) * [\_SYNC\_PLANS](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/sync-plans.md) *** Modified at: May 29, 2026 Source: [/cloud/reference/sql-reference/code-generation-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/) (An index of the documentation is available at /llms.txt) --- # Comparison Operators and Functions The following comparison operators and functions are supported. | Name | Description | | ------------------------- | ---------------------------------------------------------------- | | `BETWEEN ... AND ...` | Check if a value is within a range | | `NOT BETWEEN ... AND ...` | Check if a value is not within a range | | `=` | Equal operator | | `<=>` | NULL-safe equal operator | | `>` | Greater than operator | | `>=` | Greater than or equal operator | | `<` | Less than operator | | `<=` | Less than or equal operator | | `IN()` | Check whether a value is in a comma-separated list of values | | `NOT IN()` | Check whether a value is not in a comma-separated list of values | | `IS` | Compare a value to a boolean | | `IS NOT` | Compare a value to the complement of a boolean | | `IS NULL` | Check if a value is NULL | | `IS NOT NULL` | Check if a value is not NULL | | `ISNULL()` | Test whether the argument is NULL | | `LIKE` | Check if a value matches a simple pattern | | `NOT LIKE` | Check if a value does not match a simple pattern | | `!=, <>` | Not equal operator | | `strcmp()` | Compare two strings and return -1, 0, or 1 | ## In this section * [BETWEEN, NOT BETWEEN](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/between-not-between.md) * [NULL Handling](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/null-handling.md) * [LIKE, NOT LIKE](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/like-not-like.md) * [IN(), NOT IN()](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/in-not-in.md) * [STRCMP()](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/strcmp.md) * [Where operators can be used](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/where-operators-can-be-used.md) * [Using Comparison Operators with Date and Time Functions](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/using-comparison-operators-with-date-and-time-functions.md) *** Modified at: April 24, 2021 Source: [/cloud/reference/sql-reference/comparison-operators-and-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/) (An index of the documentation is available at /llms.txt) --- # Conditional Functions ## In this section * [CASE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/case.md) * [COALESCE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/coalesce.md) * [CAST or CONVERT](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/cast-or-convert.md) * [DECODE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/decode.md) * [ELT](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/elt.md) * [GREATEST](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/greatest.md) * [IF](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/if.md) * [ISNULL](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/isnull.md) * [ISNUMERIC](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/isnumeric.md) * [LEAST](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/least.md) * [NULLIF](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/nullif.md) * [NVL and IFNULL](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/nvl-and-ifnull.md) * [SLEEP](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/sleep.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/conditional-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/) (An index of the documentation is available at /llms.txt) --- # Data Definition Language DDL ## In this section * [ALTER DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-database.md) * [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) * [ALTER VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-view.md) * [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) * [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) * [CREATE MILESTONE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-milestone.md) * [CREATE PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-projection.md) * [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) * [CREATE TABLE with an AUTO\_INCREMENT Column](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table-with-an-auto-increment-column.md) * [CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md) * [DESCRIBE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/describe.md) * [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md) * [DROP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-database.md) * [DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-index.md) * [DROP PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-projection.md) * [DROP TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-table.md) * [DROP VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-view.md) * [FLUSH TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/flush-tables.md) * [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) * [TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/truncate.md) * [UNLOCK TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/unlock-tables.md) *** Modified at: May 11, 2026 Source: [/cloud/reference/sql-reference/data-definition-language-ddl/](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/) (An index of the documentation is available at /llms.txt) --- # Data Manipulation Language DML ## In this section * [AGGREGATOR SYNC AUTO\_INCREMENT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/aggregator-sync-auto-increment.md) * [BEGIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/begin.md) * [CLEAR LOAD ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/clear-load-errors.md) * [COMMIT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/commit.md) * [CUBE and ROLLUP](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/cube-and-rollup.md) * [DELETE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/delete.md) * [EXCEPT and MINUS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/except-and-minus.md) * [EXISTS AND NOT EXISTS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/exists-and-not-exists.md) * [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) * [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) * [INTERSECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/intersect.md) * [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) * [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) * [PIVOT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/pivot.md) * [REOPTIMIZE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize.md) * [REOPTIMIZE MARK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize-mark.md) * [REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/replace.md) * [ROLLBACK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/rollback.md) * [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) * [TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/table.md) * [UNION](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/union.md) * [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) * [USE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/use.md) *** Modified at: February 18, 2026 Source: [/cloud/reference/sql-reference/data-manipulation-language-dml/](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/) (An index of the documentation is available at /llms.txt) --- # Data Types SingleStore Helios supports the following data types. The sizes given are for in-memory rowstore tables. Columnstore (on-disk) tables are stored differently, and often compressed. The sections below provide in-depth information about various data types as well as data type conversion and comparing mismatched data types. ## In this section * [Integer Numbers](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/integer-numbers.md) * [Real Numbers](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/real-numbers.md) * [Time and Date](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/time-and-date.md) * [String Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/string-types.md) * [Encoding String Data in Columnstore Tables](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/encoding-string-data-in-columnstore-tables.md) * [Binary String Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/binary-string-types.md) * [JSON Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/json-type.md) * [BSON Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/bson-type.md) * [Geospatial Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/geospatial-types.md) * [SingleStore Helios Procedural SQL (PSQL) Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/singlestore-helios-procedural-sql-psql-types.md) * [Other Types](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/other-types.md) * [Vector Type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md) * [Comparing Mismatched Datatypes](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/comparing-mismatched-datatypes.md) * [Data Type Conversion](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/data-type-conversion.md) *** Modified at: March 5, 2026 Source: [/cloud/reference/sql-reference/data-types/](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/) (An index of the documentation is available at /llms.txt) --- # Database Object Case-Sensitivity Database objects allow you to store, reference, and operate on data. For example, tables store data, views and indexes reference data, and stored procedures and functions operate on data. Columns are not database objects. > **📝 Note**: You can only change the setting of the variable `table_name_case_sensitivity` when the cluster is empty. That is, the cluster must contain no user databases. The engine variable `table_name_case_sensitivity` defines the case-sensitivity of a database object. When this variable is set to `ON` (the default setting), all database objects are case-sensitive, except: * Stored procedures * User-defined scalar-valued functions (UDFs) * User-defined aggregate functions (UDAFs) * `information_schema` table names When the variable is set to `OFF`, the four database objects noted in the previous list are case-insensitive, in addition to the following database objects. * Tables * Views * Table aliases * User-defined table-valued functions (TVFs) * External functions Pipeline names are always case-sensitive. The following example shows how you can refer to an existing table `test_table`. This example assumes `table_name_case_sensitivity` is set to `OFF`. ```sql SELECT @@table_name_case_sensitivity; ``` ```output +-------------------------------+ | @@table_name_case_sensitivity | +-------------------------------+ | 0 | +-------------------------------+ 1 row in set (0.04 sec) ``` Please contact [SingleStore support](https://support.singlestore.com) if you wish to change the value of `table_name_case_sensitivity` on your SingleStore Helios cluster. ```sql INSERT INTO Test_Table(a) VALUES (10); ``` ```sql SELECT * FROM TEST_Table; ``` When you create a database object which is not case-sensitive, you must use a unique and case-independent name. For example, when `table_name_case_sensitivity` is set to `OFF`, running the following two commands results in an error: ```sql CREATE TABLE test_table_2(a INT); ``` ```sql CREATE TABLE test_TABLE_2(a INT); ``` The following example illustrates that column names are not subject to case-sensitivity even when `table_name_case_sensitivity` is set to `ON`: ```sql select @@table_name_case_sensitivity; +-------------------------------+ | @@table_name_case_sensitivity | +-------------------------------+ | 1 | +-------------------------------+ ``` ```sql CREATE TABLE test_table_3 (a int, b INT); ``` ```sql INSERT INTO test_table_3 (a, b) VALUES (1, 2); ``` ```sql SELECT a, b FROM test_table_3; ``` ```output +------+------+ | a | b | +------+------+ | 1 | 2 | +------+------+ ``` ```sql SELECT A, B FROM test_table_3; ``` ```output +------+------+ | A | B | +------+------+ | 1 | 2 | +------+------+ ``` *** Modified at: April 11, 2024 Source: [/cloud/reference/sql-reference/database-object-case-sensitivity/](https://docs.singlestore.com/cloud/reference/sql-reference/database-object-case-sensitivity/) (An index of the documentation is available at /llms.txt) --- # Date and Time Functions ## In this section * [ADDTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/addtime.md) * [CONVERT\_TZ](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/convert-tz.md) * [CURRENT\_DATE And CURDATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-date-and-curdate.md) * [CURRENT\_TIME And CURTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-time-and-curtime.md) * [CURRENT\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-timestamp.md) * [DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date.md) * [DATEDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/datediff.md) * [DATE\_ADD](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-add.md) * [DATE\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-format.md) * [DATE\_SUB](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-sub.md) * [DATE\_TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-trunc.md) * [DAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/day.md) * [DAYNAME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayname.md) * [DAYOFWEEK](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayofweek.md) * [DAYOFYEAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayofyear.md) * [EXTRACT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/extract.md) * [FROM\_DAYS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/from-days.md) * [FROM\_UNIXTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/from-unixtime.md) * [GET\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/get-format.md) * [HOUR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/hour.md) * [LAST\_DAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/last-day.md) * [LOCALTIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/localtimestamp.md) * [MICROSECOND](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/microsecond.md) * [MINUTE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/minute.md) * [MONTH](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/month.md) * [MONTHNAME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/monthname.md) * [MONTHS\_BETWEEN](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/months-between.md) * [NOW](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/now.md) * [QUARTER](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/quarter.md) * [SECOND](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/second.md) * [SEC\_TO\_TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/sec-to-time.md) * [STR\_TO\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/str-to-date.md) * [TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time.md) * [TIMEDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timediff.md) * [TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestamp.md) * [TIMESTAMPADD](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestampadd.md) * [TIMESTAMPDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestampdiff.md) * [TIME\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time-format.md) * [TIME\_TO\_SEC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time-to-sec.md) * [TO\_CHAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-char.md) * [TO\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-date.md) * [TO\_DAYS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-days.md) * [TO\_SECONDS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-seconds.md) * [TO\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-timestamp.md) * [TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/trunc.md) * [UNIX\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/unix-timestamp.md) * [UTC\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-date.md) * [UTC\_TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-time.md) * [UTC\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-timestamp.md) * [WEEK](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/week.md) * [WEEKDAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/weekday.md) * [YEAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/year.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/date-and-time-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/) (An index of the documentation is available at /llms.txt) --- # Distinct Count Estimation Functions ## Using Distinct Count Estimation Functions Imagine that you work at an airline that is marketing air travel to families. As a data analyst, you would like to count the number of families who have booked flights. In your total, you count a family if two or more passengers belonging to that family have booked a flight on the airline. The passengers do not need to be traveling on the same flight. Also, if a family has booked more than one flight, you count that family only once. You run your calculation on the `passenger_booking` table, which has the columns `flight_number`, `flight_date`, `passenger_id` and `family_id`. Passengers who belong to the same family share the same `family_id`. The table does not include bookings for passengers traveling alone. To find the unique family total, you run the query `SELECT COUNT(DISTINCT family_id) AS result FROM passenger_booking`. The code below creates the `passenger_booking` table and populates it with sample data. ```sql DELIMITER // CREATE PROCEDURE populate_booking_table(tbl VARCHAR(40), number_of_records INT, _flight_date DATE, _family_id INT) AS BEGIN FOR i IN 1..number_of_records LOOP /* This example doesn't populate the booking table with a flight number or a passenger ID; these values aren't needed to demonstrate the distinct count estimate. */ INSERT INTO passenger_booking (flight_date, family_id) VALUES (_flight_date, _family_id); END LOOP; END // DELIMITER ; /* The following procedure creates the passenger_booking table and populates it with sample values for four days. The procedure intentionally populates family_id with duplicate values to demonstrate the distinct count estimate later. */ DELIMITER // CREATE PROCEDURE create_sample_data() AS BEGIN CREATE TABLE passenger_booking (flight_number INT, flight_date DATE, passenger_id INT, family_id INT); FOR family_id IN 1..10000 LOOP CALL populate_booking_table('passenger_booking', 1, TO_DATE('01/01/2018','MM/DD/YYYY'), family_id); END LOOP; FOR family_id IN 5001..10000 LOOP CALL populate_booking_table('passenger_booking', 2, TO_DATE('01/01/2018','MM/DD/YYYY'), family_id); END LOOP; FOR family_id IN 8001..18000 LOOP CALL populate_booking_table('passenger_booking', 2, TO_DATE('01/02/2018','MM/DD/YYYY'), family_id); END LOOP; FOR family_id IN 15001..35000 LOOP CALL populate_booking_table('passenger_booking', 3, TO_DATE('01/03/2018','MM/DD/YYYY'), family_id); END LOOP; FOR family_id IN 30001..50000 LOOP CALL populate_booking_table('passenger_booking', 2, TO_DATE('01/04/2018','MM/DD/YYYY'), family_id); END LOOP; END // DELIMITER ; CALL create_sample_data(); ``` Find `family_id`’s distinct count estimate: ```sql SELECT COUNT(DISTINCT family_id) AS result FROM passenger_booking; ``` ```output +--------+ | result | +--------+ | 50000 | +--------+ ``` ## Getting a Distinct Count Estimate (Method 1) Calculating the distinct count of a large column can be time-consuming and memory-intensive. To reduce the cost of this calculation, you run an alternative function that gets an estimate of `family_id`’s distinct count: ```sql SELECT APPROX_COUNT_DISTINCT(family_id) AS result FROM passenger_booking; ``` ```output +--------+ | result | +--------+ | 49933 | +--------+ ``` `APPROX_COUNT_DISTINCT` is implemented using the [HyperLogLog](http://en.wikipedia.org/wiki/HyperLogLog) algorithm. As shown in the following table, `family_id`’s estimate calculation runs faster and consumes less memory, as compared to the exact calculation. | Query | Execution Time (After query compilation) | Memory Use (After query compilation) | | ---------------------------------------------------------------- | ---------------------------------------- | ------------------------------------ | | `SELECT COUNT(DISTINCT family_id) FROM passenger_booking` | `32` | `775614` | | `SELECT APPROX_COUNT_DISTINCT(family_id) FROM passenger_booking` | `9` | `128453` | ## Getting a Distinct Count Estimate (Method 2) Suppose that every day, you want to estimate the number of unique `family_id`s for flights booked up to and including that date. Rather than calling `APPROX_COUNT_DISTINCT` to recalculate the count for all `family_id` values in the `passenger_booking` table, you call the functions `APPROX_COUNT_DISTINCT_ACCUMULATE` and `APPROX_COUNT_DISTINCT_ESTIMATE`. These are low-level functions that allow you to obtain incremental estimates, consuming less time and memory than `APPROX_COUNT_DISTINCT`. In general, you work with the low-level functions as follows: You call `APPROX_COUNT_DISTINCT_ACCUMULATE` multiple times. On each invocation, you provide a data subset to summarize into a state. You then call `APPROX_COUNT_DISTINCT_ESTIMATE` to provide an estimate from the collection of input states. The following code snippets illustrate how to use the low-level functions to generate incremental `family_id` estimates. First, create a state table that the low-level functions will use. ```sql CREATE TABLE distinct_family_day_states (flight_date DATE, state VARBINARY(16384)); ``` > **❗ Important**: Use the data type `VARBINARY(16384)` to store states that you use with the low-level `APPROX_COUNT_DISTINCT` functions. At the end of the first day, create a state for that day and insert it into the table containing the states: ```sql INSERT INTO distinct_family_day_states(flight_date, state) SELECT flight_date, APPROX_COUNT_DISTINCT_ACCUMULATE(family_id) FROM passenger_booking WHERE flight_date = TO_DATE('01/01/2018', 'MM/DD/YYYY') GROUP BY flight_date; ``` Then get the distinct family estimate for the first day: ```sql SELECT APPROX_COUNT_DISTINCT_ESTIMATE(state) FROM distinct_family_day_states; ``` ```output +----------------------------------------+ | APPROX_COUNT_DISTINCT_ESTIMATE(state) | +----------------------------------------+ | 10009 | +----------------------------------------+ ``` At the end of the second day, create a state for that day and insert it into the table containing the states: ```sql INSERT INTO distinct_family_day_states(flight_date, state) SELECT flight_date, APPROX_COUNT_DISTINCT_ACCUMULATE(family_id) FROM passenger_booking WHERE flight_date = TO_DATE('01/02/2018', 'MM/DD/YYYY') GROUP BY flight_date; ``` Then get the distinct family estimate for the first and second day: ```sql SELECT APPROX_COUNT_DISTINCT_ESTIMATE(state) FROM distinct_family_day_states; ``` ```output +----------------------------------------+ | APPROX_COUNT_DISTINCT_ESTIMATE(state) | +----------------------------------------+ | 18227 | +----------------------------------------+ ``` To find the distinct family estimate for the third and fourth day, repeat the steps above. The results are shown in the table below. | Distinct family estimate for the third day | `34856` | | ------------------------------------------- | ------- | | Distinct family estimate for the fourth day | `49933` | ## Getting a Distinct Count Estimate (Method 3): Using `APPROX_COUNT_DISTINCT_COMBINE` In the [previous section](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions.md), you found a daily rolling estimate of the number of families who booked flights. Imagine that you want to make this estimate every day for multiple years. You can optimize this calculation by taking the following steps, in order, at the end of every month: * Call `APPROX_COUNT_DISTINCT_COMBINE` to merge the daily states into a monthly state. Insert the monthly state into a monthly states table. The code below shows how to create the monthly state for January 2018. Note that a `WHERE` clause is not needed in the `SELECT` statement; the daily states table contains data for one month only. ```sql INSERT INTO distinct_family_month_states SELECT 1, 2018, APPROX_COUNT_DISTINCT_COMBINE(state) FROM distinct_family_day_states; ``` * Truncate the `distinct_family_day_states` table. Prior to truncating, the table should not contain daily states for any other month. At the end of each day, insert the states from the `distinct_family_day_states` and `distinct_family_month_states` tables into a new temporary result set. Find the distinct family count estimate by running `APPROX_COUNT_DISTINCT_ESTIMATE` on the states in the result set: ```sql WITH all_states AS ( SELECT state FROM distinct_family_day_states UNION ALL SELECT state FROM distinct_family_month_states ) SELECT APPROX_COUNT_DISTINCT_ESTIMATE(state) FROM all_states; ``` ## In this section * [APPROX\_COUNT\_DISTINCT](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct.md) * [APPROX\_COUNT\_DISTINCT\_ACCUMULATE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-accumulate.md) * [APPROX\_COUNT\_DISTINCT\_COMBINE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-combine.md) * [APPROX\_COUNT\_DISTINCT\_ESTIMATE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-estimate.md) *** Modified at: July 29, 2024 Source: [/cloud/reference/sql-reference/distinct-count-estimation-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/) (An index of the documentation is available at /llms.txt) --- # Full-Text Search Functions The following Full-Text Search functions are supported in SingleStore: * [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) * [HIGHLIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/highlight.md) * [MATCH](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) See [Full Text Search](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) for more conceptual information on this feature. ## In this section * [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) * [HIGHLIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/highlight.md) * [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) *** Modified at: January 26, 2026 Source: [/cloud/reference/sql-reference/full-text-search-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/) (An index of the documentation is available at /llms.txt) --- # Geospatial Functions For more information about working with geospatial data, see [Working with Geospatial Features](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-geospatial-features.md). ## In this section * [APPROX\_GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/approx-geography-intersects.md) * [GEOGRAPHY\_AREA](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-area.md) * [GEOGRAPHY\_CONTAINS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-contains.md) * [GEOGRAPHY\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-distance.md) * [GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-intersects.md) * [GEOGRAPHY\_LATITUDE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-latitude.md) * [GEOGRAPHY\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-length.md) * [GEOGRAPHY\_LONGITUDE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-longitude.md) * [GEOGRAPHY\_POINT](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-point.md) * [GEOGRAPHY\_WITHIN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-within-distance.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/geospatial-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/) (An index of the documentation is available at /llms.txt) --- # Identifier Generation Functions SingleStore supports the following identifier generation functions: * SYS\_GUID and UUID * UUID\_TO\_BIN * BIN\_TO\_UUID * IS\_UUID ## In this section * [SYS\_GUID and UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/sys-guid-and-uuid.md) * [UUID\_TO\_BIN](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/uuid-to-bin.md) * [BIN\_TO\_UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/bin-to-uuid.md) * [IS\_UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/is-uuid.md) *** Modified at: November 7, 2024 Source: [/cloud/reference/sql-reference/identifier-generation-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/) (An index of the documentation is available at /llms.txt) --- # Information Functions ## In this section * [AGGREGATOR\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/aggregator-id.md) * [CONNECTION\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/connection-id.md) * [DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/database.md) * [FOUND\_ROWS](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/found-rows.md) * [LAST\_INSERT\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/last-insert-id.md) * [ROW\_COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/row-count.md) * [SEGMENT\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/segment-id.md) * [USER](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/user.md) *** Modified at: December 11, 2025 Source: [/cloud/reference/sql-reference/information-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/) (An index of the documentation is available at /llms.txt) --- # JSON Functions ## In this section * [JSON\_AGG](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-agg.md) * [JSON\_ARRAY\_CONTAINS\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-array-contains-type.md) * [JSON\_ARRAY\_PUSH\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-array-push-type.md) * [JSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-array.md) * [JSON\_BUILD\_OBJECT](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-object.md) * [JSON\_DELETE\_KEY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-delete-key.md) * [JSON\_EXTRACT\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md) * [JSON\_GET\_TYPE](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-get-type.md) * [JSON\_INCLUDE/EXCLUDE\_MASK](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-include-exclude-mask.md) * [JSON Keypaths](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-keypaths.md) * [JSON\_KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-keys.md) * [JSON\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-length.md) * [JSON\_MATCH\_ANY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) * [JSON\_MERGE\_PATCH](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-merge-patch.md) * [JSON\_PRETTY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-pretty.md) * [JSON\_SET\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-set-type.md) * [JSON\_SPLICE\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-splice-type.md) * [JSON\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-to-array.md) * [REDUCE](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/reduce.md) * [TO\_JSON](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/to-json.md) *** Modified at: January 7, 2025 Source: [/cloud/reference/sql-reference/json-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/) (An index of the documentation is available at /llms.txt) --- # Numeric Functions ## In this section * [ABS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/abs.md) * [SIGN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sign.md) * [EXP](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/exp.md) * [ACOS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/acos.md) * [ASIN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/asin.md) * [ATAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/atan.md) * [ATAN2, ATAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/atan-2-atan.md) * [BIT\_COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bit-count.md) * [bitwise AND (&)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-and.md) * [bitwise NOT (\~)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-not.md) * [bitwise OR (|)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-or.md) * [bitwise XOR (^)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-xor.md) * [bitwise left shift (<<)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-left-shift.md) * [bitwise right shift (>>)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-right-shift.md) * [CEIL](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/ceil.md) * [CONV](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/conv.md) * [COS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/cos.md) * [COT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/cot.md) * [LN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/ln.md) * [LOG2](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log-2.md) * [LOG10](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log-10.md) * [DEGREES](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/degrees.md) * [FLOOR](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/floor.md) * [LOG](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log.md) * [PI](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/pi.md) * [POW](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/pow.md) * [RADIANS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/radians.md) * [RAND](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/rand.md) * [ROUND](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/round.md) * [SIGMOID](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sigmoid.md) * [SIN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sin.md) * [SQRT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sqrt.md) * [TAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/tan.md) * [TO\_NUMBER](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/to-number.md) * [TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/trunc.md) * [TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/truncate.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/numeric-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/) (An index of the documentation is available at /llms.txt) --- # Operational Commands ## In this section * [ANALYZE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md) * [ANALYZE MEMORY](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze-memory.md) * [ATTACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database.md) * [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) * [CHECK BLOB CHECKSUM](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/check-blob-checksum.md) * [CHECK BOTTOMLESS](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/check-bottomless.md) * [DETACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/detach-database.md) * [FULLTEXT SERVICE STOP](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/fulltext-service-stop.md) * [KILL CONNECTION and KILL QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/kill-connection-and-kill-query.md) * [KILLALL QUERIES](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/killall-queries.md) * [REPAIR DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/repair-database.md) * [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) * [SELECT GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/select-global.md) * [SET and SET SESSION](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-and-set-session.md) * [SET AGGREGATOR](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-aggregator.md) * [SET CLUSTER](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-cluster.md) * [SET GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-global.md) * [SET LEAF](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-leaf.md) * [SHOW REPLICATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/show-replication-status.md) * [Replication Distribution Tree](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/replication-distribution-tree.md) * [SNAPSHOT DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/snapshot-database.md) * [SYNC BOTTOMLESS DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/sync-bottomless-database.md) *** Modified at: July 10, 2025 Source: [/cloud/reference/sql-reference/operational-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/) (An index of the documentation is available at /llms.txt) --- # Operations that Take Either a Database or a Cluster Level Lock Certain operations will take either a database level lock or a cluster level lock. This topic explains what these locks are, their impact, and the operations that use them. ## In this section * [Database Level Locks](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/database-level-locks.md) * [Cluster Level Locks](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/cluster-level-locks.md) * [Operations that Take Database Level Locks](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/operations-that-take-database-level-locks.md) * [Operations that Take Cluster Level Locks](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/operations-that-take-cluster-level-locks.md) *** Modified at: August 26, 2022 Source: [/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/](https://docs.singlestore.com/cloud/reference/sql-reference/operations-that-take-either-a-database-or-a-cluster-level-lock/) (An index of the documentation is available at /llms.txt) --- # Pipelines Commands ## In this section * [ALTER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/alter-pipeline.md) * [CLEAR PIPELINE ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/clear-pipeline-errors.md) * [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) * [CREATE PIPELINE ... INTO PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline-into-procedure.md) * [Additional CREATE PIPELINE Examples](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/additional-create-pipeline-examples.md) * [CREATE INFERRED PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-inferred-pipeline.md) * [DETACH PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/detach-pipeline.md) * [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) * [INFER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/infer-table.md) * [INFER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/infer-pipeline.md) * [PROFILE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/profile-pipeline.md) * [PROFILE PIPELINE JSON](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/profile-pipeline-json.md) * [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) * [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) * [TEST PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/test-pipeline.md) *** Modified at: November 14, 2025 Source: [/cloud/reference/sql-reference/pipelines-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/) (An index of the documentation is available at /llms.txt) --- # Procedural SQL Reference This section contains information about SingleStore Procedural SQL (PSQL), which encompasses stored procedures, user-defined scalar-value functions (UDFs), user-defined table-valued functions (TVFs), and user-defined aggregate functions (UDAFs). ## Wasm-based Functions SingleStore also supports Wasm-based functions. Refer to [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) for information on creating Wasm-based UDFs and TVFs. To create Wasm-based UDAFs, refer to [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md). If your logic and data structures can be implemented in PSQL with reasonable effort in a way that performs well, use PSQL. For integrating more complex logic and data structures, or using existing C/C++ or Rust code, SingleStore recommends using Wasm-based functions. For information on when to use PSQL, Wasm-based, or external functions, refer to [Choose Between PSQL, Wasm-based, and External Functions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/choose-between-psql-wasm-based-and-external-functions.md). ## Extensions Extensions in SingleStore allow you to combine user-defined objects, such as UDFs or UDAFs, into a packaged archive and then create, manage, and deploy these objects and other resources using the `CREATE EXTENSION` command. Extensions support both PSQL and Wasm-based functions. Refer to [Extensions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/extensions.md) for more information. ## Related Topics * [Procedural Extensions](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions.md) * [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md) ## In this section * [Anonymous Code Blocks](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/anonymous-code-blocks.md) * [Control Flow Statements](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/control-flow-statements.md) * [Exceptions and Exception Handling](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/exceptions-and-exception-handling.md) * [MySQL Client Delimiters](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/mysql-client-delimiters.md) * [Transactions in Stored Procedures](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/transactions-in-stored-procedures.md) * [Extensions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/extensions.md) * [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md) * [CREATE EXTERNAL FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-external-function.md) * [CREATE FUNCTION (TVF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md) * [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) * [CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-procedure.md) * [CREATE TEMPORARY PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-temporary-procedure.md) * [DROP FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-function.md) * [DROP PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-procedure.md) * [DROP AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-aggregate.md) * [SHOW AGGREGATES](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-aggregates.md) * [SHOW CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-aggregate.md) * [SHOW CREATE FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-function.md) * [SHOW CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-procedure.md) * [SHOW FUNCTIONS](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-functions.md) * [SHOW PROCEDURES](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-procedures.md) * [%ROWTYPE and %TYPE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/rowtype-and-type.md) * [ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/array.md) * [CALL](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/call.md) * [CREATE\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-array.md) * [COLLECT](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/collect.md) * [ECHO](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/echo.md) * [ECHO SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/echo-select.md) * [EXECUTE IMMEDIATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/execute-immediate.md) * [INSERT\_ALL](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/insert-all.md) * [RECORD](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/record.md) * [ROW](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/row.md) * [QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/query.md) * [SCALAR](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/scalar.md) * [TO\_QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/to-query.md) *** Modified at: March 5, 2026 Source: [/cloud/reference/sql-reference/procedural-sql-reference/](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/) (An index of the documentation is available at /llms.txt) --- # Regular Expression Functions ## Using Regular Expression Commands Suppose you want to find the starting position of a string within another string. Call the former the search string and the latter the target string. To find `dog` within `The dog chased the cat`, you would run the query `SELECT INSTR('The dog chased the cat','dog');`. This query returns `5`, indicating the position where it found `dog`. But [INSTR](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/instr.md) does not allow search strings that contain placeholders. The following examples show search strings containing placeholders, denoted by the `?` symbol. | Search String | Placeholder | | ---------------------- | --------------------------- | | `The dog chased the ?` | Any string | | `The ? chased the cat` | Any three letter string | | `The dog ? the cat` | The string`chased`or`found` | Regular expressions enable you to define search strings with placeholders. You specify these strings using patterns. For instance, the pattern `The .{3,5} chased the mouse|frog` matches many target strings. Two of them are `The dog chased the frog` and `The rat chased the mouse`. Patterns contain literals and metacharacters. Literals are characters that a pattern matcher attempts to locate in the target string exactly as the characters are written in the search string. Metacharacters describe the placeholders that the pattern uses. The pattern `The .{3,5} chased the mouse|frog` contains the literals `The`,`chased`,`the`,`mouse` and `frog`. It contains the metacharacters `.`,`{3,5}` and `|`. ## Regular Expression Functions SingleStore Helios has six built-in functions that allow you to work with regular expressions. * [`REGEXP`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/rlike-and-regexp.md): Searches a string for a regular expression pattern and returns `1` if the string is found, `0` if the pattern is not found. * [`RLIKE`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/rlike-and-regexp.md): Has the same functionality as `REGEXP`. * [`REGEXP_INSTR`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-instr.md): Searches a string for a regular expression pattern and returns an integer indicating the beginning position or ending position of the matched substring. * [`REGEXP_SUBSTR`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-substr.md): Searches a string for a regular expression pattern and returns the matching substring. * [`REGEXP_REPLACE`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-replace.md): Searches a string for the first occurrence of a regular expression pattern and replaces that occurrence with a replacement string. Can also perform a search and replace of all occurrences. * [`REGEXP_MATCH`](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-match.md): Returns a JSON array of matching substring(s) within the first match of a regular expression pattern to a string. It supports both extended and advanced regular expression formats. You can set the global variable `REGEXP_FORMAT` to specify which format you want to use. ## Regular Expression Format - Advanced Regular Expression (ARE) SingleStore Helios supports advanced regular expressions (ARE) as defined in [PostgreSQL](https://www.postgresql.org/docs/9.3/static/functions-matching.html). ARE are nearly a superset of ERE and support common Perl regular expression extensions. You can use the ARE format with the built-in SingleStore Helios functions `RLIKE`, `REGEXP`,`REGEXP_INSTR`,`REGEXP_SUBSTR`, and `REGEXP_REPLACE`, which are described in the next section. Set the global variable `regexp_format` to specify the regular expression format to be used by the built-in functions you call. To use ARE globally, run: ``` SET GLOBAL regexp_format=advanced; ``` You should set `regexp_format` to `'advanced'` if you are developing new regular expression logic. ## Advanced Regular Expression (ARE) Support SingleStore Helios supports ARE as defined in [PostgreSQL](https://www.postgresql.org/docs/9.3/static/functions-matching.html). This support is summarized below. ## Backreferences A pattern containing a backreference matches the same string that it matched earlier in the string. The earlier string must be enclosed in parentheses. A pattern may contain multiple backreferences that are denoted by `\\n`, where `n` is the number of the backreference. For example, the pattern `a(bc)d(ef)g\\1\\2` matches `abcdefgbcef`. ## Escapes Escapes make it easier to write some types of patterns. Common escapes are listed below. See section 9.7.3.3 of the [PostgreSQL pattern matching documentation](https://www.postgresql.org/docs/9.3/static/functions-matching.html) for a full explanation of the supported escapes. | Metacharacter | Meaning | | ------------- | ----------------------------------------------------------------------- | | `\d` | Matches any digit. | | `\s` | Matches a space. | | `\w` | Matches any alpha-numeric character including the underscore. | | `\D` | Matches any character except a digit. | | `\S` | Matches any character except a space. | | `\W` | Matches any non-alpha-numeric character. Does not match the underscore. | | `\m` | Matches the beginning of a word. | | `\M` | Matches the end of a word. | > **📝 Note**: In PostgreSQL, escapes in patterns begin with one `\` character.In SingleStore Helios, escapes in patterns also begin with one `\` character. Additionally, a `\` literal in an SingleStore Helios string needs to be escaped with a leading `\`. Hence, pattern strings containing escapes use two `\` characters. For example, `SELECT REGEXP_INSTR('dog58','\\d');` ## Error Handling Error handling for regular expression functions is controlled by the `regexp_error_handling_level` engine variable. The following table shows the allowed values and the error handling behavior. | Value | Behavior | | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 8.7 | Preserves prior behavior. NULL is returned when the memory limit is reached and when the pattern is not a string literal. Provided for backward compatibility. | | 8.9 | Returns an error when the memory limit is reached. Returns NULL on other errorswhen the pattern is not a string literal. | | 9.0 | Returns an error when the memory limit is reached and when the pattern is not a string literal. | ## In this section * [REGEXP\_INSTR](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-instr.md) * [REGEXP\_MATCH()](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-match.md) * [REGEXP\_REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-replace.md) * [REGEXP\_SUBSTR](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-substr.md) * [RLIKE and REGEXP](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/rlike-and-regexp.md) *** Modified at: September 25, 2025 Source: [/cloud/reference/sql-reference/regular-expression-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/) (An index of the documentation is available at /llms.txt) --- # Resource Pool Commands ## In this section * [ALTER RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/alter-resource-pool.md) * [CREATE RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/create-resource-pool.md) * [DROP RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/drop-resource-pool.md) * [SHOW RESOURCE POOLS](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/show-resource-pools.md) *** Modified at: April 24, 2021 Source: [/cloud/reference/sql-reference/resource-pool-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/) (An index of the documentation is available at /llms.txt) --- # Restricted Keywords Keywords are words that have a specific meaning in SQL, such as `SELECT`, `ARRAY`, etc. Keywords that are *restricted* (or reserved) require special treatment if they need to be used as an identifier such as a table or column name. This is because their specific meaning could cause issues when used in the context of such a name. ## In this section * [Escaping Restricted Keywords](https://docs.singlestore.com/cloud/reference/sql-reference/restricted-keywords/escaping-restricted-keywords.md) * [List of Restricted Keywords](https://docs.singlestore.com/cloud/reference/sql-reference/restricted-keywords/list-of-restricted-keywords.md) *** Modified at: April 24, 2021 Source: [/cloud/reference/sql-reference/restricted-keywords/](https://docs.singlestore.com/cloud/reference/sql-reference/restricted-keywords/) (An index of the documentation is available at /llms.txt) --- # Security Management Commands This section contains a list of all security management commands in SingleStore. For guide material around securing SingleStore, see [Learn How to Secure SingleStore](https://docs.singlestore.com/cloud/security.md). This includes RBAC permissions and commands. In this section, you can also read about the [RBAC Permissions Matrix](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/permissions-matrix.md) and [RBAC Commands Permissions Matrix](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/permissions-matrix.md). ## In this section * [ALTER USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/alter-user.md) * [CREATE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-group.md) * [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) * [CREATE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-role.md) * [CREATE USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) * [CURRENT\_SECURITY\_GROUPS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-security-groups.md) * [CURRENT\_SECURITY\_ROLES](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-security-roles.md) * [CURRENT\_USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-user.md) * [DROP GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-group.md) * [DROP LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-link.md) * [DROP ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-role.md) * [DROP USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-user.md) * [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) * [GRANT GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-group.md) * [GRANT ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-role.md) * [REVOKE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke.md) * [REVOKE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-group.md) * [REVOKE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-role.md) * [SECURITY\_LISTS\_INTERSECT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/security-lists-intersect.md) * [SET PASSWORD](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/set-password.md) * [SHOW GRANTS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-grants.md) * [SHOW GROUPS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups.md) * [SHOW GROUPS FOR ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups-for-role.md) * [SHOW GROUPS FOR USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups-for-user.md) * [SHOW LINKS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-links.md) * [SHOW ROLES](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles.md) * [SHOW ROLES FOR GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles-for-group.md) * [SHOW ROLES FOR USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles-for-user.md) * [SHOW USERS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users.md) * [SHOW USERS FOR GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users-for-group.md) * [SHOW USERS FOR ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users-for-role.md) * [Permissions Matrix](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/permissions-matrix.md) * [Default Mode Permissions Changes](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/default-mode-permissions-changes.md) *** Modified at: March 9, 2026 Source: [/cloud/reference/sql-reference/security-management-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/) (An index of the documentation is available at /llms.txt) --- # SHOW Commands ## In this section * [SHOW CDC EXTRACTOR POOL](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-cdc-extractor-pool.md) * [SHOW CHARACTER SET](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-character-set.md) * [SHOW COLLATION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-collation.md) * [SHOW COLUMNS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-columns.md) * [SHOW CREATE EXTENSION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-extension.md) * [SHOW CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-link.md) * [SHOW CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-pipeline.md) * [SHOW CREATE PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-projection.md) * [SHOW CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-table.md) * [SHOW CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-view.md) * [SHOW DATABASES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-databases.md) * [SHOW DATABASE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-database-status.md) * [SHOW DISTRIBUTED\_PLANCACHE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-distributed-plancache-status.md) * [SHOW ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-errors.md) * [SHOW EXTENSIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-extensions.md) * [SHOW FEEDBACK REOPTIMIZATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-feedback-reoptimization-status.md) * [SHOW FULLTEXT SERVICE METRICS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-fulltext-service-metrics.md) * [SHOW FULLTEXT SERVICE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-fulltext-service-status.md) * [SHOW GLOBAL STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-global-status.md) * [SHOW INDEX, SHOW INDEXES, SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) * [SHOW INDEXES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-indexes.md) * [SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-keys.md) * [SHOW LOAD ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-load-errors.md) * [SHOW LOAD WARNINGS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-load-warnings.md) * [SHOW PARTITIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-partitions.md) * [SHOW PIPELINES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-pipelines.md) * [SHOW PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-profile.md) * [SHOW PROFILE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-profile-pipeline.md) * [SHOW PLAN](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plan.md) * [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) * [SHOW PROCESSLIST](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-processlist.md) * [SHOW PROJECTIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-projections.md) * [SHOW REPRODUCTION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-reproduction.md) * [SHOW SCHEMAS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-schemas.md) * [SHOW SESSION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-session-status.md) * [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md) * [SHOW TABLE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-table-status.md) * [SHOW TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-tables.md) * [SHOW VARIABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-variables.md) * [SHOW WARNINGS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-warnings.md) * [System Status](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/system-status.md) *** Modified at: June 23, 2026 Source: [/cloud/reference/sql-reference/show-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/) (An index of the documentation is available at /llms.txt) --- # SQL Command Syntax SingleStore supports basic SQL elements, including: * Arrays * Comments * Delimiter * Identifiers * Operators * Parameters * Strings * Variables ## Arrays Arrays consist of an ordered set of elements, where each element in the array must have the same data type. For example, you cannot have integer and string elements in the same array. Refer to [ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/array.md) for more information. ## Example Acceptable formats: ```sql sample_int ARRAY(INT) = [1,2,3,4]; sample_varchar ARRAY(VARCHAR(25)) = ["apples", "oranges", "bananas"]; ``` Unacceptable formats: ```sql sample_int ARRAY(VARCHAR(10)) = [1,"apple",2, "oranges", 3,"bananas"]; ``` ## Comments SingleStore supports the following comment types: * **Single line comments**: Add `--` or `#` at the start of any line to add single line comments in your code. For example, ```sql -- this is a comment # this is a comment -- **This is also a comment -- ``` * **Multi-line or block comments**: To specify multi-line or block comments, enclose the comment in `/* */`. For example, ```sql /* This comment can span multiple lines. */ ``` ## Delimiter By default, SingleStore uses a `;` (semicolon) as the statement delimiter. You can modify the delimiter using the `DELIMITER` command. For example, ```sql DELIMITER // ``` Refer to [MySQL Client Delimiters](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/mysql-client-delimiters.md) for more information. ## Identifiers Identifiers are the names given by a database creator or user to database objects. Some examples are tables, columns, indexes, views, aliases, and other objects. Identifiers can start with capital or lowercase letters, a dollar sign, or a combination of letters and numbers. Identifiers cannot have a solely numeric name. The use of decimals, symbols, and all numeric names requires back tics. ## Examples Acceptable formats: ```sql CREATE TABLE Assets_104; CREATE TABLE 104_Assets; CREATE TABLE Assets$104; CREATE TABLE `1.04_assets`; CREATE TABLE `99999_assets`; CREATE TABLE `#@*_assets`; CREATE TABLE `*&%$@`; ``` Unacceptable formats: ```sql CREATE TABLE 104; CREATE TABLE 1.04_assets; CREATE TABLE assets_1.04; ``` ## Operators Operators are either a reserved character or word used mainly with the `WHERE` clause to perform comparative operations. SingleStore uses most of the standard mathematic, bit, and comparison operators. Here are some operators that SingleStore supports: | **Operator Type** | **Supported Operators** | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Arithmetic | `+`,`-`,`*`,`/`,`%`**Note**:SingleStorereturns`NULL`when dividing by zero. To return an error,SingleStorerecommends using a stored procedure with error/exception handling. Refer to[Exceptions and Exception Handling](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/exceptions-and-exception-handling.md)for more information. | | Assignment | `=`,`:=` | | Bitwise | `&`,`\|`,`^`,`~`,`<<`,`>>` | | Comparison | `=`,`<=>`,`>`,`>=`,`<`,`<=`,`!=``BETWEEN ... AND ...`,`NOT BETWEEN ... AND ...``IN()`,`NOT IN()`,`IS`,`IS NOT`,`IS NULL`,`IS NOT NULL`,`ISNULL()`,`LIKE`,`NOT LIKE`,`strcmp()`Refer to[Comparison Operators](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions.md)for more information. | | JSON | `::`,`::$`,`::%` | | Logical | `AND`,`OR`,`NOT``&&`,`\|\|`,`!` | | Typecast | `:>`,`!:>`Refer to[Cast Operators](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/cast-or-convert/#UUID-cc2c6e3c-477b-ad9f-074b-545f479017c1.md)for more information. | ## Parameters A parameter is used to exchange data among stored procedures and functions. Refer to [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) for more information. ## Examples ```sql CREATE FUNCTION num_parameter(emp_id INT); ``` ## Strings A string is a data type that is used to store data in a table. Strings are enclosed by single or double quotes. SingleStore supports a variety of [String Functions](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions.md). ## Examples Basic string function format is shown below: ```sql SELECT 'exp_str\3 exp_1'; SELECT "exp_str\3 exp_1"; ``` ## Variables A variable is an object that can be a single data value or a specific type (i.e., date, integer, etc.). Variables can start with capital or lowercase letters, a dollar sign, or a combination of letters and numbers. Variables cannot have a solely numeric name. The use of decimals, symbols, and all numeric names requires back tics. ## Examples Acceptable formats: ```sql SET @str = CAST(123 AS CHAR(5)); SET @var = 0; SET @$Some7_var = 0; SET @`29999` = 1010; ``` Unacceptable formats: ```sql SET @'var_name' =1010; SET @"var_name" = 1010; SET @29999 = 1010; ``` *** Modified at: April 24, 2025 Source: [/cloud/reference/sql-reference/sql-command-syntax/](https://docs.singlestore.com/cloud/reference/sql-reference/sql-command-syntax/) (An index of the documentation is available at /llms.txt) --- # SQL Commands List Here is an alphabetical list of all the SQL commands SingleStore Helios currently supports. This list is subject to change. | Command Name | Command Description | Grouping | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | [AGGREGATOR SYNC AUTO\_INCREMENT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/aggregator-sync-auto-increment.md) | Sets the auto-increment counter on the aggregators (master or child). | Data Manipulation Language | | [ALTER DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-database.md) | Specifies if an existing database should begin using synchronous or asynchronous replication. | Data Definition Language | | [ALTER PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/alter-pipeline.md) | Changes an existing pipeline’s configuration. | Pipeline | | [ALTER RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/alter-resource-pool.md) | Changes the resource settings in an existing resource pool. | Resource Pool | | [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) | Modifies the structure of an existing table. | Data Definition Language | | [ALTER USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/alter-user.md) | Alters settings associated with the user. | Security Management | | [ALTER VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-view.md) | Atomically replaces a view with a new view defined by a`SELECT`statement. | Data Definition Language | | [ANALYZE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md) | Collects statistics for a table. | Operational | | [ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/array.md) | Defines an`ARRAY`data type inSingleStore HeliosProcedural SQL (PSQL) that stores a collection of elements sharing the same data type. | Procedural | | [ATTACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/attach-database.md) | Restores an unlimited storage database at a restore point and then brings the database online. | Operational | | [BACKUP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database.md) | Backs up a database. | Operational | | [BEGIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/begin.md) | Commits any existing open transaction on the current connection and starts a new transaction. | Data Manipulation Language | | [CALL](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/call.md) | Runs the specified stored procedure. Does not return results. | Procedural | | [CHECK BLOB CHECKSUM](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/check-blob-checksum.md) | Checks for and displays columnstore blob file corruptions. Applies to the blobs in local disk and local blob cache. | Operational | | [CHECK BOTTOMLESS](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/check-bottomless.md) | Checks the checksum of the blob files stored in the remote blob store. | Operational | | [CLEAR LOAD ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/clear-load-errors.md) | Removes load errors from the`information_schema.LOAD_DATA_ERRORS`table. These errors are populated by`LOAD DATA ... ERRORS HANDLE `when it runs. | Data Manipulation Language | | [COLLECT](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/collect.md) | Runs a`SELECT`statement having the[QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/query.md)type and stores the result as an array of records. | Procedural | | [COMMIT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/commit.md) | Commits the open transaction if one exists. | Data Manipulation Language | | [CREATE EXTERNAL FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-external-function.md) | Creates an external function, which is a function that calls code that runs outside of aSingleStoredatabase. | Procedural | | [CREATE TEMPORARY PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-temporary-procedure.md) | Creates a temporary stored procedure, which is a stored procedure that is available during a user's session. | Procedural | | [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md) | Creates a user-defined aggregate function (UDAF). | Procedural | | [CREATE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-database.md) | Creates a database. | Data Definition Language | | [Trace Event Commands](https://docs.singlestore.com/cloud/reference/sql-reference/trace-event-commands.md) | Creates a trace event. | Trace Events | | [CREATE FUNCTION (TVF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md) | Creates a user-defined table-valued function (TVF). | Procedural | | [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) | Creates a user-defined scalar-valued function (UDF). | Procedural | | [CREATE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-group.md) | Creates a group on aSingleStore Helioscluster. | Security Management | | [CREATE INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-index.md) | Creates an index on the specified table. | Data Definition Language | | [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) | Creates a new connection link to S3, Azure, GCS, HDFS, HTTP,Kafka,MongoDB®, or MySQLfor a permitted user. | Security Management | | [CREATE MILESTONE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-milestone.md) | Records a milestone, which is a named point to which you can restore an unlimited storage database. Restoring to the milestone will bring the database to the point in time when the milestone was created. | Data Definition Language | | [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) | Creates a new[Pipeline](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines.md)to continuously extract, shape, and load data into a table, or to shape and load a set of files fast into a table, in parallel. | Pipeline | | [CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-procedure.md) | Creates a stored procedure. | Procedural | | [CREATE PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-projection.md) | Creates a projection. | Data Definition Language | | [CREATE RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/create-resource-pool.md) | Creates a resource pool. | Resource Pool | | [CREATE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-role.md) | Creates a role on aSingleStore Helioscluster. | Security Management | | [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) | Creates a new table. | Data Definition Language | | [CREATE USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) | Creates a new user account. | Security Management | | [CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view.md) | Creates an unmaterialized view. | Data Definition Language | | [CREATE\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-array.md) | Creates a new array of the specified length that is initialized with default values for the array’s inferred data type. | Procedural | | [CURRENT\_SECURITY\_GROUPS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-security-groups.md) | Returns a comma separated list of the current user's security groups on a SingleStorecluster. | Security Management | | [CURRENT\_SECURITY\_ROLES](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-security-roles.md) | Returns a comma separated list in one row of the current user's security roles on a SingleStorecluster. | Security Management | | [CURRENT\_USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/current-user.md) | Returns the details (username and hostname) of the user executing the query. | Security Management | | [DELETE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/delete.md) | Deletes rows in a table. | Data Manipulation Language | | [DESCRIBE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/describe.md) | Describes the specified table. | Data Definition Language | | [DETACH DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/detach-database.md) | Takes an unlimited storage database offline. | Operational | | [DETACH PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/detach-pipeline.md) | Detaches a pipeline from its target table, similar to the`STOP PIPELINE`command, while allowing`ALTER TABLE`operations on the table currently referenced by the pipeline. | Pipeline | | [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md) | Forces a plan to be optimized and generated from scratch the next time it is run. | Data Definition Language | | [DROP AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-aggregate.md) | Removes a single user-defined aggregate function (UDAF) from the specified database. | Procedural | | [DROP DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-database.md) | Drops a local storage database. | Data Definition Language | | [Trace Event Commands](https://docs.singlestore.com/cloud/reference/sql-reference/trace-event-commands.md) | Drops a trace event. | Trace Events | | [DROP FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-function.md) | Removes a single function from the specified database, including user-defined scalar-valued functions (UDFs) and user-defined table-valued functions (TVFs). | Procedural | | [DROP GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-group.md) | Removes a group from aSingleStore Helioscluster. | Security Management | | [DROP INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-index.md) | Drops the specified index on the specified table. | Data Definition Language | | [DROP LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-link.md) | Removes a connection link on S3, Azure, GCS, HDFS, or Kafka for a permitted user. | Security Management | | [DROP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/drop-pipeline.md) | Deletes a pipeline from yourSingleStore Helioscluster. If the pipeline is currently running, it will stop it before deleting it. | Pipeline | | [DROP PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/drop-procedure.md) | Removes a single stored procedure from the specified database. | Procedural | | [DROP PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-projection.md) | Drops an existing projection. | Data Definition Language | | [DROP RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/drop-resource-pool.md) | Drops the specified resource pool. | Resource Pool | | [DROP ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-role.md) | Removes a role from aSingleStore Helioscluster. | Security Management | | [DROP TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-table.md) | Drops the specified table. | Data Definition Language | | [DROP USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-user.md) | Removes one or more user accounts. | Security Management | | [DROP VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-view.md) | Drops a view. | Data Definition Language | | [ECHO](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/echo.md) | Runs a stored procedure and returns a set of rows as a result. | Procedural | | [ECHO SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/echo-select.md) | Runs the specified`SELECT`statement within a stored procedure and returns the resulting rows to the console or to the application that called the stored procedure. | Procedural | | [EXCEPT and MINUS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/except-and-minus.md) | Compare the results of two queries and return the distinct rows from the first query that do not appear in the second query. | Data Manipulation Language | | [EXECUTE IMMEDIATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/execute-immediate.md) | Runs a SQL query. | Procedural | | [EXISTS AND NOT EXISTS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/exists-and-not-exists.md) | Use a subquery to determine whether the subquery returns any rows. | Data Manipulation Language | | [EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md) | Returns detailed information on how the query runs. | Data Manipulation Language | | [FULLTEXT SERVICE STOP](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/fulltext-service-stop.md) | Stops the full-text V2 service. | Operational | | [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) | Grants the specified privileges to an existing user. | Security Management | | [GRANT GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-group.md) | Adds a user to a group. | Security Management | | [GRANT ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant-role.md) | Adds a role to a group. | Security Management | | [INSERT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/insert.md) | Inserts data into a table. | Data Manipulation Language | | [INSERT\_ALL](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/insert-all.md) | Inserts an array of records into a table in a single operation. | Procedural | | [INTERSECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/intersect.md) | Combines the result set of two queries and returns distinct rows that are returned by both queries. | Data Manipulation Language | | [KILL CONNECTION and KILL QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/kill-connection-and-kill-query.md) | Kill the connection on the specified thread. Rolls back any query running on the thread. | Operational | | [KILLALL QUERIES](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/killall-queries.md) | Kills all running queries or closes all connections. | Operational | | [LOAD DATA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md) | Imports data stored in a CSV, JSON, BSON, Avro, or Parquet file into aSingleStoretable (referred to as the destination table in this topic). | Data Manipulation Language | | [OPTIMIZE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/optimize-table.md) | Performs optimizations on the storage of rows in a table. | Data Definition Language | | [PIVOT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/pivot.md) | Lets you transform non-aggregated data into a pivot table output format. | Data Manipulation Language | | [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) | Provides detailed resources usage metrics about a query. | Data Manipulation Language | | [PROFILE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/profile-pipeline.md) | Runs a single batch in the foreground to provide detailed resource metrics for each operation in your pipeline. The batch itself is not persisted. | Pipeline | | [QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/query.md) | SingleStore HeliosProcedural SQL (PSQL) supports the`QUERY`data type, whose value represents a SQL`SELECT`statement, optionally with bound variable values. | Procedural | | [RECORD](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/record.md) | SingleStore HeliosProcedural SQL (PSQL) supports the`RECORD`data type, which is an encapsulated group of named fields. | Procedural | | [REOPTIMIZE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize.md) | Reoptimizes the most recently profiled query based on previous executions of it. | Data Manipulation Language | | [REOPTIMIZE MARK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/reoptimize-mark.md) | Manually marks a plan for[Feedback Reoptimization](https://docs.singlestore.com/cloud/query-data/query-tuning/feedback-reoptimization.md). | Data Manipulation Language | | [REPAIR DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/repair-database.md) | This command is deprecated. | Operational | | [REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/replace.md) | If the table has a`PRIMARY KEY`or`UNIQUE`index,`REPLACE`will check rows for a matching value and, if a match is found, will delete the old row and replace it with the new row. | Data Manipulation Language | | [RESTORE DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/restore-database.md) | Restores a database from a binary backup file. | Operational | | [REVOKE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke.md) | Revokes privileges from aSingleStore Heliosuser or role. | Security Management | | [REVOKE GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-group.md) | Removes a user from a group. | Security Management | | [REVOKE ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/revoke-role.md) | Removes a role from a group. | Security Management | | [ROLLBACK](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/rollback.md) | Rolls back the open transaction if one exists. | Data Manipulation Language | | [SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md) | Retrieves data from a table. | Data Manipulation Language | | [SELECT GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/select-global.md) | Inspects the setting of a global engine variable. | Operational | | [SET and SET SESSION](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-and-set-session.md) | Sets a session variable. Its value remains in effect for the duration of the current database connection. | Operational | | [SET AGGREGATOR](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-aggregator.md) | Sets a sync variable to propagate to all aggregators in thecluster. | Operational | | [SET CLUSTER](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-cluster.md) | Sets a sync variable. Depending on the variable, the updated value is propagated to all nodes in theclusteror all aggregators in thecluster. | Operational | | [SET GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-global.md) | Sets an engine variable globally. | Operational | | [SET LEAF](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/set-leaf.md) | Sets a sync variable to propagate to all leaves in thecluster. | Operational | | [SET PASSWORD](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/set-password.md) | Sets a user’s password. | Security Management | | [SHOW AGGREGATES](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-aggregates.md) | Lists any existing user-defined aggregate functions (UDAF) in the specified database. | Procedural | | [SHOW CDC EXTRACTOR POOL](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-cdc-extractor-pool.md) | Displays information about the CDC-in pipelines. | Show Commands | | [SHOW CHARACTER SET](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-character-set.md) | Shows supported character sets and the default collation for each character set. | Show Commands | | [SHOW COLLATION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-collation.md) | Lists available collations and character sets. | Show Commands | | [SHOW COLUMNS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-columns.md) | Shows the list of columns within a given table. | Show Commands | | [SHOW CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-aggregate.md) | Shows the function definition and other attributes for a user-defined aggregate function (UDAF). | Procedural | | [SHOW CREATE FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-function.md) | Outputs configuration information about an existing user-defined function, including user-defined scalar value functions (UDFs) and user-defined table-valued functions (TVFs). | Procedural | | [SHOW CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-link.md) | Shows the`CREATE LINK`command used to create a link. | Show Commands | | [SHOW CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-pipeline.md) | Returns the`CREATE PIPELINE`statement that was issued to create a pipeline. The output can be used to duplicate the configuration and state of an existing pipeline. | Show Commands | | [SHOW CREATE PROJECTION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-projection.md) | Shows the`CREATE PROJECTION`statement that was used to create the projection. | Show Commands | | [SHOW CREATE PROCEDURE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-create-procedure.md) | Shows the definition, procedure body, and other attributes for a stored procedure. | Procedural | | [SHOW CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-table.md) | Shows the`CREATE TABLE`statement that was used to create the table. | Show Commands | | [SHOW CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-create-view.md) | Shows the`CREATE VIEW`statement that was used to create the view. | Show Commands | | [SHOW DATABASE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-database-status.md) | ShowsSingleStore Heliosdatabase status. | Show Commands | | [SHOW DATABASES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-databases.md) | Shows the list of databases that exist on thisSingleStore Heliosinstance. | Show Commands | | [SHOW DISTRIBUTED\_PLANCACHE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-distributed-plancache-status.md) | Provides information about the status and performance of the[Distributed Plancache](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache.md). | Show Commands | | [SHOW ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-errors.md) | Displays errors as a result of execution of an invalid statement. | Show Commands | | [SHOW FEEDBACK REOPTIMIZATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-feedback-reoptimization-status.md) | Displays system-wide Feedback Reoptimization (FR) information. | Show Commands | | [SHOW FULLTEXT SERVICE METRICS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-fulltext-service-metrics.md) | Displays metrics for the JLucene full-text search in JSON format. | Show Commands | | [SHOW FULLTEXT SERVICE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-fulltext-service-status.md) | Shows the status of the full-text V2 service. | Show Commands | | [SHOW FUNCTIONS](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-functions.md) | Lists any existing user-defined functions in the specified database, including user-defined scalar-valued functions (UDFs) and user-defined table-valued functions (TVFs). | Procedural | | [SHOW GLOBAL STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-global-status.md) | Shows globalSingleStore Heliosserver status information. | Show Commands | | [SHOW GRANTS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-grants.md) | Shows permissions (privileges) for the specified user (or role). | Security Management | | [SHOW GROUPS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups.md) | Shows the list of groups on aclustergroup. | Security Management | | [SHOW GROUPS FOR ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups-for-role.md) | Shows groups for a specific role. | Security Management | | [SHOW GROUPS FOR USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-groups-for-user.md) | Shows groups for a specific user. | Security Management | | [SHOW INDEX, SHOW INDEXES, SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) | Displays the list of indexes associated with a given table. | Show Commands | | [SHOW LINKS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-links.md) | Shows all connection links on S3, Azure, GCS, HDFS, Kafka, or other data sources for a permitted user. | Security Management | | [SHOW LOAD ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-load-errors.md) | This command is deprecated. Query the[LOAD\_DATA\_ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md)information schema view to see the errors that occurred after running the`LOAD DATA`command. | Show Commands | | [SHOW LOAD WARNINGS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-load-warnings.md) | This command is deprecated. Query the[LOAD\_DATA\_ERRORS](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data.md)information schema view to see the warnings that occurred after running the`LOAD DATA`command. | Show Commands | | [SHOW PARTITIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-partitions.md) | Shows partitions and relevant information about them. | Show Commands | | [SHOW PIPELINES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-pipelines.md) | Shows all existing pipelines in a database and their current state. | Show Commands | | [SHOW PLAN](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plan.md) | Displays the[EXPLAIN](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/explain.md)plan of a query as per the plan ID. | Show Commands | | [SHOW PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-plancache.md) | Shows all query statements thatSingleStore Helioshas compiled and executed, as well as cumulative query execution statistics associated with each plan. | Show Commands | | [SHOW PROCESSLIST](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-processlist.md) | Shows details about currently running threads. | Show Commands | | [SHOW PROCEDURES](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/show-procedures.md) | Lists existing stored procedures in the current or the specified database. | Procedural | | [SHOW PROJECTIONS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-projections.md) | Shows projections created on the current database or specified table. | Show Commands | | [SHOW REPLICATION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/show-replication-status.md) | Shows the list of all connections between the replication master and secondary databases. | Operational | | [SHOW REPRODUCTION](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-reproduction.md) | Collects troubleshooting data for query optimizer and code generation issues for queries that fail in the compilation phase. | Show Commands | | [SHOW RESOURCE POOLS](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/show-resource-pools.md) | Lists all resource pools and their field values. | Resource Pool | | [SHOW ROLES](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles.md) | Shows the list of roles on aclustergroup. | Security Management | | [SHOW ROLES FOR GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles-for-group.md) | Shows roles for a specific group. | Security Management | | [SHOW ROLES FOR USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-roles-for-user.md) | Shows roles for a specific user. | Security Management | | [SHOW SCHEMAS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-schemas.md) | Shows the list of schemas that exist on thisSingleStore Heliosinstance. | Show Commands | | [SHOW SESSION STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-session-status.md) | ShowsSingleStore Heliosserver status information for a session. | Show Commands | | [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md) | ShowsSingleStore Heliosserver status information with detailedSingleStore Helios-specific memory and performance metrics. | Show Commands | | [SHOW TABLE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-table-status.md) | Shows tables status information in aSingleStore Heliosdatabase. | Show Commands | | [SHOW TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-tables.md) | Shows the list of tables in the currently selected database, or in another database if db\_name is specified. | Show Commands | | [SHOW USERS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users.md) | Shows users on aSingleStore Helioscluster. | Security Management | | [SHOW USERS FOR GROUP](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users-for-group.md) | Shows users for a specific group. | Security Management | | [SHOW USERS FOR ROLE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-users-for-role.md) | Shows users for a specific role. | Security Management | | [SHOW VARIABLES](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-variables.md) | Shows a list of variable bindings. | Show Commands | | [SHOW WARNINGS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-warnings.md) | Displays warnings as a result of an invalid statement execution. | Show Commands | | [SNAPSHOT DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/snapshot-database.md) | Takes a snapshot of the given database and truncates the transaction log. | Operational | | [START PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/start-pipeline.md) | Starts an existing pipeline in aSingleStore Heliosdatabase. An error will occur if the pipeline is already running. | Pipeline | | [STOP PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/stop-pipeline.md) | Stops a currently running pipeline in aSingleStore Heliosdatabase. | Pipeline | | [SYNC BOTTOMLESS DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/sync-bottomless-database.md) | Synchronizes any currently committed data in the database with the remote object storage. | Operational | | [\_SYNC\_PLANS](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/sync-plans.md) | Synchronizes a node's local PPC with the DPC for a specific database. | Operational | | [TEST PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/test-pipeline.md) | Tests an existing pipeline and looks for any new files to import. | Pipeline | | [TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/truncate.md) | Removes all rows from the table. | Data Definition Language | | [UNION](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/union.md) | Combines results from multiple[SELECT](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select.md)statements. | Data Manipulation Language | | [UNLOCK TABLES](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/unlock-tables.md) | Releases all table locks on all databases in thecluster. | Data Definition Language | | [UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/update.md) | Modifies rows in a table. | Data Manipulation Language | | [USE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/use.md) | Specifies the database or schema to use. | Data Manipulation Language | | [WITH (Common Table Expressions)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md) | Specifies a temporary named result set, referred to as aCommon Table Expression (CTE) (A named temporary result set that exists within the scope of a single statement and that can be referred to later within that statement, possibly multiple times.). | Data Manipulation Language | *** Modified at: July 21, 2026 Source: [/cloud/reference/sql-reference/sql-commands-list/](https://docs.singlestore.com/cloud/reference/sql-reference/sql-commands-list/) (An index of the documentation is available at /llms.txt) --- # SQL Functions List Here is an alphabetical list of all the SQL functions SingleStore Helios currently supports. This list is subject to change. | Function | Function Description | Grouping | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | [ABS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/abs.md) | Returns the positive value of a given number, regardless of sign. | Numeric | | [ACOS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/acos.md) | Returns the arc cosine of**x**in radians, or NULL if**x**is not in the range \[-1, 1]. | Numeric | | [ADDTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/addtime.md) | Adds the given time object to a time or datetime object. | Date and Time | | [AES\_DECRYPT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/aes-decrypt.md) | Decrypts the given ciphertext using the[AES (Advanced Encryption Standard)](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard)algorithm, with a 128-bit keyor a 256-bit key. | String | | [AES\_ENCRYPT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/aes-encrypt.md) | Encrypts the given plaintext using the[AES (Advanced Encryption Standard)](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard)algorithm with a 128-bit keyor a 256-bit key. | String | | [AGGREGATOR\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/aggregator-id.md) | Returns the unique ID (`node_id`) for the current aggregator. | Information | | [ANY\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/any-value.md) | An aggregate function that arbitrarily picks one value from the group. This can include a NULL value if one is present in the group. | Aggregate | | [APPROX\_COUNT\_DISTINCT](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct.md) | An aggregate function that returns an estimate of the number of distinct values in a collection. | Distinct Count Estimation | | [APPROX\_COUNT\_DISTINCT\_ACCUMULATE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-accumulate.md) | Creates a new[HyperLogLog](http://en.wikipedia.org/wiki/HyperLogLog)state and returns a value representing the new state for a collection of values. | Distinct Count Estimation | | [APPROX\_COUNT\_DISTINCT\_COMBINE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-combine.md) | Merges two or more[HyperLogLog](http://en.wikipedia.org/wiki/HyperLogLog)states into a new state and returns a value of type`VARBINARY(16384)`representing the new state. | Distinct Count Estimation | | [APPROX\_COUNT\_DISTINCT\_ESTIMATE](https://docs.singlestore.com/cloud/reference/sql-reference/distinct-count-estimation-functions/approx-count-distinct-estimate.md) | Returns an estimate of the number of distinct values for one or more[HyperLogLog](http://en.wikipedia.org/wiki/HyperLogLog)states. | Distinct Count Estimation | | [APPROX\_GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/approx-geography-intersects.md) | This function is the fast approximate version of`GEOGRAPHY_INTERSECTS`. It takes in two geospatial objects and determines whether any part of them overlap. | Geospatial | | [APPROX\_PERCENTILE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/approx-percentile.md) | Calculates the approximate percentile for a set of values. The calculation occurs for a given percentage. | Aggregate | | [ASCII](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ascii.md) | Returns the ASCII value of the leftmost character of the given string. | String | | [ASIN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/asin.md) | Returns the arc sine of**x**in radians, or NULL if**x**is not in the range \[-1, 1]. | Numeric | | [ATAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/atan.md) | Returns the arc tangent of**x**. | Numeric | | [ATAN2, ATAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/atan-2-atan.md) | Returns the arc tangent of**x**and**y**. It is similar to calculating the arc tangent of**y / x**, except that the signs of both arguments are used to determine the quadrant of the result. | Numeric | | [AVG](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/avg.md) | Calculates the average value from a set of numbers. | Aggregate | | [BETWEEN, NOT BETWEEN](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/between-not-between.md) | `BETWEEN`and`NOT BETWEEN`check if values are in or not in a range that includes the two endpoints (inclusive). For example: | Comparison Operators | | [BIN](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/bin.md) | Returns a string representation of the binary value of N, where N is a longlong (`BIGINT`) number. | String | | [BIN\_TO\_UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/bin-to-uuid.md) | Converts a binary UUID value to a string UUID value. | Identifier Generation | | [BINARY](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/binary.md) | Casts the input to a binary data type. | String | | [BIT\_AND](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-and.md) | Performs an`AND`operation on each bit between expressions. | Aggregate | | [BIT\_COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bit-count.md) | Returns the number of ones in its binary representation for a given number. | Numeric | | [BIT\_OR](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-or.md) | Performs an`OR`operation on each bit between expressions. | Aggregate | | [BIT\_XOR](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/bit-xor.md) | Performs an`XOR`function on corresponding pairs of bits (i.e., the first pair of bits, the second pair of bits, etc.). | Aggregate | | [bitwise AND (&)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-and.md) | Combines two integers to produce a third integer whose binary bits are 1 if and only if both bits in the input numbers are also 1. | Numeric | | [bitwise left shift (<<)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-left-shift.md) | Moves all of the bits of`num1`to the left by`num2`places. Each place shifted is equivalent to multiplying by 2. | Numeric | | [bitwise NOT (\~)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-not.md) | Inverts all of the bits of a given number. | Numeric | | [bitwise OR (\|)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-or.md) | Combines two integers to produce a third integer whose binary bits are 1 if either bit in the input numbers are also 1. | Numeric | | [bitwise right shift (>>)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-right-shift.md) | Moves all of the bits of num1 to the right by num2 places. | Numeric | | [bitwise XOR (^)](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/bitwise-xor.md) | Combines two integers to produce a third integer whose binary bits are 1 if either bit in the input numbers are different, ie a 1 in the first number and a 0 in the second, or vice versa. | Numeric | | [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) | Used to employpartition-level[BM25](https://en.wikipedia.org/wiki/Okapi_BM25)scoring for`VERSION 2`full-text search queries. | Full-Text Search | | [BM25\_GLOBAL](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) | Used to employ global[BM25](https://en.wikipedia.org/wiki/Okapi_BM25)scoring for`VERSION 2`full-text search queries. | Full-Text Search | | [BSON\_ARRAY\_CONTAINS\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-contains-bson.md) | Returns aSingleStore`TINYINT`1 if the specified value is present in the BSON array. | BSON | | [BSON\_ARRAY\_PUSH](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-push.md) | Adds the specified BSON value to the end of the input BSON array, and returns the array with the appended value as the result. | BSON | | [BSON\_ARRAY\_SLICE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-array-slice.md) | Slices the input array and returns a subset. | BSON | | [BSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-build-array.md) | Creates a BSON array from the input values. | BSON | | [BSON\_BUILD\_OBJECT](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-build-object.md) | Creates a BSON object from the specified``pairs. | BSON | | [BSON\_COMPARE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-compare.md) | Compares two BSON values of similar types. | BSON | | [BSON\_EXTRACT\_ALL\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-all-bson.md) | Extracts and returns all the values from a BSON document or array, optionally at the specified path. | BSON | | [BSON\_EXTRACT\_BIGINT](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bigint.md) | Extracts the value for the specified key from the BSON data, converts it to theSingleStore`BIGINT`type, and returns the value. | BSON | | [BSON\_EXTRACT\_BINARY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-binary.md) | Extracts and returns the BSON binary value while ignoring the subtype, optionally at the specified path. | BSON | | [BSON\_EXTRACT\_BOOL](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bool.md) | Extracts the boolean value for the specified key from the BSON data, converts it to the SingleStore`TINYINT`type, and returns the value. | BSON | | [BSON\_EXTRACT\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-bson.md) | Extracts and returns a BSON value from the BSON object, optionally at the specified path. | BSON | | [BSON\_EXTRACT\_DATETIME](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-datetime.md) | Extracts the value for the specified key from the BSON data, converts it to aSingleStore`DATETIME(6)`type, and returns the value. | BSON | | [BSON\_EXTRACT\_DOUBLE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-double.md) | Extracts the numeric value for the specified key from the BSON data, converts it to theSingleStore`DOUBLE`type, and returns the value. | BSON | | [BSON\_EXTRACT\_STRING](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-extract-string.md) | Extracts the string value for the specified key from the BSON data, converts it to aSingleStorestring type, and returns the value. | BSON | | [BSON\_GET\_TYPE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-get-type.md) | Returns a string that specifies the BSON type of the input BSON value. | BSON | | [BSON\_INCLUDE\_MASK and BSON\_EXCLUDE\_MASK](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-include-mask.md) | Applies a mask to a BSON document and returns the subset of the original document. | BSON | | [BSON\_INPUT\_STREAM\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-input-stream-to-array.md) | Converts a BSON array of documents or concatenated BSON documents to a PSQL Array. | BSON | | [BSON\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-length.md) | Returns the number of keys present in the BSON object or array. | BSON | | [BSON\_MATCH\_ANY and BSON\_MATCH\_ANY\_EXISTS](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-match-any-and-bson-match-any-exists.md) | Checks for the existence of data within a BSON document that matches specified filter conditions. Is more succinct than extracting values and checking them using SQL filters. | BSON | | [BSON\_MERGE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-merge.md) | Merges two BSON objects.`BSON_MERGE`performs the union of keys in both objects. | BSON | | [BSON\_NORMALIZE\_\*](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-normalize.md) | Set of four BSON functions that normalize BSON documents. | BSON | | [BSON\_SET\_BSON](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-set-bson.md) | Sets (or overwrites) the value at the specified key in a BSON object or array and returns the BSON object or array. | BSON | | [BSON\_UNWIND](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-unwind.md) | Converts a BSON document to a PSQL array. | BSON | | [BSON\_UPDATE](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/bson-update.md) | Updates a BSON document. | BSON | | [CASE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/case.md) | Switches statement construct. Evaluates the specified expression/condition and returns the result from the matching expression. | Conditional | | [CEIL](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/ceil.md) | Rounds a number up to the next highest integer. | Numeric | | [CHAR](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/char.md) | Processes each argument`N`as an integer and then returns a string consisting of the characters representing the code values of those integers. | String | | [CHARACTER\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/character-length.md) | Returns the Unicode character length of the given string. | String | | [CHARSET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/charset.md) | Takes a string argument and returns the character set of that argument. | String | | [COALESCE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/coalesce.md) | Given a list of values, returns the first non-NULL value. | Conditional | | [CONCAT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/concat.md) | Concatenates all of the values in its argument list. | String | | [CONCAT\_WS](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/concat-ws.md) | `CONCAT_WS`stands for concatenation with separator. | String | | [CONNECTION\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/connection-id.md) | Returns the unique connection ID of the current connection. | Information | | [CONV](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/conv.md) | Converts numbers between different number bases. | Numeric | | [CAST or CONVERT](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/cast-or-convert.md) | Casts the input to the given data type. | Conditional | | [CONVERT\_TZ](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/convert-tz.md) | Converts a given datetime object from one timezone to another. | Date and Time | | [COS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/cos.md) | Computes the cosine of**x**, where**x**is given in radians. | Numeric | | [COT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/cot.md) | Returns the cotangent of**x**, where**x**is given in radians, or NULL if**x**is invalid. | Numeric | | [COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/count.md) | Counts the number of rows passed in. | Aggregate | | [CRC32](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/crc-32.md) | Hashes the given string or binary data using the[CRC32C (Castagnoli CRC32) algorithm](https://en.wikipedia.org/wiki/Cyclic_redundancy_check). | String | | [CURRENT\_DATE And CURDATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-date-and-curdate.md) | Return the current date. | Date and Time | | [CURRENT\_TIME And CURTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-time-and-curtime.md) | Return the current time of day. | Date and Time | | [CURRENT\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/current-timestamp.md) | Returns the current date and time.`NOW()`return a second precision result and`NOW(6)`returns a microsecond precision result. | Date and Time | | [DATABASE](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/database.md) | Returns the name of the currently selected database. | Information | | [DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date.md) | Extracts the date part from a datetime object. | Date and Time | | [DATE\_ADD](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-add.md) | Adds the given interval of time to a date or datetime object. | Date and Time | | [DATE\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-format.md) | Converts the input datetime object to a string according to the specified format. | Date and Time | | [DATE\_SUB](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-sub.md) | Subtracts the given interval of time to a date or datetime object. | Date and Time | | [DATE\_TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/date-trunc.md) | Truncates a timestamp using the specified date part. | Date and Time | | [DATEDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/datediff.md) | Subtracts to find the number of days between two dates. | Date and Time | | [DAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/day.md) | Extracts the day of the month from the given datetime. | Date and Time | | [DAYNAME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayname.md) | Extracts the name of the day of the week from a date. | Date and Time | | [DAYOFWEEK](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayofweek.md) | Extracts the day of the week from a date. Sunday is 1, Monday is 2, etc. | Date and Time | | [DAYOFYEAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/dayofyear.md) | Extracts the day of the year from a date. | Date and Time | | [DECODE](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/decode.md) | Searches for a code in a code-value list and returns the corresponding value. | Conditional | | [DEGREES](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/degrees.md) | Converts**x**from radians to degrees. | Numeric | | [DENSE\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/dense-rank.md) | Returns the rank of the current row within the partition as specified by the order by clause. | Window | | [DOT\_PRODUCT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/dot-product.md) | Returns the scalar product or dot product of two vectors. | Vector | | [ELT](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/elt.md) | Returns the Nth value in its list of arguments. | Conditional | | [EUCLIDEAN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/euclidean-distance.md) | Returns the euclidean distance between two vector values. | Vector | | [EXP](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/exp.md) | Returns the result of e raised to the power of the argument specified. | Numeric | | [EXTRACT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/extract.md) | Extracts specified components from a given datetime. | Date and Time | | [FIELD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/field.md) | Returns the first index of the`str`argument in the`str1`,`str2`, … list, or 0 if it doesn’t exist. | String | | [FIRST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/first.md) | Returns the first value of a set of input values, defined as the value associated with the minimum time. | Time Series | | [FIRST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/first-value.md) | Returns an expression evaluated at the first row of the window frame. | Window | | [FLOOR](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/floor.md) | Rounds a number down to the next lowest integer. | Numeric | | [FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/format.md) | Rounds and formats a number with thousands separators and a fixed-length decimal part. | String | | [FOUND\_ROWS](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/found-rows.md) | Returns the number of rows when using commands that return a resultset, such as`SELECT`,`DESC`, and`SHOW`. | Information | | [FROM\_BASE64](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/from-base-64.md) | Takes a base-64 encoded string and returns the decoded result as a binary string. | String | | [FROM\_DAYS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/from-days.md) | Returns the date object corresponding to the input day number. | Date and Time | | [FROM\_UNIXTIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/from-unixtime.md) | Returns the corresponding datetime for a given integer Unix timestamp. | Date and Time | | [GEOGRAPHY\_AREA](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-area.md) | Measures the square meter area of a given polygon. | Geospatial | | [GEOGRAPHY\_CONTAINS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-contains.md) | Takes in two geospatial objects and determines whether the first one completely covers the area of the second. | Geospatial | | [GEOGRAPHY\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-distance.md) | Takes in two geospatial objects and measures the shortest distance between them, in meters, using the standard metric for[distance on a sphere](https://en.wikipedia.org/wiki/Great-circle_distance). | Geospatial | | [GEOGRAPHY\_INTERSECTS](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-intersects.md) | Takes in two geospatial objects and determines whether any part of them overlap. | Geospatial | | [GEOGRAPHY\_LATITUDE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-latitude.md) | Extracts the latitude portion of a`GeographyPoint`. | Geospatial | | [GEOGRAPHY\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-length.md) | Measures the length of a path or the total perimeter of a polygon, in meters. | Geospatial | | [GEOGRAPHY\_LONGITUDE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-longitude.md) | Extracts the longitude portion of a`GeographyPoint`. | Geospatial | | [GEOGRAPHY\_POINT](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-point.md) | Takes two floats or doubles and returns a`GeographyPoint`type. | Geospatial | | [GEOGRAPHY\_WITHIN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/geospatial-functions/geography-within-distance.md) | Takes in two geospatial objects and determines whether any part of them is within the given distance, in meters. | Geospatial | | [GET\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/get-format.md) | Return a format string for the specified datetime type and standard. | Date and Time | | [GREATEST](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/greatest.md) | Returns the highest value observed in its arguments. | Conditional | | [GROUP\_CONCAT](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/group-concat.md) | Concatenates all of the values passed in during aggregation. | Aggregate | | [HEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/hex.md) | Returns the hexadecimal representation of numeric, string, or binary data. | String | | [HIGHLIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/highlight.md) | Returns the fragment of text near the matching words from a full-text query. | Full-Text Search | | [HOUR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/hour.md) | Extracts the hour of the day from the given datetime, zero-indexed. | Date and Time | | [IF](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/if.md) | Returns second argument, if the first argument is true. Returns third argument, if the first argument is false or NULL. | Conditional | | [IN(), NOT IN()](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/in-not-in.md) | Filter the results after checking if the values are in or not in the list of parameters specified. | Comparison Operators | | [INET\_ATON](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-aton.md) | Converts an IPv4 address to a 32-bit integer. | String | | [INET\_NTOA](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-ntoa.md) | Converts a 32-bit integer to an IPv4 address. | String | | [INET6\_ATON](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-6-aton.md) | Converts an IPv6 or IPv4 internet address from its text representation to its binary representation. | String | | [INET6\_NTOA](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-6-ntoa.md) | Converts an IPv6 or IPv4 internet address from its binary representation to its text representation. | String | | [INITCAP](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/initcap.md) | Converts the first letter of each word in a string to upper case. | String | | [INSTR](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/instr.md) | Returns the first position of the given search string. | String | | [IS\_BSON\_NULL](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/is-bson-null.md) | Returns a`TINYINT`1 for nullable BSON input values, and returns a`TINYINT`0 for other values. | BSON | | [IS\_UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/is-uuid.md) | Validates a UUID value. | Identifier Generation | | [ISNULL](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/isnull.md) | Returns true if the value is NULL. | Conditional | | [ISNUMERIC](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/isnumeric.md) | Determines whether the provided expression is a valid numeric type. | Conditional | | [JSON\_AGG](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-agg.md) | `JSON_AGG`combines values and returns the combined data as a single JSON row. | JSON | | [JSON\_ARRAY\_CONTAINS\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-array-contains-type.md) | Set of three JSON functions, which return true if the given value is contained in the JSON array. | JSON | | [JSON\_ARRAY\_PACK](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/json-array-pack.md) | Converts a JSON array of zero or more floating point numbers to an encoded blob. | Vector | | [JSON\_ARRAY\_PUSH\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-array-push-type.md) | Set of three JSON functions, which add the given element to the end of a JSON array. | JSON | | [JSON\_ARRAY\_UNPACK](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/json-array-unpack.md) | Converts an encoded blob representing a vector to a JSON array representing the same vector. | Vector | | [JSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-array.md) | Evaluates a (possibly empty) list of values and returns a JSON array containing those values. | JSON | | [JSON\_BUILD\_OBJECT](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-object.md) | Converts a comma-separated argument list to a JSON object. | JSON | | [JSON\_DELETE\_KEY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-delete-key.md) | Removes a key/value pair from a JSON map or array. | JSON | | [JSON\_EXTRACT\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-extract-type.md) | Extracts data from a JSON object or array using a path specification. The function returns NULL if the path does not match or if either argument is`NULL`. | JSON | | [JSON\_GET\_TYPE](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-get-type.md) | Returns the Javascript type of the given JSON object. | JSON | | [JSON\_INCLUDE/EXCLUDE\_MASK](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-include-exclude-mask.md) | Applies an include/exclude mask to a JSON document so only a subset of the original document is returned. | JSON | | [JSON\_KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-keys.md) | Returns the top-level keys of a JSON object in the form of a JSON array. | JSON | | [JSON\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-length.md) | Computes the length of a JSON map or array. | JSON | | [JSON\_MATCH\_ANY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) | Checks for the existence of values within JSON data that match specified filters. It is more succinct than extracting values and checking them using SQL filters. | JSON | | [JSON\_MERGE\_PATCH](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-merge-patch.md) | Takes two JSON objects and merges them into a single JSON object. | JSON | | [JSON\_PRETTY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-pretty.md) | Outputs the given JSON object or JSON array in an indented and legible format. | JSON | | [JSON\_SET\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-set-type.md) | Set of three JSON functions, which sets the value of a given key in a JSON map or array. | JSON | | [JSON\_SPLICE\_\](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-splice-type.md) | Set of three JSON functions, which deletes the given element indexes from an array, and optionally splices the new values in. | JSON | | [JSON\_TO\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-to-array.md) | Converts a JSON array to aSingleStorearray. | JSON | | [LAG](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lag.md) | Returns an expression evaluated at some offset before the current row. | Window | | [LAST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/last.md) | Returns the last value of a set of input values, defined as the value associated with the maximum time. | Time Series | | [LAST\_DAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/last-day.md) | Finds the month of the input date, and then returns a date object containing the last day of that month. | Date and Time | | [LAST\_INSERT\_ID](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/last-insert-id.md) | Returns the last value inserted into an`AUTO_INCREMENT`column. | Information | | [LAST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/last-value.md) | Returns an expression evaluated at the last row of the window frame. | Window | | [LCASE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/lcase.md) | Converts the given string to lower case. | String | | [LEAD](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lead.md) | Returns an expression evaluated at some offset after the current row. | Window | | [LEAST](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/least.md) | Returns the lowest value observed in its arguments. | Conditional | | [LEFT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/left.md) | Extracts the specified number of characters from the left end of a string | String | | [LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/length.md) | Returns the byte length of a given string, array, or binary object. | String | | [LIKE, NOT LIKE](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/like-not-like.md) | The`LIKE`and`NOT LIKE`operators are useful for matching patterns containing simple wildcards. | Comparison Operators | | [LN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/ln.md) | Returns the logarithm of the given argument to base`e`. | Numeric | | [LOCALTIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/localtimestamp.md) | Returns the current time, in the local time zone. | Date and Time | | [LOCATE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/locate.md) | Returns the first position of the given search string. | String | | [LOG](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log.md) | Returns the logarithm of the given argument in the given base. | Numeric | | [LOG10](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log-10.md) | Returns the logarithm of the given argument to base 10. | Numeric | | [LOG2](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/log-2.md) | Returns the logarithm of the given argument to base 2. | Numeric | | [LPAD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/lpad.md) | Pads the beginning of a string up to the specified character length. | String | | [LTRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ltrim.md) | Removes leading space characters from the given string. | String | | [MATCH ... AGAINST](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/match.md) | For columnstore tables created with a[FULLTEXT](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md)index, the text columns in that table can be searched by using the`MATCH AGAINST`syntax. | Full-Text Search | | [MAX](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/max.md) | Returns the highest value observed in all rows in an aggregation. | Aggregate | | [MD5](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/md-5.md) | Hashes the given string or binary data using the[message digest algorithm](https://en.wikipedia.org/wiki/MD5), version 5. | String | | [MEDIAN](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/median.md) | Returns the median of an expression (middle value in a sorted data set), which separates the higher half in a distribution from the lower half. | Aggregate | | [MICROSECOND](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/microsecond.md) | Extracts the microseconds (fractional seconds part) from the given datetime. | Date and Time | | [MIN](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/min.md) | Returns the lowest value observed in all rows in an aggregation. | Aggregate | | [MINUTE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/minute.md) | Extracts the minutes of the hour from the given datetime. | Date and Time | | [MOD](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/mod.md) | `Mod`stands for Modulo. Calculates the remainder of a number divided by another number. | Aggregate | | [MONTH](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/month.md) | Extracts the month from the given datetime. | Date and Time | | [MONTHNAME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/monthname.md) | Extracts the name of the month from the given datetime. | Date and Time | | [MONTHS\_BETWEEN](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/months-between.md) | Subtracts to find the number of months between two dates. | Date and Time | | [NOPARAM](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/noparam.md) | Disables the parameterization of constants in a query before a query plan is compiled. | Code Generation | | [NOW](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/now.md) | Returns the current date and time in the configured time zone. | Date and Time | | [NTH\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/nth-value.md) | Returns an expression evaluated at the nth row of the window frame. | Window | | [NTILE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/ntile.md) | Divides the input into the specified number of buckets and returns the bucket number of the current row. | Window | | [NULLIF](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/nullif.md) | Returns`NULL`if the arguments are equal, otherwise the first argument. | Conditional | | [NVL and IFNULL](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/nvl-and-ifnull.md) | Checks if an expression evaluates to`NULL`, and if so, evaluates a replacement expression and returns its value. | Conditional | | [PERCENT\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percent-rank.md) | Returns the percent rank of the current row within the partition as specified by the order by clause. | Window | | [PERCENTILE\_CONT and MEDIAN](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-cont-and-median.md) | An inverse distribution function that assumes a continuous distribution between values of the expression in the sort specification, and then interpolates the value of that expression at the given percentile, performing a linear interpolation.InSingleStore, percentile functions are available as window functions and aggregate functions. | Window | | [PERCENTILE\_DISC](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-disc.md) | An inverse distribution function that assumes a discrete distribution between values of the expression in the sort specification. It returns the first element whose sort position among all non-null rows is greater than or equal to the percentile.InSingleStore, percentile functions are available as window functions and aggregate functions. | Window | | [PI](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/pi.md) | Returns the value of pi to 7 decimal places. | Numeric | | [POW](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/pow.md) | Returns the given number exponentiated to the given power. | Numeric | | [QUARTER](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/quarter.md) | Extracts the quarter of the year from the given datetime, starting with 1. | Date and Time | | [QUOTE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/quote.md) | Returns a string enclosed in single quotes. | String | | [RADIANS](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/radians.md) | Converts**x**from degrees to radians. | Numeric | | [RAND](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/rand.md) | Returns a random float or double between 0 and 1.0. | Numeric | | [RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/rank.md) | Returns the rank of the current row within the partition as specified by the order by clause. | Window | | [REDUCE](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/reduce.md) | Applies an expression to each element of an array and returns a single value as the result. It allows you to aggregate multiple values in an array into a single value. | JSON | | [REGEXP\_INSTR](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-instr.md) | Searches a string for a[regular expression pattern](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions.md)and returns an integer indicating the beginning position or ending position of the matched substring. | Regular Expression | | [REGEXP\_MATCH()](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-match.md) | Returns a JSON array of matching substring(s) within the first match of a regular expression pattern to a string. | Regular Expression | | [REGEXP\_REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-replace.md) | Searches a string for the first occurrence of a[regular expression pattern](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions.md)and replaces that occurrence with a replacement string. | Regular Expression | | [REGEXP\_SUBSTR](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/regexp-substr.md) | Searches a string for a[regular expression pattern](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions.md)and returns the matching substring. | Regular Expression | | [REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/replace.md) | Replaces all occurrences of a substring by another string. | String | | [REVERSE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/reverse.md) | Reverses the character order of a string or returns a`NULL`if the parameter is`NULL`. | String | | [RIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/right.md) | Extracts the specified number of characters from the right end of a string | String | | [RLIKE and REGEXP](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions/rlike-and-regexp.md) | Searches a string for a[regular expression pattern](https://docs.singlestore.com/cloud/reference/sql-reference/regular-expression-functions.md). | Regular Expression | | [ROUND](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/round.md) | Rounds a number to the given precision. | Numeric | | [ROW](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/row.md) | Constructs a`RECORD`object that may be used in an expression or assigned into a`RECORD`type variable. | Procedural | | [ROW\_COUNT](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/row-count.md) | Returns the number of rows inserted, updated, or deleted by the preceding statement. | Information | | [ROW\_NUMBER](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/row-number.md) | Returns the number of the current row within its partition. | Window | | [RPAD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/rpad.md) | Pads the end of a string up to the specified character length. | String | | [RTRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/rtrim.md) | Removes trailing space characters from the given string. | String | | [SCALAR](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/scalar.md) | Executes a query that returns a single value. | Procedural | | [SCALAR\_VECTOR\_MUL](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/scalar-vector-mul.md) | Multiples each element in a vector with a scalar value. The function returns a vector. | Vector | | [SEC\_TO\_TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/sec-to-time.md) | Returns a time object corresponding to the input second count. | Date and Time | | [SECOND](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/second.md) | Extracts the seconds of the minute from the given datetime. | Date and Time | | [SECRET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/secret.md) | Provides the ability to hide credentials from queries. | String | | [SECURITY\_LISTS\_INTERSECT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/security-lists-intersect.md) | Returns a list of a assigned roles for a user. | Security Management | | [SET](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables/set.md) | Sets a value for a user-defined variable for use in SQL statements. This is analogous to the`SELECT ... INTO`user defined variables discussed here:[SELECT ... INTO User Defined Variable](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables/select-into-user-defined-variable.md). | User-Defined Variables | | [SHA1](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/sha-1.md) | Hashes the given string or binary data using the so-called[secure hash algorithm](https://en.wikipedia.org/wiki/SHA1), version 1. SHA1 has been deprecated as a means of securely signing data. | String | | [SHA2](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/sha-2.md) | Hashes the given string or binary data using the[secure hash algorithm](https://en.wikipedia.org/wiki/SHA1), version 2. As of this writing (early 2015), SHA2 is considered secure for generating collision-resistant hashes. | String | | [SIGMOID](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sigmoid.md) | Returns the sigmoid function of**x**. | Numeric | | [SIGN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sign.md) | Returns`1`if the argument is positive,`-1`if it is negative, and`0`if it is 0. | Numeric | | [SIN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sin.md) | Computes the sine of**x**, where**x**is given in radians. | Numeric | | [SLEEP](https://docs.singlestore.com/cloud/reference/sql-reference/conditional-functions/sleep.md) | Halts execution of the thread for the given number of seconds. | Conditional | | [SPLIT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/split.md) | Splits a string by the specified separator into an array of substrings and then returns the array. | String | | [SQRT](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/sqrt.md) | Returns the square root of a given number, accurate to within the representation of doubles. | Numeric | | [STD](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/std.md) | Calculates the population standard deviation value from a set of numbers. | Aggregate | | [STDDEV](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev.md) | Returns the standard deviation of all the values in a specified expression. | Aggregate | | [STDDEV\_POP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev-pop.md) | Returns the population standard deviation for non-NULL values. | Aggregate | | [STDDEV\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/stddev-samp.md) | Calculates the sample standard deviation value from a set of numbers. | Aggregate | | [STR\_TO\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/str-to-date.md) | Extracts datetime value from input string according to a specified format. | Date and Time | | [STRCMP()](https://docs.singlestore.com/cloud/reference/sql-reference/comparison-operators-and-functions/strcmp.md) | Takes two string values, for example`STRCMP(v1, v2)`. It returns`0`if they are equal,`-1`if v1 is less than v2, and`1`if v2 is greater than v1. | Comparison Operators | | [STRING\_BYTES](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/string-bytes.md) | Converts a string to aSingleStore HeliosProcedural SQL (PSQL) array of integers (TINYINT UNSIGNED) where each array element is the integer byte value of a character, or a byte within a multiple byte character, in the input string. | String | | [SUBSTRING](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/substring.md) | Extracts a range of characters from a string. | String | | [SUBSTRING\_INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/substring-index.md) | Extracts the portion of a string up to the given number of occurrences of a delimiter. | String | | [SUM](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/sum.md) | Calculates the sum of a set of numbers. | Aggregate | | [SYS\_GUID and UUID](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/sys-guid-and-uuid.md) | Generates and returns a new globally unique identifier (GUID) each time it is called. | Identifier Generation | | [TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/table.md) | Converts a PSQL array to a set of rows. | Data Manipulation Language | | [TAN](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/tan.md) | Returns the tangent of**x**, where**x**is given in radians. | Numeric | | [TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time.md) | Extracts the time part from a datetime object. | Date and Time | | [TIME\_BUCKET](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/time-bucket.md) | Normalizes time to the nearest bucket start time. | Time Series | | [TIME\_FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time-format.md) | Converts the input time object to a string according to the specified format. | Date and Time | | [TIME\_TO\_SEC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/time-to-sec.md) | Returns the number of seconds from midnight. | Date and Time | | [TIMEDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timediff.md) | Subtracts to find the amount of time between two datetimes or times. | Date and Time | | [TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestamp.md) | Converts the input argument to a datetime object. | Date and Time | | [TIMESTAMPADD](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestampadd.md) | Adds the given interval of time to a date or datetime object | Date and Time | | [TIMESTAMPDIFF](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/timestampdiff.md) | Finds between two date or datetime objects the time difference in the specified unit. | Date and Time | | [TO\_BASE64](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/to-base-64.md) | Returns the argument**str**converted to base-64 encoding. | String | | [TO\_CHAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-char.md) | Converts a`DATE`,`DATETIME`,`DATETIME(6)`,`TIME`,`TIMESTAMP`or`TIMESTAMP(6)`value to a string. | Date and Time | | [TO\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-date.md) | Converts a string to a`DATE`or`DATETIME`value. | Date and Time | | [TO\_DAYS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-days.md) | Returns the day number of the input date object. | Date and Time | | [TO\_JSON](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/to-json.md) | Converts a table column, an entire table, a scalar value, a single row, or a PSQL array or record to a JSON value. | JSON | | [TO\_NUMBER](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/to-number.md) | Converts a`CHAR`or`VARCHAR`type value to a`DECIMAL`value. | Numeric | | [TO\_QUERY](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/to-query.md) | Allows you to convert a[dynamic SQL](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/dynamic-sql.md)statement to a query type value. | Procedural | | [TO\_SECONDS](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-seconds.md) | Returns the number of seconds counting from year 0 to the input datetime. | Date and Time | | [TO\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/to-timestamp.md) | Converts a string to a`TIMESTAMP(6)`value. | Date and Time | | [TRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/trim.md) | Removes padding from the ends of the given string. | String | | [TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/trunc.md)- Date function | Truncates a date using the specified format. | Date and Time | | [TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/trunc.md)- Numeric function | Returns the number**x**truncated to**d**decimal places. This is similar to the[TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/truncate.md)function except that the precision (d) is not required. | Numeric | | [TRUNCATE](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/truncate.md) | Returns the number**x**truncated to**d**decimal places. This is similar to the[TRUNC](https://docs.singlestore.com/cloud/reference/sql-reference/numeric-functions/trunc.md)function, except that the precision (d) is required. | Numeric | | [UCASE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ucase.md) | Converts the given string to uppercase. | String | | [UNHEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/unhex.md) | Converts a hexadecimal representation to the binary equivalent. | String | | [UNIX\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/unix-timestamp.md) | Returns the current Unix time as an integer, or converts the given date into unixtime. | Date and Time | | [USER](https://docs.singlestore.com/cloud/reference/sql-reference/information-functions/user.md) | Returns the name of the current user. | Information | | [UTC\_DATE](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-date.md) | Returns the current date, in the UTC time zone. | Date and Time | | [UTC\_TIME](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-time.md) | Returns the current time, in the UTC time zone. | Date and Time | | [UTC\_TIMESTAMP](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/utc-timestamp.md) | Returns the current time, in the UTC time zone. | Date and Time | | [UUID\_TO\_BIN](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions/uuid-to-bin.md) | Converts a string UUID value to a binary UUID value. | Identifier Generation | | [VARIANCE](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/variance.md) | Calculates the population variance value from a set of numbers. | Aggregate | | [VAR\_SAMP](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions/var-samp.md) | Calculates the sample variance value from a set of numbers. | Aggregate | | [VECTOR\_ADD](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-add.md) | Adds the two vector arguments and returns a vector which is the result of that addition. | Vector | | [VECTOR\_ELEMENTS\_SUM](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-elements-sum.md) | Sums all elements of a vector. It returns a single value – the sum of the elements. | Vector | | [VECTOR\_KTH\_ELEMENT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-kth-element.md) | Returns the k-th element in a vector expression. | Vector | | [VECTOR\_MUL](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-mul.md) | Multiples two vectors element by element and returns a vector which is the result of that multiplication. | Vector | | [VECTOR\_NUM\_ELEMENTS](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-num-elements.md) | Returns the number of elements in the vector. | Vector | | [VECTOR\_SORT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sort.md) | Sorts the elements in the vector expression and returns the sorted vector. | Vector | | [VECTOR\_SUB](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sub.md) | Subtracts the second vector from the first vector and returns a vector which is the result of that subtraction. | Vector | | [VECTOR\_SUBVECTOR](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-subvector.md) | Derives a vector expression from another vector expression. | Vector | | [VECTOR\_SUM](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sum.md) | Adds all vectors in a column and returns a vector which is the sum of those vectors. | Vector | | [WEEK](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/week.md) | There are eight ways to number weeks, described below. | Date and Time | | [WEEKDAY](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/weekday.md) | Extracts the day of the week from a date. Monday is 0, Tuesday is 1, etc. | Date and Time | | [YEAR](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions/year.md) | Extracts the year from the given datetime. | Date and Time | *** Modified at: July 22, 2026 Source: [/cloud/reference/sql-reference/sql-functions-list/](https://docs.singlestore.com/cloud/reference/sql-reference/sql-functions-list/) (An index of the documentation is available at /llms.txt) --- # String Functions ## In this section * [AES\_DECRYPT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/aes-decrypt.md) * [AES\_ENCRYPT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/aes-encrypt.md) * [ASCII](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ascii.md) * [BIN](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/bin.md) * [BINARY](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/binary.md) * [CHAR](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/char.md) * [CHARACTER\_LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/character-length.md) * [CHARSET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/charset.md) * [CONCAT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/concat.md) * [CONCAT\_WS](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/concat-ws.md) * [CRC32](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/crc-32.md) * [FIELD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/field.md) * [FORMAT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/format.md) * [FROM\_BASE64](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/from-base-64.md) * [QUOTE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/quote.md) * [HEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/hex.md) * [INET\_ATON](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-aton.md) * [INET\_NTOA](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-ntoa.md) * [INET6\_ATON](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-6-aton.md) * [INET6\_NTOA](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/inet-6-ntoa.md) * [INITCAP](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/initcap.md) * [INSTR](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/instr.md) * [LCASE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/lcase.md) * [LEFT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/left.md) * [LENGTH](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/length.md) * [LOCATE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/locate.md) * [LPAD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/lpad.md) * [LTRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ltrim.md) * [MD5](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/md-5.md) * [REPLACE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/replace.md) * [REVERSE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/reverse.md) * [RIGHT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/right.md) * [RPAD](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/rpad.md) * [RTRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/rtrim.md) * [SECRET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/secret.md) * [SHA1](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/sha-1.md) * [SHA2](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/sha-2.md) * [SPLIT](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/split.md) * [STRING\_BYTES](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/string-bytes.md) * [SUBSTRING](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/substring.md) * [SUBSTRING\_INDEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/substring-index.md) * [TO\_BASE64](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/to-base-64.md) * [TRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/trim.md) * [UCASE](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/ucase.md) * [UNHEX](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/unhex.md) *** Modified at: February 6, 2023 Source: [/cloud/reference/sql-reference/string-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/) (An index of the documentation is available at /llms.txt) --- # Time Series Functions In addition to the functions listed below, another time series feature is the `SERIES TIMESTAMP` clause, which specifies a table column as the default timestamp. It can be defined at the time of table creation or alteration. For details, see [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) or [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md) topic. ## In this section * [FIRST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/first.md) * [LAST](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/last.md) * [TIME\_BUCKET](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/time-bucket.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/time-series-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/time-series-functions/) (An index of the documentation is available at /llms.txt) --- # Trace Event Commands ## CREATE EVENT TRACE The `CREATE EVENT TRACE` command creates a trace event. ## Syntax ``` CREATE EVENT TRACE event_name [ WITH (, ...) ]; ``` Where `` is an key/value pair, such as: * `Query_text = on` captures the query text * `Duration_threshold_ms = 1000` traces all queries with runtimes over 1 second * `Trace_all_failed_executions= on` traces all queries that have failed, even when their runtimes are less than `Duration_threshold_ms` ## Remarks * The `event_name` and `event_characteristic` keys are not case-sensitive. * Currently, `Query_completion` is the only usable event type. * The [ALTER EVENT TRACE](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) permission is required to create trace events. * Since `event_characteristic` key-value pairs are optional, each event trace has its own default values for a default configuration. ## DROP EVENT TRACE The `DROP EVENT TRACE` command drops a trace event. ## Syntax ``` DROP EVENT TRACE event_name; ``` ## Remarks * The `ALTER EVENT TRACE` [permission](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) is required to drop trace events. *** Modified at: July 29, 2024 Source: [/cloud/reference/sql-reference/trace-event-commands/](https://docs.singlestore.com/cloud/reference/sql-reference/trace-event-commands/) (An index of the documentation is available at /llms.txt) --- # User-Defined Variables SingleStore allows users to define session variables to store values and reference these variables in subsequent queries. These user-defined variables are session variables meaning that they exist only during your session and are local in scope to your session. You can use `SELECT ... INTO` or `SET` to create user defined session variables. ```sql SELECT ... INTO @user_defined_variable [,...] ``` ```sql SET @user_defined_variable = ``` For stored procedures and anonymous code blocks, you can use `SELECT ... INTO ` to create variables; however, those variables are *locally scoped* variables, not session variables. ## Related Topics * [Anonymous Code Blocks](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/anonymous-code-blocks.md) * [Stored Procedures](https://docs.singlestore.com/cloud/developer-resources/procedural-extensions/stored-procedures.md) ## In this section * [SELECT ... INTO User Defined Variable](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables/select-into-user-defined-variable.md) * [SET](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables/set.md) *** Modified at: April 3, 2024 Source: [/cloud/reference/sql-reference/user-defined-variables/](https://docs.singlestore.com/cloud/reference/sql-reference/user-defined-variables/) (An index of the documentation is available at /llms.txt) --- # Vector Functions For more information about vectors, refer to [Working with Vector Data](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-vector-data.md) and the [Vector data type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/vector-type.md). ## In this section * [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) * [Vector Range Search](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-range-search.md) * [Using Vector Functions](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/using-vector-functions.md) * [Cosine Similarity and Cosine Distance](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/cosine-similarity-and-cosine-distance.md) * [DOT\_PRODUCT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/dot-product.md) * [EUCLIDEAN\_DISTANCE](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/euclidean-distance.md) * [JSON\_ARRAY\_PACK](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/json-array-pack.md) * [JSON\_ARRAY\_UNPACK](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/json-array-unpack.md) * [SCALAR\_VECTOR\_MUL](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/scalar-vector-mul.md) * [Vector Normalization](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-normalization.md) * [VECTOR\_ADD](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-add.md) * [VECTOR\_ELEMENTS\_SUM](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-elements-sum.md) * [VECTOR\_KTH\_ELEMENT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-kth-element.md) * [VECTOR\_MUL](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-mul.md) * [VECTOR\_NUM\_ELEMENTS](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-num-elements.md) * [VECTOR\_SORT](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sort.md) * [VECTOR\_SUB](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sub.md) * [VECTOR\_SUM](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sum.md) * [VECTOR\_SUBVECTOR](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-subvector.md) *** Modified at: March 5, 2026 Source: [/cloud/reference/sql-reference/vector-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/) (An index of the documentation is available at /llms.txt) --- # Window Functions This section contains a list of all window functions in SingleStore, excluding aggregates, which are covered in [Aggregate Functions](https://docs.singlestore.com/cloud/reference/sql-reference/aggregate-functions.md). For an overview of window functions in SingleStore Helios, see [Window Functions Guide](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-window-functions.md). ## In this section * [DENSE\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/dense-rank.md) * [FIRST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/first-value.md) * [LAG](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lag.md) * [LAST\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/last-value.md) * [LEAD](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/lead.md) * [NTH\_VALUE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/nth-value.md) * [NTILE](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/ntile.md) * [PERCENT\_RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percent-rank.md) * [PERCENTILE\_CONT and MEDIAN](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-cont-and-median.md) * [PERCENTILE\_DISC](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/percentile-disc.md) * [RANK](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/rank.md) * [ROW\_NUMBER](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/row-number.md) *** Modified at: November 17, 2022 Source: [/cloud/reference/sql-reference/window-functions/](https://docs.singlestore.com/cloud/reference/sql-reference/window-functions/) (An index of the documentation is available at /llms.txt) --- # System Databases SingleStore has several internal or "System" databases (views): * `cluster`: Stores cluster-level metadata that is replicated to all nodes in the cluster. This includes information about nodes host/ports etc. and information about databases and database partitions). Cluster replicas (named something like: `cluster_{uint64_t}`) are created during DR (disaster recovery) cluster-to-cluster replication. * `information.schema`: Read-only. A large set of [system tables](https://docs.singlestore.com/cloud/reference/information-schema-reference.md) containing metadata about clusters and all the components within them. * `memsql`: Stores metadata local to a given node, including which local databases exist, which local users exist, etc. Other than `information.schema`, these databases are not intended for users. Some commands are not allowed for these databases. For example: `ATTACH` / `DETACH DATABASE`, `BACKUP DATABASE`, `DROP DATABASE`, `REPLICATE DATABASE`, `REBALANCE PARTITIONS`, `RESTORE REDUNDANCY`. *** Modified at: July 12, 2022 Source: [/cloud/reference/system-databases/](https://docs.singlestore.com/cloud/reference/system-databases/) (An index of the documentation is available at /llms.txt) --- # Troubleshooting Reference ## Overview This section lists errors that you may encounter while using SingleStore Helios and explains how to resolve them. ## In this section * [Network Errors](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/network-errors.md) * [Operations Errors](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/operations-errors.md) * [Query Errors](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors.md) * [Identifying and Reducing Memory Usage](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage.md) *** Modified at: April 22, 2025 Source: [/cloud/reference/troubleshooting-reference/](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/) (An index of the documentation is available at /llms.txt) --- # Identifying and Reducing Memory Usage An example of a typical question about memory usage is, "*I only have 10GB of tables but I'm using 67GB of RAM? What is that additional memory being used for?*" SingleStore Helios keeps detailed accounting of how memory is being used. You can run the command [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md) on an individual SingleStore Helios instance to see a detailed break down of memory usage. ## SHOW STATUS EXTENDED The `SHOW STATUS EXTENDED` command displays a list of engine variable values as described in the following sections. ## Summary variables The following are summary variables that describe overall memory use: * `Total_server_memory`: Tracks the server’s overall memory use. SingleStore Helios does not let this value grow higher than `maximum_memory`. When `Total_server_memory` reaches `maximum_memory`, memory allocations start failing. Queries then fail with the error `1712 - "Not enough memory available to complete the current request. The request was not processed."` In addition, the trace log shows the following: `"Nonfatal buffer manager memory allocation failure. The maximum_memory parameter (XXXXX MB) has been reached."` * `Alloc_table_memory`: Tracks the memory stored inside of all rowstore tables (memory for rows, indexes, variable-length columns like `VARCHAR` or `JSON` that are stored off row). Once `Alloc_table_memory` reaches `maximum_table_memory`, `INSERT`, `UPDATE`, and `LOAD DATA` operations against the tables receive the following error: `1720 - "Memory usage by SingleStore Helios for tables (XXXXX MB) has reached the value of 'maximum_table_memory' global variable (YYYYY MB). This query cannot be executed.".` * `Buffer_manager_memory`: Tracks memory that is allocated by the Buffer Manager for SingleStore Helios’s built-in memory allocators. The Buffer Manager is a component that consumes memory from the Linux OS in 128KB blocks and manages that memory out to memory allocators used by rowstore tables or by query execution. If the application makes heavy use of rowstore tables, it’s normal for `Buffer_manager_memory` to be a large percentage of `Total_server_memory`. * `Buffer_manager_cached_memory`: Tracks memory that was allocated by the Buffer Manager, but is now cached and not in use. If the overall memory usage for SingleStore Helios is much higher than the table memory usage, this cache may be the reason. `Buffer_manager_cached_memory` is capped at 25% of `maximum_memory`. SingleStore Helios returns freed memory to Linux once `Buffer_manager_cached_memory` is at 25% of `maximum_memory`. For more information, see [Configuring Memory Limits](https://docs.singlestore.com/cloud/user-and-cluster-administration/#managing-memory.md). * `Alloc_query_execution`: Tracks memory allocated by currently executing queries for sorts, hash tables, result tables, etc. If no queries are running, this value should be 0. * `Alloc_variable`: Tracks memory allocated for variable-length columns inside rowstore tables, or for other variable-length memory allocations inside query execution (i.e. temporary allocations inside of string expressions, etc.). ## Row store variables Row store has a set of allocators it uses for various part of an index. These values can be helpful when determining rowstore table size. * `Alloc_skiplist_towers`: Tracks memory used by the towers for skiplist indexes. Each skiplist index uses on average 40 bytes of memory per row using this allocator. The exact amount of memory per row is probabilistic. It depends on the randomized tower height of the particular row. * `Alloc_table_primary`: Tracks memory used for on-row data for rowstore tables. SingleStore Helios tables share a single row memory allocation amongst all indexes on a particular table. Variable-length columns are not stored in this allocator (`VARCHAR`, `VARBINARY`, `BLOB`, `TEXT`, `JSON`, etc). Instead, they are stored in `Alloc_variable` that was previously discussed in this topic. * `Alloc_deleted_version`: Tracks memory used to mark rows as deleted in rowstore tables. `DELETE` queries in SingleStore Helios don’t free up memory when they commit. They mark rows as deleted and the garbage collector frees this memory up when it is safe to do so (i.e. no query or operation is using the deleted row anymore). If this number is large, it means the garbage collector is behind or some operation is preventing the garbage collector from physically freeing the memory used by deleted rows. Examples of this could be a snapshot or a backup, or a long running query, etc. * `Alloc_hash_buckets`: Tracks memory used for HASH index buckets (by default 4 million buckets per index, which would use 32 MB). ## Other variables There are a few variables that describe memory used by components not directly related to running queries or storing data: * `Alloc_replication`: Tracks the amount of memory used during replication. * `Malloc_active_memory`: Tracks memory allocated directly from the Linux OS and managed by the C runtime allocators (not SingleStore Helios’s built-in memory allocators that use the Buffer Manager). The memory use here should be approximately 1-2 GBs for most workloads. Column store tables, open connections, and memory for metadata about tables, columns, etc. are the biggest consumers of memory. * `Alloc_thread_stacks`: Tracks memory used by thread stacks. SingleStore Helios caches threads used to run queries. Each thread has a 1 MB stack by default. This can be controlled by the `thread_stack` session variable, but SingleStore does not recommend changing this value. SingleStore Helios kills threads it hasn’t used for 24 hours which frees up stack memory (this can be controlled by the `idle_thread_lifetime_seconds` variable). ## In this section * [Deleting Row Store Table Data When at the Memory Limit](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/deleting-row-store-table-data-when-at-the-memory-limit.md) * [Reducing Memory Use by Row Store Tables](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/reducing-memory-use-by-row-store-tables.md) * [Columnstore Table Considerations](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/columnstore-table-considerations.md) * [Understanding Memory and Disk Usage with Studio](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/understanding-memory-and-disk-usage-with-studio.md) * [Investigating Memory Usage Discrepancies](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/investigating-memory-usage-discrepancies.md) *** Modified at: May 11, 2026 Source: [/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/identifying-and-reducing-memory-usage/) (An index of the documentation is available at /llms.txt) --- # Network Errors ## ERROR 1158 (08S01): Leaf Error (): Error reading packet ### from the connection socket (): Connection timed out **Issue** The presence of extremely large numbers of duplicates in combination with `LOAD DATA IGNORE` can cause the leaves to have to wait so long they time out. **Solution** If you see this error when running `LOAD DATA IGNORE`, verify that the data does not have a lot of duplicates. ## ERROR 1735: Unable to connect … Timed out reading from socket **Issue** A SingleStore Helios node is unable to connect to another SingleStore Helios node. This may be because there is no network connectivity (such as a network problem or a firewall blocking connectivity), or because a node is overloaded with connection requests. **Solution** Here are some possible solutions to solve this problem: * Ensure that all nodes are able to connect to all other nodes on the configured port (the default is 3306). Update any firewall rules that block connectivity between the nodes. One way to verify connectivity is to run the command `FILL CONNECTION POOLS` on all SingleStore Helios nodes. If this fails with the same error, then a node is unable to connect to another node. Some queries require different amounts of connectivity. For example, some queries only require aggregator-leaf connections while others require aggregator-leaf as well as leaf-leaf connections. As a result, it is possible for some queries to succeed while others fail with this error. * If all nodes are able to connect to all other nodes, the error is likely because your query or queries require opening too many connections at once. Run `FILL CONNECTION POOLS` on all SingleStore Helios nodes to pre-fill connection pools. If the connection pool size is too small for your workload, adjust the [max\_pooled\_connections](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#non-sync-variables-list.md) configuration variable, which controls the number of pooled connections between each pair of nodes. ## ERROR 1970 (HY000): Subprocess /var/lib/memsql/master-3306/extractors/kafka-extract –get-offsets –kafka-version=0.8.2.2 timed out **Issue** This error occurs when there are connectivity issues between a SingleStore Helios node and the data source (e.g. Kafka cluster or S3). This error is particularly common when using S3 pipelines because of throttling and other S3 behavior. **Solution** To solve this issue, edit the value of `pipelines_extractor_get_offsets_timeout_ms`. The default value is 10000. Increase this value to eliminate the timeout error. ## ERROR 2002 (HY000): Can’t connect to local MySQL server through socket “/var/run/mysqld/mysqld.sock” **Issue** When the MySQL client connects to `localhost`, it attempts to use a socket file instead of TCP/IP. The socket file used is specified in `/etc/mysql/my.cnf` when the MySQL client is installed on the system. This is a MySQL socket file, which SingleStore Helios does not use by default. Therefore, connecting with `localhost` attempts to connect to MySQL and not SingleStore Helios. **Solutions** There are two solutions to solve this problem: 1. Specify `127.0.0.1` as the host instead of `localhost`. That is, `mysql -h 127.0.0.1 -u root` instead of `mysql -h localhost -u root`. If you omit the host (`mysql -u root`), the MySQL client will implicitly use `localhost`. 2. For SingleStore Helios, change the `socket` value in the `/etc/mysql/my.cnf` file to the location of your SingleStore Helios socket file as shown in the example below: ``` [client] port = 3306 socket = /var/lib/memsql/data/memsql.sock ``` ## ERROR 2026 (HY000): SSL connection error: SSL\_CTX\_set\_default\_verify\_paths failed **Issue** This error occurs when the incorrect path is provided for the ca-cert-pem file when using the `--ssl_ca` flag in the connection string to the SingleStore Helios node. **Solution** The solution is to verify you are using the correct path to the ca-cert.pem file. ## ERROR: Distributed Join Error. Leaf X cannot connect to Leaf Y. **Issue** When a distributed join occurs, the leaves within the cluster must reshuffle data amongst themselves, which requires the leaves to connect to one another. If the leaves are not able to communicate with one another, and a distributed join is touching those leaves, the distributed query will not run successfully. The inter-leaf communication needed for distributed join queries relies on the DNS cache on each leaf. If this cache is out of sync with the current state of the leaves, the distributed join will fail. **Solution** Use the following steps to troubleshoot this scenario: 1. Confirm you are able to access SingleStore Helios from one leaf to another in the cluster. This will eliminate network connection issues. > **📝 Note**: You are able to connect manually from one leaf to another because doing so does not utilize the DNS cache on the leaf. 2. Run `SHOW LEAVES` on an affected leaf (e.g. leaf X) in the cluster. The `Opened_Connections` columns should reveal what leaves the affected leaf has open connections with. Verify that leaf Y is not in this list. 3. When leaves connect to each other, they cache connection information (leaf-1 is at IP 192.0.2.1, leaf-2 is at IP 192.0.2.2, etc.). If the IPs of these leaves ever change the cache will not automatically update. This will ultimately result in an unsuccessful connection attempt because the other leaves in the cluster are using old IP address information. The solution is to flush the DNS cache and connection pools on all affected nodes. You can do so by running the following: `sql FLUSH HOSTS; FLUSH CONNECTION POOLS;` `FLUSH HOSTS` clears the DNS cache on the node. This must be performed all affected nodes in the cluster. `FLUSH CONNECTION POOLS` shuts down all existing connections and closes idle pooled connections. *** Modified at: November 22, 2022 Source: [/cloud/reference/troubleshooting-reference/network-errors/](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/network-errors/) (An index of the documentation is available at /llms.txt) --- # Operations Errors ## ERROR 2625: Operation 'BACKUP DATABASE' can't be executed on internal database('') utilized by SingleStore. **Issue** This error occurs when you attempt to backup the `cluster` or `memsql` databases. **Solution** These databases are internal system databases, and as such, they cannot and do not need to be backed up. ## ERROR: Partition’s table metadata are out of sync for table **Issue** The most common cause for this error is an `ALTER TABLE` query on a sharded table which ran partially but didn’t complete successfully. This may leave table metadata in an inconsistent state where some partitions have the old metadata while some have the new. **Solutions** If the cause of the issue was a failed `ALTER TABLE` as described above, possible solutions are: * Recreate the table. This can be done by insert-selecting the data into a newly created table or reloading the data. For example: `sql CREATE TABLE t_copy AS SELECT * FROM t; DROP TABLE t; ALTER TABLE t_copy RENAME AS t;` * Find the partition(s) with inconsistent table metadata, and manually run `ALTER` on the leaf partitions to correct them. *** Modified at: August 28, 2023 Source: [/cloud/reference/troubleshooting-reference/operations-errors/](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/operations-errors/) (An index of the documentation is available at /llms.txt) --- # Query Errors ## ISSUE: Long Running Queries Blocking DDL Operations and Workload SingleStore supports execution of online DDL statements, which means that you can perform read and write operations while the DDL command is being executed on a table. The execution of a DDL command does not begin until all the in-progress DML queries running on the table finish. As soon as the in-progress read and write operations complete and the DDL command begins executing, new reads and writes proceed as normal. This blocking period usually lasts on the order of milliseconds. If there are long running DML queries in your workload, it can extend the blocking period because the DDL must wait until those queries complete. The DDL waits and blocks until the `default_distributed_ddl_timeout` is reached (default value is 180000 milliseconds or 3 minutes), after which it throws a timeout error. If you frequently run DDL statements and have a lot of long-running queries, then it negatively impacts the workload since the DDL operation blocks other queries from starting while it waits for completion of long-running queries. Increasing the timeout duration for a DDL operation is not a recommended option since it delays the start of the operation. While the DDL operation is delayed from starting, it will continue to block new queries from executing either. Therefore, raising the timeout can cause longer delays in regular queries in your workload. **Solution** * Optimize your workload. If you have long running queries, then you can optimize your current workload by running frequent maintenance and backups, or by using a different workload. For example, run the analytical queries in your workload during the lowest traffic time. Additionally, you can use cluster replication to replicate the data to a secondary read-only cluster, and then run all long running or analytical queries on the secondary cluster. This drastically reduces the workload on the primary cluster and improves workload performance. ## In this section * [ERROR 2408 (HY000): ER\_COMPILATION\_TIMEOUT: Query compilation timed out and cannot be executed](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-2408-hy-000-er-compilation-timeout-query-compilation-timed-out-and-cannot-be-executed.md) * [ERROR 1064 ER\_PARSE\_ERROR: Unhandled exception Type: ER\_PARSE\_ERROR (1064)](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-1064-er-parse-error-unhandled-exception-type-er-parse-error-1064.md) * [ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-1205-hy-000-lock-wait-timeout-exceeded-try-restarting-transaction.md) * [ERROR 1706 (HY000): Feature Multi-table UPDATE/DELETE with a reference table as target table is not supported by MemSQL](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-1706-hy-000-feature-multi-table-update-delete-with-a-reference-table-as-target-table-is-not-supported-by-memsql.md) * [ERROR 1706 (HY000): Leaf Error (127.0.0.1:3307): Feature INSERT IGNORE … ON DUPLICATE KEY UPDATE is not supported by MemSQL](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/error-1706-hy-000-leaf-error-127-0-0-1-3307-feature-insert-ignore-on-duplicate-key-update-is-not-supported-by-memsql.md) * [Why do I get errors about UNIQUE KEYs?](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/why-do-i-get-errors-about-unique-keys.md) *** Modified at: November 22, 2022 Source: [/cloud/reference/troubleshooting-reference/query-errors/](https://docs.singlestore.com/cloud/reference/troubleshooting-reference/query-errors/) (An index of the documentation is available at /llms.txt) --- # Release Notes ## SingleStore Helios Release Notes New features added to SingleStore Helios listed by month. ## July 2026 ## Enhancements * Creating and deleting delegated entities now requires the `Manage Delegated Entities` permission, and viewing delegated entities requires the `View` permission. By default, the `Manage Delegated Entities` permission is included in the `Owner`, `Restricted Owner`, `Operator`, and `Restricted Operator` predefined cluster group roles. Refer to [Cloud Workload Identity and Delegated Entities](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities.md) for more information. * Removed password validation checks for users configured with `REQUIRE SUBJECT`. ## Bug Fixes * Fixed an issue that may stall reference partition replication when replicating from versions earlier than 8.9 to version 8.9 or later. * Fixed an issue in pipelines with offset metadata garbage collection enabled that may cause a deadlock. * Updated Jackson dependencies and `jq` to address multiple CVEs in Java components and container images. ## June 2026 ## Features **Continuous Backup**: This feature continuously and transparently stores all changes in object storage without affecting running workloads and enables self-serve recovery within the retention window. It provides assurances against bad updates or accidental data loss without interrupting running applications. **I/O Optimized Clusters**: This feature increases provisioned IOPS (input/output operations per second) and TP (throughput) for persistent cache and is ideal when optimizing SingleStore for I/O sensitive workloads. **Stage**: On the Cloud Portal, Stage now provides a unified page under **Ingestion** where users can view Stage files across all clusters from a single place. With project/cluster filters and visible Stage storage location, it simplifies file management for ingestion workflows and gives users better visibility into where the data is stored. ## Enhancements * Improved Kafka pipelines caching to use a per-partition, per-pipeline scope. This reduces resource contention across pipelines. * Added support for tracing DDL activity, allowing users to audit recent DDL statements across the cluster via Query History and the `MV_TRACE_EVENTS` information schema view. Refer to [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) for more information. * Improved `IN`-subselect rewrite handling by not adding `ISNULL()` checks for non-nullable columns. ## Bug Fixes * Fixed an issue in `JSON_MERGE_PATCH()` where key comparisons included the surrounding quotation marks, which returned incorrect results when JSON object keys shared a prefix. Key comparisons now exclude the surrounding quotes. * Fixed an issue in the `MV_ACTIVITIES_CUMULATIVE` information schema view related to the `ACTIVITY_TRACKING_ID` column and removed warnings that may interfere with queries on the view. * Updated Netty dependencies to address multiple CVEs in Java-based components. * Fixed a crash that occurred when a subselect is used in the `WHERE` predicate of a `LOAD DATA` query. * Fixed a crash when writing to a view if the column names are cased differently from the column name in the view definition. * For invalid, NULL, or empty UDF names, DML queries are now routed to the default resource pool. * Fixed incorrect results for correlated `EXISTS` subqueries that contain `TABLE(SPLIT)`. * Updated third-party dependencies to address multiple Common Vulnerabilities and Exposures (CVEs), including fixes for the Go standard library, Netty, Python 3, glibc, OpenSSL, Kerberos, systemd, Apache Commons Configuration, and Apache Thrift. ## May 2026 ## Enhancements * Improved query plan printing efficiency, lowering query compilation overhead. * Log detailed, step-by-step timings for each operation within `BATCH_PARTITION_TIME`. This behavior is enabled with the `pipelines_extractor_debug_logging` engine variable. * Update Golang to version 1.26.2. ## Bug Fixes * Fixed a bug that may lead to a crash or undefined behavior during query execution. * Fixed a bug that may cause asynchronous LLVM compilation to fail. * Fixed a bug that prevented the table creator from querying it. * Fixed a bug that potentially caused queries using the `BM25_GLOBAL()` function Global Full-Text statistics. * Updated to Temurin OpenJDK 11 JRE 11.0.31+11 to fix CVEs in OpenJDK. * Prevent messages with multiple nested "Leaf Error"s. Added the `allow_nested_leaf_errors` engine variable which disables the new behavior and reverts to prior behavior allowing nested "Leaf Error" messages. * Fix CVE vulnerabilities in Java third-party packages. ## April 2026 ## Features **Expanded Capacity Upgrade**: Updates now provision additional temporary capacity instead of reducing capacity when applying enhancements and bugfixes. This reduces update-related performance impact to running workloads and delivers a smoother and more transparent update process. **Online AZ Migration**: This feature enables you to reconfigure a deployment’s availability-zone (AZ) configuration between single AZ and multi AZ without application downtime. With this feature users can now improve resilience against AZ failures or optimize cost with minimal operational disruption. ## Enhancements * Added proper refresh handling for STS tokens during long-running operations. * Allowed Out-of-Memory (OOM) threads to exit early when waiting for the garbage collection mechanism to provide memory buffers. * Added the ability for an optimizer rewrite to merge a derived table with subselects; unwrapping a derived table with subselects is now configurable. * Upgraded librdkafka to version 2.13.2. * Improved join selectivity estimation for predicates of the form `col1 = IFNULL(col2, col3)` by leveraging column cardinality statistics. * Added the ability to avoid group by placement rewrites when grouped tables have no join predicates, improving query plan stability for multi-table aggregates. ## Bug Fixes * Fixed a bug where an optimization on `GROUP BY` fields was incorrectly applied when grouping by a typecast expression. * Improved optimizer handling of Out-of-Memory (OOM) conditions. * Reverted a change to the `JSON_EXTRACT_BIGINT` function that could cause incorrect results or system crashes.  * Allowed Out-of-Memory (OOM) threads to exit early when waiting for the garbage collection mechanism to provide memory buffers. * Fixed a filesystem (FS) pipeline error that occurred when files were deleted while the pipeline was still running. * Fixed an issue where query-time variables (QTVs) referenced in common table expressions (CTEs) caused a "no such table" error. * Fixed a nullability issue in result table columns of an outer join that may lead to a leaf crash. * Fixed an issue where the `PIPELINES_ICEBERG_FILES` view crashed if pre-upgrade partitions were present in the metadata. ## March 2026 ## Features **Manage Projects**: This feature adds Helios management workflows around the project abstraction so users can view and manage project groupings for resources, not just create resources inside them. Users can now apply project filtering to the UI to more easily access and manage separate workloads, and larger customers can keep environments organized, visible, and permissioned at the appropriate scope. **Read Replicas (Smart)**: This feature delivers independent and isolated read-replicas on shared data leveraging SingleStore’s zero-copy data fabric. This simplifies the process of leveraging existing application databases to power additional workloads, without managing complex data movement or ETL, all while delivering independent resource usage and isolation for each workload. **Cluster and Database UI**: Updated [Cloud Portal](https://portal.singlestore.com) UI experience so users can browse resources, view relationships, and perform common actions more directly from the UI. This simplifies day-to-day administration of shared databases, improves search and filtering, and enables upcoming features leveraging SingleStore’s zero-copy data fabric. ## Enhancements * Added support for the `IF EXISTS` option in the `DROP PROJECTION` command. * Extended the range of values for the `row_count` hint to allow values up to the maximum float value. * Added engine variable `skip_sharding_warmup_query`. This variable disables sending single-partition queries to all leaves to warmup. * Added `SNAPSHOT_ID` and `SNAPSHOT_TIMESTAMP` to `PIPELINES_ICEBERG_FILES` information schema view. * Updated the `MV_COLUMNAR_SEGMENT_INDEX` information schema view to return the correct `INDEX_NAME` for multi-value hash indexes on JSON/BSON columns enabling accurate per-index size reporting. * Added column `ACTIVITY_TRACKING_ID` in the `MV_ACTIVITIES`, `MV_ACTIVITIES_CUMULATIVE`, `MV_ACTIVITIES_EXTENDED`, and `MV_ACTIVITIES_EXTENDED_CUMULATIVE` information schema views. * Updated the `KILLED_QUERIES` counter in Resource Governor pools to include queries terminated due to exceeding `QUERY_MEMORY_PERCENTAGE`. * Improved performance of queries on `information_schema.MV_CLUSTER_STATUS` through RPC parallelization. * Improved `JSON_EXTRACT_BIGINT` performance by performing encoding-level extraction instead of full JSON decoding. * Improved CDC pipeline error reporting. Multiple parsing errors now display detailed messages, including the associated DDL statement. ## Bug Fixes * Fixed an issue that caused duplicate repartition on join between derived table and reference table. * Fixed an issue where `SHOW GRANTS` no longer displayed `REQUIRE SSL` status after upgrade. * Fixed an issue where the `ADD AGGREGATOR` command failed for node IDs greater than 16384. * Fixed vulnerabilities in aircompressor, jackson-core, and avro. * Fixed an issue where the system allowed creation of semantically identical multi-value indexes on the same column or JSON path. * Fixed an issue where `json_collation` was not synchronized with `character_set_server` after upgrading from version 8.5 or earlier to version 8.7 or later. * Removed password validation for `ALTER USER` statements that do not modify the password. * Fixed an issue where security built-in functions like `CURRENT_SECURITY_ROLES()` and `CURRENT_SECURITY_GROUPS()` returned incorrect results when used within a `CREATE TABLE AS SELECT` (CTAS) statement . * Fixed missing value for `Type` column in `information_schema.USERS` view. * Fixed a crash where one process queried the `MV_PROCESSLIST` view while another updated the status being collected. ## February 2026 ## Features **Custom RBAC Roles**: Enables administrators to define role-based permissions beyond the built-in roles and manage access to Helios resources at discrete levels. Teams can model least-privilege access more precisely around real job functions instead of relying only on predefined role sets. **Projects**: [Projects](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions/#section-id235175723125225.md) group resources for a specific workload, with the edition set at the project level and resources assigned to a project when they’re created. Organizations can now separate workloads cleanly, manage multiple editions in one organization, and give teams clear administrative boundaries. ## Enhancements * Result tables now explicitly set table-level collation. ## Bug Fixes * Fixed a crash that occurred when using `PERCENTILE_CONT()` with a constant `ORDER BY` clause. ## January 2026 ## Feature **AWS Marketplace Billing**: Lets customers subscribe to SingleStore Helios through AWS Marketplace, register/link their AWS account to a SingleStore organization, and have usage metered back to AWS. This simplifies procurement and billing through existing AWS purchasing and invoicing processes instead of requiring a separate vendor workflow. ## Enhancements * Backfilled index type and index size of vector index metadata during information schema queries. * Introduced background IO submission thread for sync durability and batching multiple IO requests. * Disabled prewritten ingest transformations by default. * Added support for an alter node port. * Extended the `TABLE()` function to have an ordinality column when used with `WITH ORDINALITY` clause. * Added new column `INDEX_NAME` to the `VECTOR_INDEX` information schema view. * Added the `FUNCTION_NAME` column to the `MV_PROCESSLIST` information schema view. When a query is invoked from a stored procedure, this column will display the stored procedure name. * Added support for vended credentials in Iceberg pipelines with S3 storage. * Added display of original line number when re-raising exceptions with `RAISE` statement in stored procedures. * Engine now displays NULL for vector index size and type for vector indexes created in versions older than 9.0. ## Bug Fixes * Fixed a crash when pipelines insert data into writable views. * Fixed a bug that caused stack overflow errors when executing complex queries with multiple joins in `compile_lite` and `compile_lite_first` interpreter mode. * Fixed an internal query conversion issue that caused a mismatch in vector join decisions between the aggregator and leaf nodes. * Updated Golang dependencies to address known Common Vulnerabilities and Exposures (CVEs) in container images and related components. * Fixed double accounting of memory via buffer allocators (`Alloc_spill`, `Alloc_arena`). Now, if memory is used from cache memory, total server memory remains unused. * Fixed a bug that caused a crash when a built-in Table-Valued Function (TVF) table is joined with a table with a join condition involving `JSON_ARRAY_CONTAINS_`. * Fixed a bug in parsing when `LATERAL JOIN` is used with a Table-Valued Function (TVF) with a recursive Common Table Expression (CTE). * Fixed an issue in stored procedures where the callstack is incomplete for nested `RAISE` statements and only displays the last frame instead of all the `RAISE` statement frames. * Fixed an issue promoting demoted voting members when a database does not have a R/W attachment. * Fixed a Common Table Expression (CTE) preprocessing bug involving nested CTEs within `INTERSECT` and `EXCEPT` queries. * Fixed an issue in which a scale operation on a cluster with a R/O attachment of a database could result in temporary tables being dropped from the R/W attachment of the same database in another cluster. * Fixed the affected row count for `SELECT .. FROM .. into FS/S3 ...` queries when row count is greater than 32-bit integer limits. * Fixed a stack corruption bug exposed in very large queries. * Fixed an issue where memory allocated during the spilling phase was not following resource governor memory limits. ## December 2025 ## Features **SCIM (Okta & Entra ID)**: This feature automatically provisions and deprovisions users and teams from Okta or Entra ID into SingleStore Helios via SCIM, with identity-provider changes synced to the database. Onboarding, offboarding, and team-based access can be managed centrally through the identity provider instead of manually in Helios. **mTLS**: Enables SingleStore to authenticate clients with mutual TLS certificates during connection setup, validating both sides against trusted certificate authorities and optionally binding access to a specific certificate identity. Replaces shared-secret-only authentication with stronger trust-based access and rejects untrusted clients before login completes. Refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md) for related information. ## Highlights * Added the ability to skip heuristic checks when using projection for JSON documents for `ORDER BY` self join rewrites. * Added support for increasing the length of variable-length string type columns using `ALTER TABLE MODIFY COLUMN` on columnstore tables. * Allowed users that do not require X509 (mTLS) to connect regardless of the `ssl_ca_for_client_cert` value and presence of the file it points to (if set). * Updated vector index and search to use internal memory allocator instead of malloc. * Added an optimizer rewrite to push `LIMIT`s below external User Defined Functions (UDFs). This optimization can be turned off with the engine variable: `optimizer_disable_limit_pushdown_below_external_udf`. * Added a check to `CREATE` or `ALTER USER` that verifies that the Subject in the client certificate provided during a mutual Transport Layer Security (mTLS) connection matches the subject specified in the `REQUIRE SUBJECT` clause. * Added support for `COUNT()` in aggregated views. * Limited the vector index cache size to keep it from getting to close to the maximum server memory. * Added the TLS cipher and application name to the auditlog. * Improved phonehome security by adding remote endpoint verification. * Upgraded the Almalinux version to Almalinux 10. * Added `ssl_system_capath` engine variable that defines the system path to the certificate directory. * Added collection of enhanced query stats from kill query Recursive Common Table Expressions (RCTE) writer statistics and sorted iterator error cases. * Added Unix timestamp versions of `start_time` and `event_time` fields in event trace samples exposed by monitoring exporter. * Added option to merge credentials with external secret storage. * Made Resource Governance aware of cgroup limits in Kubernetes clusters. The minimum of host cores and cgroup cores is now used as the CPU limit for governing resource pools. * Modified ingestion to skip `NULL` records while using `SKIP ALL ERRORS` and `SKIP PARSER ERRORS` in Kafka pipelines. * Added a new `BLOB_COMPUTE_ID` column for `COLUMNAR_SEGMENTS` and `MV_COLUMSTORE_FILES` information schema views. * Added a new column `MEMORY_USAGE_BYTES` to `LMV_RESOURCE_POOL_STATUS` and `MV_RESOURCE_POOL_STATUS` information schema views. * *Bug Fixes* - Resolved a crash that could occur when reading the `VECTOR_INDEX` information schema view.- Fixed an issue with lateral join. - Fixed vulnerabilities in Netty and OpenJDK CVEs. - Fixed charset UI to show `*_bin` as default collation rather than `*_general_ci`. - Fixed an issue in `ORDER BY` self-join rewrite.- Fixed Transport Layer Security (TLS) connectivity issue when using .NET connection pool. - Fixed an issue with JSON attribute validation for mutual Transport Layer Security (mTLS) in `ALTER USER`.- Disabled the optimize JSON computed column feature. - Fixed an aggregated view crash caused by aliases. - Fixed an issue where unsuccessful distributed transactions could lead to increased memory usage on aggregators. - Fixed `RESTORE DATABASE` in highly-available master aggregator deployments. - Fixed an issue where `information_schema.MV_CLUSTER_STATUS` queries would be recursively forwarded to the master aggregator. - Updated the zookeeper version. - Resolved a crash that could occur when a `TABLE()` function appeared inside a subquery that was part of an `OR` or `AND` condition (for example nested `IN` and `EXISTS` clauses).* Fixed incorrect aggregation of leaf query stats in Query\_completion tracing. * Fixed results for `NOT IN` subselects when the left input table has NULL values. * Fixed a bug with some `NOT IN` subselect query shapes. * Fixed logic for `NOT IN` subselects when nested under `OR`s. * Fixed an issue that caused a stalled cluster in the presence of many concurrent `DROP TEMPORARY TABLE` commands. * Fixed memory tracking for some queries which were not being tracked properly, result tables to be specific. * Fix correctness when using `NOT IN` for some query shapes. ## October 2025 ## Features Feature: **Sequences** - The addition of a `SEQUENCE` modifier for `AUTO_INCREMENT` ensures that a unique sequence of numbers is generated, while greatly reducing the difference between the generated numbers. Additionally, the `SEQUENCE` modifier allows you to set the column type for `AUTO_INCREMENT` columns to other integer types and specify custom starting values. Refer to [AUTO\_INCREMENT AS SEQUENCE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table/#section-idm234966854099799.md) for more information. Feature: **Automatic Query Reoptimization with Feedback** - This feature, also referred to as [Feedback Reoptimization](https://docs.singlestore.com/cloud/query-data/query-tuning/feedback-reoptimization.md) (FR), uses statistics from previous query runs to improve query plans and performance. FR reoptimizes queries using statistics collected during query execution so the query optimizer can generate plans based on actual runtime data instead of estimates. Feature: **Multi-value Index for JSON** - The new [multi-value hash index](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json.md) enables 100x faster analytics performance on JSON arrays. By indexing each value in arrays in JSON documents, the new index avoids scanning large JSON collections and improves query performance for `JSON_MATCH_ANY` queries. Feature: **Load Data Wizard** - The new wizard for loading data simplifies the data ingestion process and allows users to import data from AWS S3, local CSVs, MySQL, PostgreSQL, Oracle, Microsoft SQL Server, and Snowflake without code. These wizards offer an intuitive, step-by-step interface that guides users through the loading process, minimizing complexity and reducing setup time. Feature: **Pipeline Monitoring** - The [Pipeline Dashboards](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring/#section-idm4579546351372833664771243305.md) enhance user experience by providing real-time insights into data ingestion processes and resources consumed by pipelines. Users can easily access metrics related to pipeline performance, including data speed, row counts, and resource consumption to quickly diagnose issues and optimize their workflows. Feature: **Shared Edition Clusters** - This feature provides low-cost development clusters for non-production workloads. Providing core engine features, these shared clusters offer a cost-effective alternative for developing new applications and allowing easy migration to production when ready. Feature: **AWS On-demand Billing** - Customers can now conveniently pay for SingleStore on-demand directly through AWS, simplifying the billing process and integrating seamlessly into their existing cloud frameworks. This feature provides customers unified cost management and flexible usage. Feature: **Job Scheduling with Cron**: This feature allows users to schedule notebook code as jobs using Linux’s cron syntax. Jobs can be scheduled at specific times and on specific schedules and provides reliable control over data workflows. Feature: **Notebook Versioning** - [This feature](https://docs.singlestore.com/cloud/container-services/notebooks/notebook-version-history.md) allows users to easily track and manage the evolution of their notebooks through a Git-like interface. This functionality mitigates the risk of data loss and removes the hassle of manual version management. Enhancement: **Database Branching on GCP** - Database branching offers Git-style agility for your data. You can branch, experiment, and roll back data changes in moments, not hours. You can also create an instant, independent, and updatable clone of your primary database to safely and easily develop prototypes, run performance tests, and conduct feature tests. Database Branching was already available on AWS and is now available on GCP. Enhancement: **Enhanced Query History** - The updated [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) feature provides users with a powerful tool for monitoring and optimizing SQL query performance. This feature enables high-frequency event tracing, allowing users to identify which queries consume substantial resources or fail during execution and then trace and debug those query executions. Enhanced metrics like CPU time, disk time, network time, various lock times, and more are now included. Preview Feature: **Flow on SingleStore Helios** - [Flow](https://docs.singlestore.com/cloud/load-data/load-data-with-singlestore-flow-on-helios.md) is a no-code data transfer and CDC tool tailored for ingesting data into SingleStore from databases including Oracle, SQL Server, MySQL, PostgreSQL, and Snowflake. Preview Feature: **Distributed Plancache (DPC)**  - The [Distributed Plancache](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache.md) adds a third layer to the plancache, supplementing the existing in-memory plancache and on-disk persistent plancache (PPC). The DPC allows nodes to share plans. Nodes newly added to the cluster will start with a warm plancache instead of building one from scratch. The DPC improves query performance and reduces CPU usage after operations like scaling or rebalance. Plans compiled on one aggregator will be asynchronously synced to other aggregators so that plans do not need to be compiled separately on each aggregator. * Enabled synchronous Distributed Plancache lookup during query execution. * Distributed Plancache pool size adjustment fails gracefully. * Prevented the drop of Distributed Plancache files on regular Persistent Plancache GC. Preview Feature: **GPU-powered Notebooks** - This feature allows you to run complex data processing tasks in Python that take advantage of higher CPU and GPU compute for both interactive and job workloads. Preview Feature: **Cloud Functions** - This feature offers Lambda-style serverless functions. [Cloud Functions](https://docs.singlestore.com/cloud/container-services/cloud-functions.md) enable the deployment of serverless applications alongside database services, allowing developers to run custom code with the same performance and security as their database transactions. Cloud Functions simplifies application development by eliminating the overhead of traditional infrastructure management. Preview Feature: **Container App RBAC** - Aura Container Apps now support role-based access control (RBAC) for security and access management, which allows you to define access controls for deployed container apps. Preview Feature: **Continuous Backup and Online PITR on GCP** - [Continuous backups](https://docs.singlestore.com/cloud/manage-data/continuous-backups-and-restore.md) and online [Point-in-Time-Recovery (PITR)](https://docs.singlestore.com/cloud/manage-data/database-storage/online-point-in-time-recovery-pitr.md) allow you to back up and recover instantly, without disrupting live workloads. PITR provides self-service and instant recovery to any point in time within the retention period. Preview Feature: **Smart DR on GCP** - The [Smart DR](https://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr.md) feature in Helios is an advanced disaster recovery (DR) solution that supports flexible and predictable cross-region disaster recovery for effortless failover and failback. Smart DR is also cost effective as no idle compute is required. Smart DR is Public Preview on AWS and is now Public Preview on GCP. Preview Feature: **Projects** - Projects allow organizations to deploy multiple product editions—such as Shared, Standard and Enterprise—simultaneously from a single account, across different cloud service providers and regions. This enables flexible, hierarchical management and centralized billing, allowing teams to seamlessly scale deployments across departments and workloads without the limitations of earlier edition choices. ## Changes in Default Behavior ## Collation Change The default collation for new (9.0+) clusters has been changed to `utf8mb4_bin` from `utf8mb4_general_ci` to improve default performance of string comparisons, which can substantially improve query speed. The `utf8mb4_bin` collation is case-sensitive and provides performance advantages when case-sensitive comparisons are required. * For existing clusters that are upgraded to 9.0+, the collation is not changed. * For new (9.0+) clusters, `utf8mb4_bin` is the default collation. * The default collation for the `utf8` and `utf8mb4` charsets has been changed. If a charset, e.g. `utf8mb4` is used in new (9.0+) clusters without an explicit collation, the default collation will be `utf8mb4_bin` instead of `utf8mb4_general_ci`. * Refer to [Specifying Character Set and Collation for Clusters](https://docs.singlestore.com/cloud/reference/sql-reference/character-encoding/specifying-character-set-and-collation-for-clusters.md) for additional information. ## Enhancements ## Full-text Search * Added full-text pushdown optimization that works with any filter in the `WHERE` clause. * Added a new metadata table to track full-text search indexes eligible for merging. * Added support for custom stopword lists for [full-text search version 2](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md). * Added new nori (Korean) analyzer customizations for full-text search version 2. ## Iceberg * Implemented support for automatically generated column mappings (for primitive data types and structures) as part of Iceberg schema inference. * Added support for EKS IRSA with assumeRole for Iceberg pipeline authorization. * Added support for file filtering based on the WHERE clause in the pipeline declaration. Refer to [Iceberg Ingest](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md) for more information. ## JSON * Introduced [Multi-Value Hash Index](https://docs.singlestore.com/cloud/create-a-database/multi-value-hash-index-json.md) for JSON columns. This index optimizes key-path and array-value lookups in JSON columns. * Extended a rewrite to allow merging derived tables with JSON functions. * Added a new [JSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-array.md) function. * Added support for JSON arrays in pipelines loading JSON files. Each JSON record in the array is loaded as a separate row. * Enabled the use of the `MATCH_ANY` tokenizer option with underscore in multi-value index settings. * Added charset validation for JSON type values in external functions. * Added support for the JSON data type in external functions. ## Observability * Improved performance of profiled query plans by introducing a new profile codegen mode. * Included aggregator activity statistics in Query\_completion event traces. * Added `activity_tracking_id` field in activity structure to track the resource consumption by pipelines. * Updated the default values of `enable_auto_profile` and `auto_profile_type` engine variable to `ON` and `SMART`, respectively. * Added the `SHOW BOTTOMLESS_PLANCACHE STATUS` command that returns observability metrics for the distributed plancache. * Added estimates for query memory usage based on static row count and row size estimates to improve Workload Manager (WM) memory management. * Added an internal allocator `Alloc_connection_context` which tracks certain per-connection allocations that were previously tracked using the standard allocator. * Added an option to run queries that failed asynchronous compilation, allowing users to introspect the compilation results. * Added logging for LRU compiled unit eviction. ## Pipelines * Added support for the `PIPELINE ` clause to the [SHOW PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) command which allows you to view profile information for the specified pipeline. * Added support for `SHOW PROFILE PIPELINE` on Helios. * Added the `CREATE_USER` and `ALTER_USER` fields in the `PIPELINES` information schema view. For more information, refer to [PIPELINES](https://docs.singlestore.com/cloud/reference/information-schema-reference/data-ingest/pipelines.md). * Added a `HEADER DETECTION` `ON`/`OFF` clause to control inference from CSV files. For more information, refer to [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md). * Added support for inferring Parquet files with the `INFER PIPELINE` command. For more information, refer to [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md). * Added support for inferring Iceberg schemas automatically on pipeline creation with the `INFER PIPELINE` command. For more information, refer to [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md). * Added `SHOW CREATE LINK` command that allows you to view the `CREATE` statement for the specified link. * Added support for `TIME(0)`, `TIME(6)`, `DATETIME(0)`, and `DATETIME(6)` types for the `CAST` command for use with computed columns and the `SET` clause for use with pipelines and `LOAD DATA`. * Added support to infer CSV files with a single column when the file contains no field terminators in any record. * Implemented exponential backoff sleep between batch retries. * Added support for metadata garbage collection for Azure pipelines. * Enhanced [Pipeline Dashboards](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring/#section-idm4579546351372833664771243305.md) to include items such as speed of execution, number of batches processed, data transferred, and lag over time. ## Query Execution * Added Join Memory Reduction Optimization. This optimization is a new query optimization feature designed to help prevent memory overflow (spilling) during hash joins on large datasets. When enabled, it automatically rewrites qualifying joins into smaller, manageable pieces—reducing memory pressure and the risk of out-of-memory (OOM) errors. This feature is especially useful when joining large tables on high-cardinality columns, where traditional hash joins could exceed memory limits. Added engine variables `cardinality_threshold_for_join_split_rewrite` and `target_group_size_for_join_split_rewrite` to control the behavior of this optimization. Refer to [Join Memory Reduction Optimization](https://docs.singlestore.com/cloud/query-data/query-tuning/join-memory-reduction-optimization.md) for details.  * Added segment metadata that allows the engine to determine if columns are fully enclosed in (or pass) a filter and allows the engine to do segment elimination on queries with an `IS NULL` filter. * Added the `CHARACTER SET ` and `COLLATE ` clauses to the `ALTER TABLE` command that allows converting the character set and collation of a table, respectively. * Added support for parameterized `IN`-lists in BSON multi-value index optimization enabling improved performance for queries with same-type constant literals and single-argument built-in expressions. Previously limited to hex literals. * Disabled the pushdown of `INSERT SELECT` queries to leaf nodes for shard key affecting columns. * Enabled expression pushdown from join into hash builds on materialized CTEs. * Enabled the rewriting of correlated subselects that depend on more than one outer table with correlation conditions other than `=` and `AND`. * Extended predicate pushdowns to identify more cases. * Extended rewrite for `EXISTS` subselects to handle arbitrarily nested `EXISTS` subselects under `AND` or `OR`. * Added the ability to ignore join selectivity estimates of zero. ## Security * Enabled mutual TLS authentication between the client and server. * Added support for manual password activation and expiration to the `ALTER USER` command. * Added the `password_expiration_mode` engine variable. When set to `LIMITED_ACCESS`, users are allowed to log in even after password expiration and run password update commands, such as `ALTER USER` or `SET PASSWORD`. * Internal roles and groups cannot be deleted anymore. * Passwords are now masked in extractor subprocess logs. * A warning is now raised for every query if the user's password is expiring soon. ## SQL Editor * Autocomplete provides real-time, context-aware query completion as the SQL query is written. This functionality helps users quickly identify relevant tables, views, and columns, and also helps reduce syntax errors. * Multi-tab provides multiple tabs in the SQL Editor so you can better organize your SQL scripts and queries. ## Vector Search * Added the `VECTOR_INDEX` information schema view which shows the amount of memory and disk used by each vector index. * Added a fallback mechanism so that when the engine detects that an indexed vector search followed by filters yields fewer than `LIMIT` rows, the engine falls back to a full table scan to ensure sufficient rows are produced. The fallback mechanism can be disabled by setting the engine variable `vector_index_fallback_non_index_scan` to FALSE. * Extended the `VECTOR_SUM` aggregate function to support the `VECTOR` data type. * Added a vector index cache to limit the amount of memory used by vector indexes. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for details. ## General Updates * Added events to indicate the start and end of asynchronous upgrades. * Added the `ENSURE_ONLINE_NODES` clause to the `REBALANCE ALL DATABASES` and `REBALANCE PARTITIONS` commands which allows rebalancing to run only if all the leaf nodes are online. * Allowed background merger to run while `OPTIMIZE TABLE` is running. * Connected the distributed plancache to the plancache manager. * Disabled unnecessary collection of block statistics. * All distributed plancache operations are now asynchronous. * Added support for reusing autostats file on recovery. * Added support for spilling if input data has NULL rows. * Aggregators now periodically synchronize query plans across the deployment. * `DROP RESOURCE POOL` command cannot delete the current resource pool anymore. * The engine now waits until asynchronous upgrade steps complete before taking a snapshot. * Added support for NULL type in Wasm UDF return types and parameters. * Added support for expressions, builtin functions, and user-defined variables in addition to hex literals in Multi-Value index for BSON. Refer to [Multi-Value Hash Index (BSON)](https://docs.singlestore.com/cloud/reference/sql-reference/bson-functions/multi-value-hash-index-bson.md) for more information. * Extended redundant `GROUP BY` detection to outer join predicates. * Users with database visibility can now query the `*_BOTTOMLESS_*` information schema views in the respective database. `CLUSTER` permissions are no longer enforced for querying these views. * Columns with extended types now use the correct default table collation. * Enabled correlating fields in subselects to reference outer selects more than one level up. * Enabled skipping of non-alterable tables from asynchronous DDL queries. * Updated to use a consistent snapshot while performing a shallow table copy. * Disabled segment elimination for `IN` clauses for which the left-hand expression is not a table column. * Improved the error message returned when invalid key is specified in the CONFIG/CREDENTIALS JSON. * Improved the error message for parameter count mismatch during query parsing. * Users with `SHOW METADATA` privilege can now access the `LOAD_DATA_ERRORS` information schema view. * Added support for delimited batch sets in external functions. * Added support for the `LIMIT` clause in prepared statements. * Computed column definitions now support the `SPLIT` function. * The `DROP` and `ALTER TABLE` commands no longer have to wait for the plan garbage collector. * Improved the garbage collection behavior for plancache. * Introduced support for placeholders for partition ID and timestamp in the `SELECT INTO ... filename` command. * Added Lucene logs to the cluster report. * Added statistics related to AWS S3 lookup latency. * Added knob to disable cardinality estimates on JSON/BSON columns during analyze. ## Engine Variables This update adds the following new engine variables: * `disk_plan_gc_pause_seconds_on_startup`: Disables disk plan garbage collection on startup. This variable prevents hot plans from being unintentionally disk garbage collected. * `disk_plan_gc_pause_minutes_for_detached_tables`: Specifies the number of seconds to cache detached table modules before eviction from disk. * `enable_gc_events`: Enables reporting of garbage collection events. Gates the `MV_GC_EVENTS` information schema view that provides visibility into garbage collection passes. * `info_schema_show_null_dbs`: Allows non-privileged users to view the rows in `*_BOTTOMLESS_*` information schema views where associated database name is NULL. * `managed_functions_batch_size`: Controls the batch size for managed external functions including Python UDFs and AI/ML Functions. * `optimizer_disable_analyze_cardinality_on_json`: Controls if the cardinality of JSON columns is collected by autostats in `PERIODIC` mode. * `optimizer_max_table_row_count_as_default_cardinality`: Controls the maximum estimated table row count to be used as the per column cardinality when no column-level statistics exists. * `password_expiration_mode`: When set to `LIMITED_ACCESS`, users are allowed to log in even after password expiration and run password update commands, such as `ALTER USER` or `SET PASSWORD`. * `regexp_error_handling_level`: Controls the behavior of error handling for regexp operations. * `scheduler_slow_loop_seconds`: Specifies the threshold for triggering the verbose logging of scheduler thread timing. * `ssl_ca_for_client_cert`: Support for client certification. * `suppress_dml_warnings`: Suppresses all DML query warnings. * `vector_index_fallback_non_index_scan`: Controls a fallback mechanism so that when the engine detects that an indexed vector search followed by filters yields fewer than `LIMIT` rows, the engine falls back to a full table scan to ensure sufficient rows are produced.  * [Distributed Plan Cache](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache.md): Added the following engine variables to control this feature: `enable_distributed_plancache`, `distributed_plancache_worker_threads`, `distributed_plancache_max_download_plans`, `distributed_plancache_agg_sync_s`, `enable_periodic_distributed_plancache_agg_sync`, `enable_synchronous_dpc_lookup`. * [Join Memory Reduction Optimization](https://docs.singlestore.com/cloud/query-data/query-tuning/join-memory-reduction-optimization.md) : Added `cardinality_threshold_for_join_split_rewrite` and `target_group_size_for_join_split_rewrite` to control the behavior of this optimization. ## Information Schema This update adds the following information schema views and columns: * `MV_ACTIVE_METADATA_VERSIONS`. Provides visibility into active metadata transaction read versions. * `MV_COLUMNSTORE_ACTIVE_MERGE_PLANS` and `LMV_COLUMNSTORE_ACTIVE_MERGE_PLANS`. Show internal details on currently running merge plans. * `MV_GC_EVENTS`. Provides visibility into garbage collection passes. This view is gated behind the preview feature and engine variable `enable_gc_events`. * `MV_TABLE_COUNTERS` and `LMV_TABLE_COUNTERS`. Show cumulative per-table metrics, i.e., the amount of work performed by the background merger. * `VECTOR_INDEX`. Shows the amount of memory and cache used by each vector index. * Added the `UPLOAD_BLOB_LEAD_PAGES` column to `information_schema.MV_BOTTOMLESS_SUMMARY`. Ensures that uploading log chunks to the remote storage does not keep falling behind. * Added more metrics to `information_schema.MV_CLOUD_PER_COMPUTE_REMOTE_STATS` and `information_schema.MV_CLOUD_PER_STORAGE_REMOTE_STATS`. Tracks retention log chunks and snapshots. * Added the `ATTACH DATABASE`, `DETACH DATABASE`, `CREATE DATABASE`, and `DROP DATABASE` columns to `information_schema.MV_EVENTS`. * Added the `CREATE_USER` and `ALTER_USER` fields to `information_schema.PIPELINES`. * `source_type` is now included as part of the `config_json` in `information_schema.pipelines`. * *Bug Fixes* * Fixed a case where long running background work was blocking in-memory garbage collection of database metadata. * Fixed a memory allocation bug during query compilation. * Limited the number of retries on plan compilation errors; this fix limits resource usage. * Fixed a deadlock in result table creation during reshuffle with the always-false filter. * Fixed a crash when a `BM25` function in a subselect referenced a table in the outer select. * Fixed a crash during replay when a table uses an FTS index with JSON options. * Fixed a potential replay failure on a table that uses full-text search version 2. * Fixed how `JSON_PRETTY` builtin function escapes characters. * Fixed a bug that caused `JSON_TO_ARRAY` join queries to return wrong results where the quotes on strings were removed incorrectly. * Fixed using delimited batch sets with JSON format. * Fixed performance regression for queries involving JSON string extraction (e.g. `JSON_EXTRACT_STRING` and `JSON_ARRAY_CONTAINS_STRING`) in situations where there are a large number of JSON values. * Fixed incorrect aggregation of leaf query stats in Query\_completion tracing. * Fixed an issue that caused an error if column names had trailing spaces in `CREATE PIPELINE` statements. * Fixed an issue where monitoring pipelines would sometimes fail to produce data for large source clusters. * Fixed a wrong result in `ORDER BY LIMIT` query that skipped rows with NULL values in the sort key. * Added support for the `IF NOT EXISTS` clause to the `CREATE VIEW` command. * Added support for table-level collations. * Improved optimization speed of parameterized `IN`-lists by limiting traversal depth. * `OPTIMIZE TABLE ... INDEX` is now more responsive to `KILL QUERY` statements. * Enabled using the estimated table row count as default column cardinality for better join estimation. * Added support for PSQL ARRAY type arguments to the `TO_JSON` built-in function. * Removed the `CAST` operator if a redundant cast is performed. * Improved error handling for regular expressions: errors are now properly raised instead of returning SQL NULL. To avoid breaking existing apps the `regexp_error_handling_level` engine variable was added. This variable controls the behavior of error handling for regexp operations. * Fixed an issue where concurrent DDL and `PROMOTE AGGREGATOR TO MASTER` statements were not forwarded to the Master Aggregator. * Fixed the projection DDL output in the debug profile. * Fixed an error named `ER_BAD_TABLE_ERROR` on `UPDATE` or `DELETE` queries with a specific shape. * Fixed an issue in CTE query rewrites caused when a CTE referenced a recursive CTE. * Fixed the predicate pushdown logic for materialized common table expressions. * Fixed query performance issue caused by heuristic estimate selectivity being overwritten. * Fixed row count estimation for union subquery containing Materialized Common Table Expression (MCTE)s. * Resolved an issue where the regular expression engine would return SQL NULL when hitting the memory limit during execution. It now raises an appropriate error instead. * Fixed resource pool and workload management CPU limiting. * Fixed a locking bug related to table module eviction during recovery. * Fixed a deadlock in a rare scenario related to low memory conditions. * Fixed a rare race condition which caused a node to get stuck in offline mode after recovery. * Fixed an issue that led to undefined behavior if a node failed concurrently with the computation of the replication distribution tree. * Fixed a race condition that caused autostats file leak while reprovisioning. * Fixed incorrect autostats in a rare failover case during asynchronous replication. * Fixed a race condition in autostats between table refresh and table deletion. * Fixed a race condition that could cause shutdown to hang. * Fixed a rare crash in synchronized variables initialization. * Fixed a bug in the deadlock avoidance algorithm that could lead to deadlocks. * Fixed a bug that caused nodes to crash when `ALTER TABLE` queries ignored foreign keys. * Fixed a rare deadlock related to temporary tables. * Fixed a crash in `SHOW PROCESSLIST` in a rare race condition scenario. * Fixed a bug in CTE rewrites. * Fixed a leak in a rare scenario. * Fixed a bug in parser. * Fixed a crash in a shallow copy of a table with a pending ALTER operation. * Fixed an issue where IN-list factorization did not work with newline characters around the IN-list. * Fixed an issue where periodic autostats was disabled when it should not have been. * Fixed a crash in lockfree hashtable when the engine is under high memory pressure. Reports an "out of memory" error instead. * Fixed a bug in lateral join. * Updated third party libraries to fix security vulnerabilities. * Increased the \_bt timeout from 60 seconds to 180 seconds. * Implemented shallow copy of a table with a dropped index, the shallow copy now includes the in-memory segment. * Fixed incorrect aggregation of leaf query stats in Query\_completion tracing. * Fixed an issue where certain commands (e.g. DROP TABLE) could hang due to aborted queries caused by stuck collocated services triggered by external functions. * Fixed a bug in the detection of shard key matches in the window function partition by list. * Fixed an error in backup subprocess and enhanced subprocess tracing to use structured logging. ## September 2025 ## Highlights * **Enhancement**: Added the ability to ignore join selectivity estimates of zero. * **Enhancement**: Enabled using the estimated table row count as default column cardinality for better join estimation. Added session variable `optimizer_max_table_row_count_as_default_cardinality` which controls the maximum estimated table row count to be used as the per column cardinality. * *Bugfixes* - **Bugfix**: Fixed a case where long running background work was blocking in-memory garbage collection of database metadata. - **Bugfix**: Fixed a memory allocation bug during query compilation. - **Bugfix**: Fixed a bug where large queries can crash the engine in `compile_lite` and `compile_lite_first` interpreter modes. - **Bugfix**: Fixed stack overflows in `compile_lite` and `compile_lite_first` interpreter mode. - **Bugfix**: Fixed a bug with some `NOT IN` subselect query shapes. - **Bugfix**: Limited the number of retries on plan compilation errors; this fix limits resource usage. - **Bugfix**: Fixed a deadlock in result table creation during reshuffle with the always-false filter.* **Bugfix**: Fixed an issue with Java 11 path (`java_pipelines_java11_path` engine variable) configuration while deploying SingleStore. * **Bugfix**: Fixed a bug in the detection of shard key matches in the window function partition by list. * **Bugfix**: Fixed using delimited batch sets with JSON format. * **Bugfix**: Fixed an issue where the `ALTER TABLE` command could hang and become unkillable due to a type conversion error occurring during the alter operation. ## August 2025 ## Highlights * **Enhancement**: Removed the `CAST` operator if a redundant cast is performed. * **Enhancement**: Changed the approach for delimited batch sets. Delimited batch sets now return and accept original rowIDs. * **Enhancement**: Added a `suppress_dml_warnings` global engine variable to suppress all DML query warnings. * **Enhancement**: Added `ATTACH DATABASE`, `DETACH DATABASE`, `CREATE DATABASE`, and `DROP DATABASE` to `information_schema.MV_EVENTS`. * **Enhancement**: Added support for metadata garbage collection for Azure pipelines. * **Enhancement**: Added knob to disable cardinality estimates on JSON/BSON columns during analyze. * **Enhancement**: Added the `managed_functions_max_connections` engine variable which specifies the maximum number of HTTP connections per node for managed external functions, such as Python User Defined Function (UDF)s. * *Bugfixes* * **Bugfix**: Fixed an error in backup subprocess and enhanced subprocess tracing to use structured logging. * **Bugfix**: Fixed a crash when BM25 in a subselect references a table from the outer select. * **Bugfix**: Fixed an issue where certain commands (e.g. `DROP TABLE`) could hang due to aborted queries caused by stuck Collocated Services triggered by External Functions. * **Bugfix**: Fixed performance regression for queries involving JSON (like `JSON_EXTRACT_` builtins). * **Bugfix**: Fixed query performance issue caused by heuristic estimate selectivity being overwritten. * **Bugfix**: Updated third party libraries to fix security vulnerabilities. * **Bugfix**: Fixed an issue where monitoring pipelines would sometimes fail to produce data for large source clusters. * **Bugfix**: Fixed row count estimation for union subquery containing Materialized Common Table Expression(MCTE)s. ## July 2025 ## Highlights * **Enhancement**: Added support for fully enclosed optimization on nullable columns and segment elimination on `IS NULL` filter. * **Enhancement**: Updated the bundled OpenSSL version to 3.1.2. * *Bugfixes* * **Bugfix**: Fixed results for `NOT IN` subselects when the left input table has `NULL` values. * **Bugfix**: Fixed incorrect results in an `ORDER BY LIMIT` query, caused by an issue where rows with null values in the sort key may be skipped. * **Bugfix**: Bumped the \_bt timeout from 60 seconds to 180 seconds. * **Bugfix**: Fixed logic for `NOT IN` subselects when nested under `OR`s. * **Bugfix**: Fix correctness when using `NOT IN` for some query shapes. * **Bugfix**: Fixed incorrect aggregation of leaf query stats in Query\_completion tracing. ## June 2025 ## Highlights * **Enhancement**: Added Lucene logs to the cluster report. * **Enhancement**: Users with database visibility can now query the `*_BOTTOMLESS_*` information schema views in the respective database. `CLUSTER` permissions are no longer enforced for querying these views. * **Enhancement**: Improved the error message for parameter count mismatch during query parsing. * **Enhancement**: Disabled segment elimination for `IN` clauses for which the left-hand expression is not a table column. * *Bugfixes* * **Bugfix**: Fixed a bug in lateral join. * **Bugfix**: Fixed the predicate pushdown logic for materialized common table expressions. * **Bugfix**: Fixed an issue where periodic autostats was disabled when it should not have been. * **Bugfix:** Fixed crash in lockfree hashtable when engine is under high memory pressure. Report out of memory error instead. * **Bugfix**: Fixed a leak in a rare scenario. * **Bugfix**: Fixed the `ER_BAD_TABLE_ERROR` error on `UPDATE` or `DELETE` queries with a specific shape. * **Bugfix**: Fixed a crash in a shallow copy of a table with a pending `ALTER` operation. * **Bugfix**: Fixed a bug in CTE query rewrites. * **Bugfix**: Fixed a potential replay failure on a table that uses full-text version 2. * **Bugfix**: Fixed an issue where `IN`-list factorization did not work with newline characters around the `IN`-list. ## May 2025 ## Highlights * **Enhancement**: Extended a rewrite to allow merging derived tables with json functions. * **Enhancement**: Changed the default blob cache size to 75% of the total disk for small disk config. * **Enhancement**: Enabled rewriting correlated subselects that depend on more than one outer table with conditions other than equality. * **Enhancement**: Prevented the destination table from being replaced by projection on update or delete. * **Enhancement**: Renamed GIN index to Multi-Value Hash Index. GIN keyword deprecated. * **Enhancement**: Enable the use of the `MATCH_ANY` tokenizer option with underscores in GIN index settings. * **Enhancement**: Added support for parameterized  `IN` -lists in BSON GIN index optimization enabling improved performance for queries with same-type constant literals and one-argument built-in expressions. Previously limited to hex literals. * **Enhancement** :New variables (Note, these should only be set with guidance from SingleStore. * `repl_page_desc_limit_before_stall`: limits the number of page descriptors in the page map that a 'ReplLog' can use before stalling new txs. See 'ReplLog::StallForPageMap'. The default 256K page desciptor limit means the hash table will not have load greater than 4. * `repl_io_buffer_limit`: limits the number of IO buffers that a 'ReplLog' can use before stalling new txs. See 'ReplLog::StallForPageMap'. 128 IO buffers is 16MB since each IO buffer is 128K. * `[dr_]repl_network_buffer_size`: replication tries to set both send and receive buffer sizes to reduce waits on TCP ack packets. 8MB is a sane battle-tested value. Can be set for both DR and non-DR replicas. * `[dr_]repl_max_chunk_size`: when sending logs or snapshots to a replica limits chunk size to this value. Too long of a chunk increases the chance of timeout waiting for the replica's response packet (since the replica must process the entire chunk before acking). This defaults to 8MB - a sane limit chunk size to this value. Maximum value is 1GB and it must be divisible by 128kb so it is a multiple of io buffer size. Can be set for both dr and non-DR replicas. * **Enhancement**: Added more metrics to the views,  `MV_CLOUD_PER_COMPUTE_REMOTE_STATS`  and  `MV_CLOUD_REMOTE_STATS` , for tracking of retention log chunks and snapshots. * **Enhancement**: Added support for stopwords `token_filter` for custom analyzers for FTS VERSION 2. * *Bugfixes* - **Bugfix**: Fixed a bug that caused filter-aware optimizations to not be applied above MCTE when involved in hash joins. - **Bugfix**: Fixed a bug that occurred when CTEs are rewritten. - **BugFix**: Fixed resource pool and workload management CPU limiting. - **BugFix**: Fixed an isue to use full table scan instead of vector index scan if enough rows are not produced. - **BugFix**: Fixed a wrong results bug in  `JSON_TO_ARRAY ()` join queries where the quotes on strings were incorrectly removed. - **BugFix**: Fixed a rare deadlock involving temp tables.* **BugFix**: Fixed invalid optree from rewrite on union queries matching a very specific shape. ## April 2025 ## Highlights * **Enhancement**: Added support to the GIN Index for BSON for expressions, built-in functions, user-defined variables (UDVs), and typecasting. User-defined functions (UDFs) and non-deterministic built-in functions, such as rand(), remain unsupported. * **Enhancement**: Added estimates for query memory usage based on static row count and row size estimates to improve Workload Manager (WM) memory management. * **Enhancement**: Added a `KILL` option for the `DETACH DATABASE` command. * **Enhancement**: General improvements and optimizations for `CHECK BOTTOMLESS CHECKSUM` command. * **Enhancement**: Added support for JSON arrays in pipelines loading JSON files. Each JSON record in the array is loaded as a separate row. * **Enhancement**: Improved optimization speed of parameterized `IN`-lists by limiting traversal depth. * **Enhancement**: Added support for delimited batch sets in external functions. * **Enhancement**: High Availability for the Master Aggregator (HA for MA) is now enabled exclusively for Enterprise edition. * **Enhancement**: `OPTIMIZE TABLE ... INDEX` is now more responsive to `KILL QUERY` statements. * *Bugfixes* - **Bugfix**: Fixed an issue where a node that had not been upgraded could trigger undefined behavior if the node's `processlist` was queried while the node was processing an internal RPC. - **Bugfix**: Fixed a `NULL` value inline bug by adding typecast operations above the referencing field. - **Bugfix**: Fixed an issue that could lead to undefined behavior when a node failed concurrently with the computation of the replication distribution tree. - **Bugfix**: Fixed a bug that caused `DROP DATABASE` to hang in a rare condition. - **Bugfix**: Fixed an issue that returned wrong results in pipelines in a rare race condition. - **Bugfix**: Fixed an issue that caused a crash when a query referenced too many tables. - **Bugfix**: Changed options for loading libpam.so. - **Bugfix**: Fixed a race condition that caused `DETACH DATABASE` to hang. - **Bugfix**: Fixed a buffer overflow issue that occurred while de-parameterizing a multi-column `IN`-list filter. - **Bugfix**: Fixed an issue to prevent LRU eviction from stalling when several table modules are stale. - **Bugfix**: Fixed a bug in `JSON_TO_ARRAY()` join pushdown optimization that occurred when the JSON value being extracted from `table_col` is a JSON array.* **Bugfix**: Fixed `ALTER RESOURCE POOL` to respect queue depth constraints. * **Bugfix**: Fixed a bug that produced incorrect results when performing a hash join with a condition that compares mismatched types where one of the types is BSON. * **Bugfix**: Fixed a crash caused by libgcc btree bug. * **Bugfix**: Reduced contention on a global lock controlling modules, affecting code loading and unwinding of exceptions. * **Bugfix**: Removed garbage collection from cleaning up arrangements when such garbage collection is already in process. * **Bugfix**: Disabled vector range search filter during `JOIN`s to avoid an engine crash. ## March 2025 ## Highlights * **Enhancement**: Added an engine variable `optimizer_disable_subselect_to_join_cte_preprocess` which can be used to disable the subselect to join rewrite for CTE preprocessing before inlining or materialization. * **Enhancement**: Added an engine variable `optimizer_enable_merge_unioned_queries_rewrite` which can be used to enable or disable the rewrite that merges union queries. This variable is `ON` by default. * **Enhancement**: Added aggregator activity stats in `Query_completion` event tracing. Refer to [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) for details. * **Enhancement**: Added the ability for a user to change the login password when the variable `password_expiration_mode` is set to LIMITED\_ACCESS. Now the user is allowed to login even after the password expires but can only execute commands that update the password such as `ALTER USER` or `SET PASSWORD`. * **Enhancement**: Added a password expiry warning message that is raised each time the user executes a query. This warning starts appearing 14 days before the password actually expires. * **Enhancement**: Added auditlogging mode for only the root user. It is a startup-only variable similar to other audit logging configuration variables and should be set along with the `auditlog_level`. * **Enhancement**: Added support for the `LIMIT` clause in prepared statements. * **Enhancement**: Added an option to run queries that failed asynchronous compilation, allowing users to introspect the compilation results. * *Bugfixes* - **Bugfix**: Fixed an issue impacting creating or altering a resource pool with a CPU percentage limitation after a MA failover.- **Bugfix**: Fixed CTE subselect to join preprocessing in the `INTERSECT`/`EXCEPT` CTE case. - **Bugfix**: Reserved memory space for `segment_id` in the pseudo column's MemoryFile. - **Bugfix**: Fixed a bug that occurred when a query used `I0` as a column name. - **Bugfix**: Fixed a crash that was due to an issue with the rewrite that merges union queries.* **Bugfix**: Fixed a bug that caused incorrect results when running `INFER PIPELINE CSV`. * **Bugfix**: Fixed a rare deadlock occurring when DETACH DATABASE is run. * **Bugfix**: Fixed a bug that caused a node to crash when the variable `ignore_foreign_keys` is ON and the `ALTER TABLE` command is used to add a foreign key. * **Bugfix**: Changed the output projection DDL to `CREATE PROJECTION` instead of `CREATE TABLE _$_table_name_$` in debug profile. * **Bugfix**: Fixed an issue where DDL statements that were concurrent with PROMOTE AGGREGATOR TO MASTER did not get forwarded to the new Master Aggregator. * **Bugfix**: Fixed an issue to prevent a crash when `JSON_TO_ARRAY` optimization is run against non-nullable JSON columns. * **Bugfix**: Fixed a bug to prevent a crash when using FTS v2 with CTE. ## February 2025 * **Enhancement**: Added an internal allocator `Alloc_connection_context` which tracks certain per-connection allocations that were previously tracked using the standard allocator. * **Enhancement**: Added a vector index cache to limit the amount of memory used by vector indexes. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for details. * **Enhancement**: Extended the [VECTOR\_SUM](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-sum.md) aggregate function to support the `VECTOR` data type. * **Enhancement**: Computed column definitions now support the `SPLIT` function. * **Enhancement**: The `DROP` and `ALTER TABLE` commands no longer have to wait for the plan garbage collector. * *New or Modified Engine Variables* Refer to [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for information on each of the following engine variables.- **Enhancement**: Added an engine variable `disk_plan_gc_pause_seconds_on_startup` that disables disk plan garbage collection on startup. This variable prevents hot plans from being unintentionally disk garbage collected. * *Bugfixes* * **Bugfix**: Fixed the `userDictionary` parameter for the korean full-text tokenizer. * **Bugfix**: Fixed a crash that occurred when aggregator functions are used with `VECTOR` type data inside other built-ins. * **Bugfix**: Fixed a crash occurring when a table-valued function (TVF) column is used in a `WHERE` clause without a wrapping `JSON_EXTRACT` function in a `TABLE(JSON_TO_ARRAY())` join. * **Bugfix**: Fixed a bug that caused CDC-in pipelines to fail while inferring the table schema, with the "Failed to allocate slot in the extractor pool" error. * **Bugfix**: Fixed a deadlock between `ALTER` and failover in a rare race condition. * **Bugfix**: Fixed a critical issue where clusters were failing to connect to AWS remote storage due to CURL request timeouts. * **Bugfix**: Fixed a bug where a CDC pipeline gets stuck while waiting for data. * **Bugfix**: Fixed a bug that prevented proper error handling on socket timeout. * **Bugfix**: Fixed a bug in `JSON_ARRAY_CONTAINS_STRING` query rewrite for query shapes with the `LIMIT` clause. * **Bugfix**: Fixed support for plan pinning in `IN` list factorization. ## January 2025 ## Highlights ## Full-Text Search - Analyzers and Tokenizers Full-text search using SingleStore's VERSION 2 full-text index has been enhanced with support for custom analyzers and tokenizers. With this enhancement, SingleStore full-text indexes can be created to support languages other than English, for text that contains emails and URLs, with custom whitespace processing, and more. The n\_gram tokenizer, which breaks words into [n-grams](https://en.wikipedia.org/wiki/N-gram), sequences of n adjacent symbols is included. The full set of Apache Lucene analyzers and tokenizers is supported. Refer to [Full Text VERSION 2 Custom Analyzers](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/full-text-version-2-custom-analyzers.md) for more information. ## Full-Text Search - Enhanced BM25 Scoring Full-text search using SingleStore's VERSION 2 full-text index has updated BM25 scoring functionality.  The `BM25` function has been enhanced with support for boolean and boost queries, phrase and proximity search queries, and queries over multiple columns. Refer to [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) for more information and examples. A new function, `BM25_GLOBAL`, has been added to provide BM25 scoring across all partitions. With this new function, all rows in a table are scored together; collection and term statistics are calculated for a table, ensuring accurate scores relative to all rows in a table. The `BM25_GLOBAL` function augments the existing `BM25` and `MATCH` functions, and it is more accurate and more expensive than both of these functions. Refer to [BM25](https://docs.singlestore.com/cloud/reference/sql-reference/full-text-search-functions/bm-25.md) for more information on the `BM25_GLOBAL` function. ## Iceberg Continuous Ingest Added support for continuous ingest of data from Iceberg tables. Upsert and append-only workloads are supported. In addition, manual upserts with the `CREATE OR REPLACE` command are supported. Refer to [Iceberg Ingest](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md) for more information. ## Iceberg - New Catalogs Added support for Snowflake, REST, JDBC, Hive, and Polaris Catalogs for Iceberg Ingest using pipelines. Refer to [Iceberg Ingest](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md) for more information. ## Enhanced Disk Spilling Added disk spilling for `RIGHT` and `FULL OUTER JOIN`. ## Writable Views Writable views allow users to run `UPDATE`, `INSERT`, and `DELETE` queries on views. To enable writable views, set the `enable_writable_views` global variable to `1`. Query the `information_schema.VIEWS` view to inspect if a view can be updated. Refer to [CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view/#section-idm234617498191001.md) for more information. ## Other Improvements and Fixes ## Vector Index on Nullable Column Vector Indexes can be created on columns that are nullable. Prior to this improvement, vector indexes could only be created on columns that were declared `NOT NULL`. With this improvement, a user can insert a row containing text and a `NULL` vector value into a table with a vector index. The user can subsequently obtain a vector embedding for the text and update the row with that vector value. The updated value will be added to the vector index. ## Vector Index Memory Tracking The memory used by vector indexes can be tracked using the `alloc_vector_index` metric which is now available in [SHOW STATUS EXTENDED](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-status-extended.md). Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) and [Tuning Vector Indexes and Queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/tuning-vector-indexes-and-queries.md) for more information. ## Vector Index Merger Added the vector index merger which combines per-segment vector indexes to create a cross-segment vector index and improve the performance of vector search queries. Refer to [Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing.md) for more information. * *Other Performance Enhancements* * **Enhancement**: Added sub-segment elimination for flexible parallelism. Refer to [Flexible Parallelism](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/flexible-parallelism.md) for more information. * **Enhancement**: Improved the performance of `LOAD DATA` queries that include the `CHARACTER SET` clause. * **Enhancement**: Optimized full-text queries with `ORDER BY ... LIMIT` on a full-text score and that optionally filter on the same full-text clause. * **Enhancement**: Significantly improved the performance (\~20x) of certain JSON-based SQL queries and queries run using the Kai endpoint, when JSON objects contain arrays of sub-objects. This optimization reduces the need to normalize the data into multiple tables to achieve high analytics performance. Queries that expand JSON arrays (with or without any aggregations) and/or perform the following operations benefit from this optimization: * Group by a field outside the array (in the `GROUP BY` clause) * Filter on the fields in the array * *New Information Schema Views and Columns* * **Enhancement**: Added the `TABLE_NAME` column to the `LOAD_DATA_ERRORS` information schema view. `TABLE_NAME` is the name of the table associated with the error. * **Enhancement**: Added the `NODE_ID` column to the `MV_RECOVERY_STATUS` information schema view that specifies the ID of the node from which the database is being recovered. * **Enhancement**: Added the `MV_BOTTOMLESS_API_EVENTS_SUMMARY` information schema view that contains a summary of remote API calls made from the engine. Refer to [MV\_BOTTOMLESS\_API\_EVENTS\_SUMMARY](https://docs.singlestore.com/cloud/reference/information-schema-reference/unlimited-storage/mv-bottomless-api-events-summary.md) for more information. (8.7.20) * **Enhancement**: Added the following columns (metrics) to the `MV_BOTTOMLESS_STATUS_EXTENDED` and `MV_BOTTOMLESS_SUMMARY` information schema views: * `IS_BOTTLE_SERVICE_UP`: Indicates whether the bottle service is up. * `BOTTLE_SERVICE_UPTIME_DOWNTIME_SECS`: Specifies the minimum uptime when the bottle service is up and the maximum downtime when the bottle service is down. * *New Commands and Functions* * **New feature**: Added the following [Identifier Generation Functions](https://docs.singlestore.com/cloud/reference/sql-reference/identifier-generation-functions.md): * `UUID_TO_BIN` * `BIN_TO_UUID` * `IS_UUID` * **New feature**: Added a `SHOW FULLTEXT SERVICE METRICS` command that displays the diagnostic metrics for the JLucene full-text search in JSON format. Refer to [SHOW FULLTEXT SERVICE METRICS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-fulltext-service-metrics.md) for more information. * **New feature**: Added a `SHOW CDC EXTRACTOR POOL` command that displays information about the CDC-in pipelines. Refer to [SHOW CDC EXTRACTOR POOL](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-cdc-extractor-pool.md) for more information. (8.7.20) * **New feature**: Added a new `JSON_MERGE_PATCH` function that merges two JSON objects into a single JSON object. Refer to [JSON\_MERGE\_PATCH](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-merge-patch.md) for more information. * **New feature**: Added support for Lateral Join. Lateral join allows a subquery in the `FROM` clause of a SQL query to reference another table in that same `FROM` clause, which can simplify query syntax. Refer to [Lateral Join](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select/#lateral-join.md) for more information. * *New or Modified Engine Variables* Refer to [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for information on each of the following engine variables.- **Enhancement**: Added a new engine variable `scheduler_slow_loop_seconds` that specifies the threshold for triggering the verbose logging of scheduler thread timing. - **Enhancement**: Added a new engine variable `scheduler_slow_ready_queue_seconds` that specifies the threshold for triggering logging of slow ready queue draining. - **Enhancement**: Introduced global engine variable `pipelines_iceberg_data_workers_heap_size` to control memory consumption on leaves. - **Enhancement**: Added a new engine variable `enable_writable_views` that enables creation of writable views. Refer to [CREATE VIEW](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-view/#section-idm234617498191001.md) for more information. - **Enhancement**: Updated the minimum value of `json_document_max_children` engine variable to `1` (from `128` previously). - **Enhancement**: Added a new engine variable `enable_block_level_stats_collection` that controls the collection of block-level statistics for sub-segment elimination for flexible parallelism. - **Enhancement**: Added a new engine variable `enable_block_stats_use_in_query` that controls whether the block-level statistics are read and used during scan as part of sub-segment elimination for flexible parallelism. - **Enhancement**: Added a new engine variable `pipelines_iceberg_heap_size` to control heap size specifically for Iceberg pipelines. - **Enhancement**: Added `json_collation` global variable to control collation of JSON. The value of `json_collation` can be either `utf8_bin` or `utf8mb4_bin`. - **Bugfix**: During the upgrade to SingleStore 8.9, if the value of `fts2_max_connections` is equal to `100000`, it is set to `32`. * *Miscellaneous* - **Enhancement**: Improved pipeline error clearing. Additional errors are cleared when `CLEAR PIPELINES ERRORS` is run or when the `pipelines_errors_retention_minutes` limit is reached. - **Enhancement**: Improved performance of `OBSERVE` queries on columnstore tables. - **Enhancement**: Improved performance of `OBSERVE` queries by only decoding primary key or internal ID columns for columnstore segment deletes. - **Enhancement**: `OBSERVE` queries no longer block DDL operations while reading columnstore segments. - **Enhancement**: `OBSERVE` queries no longer require timeouts to prevent blocking database drop operations. - **Bugfix**: Fixed a bug in lateral join to preserve projection field aliases for lateral join subselects. - **Bugfix**: Fixed an issue that occurred when `OBSERVE` query was run on columnstore tables after `ALTER` operations.- **Enhancement**: Improve behavior of garbage collection for plancache. - **Bugfix**: Fixed a bug caused by using a (cross-segment) vector index ([Vector Indexing](https://docs.singlestore.com/cloud/reference/sql-reference/vector-functions/vector-indexing/#section-idm234571614690074.md)) with null vectors.- **Bugfix**: Fixes a data-dependent crash condition occurring in certain `TABLE(JSON_TO_ARRAY(...` joins. - **Enhancement**: Added new nori (Korean) analyzer customizations for Full-Text Search V2. - **Enhancement**: Added logging for LRU compiled unit eviction. - **Enhancement**: Fixed out of memory (OOM) errors and extra memory usage in Iceberg ingest. - **Enhancement**: Introduced new [JSON\_BUILD\_ARRAY](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-build-array.md) function. - **Enhancement**: Introduced support for placeholders for partition ID and timestamp in the `SELECT INTO ... file name` command. - **Bugfix**: Fixed a bug causing accumulation of .rem files on disk. - **Bugfix**: Fixed a small memory bug in columnstore scans that use the `JSON_TO_ARRAY` join optimization.- **Enhancement**: Added Korean language analyzer for full-text search V2. - **Enhancement**: Added support to infer CSV files with a single column when the file contains no field terminators in any record. - **Enhancement**: Relaxed dependency on partition count of leaf nodes for leaf plans.- **Bugfix**: Fixed out-of-memory errors and extra memory usage in Iceberg ingest.- **Bugfix**: Fixed a crash in `JSON_EXTRACT_STRING`. - **Bugfix**: Fixed a bug with updates and asserts using `JSON_ARRAY_CONTAINS_` predicates. - **Bugfix**: Fixed bottomless upload throttling criteria. - **Bugfix**: Blocked creation of temporary table as a shallow copy. - **Bugfix**: Resolved an issue where an in-development subsystem can leak files on disk- **Bugfix**: Fixed handling of heartbeat messages in the MongoDB® extractor in debug mode. - **Bugfix**: Fixed display of default BSON and string values with null-terminators in the information schema. - **Bugfix**: Removed trailing dot for decimal column types that have scale equal to 0. - **Bugfix**: Fixed `IN`-list index matching for columnstore tables when a query has multiple `IN`-list predicates. - **Bugfix**: Fixed a bug that caused an invalid optree error after the `JsonArrayContainsToTableBuiltin` rewrite.- **Bugfix**: Fixed a crash that occurred during spilling when executing a query with a large number of `GROUP BY` columns. - **Bugfix**: Fixed performance regression in Vector Search when using `DOT_PRODUCT` metric. - **Bugfix**: Added an optional 'swap\_time' argument to `UUID_TO_BIN` and `BIN_TO_UUID` functions. - **Bugfix**: Allowed cached table memory to be freed for empty tables in replica databases. - **Bugfix**: Fixed a lateral join parsing bug.* **Enhancement**: Added the `DETERMINISTIC` clause to the `CREATE FUNCTION` (UDF) command that instructs the query optimizer to assume that the created function is deterministic. Refer to [CREATE FUNCTION (UDF)](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) for more information. * **Enhancement**: Added support for the `IGNORE LINES` clause to the `INFER PIPELINE` command for CSV files. Refer to [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md) for more information. * **Enhancement**: Added support for using the `LOAD DATA` command to load files from a Stage without using pipelines. Refer to [Stage](https://docs.singlestore.com/cloud/load-data/load-data-from-files/stage.md) for more information. * **Enhancement**: Added support for using the `SKIP ALL ERRORS` clause during creation of Kafka pipelines for ingesting JSON formatted data. * **Enhancement**: Added support for `SKIP ALL ERRORS` and `SKIP PARSER ERRORS` clauses during creation of Kafka pipelines for ingesting Avro formatted data. * **Enhancement**: Added the ability to load Kafka properties and headers with the `get_kafka_pipeline_prop("")` function. * **Enhancement**: Added the ability to override the `pipelines_max_offsets_per_batch_partition` global variable for each Kafka pipeline using the `MAX_OFFSETS_PER_BATCH_PARTITION` pipeline variable in `CREATE PIPELINE` and `ALTER PIPELINE` commands. * **New feature**: Added support for the following parameters in the `CONFIG` clause of `CREATE PIPELINE AS ... LOAD DATA S3` statement: * `file_compression`: Decompresses files with the specified extensions. * `file_time_threshold`: Only ingest files modified after the specified timestamp. Refer to [S3 Configurations](https://docs.singlestore.com/cloud/load-data/data-sources/configuration-options-for-different-sources/#section-idm234488319463038.md) for more information. * **Enhancement**: Added the ability to use connection links to load Avro and Parquet formatted data stored in an AWS S3 bucket. * **Enhancement**: Enabled auto [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) for `INSERT...SELECT` and `REPLACE...SELECT` query shapes. * **New feature**: Added the `ENABLE_OVERWRITE` clause in the `SELECT ... INTO S3` and `SELECT ... INTO LINK` statements that enables the overwriting of existing files. Refer to [SELECT … INTO S3](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select/#select-into-s-3.md) for more information. * **Enhancement**: Updated [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) to show `number_of_blocks_tested_for_block_elim` and `number_of_blocks_eliminated_for_block_elim` for sub-segment elimination for flexible parallelism. * **Enhancement**: Added the ability to load CSV and JSON files from an Amazon S3 bucket using a `LOAD DATA` query. * **Enhancement**: Enhanced the `TABLE()` function to support `DISTINCT`. * **Enhancement**: Improved recovery time for tables with incremental autostats by recovering statistics from disk instead of rebuilding them from scratch. * **Bugfix**: Fixed an issue where attaching a leaf node to the Master Aggregator (MA) failed if the MA was still starting up. * **Enhancement**: Improved columnstore hash index performance on low cardinality column. * **Enhancement**: Added row count estimate for joins in the output of `EXPLAIN` and `PROFILE` queries. * **Enhancement**: Added support for `LIMIT` and `OFFSET` clauses in non-equality `WHERE` conditions in subselects. * **Enhancements**: Added support for `LATERAL` joins for table-valued functions (TVFs). * **Enhancement**: `SYNC DURABILITY` is always enabled for reference databases. Reference tables in user databases that use async durability may notice a decrease in performance for DDL and DML statements. * **Enhancement**: Added support for `HIGHLIGHT ... AGAINST` as a computed column expression when creating a new table. * **Enhancement**: Added support for sub-select to join rewrites for correlated subselects and nested scalar subselects. * **Enhancement**: Added support for null-accepting projections in scalar subselect queries. * **Enhancement**: Added support for `LATERAL` join subselects to reference any level of outer tables. * **Enhancement**: Changed the default collation to `utf8mb4_general_ci` and default character set to `utf8mb4` for `TEXT` and `ENUM` type columns for `CSV`, `JSON`, `AVRO`, and parquet formats in `INFER PIPELINE AS LOAD DATA` statements. * **Bugfix**: Fixed an issue that caused dangling compute sessions after a failed `ALTER DATABASE` command. (8.7.20) * **Bugfix**: Fixed an issue where concurrent `ALTER DATABASE` and `REBALANCE` operations create unlimited storage partitions with wrong compute ID. * **Bugfix**: Fixed a race condition between the transaction log garbage collection and database transition-to-master that could result in unrecoverable partitions. * **Enhancement**: Added support for multi-column `IN` list predicate in the `WHERE` clause of a query. * **Enhancement**: Function mapped `IN`-lists now share the same signature in the plancache for the same set of built-in functions. * **Bugfix**: Removed soft lock on the `CHARACTER SET` clause and added a warning to indicate invalid character set value in the `LOAD DATA` statement. * **Bugfix**: Database names can no longer end with big numbers, such as `db_`, to avoid conflicts with internal databases used in replication. * **Enhancement**: Each node in the cluster now validates the availability of bottle service every minute and records any consecutive failures in the `LMV_EVENTS` information schema view. * **Enhancement**: Added more information to the out-of-memory (OOM) errors. * **Bugfix**: Fixed an issue that occurred while parsing manifest files (associated with backup or restore operations) having more than 4096 characters. * **Bugfix**: Fixed an issue that caused duplication of storage blobs that have not been repaired yet for ongoing repair operations. * **Bugfix**: Fixed an issue in MySQL CDC-in pipelines where some MySQL tables with names containing `_` were not being replicated. * **Bugfix**: Fixed a race condition that caused shutdown to wait on idle async compile manager thread. * **Bugfix**: Fixed an issue where repair operation gets stuck while converting milestones. (8.7.20) * **Bugfix**: Fixed a synchronization issue where the database was not immediately available after some clustering operations. * **Enhancement**: Added the estimated number of partitions returned by the query optimizer for cost-based join order optimization to the debug profile. * **Enhancement**: Added support for `VECTOR` type in `REDUCE` built-in function. * **Enhancement**: Improved the lockdown message when changing collation or character set related engine variables within `utf8mb4` character set. * **Bugfix**: Fixed an issue with `PROMOTE AGGREGATOR ... MASTER` command where a restart at the end of the command, followed by manually finishing the promote operation, caused reprovisioning of the old Master Aggregator. * **Bugfix**: Fixed an issue that caused an empty network prefetch queue. * **Bugfix**: Fixed a distributed deadlock caused by a query blocking reference database reprovisioning in one part of the deadlock cycle. * **Enhancement**: Added the following headers to Data API responses: * `Cache-control: no-store` * `Strict-Transport-Security: max-age=31536000` * `X-Content-Type-Options: nosniff` * **Bugfix**: Fixed an issue with counting common table expressions (CTEs) references in a query. * **Bugfix**: Fixed a memory corruption issue caused by a rare race condition involving `MV_ACTIVE_TRANSACTIONS` information schema view. * **Enhancement**: Sharding planner now recognizes non-union style single partitioned derived table. * **Enhancement**: Disabled pipeline batches sample by default in `memsql_exporter`. * **Enhancement**: Improved message for an error where a global variable cannot be set because of a table sharded on a computed column. * **Enhancement**: Improved connection stability for MongoDB® CDC-in pipelines. * **Enhancement**: Upgraded the [librdkafka](https://github.com/confluentinc/librdkafka) library to version 2.4.0-3. * **Bugfix**: Fixed an issue where the Master Aggregator (MA) temporarily stopped behaving as the MA after being restarted. * **Enhancement**: Improved the performance of cluster operations through distributed plancache. * **Bugfix**: Fixed a bug with incorrect privilege checks. * **Enhancement**: Optimized row locking for internal transactions. * **Enhancement**: Improved performance of `REVOKE` in case of errors. * **Bugfix**: Fixed an engine crash caused when the Master Aggregator received a `REMOVE AGGREGATOR` query with its own `:`. * **Enhancement**: Reduced the chances of reference database reprovisioning in universal storage if a snapshot is taken concurrently with Master Aggregator shutdown. * **Enhancement**: `TO_JSON()` and `JSON_BUILT_OBJECT()` built-in functions now convert `VECTOR` type arguments to JSON array instead of a JSON string. * **Enhancement**: Added support for ZSTD compressed Kafka topics to Kafka pipelines. * **Bugfix**: Fixed an issue where `ClampTimestamp` spammed the tracelog. * **Enhancement**: Limit DR connection attempts to the primary node when it is failing connections, avoiding a substantial increase in `TIME_WAIT` sockets. * **Enhancement**: Improved error messages related to reprovisioning. * **Enhancement**: Implemented rotation and deletion of webproxy socket logs. * **Enhancement**: Ingest Kafka headers into the SingleStore table if they are included in the Kafka message. * **Enhancement**: Lockdown hints for common table expressions (CTEs). * **Enhancement**: Locked select and row count hints for multi-table views. * **Bugfix**: Fixed an issue where slow snapshots blocked clustering operations. * **Bugfix**: Fixed an issue where unrecoverable reference databases did not auto-heal for a prolonged period of time and get blocked, for example, by alter operations. * **Enhancement**: Improved the error message for `Correlated subselect that cannot be transformed and does not match on shard keys` errors. * **Bugfix**: Fixed data loss in sync durability in a rare race condition. * **Enhancement**: Improved processing of queries with redundant (superfluous) `EXISTS` subselects. * **Bugfix**: Fixed the reference count of committed blobs on upgrade. * **Enhancement**: Improved enforcement of `internal_columnstore_max_uncompressed_blob_size` engine variable. * **Enhancement**: Information schema queries are no longer case-insensitive with respect to database names. * **Bugfix**: Fixed a snapshot not found error. * **Bugfix**: Fixed an issue where the result of a seek operation was not handled correctly. * **Enhancement**: Detect scalar subselect requirement at runtime rather than rewrite time for certain cases. * **Bugfix**: Fixed a bug in full-text search query compilation. * **Enhancement**: Enabled `LOAD DATA` queries to use the specified `CHARACTER SET` to ingest [string type](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/string-types.md) fields. * **Bugfix**: Fixed an issue in the `OBSERVE` query that corrupted the blob cache when changes were captured from an unlimited storage database. * **Enhancement**: Improved performance of queries that use the `RAND()` builtin or constant expressions involving `PARTITION_ID`. * **Enhancement**: Added support for a query rewrite when `UNION` is used to merge queries that reference the same tables and have mutually exclusive filters. * **Enhancement**: Improved message for an error that occurs when `OBSERVE` queries are run without selecting a database. * **Bugfix**: Fixed a rare issue where `ALTER` operation and two-phase commit (2PC) caused index corruption. * **Bugfix**: Fixed an issue that caused a crash when the global collation or character was set to a value with the prefix "utf8mb3". * **Bugfix**: Fixed an issue that returned incorrect results when `JSON_MATCH_ANY()` and `BSON_MATCH_ANY()` queries contained nested `JSON_EXTRACT_()`, `JSON_MATCH_ANY()`, `BSON_EXTRACT_`, or `BSON_MATCH_ANY()` functions. * **Bugfix**: Fixed a rare undefined behavior issue caused by `SHOW CLUSTER STATUS` command. * **Bugfix**: Fixed an issue caused when multi-part names were used in common table expressions (CTEs) within UNION. * **Bugfix**: Fixed an issue that caused `DROP DATABASE` and some other cluster management commands to wait indefinitely for resources. * **Bugfix**: Fixed a rare issue that caused autostats file leak. * **Bugfix**: Fixed a rare issue that blocked kill query operations. * **Bugfix**: Fixed CVE-2024-45772 and BDSA-2024-0720 security vulnerabilities. * **Bugfix**: Fixed an error in `BSON_MATCH_ANY()` pushdown. * **Bugfix**: Fixed a bug that caused `OBSERVE` queries to reuse bad plans across database partitions. * **Enhancement**: Improved error message returned when an `OBSERVE` query is run while `enable_observe_queries` engine variable is disabled. * **Bugfix**: `OBSERVE` queries now return correct data when the `vector_type_project_format` engine variable is set as `JSON`. * **Bugfix**: Fixed the result of `ORDER BY LIMIT` query on tables with a sort key column defined in the descending order. * **Bugfix**: Fixed an issue that caused uneven time distribution in the CDC extractor pool. * **Bugfix**: Fixed an issue that delayed restart of nodes with multiple reference databases. * **Bugfix**: Fixed an issue where some point-in-time-recovery (PITR) operations failed when started from a restored snapshot. * **Enhancement**: Upgraded Apache Lucene search engine library to 10.0.0. * **Bugfix**: Fixed a bug that caused a query to fail in a specific situation when using `SEARCH OPTIONS` with vector index search. * **Bugfix**: Fixed an edge case; common table expressions (CTEs) are now recognized in subselect to join rewrite. * **Bugfix**: Fixed a bug that led to a master aggregator crash when using dynamic resource pools. * **Enhancement**: Added support for Iceberg v2 merge-on-read ingest mode. * **Bugfix**: Fixed ingest streaming logic for large Iceberg v2 tables. ## November 2024 * **Enhancement**: Improved out of memory (OOM) error message to make the message more accurate. * *Bugfixes* * **Bugfix**: Fixed a crash that could occur when columns were dropped with `ALTER TABLE`. * **Bugfix**: Fixed an issue where left join sometimes returned an incorrect result when spilling occurred. * **Bugfix**: Fixed an issue that could lead to new connections hanging. ## October 2024 * **Enhancement:** Improved the performance of regular expression (regex) operations. * **Enhancement:** Added a new filter derivation rewrite controlled by the `optimizer_not_null_filter_derivation` engine variable. * **Enhancement**: Improved the performance of `REPLACE` query into a columnstore table when table-level locking is triggered. * **Enhancement:** Added a variable, `max_autostats_update_workers`, to tune the maximum number of background autostats update workers. * **Enhancement**: Improved the Debezium DDL statements parser. * *Bugfixes* * **Bugfix**: Additional binlog position updates for CDC-in pipelines in the extractor pool queue. * **Bugfix**: Disable `JSON_ARRAY_CONTAINS_` rewrite in common table expressions (CTEs). * **Bugfix:** Fixed an edge case certificate refresh issue for connections to remote storage in long running clusters. * **Bugfix**: Improved the error message for CDC-in pipelines for insufficient permissions errors. * **Bugfix**: Fixed deadlock in database recovery due to evicting table module holding an open LOCK. * **Bugfix**: Fixed an issue where the garbage collection of a partition database's 2PC transaction table could unnecessarily delay the reprovisioning of the reference database. * **Bugfix**: Fixed an issue with alter repair where the repair job would be created before alter commits. * **Bugfix**: Fixed a race condition where node shutdown could get stuck if it happens concurrently with a partition failover. * **Bugfix**: Fixed a rare bug where `STOP PIPELINE` could write an old extractor cookie into pipeline metadata. ## September 2024 * **Enhancement**: Added single-parameter `JSON_EXTRACT` builtins. * **Bugfix**: Fixed a crash when using a unique key to upsert into a table when JSON column optimization is enabled. * **Bugfix**: Fixed a certificate refresh issue for connections to remote storage in long-running clusters. * **Enhancement**: Both `END AT TAIL` and the DDL gracefully end `OBSERVE` queries instead of producing an error. * **Enhancement**: Added a new `observe_agg_timeout_secs` global variable that sets the number of seconds an `OBSERVE` query can block on sending data before giving the consumer a network error. * **Bugfix**: Fixed an undefined behavior in the failure path of an `OBSERVE` query. * **Enhancement**: Where appropriate, the `DELETE` records for `OBSERVE` queries now populate the primary key for tables instead of the internal ID. * **Enhancement**: Removed the `TxPartitions` column and added a new `TxTimestamp` column for `OBSERVE`. * **Enhancement**: The internal IDs for `OBSERVE` are now unique across partitions. * **Enhancement**: `OBSERVE` now immediately flushes the result-set metadata to the client, eliminating the need to wait for rows to be returned to receive result metadata. * **Bugfix**: Fixed the failure to reset the socket-write timeout after an `OBSERVE` query has completed. * **Enhancement**: Reading segments in `OBSERVE` now respects `KILL QUERY`. * **Bugfix**: Fixed misleading `KILL QUERY` tracelog messages in `OBSERVE`. * **Bugfix**: Fixed connection reset errors when the value in the `observe_leaf_timeout_secs` engine variable was not used correctly with `OBSERVE`. * **Bugfix**: Fixed `BEGIN AT` in `OBSERVE` when the offset is in the middle of a snapshot on an unlimited storage database. * **Enhancement**: Added an `information_schema.OBSERVE_DATABASE_OFFSETS` view for obtaining various `OBSERVE` offsets. * **Enhancement**: Added support for resumable offsets into columnar segments for `OBSERVE`. The `OBSERVE` query now returns 28-byte offsets. While 24-byte offsets are still valid with `BEGIN AT`, the "strictly increasing" guarantee does not hold when comparing 24-byte and 28-byte offsets. * **Bugfix**: Fixed `OBSERVE` returning the wrong transaction ID (txId) when resuming from some transactions. * **Bugfix**: Fixed an issue where the output of `OBSERVE` would be incorrect when performing a delete on a large columnstore segment. * **Bugfix**: Fixed some cases of an incorrect transaction ID (txId) for snapshot records. `BeginSnapshot/CommitSnapshot` can now report nested `BeginTransaction/CommitTransaction` when there are unresolved distributed transactions while the snapshot is taken. * **Enhancement**: Added metrics to track the availability of unlimited storage and the bottle service. * **Enhancement**: Added support for query rewrite that enables hash joins when two tables are joined together using the `JSON_ARRAY_CONTAINS_` function. * **Enhancement**: Added bottle service reliability metrics to `MV_BOTTOMLESS_STATUS_EXTENDED`. * **Enhancement**: Fixed the `KILLED_QUERIES` counter in `MV_RESOURCE_POOL_STATUS`. * **Enhancement:** Added support for query rewrite for certain query patterns with `EXISTS` and subselects. * **Enhancement**: Added support for BM25 over multiple columns. * **Enhancement**: Added support for BM25 partition-scoped scoring for Boolean and boost queries. * **Bugfix**: Fixed a rare issue where a membership-related clustering operation could delay the re-provision or deletion of a cluster database. * **Enhancement**: Added the `json_collation` engine variable to control JSON collation, the value of which can either be `utf8_bin` or `utf8mb4_bin`. * **Enhancement**: Improved the logic for selecting vectors close to the distance threshold during vector range search. * **Bugfix**: Fixed an issue where `JSON_MATCH_ANY()` queries with `MATCH_ELEMENTS` evaluated incorrectly on non-array values. * **Enhancement**: Updated `JSON_MATCH_ANY()` simplified syntax to allow specifying `MATCH_ELEMENTS` by appending a '`*`' to the end of the keypath. * **Enhancement**: Added support for BM25 partition-scoped scoring for phrase and proximity search queries. * **Bugfix**: Fixed sporadic reading failures from `MV_SYSINFO_CPU`. * **Bugfix**: Clarified the error message produced in some cases where a snapshot gets aborted. * **Bugfix**: Fixed schema history in incremental snapshot mode for MySQL CDC. ## August 2024 * **New Feature**: Added the new `FULLTEXT SERVICE STOP` command. This command stops the full-text V2 service running on any node connected to the aggregator on which the command is run. Syntax: ```sql FULLTEXT SERVICE STOP ``` `FULLTEXT SERVICE STOP` requires the `PROCESS` permission. This command is useful when making configuration changes to the full-text service via engine variables that only take effect when the service (re)starts (`fts2_max_connections`, for example). Also can be used to force a reclaim of resources the service might be using before the idle-out time is reached. The service run state can be checked using `SHOW FULLTEXT SERVICE STATUS`. (8.7.12) * **Enhancement**: Significant overall performance improvements for the full-text V2 service. * **Bugfix**: Fixed an intermittent hang under load issue with the full-text V2 service. * **Bugfix**: Reduce `server_id` collision probability for MySQL CDC. * **Enhancement**: When `parametrize_json_keys` is enabled, numeric keys will be parametrized as well. Also fixed a crash in queries with json extracts within JSON MATCH ANY (when `parametrize_json_keys` is switched on). * **Enhancement**: Update to certificate loading behavior. Prior to this change, only the first certificate was loaded from a server's .pem file specified via the server configuration. Now all certificates are loaded, so if a .pem file contains a certificates chain, then intermediate CA certificates will be returned by the server. * **Enhancement**: Updated the distributed OpenSSL license file to to 1.0.2zj * **Bugfix**: Fixed an issue that caused INFER of CDC-in pipelines to fail with "Failed to allocate slot in the extractor pool" error after increasing the limit. * **Bugfix**: Fixed an issue where concurrent `DROP TABLE`s could block a snapshot from running indefinitely. * **Bugfix**: Address an issue with rewriting of scalar subqueries with null-accepting projections. * **Enhancement**: Added `X-Content-Type-Options: nosniff`, `Strict-Transport-Security: max-age=31536000`, `Cache-control: no-store` headers to Data API responses. * **Enhancement**: Added new engine variables for full-text search, used to fine-tune Lucene service connections: `fts2_socket_timeout_ms` and `fts2_socket_ttl_ms`. These variables control the service's socket timeout and time-to-live. * **Enhancement**: The default value for the `fts2_max_connections` engine variable has been increased to 100000. * **Enhancement**: Improved throughput for full-text v2 queries. * **Enhancement**: Optimized full-text queries that contain an `ORDER BY ... LIMIT ...` over a full-text score and optionally filter on the same full-text clause. * **Bugfix**: Fixed an issue in vector range search with `DOT PRODUCT` as the distance metric. * **Bugfix**: Fixed an issue with lateral join queries with identical aliases. * **Bugfix**: Fixed a possible race condition with DR connections that could cause repeated connection failures when attempting to replicate from the primary. * **Bugfix**: Fixed a rare race condition where PITR would incorrectly operate on an unrecoverable database. ## July 2024 * **Bugfix**: Fixed a password validation error on `ADD AGGREGATOR`. * **Enhancement**: Added a new engine variable: `assume_udfs_deterministic`. This variable controls behavior where SingleStore will do extra work to avoid issues caused by UDFs that return different values when repeatedly invoked (e.g., are non-deterministic). This variable defaults to false, preserving the existing behavior (since SingleStore version 8.0), where SingleStore will ensure these computations take place once on an aggregator node. When set to true, SingleStore will avoid this extra layer of coordination (and bottleneck) allowing leaf nodes to independently coordinate. * **Enhancement**: Added a new engine variable: `external_functions_service_buffer_mb`. Sets the maximum size (in mb) of the memory-mapped region used to communicate between the engine and collocated services. * **Bugfix**: Fixed an issue that could occur when replacing JSON expressions with computed columns in `GROUP BY` lists. * **Bugfix**: Fixed potential deadlock in Lucene service under high load. * **Bugfix**: Fixed an issue where in a VM environment `SHOW LEAVES` could show corrupted latency values due to incorrect values from the system clock if the VM is stopped or migrated. * **Bugfix**: Fixed a vector index issue in unlimited storage that could make a vector index blob file in accessible. * **Enhancement**: Added a new engine variable, `sampling_estimate_approach`, used to configure sampling in columnstore tables. See [Statistics and Sampling Concepts](https://docs.singlestore.com/cloud/query-data/query-tuning/statistics-and-sampling/statistics-and-sampling-concepts/#section-idm4556665136652832788440675389.md) for more information. * **Bugfix**: Fixed an issue where indexes were not case-sensitive when performing full-text search against JSON keys. * **Bugfix**: Fixed an issue where `ALTER TABLE` could throw errors related to VECTOR-type columns, even if those columns were not part of the `ALTER`. * **Bugfix**: Fixed an issue in CDC-in pipelines causing delays between batches. * **Enhancement**: Added Feedback Reoptimization via the `REOPTIMIZE` SQL command. The command is used to reoptimize a query based on previous executions of it. You can reoptimize a query multiple times, pick execution plans from any iteration, and then choose the best iteration's plan to commit to use. Syntax: ```sql REOPTIMIZE [[EXPLAIN] | [COMMIT]] [] ``` For more information, see `REOPTIMIZE`. (8.7.7) * **Enhancement**: Added websocket\_proxy.log to the files managed via Trace Log Rotation. * **Enhancement**: Added support for numeric range queries when doing full-text search against JSON fields. Refer to [numeric range queries](https://docs.singlestore.com/cloud/developer-resources/functional-extensions/working-with-full-text-search.md) for more information. * **Enhancement**: Enabled lateral join subselects to reference fields in grandparent `SELECT`. * **Bugfix**: Fixed an issue that could occur when replacing JSON expressions with computed columns in `GROUP BY` lists. * **Enhancement**: Enabled `log_file_size_partitions` and `log_file_size_ref_dbs` engine variables on SingleStore Helios. * **Bugfix**: Fixed display of microseconds for default TIMESTAMP(6) column values in system information commands (such as `DESCRIBE TABLE` or `INFORMATION_SCHEMA.COLUMNS.COLUMN_DEFAULT`). * **Bugfix**: Fixed a potential false positive output for OBSERVE queries during the first delete of large columnstore segments. * **Enhancement**: Allow leading 0x prefix for `OBSERVE` offsets. * **Bugfix**: Fixed an issue where the output of the `OBSERVE` query for the first delete on a columnstore segment was the inverse of expected. * **Bugfix**: Fixed an issue where user defined variables could read collation-related variables incorrectly during a `SET @var = ...` command. * **Bugfix**: Fixed a table eviction issue that can crash a node when replaying straight-to-disk inserts on unique key tables during database recovery. * **Bugfix**: Fixed an issue in MongoDB® CDC-in pipelines which blocks data ingestion after `ALTER PIPELINE`. * **Enhancement**: Allow NULL values in JSON columns when used with Full-Text search. * **Bugfix**: Fixed several system warnings that occur when running a `SELECT` from the `MV_VERSION_HISTORY` information schema view. * **Enhancement**: Improved performance of VECTOR-type User-Defined Variables(UDVs). No longer require using extra typecast to BLOB datatype. * **Bugfix**: Fixed an issue to now avoid prematurely assigning column reference to namespace. * **Enhancement**: Added support for `INNER`/`LEFT` lateral joins. * **Bugfix**: Fixed issue that caused protocol errors in the pipelines. * **Bugfix**: Temporarily disallow shallow table copy for tables created on version 8.0 or before. * **Enhancement**: Enhance table pushdown to union rewrite to enable derived table pushdown based on row count estimation. * **Enhancement**: Added support for `ON NODE` clause for `SHOW PROFILE PROCESS|PLAN`, which forwards the command to another aggregator. * **Enhancement**: Improved performance of the `CREATE PROJECTION` command. * **Bugfix**: Unneeded sort fields will no longer be projected to the aggregator. * **Enhancement**: Improved the performance of `CREATE TABLES AS INFER PIPELINE`. * **Bugfix**: Fixed an issue that could occur where a join with `TABLE(JSON_TO_ARRAY(JSON_EXTRACT_JSON()))` is performed with numeric keys in the `JSON_EXTRACT_JSON()` statement could possibly lead to incorrect output or crashes. * **Bugfix**: Identified an upgrade scenario that causes a blob leak and added a command to delete the leaked blobs. As a result, temporarily disallowing shallow table copy for tables created on version 8.0 or before. * **Enhancement**: Set collation utf8\_bin for the `JSON_TO_ARRAY` builtin in cases where the input has utf8 charset, and set collation utf8mb4\_bin in cases where the input has utf8mb4 charset. * **Bugfix**: An error is reported when inserting incorrect data into vector column for any `data_conversion_compatibility_level`. * **Bugfix**: Fixed a possible connectivity issue and eliminated delay when starting the Lucene service for the first time. * **Enhancement**: Updated the `MV_BOTTOMLESS_STATUS_EXTENDED` information schema table: * Removed the `LATEST_API_FAILURE_RATE` and `AVERAGE_API_LATENCY_US` columns. * Added the following columns: `SUCCESS_API_PER_SECOND` - Number of successful remote storage api calls per second over the last 30 seconds. `FAILED_API_PER_SECOND` - Number of failed remote storage api calls per second over the last 30 seconds. `AVERAGE_API_EVENT_LATENCY_US` - Average latency to complete a remote storage api event over the past 30 seconds. Includes time taken on retries or waiting for rate limiting. `AVERAGE_API_RESPONSE_LATENCY_US` - Average latency of receiving a response for a remote storage api request. Excludes waiting for rate limiting or retries. NOTE: This is relevant only for S3. otherwise this field is identical to `AVERAGE_API_EVENT_LATENCY_US`. ## June 2024 The following updates have been made this month: * **Enhancement**: Added the `enable_use_of_stale_incremental_stats` engine variable which allows the optimizer to use stale stats instead of querying for stats during optimization. * **Enhancement**: Added the `throttle_replay_behind_tail_mb` engine variable which controls how far the system allows a child aggregator to lag behind replicating the transaction log of its master aggregator before throttling is applied to the master aggregator. * **Bugfix**: Fixed issues with the `TO_CHAR()` function where results were truncated or reading communication packet errors were generated. * **Bugfix**: Fixed an integer overflow issue in unique integer columns by changing the storage type to a 32-bit unsigned integer. * **Bugfix**: Fixed incorrect results in an uncommon scenario where hash join operators with an optimization path need to spill. * **Enhancement**: The `memsql_exporter` now collects additional fields from the `mv_activities_extended_cumulative` information schema view. * **Enhancement**: Updated the default `client.id` for the Kafka configuration to the SingleStore partner ID. * **Enhancement**: Added the following monitoring metrics to the `information_schema.mv_bottomless_status_extended` view: `NUM_SUCCESS_API_REQUESTS`, `NUM_FAILED_API_REQUESTS`, `LATEST_API_FAILURE_RATE`, `AVERAGE_API_LATENCY_US`, `IS_GC_RUNNING`, `NUM_REMOTE_FILES_DELETED`, and `NUM_SLOW_DOWNS`. * **Bugfix**: Fixed a crash that occurred when spilling a hash join operation with at least 3 tables. * **Bugfix**: Fixed a crash that occurred when running the `SHOW CREATE PROJECTIONS` command when using the database name in the syntax (`SHOW CREATE PROJECTIONS database_name.projection_name...`). * **Bugfix**: Fixed a bug where the full-text service controller would periodically restart the process when the started process hadn't fully completed. * **Bugfix**: Fixed an issue where `INSERT ... SELECT` into `VECTOR` type columns returned an error. * **Enhancement**: Added support for Iceberg ingest via Snowflake, REST, and JDBC. * **Enhancement**: Added support for BM25 full-text search (FTS) as a built-in feature. * **Bugfix**: Fixed an edge case regression with SingleStore Kai and information schema queries when using multiple clusters and databases. * **Bugfix**: Fixed a crash that occurred when dropping a database with tables containing very high and very low column IDs. ## May 2024 * **Bugfix**: Addressed a crash issue with `REGEX_REPLACE`. * **Bugfix**: Fixed an issue in `ALTER TABLE` when adding a computed column containing `JSON_BUILD_OBJECT`. * **Bugfix**: Reduced stack usage during compilation for some `CASE` expressions with many `WHEN-THEN` statements. * **Bugfix**: Fixed a potential deadlock issue between background garbage collector threads and foreground query execution threads. * **Bugfix**: Fixed a collation issue in the `JSON_COLUMN_SCHEMA` information schema table. * **Enhancement**: SingleStore now supports module eviction of idle tables with an `ALTER` history. This means that, prior to the 8.5.20 release, tables that had been altered could not be evicted. Altered here includes `CREATE INDEX`, `ALTER TABLE ADD COLUMN`, and some others. * **Bugfix**: Fixed a bug with Wasm UDFs where `REPLACE FUNCTION` did not correctly evict from the plancache, causing continued usage of the old UDF after replacement. * **Bugfix**: Fixed an upgrade issue present in version 8.5.17. * **Bugfix**: Fixed a BSON issue that could occur when large documents are read and mistakenly reported as corrupted. * **Enhancement**: Upgraded OpenSSL 1.0.2u to 1.0.2zj * **Bugfix**: Fixed a crash issue that could occur under a rare race condition in some distributed join queries. * **Bugfix**: Fixed statement atomicity in transactions writing to tables that have projections on them. * **Enhancement**: Allow Kerberos users to skip password validation for users who are authenticated with plugins. ## April 2024 * **Enhancement**: Added support for loading data from compressed Avro datafiles. * **Bugfix**: Fixed an issue that occurred when primary key is NULLABLE when copying a table using "Create table as select" statement. * **Bugfix**: Addressed a crash issue occurring while executing queries with vector builtins on clusters running via Docker containers on Mac machines equipped with Apple silicon processors (M1). * **Bugfix**: No longer show information\_schema pipeline metadata of databases that are not attached to the cluster. * **Enhancement**: Use SeekableString encoding by default for storing VECTOR datatype columns in columnstore tables. * **Bugfix**: Fixed an issue that caused rare "Arrangement already exists in namespace" errors in recursive CTE queries. * **Bugfix**: Improved the hashGroupBy spilling condition to trigger (spill) when appropriate. * **Enhancement**: Added a new engine variable `costing_max_djo_tables` which sets the maximum amount of tables we allow in full DQO estimate search when filling table row estimates for distributed join hints. * **Enhancement**: Made the following engine variables configurable on SingleStore Helios: `multi_insert_tuple_count, columnstore_row_value_table_lock_threshold, internal_columnstore_max_uncompressed_blob_size, internal_columnstore_max_uncompressed_rowindex_blob_size, columnstore_validate_blob_before_merge, default_max_histogram_buckets, enable_varbuffer_dictionary_compression, enable_alias_space_trim, skip_segelim_with_inlist_threshold, default_autostats_histogram_mode, default_autostats_columnstore_cardinality_mode, default_autostats_rowstore_cardinality_mode, default_autostats_columnstore_sampling, experimental_disable_multipartition_parallel_read, internal_enable_parallel_query_throttling, enable_spilling, spilling_node_memory_threshold_ratio, spilling_query_operator_memory_threshold, regexp_compile_mem_mb, selectivity_estimation_min_sample_size, repartition_minimum_buffer_size, use_seekable_json, json_document_max_children, json_document_max_leaves, json_document_sparse_children_check_threshold, json_document_sparse_children_check_ratio, json_document_absolute_sparse_key_check_ratio, json_document_page_data_soft_limit, ignore_insert_into_computed_column, max_subselect_aggregator_rowcount, leaf_pushdown_enable_rowcount, reshuffle_group_by_base_cost, max_broadcast_tree_rowcount, enable_broadcast_left_join, display_full_estimation_stats, force_bushy_joins, force_heuristic_rewrites, force_table_pushdown, force_bloom_filters, explain_joinplan_costs, statistics_warnings, optimizer_warnings, optimize_constants, optimize_huge_expressions, optimize_expressions_larger_than, optimize_stmt_threshold, quadratic_rewrite_size_limit, batch_external_functions, batch_external_functions_size, optimizer_enable_json_text_matching, optimizer_enable_orderby_limit_self_join, distributed_optimizer_broadcast_mult, distributed_optimizer_old_selectivity_table_threshold, distributed_optimizer_selectivity_fallback_threshold, distributed_optimizer_verbose, old_local_join_optimizer, optimizer_hash_join_cost, optimizer_merge_join_cost, optimizer_nested_join_cost, optimizer_disable_right_join, interpreter_mode_sampling_threshold, hash_groupby_segment_distinct_values_threshold, sampling_estimates_for_complex_filters, enable_histogram_to_unwrap_literals, estimate_zero_rows_when_sampling_data_is_missing, exclude_scalar_subselects_from_filters, leaf_pushdown_default, distributed_optimizer_nodes, optimizer_num_partitions, enable_binary_protocol, enable_multipartition_queries, enable_local_shuffle_group_by, enable_skiplist_sampling_for_selectivity, columnstore_sample_per_partition_limit, disable_sampling_estimation, disable_histogram_estimation, inlist_precision_limit, allow_materialize_cte_with_union, optimizer_cross_join_cost, distributed_optimizer_run_legacy_heuristic, distributed_optimizer_min_join_size_run_initial_heuristics, distributed_optimizer_unrestricted_search_threshold, singlebox_optimizer_cost_based_threshold, distributed_optimizer_estimated_restricted_search_cost_bound, disable_reference_table_leftjoin_where_pushdown, disable_sampling_estimation_with_histograms, disable_subquery_merge_with_straight_joins, default_columnstore_table_lock_threshold, default_spill_dependent_outputters, query_rewrite_loop_iterations, verify_fields_in_transitivity, optimizer_min_reference_tables_for_gathered_join, optimizer_min_reference_rows_for_gathered_join, max_expression_query_limit, max_expression_item_limit, optimizer_empty_tables_limit, optimizer_beam_width, optimizer_disable_subselect_to_join, disable_remove_redundant_gby_rewrite, subquery_merge_with_outer_joins, optimizer_disable_operator_tree_nullability_check, clamp_histogram_date_estimates, varchar_column_string_optimization_length, histogram_column_correlation, use_column_correlation, consider_secondary_projection, optimizer_disable_semi_join_reduction, optimizer_gb_placement_table_limit, optimizer_disable_transitive_predicates` * **Enhancement**: Added more config information about pipelines to memsql\_exporter. * **Enhancement**: Improved error handling for VECTOR type. * **Enhancement**: Added support for aggregate functions (`FIRST`, `LAST`) with VECTOR datatype. * **Enhancement**: Added support for Kafka key (producer) in the [SELECT ... INTO KAFKA](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/select/#select-into-kafka.md) command. * **Enhancement**: Improved Kafka error messages to make them more specific. * **Bugfix**: Fixed a case where a killed query could incorrectly return a JSON error rather than query interrupted message. * **Bugfix**: `MV_DISK_USAGE` queries can no longer block garbage collection. * **Bugfix**: Fixed a crash-on-startup issue on Centos9. * **Bugfix**: Fixed potential deadlock between table module eviction and GC manager. * **Bugfix**: Fixed a crash that could occur when a scalar subselect is passed as an argument to an external TVF. * **Bugfix**: Fixed an issue with restoring from public backups. * **Bugfix**: Backups now throttle after snapshot. * **New Feature**: Added ability to [create a Projection](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-projection.md) which is a copy of some or all columns in a table, and may be sharded and sorted differently than the primary table. Projections can be used to speed up several types of queries including range filters, `ORDER BY`/`LIMIT`, `GROUP BY`, `count(DISTINCT...)`, `DISTINCT`, joins, and row lookups on secondary keys. Projections depend on the table and are updated in real time when you update the table. Related information schema table: [PROJECTIONS](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/projections.md). * **Bugfix**: Removed an unnecessary metadata copy to improve JSON performance. * **Enhancement**: Multi-insert queries now respect lower `columnstore_segment_rows` settings. * **Enhancement**: Subprocess and S3 API tracing are now also enabled via the `enable_subprocess_tracing` engine variable. * **Enhancement**: Added the optional `IF NOT EXISTS` clause to the `CREATE TABLE ... AS INFER PIPELINE` statement. Refer to [Replicate MongoDB® Collections using SQL](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-sql.md) for more information. ## March 2024 * **Bugfix**: `LAST_INSERT_ID` will now correctly retrieve the last inserted ID for a forwarded `INSERT` query. * **Bugfix**: The rule based optimization warning message will only be shown when cost based optimization is meaningful. * **Bugfix**: Fixed a minor performance issue where a table module could be evicted prematurely during database recovery. * **Bugfix**: Fixed a very rare deadlock between `ALTER` of a table and a `DROP` of a database. * **Bugfix**: Fixed a JSON insertion failure to old utf8 sample tables that contain utf8mb4 characters by encoding it with base64 format beforehand for backward compatibility. * **Enhancement**: Disk Spilling now takes `QUERY_MEMORY_PERCENTAGE` resource pool setting into consideration. * **Bugfix**: Fixed an issue with a blob file leak in a rare crash scenario. * **Bugfix**: Fixed a potential deadlock in a scenario involving DDL, a 2PC transaction, and a concurrent failover. * **Bugfix**: Fixed a deadlock involving `BACKUP` and `ALTER` in a rare case. * **Enhancement**: Added support for utf8mb4 symbols in column comments, table comments, and user comments. * **Enhancement**: Enabled additional query hint for non-distributed queries so that first run of those queries can be queued. * **Enhancement**: Updated timezone data, now using IANA's tzdata version 2024a; new timezones Kyiv, Ciudad\_Juarez, and Kanton supported. * **Enhancement**: Idle Table Eviction improvement. In the 8.5 GA version, an idle table's objects cannot be evicted from memory if the table has one or more UNIQUE key columns (both PRIMARY KEYs and any other keys with a UNIQUE constraint). In the 8.5.11 patch, idle tables' code objects can be evicted from memory if the table has one or more UNIQUE key columns. * **Bugfix**: Fixed an issue in background merger that allowed concurrent deletes of empty segments. * **Bugfix**: Fixed an issue to ensure preserving a field of a table correctly. * **Enhancement**: Added `numInferredPaths` and `schemaSize` as new columns to `JSON_COLUMN_SCHEMA` table. `numInferredPaths` is the number of key paths inferred for the segment. `schemaSize` is the size of `schema_json` in bytes. * **Bugfix**: Added safety checks around dropping databases without exclusive access to storage. * **Bugfix**: Fixed an issue where explicitly defining a JSON type in the RETURNS of a Wasm TVF can cause an error when it is run. * **Bugfix**: Fixed a rare crash scenario. * **Bugfix**: Fixed an issue that could lead to the risk of undefined behavior when running `DROP EXTENSION IF EXISTS`. * **Enhancement**: `IN`-lists will now use hashmap optimization for matching parameters in `*_MATCH_ANY` statements. * **Enhancement**: Made `default_distributed_ddl_timeout` a sync variable on all nodes. * **Enhancement**: Modified the conversion logic for converting from VECTOR to BSON types. Now, when casting VECTOR(F32), it will generate a BSON array of doubles, rather than a combination of numeric types. * **Bugfix**: Addressed a family of issues relating to optimal execution of JSON extracts on schema'd data. ## February 2024 * **Bugfix**: Fixed an issue with regression in replay performance for databases with many tables. * **Enhancement**: Added support for usage of vector built-ins with string/JSON arguments without requiring an explicit typecast to VECTOR (e.g., 'SELECT @vec<\*> '\[1,2,3]'). * **Enhancement**: Added spilling metrics for TopSort (`ORDER BY` with `LIMIT`). * **Enhancement**: Added support to the VECTOR built-ins for all VECTOR elements types (e.g., F32, F64, I8, I16, I32, and I64). * **Enhancement**: Added support for creating numeric histograms based on JSON values. (Feature flag gated, contact your SingleStore representative) * **Enhancement**: Added new metrics to `memsql_exporter` based on the `information_schema.mv_sysinfo_disk` columns. * **Enhancement**: Creation of computed columns with VECTOR data type is now allowed. * **Enhancement**: Added ability for `information_schema.optimizer_statistics` to display JSON keys. * **Enhancement**: Added ability to skip eviction when log replaying hits the blob cache space limit, providing a greater chance to succeed. * **Enhancement**: Improved JSON histograms to support numeric histograms, analyze command, `info_schema.optimizer_statistics`, and displays JSON histograms. * **Enhancement**: Added support for warming blob cache with table's column data. Syntax is: `OPTIMIZE TABLE WARM BLOB CACHE FOR COLUMN `: ```sql OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN c1, c2; OPTIMIZE TABLE t WARM BLOB CACHE FOR COLUMN *; ``` * **Enhancement**: Disabled the default semi join reduction rewrite. * **Enhancement**: Now recognize more `EXTRACT`/`MATCH` functions as candidates for pushdown/computed column matching. * **Enhancement**: Replaced expression pushdown (EPD) approach with more complete version that does not over project rows. * **Bugfix**: Fixed an issue where the existing websocket connection would close when variables are updated in global scope. * **Bugfix**: Removed the hard-coded 'collation\_server' from `constant.StrictModeParams`. * **Bugfix**: Fixed an issue with a sensitive information leak inside of out-of-memory reports. * **Bugfix**: Fixed the result collation from `VECTOR` built-ins to be `utf8mb4_bin` instead of binary. * **Bugfix**: Fixed an issue with nested extracts in the `JSON_MATCH_ANY()` statement. * **Bugfix**: Fixed an issue with backup restore when the backup has a corrupted GV timestamp in the about snapshot record. * **Enhancement**: The new [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) `parametrize_json_keys` allows JSON keys to be parametrized and plans can be reused. * **Bugfix**: Fixed an issue that causes `CREATE TABLES AS INFER PIPELINE` to fail with an `Out Of Memory` error. * **Bugfix**: Fixed an issue with exporter not working with Public-Key Cryptography Standards #8(PKCS#8) pem format. * **Bugfix**: Fixed an issue with potential crashes when specifying a vector index in index hint. * **Bugfix**: Fixed a crash that occurred when trying to grant access to the table with no context database in per-privilege mode. * **Bugfix**: Fixed a memory issue, when calling a Table-Valued Function (TVF) that has arguments with default values. * **Bugfix**: Fixed an issue with the RG pool selector function return type check. * **Enhancement**: Added ability for `information_schema.mv_connection_attributes` to show `tls_version` and `tls_cipher` for connections where `SSL` is enabled. * **Enhancement**: Added ability for predicate pushdown with `NOW()` and user defined functions (UDF). * **Enhancement**.: Reduced amount of memory used by unlimited storage file listing, including point-in-time recovery (PITR). * **Enhancement**. Added an engine variable, `innodb_lock_wait_timeout.` This variable exists for backwards compatibility with MySQL and is non-operational in SingleStore Helios. * **Enhancement**. Optimized performance when using `VECTOR` built-ins with the `VECTOR` data type. * **Bugfix**: Fixed an issue where a backup would inadvertently carry metadata about the retention period of the backed-up unlimited storage database. * **Bugfix**: Fixed the change to aggregator activity queries in `memsql_exporter` not being applied properly. * **Bugfix**: Fixed an issue with `START PIPELINE FOREGROUND` skipping files in some circumstances. * **Enhancement**: Added ability to `memsql_exporter` to always return all sub-activities of an aggregator activity. * **Enhancement**: Improved ability to configure Kafka extract/egress by allowing additional options. * **Enhancement**: Improved performance for certain query shapes used by Kai. * **Bugfix**: Fixed the query-events endpoint link on the `memsql_exporter` landing page. * **Bugfix**: Fixed an issue where a segmentation fault (crash) appeared after terminating a query on the remote node. * **Bugfix**: Fixed an issue with crashes occurring when using certain `JSON_EXTRACT`s in the predicate expression of `JSON_MATCH_ANY()`. * **Bugfix**: Fixed an issue where the number of rows affected with `SELECT INTO` object storage was erroneously reported as 0 rows. ## January 2024 **Smart DR** Smart DR creates and manages a continuous replication of data to a geographically separate secondary region thereby allowing you to failover to the secondary region with minimal downtime. For more information, refer to [Smart Disaster Recovery (DR): SmartDR](https://docs.singlestore.com/cloud/manage-data/smart-disaster-recovery-dr-smartdr.md) **Database Branching** Database branching enables you to quickly create private, independent copies of your database for development, testing and other scenarios. For more information, refer to [Database Branching](https://docs.singlestore.com/cloud/developer-resources/database-branching.md) **Extensions** Extensions in SingleStore allow you to combine user-defined objects, such as UDFs or UDAFs, into a packaged archive (the extension) and then create, manage, and deploy these objects and other resources using a single command. Extensions support both Wasm-based and PSQL functions. For more information, refer to [Extensions](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/extensions.md). **Trace Events and Query History** Added the ability to trace query completions as events, which is the initial installment of the larger event tracing framework. The Query History feature relies on query event tracing, and can be used to display query trace events over time. The Query History feature can therefore be used to troubleshoot and optimize query performance, including, but not limited to, tracing and recording expensive queries, resolving unexpected slowdowns, and viewing and optimizing workloads in real time. Refer to [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) for more information. **Improved Memory Management for Resource Pools** Added the `QUERY_MEMORY_PERCENTAGE` option for resource pools, which restricts memory usage in the pool on a per-individual query basis. This in contrast to `MEMORY_PERCENTAGE` which restricts usage based on total memory used within the current pool. For example, when creating or altering a resource pool, setting `MEMORY_PERCENTAGE` to `60%` and `QUERY_MEMORY_PERCENTAGE` to `50%` would configure the system so that all queries running within the specified resource pool should together use a maximum of 60% of system memory, and any single query running within the pool should use, at most, 50% of system memory. Example syntax: ```sql CREATE RESOURCE POOL rpoolmain WITH MEMORY_PERCENTAGE = 60, QUERY_MEMORY_PERCENTAGE = 50, SOFT_CPU_LIMIT_PERCENTAGE = 65, MAX_CONCURRENCY = 40; ``` **Load Data Updates** SingleStore now supports loading data using the Change Data Capture (CDC) pipelines from the following data sources: MongoDB® and MySQL. Refer to [Replicate MongoDB® Collections to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore.md) or [Replicate Data from MySQL](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql.md) for information on loading data from the respective data source. **Enhancement**: Other Performance Enhancements * SingleStore now supports creating shallow copies of tables. The `WITH SHALLOW COPY` [feature](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) copies an existing table and creates a new table that will have the same structure as the original table. The data is not physically copied to the new table, but referenced against the original table. * SingleStore now supports sorted scan query plan operators for queries containing `ORDER BY/LIMIT` clauses when utilizing flexible parallelism. Before this enhancement, there could be performance regressions for this query shape using flexible parallelism. * Improved performance when completing large sets of security operations (creating a lot of groups/users/roles, etc.). * Added the ability to use named argument notation when calling a PSQL SP or function. Can reduce total lines of code and make code more readable. * Added reduction of memory pre-allocation during columnstore JSON reads. * Added ability to check if all leaf node partitions are available, before processing new batches. * Addressed a table resolution issue for embedded recursive Common Table Expressions (CTEs). * SingleStore now natively supports the BSON data type. **Enhancement**: Query Optimization Enhancements * Added support for Row Count and Selectivity hints in views. * Added new join logic to recognize when a non-reference table is being joined exclusively to reference tables and then gather the non-reference table to avoid duplicating work across every partition. * Remove redundant aggregation functions and `GROUP BY` statements. * JSON expressions are properly pushed down. * Allow columnstore optimization for `JSON_MATCH_ANY` with `JSON_EXTRACT` in predicate. * Added support for hash joins on null-accepting expressions in the `ON` condition of outer joins. * Automatically rewrite `A=B OR (A IS NULL AND B IS NULL)` to null safe equal `(A<=>B)` so that many important optimizations (e.g. shard key joins, hash joins) will work. * Perform a subselect to join rewrite in an `UPDATE` statement when there are multiple columns in the subselect. * Removed some query shape lockdowns. * Added support for flipping join order for full outer join. * Improved performance by not executing query optimization procedures for read queries during the process of persistent plan cache lookup. This optimization strategy has resulted in improved lookup performance, leading to faster data retrieval operations. * Added support for statistics on correlations between columns in cases where highly correlated filters are used. * The Data API now supports HTTPS for connections where `ssl_key` is encrypted with `ssl_key_passphrase`. * Removed parametrization of `LIMIT 0` and `LIMIT 1` to unlock more rewrites, especially for subselects. * Modified computed column matching to accurately evaluate JSON expressions containing equality and non-safe equality comparisons. * Fixed an issue where filtering with a `JSON_EXTRACT_` function performs inconsistently. * The query optimizer now considers more `LEFT JOIN` elimination cases. **Enhancement**: New Information Schema Views and Columns * Added a new view, `correlated_column_statistics`, to provide [metadata on correlated columns](https://docs.singlestore.com/cloud/reference/information-schema-reference/query-performance-workload-management-and-statistics/correlated-column-statistics.md). * Added a new view, `RESOURCE_POOL_PRIVILEGES`, to provide [information](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/resource-pool-privileges.md) about resource pool grants and privileges. * Added the following new columns to `MV_BACKUP_HISTORY`: * `error_code`: Error code for failed backups. * `error_message`: Error message for failed backups. * Added the following new columns to `MV_SYSINFO_DISK`: * `read_operations_cumulative_per_device`: Number of read operations performed by the device since start up. * `write_operations_cumulative_per_device`: Number of write operations performed by the device since start up. * `device_name`: Name of the device to which the values in `read_operations_cumulative_per_device` and `write_operations_cumulative_per_device` are associated. * Added the following to support trace events that are used by the [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) feature: * `MV_TRACE_EVENTS`: A snapshot of all trace events, the size of which is dictated by the `trace_events_queue_size` variable * `MV_TRACE_EVENTS_STATUS`: A view that reflects the status of current trace events * `LMV_TRACE_EVENTS`: A snapshot of each node's trace events * Added the following new column to `ADVANCED_HISTOGRAMS`, `L/MV_QUERY_PROSPECTIVE_HISTOGRAMS`, and `L/MV_PROSPECTIVE_HISTOGRAMS`: * `JSON_KEY`: an entry for each (column, json\_key) pair. For non-json columns `JSON_KEY` is NULL. * Added the `blob_cache_miss_b` and `blob_cache_wait_time_ms` columns to the following information schema views: `information_schema.plancache`, `information_schema.MV_ACTIVITIES`, `information_schema.MV_ACTIVITIES_CUMULATIVE`, `information_schema.MV_TASKS`, `information_schema.MV_FINISHED_TASKS`, and `information_schema.MV_QUERY_ACTIVITIES_EXTENDED_CUMULATIVE`. * **Bugfix**: Updated the `information_schema.USERS` view to reflect the account status for locked users. **Enhancement**: New Commands and Functions * Added support for the `REGEXP_MATCH()` function. This function returns a JSON array of matching substring(s) within the first match of a regular expression pattern to a string. * Added support for `CUBE` and `ROLLUP` grouping operations to Wasm-based user-defined aggregate functions (UDAFs) in SingleStore. For more information, refer to [CREATE AGGREGATE](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-aggregate.md). * Added the following to support trace events that are used by the [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) feature: * `CREATE EVENT TRACE` to create a trace event * `DROP EVENT` to drop a trace event * Added ability to use `DELETE` on identical keys with the `ON DUPLICATE KEY` clause. This is in addition to existing "upsert" support with `ON DUPLICATE KEY UPDATE`. This allows new scenarios such as the ability to manage streaming aggregation with `INSERT … ON DUPLICATE KEY UPDATE … ELSE DELETE ….` * `INFER PIPELINE` for MongoDB® CDC-in now generates tables with BSON column types. **Enhancement**: New or Modified Engine Variables * Added `enable_idle_table_optimizations` and `enable_idle_table_eviction`, which are used to reduce table memory overhead for idle tables on a cluster. The feature is enabled by default on all new *and existing* clusters. The variable `enable_idle_table_optimizations` needs to be set at runtime and requires a restart for changes to take effect. It can be set to OFF or ON (default). The other related variable, `enable_idle_table_eviction`, can be set during a session (though a very small amount of overhead will remain until the server is restarted) and can be set to Full, SkipListsOnly, and Off. It defaults to SkipListsOnly, which means SingleStore will only evict skiplist indexes for idle tables on the cluster. Full means it will evict skiplists and table modules, and Off means no eviction. * Added `privilege_caches_update_mode`, which can be used to address some performance issues that occur when performing large sets of security operations (creating a lot of groups/users/roles, etc.). * The `query_parallelism` engine variable (which was deprecated in 8.1) now is non-functional. To modify Flexible Parallelism settings, use `query_parallelism_per_leaf_core` instead. * Added the `use_user_provided_index_types_in_show` engine variable which controls what will be displayed via the `DESCRIBE ` or `SHOW COLUMNS` syntaxes for backward compatibility. * Added the `throttle_replay_behind_tail_mb` engine variable which controls how far the system allows a child aggregator to lag behind replicating the transaction log of it's master aggregator before throttling is applied to the master aggregator. * Added the ability to use the `ANALYZE` command with JSON keys to create histograms when the new engine variable `enable_json_statistics` is enabled. The engine variable `enable_json_statistics` is disabled by default. * Added `trace_events_queue_size` to capture trace events, the first of which is query event tracing. This engine variable is enabled by default (set to a value of 16 MB, where the value must be provided in bytes). Refer to [Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md) for more information. * Added the `optimizer_disable_transitive_predicates` engine variable which disables predicate transitivity on query rewrites if set to `TRUE`. This engine variable defaults to `FALSE`. * Added a new engine variable `pipelines_cdc_java_heap_size` to specify the JVM heap size limit for CDC-in pipelines. * Added the `bottomless_experimental_blobstore_mode` engine variable, which when enabled, completes additional verification of persisted files immediately after upload. This mode is experimental and may reduce upload speed. Please use with caution. * The new [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-2.md) `max_table_memory_room_mb` sets the maximum amount of memory required when creating or attaching a database. Configuring this engine variable allows more control over whether a detached database can be reattached. * The [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-3.md) `backup_multipart_upload_concurrency` maximum value has been increased to 15. **Miscellaneous Enhancements and Bugfixes**: * **Enhancement**: Introduced a new `VECTOR` data type that is recommended over the `BLOB` data type for vector operations and use with ANN indexes. The `VECTOR` type improves code readability, error checking, and reduces total lines of code. * **Enhancement**: Added support for DDL forwarding for `CLEAR BACKUP_HISTORY`.`CLEAR BACKUP_HISTORY` works on the DML endpoint now. * **Enhancement**: Improved retry logic for `connection reset: write` errors. * **Enhancement**: SingleStore now natively supports the BSON data type. * **Enhancement**: Added support for `collection.exclude.list`, `database.include.list`, and `database.exclude.list` parameters to the `CONFIG`/`CREDENTIAL` clause of the `CREATE AGGREGATOR PIPELINE ... AS LOAD DATA MONGODB` statement. Refer to [Replicate MongoDB® Collections to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore.md) for more information. * **Enhancement**: Added support for Approximate Nearest Neighbor (ANN) vector search using inverted file (IVF) and hierarchical navigable small world (HNSW) indexes, and variants of them based on product quantization (PQ). Enables support of larger-scale semantic search and generative AI applications. * **Enhancement**: Increased name length limit to 256 characters for tables, views, table columns, view columns, procedures, functions, and aliases. Database names are still limited to 64 characters. Note: Column aliases are truncated during projection (`SELECT`) to 256 bytes (not characters) which impacts names that use multi-byte characters. The full names are stored in the database and can be queried. Information schema tables can be used to view the full names. * **Enhancement**: Added the ability to truncate plancache file names if they exceed the operating system's specified limit (255 bytes). * **Bugfix**: Both `SHOW PIPELINES` and `SELECT * FROM information_schema.pipelines` now show consistent pipeline state information across master and child aggregators. * **Bugfix**: Specific pipeline built-ins like `pipeline_source_file()` and `pipeline_batch_Id()` should not be used in `UPSERT` clause when creating a pipeline. * **Bugfix**: Improved the ability to terminate expressions containing JSON built-ins. * **Enhancement**: Added error-handling details for pipelines, including state, error, and performance-related metrics through monitoring solutions. * **Enhancement**: MemSQL Procedural SQL (MPSQL) has been renamed to simply Procedural SQL (PSQL). The name change will only show in some `SHOW` command output and information schema views. For example, `SHOW FUNCTIONS` output changed. * **Enhancement**: Added support for `%ROWTYPE` and `%TYPE` for use in declaring scalar type variables and parameters. Employing these abbreviations in Procedural SQL (PSQL) can lead to a reduction in the required lines of code. * **Enhancement**: Introduced blob cache profiling metrics for columnstore tables on unlimited storage databases. Refer to the [PROFILE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/profile.md) page for more information on what blob cache metrics are available. * **Enhancement**: Added infix operators for dot\_product (<\*>) and euclidean\_distance (<->). * **Enhancement**: Added ability to delay retry attempts for pipeline retries. * **Bugfix**: Updated the output for the `Key_name` and `Index_type` columns in the `SHOW INDEX`, `SHOW INDEXES`, and `SHOW KEYS` commands for primary keys on columnstore tables. Refer to the [SHOW INDEX, SHOW INDEXES, SHOW KEYS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-index-show-indexes-show-keys.md) page for more information. * **Bugfix**: Improved the error message displayed when trying to create a primary key on an existing table. * **Bugfix**: Improved the error message displayed when a `GRANT` command fails due to missing permissions. The error message will now show the missing permissions: ```sql GRANT SELECT, UPDATE, DELETE, EXECUTE ON *.* TO test2; ``` ```output ERROR 1045 (28000): Current user is missing UPDATE, DELETE permission(s) for this GRANT ``` * **Bugfix**: Fixed a case when `UNIX_TIMESTAMP()` was incorrectly returning 999999999.999999 for `DATETIME` data types with precision. * **Bugfix**: Fixed an issue where a user could be erroneously marked as deleted. * **Bugfix**: Fixed an issue where a child aggregator could crash if it ran out of memory during query forwarding. * **Bugfix**: Fixed an issue with blob cache LRU2 eviction that could occur when a query fetches a blob, evict it, and fetches it again. * **Bugfix**: Fixed an issue that could cause information for a blob to be missing from an Information Schema table. * **Bugfix**: Disk Spilling now takes the resource pool settings into consideration. * **Enhancement**: Auto user creation is deprecated and the `NO_AUTO_CREATE_USER` variable is enabled by default. * **Bugfix**: Fixed an erroneous access denied issue to views selecting from shard tables with computed columns. * **Bugfix**: Fixed a rare issue where the incorrect timezone could be used in logging. * **Bugfix**: Fixed an issue where using user-defined variables inside `ORDER/GROUP BY` statements could cause a crash. * **Enhancement**: Added column name to error messages when invalid date/time is inserted into a column. * **Enhancement**: Added BSON columnstore functionality over Parquet storage. * **Enhancement**: Added support for `SELECT ... INTO KAFKA` using OAUTH credentials. * **Bugfix**: Prevent the ability to create a Kafka Pipeline using Parquet Format. * **Bugfix**: Queued Time is now excluded from the cost estimate for workload management leaf memory. * **Bugfix**: Specific error messages are now logged for GCS subprocess failures. * **Bugfix**: Fixed a network communication error that occurred after query rewrites. * **Bugfix**: Fixed an error that occurred when a user attempts to access a view that is based on a table that has no privileges granted on it. * **Enhancement**: Improved the ability to kill queries containing JSON built-in functions. * **Bugfix**: `JSON_KEY` escape characters were not working as expected. * **Bugfix**: `ALTER PIPELINE` setting `max_partitions_per_batch` to use a default 0 value is now allowed. * **Enhancement**: Improved performance for `JSON_EXTRACT_` built-ins in `ORDER BY` clauses. * **Enhancement**: Now suppressing a harmless `traceSuspiciousClockUpdate` trace message during recovery. * **Bugfix**: Fixed an issue where `INSERT...SELECT` queries with a `partition_id()` filter generating an error. * **Bugfix**: Fixed an issue with memory crashing when using `REGEXP_MATCH`, `JSON_INCLUDE_MASK`, or `JSON_EXCLUDE_MASK` built-ins. * **Enhancement**: Improved performance by optimizing joins on `TABLE(JSON_TO_ARRAY())` queries. * **Bugfix**: Fixed an allocation issue that caused poor performance on high load insertion queries. * **Enhancement**: Enabled support for external UDFs used in `INSERT` statements with multiple `VALUE` clauses. * **Enhancement:** Added BSON fundamentals and column type support for SingleStore Kai. * **Bugfix**: Fixed a bug that could result in unrecoverable databases if the database had 1024 or more tables. * **Bugfix**: Fixed an optimization out-of-memory issue cause by operators generated for wide user tables. * **Bugfix**: Fixed a crash that occurs in rare scenarios involving `ALTER TABLE` and failovers. * **Bugfix**: Fixed ineffective search options that change in subsequent vector search queries. * **Bugfix**: Resolved an issue related to `FROM_BASE64` and `TO_BASE64` builtins when processing large string inputs thereby preventing potential errors in reading communication packets. * **Bugfix**: Fixed the code involved in backups to improve download error messaging. * **Bugfix**: Fixed an bug that could cause scans using non-unique indexes which could return incorrect results or cause crashes. ## December 2023 * **Enhancement**: Added support for creating a `DEEP COPY` of tables with computed columns. * **Enhancement**: Added additional config and credential option validation while creating pipelines. * **Bugfix**: Fixed an issue where valid `LINK` config and credential parameters were not supported for both reading from and writing to a datasource. ## November 2023 * **Enhancement**: Notebooks now have autosave (currently saves every 5 seconds). * **Enhancement**: Added additional node metrics to the /cluster-metrics endpoint of the `memsql_exporter`. * Added three new fields to `MV_SYSINFO_DISK`: * `READ_OPERATIONS_CUMULATIVE_PER_DEVICE`: Number of read operations performed by the device since start up. * `WRITE_OPERATIONS_CUMULATIVE_PER_DEVICE`: Number of write operations performed by the device since start up. * `DEVICE_NAME`: Name of the device to which the values in `read_operations_cumulative_per_device` and `write_operations_cumulative_per_device` are associated. * **Enhancement**: Changed the SingleStore Helios cluster default and range for the `pipelines_cdc_row_emit_delay_us` engine variable. Throttling default value is set to 1. The supported range is from 0 to 1000000. * **Enhancement**: Addressed some performance issues that occur when performing large sets of security operations (creating a lot of groups/users/roles, etc.) via the new `privilege_caches_update_mode` engine variable. * **Enhancement**: Improved performance during snapshotting for CDC-in pipelines. ## October 2023 * **Enhancement**: Added ability to infer CSV data with text boolean values. * **Enhancement**: Added support for simple multi-column update with sub-query. * **Enhancement**: Added ability to use SSL keys with a password in the HTTP API. * **Bugfix**: Fixed an issue preventing nodes from attaching during upgrade. * **Bugfix**: Fixed an upgrade issue where some databases could temporarily become unrecoverable if snapshots were skipped during a pre-upgrade on a recently attached unlimited database. * **Enhancement**: The `singlestore_bundle.pem` file, which SQL clients can use to connect to SingleStore Helios, will be updated as of October 20, 2023. As a consequence, connecting to SingleStore Helios may not be possible until this file has been (re-)downloaded. Refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md) for more information. ## September 2023 * **New Feature**: SingleStore Spaces - Find a gallery of notebooks to learn about scenarios that SingleStore covers at: * **Enhancement**: Notebooks have been improved with the addition of the following features: * SQL Notebooks * Hints to connect to external sources for the notebook firewall settings * Performance improvements around loading time * Jupyterlab 4.0 * **New Feature**: [Datadog integration](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-datadog.md). Monitor the health and performance of your SingleStore Helios clusters in Datadog. * **Enhancement**: Enhanced the performance of DDL statements for role manipulation. * **Enhancement**: Added two [engine variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-4.md), `jwks_username_field` and `jwks_require_audience` to add flexibility and improve security. * **Enhancement**: Added two [engine variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-5.md): `max_expression_query_limit` which sets a limit on the number of expressions within an entire query and `max_expression_item_limit` which sets a limit on the number of expressions within a query item. Both can be set to a range between `100` and the [maximum unsigned `INT` value](https://docs.singlestore.com/cloud/reference/sql-reference/data-types/integer-numbers.md). Setting these engine variables to the maximum unsigned `INT` value disables both features. * **Enhancement**: Added support for materializing CTEs without recomputing them when the query contains `UNION`, `UNION ALL`, and other `SET` operations. To enable the feature, set the [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-6.md) `allow_materialize_cte_with_union` to `TRUE`. * **Bugfix**: Fixed several issues causing slow compilation for queries over wide tables. * **New Feature**: Persistent Cache/Disk Monitoring - Monitoring dashboard to help explain "What's consuming Persistent Cache" as well as the Blob Cache downloaded/evicted rate. * **Bugfix**: Fixed an issue where a crash occurs where the engine improperly rewrites queries with a `UNION` in an `EXCEPT` clause. ## August 2023 * **New Feature**: Added support for the `REGEXP_MATCH()` function. This function returns a JSON array of matching substring(s) within the first match of a regular expression pattern to a string. * **Enhancement**: Improved performance of multi-part GCS backups. * **Enhancement**: Improved memory consumption in json decoding. * **Enhancement**: Introduced a new global engine variable, `json_document_max_leaves` which limits the number of JSON key paths inferred within a segment. The default value is `10000`. * **Enhancement**: Introduced a new global variable, `dr_min_connection_timeout_ms`, which allows users to adjust the minimum timeout period in Disaster Recovery (DR) replication. * **Enhancement**: Added support for multiple uncorrelated IN-subselects in more query shapes. * **Enhancement**: `SKIP PARSER ERRORS` is now supported for Kafka. Additionally, a new related engine variable, `pipelines_parse_errors_threshold`, has been added. * **Enhancement**: SingleStore automatically rewrites A=B OR (A IS NULL AND B IS NULL) to null safe equal (A<=>B) to enable hash joins. * **Enhancement**: Added support for hash joins on null-accepting expressions in the ON condition of outer joins. * **Bugfix**: Fixed an issue with significant memory reduction from spilling for hash join queries with variable length strings (varchar, text, etc.) involved. * **Bugfix**: Fixed an issue where `BACKUP DATABASE WITH INIT` would fail under out-of-memory conditions. * **Bugfix**: Fixed a potential crash in NFS backup when encountering an IO error. ## July 2023 * **Enhancement**: Added support for Control Group v2 (cgroup v2). * **Bugfix**: Fixed an issue where a large spike in the query memory consumption on a any SingleStore node could cause replicas on the same node to become unrecoverable. * **Bugfix**: Fixed a pipeline wrong state issue caused by an error in a table with a computed column. * **Bugfix**: Fixed a potential issue where `REBALANCE PARTITION` would not stabilize to a partition placement on a read replicas cluster. * **Bugfix**: Fixed a potential crash in some edge cases when using Parquet pipelines. * **Enhancement**: The SingleStore Python Client is now the standard for our [notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md). This upgrade supports the ingestion of dataframes with specialized data types, including geospatial or vector data, into the database. Additionally, it incorporates the Ibis component, enabling Python to directly interact with the database. This allows dataframes to be executed within the database itself, greatly enhancing performance. * **Enhancement**: Default values for BLOB/TEXT and JSON columns are allowed, as well as NULL and empty strings. * **Bugfix**: Fixed a crash that may occur when using `SELECT … FOR UPDATE … LIMIT ` in a multi-statement transaction when `two_phase_commit` is ON. * **Bugfix**: Fixed a bug where the permissions do not clear for Data Manipulation Language (DML) queries and then leak to subsequent Data Definition Language (DDL) queries. * **Bugfix**: Fixed a bug where the `mv_disk_usage` table would show incorrect results for the plancache directory. * **Bugfix**: Fixed a crash that may occur when a function calls more than one external function. In the following example `get_embedding()` is an external function: ```sql SELECT DOT_PRODUCT(JSON_ARRAY_PACK(get_embedding('house')), ``` * **Enhancement**: Added an opt-in optimization to Kerberos for HDFS pipelines to reduce the amount of requests. * **Bugfix**: Fixed a JSON formatting issue for the `PARTITION_UNRECOVERABLE` event in `MV_EVENTS` details column. * **Bugfix**: Fixed the dependency on `PIPES_AS_CONCAT sql_mode` inside user defined functions. The `sql_mode` state is now stored when a user defined function is created and used whenever the function is executed. ## June 2023 * **Enhancement**: Introduced the 8.1 `data_conversion_compatibility_level` which provides additional out-of-range checks for the `TIMESTAMP` data type. * **Bugfix**: Fixed a bug where a `HashJoin` on a materialized CTE gets executed as a `NestedLoopJoin` in some query shapes causing slow join performance. * **Bugfix**: Fixed the `SPLIT` [string function](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/split.md) that was not detecting the correct argument type when used inside a [table-valued function](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-tvf.md) (TVF) or a [user-defined scalar-valued function](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-function-udf.md) (UDF). * **Bugfix**: Reduced memory usage in situations where tables are repeatedly created and dropped. * **Enhancement**: Added `mv_disk_usage` and `mv_data_disk_usage` information schema tables that report the breakdown of disk utilization by SingleStore Helios. * **Enhancement**: Added the `max_execution_time` [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#non-sync-variables-list.md). It is unused and setting this variable has no effect. It exists so certain MySQL-compatible apps that have this variable set will not fail when connected to SingleStore Helios. * **Bugfix**: Fixed an issue on sample tables that can be created with different collations on different partitions. * **Bugfix**: Fixed an issue where JSON computed column matching incorrectly triggers a JSON function. * **Bugfix**: Fixed a rare case where the query optimizer re-wrote a query into a non-equivalent query which produced more result rows. * **Bugfix**: Fixed unlimited storage S3 multipart uploads to properly retry on 200 responses which contain an embedded "SlowDown" error, instead of failing immediately. * **Bugfix**: Fixed an issue where memory accumulates over time if reshuffle or repartition operators are executed for many unique query shapes. * **Bugfix**: Fixed an issue with crashes that occur when SSL is enabled on universal storage databases. ## May 2023 **SingleStore Kai (Preview)** SingleStore Kai allows you to run MongoDB® queries natively in a SingleStore Helios cluster. This feature is currently available in [these regions](https://docs.singlestore.com/cloud/reference/singlestore-kai/#section-idm457032586024963367437811064.md). You can enable this feature while creating a cluster. Each SingleStore Kai-enabled cluster has an additional `mongodb://` endpoint, that can be used to connect from supported MongoDB® tools/applications to run MongoDB® queries. For more information, refer to [SingleStore Kai](https://docs.singlestore.com/cloud/reference/singlestore-kai.md). Here's a few additional references: * [Migrate from MongoDB® to SingleStore](https://docs.singlestore.com/cloud/reference/singlestore-kai/migrate-from-mongodb-to-singlestore.md) * [MongoDB® to SQL Mapping](https://docs.singlestore.com/cloud/reference/singlestore-kai/mongodb-to-sql-mapping.md) * [SingleStore Extension Commands](https://docs.singlestore.com/cloud/reference/singlestore-kai/singlestore-extension-commands.md) * [Supported MongoDB® Commands, Data Types, and Operators](https://docs.singlestore.com/cloud/reference/singlestore-kai/supported-mongodb-commands.md) **New Feature**: Introduced a [columnstore row](https://docs.singlestore.com/cloud/create-a-database/columnstore/how-the-columnstore-works.md) data structure (`COLUMN GROUP`) that will create a materialized copy of individual rows as a separate structure in a columnstore table. This index will speed up full row retrieval and updates. **New Feature**: SingleStore Helios now supports creation of Wasm-based user-defined aggregate functions (UDAFs). Refer to `CREATE AGGREGATE` for more information. **Enhancement**: Introduced the `autostats_flush_interval_secs` engine variable. It determines when autostats are flushed to disk if they are not used within the specified time. The default value is 600 seconds. If the engine variable is set to 0, autostats will always stay in memory. **Enhancement**: Introduced two new Sync variables control the dynamic reuse of WM queues: `workload_management_dy-namic_resource_allocation` and `workload_management_queue_size_allow_upgrade`. The default value for `workload_management_queue_size_allow_upgrade` is 1. This means we can upgrade a medium queue to a large queue until the large queue becomes equal to 1. **Enhancement**: Added two new monitoring dashboards: Pipeline Summary and Pipeline Performance. Currently, both dashboards are in preview mode. **Enhancement**: Improved JSON encoding speed for sparse schemas (JSON schemas with a very large number of distinct property names across the full set of documents, with most properties missing in each document). This applies only to universal storage. **Enhancement**: A disk manager now brokers the usage of disk space between competing uses like data, log, cache, and spilling. **Enhancement**: Added capability to create Wasm functions in SingleStore Helios clusters with heterogeneous hardware, e.g., with different levels of SIMD support. **New Feature**: Added a `METADATA_ONLY` [argument](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md) for shard keys which will prevent an index from being created on shard keys thereby saving memory. It can cause queries that would have used that index to run slower. **Enhancement**: Reduced processing costs for queries containing an `ORDER BY` clause by minimizing the amount of data that needs to be processed. This is achieved by only projecting the primary key at the lower levels of the query plan and then joining the primary key with the original data to retrieve the remaining columns. This can significantly improve performance when dealing with large amounts of data and when only a small portion of the data needs to be retrieved. **New Feature**: Added support for [query profiling](https://docs.singlestore.com/cloud/query-data/query-tuning/query-performance-tools.md) that is efficient enough to be enabled all the time by introducing the `auto_profile_type` engine variable with `FULL` and `LITE` options for automatic profiling. The `enable_auto_profile` engine variable must be set to `ON` for the `FULL` or `LITE` options to work. **Enhancement**: The updated full-text search tokenizer in version 8.1 supports utf8mb4. The tokenizer properly tokeni- zes emojis and other extended 4-byte characters. In addition, certain emoji and glyph characters in utf8mb3 are also recognized as their own token rather than being treated as blanks. **Enhancement**: Enhanced support for recursive common table expressions (CTEs) by expanding the range of query shapes allowed, improving column type checking and resolution between the base and recursive branches, and adding cross database support. Also, resolved the issue that caused the "ER\_UN- SUPPORTED\_RECURSIVE\_CTE\_SHAPE" error with the accompanying message about dependen- cy on a streaming result table outside the cycle. **New Feature**: Added two [Information Schema views](https://docs.singlestore.com/cloud/reference/information-schema-reference.md) for tracking connection attributes: `LMV_CONNECTION_ATTRIBUTES` and `MV_CONNECTION_ATTRIBUTES`. **Enhancement**: Added support to optimize table scans with an Order-By-Limit by (a) do the order and limit first, with only the minimum needed columns, and then (b) using a self-join to retrieve the additional necessary columns for only the qualifying rows. This optimization can be enabled/disabled via the `optimizer_enable_orderby_limit_self_join` session variable, which is `ON` by default. **New Feature**: Added `ATTRIBUTE` and `COMMENT` fields for users. These can be set via the `CREATE USER` and `ALTER USER` commands. The values are shown in the `INFORMATION_SCHEMA.USERS` [view](https://docs.singlestore.com/cloud/reference/information-schema-reference/resource-and-user-management/users.md). **New Feature**: [Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) feature (Preview) is available to all customers through our Cloud service. It gives users the ability to marry SQL and Python interoperably and in a secure way. This is available in the Develop section within the Cloud Portal. **Bugfix**: The `sql_mode` session variable now shows up in `SHOW VARIABLES` and the mv\_global\_variables view. **Bugfix**: Fixed an issue that could cause an incorrect error message to be generated on an out of memory error. **Bugfix**: Fixed an issue where a NULL could be pushed down to a union which could cause the type-cast/collation to be lost, resulting in an error: "Operation 'Scanning a hash join build' is not allowed". **Bugfix**: Fixed an issue with `DATETIME` columns with `ON UPDATE CURRENT_TIMESTAMP` defaults that could cause incorrect results. **Enhancement**: Removed `EXECUTE` privilege requirement for Kai internal UDFs. **Enhancement**: Changed system information built-ins to use `utf8_general_ci` collation for the `collation_server`. **Enhancement**: Removed the restrictions on the names of [JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md) users. Customers may now create users with names that look like UUIDs and emails. **Enhancement**: Improved the performance of a left join of a reference table with a sharded table by pushing the reference table clauses to the leaves. The engine variable `disable_reference_table_leftjoin_where_pushdown` must be set to "OFF" to enable this operation. **Bugfix**: Fixed an issue with optimizing queries containing no-operation predicates such as 1=1. **Bugfix**: Fixed rare issue with sync durability and unlimited storage that could cause log file corruption in the blob storage. **Enhancement**: Added more information about blob files that need to be downloaded from unlimited storage to the blob cache when running `PROFILE`. **Bugfix**: Fixed an issue where the query optimizer could choose incorrectly between using the `HashGroupBy` or the `StreamingGroupBy` operators. **Bugfix**: Disabled semi-join operator reduction optimization. ## April 2023 **Bugfix**: Fixed an issue where errors could occur during backup when large files are copied. **Bugfix**: Fixed an issue where S3 pipelines could stop loading data from the bucket if files are deleted from the bucket while the pipeline is active. **Bugfix**: Fixed an issue where, in rare cases, an UPDATE statement with a left join in the filter clauses could fail to execute. **Bugfix**: Fixed an issue where replica blobs could be loaded into the blob cache with an incorrect LRU timestamp. **New Feature**: Added support for LOAD DATA from S3 for [Avro](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data/#section-idm4580319144742433635119437575.md) and [Parquet](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/load-data/#section-idm46104602580400336351264225.md) data. **New Feature**: Added  `JSON_INCLUDE/EXCLUDE_MASK` function. When applied to a JSON document it will return a subset of the document based on the mask. **Bugfix**: Fixed a potential crash issue in `UPDATE` queries that involve joins and that have scalar subselects in `SET` clauses. **Bugfix**: Fixed an issue where running `DROP PLAN FROM PLANCACHE ... FOR QUERY` on a query with invalid syntax could cause a crash. **Enhancement**: Updated timezone metadata to include Mexico's latest timezone change. **Enhancement**: Added a new information schema view internal\_table\_statistics which shows memory use of SingleStore internal metadata tables. The columns displayed are the same as those shown for table\_statistics. **Bugfix**: Fixed a column name resolution issue for recursive CTEs when the column type is the same across the base branch and the recursive branch. **Bugfix**: Fixed an issue where `CLEAR ORPHAN DATABASE` could cause data loss if run when the master aggregator has a database in an unrecoverable state. **Bugfix**: Added the option to use HTTPS with memsql\_exporter. To use HTTPS, enable the engine variables `exporter_use_https`, `exporter_ssl_cert`, and `exporter_ssl_key`. **Bugfix**: Fixed an issue with missing row counts during optimization when sampling is disabled. **Bugfix**: Fixed an issue where background mergers were not stopping quickly enough, resulting in delayed DDL operations or `REBALANCE` commands. **Bugfix**: Fixed an issue in information schema view JSON\_COLUMN\_SCHEMA where incorrect details were being shown for the leaf columns. **Bugfix**: Fixed an issue where the memory used by an external function was not being freed quickly enough. **Enhancement**: The `default_partitions_per_leaf` global variable will no longer be user-settable in SingleStore Helios, ensuring that it's always defined for the most optimal performance, according to the current resource size. **Bugfix**: Fixed an issue where filters using the `range_stats` column in the `information_schema.optimizer_statistics` table were not allowed. **Bugfix**: Fixed an issue where incorrect results could be returned when segment-level flexible parallelism is used inside a subquery containing both a shard key filter and either an aggregate, a window function, or the limit clause. **Bugfix**: Fixed an issue where an internal result table was not created for a recursive CTE involving data across databases. **New Feature**: Added self-service [historical monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md) that allows you to quickly and easily understand your application workloads and debug performance-related issues. ## March 2023 **Enhancement**: Improved column type resolution for base and recursive branches in recursive common table expressions (CTEs). **Bugfix**: Fixed an error that could occur when attaching databases with a snapshot file of greater than 5 GB. **Bugfix**: Fixed a bug where too many rows are sampled from columnstore tables with more than 1 billion rows. **Bugfix**: Fixed an issue with histogram estimation in columns with a negative value `TIME` type. **Bugfix**: Fixed "table partition count mismatch errors" that occur due to the following conditions: the system variable `enable_workspace_ha` is set and there is an upgraded cluster with an attached read replica. **Bugfix**: Fixed an issue with DDL endpoint queries using a lower-than-specified query parallelism setting when clusters are enabled. **Bugfix**: Fixed a bug that prevents `GROUP BY Push-Down` optimization if the join filter contains a mismatched column type. **Bugfix**: Fixed a data conversion compatibility level 8.0 error that may occur when sampling columnstore tables. **Bugfix**: Fixed a possible deadlock that may occur between the blob cache and the Rowstore Garbage Collection (GC) when the blob cache encounters an out-of-memory error. **Bugfix**: Fixed an error caused by setting the `collation_server` global variable to a non-default value when performing a `REBALANCE PARTITIONS` resource availability check. **Enhancement**: Improved the parsing performance of queries that contain several tables. **Enhancement**: The  `read_advanced_counters`, `snapshot_trigger_size`, and  `snapshot_to_keep`  engine variables can now be set on SingleStore Helios. **New Feature**: Added new session variable `disable_remove_redundant_gby_rewrite` to prevent the `GROUP BY` columns from being removed when used in an `ORDER BY` clause. **Enhancement**: Introduced disk and memory availability checks that run before a database is allowed to be attached to a cluster. **Enhancement**: Added the ability to cache histogram results during optimization to reduce the work performed by the histograms. **Enhancement**: Improved the performance of S3 pipelines when Garbage Collection (GC) is enabled. **Enhancement**: Added the ability to backup a database to an HTTPS S3 target with an unverified SSL certificate when using the option: `verify_ssl: false`. **New feature**: `ORDER BY SELF JOIN`, it creates a self join on ORDER BY LIMIT queries to take advantage of differences in bandwidth. **Bugfix**: Fixed an issue with column type checking on base and recursive case of a recursive common table expression. **Bugfix**: Fixed an issue that may cause a "Table doesn't exist" error when a multi-insert contains expressions and the target table has a computed column as its shard key. **Enhancement**: Expanded existing Unicode characters to support Private Use Area (PUA) code points. Including one in the Basic Multilingual Plane (U+E000–U+F8FF) and one in each plane 15 and 16 (U+F0000–U+FFFFD, U+100000–U+10FFFD). **Bugfix**: Fixed a crash that could occur when a computed column definition refers to a table name. **New Feature**: Added the ability to set the `maximum_blob_cache_size_percent` global variable for clusters. **Bugfix**: Fixed an issue with promote lock timeout errors that may occur during a rebalance due to a heavy ingest workload, which causes the merger to be slow to pause. **New Feature**: Introduced a new global variable `subprocess_max_retries`, which is used for retrying on retry-able connection failures during select into/backup queries for S3 and GCS. ## February 2023 **Enhancement**: Background [snapshots](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/backup-database/#UUID-8b6b8e22-d95a-f708-4d77-92d995d5594d.md) are now allowed to run during `BACKUP DATABASE` commands. This prevents increased disk usage by logs during a long-running backup. **Bugfix**: The `CREATE_TIME`, `CREATE_USER`, `ALTER_TIME`, and `ALTER_USER` columns in the `information_schema.TABLES` [view](https://docs.singlestore.com/cloud/reference/information-schema-reference/cluster-component/tables.md) are now properly set for views and TVFs (table-valued functions). **Bugfix**: Fixed an issue that occurred when the `MATCH_PARAM_` argument of the `JSON_MATCH_ANY`[function](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) was not in a predicate expression. **Enhancement**: Improved the performance of the `JSON_MATCH_ANY_EXISTS` [function](https://docs.singlestore.com/cloud/reference/sql-reference/json-functions/json-match-any.md) over columnstore tables. **Bugfix**: Fixed a profiling issue specific to non-collocated hash joins where the memory usage and disk spilling are missing under the join operators. **Enhancement**: Background snapshots are now allowed to run during `BACKUP` commands. This prevents increased disk use by logs during a long running `BACKUP`. **Bugfix**: Attaching a database that exceeds available cluster memory or persistent cache is now automatically blocked. **Enhancement**: Cluster creation and resume times are reduced by running operations in parallel. **Bugfix**: Fixed an issue where the database engine locks up on certain out-of-memory errors. **Bugfix**: Fixed a parsing issue for queries containing multi-line comments near `GROUP BY` or `ORDER BY` clauses. **Enhancement**: The `num_background_merger_threads` [engine variable](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-9.md) is now settable on Cloud. **Enhancement**: The `ORDER BY ALL [DESC|ASC]` (or `ORDER BY *`) syntax is now supported.  **Enhancement**: The `GROUP BY ALL [DESC|ASC]` (or `GROUP BY *`) syntax is now supported.  **Enhancement**: Improved the query execution performance of JSON columns under a higher level of parallelism. **Enhancement**: Expanded support for encoded `GROUP BY` query shapes containing expressions in aggregates. **Bugfix**: Fixed an issue where extra CPU was used when a read-only database is attached to a cluster without any writable mount for the read-only database. **Enhancement**: Sampling will no longer be used for table size estimation when statistics are present. **Enhancement**: Added the `/api/v2/jwks_setup` endpoint to Data API to allow users to enable JWT Auth in Data API on Cloud. See [jwks\_setup](https://docs.singlestore.com/cloud/reference/data-api/data-api-endpoint-reference/jwks-setup.md) for more information. **Enhancement**: Improved the code generation performance of tables with a large number of indexes. **Bugfix**: Fixed an issue causing incorrect trace messages in master logs where clocks were incorrectly advancing from "0". **Enhancement**: Added the `DATETIME_PRECISION` column to both `PARAMETER` and `ROUTINES`[information\_schema](https://docs.singlestore.com/cloud/reference/information-schema-reference/information-schema-view-list.md) views. Also, the `DATETIME_PRECISION` column will include `TIME` and `TIMESTAMP` data types in the `COLUMNS`information\_schema view. **Enhancement**: Added the `REVERSE()` built-in [string function](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/reverse.md) that reverses the target string. ## January 2023 **Bugfix**: Fixed some error handling issues with unlimited storage download and upload processes. **Enhancement**: The `SHOW TABLE STATUS` command now displays the memory usage by GLOBAL TEMPORARY tables. **Bugfix**: Fixed a crash when parsing certain Parquet data into a pipeline. **Enhancement**: Added support for using a connection link for inserting data with the `FORMAT PARQUET` option. **Bugfix**: Fixed an issue with aggregate functions using incorrect enum/set types that may result in inaccurate output in the operator tree. **Bugfix**: Fixed an issue with a transaction leak on the master aggregator when running `CREATE TABLE AS SELECT` on a child aggregator using autocommit=0 on the connection. **Bugfix**: Fixed a bug that may cause a query to hang when comparing an utf8 column with an utf8mb4 constant. This issue occurs when collation\_server is set to one of the utf8mb4 collations. **Bugfix**: Improved the accuracy of network time reporting in query profiles regarding the time spent sending the results back to the user connection. **Bugfix**: Fixed an edge case issue causing a potential memory leak when running an `UPSERT` statement against a columnstore table. **Bugfix**: Fixed an issue that could cause the engine to crash from info\_schema query submissions. **Enhancement**: Improved the performance of bushy join rewrites. **Bugfix**: Fixed an edge case issue where the engine could crash when performing multi-inserts. **Bugfix**: The avro schema registry URL portion of the `CREATE PIPELINE` syntax is now redacted in processlist. **Bugfix**: Fixed an issue where the engine could crash during recursive set operations. **Bugfix**: The `information_schema.statistics` "collation" column now correctly indicates whether an index is ascending ("A") or descending ("D"). **Enhancement**: Improved performance of comparing utf8mb4 strings. **Bugfix**: Fixed an edge case issue which could cause the engine to hang during shutdown. **Enhancement**: Added the `skip_segelim_with_inlist_threshold` engine variable, which will skip segment elimination with the IN list if its size is larger than threshold (default is 1000 elements). **Bugfix**: `information_schema.table_statistics` now correctly shows information about global temporary tables. ## December 2022 The following features may require you to enable them manually. **New Feature**: Improved Seekability in Universal Storage Tables delivers large performance gains for transactional workloads on universal storage tables.  * Added support for fast seeking into JSON columns in a universal storage table using subsegment access. * Improved seek performance for string data types for universal storage for LZ4 and run-length encoded (RLE) data. **New Feature**: Recursive common table expressions (CTE) are now supported by SingleStore. Previously, complex operations including temporary tables within a stored procedure would be needed to perform the actions that a simple recursive CTE query can handle. For more information, see [WITH (Common Table Expressions)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/with-common-table-expressions.md). **Enhancement**: New Information Schema Views * Added the `MV_RECOVERY_STATUS` view which includes information about the status of the current recovery process. * Added several Replication Management views. **Enhancement**: Subselect lockdown messages are now more informative and they indicate the line number and character offset of the subselect that caused the error. In addition, up to 100 bytes of text from the beginning of the referred subselect is also displayed. ``` SELECT (SELECT DISTINCT t1.a FROM t ORDER BY a) FROM t t1; ``` Old output: "Feature 'subselect containing dependent field inside group by' is not supported by SingleStore." New output: "Feature 'subselect containing dependent field inside group by' is not supported by SingleStore.    Near '(SELECT DISTINCT t1.a FROM t ORDER BY a) FROM t t1' at line 1, character 7." **Enhancement**: Decreased the memory overhead for columnstore cardinality statistics by 25% as the first phase of an overall project to improve memory for auto-stats in general. **Enhancement**: Improved performance for user-defined functions (UDFs) and Stored Procedures that take JSON arguments, and the `JSON_TO_ARRAY` command. **Enhancement**: Updated the supported syntax for `DROP … FROM PLANCACHE` so plans on a specified node and plans from all aggregators based on the query text can be dropped. ``` DROP plan_id FROM PLANCACHE ON NODE node_id; DROP PLAN FROM PLANCACHE [ON AGGREGATORS] FOR QUERY ; ``` **Enhancement**: Setting Collation for String Literals You can set the collation for string literals explicitly: ``` SELECT "My string" COLLATE utf8mb4_unicode_ci; ``` **Enhancement**: Created the ALTER USER permission. Users must have this permission or the GRANT permission to be able to execute the `ALTER USER` command. **Enhancement**: Added `ALTER USER ... ACCOUNT LOCK` to manually lock accounts: ``` ALTER USER 'test'@'%' ACCOUNT LOCK; ALTER USER 'test'@'%' ACCOUNT UNLOCK; ``` **Enhancement**: Added sampling (a small portion of the rows in the table are used for analysis) for Reference tables as part of query optimization. **Enhancement**: Improved the performance of the `PROFILE` functionality such as lower memory overheads, lower performance impacts to OLAP queries, and better statistics collecting. **Enhancement**: Added support for improved segment elimination in queries with WHERE clauses containing [DATE and TIME functions](https://docs.singlestore.com/cloud/reference/sql-reference/date-and-time-functions.md). The functions that are supported for segment elimination are `DATE`, `DATE_TRUNC`, `TIMESTAMP`, `UNIX_TIMESTAMP`, and `YEAR`. **Enhancement**: The `data_conversion_compatibility_level` engine variable can now be set to '8.0' for stricter data type conversions. This will now be the default value. This new `data_conversion_compatibility_level` setting additionally flags invalid string-to-number conversion in `INSERT` statements. **Enhancement**: The `sync_permissions` engine variable default value is now ON. The default value only impacts newly installed clusters. Existing clusters must be manually updated to the variable. **Enhancement**: The `enable_auto_profile` engine variable now has a third value: LITE. LITE is the new default value for new customers. It has a lower memory overhead that ON. The default value for existing customers is ON. **Enhancement**: The `columnstore_small_blob_combination_threshold` engine variable default value has been changed to 5242880 bytes. Prior to the 8.0 release, the default value was 33554432 bytes. **Enhancement**: Added support for encoded GROUP BY clauses in queries containing conditional and character expressions in aggregate functions. **Enhancement**: Expanded the type of query execution operations (hash joins, window functions, and sort operations) to offload memory to disk using spilling to allow a large memory footprint query to succeed at the cost of query execution times in a memory constraint environment. **Enhancement**: Added support for ? and \[ ] glob patterns to FS pipelines. **Enhancement**: Added the optional parameter `DEFINER` for `CREATE PROCEDURE`, `FUNCTION`, and `AGGREGATE`. **Enhancement**: Added ability for a JSON computed column to be returned in a query instead of the entire document. **Enhancement**: Added ability to use use the `ORDER BY` clause with the `JSON_AGG` function. **Enhancement**: Expressions can be assigned to system variables. System variables, literals, or any combination of these can be referenced using built-ins like `CONCAT` as a variant of complex expressions. **Enhancement**: For unlimited storage databases, SingleStore caches data within the cluster. It uses a modified least-recently-used (LRU(2)) replacement policy. Information is retained to indicate if objects are frequently-accessed. This reduces the chance that a single large query will flush frequently-accessed data from the cache. **Enhancement**: Added support for the `AUTO` option in the computed column definition clause of a `CREATE TABLE` statement to automatically infer the data type of a computed column expression. For more information, see [CREATE TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/create-table.md). **Enhancement**: Added two Workload Management engine variables: `workload_management_queue_size_allow_upgrade` and `workload_management_dynamic_resource_allocation`. These variables work together to dynamically move queries to another queue if the original queue is saturated. **Enhancement**: Storage of `CHAR()` as `VARCHAR()`: For a column defined as type CHAR of length len, SingleStore will store the column as a `VARCHAR` of length len if len greater than or equal to the value of the new engine variable `varchar_column_string_optimization_length`. If the value of the variable is 0, the column is not stored as a VARCHAR. **Enhancement**: After scaling a cluster, the persistent cache on the cluster is warmed with copies of blobs before new resources begin handling queries. It is fully automatic. **Enhancement**: The `BACKUP` command no longer blocks the `ALTER TABLE` and several other commands for the duration of the backup. This allows you to run commands like `TRUNCATE` on your tables even during the backup of a very large deployment. **Enhancement**: Added the ability to use `JSON_MATCH_`. Returns true if, in the JSON, there is a value at the specified filter path which evaluates the optional filter predicate as true. If no filter predicate is provided, will return true if the filter path exists. **Enhancement**: Made the following Selectivity Estimation improvements: * Enabled sampling for reference tables.Improved date/time histogram estimates by utilizing a heuristic when the current date/time is outside of the histogram range. * Added selectivity estimation for filters containing uncorrelated scalar subselects. This behavior can be controlled by the engine variable `exclude_scalar_subselects_from_filters`. This change has the side-effect of enabling bloom filters more often. * Changed the estimation source to heuristics when sampling is turned on but the total sampled rows are zero. * Added ability to use histogram estimation for filtering predicates that use a stored procedure parameter. * Increased the default value for engine variable optimizer\_cross\_join\_cost to reduce the chance of Cartesian Joins being included when there are incorrect estimations. * Improved the GROUP BY cardinality estimates for predicates using OR expressions. * Enabled ability to combine histogram and sampling selectivity estimates by default. **Enhancement**: Made the following Query Optimization enhancements: * Moved sub-queries for some outer joins from the ON clause to a WHERE clause to enable subselects to be rewritten as joins. * Enabled repartition on expressions. * Added ability to use GROUP BY push down for outer joins. * Enhanced column pruning by eliminating derived duplicate columns. * Removed redundant GROUP BY clauses that are implied by equi-joins. **Bugfix**: Fixed an issue where `REGEXP` and `RLIKE` were case-insensitive. They are now case-sensitive. ## June 2022 **New Feature**: Management API now supports clusters. For more information see, Management API Reference. The following features may require you to enable them manually. * *Code Engine - Powered by Wasm* The Code Engine feature allows you to create UDFs using code compiled to WebAssembly (Wasm). This feature supports any language that can compile to the Wasm core specification.For more information, see [Code Engine - Powered by Wasm](https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm.md). * *Clusters* The Cluster feature allows you to spin up compute resources and size them up or down on-demand independent of storage.Clusters also provide greater flexibility than clusters by allowing databases to be shared across multiple clusters thereby eliminating the need of maintaining data across multiple workloads.See [Compute Clusters](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/compute-clusters.md) for more information.The [SingleStore Management API](https://docs.singlestore.com/cloud/reference/management-api.md) now supports clusters. For more information see, [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md). * *`OUTBOUND` privilege* The `OUTBOUND` privilege can be used to mitigate security risks. The privilege can be assigned to users who are allowed to create outbound internet connectivity.For more information, see [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md). ## April 2022 The following features may require you to enable them manually. **New Feature**: Flexible Parallelism allows multiple cores on the same node to access the same database partition. With Flexible Parallelism, as database partitions are created they are divided into sub-partitions. As a query runs on a leaf node, multiple cores working on behalf of the query can process different sub-partitions of a partition in parallel. As an example, if you are currently at one partition per core with Flexible Parallelism, doubling the size of your cluster and then rebalancing will result in two cores for each partition. As a result, a simple query that scans and aggregates all the data in a single partition will now execute more quickly than it did before. Added new engine variables used for enabling and configuring Flexible Parallelism: `sub_to_physical_partition_ratio`, `query_parallelism_per_leaf_core`, and `expected_leaf_core_count`. The existing engine variable `node_degree_of_parallelism` is deprecated. For more information, see [Flexible Parallelism](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/flexible-parallelism.md). **New Function**: Added the `ISNUMERIC` function, used to determine whether the provided expression is a valid numeric type. **New Function**: Added the `SESSION_USER` function, used to return the user name you specified when connecting to the server, and the client host from which you connected. **New Function**: Added the `SET` function, used to initialize a user-defined session variable. **New Function**: Added new vector functions, namely `VECTOR_ELEMENTS_SUM`, `VECTOR_KTH_ELEMENT`,`VECTOR_NUM_ELEMENTS`, `VECTOR_SORT`, and `VECTOR_SUBVECTOR`. **Enhancement**: Added support for `TRIM` string function. `TRIM` is now multi-byte safe which means the result of an operation using [TRIM](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/trim.md) is either a valid string or an unmodified string. **Enhancement**: Unlimited storage databases now support the `BACKUP … WITH SPLIT PARTITIONS` command. **Enhancement**: The `DROP MILESTONE` command is used to delete a milestone of a currently attached database. **Enhancement**: Improved performance for columnstore seeks into string columns – now it is no longer necessary to scan an entire segment to look up the data for a string value for a row when seeking to find that one row. **New Function**: A new clause "AS new\_db\_name" has been added to the `RESTORE DATABASE` command which allows the use of the full original backup path if trying to restore to a new database name. **New Function**: `SECRET` - Added the ability to hide credentials from queries. Passing credentials in queries can leave them exposed in plain text during parameterization and they can be seen in logs and the process list. To counter this, you can use the `SECRET` function (similar in function to `NOPARAM`). `SECRET` takes a string (such as a password or other sensitive information) and replaces it with the literal string "\" during parameterization. The string is unchanged for the query however. ``` CALL db.do_something_useful('root', SECRET('super-secret-password')); ``` See [SECRET](https://docs.singlestore.com/cloud/reference/sql-reference/string-functions/secret.md) for more information. **Enhancement**: Added per privilege transferability from one user to another via the new TRANSFERABLE clause and SYSTEM\_VARIABLES\_ADMIN grant in the REVOKE security management command. A new engine variable, `privilege_transfer_mode`, must be set to per\_privilege for this functionality to work as expected. Also, this new functionality will affect the results of the `SHOW GRANTS` command. If the `privilege_transfer_mode` engine variable remains on the default value of `grant_option`, then the output is one row and can include the WITH GRANT OPTION privilege. If the value of `privilege_transfer_mode` is per\_privilege, then the output can be two rows. The first row will display the non-transferable privileges. The second row will display the transferable privileges. **Enhancement**: Added new EXPLAIN and PROFILE reproduction clause syntax. `EXPLAIN REPRO` outputs the explain information in JSON format and provides important debugging information. EXPLAIN REPRO will work for SELECT queries only. The `PROFILE REPRO` syntax will replace the need to set the engine variable `set_profile_for_debug` to on. The engine variable will continue to be supported for backward compatibility. **Enhancement**: Added ability to match a computed column expression and the same expression appearing in a query, to improve query performance, especially for indexed computed JSON fields. The enhancement promotes data independence between the physical and application layer. **Enhancement**: Spilling for `GROUP BY` statements is enabled by default starting in engine version 7.8. Added an additional engine variable, `spilling_minimal_disk_space`. If a node has less disk space than `spilling_minimal_disk_space` (default is 500MB), queries on that node that require spilling will fail instead of spilling to disk. **Enhancement**: Materialized CTEs are now on by default and no longer considered a preview feature.Reduced the memory usage of `approx_count_distinct` by using a more compact representation. **Enhancement**: Existing queries are no longer recompiled on minor upgrades (from 7.8.x to 7.8.y for example). **New Function**: Added ALTER\_TIME, ALTER\_USER, and CREATE\_USER to the `information_schema.TABLES` table, to show the time of the latest update the table, the user who made the change, and the user who created the table. For existing tables, the ALTER\_TIME value will be NULL until the table is altered. For new tables, ALTER\_TIME will be the same as CREATE\_TIME. **New Function**: Added FLAGS column to the `information_schema.VIEWS` table, to indicate whether a view is a Table Valued Function (TVF). A value of IS\_TABLE\_VALUED\_FUNCTION indicates a TVF. **Enhancement**: An internal component, the LLVM code generation framework, was updated to version 10 from version 3.8. This improves performance of query compilation for DELETES on tables with a very large number of columns. ## November 2021 **New Feature**: Point-in-time recovery (PITR) has moved from preview to production status, and is now supported for production use cases. **Enhancement**: By default, all backups are now lock-free. Distributed write transactions no longer have to wait when a backup starts. **New feature**: Introduced row-level decompression for the string data type which will increase performance on reads against columnstore tables.  Before this improvement, decompression occurred on the order of blocks of data which consists of 4096 rows. **New Feature**: Added support for cross-database `INSERT...SELECT` into columnstore temporary tables. **Enhancement**: Allow spilling hash GROUP BY operator. **New Feature**: Added support for `SELECT ... INTO AZURE`. This command supports the WITH COMPRESSION option, which is described in the next release note. **New Feature**: Added the WITH COMPRESSION option to `SELECT … INTO FS`, `SELECT … INTO GCS`, and `SELECT … INTO S3`. WITH COMPRESSION writes the SELECT query results, in compressed .gzip files, to an object store. **New Feature**: Added support for new vector functions, namely `JSON_ARRAY_UNPACK`, `SCALAR_VECTOR_MUL`, `VECTOR_ADD`, and `VECTOR_MUL`. **New Feature**: Added support for the current user security model in stored procedures. In this model, when the current user executes a stored procedure, the stored procedure is executed using the security permissions of that user. **New Feature**: Added support for external functions, as a preview feature. An external function calls code that is executed outside of a SingleStore database. For more information, see [CREATE \[OR REPLACE\] EXTERNAL FUNCTION](https://docs.singlestore.com/cloud/reference/sql-reference/procedural-sql-reference/create-or-replace-external-function.md). **Enhancement**: Improved full-text filter performance when used with other secondary hash index filters. For highly selective full-text filters, the improvement in execution speed can be 10 times faster. **Enhancement**: Introduced new logic to determine when to evict a compiled image of a query plan. The logic will sort on the oldest number of plans while considering the explicitly set memory limit usage of each plan. The feature is disabled by default. To enable the logic, the engine variable `enable_compiled_images_eviction` must be set to ON. The engine variable `compiled_images_eviction_memory_limit_mb`  is used to set the memory limit. **New Feature**: Added a new function, `JSON_KEYS`, which returns the top-level keys of a JSON object in the form of a JSON array. Optionally, if a keypath is defined, returns the top-level keys from the keypath. **Enhancement**: Added support for more query shapes with FULL JOIN or correlated subselects when reference tables are involved. Prior to this release, these query shapes would hit a lockdown error. **New Feature**: Added support for query shapes that include repartitioned subqueries containing `SELECT` statements with aggregated column(s) without a GROUP BY clause. Prior to this release, these query shapes would hit a lockdown error. **New Feature**: Added support for LEFT JOIN when the left table is a reference table without a primary key. Prior to this release, this query shape would hit a lockdown error. **Enhancement**: Improved query execution for repartition DELETE FROM ...LIMIT and broadcast LEFT JOIN. **Enhancement**: Improved selectivity estimate for RIGHT JOIN query shapes when doing BloomFilter decision. See [Query Plan Operations](https://docs.singlestore.com/cloud/query-data/query-plan-operations.md) for a detailed explanation of BloomFilter and other filtering methods. **Enhancement**: Improved query performance using `SORT KEY()` and `KEY () WITH CLUSTERED COLUMNSTORE` columns with integer data types. **Enhancement**: Added password\_expiration column to the `information_schema.USERS` table. If the `password_expiration_seconds` engine variable is not enabled, the password\_expiration column will be NULL. If the `password_expiration_seconds` engine variable is enabled, the password\_expiration column will display the number of seconds remaining for the password to expire. **Enhancement**: Improved the performance of selective filters using string columns in columnstore tables. **New Feature**: Added support for UNION between reference and sharded tables.  Prior to this, this query shape would hit a lockdown error. **Enhancement**: Improved EXPLAIN output to clarify a result table for a broadcast LEFT JOIN or for a MATERIALIZE\_CTE as they can have the same result table name. For broadcast LEFT JOIN, a branch operator is added so that the branching operation on the shared result table is reflected. See [Query Plan Operations](https://docs.singlestore.com/cloud/query-data/query-plan-operations.md) for a detailed explanation of broadcasts and other distributed data movement. **New Feature**: Ingest, Added support for transactions in Kafka pipelines. ## In this section * [SingleStore Release Types](https://docs.singlestore.com/cloud/release-notes/singlestore-release-types.md) *** Modified at: June 1, 2026 Source: [/cloud/release-notes/](https://docs.singlestore.com/cloud/release-notes/) (An index of the documentation is available at /llms.txt) --- # SingleStore Release Types SingleStore systematically releases features to ensure quality, stability, and customer satisfaction. The following table, which summarizes SingleStore's release types, is intended to provide a clear, at-a-glance reference for what to expect at each stage of a feature's lifecycle. | | **Private Preview** | **Public Preview** | **General Availability (GA)** | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | **Who Can Use?** | Selected design partners or internal users | All customers | All customers | | **For Production?** | Non-production use only | Can be used in productionFeature may have limitations or issues | Fully supported | | **Interface Stability** | Not fully stable | Largely stable | Fully stable | | **SLA** | None | Full SLA applies to the clusterBest-effort support for the feature | Full SLA applies | | **Support** | Limited / Engineering team | Standard supportFeature may not be fully functional | Standard supportFeature is fully functional | | **Release State** | Invite-onlyFeature is in early testing | Broader testingKnown limitations documented | Official production releaseFinal customer success materials available | | **Documentation** | Documentation may be available on[docs.singlestore.com](https://docs.singlestore.com/)labeled as "Private preview" with a link to request access | Documentation available on[docs.singlestore.com](https://docs.singlestore.com/)labeled as "Preview" | Final documentation available on[docs.singlestore.com](https://docs.singlestore.com/) | *** Modified at: October 22, 2025 Source: [/cloud/release-notes/singlestore-release-types/](https://docs.singlestore.com/cloud/release-notes/singlestore-release-types/) (An index of the documentation is available at /llms.txt) --- # Security ## Overview From development to delivery, SingleStore ensures that security is considered, designed, reviewed, and implemented so that the data of our customers, and their customers, is safeguarded as if it were our own. SingleStore has built security into all its products, to those hosted by customers on their own infrastructure, to those SingleStore hosts on our customers’ behalf. Encryption, authentication, access, and monitoring are all things SingleStore is concerned with for you, so you can focus on the data and the value it can add. ## Security Program SingleStore has a holistic approach to information security combining a set of controls, both logical and administrative, that ensure the business meets both its compliance objectives and ensures the data of our customers is secure. SingleStore employs a combination of internal audit and external third-parties to ensure that all controls, all elements of the product, our approach to disaster recovery, and business continuity are tested annually. SingleStore believes in transparency, and all of our policies, audits, certifications, and details can be reviewed on our [Security Profile](https://singlestore.trust.site/). ## Certifications SingleStore has undergone an 18-month effort to transform the way the business approaches information security and data privacy. SingleStore has secured industry-leading security certifications including SOC 2 Type 2, and is also fully compliant to the requirements of HIPAA, CCPA, and GDPR. SingleStore continues to mature its information security posture and is passionate about meeting the security and compliance requirements of our customers. | ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc7c636b719336bcb/6a7adb7473d9bdeb202924cd/SOC-2-Badge-b7HQdH.png) | ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5466e6e22eda9a6a/6a7adb59c5d1934ed95db8f1/GDPR-Ready-Badge-3LhG76.png) | | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ## In this section * [Firewall](https://docs.singlestore.com/cloud/security/firewall.md) * [Privacy](https://docs.singlestore.com/cloud/security/privacy.md) * [Third Party Sub-processors](https://docs.singlestore.com/cloud/security/third-party-sub-processors.md) * [Portal Access](https://docs.singlestore.com/cloud/security/portal-access.md) * [Database Access](https://docs.singlestore.com/cloud/security/database-access.md) * [Administration](https://docs.singlestore.com/cloud/security/administration.md) * [Encryption](https://docs.singlestore.com/cloud/security/encryption.md) * [Audit Logging](https://docs.singlestore.com/cloud/security/audit-logging.md) * [Disaster Recovery](https://docs.singlestore.com/cloud/security/disaster-recovery.md) *** Modified at: July 17, 2026 Source: [/cloud/security/](https://docs.singlestore.com/cloud/security/) (An index of the documentation is available at /llms.txt) --- # Administration ## Overview This section covers configuring user accounts, including adding/removing users, updating passwords, and managing user permissions. ## In this section * [Role-Based Access Control (RBAC) for SingleStore Helios](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios.md) * [Role-Based Access Control (RBAC) at Database Level](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-at-database-level.md) * [Row-Level Security (RLS) Deployment Guide](https://docs.singlestore.com/cloud/security/administration/row-level-security-rls-deployment-guide.md) *** Modified at: November 22, 2022 Source: [/cloud/security/administration/](https://docs.singlestore.com/cloud/security/administration/) (An index of the documentation is available at /llms.txt) --- # Role-Based Access Control (RBAC) at Database Level This guide outlines how to secure SingleStore Helios [database users](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations.md) in a reduced-privileges, role-separated environment. ## Role-based Operations Users can access a database and execute their functions and responsibilities through the creation of [users](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md), [roles](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-role.md) and [groups](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-group.md), and [granting](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) of correct permissions (privileges). Here is a sample of standard roles. | Role | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Compliance Officer | Management for roles and schema authorizations. | | Security Officer | Full authority to view, modify, and create users and groups. Manages user passwords. | | Database Administrator | This role cannot execute backups, nor can it read any of the data within the database. Responsible for creating and removing databases. Ability to restore backups. | | ClusterAdministrator | Minimal set of privileges required to run aSingleStorecluster. | | Backup Operator | Authorization to performclusterbackups. | | Application Schema Owner | Dedicated, per-application role, authorized to execute create, alter, and delete DDL statements. Cannot view application data. | | Application Service Account | Dedicated, per-application role, authorized to execute select, update, insert, delete DML. | ## Relation Between Users, Roles and Groups ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt16233ac43b7200d0/6a7adb441967eadb5b325063/diagram_RBAC-in-SingleStore-FvA82r.jpg) 1. A role can have multiple privileges. 2. A group can have multiple roles. 3. A group can have multiple users. 4. A user can have multiple roles. 5. A user can be assigned to multiple groups. Users inherit the permissions, and roles of the groups they are assigned to. ## Role Creation SingleStore recommends that the following roles be used as a starting point for all use of the RBAC functionality. It is strongly suggested that these commands be kept in a separate, version-controlled, file and loaded into SingleStore. These scripts should be executed against the cluster endpoint. **Compliance Officer** ```sql CREATE ROLE 'compliance_role'; GRANT USAGE on *.* to ROLE 'compliance_role' WITH GRANT OPTION; CREATE GROUP 'compliance'; GRANT ROLE 'compliance_role' to 'compliance'; ``` **Security Officer** ```sql CREATE ROLE 'security_role'; GRANT CREATE USER on *.* to ROLE 'security_role'; CREATE GROUP 'security'; GRANT ROLE 'security_role' to 'security'; ``` **Database Administrator** ```sql CREATE ROLE 'dba_role'; GRANT CREATE DATABASE, DROP DATABASE on *.* to ROLE 'dba_role'; GRANT RELOAD on *.* to ROLE 'dba_role'; GRANT SHOW METADATA on *.* to ROLE 'dba_role'; CREATE GROUP 'dba'; GRANT ROLE 'dba_role' to 'dba'; ``` **Cluster Administrator** ```sql CREATE ROLE 'cluster_role'; GRANT CLUSTER on *.* to ROLE 'cluster_role'; GRANT SHOW METADATA on *.* to ROLE 'cluster_role'; CREATE GROUP 'cluster'; GRANT ROLE 'cluster_role' to 'cluster'; ``` **Backup Operator** ```sql CREATE ROLE 'backup_operator_role'; GRANT BACKUP, RELOAD on *.* to ROLE 'backup_operator_role'; CREATE GROUP 'backup_admins'; GRANT ROLE 'backup_operator_role' to 'backup_admins'; ``` **Application Schema Owner** Create one for each distinct application: ```sql CREATE ROLE 'app__schema_role'; GRANT CREATE, ALTER, DROP on .* to ROLE 'app__schema_role'; GRANT CREATE VIEW, ALTER VIEW, DROP VIEW on .* to ROLE 'app__schema_role'; GRANT SHOW VIEW on .* to ROLE 'app__schema_role'; GRANT CREATE TEMPORARY TABLES on .* to ROLE 'app__schema_role'; CREATE GROUP 'app__schema'; GRANT ROLE 'app__schema_role' to 'app__schema'; ``` **Application Service Account** Create one for each distinct application: ```sql CREATE ROLE 'app__role'; GRANT SELECT, INSERT, UPDATE, DELETE on .* to ROLE 'app__role'; GRANT SHOW VIEW on .* to ROLE 'app__role'; GRANT LOCK TABLES on .* to ROLE 'app__role'; GRANT CREATE TEMPORARY TABLES on .* to ROLE 'app__role'; CREATE GROUP 'app_'; GRANT ROLE 'app__role' to 'app_'; ``` ## Account Creation SingleStore recommends the following roles be used as a starting point for all use of the RBAC functionality. It is strongly suggested that these commands be kept in a separate, version-controlled, file and loaded into SingleStore. Depending on requirements, these scripts should be executed on all nodes where users will connect, typically all aggregators and optionally leaves. **Compliance Officer** ```sql GRANT USAGE ON *.* TO 'compliance_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'compliance' TO 'compliance_user'; ``` **Security Officer** ```sql GRANT USAGE ON *.* TO 'security_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'security' TO 'security_user'; ``` **Database Administrator** ```sql GRANT USAGE ON *.* TO 'dba_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'dba' TO 'dba_user'; ``` **Cluster Administrator** ```sql GRANT USAGE ON *.* TO 'cluster_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'cluster' TO 'cluster_user'; ``` **Backup Operator** ```sql GRANT USAGE ON *.* TO 'backup_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'backup_admins' TO 'backup_user'; ``` **Application Schema Owner** Create one for each distinct application. ```sql GRANT USAGE ON *.* TO 'app__schema_user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'app__schema' TO 'app__schema_user'; ``` **Application Service Accounts** Create one for each distinct application. For additional release validation, one may consider creating a number of accounts for the same application and rotating them during code releases. They should all have unique names and unique passwords. ```sql GRANT USAGE ON *.* TO 'app__user' IDENTIFIED BY 'AVery$ecurePassword123'; GRANT GROUP 'app__schema' TO 'app__user'; ``` ## Appendix: Role-Based Access Control Command Reference The following commands are available to query the status of the different users, groups, and roles in SingleStore running in RBAC mode. ```sql SHOW USERS; SHOW USERS FOR ROLE 'role'; SHOW USERS FOR GROUP 'group'; SHOW GROUPS; SHOW GROUPS FOR ROLE 'role'; SHOW GROUPS FOR USER 'user'@'%'; SHOW ROLES; SHOW ROLES FOR USER 'user'@'%'; SHOW ROLES FOR GROUP 'group'; ``` *** Modified at: June 4, 2025 Source: [/cloud/security/administration/role-based-access-control-rbac-at-database-level/](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-at-database-level/) (An index of the documentation is available at /llms.txt) --- # Role-Based Access Control (RBAC) for SingleStore Helios SingleStore Helios role-based access control (RBAC) framework controls access to functionality in the SingleStore portal. It allows you to add RBAC functionalities and control access at different levels such as organizations and cluster groups. Actions may be disabled or hidden in the SingleStore Helios portal based on the roles granted to a user. This RBAC framework controls access only to the administrative features of SingleStore Helios. The SingleStore database controls access to the databases. ## Benefits of RBAC * **Increased security**: With RBAC, access to SingleStore Helios is strictly limited to those who need it. This greatly reduces the risk of unauthorized access or changes to the network. * **Least privilege principle**: RBAC aligns with the principle of least privilege, which means that users only have the rights they need to perform their job functions and nothing more. This prevents excessive permissions that could lead to security risks. * **Efficient access management**: RBAC allows you to manage users' access rights efficiently. Instead of manually assigning or changing rights for every individual user, you can assign or revoke roles with predefined permissions. * **Scalability:** RBAC scales well as your organization grows. As you add more users, it becomes more beneficial to manage access rights through roles rather than on an individual basis. * **Regulatory compliance**: Many regulatory standards require strict control over who has access to critical data and systems. RBAC helps meet such requirements. * **Reduced chance of human error**: By limiting who can make changes to access via the SingleStore Helios portal, you reduce the chance of human error leading to network issues. ## RBAC Framework Implementation SingleStore Helios RBAC restricts individual database users' access to resources by setting up a role-based environment. Each user is assigned one or more roles based on their functionality. User permissions in a cloud deployment are tied to the hierarchy of the objects associated with the organization. ![Diagram showing relationships between organization users.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltc7e9edaec4bb4e9c/6a7adb9d2f00b213afef366d/Cloud_RBAC_new_001-9mfi9G.jpeg) ## Objects Associated with Users and Roles in the Cloud RBAC Framework * **Organization**: This is the highest level resource that is tied to a customer. Organizations have a 1:1 mapping to customers. * At this level, you can view all the clusters, cluster groups, databases, etc. * Billing is done at the organization level but visibility is available at the cluster group level. * Members can be added and removed from the organization at this level. * **Team**: This is a group of users. A predefined set of teams corresponding to the organization level roles are created when RBAC is enabled. Additional teams may be created to identify groups of users with specific access roles. Users can be added or removed from the team(s). * **Cluster group**: This is a set of clusters grouped together. * **Cluster**: This is an independent deployment of compute resources that can be used to run a workload. Clusters are isolated instances and can read multiple databases. They are useful in maintaining separate environments for example for production and development. * **Database**: A database contains objects such as tables, views, stored procedures, pipelines etc. The RBAC framework for organization users in the SingleStore Helios portal does not enforce any access control on databases. All data access is controlled by the RBAC framework for database users in the database engine. ## Difference Between Teams and Roles Teams simplify granting access to users with similar functions. Users can be added to any of the default teams or any custom team when they are invited into an organization or later. This allows a user to be added to the organization for a specific function (for example, an Operator or an Observer) as part of the invitation process. You do not have to explicitly grant or revoke roles to each user at the organization level. If you want to control access at a lower level, i.e., at the individual cluster group level, then you will need to grant roles. You can grant roles directly to users or you can create teams that are granted roles and then just manage membership in these team(s). ## Cloud User and Role Synchronization with the Database Engine SingleStore Helios RBAC does not directly provide any access control to actions performed in the SingleStore database engine. In the engine, access is controlled by the [database RBAC features](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-at-database-level.md). Cloud user and role synchronization provides a unified user experience for basic security patterns. More complex patterns, such as limited access to specific databases or defining custom database roles, are performed with [SQL security management commands](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands.md) in the engine. Hence, SingleStore Helios cannot be used to grant access to individual databases. You must use DB (SQL security management) RBAC to control access to specific databases. The built-in roles (e.g., Reader or Writer) correspond to engine roles that grant access to all databases in the cluster group. When a user is granted any role in a cluster group, synchronization automatically takes place with a corresponding user group existing in the engine. In the backend, engine user groups are granted roles with permissions appropriate for the Cloud RBAC role. Refer [here](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/predefined-roles-for-cluster-groups-in-an-organization/#section-idm4583707817452833931179634459.md) for the pre-defined mapping between Cloud RBAC roles and the engine user groups and roles. ## Rules for the RBAC Framework * **Ownership of objects**: Owners of a parent object are the only users allowed to create child objects. For example, only organization owners can create a cluster group. Object owners inherit the owner role on all child objects. Organization owners are owners of all objects in the organization. * **Inheritance of roles in object hierarchy**: There is an inheritance of ownership of objects. For example, owners of cluster groups are also owners of all clusters in the group. * **Granting of privileges and roles**: This is not limited to users but can also be given to a group which is a collection of users. ## In this section * [Predefined Organization Teams and their Corresponding Roles](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/predefined-organization-teams-and-their-corresponding-roles.md) * [Predefined Roles for Cluster Groups in an Organization](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/predefined-roles-for-cluster-groups-in-an-organization.md) * [How to Use SingleStore Helios RBAC](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/how-to-use-singlestore-helios-rbac.md) * [RBAC Best Practices and Use Cases](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/rbac-best-practices-and-use-cases.md) *** Modified at: May 12, 2026 Source: [/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/) (An index of the documentation is available at /llms.txt) --- # Row-Level Security (RLS) Deployment Guide ## Row-Level Security (RLS) Configuration Row-Level Security (RLS) allows only those users who have the required permissions to access data by rows in a database. For example, RLS can be used to restrict each salesman to access only those rows in a table that are relevant to their sales details. RLS restricts users' access to data at the database level instead of the application level. The database applies this access control to the table whenever a query runs, irrespective of which application needs the data. Row-level security in SingleStore Helios is achieved by creating a view on a table with a special roles column. RLS works for read-only, you cannot restrict write access on a table using this configuration. ## Setting up a table for row-level security For a table to be used with row-level security, it must have a VARBINARY column where a row entry in the column contains a comma separated list of roles which have access to that row. There are special formatting constraints for the roles columns which are discussed below. Consider the following table containing 4 rows: | ACCESS\_ROLES | DATA\_1 | DATA\_2 | DATA\_3 | | ----------------- | ------- | ------- | ------- | | ,ROLE\_A,ROLE\_B, | xxxxxx | xxxxxx | xxxxxx | | ,ROLE\_A,ROLE\_C, | xxxxxx | xxxxxx | xxxxxx | | ,ROLE\_D, | xxxxxx | xxxxxx | xxxxxx | | , | xxxxxx | xxxxxx | xxxxxx | For a given role, the `ACCESS_ROLES` field will be used to specify which roles have access to that row. The `DATA_1` through `DATA_3` columns are data stored in a table. It is important that each role name in `ACCESS_ROLES` be surrounded by a preceding and trailing comma. In addition to applying RLS using an access-control list field, you can also restrict access based on the data in the rows of a table. For example, grant access based on an owner id or restrict access to only rows for a specific year, 2022. ## Creating a new table with row-level security To create a new table with a roles column use an appropriate version of the following command: ```sql CREATE TABLE (ACCESS_ROLES VARBINARY() DEFAULT ",", …); ``` It is important that the default value for `ACCESS_ROLES` be a comma: `","` for row-level security to work correctly. The `` of the `ACCESS_ROLES` column should be set to match the expected number of roles. Note that the size can always be changed using: ```sql ALTER TABLE
MODIFY COLUMN ACCESS_ROLES VARBINARY(); ``` ## Updating an existing table with row-level security To update an existing table to work with row-level security, use the following command: ```sql ALTER TABLE
ADD COLUMN ACCESS_ROLES VARBINARY() DEFAULT ","; ``` Once again it is important to set the default value of the `ACCESS_ROLES` column to a comma (`","`) for row-level security control to function correctly. ## Granting row access to a role To give a new role access to a column, append the role name followed by a comma to the column of a table containing role access information. If the role access column is called `ACCESS_ROLES`, then the following query gives the role `ROLE` access to all rows whose ID column is 5. ```sql UPDATE
SET ACCESS_ROLES=CONCAT(ACCESS_ROLES, "ROLE,") WHERE ID=5; ``` To give access to multiple roles R1, R2, R3 to rows whose ID column is 5: ```sql UPDATE
SET ACCESS_ROLES=CONCAT(ACCESS_ROLES, "R1,R2,R3,") WHERE ID=5; ``` It is important that each new role added is suffixed with a comma. ## Revoking row access from a role To revoke access from a role, R, on rows whose ID column in a table is 5: ```sql UPDATE
SET ACCESS_ROLES=REPLACE(ACCESS_ROLES, ',R,', ',') WHERE ID=5; ``` It is important for the role name being revoked be surrounded by commas in the `REPLACE` function. ## Updating and inserting rows in a row-level security table When updating rows in a row-level security table, it is important to not update the roles access column. When inserting into a row-level security table, it is important that new values in the roles access column have a default value of a comma. If the table is configured correctly, a comma will be the default value of the roles access column. The following query inserts values into a table with 4 rows (ACCESS\_ROLES, A, B, C), the first of which is the roles access column: ```sql INSERT INTO
(A, B, C) VALUES(…); ``` ## Creating Views on row-level security Tables To restrict access on a row-level security table whose role access column is called `ACCESS_ROLES`, the following view must be created on the table: ```sql CREATE VIEW AS SELECT COLUMNS FROM
WHERE SECURITY_LISTS_INTERSECT(CURRENT_SECURITY_ROLES(), ACCESS_ROLES); ``` ## Limitations RLS restricts the results of a query returned for each user based on their permissions but does not guarantee security against side-channel attacks. For example, a malicious user with access to an RLS view may connive another user with a different set of permissions on a table to extract information. A carefully designed query can trigger exceptions on non-permitted data, timing dependence, EXPLAIN command options, etc. For example, using a query with  "1/(total\_value - 200000)" on a Tenders table may result in a “divide by zero” error, thereby indicating there is at least one tender with this value. There are mitigations that can be introduced with RLS, for example, turning on audit logging to check for suspicious queries. *** Modified at: February 15, 2023 Source: [/cloud/security/administration/row-level-security-rls-deployment-guide/](https://docs.singlestore.com/cloud/security/administration/row-level-security-rls-deployment-guide/) (An index of the documentation is available at /llms.txt) --- # Audit Logging > **📝 Note**: Audit logging is available when using [Enterprise edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md), where logs can be streamed to, and accessed through, third-party audit tools. SingleStore Helios logs activities on both the **Control Plane** and **Data Plane**. This feature is useful for performing common information security tasks such as auditing, investigating suspicious activity, and validating access control policies. ## Audit Logging Architecture ## Control Plane The Control Plane logs actions performed in the [Cloud Portal](https://portal.singlestore.com), for example, cluster management, user administration, authentication events, changes to Cloud Portal configuration, and more. These logs help administrators monitor changes to the infrastructure, ensure compliance with security policies, and investigate access-related events. Access Control Plane logs via the `/auditlogs` endpoint in the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md). Refer to [Control Plane Audit Logs](https://docs.singlestore.com/cloud/security/audit-logging/control-plane-audit-logs.md) for more information. ## Data Plane The Data Plane logs database related operations, for example, queries, system responses, administrative operations, etc. Additionally, SingleStore provides multiple logging levels for the Data Plane, and each level provides limited or exhaustive information about user actions and database responses. These logs are essential for monitoring user activity, detecting abnormal behavior, and analyzing data access patterns. Data plane audit logs can be forwarded to and accessed using third-party audit tools. Refer to [Data Plane Audit Logs](https://docs.singlestore.com/cloud/security/audit-logging/data-plane-audit-logs.md) for more information. ## In this section * [Control Plane Audit Logs](https://docs.singlestore.com/cloud/security/audit-logging/control-plane-audit-logs.md) * [Data Plane Audit Logs](https://docs.singlestore.com/cloud/security/audit-logging/data-plane-audit-logs.md) *** Modified at: December 22, 2025 Source: [/cloud/security/audit-logging/](https://docs.singlestore.com/cloud/security/audit-logging/) (An index of the documentation is available at /llms.txt) --- # Control Plane Audit Logs SingleStore identifies and logs the user actions in the Control Plane that can be used to track user activity. Logged [Cloud Portal](https://portal.singlestore.com) activities include, but are not limited to, the following: | Category | Event | | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Cluster |
  • Create, resize, suspend, or delete a cluster
  • Detach a database from a cluster
  • Modify auto-suspend configuration of a cluster
  • Create or delete private connections to a cluster
| | Clustergroup |
  • Create or delete a cluster group
  • Add or update firewall rules for a cluster group
  • Update the password
  • Add RBAC rules at the cluster group level
  • Change the update window for a deployment
  • Create a database
| | Observability |
  • Add, update, or remove an alert for the database operations
  • Add, update, or remove subscribers to the alert
| | Cloud Portal |
  • Generate API keys
  • Add a user
  • Add a RBAC property at the organization level
  • Add a payment method
  • Change the database admin password
| | Team |
  • Create, update, or delete a team
  • Add or delete members in a team
| | Authentication |
  • User login events
  • Add, update, or delete a SCIM connection
| | Notebook |
  • Create or delete a notebook on the Cloud Portal
| ## Access Control Plane Audit Logs Use the `AuditLogs` path (`/v1/auditLogs` endpoint) in the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) to access the Control Plane audit logs. For example: ```shell curl -X 'GET' \ 'https://api.singlestore.com/v1/auditLogs' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' ``` The following parameters can be used to filter the audit logs fetched from the `Management` API: | Parameter | Description | | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type``source``firstName``lastName``email``userID` | Returns audit log entries that match the specified value for the respective field in the audit log. Refer to[Control Plane Audit Log Format](https://docs.singlestore.com/#section-id235122503466303.md)for related information. | | `startDate` | Returns audit log entries where the`createdAt`timestamp is greater than the specified`startDate`. | | `endDate` | Retrieves audit log entries where the`createdAt`timestamp is earlier than the specified`endDate`. | | `limit` | Limits the number of returned audit log entries to the specified value. | | `nextToken` | Returns the audit log entries generated after the specified`nextToken`value. | | `sortByDescending` | If enabled, audit log entries are returned in descending order by the`createdAt`timestamp (newest first). | Refer to [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. ## Control Plane Audit Log Format The `Management` API returns the Control Plane audit logs formatted as JSON records. For example: ```json { "nextToken": "2025-08-23T20:10:24.566869Z", "auditLogs": [ { "orgID": "d4226b30-0000-1000-9000-e56cc36be262", "auditID": "9a189082-0000-1000-9000-d574adbbe889", "type": "WorkspaceGroupCreate", "source": "Portal", "createdAt": "2025-07-22T03:15:06.693055Z", "reason": "Workspace group (1baae325-0000-1000-9000-17b67a0af1ce) created in project (3c8d7c87-0000-1000-9000-feebd4f92dcf)", "userType": "Employee", "version": 0 } ]} ``` Each Control Plane audit log entry may contain a subset of the fields listed in the following table: | **Field** | **Description** | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `orgID` | ID of the organization associated with this event. | | `auditID` | ID of the audit log entry. | | `type` | Type of the audit log event. Refer to[Audit Log Event Types](https://docs.singlestore.com/#section-id235122508390412.md)for more information. | | `userEmail` | Email ID of the user that triggered this event. | | `userID` | ID of the user that triggered this event. | | `userType` | Type of the user that triggered this event. It can have one of the following values:`Unspecified`,`System`,`Employee`,`Customer`,`Automation`,`Studio`,`CIAutomation`,`SharedTier`,`SNIProxy`,`NimbusGateway`, or`Cluster`. | | `workspaceID` | ID of theclusterassociated with this event. | | `projectID` | ID of the project associated with this event. | | `sessionID` | ID of the authorization session associated with this event. | | `teamID` | ID of the team associated with this event. | | `source` | Source of the audit log entry. It can have one of the following values:`Portal`,`Admin`, or`SystemJob`. | | `createdAt` | The timestamp (in the RFC3339Nano format) of when the audit log entry was created. | | `attributes` | Additional keys and values specific to the audit log. | | `error` | Error message, if any, related to this audit log entry. | | `reason` | A detailed description of the event. | | `firstName` | First name of a redacted user. | | `lastName` | Last name of a redacted user. | | `labels` | List of audit-related keywords. | | `nextToken` | Timestamp of the latest audit log entry.If specified in the API call, the API only returns the audit log entries generated after the specified timestamp. | ## Control Plane Audit Log Event Types The following is a list of logged event types for each category: | **Category** | **Event Type** | | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Notebook Management |
  • NotebookCreate
  • NotebookDelete
| | | Cluster Operations |
  • ClusterCreate
  • ClusterTerminate
| | | Workspace Management |
  • WorkspaceGroupTerminate
  • WorkspaceTerminate
  • WorkspaceCreate
  • WorkspaceGroupCreate
|
  • WorkspaceScale
  • WorkspacePause
  • WorkspaceResume
  • WorkspaceGroupDelete
| | Alert Management |
  • SubscribedAlert
  • UnsubscribedAlert
|
  • SubscriberAlertUpdated
| | User Management |
  • InvitedNewUserToPortal
  • AddedNewUserToPortal
|
  • UserRemovedFromPortal
| | Team Management |
  • TeamCreate
  • TeamUpdate
  • TeamDelete
|
  • TeamAddMember
  • TeamRemoveMember
| | Billing & Payment |
  • NewPaymentMethod
  • NovaPoolBillingUpdate
  • ComputeOverageChange
  • BalanceCorrection
|
  • BalanceIncrease
  • SubscriptionBundleCreated
  • SubscriptionBundleDeleted
  • SubscriptionBundleUpdated
| | Security & Access |
  • GenerateAPIKey
  • RevokeAPIKey
  • WorkspaceGroupAccessGranted
  • WorkspaceGroupAccessRevoked
  • OrganizationAccessGranted
  • OrganizationAccessRevoked
  • NovaAppAccessGranted
  • NovaAppAccessRevoked
  • NovaServiceAPIKeyCreate
  • NovaServiceAPIKeyRevoke
|
  • ChangeAdminDatabasePassword
  • WorkspaceAccessGranted
  • WorkspaceAccessRevoked
  • AgentDomainAccessGranted
  • AgentDomainAccessRevoked
  • SecretAccessGranted
  • SecretAccessRevoked
  • NotifySAMLCertExpire
  • UpdateOAuthIntegrationAvailability
  • MFAMethodChange
| | Maintenance & Config |
  • MaintenanceWindowUpdate
  • AutoSuspendDisableOnResume
|
  • UpdateAutoSuspendConfiguration
| | Firewall Management |
  • FirewallRuleAdded
  • FirewallRuleUpdate
|
  • FirewallRangesUpdate
| | Storage Management |
  • BottomlessRetentionPeriodUpdate
| | | Job Scheduling |
  • CreateScheduledJob
  • ScheduledJobUpdate
|
  • ScheduledJobRemove
| | Database Operations |
  • DatabaseCreated
  • DatabaseDetached
|
  • DatabaseRemoved
| | Network & Connectivity |
  • PrivateConnectionCreated
  • PrivateConnectionDeleted
| | | SCIM Integration |
  • SCIMCreateConnection
  • SCIMUpdateConnection
|
  • SCIMDeleteConnection
| | Operations API |
  • OpsAPIRequest
| | | Encryption Management |
  • CMEKScheduleDeletionAlert
| | *** Modified at: April 24, 2026 Source: [/cloud/security/audit-logging/control-plane-audit-logs/](https://docs.singlestore.com/cloud/security/audit-logging/control-plane-audit-logs/) (An index of the documentation is available at /llms.txt) --- # Data Plane Audit Logs SingleStore logs database activities and writes the generated logs to an external location. You can forward Data Plane audit logs and access them through third-party audit tools such as Amazon CloudWatch, Datadog, and others. Contact [SingleStore Support](https://singlestore.com/support) to set up audit log forwarding with a supported third-party tool. ## Forwarding Data Plane Audit Logs > **⚠️ Warning**: Including customer credentials or other sensitive or confidential information through Zendesk or email puts both you, our customer, and SingleStore at risk, as the subject credentials may be exposed to unauthorized parties.Please use a secure secrets management solution when sharing credentials via any means with SingleStore or any other third party. SingleStore Helios supports forwarding Data Plane audit logs to the following third-party tools/integrations: * Amazon CloudWatch * Amazon S3 * Azure Blob * Azure Log Analytics * Datadog * Google Cloud Logging (via [Stackdriver](https://docs.fluentbit.io/manual/data-pipeline/outputs/stackdriver)) * New Relic * Splunk ## Configure Audit Log Forwarding Contact [SingleStore Support](https://singlestore.com/support) and provide the requested information to configure audit log forwarding, including the audit logging level. ## Requirements Here are the requirements for some common supported third-party integrations: ## Amazon CloudWatch * Your AWS access key ID and secret access key. * `log_group_name`: The name of the CloudWatch Log Group to send records to. * `log_stream_name`: The name of the CloudWatch Log Stream to send records to. * `region`: The AWS region. Refer to [Amazon CloudWatch](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch) for more information. ## Amazon S3 * Your AWS access key ID and secret access key. * `bucket`: The name of your S3 bucket. * `region`: The AWS region of your S3 bucket. Refer to [Amazon S3](https://docs.fluentbit.io/manual/pipeline/outputs/s3) for more information. ## Datadog * `apikey`: Your Datadog API key. * Host: The Datadog server to send logs to. * TLS: Enable/disable end-to-end security communications protocol. Refer to [Datadog](https://docs.fluentbit.io/manual/pipeline/outputs/datadog) for more information. ## Access Data Plane Audit Logs You can view the Data Plane audit logs once they are configured and forwarded to the specified third-party tool. Here's a sample audit log from a log group configured in Amazon CloudWatch: ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt61f21fa5acbbbe2d/6a7adb4af6825714cb58d973/image1-ULriag.png) ## Audit Log Formats Data Plane audit logs are parsed and represented as JSON-formatted records in third-party tools. Each forwarded audit log record follows a schema specific to its audit log type. The following sections describe the corresponding record schema for each audit log type. ## First Line Schema The first line of a log file has a unique schema that describes the event or action that resulted in the creation of a new log file. It has the following schema: | **Field** | **Description** | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Represents the unique identifier for the log entry within the scope of a single log file. This value is always`0`for the first line in a log file. | | `time` | Specifies the local system time when the file was opened or rotated, using the same timestamp formatting as other audit log records. | | `origin` | Describes the reason or event that caused the creation of the log file. It can have the following values:
  • Log rotated: Indicates that a new file was required because rotation conditions were met.
  • Log opened on MemSQL startup: Indicates that a new file was created because the node was either started or restarted.
| | `version` | Specifies current audit logging format version. | | `action` | Describes the audit logging state associated with the new file. It can have the following values:
  • Log level initiated at <auditlog-level>: Indicates that audit logging was first enabled at a given level and then the log file was opened.
  • Log level changed to <auditlog-level>: Indicates that logging was already enabled, but the audit logging level was changed.
  • Logging at level <auditlog-level>: Indicates that the system was restarted or the log was rotated, but the audit log level remained the same.
| For example: ```json { "id": "0", "time": "2026-06-21 18:49:25.888", "origin": "Log opened on MemSQL startup", "version": "Format version 4", "action": "Logging at level ADMIN-ONLY-INCLUDING-PARSE-FAILS" } ``` ## Login and Logout Schema Each login attempt and logout event is logged using the following schema: > **📝 Note**: The logout event log schema does not contain the `authtype` and `status` fields. | **Field** | **Description** | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Represents the unique identifier for the log entry within the scope of a single log file. | | `time` | Specifies the node’s local system time when an operation was executed. | | `timezone` | Specifies the local time zone when the event was logged. | | `endpoint` | Specifies the hostname or IP address of theSingleStore Heliosclusterin the`:`format. | | `nodetype` | Specifies the type of node. It can have one of the following values:
  • agg: For aggregators.
  • leaf: For leaf nodes.
| | `type` | Specifies the login or logout event type. It can have the following values:
  • USER_LOGIN: For login attempts.
  • USER_LOGOUT: For logout event.
| | `threadid` | Specifies the ID of the thread that is running the statement. | | `username` | Specifies the username of the user performing the login or logout. | | `remotehost` | Specifies the hostname or IP address that the client is using to connect to thecluster. This value is detected by the server and not reported by the client. | | `usergrant` | Specifies the`'username'@'hostname'`specification from the`GRANT`that defines the user being authenticated or logged out. | | `authtype`(Login attempts only) | Specifies the authentication method for the user, for example,`password`,`PAM`,`Kerberos`, etc. | | `status`(Login attempts only) | Indicates the success or failure of the login attempt. It can have the following values:
  • SUCCESS: For a successful login.
  • FAILURE: reason: For a failed login attempt. Optionally, includes a reason that specifies why the login attempt failed.
| | `tlscipher` | Specifies the TLS encryption algorithm used during the session. | | `appname` | Specifies the client application name retrieved from the session's`program_name`attribute, if set. | | `osuser` | Specifies the client OS user retrieved from the session's`_os_user`attribute, if set. | For example: ```json { "id": "363750385", "time": "2026-08-11 18:53:18.150,UTC", "endpoint": "c770dd909a9c:3306", "nodetype": "agg", "type": "USER_LOGIN", "threadid": "100000", "username": "pac", "remotehost": "192.168.0.1", "usergrant": "pac@%", "authtype": "mysql_native_password", "status": "SUCCESS", "tlscipher": "ECDHE-RSA-AES256-GCM-SHA384", "appname": "mysql", "osuser": "osuser" } ``` ## Completion Record Schema for Query Execution Completion records capture the final outcome of a query after execution. They are emitted as a separate record, associated with the original query record by `id`. Completion records have the following schema: | Field | Description | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | Specifies the audit-log record ID of the original query record, used to correlate the command completion log to the original query. | | `record_type` | A record-type marker which indicates a query record. Its value is always`C`for a completion record. | | `status` | Specifies the final status of the query. It can have the following values:
  • SUCCESS: Indicates successful execution of the query.
  • FAILURE:<error_code>: Indicates query failure and includes the error returned by the SingleStore engine.
| | `records_affected` | Specifies the number of affected or returned records recorded at completion time. | For example: ```JSON { "id": 123456, "record_type": "C", "status": "SUCCESS", "records_affected": 42 } ``` ## Schema for General Entries Queries are logged as general entries in the following format: ```json { "id": "28151", "endpoint": "node-37775b0a-6feb-4b61-98a7-master-0:3306", "nodetype": "agg", "aggregatorid": "1", "threadid": "99996", "username": "root", "database": "[unknown]", "queryhash": "3505599087362974304", "query": "GRANT CREATE LINK\\,DROP LINK\\,SHOW LINK ON *.* TO 'admin'" } ``` | **Field** | **Description** | | -------------- | ------------------------------------------------------------------------------------------------------ | | `id` | Represents the unique identifier for the log entry within the scope of a single log file. | | `endpoint` | Specifies the hostname or IP address of theSingleStore Helioscluster. | | `nodetype` | Specifies the type of node. It can have one of the following values:`agg`(aggregator) and`leaf`(leaf). | | `aggregatorid` | Specifies the ID of the aggregator that is running the query. | | `threadid` | Specifies the ID of the thread that is running the query. | | `username` | Specifies the username that is running the query. | | `database` | Specifies the database name. | | `queryhash` | Specifies the unique hash of the query signature or the query text. | | `query` | Specifies the query signature or the query text. | ## Audit Logging Levels There are 11 logging levels that are organized into the following categories, each with increasing levels of verbosity: * Log only valid statements and queries: * `LOGINS-ONLY` * `ADMIN-ONLY` * `WRITES-ONLY` * `ALL-QUERIES` * `ALL-QUERIES-PLAINTEXT` * `ALL-RESULTS` * Log valid and invalid statements and queries: * `ADMIN-ONLY-INCLUDING-PARSE-FAILS` (default) * `WRITES-ONLY-INCLUDING-PARSE-FAILS` * `ALL-QUERIES-INCLUDING-PARSE-FAILS` * `ALL-QUERIES-PLAINTEXT-INCLUDING-PARSE-FAILS` * `ALL-RESULTS-INCLUDING-PARSE-FAILS` A valid statement or query is one that can be successfully parsed by SingleStore. Invalid statements or queries include those with misspellings or improper syntax. > **⚠️ Warning**: User credentials and PII information contained in all valid statements and queries is obfuscated in audit logs. When invalid statements cannot be parsed, the literal query text is included in the log entry. This text may contain sensitive information. For example, if a user attempts to connect to the database with an invalid statement that contains their username and password, these values will be logged in plain text. You can optionally filter logs for the root user only and exclude the audit of information schema queries. To change the current audit logging level contact [SingleStore Support](https://support.singlestore.com/). ## LOGINS-ONLY The `LOGINS-ONLY` level logs the successful and failed login attempts into the database. These login activities coexist with audit data from existing levels, but the information about logins follows a different format than database activities. The `LOGINS-ONLY` level does not include the `PARSE-FAILS` option for logins, because this option is reserved for queries. The `LOGINS-ONLY` level is the lowest level of the audit log, i.e. every other audit level is inclusive of the login information. ## ADMIN-ONLY and ADMIN-ONLY-INCLUDING-PARSE-FAILS The `ADMIN-ONLY` level is inclusive of the `LOGINS-ONLY` level logs, and it also logs DDL operations such as `CREATE`, `DROP`, `ALTER`, etc. Additionally, if a query contains passwords (such as `SET PASSWORD`), the password’s value will be omitted from the log. The `ADMIN-ONLY-INCLUDING-PARSE-FAILS` level is inclusive of `ADMIN-ONLY` but also logs invalid statements that fail to parse. These invalid statements may include sensitive information that would normally be obfuscated in a log entry. ## WRITES-ONLY and WRITES-ONLY-INCLUDING-PARSE-FAILS The `WRITES-ONLY` level is inclusive of `ADMIN-ONLY` but also logs DML operations such as `INSERT`, `DELETE`, `UPDATE`, with one exception: `SELECT` queries are not logged. The `WRITES-ONLY-INCLUDING-PARSE-FAILS` level is inclusive of `WRITES-ONLY` but also logs invalid statements that fail to parse. These invalid statements may include sensitive information that would normally be obfuscated in a log entry. ## ALL-QUERIES and ALL-QUERIES-INCLUDING-PARSE-FAILS The `ALL-QUERIES` level is inclusive of `WRITES-ONLY` but also logs read operations such as `SELECT` statements. The `ALL-QUERIES-INCLUDING-PARSE-FAILS` level is inclusive of `ALL-QUERIES` but also logs invalid statements that fail to parse. These invalid statements may include sensitive information that would normally be obfuscated in a log entry. ## ALL-QUERIES-PLAINTEXT and ALL-QUERIES-PLAINTEXT-INCLUDING-PARSE-FAILS The `ALL-QUERIES-PLAINTEXT` level is inclusive of `ALL-QUERIES` but also logs the entire literal query, not just the parameterized version without literal strings and numbers. The `ALL-QUERIES-PLAINTEXT-INCLUDING-PARSE-FAILS` level is inclusive of `ALL-QUERIES-PLAINTEXT` but also logs invalid statements that fail to parse. These invalid statements may include sensitive information that would normally be obfuscated in a log entry. ## ALL-RESULTS and ALL-RESULTS-INCLUDING-PARSE-FAILS The `ALL-RESULTS` level is inclusive of `ALL-QUERIES-PLAINTEXT` but also logs the database’s responses. This is the most verbose logging level available, and accordingly it produces a large number of log entries. When this level is selected, the entry ID for a query is shared with the results entries. The `ALL-RESULTS-INCLUDING-PARSE-FAILS` level is inclusive of `ALL-RESULTS` but also logs invalid statements that fail to parse. These invalid statements may include sensitive information that would normally be obfuscated in a log entry. *** Modified at: July 3, 2026 Source: [/cloud/security/audit-logging/data-plane-audit-logs/](https://docs.singlestore.com/cloud/security/audit-logging/data-plane-audit-logs/) (An index of the documentation is available at /llms.txt) --- # Database Access ## In this section * [Authenticate via JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md) * [Authenticate via Browser-based SSO using JWTs](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-browser-based-sso-using-jwts.md) * [Configuring a Password Policy](https://docs.singlestore.com/cloud/security/database-access/configuring-a-password-policy.md) * [Configuring and Using Connection Links](https://docs.singlestore.com/cloud/security/database-access/configuring-and-using-connection-links.md) *** Modified at: February 25, 2026 Source: [/cloud/security/database-access/](https://docs.singlestore.com/cloud/security/database-access/) (An index of the documentation is available at /llms.txt) --- # Authenticate via Browser-based SSO using JWTs > **📝 Note**: SingleStore Helios supports browser-based single sign-on (SSO) with Google Chrome, Mozilla Firefox, and Microsoft Edge. To use browser-based SSO on macOS, you will need to set your default browser to either Chrome or Firefox.SingleStore Helios supports authentication via browser-based SSO using JWTs only for the SingleStore JDBC driver and SingleStore ODBC driver. The SingleStore authentication helper utility, [singlestore-auth-helper](https://github.com/memsql/singlestore-auth-helper), enables browser-based passwordless login using the SingleStore and MySQL clients. Install the `singlestore-auth-helper` utility using Go. Perform the following steps: 1. [Download](https://go.dev/doc/install) Go installer version 1.18.2 or later. 2. Run the following command, which is tested with Go v1.18.2: ```shell go install github.com/memsql/singlestore-auth-helper@latest ``` > **📝 Note**: The `singlestore-auth-helper` is not required for browser-based SSO authentication with the SingleStore JDBC driver. The browser-based SSO authentication workflow opens the default browser to the SingleStore Helios login page. After authentication, a JWT is created for the user, which can then be used to access a SingleStore Helios cluster. ## Use JWTs with the SingleStore and MySQL Clients To use a JWT with the SingleStore client to connect to your SingleStore Helios cluster, replace the password with `singlestore-auth-helper`: ```shell singlestore -u '*' -h $HOSTNAME -P $PORT --password=`singlestore-auth-helper` --ssl=TRUE --enable-cleartext-plugin ``` Similarly, to use a JWT with the MySQL client to connect to your SingleStore Helios cluster: ```shell mysql -u '*' -h $HOSTNAME -P $PORT --password=`singlestore-auth-helper` --ssl=TRUE ``` ## Authenticate via the SingleStore JDBC Driver Using Browser-Based SSO > **📝 Note**: To authenticate your connection to the SingleStore Helios cluster via SingleStore JDBC driver using JWTs, the SingleStore user must connect via SSL and use JWT for authentication. See [Create a JWT User](https://docs.singlestore.com/#section-idm323307115459026.md) for more information. To authenticate your connection to a SingleStore Helios cluster with browser-based single sign-on (SSO), add the following parameter to the connection string: ``` credentialType=BROWSER_SSO ``` Using the `credentialType=BROWSER_SSO` parameter in the connection string tells the driver to use the `BrowserCredentialPlugin` class for authentication. This plugin queries the OS-native keyring (keystore) for the username and token to use. If this parameter is not specified, the connection uses password-based authentication. To authenticate using browser-based SSO on Linux distributions, you must have the [gnome-keyring-daemon](https://wiki.gnome.org/Projects/GnomeKeyring/RunningDaemon) and an active Login Password Keyring. If the Login Password Keyring is not configured, refer to [Install a Gnome Keyring Manager for Linux Distributions](https://docs.singlestore.com/#section-idm4585460943528033557225814158.md). Windows and MacOS have built-in OS-specific keyrings. > **📝 Note**: The keyring must have only one username-token pair for a specific user. If there is no token in the keyring, or the existing token has expired, the driver launches the default browser to initiate the authentication workflow. After a successful authentication, the driver stores the new/updated token in the keyring. The driver receives the following information from the JWT generated by SingleStore Helios: * `expiration`: Timestamp for when the token expires (in [RFC3339](https://datatracker.ietf.org/doc/html/rfc3339) format). * `token`: Base64-encoded JWT token, which is used as the password. * `username`: Username for the database. By default, it is the user’s authentication UUID. * `userID`: User ID represented as a UUID, used by the authentication helper. > **📝 Note**: Do not specify a user or password in the connection string. You must set the `connect_timeout` engine variable to a value that’s long enough to allow a user to successfully complete the browser-based login. For example, a value of `300` specifies that the user has 300 seconds to complete the login process. You can set `connect_timeout` value via the `SET GLOBAL` command. For example: ```sql SET GLOBAL connect_timeout = 300; ``` ## Authenticate via the SingleStore ODBC Driver Using Browser-Based SSO To authenticate your connection to the SingleStore Helios cluster via ODBC using browser-based SSO, add the `BROWSER_SSO=1` parameter to the connection string. When the `BROWSER_SSO` parameter is enabled, the driver launches the default browser to the Cloud Portal for the first connection request. After a successful authentication, a JWT is generated and sent from the Cloud Portal to the driver. The driver stores the token in the OS-specific keyring. To authenticate using browser-based SSO on Linux distributions, you must have the [gnome-keyring-daemon](https://wiki.gnome.org/Projects/GnomeKeyring/RunningDaemon) and an active Login Password Keyring. If the Login Password Keyring is not configured, refer to [Install a Gnome Keyring Manager for Linux Distributions](https://docs.singlestore.com/#section-idm4585460943528033557225814158.md). Windows and MacOS have built-in OS-specific keyrings. For subsequent logins, the access token is read from the OS-specific keyring. If the token in the keyring expires (or removed/lost), a new login is required. To skip the interaction with the keyring and login via the browser on each connection attempt, enable the `IGNORE_KEYRING_SSO` connection parameter. ## Install a Gnome Keyring Manager for Linux Distributions You can store and manage tokens in the Login Password Keyring on Linux distributions using a [Gnome Keyring](https://wiki.archlinux.org/title/GNOME/Keyring) manager, for example, Seahorse. Perform the following steps to install the `seahorse` package: 1. Run the following command on the terminal to install the seahorse package. **Debian** ```shell sudo apt-get install seahorse ``` **Red Hat** ```shell sudo yum install seahorse ``` 2. Once the installation is complete, open **Seahorse**. 3. Configure a Password Keyring. Select **+** (Add a new key or item) **> Password Keyring**. 4. Enter a name and password for the new keyring. 5. Set the new keyring as the default. ## Create a JWT User > **📝 Note**: Creating JWT users is only supported in SingleStore Helios clusters running SingleStore v7.8.3 and later. By default, each member of a SingleStore Helios organization that belongs to a cluster has a corresponding JWT user created for that cluster. This JWT user has a username based on the user’s Cloud Portal email address. You can also manually create a JWT user using the following command: ```sql CREATE USER 'email@example.com'@'%' IDENTIFIED WITH authentication_jwt REQUIRE SSL; ``` where: * `email@example.com` is the JWT user’s username. As of SingleStore version 8.1.5, you may also create a JWT user with a username that resembles an email address or a UUID. * The maximum length of a JWT user’s username is 320 characters. * `IDENTIFIED WITH authentication_jwt` sets the user’s authentication method to use a JWT. * `REQUIRE SSL` (mandatory) enforces the use of SSL for the JWT connection. A JWT user whose username conflicts with an organization user's email address is treated as an organization user. If the organization user is removed from the organization, the JWT user is also removed from the cluster. However, a JWT user that is not an organization user needs to be removed manually. ## Related Topics * [Authenticate via JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md) * [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md) * [The SingleStore ODBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md) *** Modified at: July 24, 2026 Source: [/cloud/security/database-access/authenticate-via-browser-based-sso-using-jwts/](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-browser-based-sso-using-jwts/) (An index of the documentation is available at /llms.txt) --- # Authenticate via JWT A JSON Web Token (JWT) (An open, industry standard typically used for authorization and information exchange.) is an open, industry standard ([RFC 7519](https://datatracker.ietf.org/doc/html/rfc7519)) that defines a compact and self-contained method for securely transmitting information between parties as a JSON object, which can be verified and trusted as it is digitally signed. JWTs are useful for both authorization (the most common scenario for using a JWTs) and information exchange (where information can be securely transmitted between parties). SingleStore Helios supports using JWTs for authentication. Refer to [JWT.io](https://jwt.io/) for more information. ## Use JWTs with the SingleStore and MySQL Clients To use a JWT with the SingleStore client, replace the password with the JWT: ```shell singlestore -u '*' -h $HOSTNAME -P $PORT --password=$JWT --ssl=TRUE --enable-cleartext-plugin ``` Similarly, to use a JWT with the MySQL client, replace the password with the JWT: ```shell mysql -u '*' -h $HOSTNAME -P $PORT --password=$JWT --ssl=TRUE ``` Specify the username as `'*'` in the `-u` (`--user`) option in the SingleStore/MySQL client command line to retrieve the username to use for connecting with SingleStore Helios from the JWT. Refer to [Validate JWTs with JWKS](https://docs.singlestore.com/#section-idm4592227523569633070889883433.md) for more information. > **📝 Note**: SingleStore also supports JWT-based authentication with [the SingleStore JDBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver.md), [the SingleStore ODBC driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-odbc/the-singlestore-odbc-driver.md), and other client libraries that support cleartext (provided cleartext is enabled). ## About JWT Users For the purposes of this document, a "JWT user" is a user that can authenticate with a JWT when connecting to a SingleStore Helios cluster. Refer to [Create a JWT User](https://docs.singlestore.com/#section-idm4582152339497633042662275189.md) for more information. For authenticating to SingleStore Helios with a JWT: * When using browser-based SSO in conjunction with the `singlestore-auth-helper` or SingleStore driver, the JWT user is identified by the user’s Cloud Portal email address, where the `sub` field of the JWT will contain the user’s email address as the username. Any user configured with a corresponding JWT user in the database can use this method to authenticate. The default expiration time for these tokens is sixteen (16) hours. Clusters accept these JWTs that are signed by the Cloud Portal. * When using the Cloud Portal SQL Editor against a running cluster, a JWT user is identified by a UUID. Only Cloud Portal users that are members of the organization which owns the cluster may access the database using this method. Clusters accept these JWTs that are signed by the Cloud Portal. * Customers can create and sign JWTs. For a customer-created JWT to be accepted by SingleStore Helios, the cluster must be configured to accept the public key corresponding to the private key used to sign the JWT. * Clusters are configured to accept JWTs using [JSON Web Key Sets (JWKS)](https://docs.singlestore.com/#section-idm4592226252089633070902698635.md). With JWKS, JWTs are matched to JSON Web Keys (JWKs) to validate the JWT. The JWK and JWT together specify the database username to use. ## Validate JWTs with JWKS You can use JSON Web Key Sets (JWKS) (A set of keys containing public keys that can be used to authenticate any JWT.) to validate the signature of a signed JWT. JWKS are a set of keys which contain public keys that can be used to authenticate any JWT. JWKS is a standard to download a batch of JWKs from a URL. JWTs are matched with JSON Web Keys (JWKs) for validation as follows: * If the JWT has a `kid` (Key ID) field, the JWKs with matching `kid` fields are validated. * If the JWT has a `kid` field that doesn’t match any JWK, the authentication request is rejected. * If the JWT has an `iss` (Issuer) field (instead of a `kid` field) that matches the `kid` in one or more JWKs, the JWKs with matching `kid` fields are validated. * If the JWT does not have a `kid` field and the `iss` field does not match the `kid` field in any JWK, then validation is attempted with all the JWKs with a matching `alg` (Algorithm) field. If the `alg` field is not specified, the `kty` (Key Type) field is used instead. The JWTs signature is validated as follows: * The JWT’s signature is validated against matching JWKs. If the JWT’s signature matches more than one JWK, validation is attempted against all matching JWKs. If the signature cannot be validated, the authentication request is rejected. * If the matching JWK includes an `aud` (Audience) field which does not match the `aud` field in the JWT, then the authentication request is rejected. The `aud` field can be a string or an array of strings. If any `aud` string of the JWT matches any `aud` string of the JWK, it is considered a match. **Note**: If the `jwks_require_audience` variable is set (non-empty), the value of this variable must be present in the `aud` field in the JWT. If the `aud` field is an array, the value of this variable must be one of the items in the array. * If the matching JWK does not define an audience (`aud`), audience checking is skipped. Note that `aud` is not a standard field in JWK. The database username is identified as follows: * When the `jwks_username_field` variable is set (non-empty), the value of this variable specifies the name of the JWT field that determines the database username. If this variable is set, all the other rules for identifying the database username are overridden/ignored. * If the matching JWK has a `usernameFrom` field, the value of this JWK field is used to identify the name of the field in the JWT that determines the database username. * If the JWK does not have a `usernameFrom` field, but the JWT has a `username` field, then the `username` field determines the database username. * If the JWK does not have a `usernameFrom` field and the JWT does not have a `username` field, then the `sub` field in the JWT determines the database username. > **📝 Note**: The database username in the JWT, once determined, must exactly match the database username used in the connection attempt. Otherwise, authentication is rejected. ## Configure JWKS To authenticate using JWKS, build a JWKS configuration file that contains your public keys and serve it from your Web server via static URL. SingleStore Helios supports the following public key signature algorithms: RS256, RS384, RS512, ES256, ES384, and ES512. Perform the following tasks to configure JWKS. 1. Configure the certificate authority used to validate the TLS transfer of the JWKS using SSL. This step should be performed only if you are using self-signed certificates. If you are not using self-signed certificates then start with step 2. 1. Upload a public key certificate corresponding to your TLS encryption certificate authority that secures the HTTP SSL transfers. Use the following command to obtain the certificate (assuming that your JWKS endpoint is **http\://some-domain/some/path**): ```shell openssl s_client --connect some-domain.com:443 --servername some-domain.com --showcerts < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > certificate-filename.pem ``` 1. On the [Cloud Portal](https://portal.singlestore.com/), select the cluster on which to enable TLS/SSL connection. 2. Under **Security**, select the **Upload Certificate** button to upload your security certificate. Once the certificate is uploaded, it is available to all the nodes in the cluster. This lets you secure outbound connections via TLS/SSL. 2. Specify the certificate file. ```sql SET GLOBAL jwks_ssl_ca_certificate='certificate-filename.pem' ``` 2. Configure the JWKS endpoint. ```sql SET GLOBAL jwks_endpoint='https://example.com/.well-known/jwks.json' ``` Updating the endpoint configuration triggers a JWKS update and validation. The `SET GLOBAL` command returns an error if: * The `GET` request fails. * There are no valid keys in the updated JWKS. A warning is generated if any JWKS is rejected, even if the JWKS has valid keys. Information on rejected keys is available in the audit log. To disable JWKS updates, set the endpoint to an empty string. ```sql SET GLOBAL jwks_endpoint='' ``` Optionally, you can set a refresh interval to update the JWKS configuration automatically (specify the value in seconds). ```sql SET GLOBAL jwks_update_interval = 3600 ``` If this value is not set, or set to `0`, the JWKS configuration updates only when the `SET GLOBAL jwks_endpoint` command is run. To manually refresh the JWKS configuration, run the `SET GLOBAL jwks_endpoint` command with its existing value. ## JWKS Update Status To check the status of the most recent JWKS update, use the `SHOW STATUS EXTENDED` command. It provides the following information: * `JWKS_update_status`: Specifies the status of the update. The status can be one of the following: `SUCCESS`, `FAILED`, or `DISABLED`. The `FAILED` status is followed by the reason behind the failure, in parenthesis. * `JWKS_update_time`: Specifies the timestamp of JWKS update. ```sql SHOW STATUS EXTENDED LIKE 'JWKS_update%'; ``` ```output +--------------------+-----------------------------------------------------------------------+ | Variable_name | Value | +--------------------+-----------------------------------------------------------------------+ | JWKS_update_status | FAILED (Failed to connect to localhost port 8082: Connection refused) | | JWKS_update_time | Mon May 2 14:03:08 2022 EST | +--------------------+-----------------------------------------------------------------------+ ``` ## Create a JWT User > **📝 Note**: Creating JWT users is only supported in SingleStore Helios clusters running SingleStore v7.8.3 and later. By default, each member of a SingleStore Helios organization that belongs to a cluster has a corresponding JWT user created for that cluster. This JWT user has a username based on the user’s Cloud Portal email address. You can also manually create a JWT user using the following command: ```sql CREATE USER 'email@example.com'@'%' IDENTIFIED WITH authentication_jwt REQUIRE SSL; ``` where: * `email@example.com` is the JWT user’s username. As of SingleStore version 8.1.5, you may also create a JWT user with a username that resembles an email address or a UUID. * The maximum length of a JWT user’s username is 320 characters. * `IDENTIFIED WITH authentication_jwt` sets the user’s authentication method to use a JWT. * `REQUIRE SSL` (mandatory) enforces the use of SSL for the JWT connection. A JWT user whose username conflicts with an organization user's email address is treated as an organization user. If the organization user is removed from the organization, the JWT user is also removed from the cluster. However, a JWT user that is not an organization user needs to be removed manually. ## Notes on JWT Authentication The Cloud Portal creates and deletes users in the cluster to ensure that every user in the organization has corresponding JWT users. SingleStore Helios creates two JWT users for each user in the organization: 1. User based on a UUID: The SQL Editor and other Cloud Portal functions use a UUID to identify a user. 2. User based on an email address: Browser-based SSO uses the email address to identify a user for access to SingleStore Helios clusters. SingleStore Helios will automatically delete any users it created when they are no longer members of the organization. However, if JWT users are manually created then they are not managed by the SingleStore Helios automatic sync process and will not be removed if they are no longer members of the organization. *** Modified at: July 24, 2026 Source: [/cloud/security/database-access/authenticate-via-jwt/](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt/) (An index of the documentation is available at /llms.txt) --- # Configuring a Password Policy In SingleStore Helios, password policies include rules for password expiration, password reuse, and password complexity. ## Password Expiration `password_expiration_seconds`: The time in seconds before a password expires. The default value is `0`, which indicates that passwords never expire. Configuring this variable is not recommended; contact [SingleStore Support](https://support.singlestore.com) for assistance. > **📝 Note**: When a user is created using `CREATE USER 'user'@'host'` or `CREATE USER 'user'@'host' IDENTIFIED BY '',` the account is assigned an empty password. If `password_min_length` is `0` (the default), an empty password is treated as a valid password. If `password_min_length` is greater than 0, an empty password does not satisfy the password policy and the statement is rejected. If `password_expiration_seconds` is set to a non‑zero value and an empty password is allowed by the policy, that password expires after the configured duration and the user is required to reset it. > **📝 Note**: A password expiration warning is raised to the user on every query, starting 14 days before the password expires. `password_expiration_mode`: Specifies what happens when a user's password expires. The default value is `NO_ACCESS`. If set to `NO_ACCESS`, the user cannot log in after the password expires. If set to `LIMITED_ACCESS`, the user can log in after the password expires but can only update their password using `ALTER USER` or `SET PASSWORD` commands. Until the password is updated, no other commands can be executed except `SET SESSION` or `SET LOCAL`. ## Password Reuse `password_history_count`: Restricts the reuse of previous user passwords. This variable is the number of previous passwords per user that SingleStore stores and disallows from reuse. Configuring this variable is not recommended, contact [SingleStore Support](https://support.singlestore.com) for assistance. SingleStore does not allow setting a user account’s password to one of the last `password_history_count` number of passwords for that user. The count includes the current password. For example, if set to `2`, setting a user’s password to its current password or the last password before the current password is disallowed. The default value is `0`, which indicates that any previous password can be reused. The maximum is `10`. ## Password Complexity The following variables control password complexity requirements. For each variable, the default value is `0` (disabled) and the accepted values are integers ranging from `0` to `100`. `password_min_length`: The minimum number of characters required. `password_min_uppercase_chars`: The minimum number of uppercase characters required. `password_min_lowercase_chars`: The minimum number of lowercase characters required. `password_min_numeric_chars`: The minimum number of numeric digit characters required. `password_min_special_chars`: The minimum number of special (non-alphanumeric) characters required. `password_max_consec_sequential_chars`: The maximum number of consecutive characters allowed. For example, if set to 3, passwords with a 4-letter sequence or longer (e.g “1234” or “abcd”) are disallowed. `password_max_consec_repeat_chars`: The maximum number of consecutive repeated characters allowed. For example, if set to 3, passwords with 4 or more consecutive repeated characters (e.g., “aaaa” or “1111”) are disallowed. When a user enters a new password, if the password does not meet the password complexity policy, the following error message is returned: `Error: password does not meet the requirements specified for in your password complexity policy. Password not changed.` ## Example Password Complexity Usage The following stored procedure (you can also use individual SET statements in the command line) creates a password complexity policy where passwords must: * be at least 12 characters long * include at least one uppercase character * include at least one lowercase character * include at least one numeric character * include at least one special character ```sql CREATE DATABASE db_security USE db_security DELIMITER // CREATE OR REPLACE PROCEDURE set_password_complexity_policy() AS BEGIN SET GLOBAL password_min_length=12; SET GLOBAL password_min_uppercase_chars=1; SET GLOBAL password_min_lowercase_chars=1; SET GLOBAL password_min_numeric_chars=1; SET GLOBAL password_min_special_chars=1; END // DELIMITER ; CALL set_password_complexity_policy(); ``` ## Password Complexity and Existing Passwords When changes are made to the password complexity policy, existing passwords are not checked. The policy only applies to new or changed passwords. *** Modified at: July 31, 2026 Source: [/cloud/security/database-access/configuring-a-password-policy/](https://docs.singlestore.com/cloud/security/database-access/configuring-a-password-policy/) (An index of the documentation is available at /llms.txt) --- # Configuring and Using Connection Links A connection link is a secure link that stores connection details (credentials and configurations) to supported data providers such as S3, Azure, GCS, HDFS, and Kafka. ## User Advantages 1. Referring to a connection link in a command is more secure than specifying the connection details directly in the command. Users need the `CREATE LINK` permission to create connection links, and only those users need to know the connection details thereby limiting the exposure of the details. 2. Commands such as `BACKUP`,`RESTORE`, `CREATE PIPELINE`, and `SELECT` support connection links. Users can run these commands without specifying the connection details. However, the user needs the `SHOW LINK` permission to use a connection link. ## Using a Connection Link Creation and use of a connection link is dependent on the [permissions granted](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) to a user. The credentials are currently stored internally in plain text and users require the following permissions to create and use connection links: * [`CREATE LINK`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md): A user with the `CREATE LINK` permission can create a connection link and only that user will know the connection details. * [`SHOW LINK`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-links.md): A user with the `SHOW LINK` permission can view and use all connection links that exist in a SingleStore Helios database. * [`DROP LINK`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-link.md): A user with the `DROP LINK` permission can remove a connection link that exists in a SingleStore Helios database. **Note**: The listed permissions can be cluster or database scoped. For example, you can grant the CREATE LINK permission on `*.*` (cluster scoped) and `database.*` (database scoped) but not `database.table`. See the [GRANT](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) topic for more details. ## Example The following example demonstrates the steps performed by a user `user1` to write all rows of the table `t1` of the database `productdb` to an S3 bucket using a connection link. Azure/GCS/HDFS/KAFKA links are created similarly via `CREATE LINK linkname AS {AZURE,GCS,HDFS,KAFKA} ...` -- see [CREATE LINK](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-link.md) for more information. 1. On user request, the DBA (who has the `CREATE LINK` permission) creates an S3 connection link `demouser_S3`: ```sql CREATE LINK productdb.demouser_S3 AS S3 CREDENTIALS '{"aws_access_key_id":"your_access_key_id","aws_secret_access_key":"your_secret_access_key"}' CONFIG '{"region":"us-east-1"}' DESCRIPTION 'Product list'; ``` 2. The DBA grants the `SHOW LINK` permission to `user1`. ```sql GRANT SHOW LINK ON productdb.* TO 'user1'; ``` This allows `user1` to use the S3 connection link `demouser_S3` and any other connection links defined in the `productdb` database. `user1` can run the `SHOW LINKS` command to view all the connection links in a database. For example, if a second connection link, `demouser2_S3` had been created in the `productdb` database, running `SHOW LINKS` would return the following results: ```sql SHOW LINKS ON productdb; ``` ```output +-------------------------+--------+-----------------------------+ | Link | Type | Description | +-------------------------+--------+-----------------------------+ | demouser_S3 | S3 | Product list | | demouser2_S3 | S3 | Brand list | +-------------------------+--------+-----------------------------+ ``` 3. `user1` runs the `SELECT .. INTO LINK` command to write the contents of the table `t1`, to the S3 bucket at the specified path, using the S3 connection link `demouser_S3` stored in the `productdb` database. ```sql USE productdb; SELECT * FROM t1 INTO LINK demouser_S3 'testing/output'; ``` *** Modified at: September 19, 2025 Source: [/cloud/security/database-access/configuring-and-using-connection-links/](https://docs.singlestore.com/cloud/security/database-access/configuring-and-using-connection-links/) (An index of the documentation is available at /llms.txt) --- # Disaster Recovery ## Overview SingleStore’s unique storage architecture replicates all cluster data to durable object storage using [Unlimited Storage](https://docs.singlestore.com/cloud/manage-data/database-storage/#section-idm457438403007043256122251662.md) for resilience and recoverability. SingleStore manages automatic full backups of your data. While automated backups can be used to recover from unexpected issues, SingleStore also enables you to [perform your own backups](https://docs.singlestore.com/cloud/manage-data/back-up-and-restore-data.md) as needed. Automated daily backups can be restored upon request, while performing your own backups allows you to self-serve restore operations. To restore a database backup, [create a support request](https://support.singlestore.com). While requests to restore a database do not have a fixed response time, these requests will be serviced as soon as possible. SingleStore provides a Enterprise offering which adds an additional recovery capability via [Point-in-Time Recovery](https://docs.singlestore.com/cloud/manage-data/database-storage/#section-idm4536710728857632830097298261.md). This feature allows a database to be restored to a previous point in time within the retention period. The backups retention period is 7 days. ## Data Retention Period Database backups are taken daily and retained for 7 days. ## Testing Disaster recovery and business continuity scenarios and approaches are tested annually with external third-parties to ensure adequacy and efficacy. *** Modified at: November 18, 2024 Source: [/cloud/security/disaster-recovery/](https://docs.singlestore.com/cloud/security/disaster-recovery/) (An index of the documentation is available at /llms.txt) --- # Encryption ## Overview This document covers SingleStore Helios data encryption in transit and at rest. ## Encryption in Transit To ensure a secure connection to SingleStore Helios, SQL clients must be properly configured to require a secure connection, and to verify the supplied server certificate. When a SingleStore Helios cluster has `REQUIRE SSL` enabled, users cannot connect to the cluster without using SSL. However, security can still be compromised with or without the use of SSL. Not using SSL can lead to a man-in-the-middle attack, where a would-be attacker can impersonate a server. Conversely, a secure connection can be established by using SSL, but perhaps to a server that’s using an illegitimate certificate. To circumvent these potential issues, SingleStore supports TLS 1.2 for data in transit and for all connections to the database. Transport Layer Security (TLS) uses a combination of symmetric and asymmetric encryption which employs a pair of keys: a public key and a private key. The SSL/TLS cipher suite used is AES128-GCM-SHA256, with SSL certificates on a one-year rotation for `svc.singlestore.com` and on a two-year rotation for the legacy `db.memsql.com`. The use of [Let’s Encrypt](https://letsencrypt.org/), which will have a 90-day certificate rotation, is planned for the future. Refer to [Connect to SingleStore Helios using TLS/SSL](https://docs.singlestore.com/cloud/connect-to-singlestore/connect-with-mysql/connect-with-mysql-client/connect-to-singlestore-helios-using-tls-ssl.md) for additional information. ## Encryption at Rest For data at rest, SingleStore uses best-practice AES-256 encryption with AWS, Azure, and GCP cloud-hosting partners. With a 256-bit key length, it is currently the strongest encryption algorithm available. In the [Standard edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md) of SingleStore Helios, the cloud provider managed key is used to encrypt all data at rest. For the [Enterprise edition](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-singlestore-helios/singlestore-helios-editions.md) of SingleStore Helios, a customer may use their own key, stored in their own key vault in the cloud key management service (KMS) to add an additional layer of security. Key access and use is captured using [AWS CloudTrail](https://aws.amazon.com/cloudtrail/). SingleStore cannot access the shared key material directly. For the Data Plane, SingleStore logs all access to each SingleStore Helios cluster, and runs each cluster with audit logging enabled. The [ADMIN-ONLY-INCLUDING-PARSE-FAILS ](https://docs.singlestore.com/cloud/security/audit-logging/data-plane-audit-logs.md) audit logging level is used for completeness. Audit logs can be streamed to third-party audit tools. Refer to [Audit Logging](https://docs.singlestore.com/cloud/security/audit-logging.md) for more information. ## In this section * [Customer Managed Encryption Keys](https://docs.singlestore.com/cloud/security/encryption/customer-managed-encryption-keys.md) * [Troubleshoot OpenSSL 3.0 Connections](https://docs.singlestore.com/cloud/security/encryption/troubleshoot-openssl-3-0-connections.md) *** Modified at: August 25, 2025 Source: [/cloud/security/encryption/](https://docs.singlestore.com/cloud/security/encryption/) (An index of the documentation is available at /llms.txt) --- # Customer Managed Encryption Keys ## Overview “Encryption” encompasses the processes and controls used to ensure that data remains inaccessible to unauthorized users and to protect the data transferred between users, client applications, and the computers/services involved. In accordance with best practices, SingleStore applies encryption to data both in transit and data at rest. Refer to [Encryption](https://docs.singlestore.com/cloud/security/encryption.md) for more information. ## Customer Managed Keys Encryption keys are created and maintained with a cloud key management service (KMS) using key material owned by the cloud service provider. SingleStore does not have access to these encryption keys, nor can it manage them. SingleStore uses these keys solely for encryption and decryption operations. A SingleStore Helios customer can use their own key stored in their cloud key vault to encrypt data at rest. This provides the customer with control over their own data, including the ability to grant and revoke permission to it using this key, and implement a key rotation schedule based on corporate policy. In the Enterprise edition, encryption of data at rest using customer managed encryption can use separate keys for encryption of the data and the backup bucket. ## Benefits of using Customer Managed Keys The benefits of using of customer managed keys for data encryption at rest include: * A customer can maintain the proper set of permissions for SingleStore to access the KMS keys. * Once an existing key is rotated, the new key will be used for data protection. A customer can also set the automatic key rotation in their key vault (such as with AWS KMS), which will rotate the key every year. This allows a customer to own the data lifecycle and protect their data based on business priorities. * A customer can control/restrict access to their data for SingleStore and the cloud service provider. In the case of a data breach, a customer can easily revoke access to their data. This approach allows Zero Trust to be achieved with SingleStore. Once key access is revoked, SingleStore will no longer have access to a customer’s data. To resume normal cluster operations, the customer must explicitly grant SingleStore access to their data. This section provides instructions on how to use customer managed keys with AWS, with instructions for Azure and GCP to be provided at a later date. ## In this section * [Cloud Key Management Service for AWS Volume Encryption](https://docs.singlestore.com/cloud/security/encryption/customer-managed-encryption-keys/cloud-key-management-service-for-aws-volume-encryption.md) * [Customer Managed Encryption Keys with Azure Key Management Service](https://docs.singlestore.com/cloud/security/encryption/customer-managed-encryption-keys/customer-managed-encryption-keys-with-azure-key-management-service.md) * [Customer Managed Encryption Keys with GCP Key Management Service](https://docs.singlestore.com/cloud/security/encryption/customer-managed-encryption-keys/customer-managed-encryption-keys-with-gcp-key-management-service.md) *** Modified at: May 19, 2025 Source: [/cloud/security/encryption/customer-managed-encryption-keys/](https://docs.singlestore.com/cloud/security/encryption/customer-managed-encryption-keys/) (An index of the documentation is available at /llms.txt) --- # Troubleshoot OpenSSL 3.0 Connections ## Overview As of SingleStore Helios 8.1, OpenSSL 3.0 is now used to establish secure connections to SingleStore Helios. As a consequence, a client certificate that uses SHA or MD5 hash functions in its signature must be replaced with a certificate that uses SHA256 at a minimum, or a secure connection to SingleStore Helios cannot be established. ## Connect After Upgrading As this file is always kept up to date, you may download the [singlestore\_bundle.pem](https://portal.singlestore.com/static/ca/singlestore_bundle.pem) certificate and use it to connect to SingleStore Helios 8.1. > **❗ Important**: If your SQL client connects using the `singlestore_bundle.pem` file with the `--ssl-mode=VERIFY_CA` flag and is unable to establish a connection to SingleStore Helios, download and use the latest `singlestore_bundle.pem` file. *** Modified at: March 16, 2026 Source: [/cloud/security/encryption/troubleshoot-openssl-3-0-connections/](https://docs.singlestore.com/cloud/security/encryption/troubleshoot-openssl-3-0-connections/) (An index of the documentation is available at /llms.txt) --- # Firewall Use IP Address Allowlisting to ensure only a specified set of IP addresses can communicate with your SingleStore Helios cluster. You can add IP addresses to the firewall allowlist within the cluster: 1. Log in to the [Cloud Portal](https://portal.singlestore.com/), and select **Clusters**. 2. Select the three dots for your cluster, and then select **Access & Security** from the list. 3. Scroll to the **Firewall** section. This section shows the IP addresses that have been allowlisted for inbound and outbound connections. 4. To add additional IP addresses, select **Edit**, and enter the address or range of addresses in the **Edit IP Address Access** dialog. 5. Select **Save**. The cluster will now allow access from the IP address(es) you added. *** Modified at: May 11, 2026 Source: [/cloud/security/firewall/](https://docs.singlestore.com/cloud/security/firewall/) (An index of the documentation is available at /llms.txt) --- # Portal Access ## In this section * [Multi-Factor Authentication](https://docs.singlestore.com/cloud/security/portal-access/multi-factor-authentication.md) * [SingleStore’s Identity Platform](https://docs.singlestore.com/cloud/security/portal-access/singlestores-identity-platform.md) * [The Overall Flow of the SSO Setup](https://docs.singlestore.com/cloud/security/portal-access/the-overall-flow-of-the-sso-setup.md) * [Identity Provider Connections](https://docs.singlestore.com/cloud/security/portal-access/identity-provider-connections.md) * [SCIM User Provisioning](https://docs.singlestore.com/cloud/security/portal-access/scim-user-provisioning.md) * [Protocols](https://docs.singlestore.com/cloud/security/portal-access/protocols.md) * [OIDC](https://docs.singlestore.com/cloud/security/portal-access/oidc.md) * [SAML](https://docs.singlestore.com/cloud/security/portal-access/saml.md) * [Renew/Rotate SAML Certificate for SSO](https://docs.singlestore.com/cloud/security/portal-access/renew-rotate-saml-certificate-for-sso.md) * [Troubleshooting SSO Connections](https://docs.singlestore.com/cloud/security/portal-access/troubleshooting-sso-connections.md) *** Modified at: June 12, 2025 Source: [/cloud/security/portal-access/](https://docs.singlestore.com/cloud/security/portal-access/) (An index of the documentation is available at /llms.txt) --- # Identity Provider Connections An Identity Provider Connection represents a connection between SingleStore Helios’s identity system and your IdP. An organization may have zero or more IdP Connections. To add an IdP, select **(ORG:) > Organization Details > Authentication**. Connections can be active or inactive, and they can be modified, copied, and deleted. Each connection has an identifier and its own URL paths that need to be configured with the customer’s Identity Provider to make the connection work. The process of establishing a connection involves exchanging information between the SingleStore Helios’s identity system and your IdP. Both OIDC 1.0 and SAML 2.0 connections are supported. ## Domains Each IDP connection is also configured with one or more domain names. The domain names control which authentication requests will be routed to the IdP connection. A domain can either be live or not live, on a per-IdP connection basis. Setting a domain live in one connection will make it *not* live in all other connections. A domain can either be verified or not verified. Verification is the process of proving that you (the customer) own (or at least control) the domain. To verify a domain, you must perform either of the following: * Create a DNS TXT record with the token as specified in the UI. * Create a web page with the token as specified in the UI. Once a domain is verified, it remains verified. To semi-automatically verify the sub-domains of a verified domain, select `Verify`. An IdP connection cannot be activated without having at least one live and verified domain. ## Multiple Organizations If you have multiple SingleStore organizations that include users with the same email domain, set up SSO with only one of the organizations. It is recommended that all users are allowed to log into the [Cloud Portal](https://portal.singlestore.com/) in the IdP settings. SSO provides only authentication. Authorization is handled separately by permissions via [group memberships](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations.md). When setting up SSO, ensure that each email domain maps to a single SSO configuration. This mapping enables the SingleStore authentication server to redirect users to the appropriate IdP for authentication. ## Settings One of the per-domain settings controls whether a non-SSO login is permitted when using an email address that matches the domain. If the **Require SSO** setting is enabled, then one cannot log into SingleStore Helios via the Cloud Portal with an email address that matches the domain. A list of email addresses of the form `username` or `username@domain` can bypass the per-domain **Require SSO** setting and log in through Keycloak. ![A dialog box appears when an identity provider is added, connection settings can be configured.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt1fa9e17f26c24516/6a7adae53ce8e20ad4cea117/IdP_Connection_Settings-XZ0Iiz.png) *** Modified at: November 26, 2024 Source: [/cloud/security/portal-access/identity-provider-connections/](https://docs.singlestore.com/cloud/security/portal-access/identity-provider-connections/) (An index of the documentation is available at /llms.txt) --- # Multi-Factor Authentication ## Overview SingleStore provides a variety of authentication methods including username/password, [JWT](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-jwt.md), [SAML](https://docs.singlestore.com/cloud/security/portal-access/saml.md), and [OIDC](https://docs.singlestore.com/cloud/security/portal-access/oidc.md). SingleStore also supports multi-factor authentication (MFA) which enhances login security when connecting to SingleStore Helios. The MFA solution is available only to non-SSO users and SSO users who are exempt from the SSO requirement when logging in through the IDP. While customers using single sign-on (SSO) with external authentication tools can enable MFA on their identity providers, SingleStore offers a default MFA solution, through a combination of either the FreeOTP or the Google Authenticator app, which is managed entirely by SingleStore. MFA is enabled on a per-user basis. Users can simply install either the FreeOTP or the Google Authenticator app on their mobile device (iOS, Android, Windows, etc.) and configure it for use with the SingleStore ## SingleStore Helios Multi-factor Authentication SingleStore Helios MFA is enforced for all users except a predefined set of exemptions. For MFA, email is set as the default authentication method. ## MFA Exemptions The following users are exempt from MFA: * Users logging in via Single Sign-On (SSO). * Users who already have MFA enabled in Keycloak. ## Keycloak MFA Keycloak-based MFA is being deprecated and will be removed in the near future. Existing users configured with Keycloak MFA should migrate to Helios-native MFA as described below to avoid any authentication disruption. When you remove the two factor authenticator configuration from your Keycloak account console by navigating to **User settings **→ **Manage Account **and select **Remove Two factor Authentication**, MFA will automatically switch to Helios MFA. ## Changing Your MFA Method 1. Sign in to the Cloud Portal and complete the current (default email) MFA verification process. 2. Navigate to <**your\_account**> → **User Settings** → **Multi-Factor Authentication**. 3. By default, **email authentication** will be displayed as the active method. 4. To switch to **Authenticator App** (TOTP): * Select **Use this method** under **Authenticator App** (TOTP). * Follow the on-screen instructions to configure TOTP as your new MFA method. **Note:** To switch back from TOTP to email, follow the same process. However, SingleStore strongly recommends using TOTP for enhanced security. ## Reconfiguring TOTP If your MFA method is set to TOTP, you can reconfigure it at any time by going to <**your\_account**> → **User Settings** → **Multi-Factor Authentication** and select the **Reconfigure** option. If you cannot access your TOTP device, for example: you have lost your mobile, you can complete MFA by choosing to verify using email for that particular session. (There is an option displayed on the MFA screen when the you use TOTP MFA). If you are unable to access neither email nor the TOTP device, then you have to raise a request with the Support team citing the reason and requesting MFA exemption for the affected user(s). After MFA exemption and subsequent successful login, you have to go to the **User Settings** page and either reconfigure TOTP on your device or configure **Email** MFA for future logins. ## Remember My Device On log in, you have the option to **remember your device** for MFA. If you select this option, you can choose from a predefined set of durations visible on the MFA screen. During the selected period, you will not be prompted for MFA when logging in from that device. ## Enforcing SingleStore Helios MFA for SSO Users By default, SSO users are exempt from SingleStore Helios MFA. However, if you want to include SingleStore Helios MFA in addition to your identity provider’s MFA then execute the following steps: 1. Go to <**your\_account**> → **Organization Details** → **Authentication**. 2. Select your **Identity Provider** (IdP). 3. Click **Update Connection**. 4. Enable the **Enforce MFA** toggle. 5. Click **Save**. *** Modified at: May 6, 2026 Source: [/cloud/security/portal-access/multi-factor-authentication/](https://docs.singlestore.com/cloud/security/portal-access/multi-factor-authentication/) (An index of the documentation is available at /llms.txt) --- # OIDC The authentication flow for OpenID Connect involves the following steps: 1. The auth client (SingleStore) creates a request and redirects to the server’s authentication endpoint. 2. The server also creates a request and redirects (or displays) a login page. 3. When login is complete, the server redirects back to the client providing an authentication code. 4. The client then makes a direct request (not the browser) to the server to exchange the authentication code for tokens (refresh, access, and ID). 5. The client stores the tokens and uses the id token to ask for user information (name, email) from the server. Access ID tokens may have short expiry times. When they expire, the token endpoint will be used to get new tokens using the refresh token. The id token is always a signed JWT (JSON Web Token) with an expiration time. The access token may be a JWT. The Refresh token is typically not a JWT but it could be one too. The Refresh token should be persisted server-side so that it can be invalidated. The access and refresh tokens may also be persisted server-side for invalidation purposes, but that is generally not required because they generally expire quickly. The configuration items available from the SingleStore Helios Portal are: * **Login Initiation URL**: This is used for an IdP-initiated login. When using this URL, the SingleStore login screen is bypassed but otherwise, this is a normal authentication flow. Login Redirect URLs: This is the URL that the IdP may redirect to when it has verified the login. * **Logout Redirect URLs**: This is where to redirect to after logout to return to a SingleStore login screen. Most of the configuration comes from the IdP. This includes: * Issuer – this is a URL that is used to identify the connection. The discovery endpoint is usually, Issuer + "/.well-known/openid-configuration”. It should be a working URL that provides an OpenID configuration block. * Client ID * Client Secret * PKCE is usually supported and it should be enabled. * The set of required scopes for the OIDC client to request. This will always include 'openid' and will usually include 'email', and 'profile'. Other scopes may also be required. The set of supported scopes can be found inside the discovery endpoint. Do not include any scope that is not supported by your IdP. A `global logout` setting, if true implies that, when a user logs out of a SingleStore site (portal, etc) it will try to log them out of their IdP as well. In the SingleStore Helios Portal this is specified as `Attempt Global Logout`. ## In this section * [Microsoft Entra ID Self Serve SSO Steps - OIDC](https://docs.singlestore.com/cloud/security/portal-access/oidc/microsoft-entra-id-self-serve-sso-steps-oidc.md) * [Jumpcloud Self Serve SSO Steps - OIDC](https://docs.singlestore.com/cloud/security/portal-access/oidc/jumpcloud-self-serve-sso-steps-oidc.md) * [Okta Self Serve SSO Steps - OIDC](https://docs.singlestore.com/cloud/security/portal-access/oidc/okta-self-serve-sso-steps-oidc.md) *** Modified at: January 10, 2024 Source: [/cloud/security/portal-access/oidc/](https://docs.singlestore.com/cloud/security/portal-access/oidc/) (An index of the documentation is available at /llms.txt) --- # Protocols ##### OIDC 1.0 vs SAML 2.0 OpenID Connect (OIDC) and SAML are quite different in how they operate. Both are supported. Each has advantages. If only one is supported by your IdP, then use that one. With SAML, there is no direct connection between the SingleStore Helios identity system and the IdP. All communication takes place using browser redirects. If the IdP is behind a firewall and inaccessible to SingleStore Helios’s identity platform, then SAML is the only choice. SAML configuration is done by exchanging XML configuration blobs and configuring, on both sides, the attributes used for first name, last name, and email address. With SAML, there is fine-grain control over what information is sent from the IdP to the Service Provider (SP). This allows sending group membership and any other arbitrary data that is desired. At the current time, SingleStore Helios does not use anything besides first name, last name, and email address. OIDC is generally easier to set up, but the instructions for any given IdP platform (Okta, Ping, etc.) are not obvious as the set of scopes is not well standardized. SingleStore Helios’s identity system supports IdP-initiated authentication with OIDC. SingleStore Helios Portal authentication sessions will last only as long as the access token granted by the customer IdP lasts or can be refreshed. This allows the IdP to force all sessions to close by invalidating the refresh token which may happen, for example, because an administrator removed the corresponding user. With the refresh token invalidated, the access token will expire and the session will close. SingleStore Helios does not limit the session length – that is up to the customer IdP’s policies. Currently, only IdP-initiated login with SAML is supported; IdP-initiated logout with SAML is not supported. ## Settings Available The following settings are available in: ## General (not specific to OIDC or SAML) * JWT Token lifetime for engine access - This is for tokens generated via a browser login for accessing the SingleStore Helios database. * A list of email addresses of te format, username\@domain or just username, that can bypass the per-domain SSO required setting and log in through the keycloak. ## OIDC * Allowed Clock drift – useful when the IdP and SingleStore disagree about the current time. ## SAML * Portal access (minutes) – determine how long should the tokens generated for the SingleStore Helios Portal last. You will have to re-authenticate when this runs out so a value like 1440 minutes (one day) is reasonable. *** Modified at: August 11, 2025 Source: [/cloud/security/portal-access/protocols/](https://docs.singlestore.com/cloud/security/portal-access/protocols/) (An index of the documentation is available at /llms.txt) --- # Renew/Rotate SAML Certificate for SSO Your SSO configuration should be either self-service (you did it yourself using the Portal) or it was set up by filing a support ticket and exchanging configuration blocks with SingleStore , who did the SingleStore-side setup. If your configuration is not self-service and you have an expiring certificate, you must switch to self-service. SingleStore will not update non-self-service configurations. For additional details refer: [SingleStore’s Identity Platform](https://docs.singlestore.com/cloud/security/portal-access/singlestores-identity-platform.md) The steps outlined below assume you are already self-service. SAML certificate rotation for SingleStore Helios Portal access, especially where Single Sign-On (SSO) is configured via self-service (example, Okta), involves re-establishing the SAML connection with a new certificate. The process is essentially equivalent to configuring SSO using updated certificate material. Certificate rotation is performed by redoing the SSO configuration, not with a separate rotation-specific workflow. The following steps outline the SAML certificate rotation: 1. **Prepare for Rotation** * Identify the expiration date of your existing SAML signing certificate. * Notify stakeholders about the planned rotation to minimize disruption. * All the following steps are just a summary of the [regular SAML instructions](https://docs.singlestore.com/cloud/security/#saml-authentication.md). You also have the option of switching to OIDC which is recommended in cases where your Identity Provider is not behind a firewall. 2. **Access your SSO Provider and the SingleStore Helios Portal** * Log into your SingleStore Helios Portal. * Log into your SAML identity provider’s admin portal (for example, Okta). 3. **Add (or Update) the Identity Provider Connection** * In the SingleStore Helios Portal, navigate to **Organization Details** > **Authentication** tab. * Use the **Add Identity Provider** list to start a new connection or edit the existing one, as appropriate. * Assign a connection name (for example, Okta SAML). 4. **Copy the Service Provider Metadata** * Download/copy SingleStore’s Service Provider Configuration (Login/Logout URLs, Entity ID) for use in your IdP. 5. **Create/Configure the SAML Application in Your IdP** * In your IdP (for example, Okta), create or update the app integration: * Input SingleStore’s URLs and Entity ID. * Set an appropriate NameID format (for example, Persistent). * Configure required attribute statements (`email`, `lastName`, `firstName`). 6. **Generate or Upload a New SAML Signing Certificate in Your IdP** * In your IdP, generate a new SAML signing certificate or upload a renewed one. * Download the IdP metadata XML (updated with the new certificate). 7. **Upload IdP Metadata to SingleStore** * In the SingleStore Helios Portal, upload the new IdP metadata XML under the SAML connection. 8. **Map User Attributes and Set Domains** * Map the user attributes in SingleStore to correspond with the IdP. * Add/verify required domains. 9. **Update IdP with SingleStore’s New Certificate (If Required)** * Optionally, if SingleStore's SP signing certificate has changed, upload the `.pem` file to your IdP and enable **Validate SAML requests with signature certificates**. 10. **Finalize and Test the Configuration** * Save and update the SSO connection in SingleStore. * Test the login workflow to ensure the new certificate is used and authentication succeeds. **This is the most important step.** * After testing succeeds, enable/activate the connection. 11. **Decommission the Old Configuration** * After validation, remove any deprecated or obsolete SAML settings. * If migrating from legacy SSO (for example, "old-style Keycloak"), ensure the previous IdP is disabled to avoid confusion. ## Remarks * If you are migrating from legacy/manual (non-self-service) SSO, you should reconfigure using the self-serve workflow rather than asking to update your existing connection. After you make the new connection live, file a [support](https://support.singlestore.com/) ticket to have the old connection disabled. * The steps for other IdPs (Azure AD, JumpCloud) are similar: create/update SAML app, upload new certificate, update IdP XML, and verify authentication. * Refer to the latest [Okta self-serve SSO steps](https://docs.singlestore.com/cloud/security/portal-access/scim-user-provisioning/idp-configuration-okta.md) and adapt based on your identity provider’s specifics. *** Modified at: June 12, 2025 Source: [/cloud/security/portal-access/renew-rotate-saml-certificate-for-sso/](https://docs.singlestore.com/cloud/security/portal-access/renew-rotate-saml-certificate-for-sso/) (An index of the documentation is available at /llms.txt) --- # SAML Most Identity Providers can export an XML configuration block or file. Many can also import an XML configuration block or file. After choosing to create a SAML Identity Provider Connection in the SingleStore Helios Portal, the XML configuration block for that connection is available for immediate download. This can be downloaded before doing any setup with the Identity Provider. Each Identity Provider Connection will have a different configuration block and you cannot use a block meant for one connection with another. Attribute mappings must be set up on both sides of the connection. On the SingleStore side, the name of the attributes used for `email`, `firstName`, and `lastName` must be set. On the IdP side, these attributes must be exported. Generally, to discover the names actually associated with the attributes, most IdPs have a UI function to preview what their assertion statement looks like. The names of the attributes can be found there. The basic flow of SP-initiated authentication via SAML is a single request and response made with browser redirects. The request that the SP (SingleStore identity platform) makes will be signed. The IdP (customer) may validate the signature or ignore it. The response from the IdP must be signed and the SP (SingleStore identity platform) will validate the signature. Both the IdP and SP are configured with the entity IDs of their counterparts and also URLs. The SAML-specific configuration items that must be filled in the SingleStore Helios Portal to enable SAML are: * IdP XML configuration block generated by the IdP * Name of the email attribute in the IdP-provided assertions * Name of the first name attribute in the IdP-provided assertions * Name of the last name attribute in the IdP-provided assertions The SAML-specific configuration items that are provided by the SingleStore Helios Portal include: * SP XML Configuration block that can be used to configure the IdP * The attribute consuming service (ACS) endpoint * The SP Entity ID These configuration items are available as soon as the Identity Provider Connection is created. No parameters from the IdP are required. ## In this section * [Microsoft Entra ID Self Serve SSO Steps - SAML](https://docs.singlestore.com/cloud/security/portal-access/saml/microsoft-entra-id-self-serve-sso-steps-saml.md) * [Jumpcloud Self Serve SSO Steps - SAML](https://docs.singlestore.com/cloud/security/portal-access/saml/jumpcloud-self-serve-sso-steps-saml.md) * [Okta Self Serve SSO Steps - SAML](https://docs.singlestore.com/cloud/security/portal-access/saml/okta-self-serve-sso-steps-saml.md) *** Modified at: November 9, 2023 Source: [/cloud/security/portal-access/saml/](https://docs.singlestore.com/cloud/security/portal-access/saml/) (An index of the documentation is available at /llms.txt) --- # SCIM User Provisioning SingleStore for Cross Identity Management (SCIM) enables user provisioning from an identity provider to the [Cloud Portal](https://portal.singlestore.com/) and SingleStore database. When SCIM is configured, changes in the identity provider synchronize automatically with the Cloud Portal. For example, assigning a user in the identity provider application automatically adds that user to the organization. SingleStore SCIM supports SCIM 2.0 protocol. When users are provisioned through SCIM from an identity provider, their default access depends on whether RBAC is enabled: * If RBAC is disabled, the system assigns **Owner** access by default, similar to inviting new users manually. * If RBAC is enabled, the system grants the user basic permissions based on the role defined for new users.This provides only limited access unless additional roles or permissions are explicitly assigned. RBAC is required for managing user permissions both in the SingleStore Helios and the SingleStore database engine. The following table shows the identity provider application assignments and the corresponding actions in SingleStore Helios: | Identity provider application assignment | SingleStore Helioscorresponding action | | ---------------------------------------- | -------------------------------------- | | Add user | Add a user to aSingleStoreorganization | | Remove user | Remove the user from the organization | | Add group | Create a new team | | Remove group | Remove the team | | Add a user to group | Add the user to team | When SCIM is configured with RBAC enabled, adding a group in the identity provider automatically creates a corresponding team in the Cloud Portal. The permissions configured for the team are then automatically granted to users added to the team. > **📝 Note**: For optimal performance, SCIM provisioning supports synchronizing up to 1000 users and 500 groups. ## Create SCIM Configuration Perform the following tasks to create a new SCIM configuration: 1. On the Cloud Portal, select **\ > Organization Details > SCIM**. 2. Select **+ New SCIM Configuration**. 3. In the **New SCIM Configuration** Dialog box, enter a **Description**, and select **Generate Secret Token** 4. Copy the generated Secret Token and secure it. The secret token is displayed only **once**. 5. Select **Save Configuration**. 6. Use the endpoint URL (`https://authsvc.singlestore.com/auth/scim/[id]`) and the generated bearer token to configure SCIM in the identity provider. Creating a new SCIM configuration deactivates the existing configuration. Only one SCIM configuration can be active at a time. ## Configure an existing SCIM To generate a new secret token for an existing SCIM configuration, select **Configure SCIM**. Select **Generate Secret Token > Save Configuration**. Generating a new secret invalidates the existing secret token. ## Deactivate an existing SCIM To deactivate an existing SCIM configuration, select the ellipsis (three dots) next to the displayed SCIM, and then select **Deactivate** from the list. Deactivating an existing SCIM configuration does not remove users; it only prevents the synchronization of users and teams from the identity provider. ## Activate an existing SCIM To activate an existing SCIM configuration,  select the ellipsis (three dots) next to the displayed SCIM, and then select **Activate** from the list. Activating an existing SCIM configuration deactivates other SCIM connections and prevents synchronization of users and teams from the identity provider. It does not remove users. ## Remove an existing SCIM To remove an existing SCIM configuration, select the ellipsis (three dots) next to the displayed SCIM, and then select **Remove SCIM** from the list. Once the SCIM configuration is removed, the associated users and teams are deleted, and the action cannot be undone. ## In this section * [IdP Configuration - Azure](https://docs.singlestore.com/cloud/security/portal-access/scim-user-provisioning/idp-configuration-azure.md) * [IdP Configuration - Okta](https://docs.singlestore.com/cloud/security/portal-access/scim-user-provisioning/idp-configuration-okta.md) *** Modified at: April 1, 2026 Source: [/cloud/security/portal-access/scim-user-provisioning/](https://docs.singlestore.com/cloud/security/portal-access/scim-user-provisioning/) (An index of the documentation is available at /llms.txt) --- # SingleStore’s Identity Platform > **📝 Note**: Currently, this is a public preview feature. SingleStore Helios’s identity platform is an authentication proxy. When using the SingleStore Helios Portal or any other SingleStore Helios site, users log in to the SingleStore Helios identity platform. The Portal is an authentication client to the identity platform. It is an OpenID Connect (OIDC) client. The identity platform is actually an identity proxy: while the Portal uses OIDC to authenticate, the login process is to authenticate with an external Identity Provider (IdP) using either OpenID Connect (OIDC) or Security Assertion Markup Language (SAML). The flow is: 1. The Portal redirects to the authentication endpoint. 2. This creates a request and redirects to a login page. 3. On the login page, after you enter your email address, if SSO is required for your email domain, you will be redirected to your IdP. If SSO is allowed but not required, you can choose to log in with SSO.. 4. When that login process completes, an authentication code is generated and given to the Portal. 5. The Portal exchanges that code for access and refresh tokens. When the access token expires (within five minutes or less) the Portal asks for a fresh token. If the proxied connection is OIDC, the SingleStore identity platform checks the expiration time of the access token it holds for the external IdP and if it has or is about to expire then it refreshes that access token before refreshing the Portal’s access token. For IdP-initiated login (OIDC only), the login page step is skipped. > **📝 Note**: Logging in with SSO does not currently grant membership in your SingleStore Helios organization. It is just authentication. Organization membership/RBAC based on IdP group membership is not supported. Users must still be added to your organization with invitations. ## When SSO with SingleStore Helios is Already Available If you already have an SSO connection set up because you followed the old instructions and opened a ticket, that SSO connection will continue to work at least for a while. You should migrate to the new self-service SSO setup. To migrate, do a new SSO setup. Test it without putting it live. Once you have it tested, then put it live and open a support ticket to take down the old SSO connection. It’s okay for there to be some overlap between the old connection and the new connection. IdP-initiated login (OIDC only) will always use the new self-service SSO connection. *** Modified at: March 7, 2024 Source: [/cloud/security/portal-access/singlestores-identity-platform/](https://docs.singlestore.com/cloud/security/portal-access/singlestores-identity-platform/) (An index of the documentation is available at /llms.txt) --- # The Overall Flow of the SSO Setup 1. In the SingleStore Helios Portal, open the **ORG:your-org** menu at the top and go to **Organization Details**. 2. Under the **Authentication** tab, use the **Add Identity Provider** menu to create an OIDC or SAML connection. 3. Name the connection. 4. Save it. 5. To continue editing the connection, select the ellipsis (three dots) under the **Actions** column, and select **Update Connection**. 6. Navigate in your IdP UI to where you can add a client/SP. 7. Add an authentication client (OIDC) or SP (SAML) to your IdP. 8. Exchange data back and forth between SingleStore and your IdP. (See specific instructions for various providers). 9. Configure bits that are not exchanged such as scopes, attributes, etc. (See specific instructions for various providers). 10. Select **Save** in the SingleStore Helios Portal to save your work. 11. Try the new connection with test logins: select the ellipsis (three dots) in the **Actions** column, and select **Perform Test Login**. This will test the ability to log in without actually logging you in. 12. If the test logins work, you get an error page that says login would have worked only if the problem (domain live and verified, connection activated, started from the Portal) were solved. 13. If the test logins do not work, debug the problem. You may have an error message that provides a clue or if not, you can go to **Debug Logins** in the **Actions** column to see how far the login attempt made it and what errors the SingleStore identity platform saw. 14. Add a domain. 15. Set the domain live. 16. Save. 17. Verify the domain by either creating a web page with the provided token or creating a DNS record with the provided token. Select **View Details** on the domain status to see the specific instructions. 18. Put the IdP connection live by using **Activate Connection** under the **Actions** column. *** Modified at: July 19, 2024 Source: [/cloud/security/portal-access/the-overall-flow-of-the-sso-setup/](https://docs.singlestore.com/cloud/security/portal-access/the-overall-flow-of-the-sso-setup/) (An index of the documentation is available at /llms.txt) --- # Troubleshooting SSO Connections When using SSO to log into SingleStore Helios the most common issues encountered include: * Attempting to perform an IdP-initiated login with SAML Response. * An incorrect SSO subject type (SAML only). * Incorrect SSO assertions for email, first name, and last name  (SAML only). * Unsigned SAML Responses. * Not providing a fully qualified domain name (FQDN) when a domain is requested on the login page. * An incomplete or incorrect set of scopes (OIDC only). * Incorrect URLs in configurations. **SAML Chrome Pane**l can be used for debugging SAML connections. > **📝 Note**: The following instructions require a Chrome browser. 1. Install the [SAML Chrome Panel](https://chrome.google.com/webstore/detail/saml-chrome-panel/paijfdbeoenhembfhkhllainmocckace?hl=en) extension (or an equivalent). 2. Open Chrome Developer Tools: 1. Click the Chrome hamburger menu (three vertical dots in the top right-hand corner of Chrome) and select `More Tools` -> `Developer Tools` 2. Click `View` in the Chrome menu bar and select `Develope`r -> `Developer Tools`. 3. In the SAML Chrome Panel, click the `SAML` tab. 4. In the Chrome address bar, navigate to the Cloud Portal (). 5. *Proceed to the next step at your first failure*. Under `Actions` click `Perform Test Login`. Log into your account using your Identity Provider, and allow it to redirect back to the Cloud Portal. 6. Upon receiving an authentication error, check the SAML Chrome Panel. There should be a host of XML output displayed. 7. In the XML output verify that: * The endpoint is correct for your Identity Provider configuration. * The SSO subject is set to `“persistent”`. * There are SSO assertions for email, first name, and last name that match the names configured in the connection. 8. If you are still having trouble after verifying all of the above, copy this XML output, create a [SingleStore Support ticket](https://support.singlestore.com/), and paste this XML output into the ticket. ## Testing in the SingleStore Helios Portal Perform the test logins as follows:: 1. In the  `Authentication actions` menu for each Identity Provider connection, select ` Perform Test Login`. 2. That should end up at an error page that says “*There was an error with authentication*” and then gives a reason why the login was not successful. 3. If you do not get there, then from that same menu, select  `Debug Logins`. 4. That will list recent login attempts and provide some details about what went wrong.  The  `Login Step` column is the most important because it will indicate how far along the authentication process gets. 5. From this screen, there is also `Test Login`  which is the same as `Perform Test Login` in the previous menu. ## How to Debug SSO Connections 1. Test the new connection with the link from the UI. 2. Review the last login attempt screen - on the last login screen, you can view how far each login attempt progressed. Look at the flow of authentication steps below as a reference. If login does not succeed, knowing where it failed is key to solving the issue. 3. The flow of authentication steps: 1. Redirect to customer Identity Provider - this happens when you have figured out which SSO configuration to use and redirect to the customer IdP. 2. Record customer IdP approval - when the user has logged into the customer IdP and it has redirected back to SingleStore Helios. 3. IdP authentication - the terms of service need signing. 4. IdP authentication - the terms of service are signed. 5. IdP authentication is completed. 6. IdP authentication is complete and the auth code for the client is generated, and redirected to the authentication client (SingleStore Helios Portal) with an auth code. 7. Authentication is fully completed when the code is exchanged for the access token. The client has exchanged the auth code for an access/refresh token. ## Adding and Verifying Domains 1. SSO is always based on matching domain names. 2. Each connection has one or more domains. 3. Use `Actions` -> `Update Connection` to edit the set of domains. 4. After adding a domain, it must be verified before it can be used. 5. Domains can be “live” or not. 6. A domain can only be “live” in one connection at a time. 7. An unverified domain is not really active. *** Modified at: November 6, 2023 Source: [/cloud/security/portal-access/troubleshooting-sso-connections/](https://docs.singlestore.com/cloud/security/portal-access/troubleshooting-sso-connections/) (An index of the documentation is available at /llms.txt) --- # Privacy Privacy is becoming increasingly important when it comes to data processing and analysis. SingleStore is both GDPR and CCPA compliant and are sensitive to the privacy challenges our customers face. Below are some key questions and answers regarding data privacy as it relates to SingleStore Helios. To receive additional information, or to pose a question not answered here, please contact . ## *Who has access to my data and is the access GDPR compliant?* Site Reliability Engineers (SREs) perform much of the infrastructure and engineering work to ensure that your database, and our SingleStore Helios offering, remain operational. The responsibilities of this role are similar for all products like our SingleStore Helios. Access is both strictly monitored and frequently reviewed. Access to end-user data for the purpose of maintenance or debugging is sometimes required, and you may request to be notified about these actions. Aside from this rare occurrence, SREs will never access end-user data housed within SingleStore Helios. Our Support team may also require access to your cluster and/or data to troubleshoot and remediate issues, but this action would be undertaken at your explicit request via support ticket. You may request to be notified about these actions as well. This access is permitted under [Article 32 of GDPR](https://gdpr-info.eu/art-32-gdpr/) to maintain service continuity and the security and integrity of your data. ## *Do we need to change all our customer contracts if we start using SingleStore?* SingleStore is considered a “data processor” when customers use our SingleStore Helios to house their data and that of their end-users. In accordance with [GDPR Article 15(1)](https://gdpr-info.eu/art-15-gdpr/), it is required to disclose "the recipients or categories of recipient to whom the personal data have been or will be disclosed, in particular recipients in third countries or international organisations." Under GDPR, while the customer, or data controller, must maintain a list of data processors, disclosing this list to customers is not required. Instead, customers only need to be made aware that data processors are used. If a contract already includes this stipulation, then no additional changes to the contract are required. *** Modified at: June 22, 2022 Source: [/cloud/security/privacy/](https://docs.singlestore.com/cloud/security/privacy/) (An index of the documentation is available at /llms.txt) --- # Third Party Sub-processors SingleStore Helios is deployed in the cloud provider of your choice. SingleStore does not directly host or process customer data. When you select to deploy in AWS, Azure, or GCP all data ingested into SingleStore Helios is hosted and processed by the third-party sub-processor (cloud provider) you choose and is stored within the geographic region of the deployment. Since SingleStore does not directly host or store customer data outside of the sub-processor selected, if you have an existing arrangement with one or more cloud providers, you can migrate to SingleStore Helios without adding additional sub-processors to your existing legal or compliance processes. ## Infrastructure and Data Sub-processors | Sub-processor | Purpose | Link | | --------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Amazon Web Services | Hosting services and storage | [Amazon Web Services (AWS) Sub-processors](https://aws.amazon.com/compliance/sub-processors/) | | Azure | Hosting services and storage | [Service Trust Portal](https://servicetrust.microsoft.com/Library?command=Download\&downloadType=Document\&downloadId=926b2cf5-6b6e-43ca-9bc3-f73e961aad5f) | | Google Cloud Platform | Hosting services and storage | [Google Cloud Platform Subprocessors \| Google Cloud](https://cloud.google.com/terms/subprocessors?hl=en) | Below are the sub-processors used for support and communication (not customer data) | Zendesk | Customer support portal | | ----------- | --------------------------- | | Google Inc. | Email Support/documentation | *** Modified at: January 22, 2025 Source: [/cloud/security/third-party-sub-processors/](https://docs.singlestore.com/cloud/security/third-party-sub-processors/) (An index of the documentation is available at /llms.txt) --- # Support The topics in this section provide an overview of SingleStore schedule updates and links to the SingleStore status page. This section presents the End of Life (EOL) policy details for SingleStore and general information about how SingleStore support works. A list of frequently asked SingleStore support-related questions covers information, such as signing up with SingleStore, services offered by the support team in different tiers, filing support tickets, service-level agreements (SLAs) for support tickets, eligibility criteria for availing technical assistance, and the like.  In addition, this section links to the various locations where customers can find product help and support for SingleStore. ## In this section * [SingleStore Helios Scheduled Updates](https://docs.singlestore.com/cloud/support/singlestore-helios-scheduled-updates.md) * [SingleStore Helios Status](https://docs.singlestore.com/cloud/support/singlestore-helios-status.md) * [SingleStore Helios Service Level Agreement](https://docs.singlestore.com/cloud/support/singlestore-helios-service-level-agreement.md) * [Support FAQ](https://docs.singlestore.com/cloud/support/support-faq.md) * [Obtain Help and Support](https://docs.singlestore.com/cloud/support/obtain-help-and-support.md) * [SingleStore Software End of Life (EOL) Policy](https://docs.singlestore.com/cloud/support/singlestore-software-end-of-life-eol-policy.md) *** Modified at: August 10, 2021 Source: [/cloud/support/](https://docs.singlestore.com/cloud/support/) (An index of the documentation is available at /llms.txt) --- # Obtain Help and Support Customers with an existing license or SingleStore Helios subscription have access to support as defined by their associated agreement. SingleStore Helios provides the same support and services for SingleStore Helios that are available for self-managed deployments of SingleStore Helios. To obtain support for SingleStore Helios, visit [support.singlestore.com](https://support.singlestore.com). SingleStore Helios trial support is available via [SingleStore documentation](https://docs.singlestore.com) and [SingleStore Forums](https://www.singlestore.com/forum/). For more information regarding license agreements and SingleStore Helios support and services, visit [www.singlestore.com/legal](https://www.singlestore.com/legal). ## Enterprise Customers, SingleStore Helios, SingleStore Self-Managed All SingleStore customers who pay for SingleStore Helios, and/or have a SingleStore Self-Managed license, have access to SingleStore Support (subject to your contract and the [SingleStore Helios Terms of Service](https://www.singlestore.com/assets/contracts/memsql-helios-terms-of-service-09-05-19.pdf) or [SingleStore Software and Services Agreement](https://www.singlestore.com/assets/contracts/memsql-software-and-services-agreement.pdf), respectively). Please access Support at [support.singlestore.com](https://support.singlestore.com/) and log in or sign up to your account. *** Modified at: November 30, 2022 Source: [/cloud/support/obtain-help-and-support/](https://docs.singlestore.com/cloud/support/obtain-help-and-support/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Scheduled Updates > **📝 Note**: SingleStore Helios deployments perform weekly online updates to ensure the service receives the latest performance enhancements and bug fixes. During these updates, there may be minor performance impacts which could last for up to two (2) hours. The scheduled update time is shown per deployment in the Updates tab of the Cloud Portal and can be modified as needed. ## About Scheduled Updates Scheduled updates allow improvements, bug fixes, and optimizations to be made to SingleStore. These updates are applied automatically across infrastructure, system software, and services so that no manual user intervention is needed. The updates may trigger an online rolling restart. SingleStore Helios is highly available, and a rolling restart of services should not typically affect running workloads, however, you may experience 2 to 5 minutes where DDL is unavailable during the update. Disruptions which may be experienced during the update window include: * Pipeline batches may fail and retry. * Resources may be added and removed as the service is updated. As a result, you may see a temporary performance reduction for compute workloads as these resources are cycled. Some actions may be blocked in the two hours preceding a scheduled update. These include scaling, suspending, and resuming deployments, as well as resetting the admin password or updating the firewall allowlist. Email reminders for each upcoming update are sent to the Organization Contact Email configured in organization settings on the Cloud Portal for accounts with an active SingleStore deployment. > **📝 Note**: You can modify the email ID for the scheduled update emails by editing the Organization Contact Email in the Settings for your organization in the Cloud Portal. Refer to [Manage Organization Users](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-organization-users/#section-idm4602608765305633275111393403.md). ## How Updates are Applied Updates are applied using an enhanced upgrade process that minimizes the impact on running workloads. During updates, SingleStore transparently provisions new nodes with the updates already applied, and then transitions the workload to the new instances. This minimizes the impacts to running workloads, while keeping deployments continuously up-to-date. ## Key Benefits * Maintains compute capacity: SingleStore Helios optimizes deployment performance throughout the update process by temporarily provisioning additional capacity. * Improves update reliability: If SingleStore Helios detects an issue during the update, it can roll back to the previous version without data loss. ## Non-Production and Production When creating a new deployment, users can set the deployment type to Non-production or Production. This tag will be used when rolling out new features and patches. Updates will be automatically applied to non-production deployments before they are rolled out to production. ## Enabling New Features Updates are automatically and transparently applied to all deployments during each scheduled update; however, some new features are not enabled automatically. This allows users to test these features with applications and workloads before enabling them. When available, a list of new features will be shown in the Updates tab in the Cloud Portal. All features not currently enabled are listed along with the Month in which the latest features were made available. Enabling new features will cause an online restart, and can be enabled immediately or scheduled for the next available update. It will not be possible to roll back these features once applied. SingleStore recommends users test new features with their applications or workloads before enabling them to ensure there are no application or performance regressions. As updates and new features are rolled out progressively, not all deployments will receive updates at the same time. This ensures the highest stability of the service can be delivered at all times. Due to the progressive nature of updates, it is not possible to cancel updates, but the schedule can be adjusted by an administrator to meet the needs of a specific application. ## Client Application Expectations Cloud deployments rely on networks and distributed services, which means some temporary failures—such as dropped connections or timeouts—are inevitable rather than exceptional. Retry logic allows an application to automatically recover from these transient issues without immediately failing a user request. Without retry logic, even brief disruptions could lead to poor user experiences, lost transactions, or inconsistent states. Properly designed applications must have retries (with backoff and limits) to help maintain system resilience while avoiding additional strain on the database. In short, retry logic is a key part of building reliable, fault-tolerant cloud applications. *** Modified at: June 17, 2026 Source: [/cloud/support/singlestore-helios-scheduled-updates/](https://docs.singlestore.com/cloud/support/singlestore-helios-scheduled-updates/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Service Level Agreement **(As of 04/23/25)** This Service Level Agreement ("SLA") sets forth SingleStore's availability commitments for the Services. Capitalized terms not defined below shall have the meanings set forth in the applicable agreement between Customer and SingleStore, Inc. ("SingleStore") (the "Agreement"). 1. **SingleStore Deployment Configurations**. Monthly Uptime Percentage (defined below) is dependent on the Availability Zone Deployment Configuration (the “AZ Configuration”) used for SingleStore clusters. SingleStore may be deployed in a single Availability Zone, or across multiple Availability Zones. Target Uptime is dependent on the Availability Zone configuration. 2. **Monthly Uptime Percentage**. Monthly Uptime Percentage is the Percentage of Time (defined below) during a particular calendar month that each of a customer's Service Term is available per the terms of this SLA. The percentage of time available is calculated by the total number of monthly minutes minus the number of minutes of Downtime, divided by the number of monthly minutes. During the Subscription Term, SingleStore shall maintain a target Monthly Uptime Percentage (the "Target Uptime") per Deployment based on the percentage of time for the Service as follows: 1. Single AZ Deployments - Monthly Uptime Percentage of 99.9% (the "Target Uptime"). 2. Multi AZ Deployments - Monthly Uptime Percentage of 99.99% (the "Target Uptime"). 3. **Service Credits**. If SingleStore fails to meet the Target Uptime in a given month due to Unexcused Downtime, and if Customer meets its obligations below, Customer will be eligible to receive a Service Credit as follows. 1. **Service Credit Calculation**. Service Credits are calculated as a percentage of the total charges paid by Customer for the portion of the Deployment affected by the Unexcused Downtime for the month in which the Unexcused Downtime occurred in accordance with the schedule below: | **Single-AZ Monthly Uptime Percentage** | **Multi-AZ Monthly Uptime Percentage** | **Service Credit** | | --------------------------------------- | -------------------------------------- | ------------------ | | < 99.9% - >= 99.0% | < 99.99% - >= 99.9% | 10% | | < 99.0% | < 99.9% | 25% | 2. **Eligibility**. To receive a Service Credit, Customer must: (i) issue a Service Credit Request within thirty (30) days after the end of the calendar month in which Customer believes SingleStore’s failure to meet the Target Uptime occurred; and (ii) not be past due on any Services payments when it issues a Service Credit Request. 3. **Investigation**. Promptly after receipt of a Service Credit Request, SingleStore will investigate the request and notify Customer that either: (i) no Service Credit is due, and state the basis of this determination; or (ii) a Service Credit is due. SingleStore's determination of Monthly Uptime Percentage and any applicable Service Credit is final and binding. 4. **Issuance**. SingleStore will apply a Service Credit to the Customer in the monthly invoice following the monthly period during which SingleStore determined the Service Credit is due. Service Credits have no cash value, and are Customer's sole and exclusive remedy for any failure by SingleStore to meet the Target Uptime. 4. **Definitions.** 1. "**Downtime**" means the time (in minutes) that a Deployment is unavailable for Customer's workload and Customer is impacted by such unavailability. A minute is considered unavailable for a given Deployment if all connection requests fail during a one minute interval. Downtime does not include time before a Deployment becomes available, such as during provisioning, nor when resuming from suspended states. 2. "**Emergency Updates**" means updates performed to fix critical functionality, vulnerabilities, or material defects that may substantially impair the usability or performance of the Service, and are done as online operations whenever possible. 3. "**Approved Downtime**" means Downtime due to Emergency Updates and/or Scheduled Updates. 4. "**Update Window**" means SingleStore-designated windows for system updates, fixes, updates, etc. SingleStore will make commercially reasonable efforts to minimize impacts on Customer's workload and set Scheduled Update Windows during off-peak business hours. 5. "**Monthly Uptime Percentage**" is calculated as the total number of minutes in a calendar month minus the number of minutes of Unexcused Downtime occurring in such calendar month, divided by the total number of minutes in such calendar month multiplied by 100. 6. "**Scheduled Updates**" is any system update performed during the Update Window. 7. "**Service Credit Request**" means a Service Credit request emailed to SingleStore to stating that Customer believes that SingleStore has failed to meet the SLA and detailing the scope and nature of such failure. 8. "**Unapproved Downtime**" means Downtime other than Approved Downtime. 9. "**Deployment**" means each SingleStore compute pool in the Service which is used by Customer to process its workloads. 5. **Exclusions**. Notwithstanding anything to the contrary stated herein, this SLA does not apply to any performance or availability issues: 1. caused by factors outside of SingleStore's reasonable control, including, without limitation, any force majeure event, or any Internet access or related problems beyond the point in the network where SingleStore maintains access and control over the SingleStore Service; 2. that result from the use of services, hardware, software, or technology not provided by or directly controlled by SingleStore including, but not limited to, the cloud platform services on which SingleStore runs, issues resulting from inadequate bandwidth, or otherwise related to third-party software or services; 3. related to preview, pre-release, beta, evaluation, or trial versions of the Service, feature, or software (as defined by SingleStore); 4. that result from the unauthorized action or failure to take required action by Customer or any of its employees, agents, contractors, or vendors, or by anyone gaining access to the Service by means of Customer's passwords or equipment, or otherwise resulting from Customer's failure to follow appropriate security practices; 5. caused by Customer's use of a Service after SingleStore advised Customer to modify Customer's use of the Service, if Customer did not modify Customer's use as advised; 6. that result from Customer's failure to adhere to all required configurations, use supported platforms, follow all policies for acceptable use, or Customer’s use of the Service in a manner inconsistent with the features and functionality of the Service (for example, attempts to perform operations that are not supported) or inconsistent with SingleStore's published guidance; 7. that result from Customer's attempts to operate a Deployment beyond normal operating parameters as set forth in the Documentation (e.g., excessive workload); 8. that result from SingleStore’s throttling of excessive or suspected abusive behavior; or 9. arising from SingleStore’s suspension or termination of Customer’s right to use the Service due to Customer’s breach of the SingleStore Main Services Agreement. *** Modified at: April 23, 2025 Source: [/cloud/support/singlestore-helios-service-level-agreement/](https://docs.singlestore.com/cloud/support/singlestore-helios-service-level-agreement/) (An index of the documentation is available at /llms.txt) --- # SingleStore Helios Status [Check the status](https://status.singlestore.com/) of SingleStore Helios. *** Modified at: June 22, 2022 Source: [/cloud/support/singlestore-helios-status/](https://docs.singlestore.com/cloud/support/singlestore-helios-status/) (An index of the documentation is available at /llms.txt) --- # SingleStore Software End of Life (EOL) Policy The SingleStore Helios service is continually updated to ensure that customers receive the latest features and functionality. Customers may select the time at which updates run to ensure that they do not conflict with active workloads. Refer to [SingleStore Helios Scheduled Updates](https://docs.singlestore.com/cloud/support/singlestore-helios-scheduled-updates.md) for more information. From time to time, features and integrations may be deprecated, at which time notifications will be sent to discontinue use, or to move to an updated component. Customers have up to twelve (12) months to migrate off of the deprecated component before it ceases to function or the component is automatically updated. Refer to [Support Terms and Conditions](https://www.singlestore.com/legal/) for more information. *** Modified at: January 7, 2026 Source: [/cloud/support/singlestore-software-end-of-life-eol-policy/](https://docs.singlestore.com/cloud/support/singlestore-software-end-of-life-eol-policy/) (An index of the documentation is available at /llms.txt) --- # Support FAQ ## Who is eligible to receive Support? Customers who have an active, paid subscription or who pay for on-demand use of SingleStore Helios and/or a SingleStore Helios self-hosted License are eligible to receive SingleStore Support during the subscription or License Term. ## What are the different Support Tiers available? Information on our SingleStore Support Tiers and the differences between them are listed on [www.singlestore.com/support](https://www.singlestore.com/support/). Existing customers may have Standard, Silver, or Platinum Support as defined in their SingleStore Agreement. ## Can I upgrade my Support Tier? Yes, please reach out to your Account Executive or contact team\@singlestore.com to upgrade your Support Tier. ## Where can I go for help if I do not have access to paid support? 1. **SingleStore Forums**. To ask questions and share knowledge with other members of the community, join our public forums at [www.singlestore.com/forum](https://www.singlestore.com/forum). SingleStore employees will also monitor the forums during business hours in California. 2. **Ask StackOverflow**. We and other SingleStore Helios experts are active on the StackOverflow community. If you can’t find what you’re looking for in our documentation, [just ask a question](http://stackoverflow.com/questions/ask). Make sure to tag it with `#singlestore`! ## When should I use SingleStore Support? You may file a support ticket if you encounter the following: * A SingleStore Helios cluster outage and you need help recovering * A critical error that impedes your workload or use of SingleStore Helios * An issue that you believe is caused by a product bug or deficiency * Unexpected behavior that requires further explanation * Questions on product functionality or you would like to file a feature request ## How do I sign up for an account on SingleStore’s Support Portal? 1. Navigate to [support.singlestore.com](https://support.singlestore.com). 2. Click **Sign In** at the top of the page. 3. In the modal window, click the **Sign up** link to create a new account. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb9335c4ac5adc157/6a7adb5836668773ce27e3a3/16083b57a0167b-7DcdIP.png) 4. Fill in your information and click **Sign Up**. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt92ce57e45bf5a44d/6a7adb6e33fa8a059a1f6f4b/16083b57a098fe-oBKElX.png) If successful, a confirmation message will direct you to check your email account for a welcome email with an account verification link. Check your junk/spam folder if you do not receive your welcome email within a few minutes. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt6e0eb2fbbb7c1098/6a7adb57c5d193bbec5db8ef/16083b57a10b97-JkgrcO.png) 5. After clicking on the verification link, you will be directed back to the support portal. Confirm your name and set a password for your account. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blta4f6d962cb98cf40/6a7adb73c2cc09f8a223dbad/16083b57a17f79-pu9b6U.png) ## How do I file a support ticket? 1. Navigate to [support.singlestore.com](https://support.singlestore.com). 2. Click **Sign In** at the top of the page and enter your credentials. 3. After signing in, click **Submit a Support Request** at the top of the page. 4. Fill out the request form by following the guidelines in this FAQ. Also please indicate whether you are using SingleStore Helios or are running SingleStore Helios in a self-hosted environment. 5. After you have finished filling out the form, click **Submit**. You will see your current ticket and its replies, ticket ID for your communication with SingleStore Support, as well as the ability to add more information or additional CC email addresses. ## What does each ticket severity level mean? ## Summary This incident escalation process guides customer-reported issues from initial report to resolution. Customer-submitted issues are triaged and prioritized. Critical incidents (P1) are escalated and managed collaboratively in a dedicated communication channel. If unresolved, additional teams and management are notified. After mitigation, the Support team communicates the solution to the customer, performs a Root Cause Analysis, and implements preventive measures to avoid future issues. ## Urgency Level The urgency level defines the timeframe in which the incident must be addressed. | Urgency Level | Description | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **U1** | **Critical**These are issues that require immediate action, often involving major system outages, security breaches, or critical business disruptions.Needs resolution immediately. | | **U2** | **High**These issues are important and require prompt attention, but not necessarily as immediately as critical issues. Examples include major product bugs or service disruptions that impact a significant number of users.Needs to be addressed within 1 business day. | | **U3** | **Medium**These issues require attention within a reasonable timeframe, such as a bug that affects a small number of users or a general service issue that doesn't require immediate action.Needs to be addressed within 5 business days. | | **U4** | **Low**These are general inquiries, minor issues, or questions that don't require immediate attention or action.Needs to be addressed within 30 business days. | ## Severity Level The severity level defines the footprint of the incident's overall severity / impact. | Severity Level | Description | | -------------- | -------------------------------------------------------------- | | **S1** | Critical production outage / emergency | | **S2** | Major impact / Cluster degradation without workaround | | **S3** | Moderate impact / Cluster largely unaffected or has workaround | | **S4** | Minimal impact / General product questions | ## Priority Level The priority level can be assessed by cross-referencing an incident's urgency level with its severity level. | **Urgency Level** | **Severity Level** | | | | | ----------------- | ------------------ | ------ | ------ | -- | | **S1** | **S2** | **S3** | **S4** | | | **U1** | P1 | P1 | P2 | P3 | | **U2** | P1 | P2 | P3 | P4 | | **U3** | P2 | P2 | P3 | P4 | | **U4** | P3 | P3 | P4 | P4 | When submitting an issue, use a priority level that best matches the scenario that you are experiencing. | **Priority Level** | **Description** | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **P1** | **Critical**Production system outageSevere production problem preventing customer or workgroup from performing critical business functions. | | **P2** | **High**Job function impaired; no workaroundCustomer or workgroup able to perform job function, but performance of job function degraded or severely limited. Non-production environment outage. | | **P3** | **Medium**Moderate impact; with workaroundCustomer or workgroup performance or job function is largely unaffected. | | **P4** | **Low**Minimal impactMinimal system impact; includes feature requests and other non‐critical questions | ## Definitions | **Term** | **Description** | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Outage** | A period whenSingleStore Heliosor another customer-facing application ([singlestore.com](https://www.singlestore.com),[Forums](https://www.singlestore.com/forum), etc.) is not available and/or not operational resulting in a disruption of production service for customers orSingleStore. | | **Non-Outage** | Everything not an outage, and may include a period whenSingleStore Heliosor another customer-facing application ([singlestore.com](https://www.singlestore.com),[Forums](https://www.singlestore.com/forum), etc.) is severely degraded and/or not operating at full capacity. | | **Production System** | A live environment where systems are actively supporting business operations and serving end-users. It processes real data or transactions, meaning any failure or downtime can lead to business impact or disruption. | | **Non-Production System** | Everything not a production system. | ## What should I include in my Support ticket? 1. Provide a detailed description of the issue and include all steps/actions which led you to the issue or error you have experienced. Error messages are helpful, but please also provide the actions you took when you encountered the error, if known. 2. Include screenshots if you believe they may be helpful to our support engineers. 3. Do not include passwords, login credentials or personally identifiable information. ## How do I share files with SingleStore Support? You may attach files as large as 20MB directly into the support ticket. Please upload any files larger than 20 MB to a shared folder on services such as Google Drive, OneDrive or DropBox, and share the link with the support engineering team within your support ticket. Additionally, customers may be granted FTP access to file share upon request. If this is the case, please use the credentials provided by the support engineer to upload necessary files. ## When can I expect a response to my Support ticket? Response time Service-Level Agreements (SLAs) for SingleStore Support vary based on your Support Tier and ticket severity level. Please refer to your SingleStore Agreement and our Support Terms and Conditions for details. ## Why am I not getting emails from SingleStore Support? Please ensure that the **@memsql.com** and **@singlestore.com** email domains are whitelisted as our ticketing system may use these domains for messaging. *** Modified at: February 13, 2026 Source: [/cloud/support/support-faq/](https://docs.singlestore.com/cloud/support/support-faq/) (An index of the documentation is available at /llms.txt) --- # User and Cluster Administration This section presents the comprehensive metrics that help determine a cluster's health and performance, predict resource constraints, diagnose issues, and incorporate workload or resource improvements. It demonstrates how SingleStore exposes statistics for performance analysis at different levels, including node-, database-, and query-levels. In addition, this section captures information about the `Management` API and the `singlestore` CLI tool, both of which help automate cluster operations. ## In this section * [Manage Organizations](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations.md) * [Manage Database Users](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users.md) * [Cloud Workload Identity and Delegated Entities](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities.md) * [Use the Workload Manager and Set Resource Limits](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits.md) * [Cluster Health and Performance](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance.md) * [Billing and Usage](https://docs.singlestore.com/cloud/user-and-cluster-administration/billing-and-usage.md) * [Management API](https://docs.singlestore.com/cloud/user-and-cluster-administration/management-api.md) * [Manage SingleStore Helios using Terraform](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-singlestore-helios-using-terraform.md) *** Modified at: June 12, 2026 Source: [/cloud/user-and-cluster-administration/](https://docs.singlestore.com/cloud/user-and-cluster-administration/) (An index of the documentation is available at /llms.txt) --- # Billing and Usage This section describes the SingleStore Helios flexible usage model and how to view compute and storage consumption. ## In this section * [Flexible Usage Model](https://docs.singlestore.com/cloud/user-and-cluster-administration/billing-and-usage/flexible-usage-model.md) *** Modified at: June 22, 2022 Source: [/cloud/user-and-cluster-administration/billing-and-usage/](https://docs.singlestore.com/cloud/user-and-cluster-administration/billing-and-usage/) (An index of the documentation is available at /llms.txt) --- # Flexible Usage Model ## Consumption-Based Pricing SingleStore Helios is priced based on the amount of compute time and volume of storage used. Compute clusters consume a set number of credits per hour, and storage is billed by the average number of GB used per month. | **Compute** | **Storage** | | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | The amount of compute available for a workload is determined byclustersize. Credits are consumed while aclusteris running. | Storage is provided at a flat rate determined by the average amount of storage consumed per month. | | Starting at $0.99/hour | Starting at $0.023 per Avg GB per month | ## Compute Compute clusters consume credits while they are running. The number of compute credits consumed is dependent on the size of the cluster, the duration it runs for, the product edition, and the cloud provider and region in which it is deployed. The total number of compute resources available is determined by selecting a cluster size. This determines the number of vCPUs and the amount of memory available for a workload. The SingleStore Helios flexible usage model allows customers to purchase compute credits on-demand at list price, or through 1-year subscriptions to monthly volumes at discounted rates. ## On-Demand This option is ideal for users who don’t know how much they expect to spend or to evaluate workloads before committing to monthly consumption levels. The number of credits consumed will be determined by the cluster size, duration the cluster is running, and the cloud and region in which the cluster is deployed. Refer to [Cloud Pricing](https://www.singlestore.com/managed-service-pricing/) for more information. ## Subscription Customers can commit to monthly compute volumes in order to receive discounted rates. Customers can always “burst” above this commit at on-demand prices, and unused compute credits are rolled over to the following month. Customers can commit to a 1-year credit subscription at any time to receive discounts on compute credits. When customers sign up for a subscription, they commit to using a number of compute credits per month, which is based on how much they expect to use. If a customer uses more than the monthly number of credits, additional credits will be billed at the on-demand rate. Any credits that remain unused at the end of the month will be rolled over to the following month, for the duration of the subscription. ## Storage Storage is billed at a flat monthly rate determined by the total average GB of storage used per month and the region in which the storage is deployed. For services deployed in multiple regions, the storage charge will be the sum of all the storage charges for each respective region. Storage is purchased monthly based on the usage for the previous month. Customers are only billed for what they have already used. ## View Consumption > **📝 Note**: Compute consumption is based on the number of seconds of compute time used.Total storage consumption is based on the average amount of storage used per month. Storage usage is polled every 30 minutes, and the total storage consumption is the average of the recorded storage usage values. ## View Overall Consumption To view compute and storage consumption: * Navigate to the organization name at the top **(ORG:) > Billing & Usage > Usage Estimate ** > **📝 Note**: The **Usage Estimate** information is accurate up to the hourly level of granularity. *** Modified at: August 26, 2025 Source: [/cloud/user-and-cluster-administration/billing-and-usage/flexible-usage-model/](https://docs.singlestore.com/cloud/user-and-cluster-administration/billing-and-usage/flexible-usage-model/) (An index of the documentation is available at /llms.txt) --- # Cloud Workload Identity and Delegated Entities > **📝 Note**: This is a Preview feature.Cloud Workload Identity and delegated entities are currently only available for clusters deployed on AWS. This feature is not supported on Shared Edition. **Cloud Workload Identity** represents a set of additional components (or resources) that are provisioned for each cluster. It consists of Kubernetes and AWS resources and configurations and enables cluster workloads to use IAM Roles for Service Accounts (IRSA) to access cloud resources without long‑lived static credentials. This identity is provisioned for each cluster and is represented by an IAM role ARN. Workloads can use this identity to authenticate to cloud provider services without storing or rotating static credentials. > **📝 Note**: Cloud workload identity is unique to a cluster group. A cluster inherits the identity configuration from the cluster group that contains it. **Delegated entities** are customer-owned AWS IAM roles that a cluster's cloud workload entity is allowed to assume. This enables pipelines and other workloads to access resources such as Amazon S3 using short‑lived credentials instead of AWS access keys. ## Permissions Required You must have the following permissions to perform the respective task: | Task | Permission Required | | -------------------------------- | --------------------------- | | Add or delete delegated entities | `Manage Delegated Entities` | | View delegated entities | `View` | Refer to [Predefined Roles for Cluster Groups in an Organization](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-for-singlestore-helios/predefined-roles-for-cluster-groups-in-an-organization.md) for roles that include the `Manage Delegated Entities` permission by default. Optionally, you can also assign these permissions to custom roles. ## Configure Delegated Entities To access cloud resources using cloud workload identity, perform the following tasks: 1. [Get the cloud workload identity ARN for the SingleStore cluster.](https://docs.singlestore.com/#section-id23539041172686.md) 2. [Configure an IAM role in your AWS account.](https://docs.singlestore.com/#section-id235390421452862.md) 3. [Add delegated entities.](https://docs.singlestore.com/#section-id235390423151351.md) > **📝 Note**: You can configure a maximum of 20 delegated entities for each cluster group. You can get the cloud workload identity and configure delegated entities using either the [Cloud Portal](https://portal.singlestore.com) or the `Management` API. Once the delegated entities are configured, access cloud resources using the ARN of your IAM role. For example, use cloud workload identity instead of static credentials to load data via S3 pipelines. Refer to [Load Data from Amazon Web Services (AWS) S3](https://docs.singlestore.com/cloud/load-data/data-sources/load-data-from-amazon-web-services-aws-s-3/#section-id235391782907589.md) for an example. ## Get the Cloud Workload Identity ARN for the SingleStore Cluster The cloud workload identity value is the IAM role ARN of your cluster, for example: `arn:aws:iam::0000000000:role/xxxxxxxxxxxxxx`. ## Using the Cloud Portal 1. Log in to the [Cloud Portal](https://portal.singlestore.com). 2. Select **\ > Security**. 3. Copy the cloud workload identity ARN of the SingleStore cluster specified under **Delegated Entities > Configure trust relationship**. ## Using the Management API Use the `Identity` path (`/v1/workspaces/{workspaceID}/identity` endpoint) in the `Management` API to obtain the cloud workload identity ARN of the SingleStore cluster. For example: ```shell curl -X 'GET' \ 'https://api.singlestore.com/v1/workspaces/xxxx/identity/' \ -H 'accept: application/json' ``` Refer to [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for related information. ## Configure an IAM Role in your AWS Account To use delegated entities, configure an IAM role in your AWS account that the cluster's cloud workload identity can assume. Perform the following tasks: 1. **Create an IAM role** in your AWS account with the necessary privileges. You can also use an existing IAM role. 2. **Update the IAM role's trust policy** to allow the cluster's cloud workload identity to assume the role. Specify the cloud workload identity ARN of the cluster copied earlier. For example: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole" } ] } ``` ## Add Delegated Entities Add delegated entities to your cluster. Once the delegated entities are created, you can access cloud resources without using persistent static credentials. Ensure that you have the [permissions required](https://docs.singlestore.com/#section-id23567013433006.md) to configure and view delegated entities. ## Using the Cloud Portal 1. Log in to the [Cloud Portal](https://portal.singlestore.com). 2. Select **\ > Security**. 3. Under the **Delegated Entities** section, select **Add Entity**. 4. Add the ARN of your IAM role to the delegated entities list. Alternatively, enter a role ARN pattern to match multiple roles that share a common naming pattern, such as `arn:aws:iam::987654321098:role/singlestore-*`. Patterns provide a flexible mechanism to configure delegated entities. 5. Save the configuration. ## Using the Management API Use the `Delegated Entities` path (`/v1/workspaces/{workspaceID}/delegatedEntities` endpoint) in the `Management` API to add one or more delegated entities (ARN of your IAM role) to the cluster. > **📝 Note**: Adding delegated entities at the cluster level also updates the delegated entities for the cluster group that contains it. You can also view and delete the delegated entities for a cluster. Refer to [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md) for more information. *** Modified at: July 24, 2026 Source: [/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities/) (An index of the documentation is available at /llms.txt) --- # Cluster Health and Performance ## In this section * [Active Queries](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/active-queries.md) * [Alerts](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/alerts.md) * [Distributed Plancache](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache.md) * [Historical Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md) * [Active Workload Monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/active-workload-monitoring.md) * [Managing Plancache Memory and Disk Usage](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/managing-plancache-memory-and-disk-usage.md) * [Workload Profiling](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/workload-profiling.md) * [Configure Monitoring with Datadog](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-datadog.md) * [Configure Monitoring with New Relic](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-new-relic.md) *** Modified at: June 12, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/) (An index of the documentation is available at /llms.txt) --- # Active Queries The [SingleStore Helios Cloud Portal](https://portal.singlestore.com) has an option that lets you monitor currently running queries. This can help in identifying and troubleshooting long-running queries. ![An example of running query text on Active Queries page to troubleshoot long-running queries.](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltcd838df2fe578144/6a7adbb0fc63ab123964186f/ActiveQueries-JtP9hP.png) To access the **Active Queries** page, select Active Queries under **Monitoring** in the left navigation. Once on the Active Queries page, you can select a cluster by using the cluster selector list at the top of this page. **Kill a Running Query** To kill a running query, select the checkbox next to the query and then select the **Kill Query** button at the top of the page. **Explain a Query** To explain a query, select the checkbox next to the query and then select the **Explain Query** button. This will take you to the [Visual Explain](https://docs.singlestore.com/cloud/query-data/query-tuning/singlestore-visual-explain.md) page where you can see the profile of the query. **Export as CSV** To export the details displayed for a query, select the checkbox next to the query and then select the **Export as CSV** button. This will download the query as a CSV file. **Additional Information** Refer to [Testing Your Queries and Performance](https://docs.singlestore.com/cloud/query-data/query-tuning/testing-your-queries-and-performance.md) and [Query Tuning](https://docs.singlestore.com/cloud/query-data/query-tuning.md) for more information about queries and query troubleshooting. *** Modified at: July 17, 2024 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/active-queries/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/active-queries/) (An index of the documentation is available at /llms.txt) --- # Active Workload Monitoring Active Workload Monitoring enables you to analyze query behavior during a specific time period by comparing metrics at the start and end of the recording window. ## How it Works 1. Cumulative Metrics view ([mv\_activities\_extended\_cumulative](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-extended-cumulative.md)) contains a cumulative collection of query metrics gathered since the system started. The metrics include total CPU time, memory used, I/O, etc., for each query. 2. Start a Snapshot (T₁): when you click **Record** the system takes a snapshot of the cumulative metrics at that specific time (T₁). 3. End Snapshot (T₂): when you stop recording, a second snapshot is taken (T₂). 4. Delta Calculation: the system calculates the difference (T₂ - T₁) in metrics for each query that was running or completed during the recording window. 5. Result - the delta provides metrics specific to the recording period, such as: * CPU utilization during the recording window. * Number of query executions. * Memory consumed. ## Total CPU Percentage Metric This metric is intended for comparison purposes and is not an absolute measure of CPU capacity used. It is the query’s CPU time during the recording window divided by the recording duration, expressed as a percentage on a single-core basis, so it is useful for comparing queries but not for measuring cluster-wide CPU utilization. The formula it uses is: (sum(cpu time) / recording time)\*100. A key detail is that this calculation is always based on one CPU core, regardless of how many cores the cluster has. This means values can be greater than 100%. For example, if you record for 10 seconds and a query accumulates 87 seconds of CPU time, the metric shows about 870%. This does not mean the query used 8.7× the cluster’s total CPU capacity. It means the query consumed CPU time equal to 8.7 times the duration of the recording window, potentially across parallel work. ## What this Metric is Useful For * Comparing queries to see which ones are more CPU-intensive relative to others running or completed during the recording time. * Ranking queries by CPU usage during a given recording period. ## What this Metric Does Not Represent * It does not show CPU percentage relative to cluster’s total CPU capacity. * It does not tell you how close your cluster is to hitting CPU limits. This metric should not be used to calculate overall CPU utilization of your cluster. *** Modified at: June 11, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/active-workload-monitoring/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/active-workload-monitoring/) (An index of the documentation is available at /llms.txt) --- # Alerts ## Overview Working alongside [historical monitoring](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring.md), the alerting feature actively notifies you about the changing state of a cluster. Alerts allow you to proactively remediate any issues that could adversely affect both the cluster and the applications that depend on it. ## Prerequisites To use alerts with your cluster, SingleStore 7.3 or later is required. ## Pre-Configured Alert Types Pre-configured alert types are included that can be configured by: * Individual cluster(s), where alerts are triggered for the specified cluster(s) * Cluster group, where alerts are triggered for all clusters in the specified cluster group These pre-configured alert types, most of which can be customized, have the following components: * A *trigger condition* (alert type) refers to what’s being monitored. * Each trigger condition has a single *condition* that can have up to three *severity levels* enabled: Critical, Warn, and Info. * Each severity level has configurable parameters that, when taken together, constitute a *threshold*. * When a severity level's threshold is reached, an email alert is sent to the list of subscribers. The one exception is the Availability alert type, which has a preset threshold that cannot be customized. As SingleStore continuously monitors the SingleStore Helios infrastructure, availability alerts are first routed to, and evaluated by, SingleStore. If the issue cannot be readily resolved, an Availability alert is sent to the customer. While this alert may arrive outside of the expected notification window, please know that SingleStore is already working to resolve the issue. The following table summarizes the available pre-configured alert types, what they measure, what an alert could mean, and the preset thresholds. | **Trigger Condition**(Alert Type) | **Condition** | **Critical** | **Warn** | **Info** | **What it measures** | **What it could mean** | **Alert Method** | | ----------------------------------- | ------------------------------------ | ------------ | -------- | -------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ---------------- | | CPU Utilization | Core CPU utilization is greater than | 95% | 90% | 80% | A workload is beginning to slow due to lack of available CPU cycles | How long at least one CPU core has been running over the specified maximum threshold | Email | | For this number of minutes | 5 | 5 | 10 | | | | | | Memory Utilization | Memory utilization is greater than | 95% | 90% | 80% | How long that thecluster’s memory has been allocated over the specified maximum threshold | Theclusteris using more memory than expected | Email | | For this number of minutes | 5 | 5 | 10 | | | | | | Persistent Cache (Disk) Utilization | Persistent cache is greater than | 95% | 90% | 80% | How long the disk has been allocated over the specified threshold | Theclusteris using more disk space than expected and warrants further investigation | Email | | For this number of minutes | 5 | 5 | 10 | | | | | | Availability | Infrastructure is unavailable | N/A | N/A | N/A | Aclustermay be either offline, or online but unreachable | Whether aclusteris available | Email | The following table specifies the alerting frequency for each severity level. | **Severity Level** | **Alerting Frequency** | | ------------------ | ---------------------- | | Critical | 1 hour | | Warn | 6 hours | | Info | 24 hours | For example, if you receive a critical CPU Utilization alert for a given cluster, you will receive the next critical CPU Utilization alert an hour later. During this one-hour interval, all other critical CPU Utilization alerts will be suppressed for this cluster. ## Enable Alerts Alerts for a cluster group can either be enabled or disabled. They are enabled by default when a new cluster group is created. When alerts are enabled, the Persistent Cache Utilization alert type is also enabled by default, as are the alert types that have been enabled by the user. To enable or disable alerts for a cluster group: 1. Under **Monitoring**, select **Alerts** in the left navigation. 2. Select your cluster group from the list at the top of the page. 3. Select the **Enable Alerts** switch. ## Edit Thresholds and Alerting Level Once alerts are enabled for a cluster group: 1. Select the checkbox next to the desired cluster(s) to enable additional alert types for it. You may also bulk-edit multiple clusters by selecting additional or all clusters. 2. Select the **Edit** icon next to this cluster. 3. Select the desired alert type’s severity level to enable it and set the desired thresholds. *** Modified at: May 11, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/alerts/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/alerts/) (An index of the documentation is available at /llms.txt) --- # Configure Monitoring with Datadog Using the Datadog integration, you can monitor the health and performance of your SingleStore Helios clusters. To integrate Datadog with SingleStore Helios, you must install the SingleStore integration in Datadog, and then configure the Datadog integration in the [Cloud Portal](https://portal.singlestore.com/). ## Prerequisites * Access to Datadog. > **📝 Note**: Because a Datadog organisation can only have a maximum of 50 API keys, ensure that your Datadog organisation has less than 50 API keys when configuring the integration. Refer to [API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/#add-an-api-key-or-client-token) for more information. * An active SingleStore Helios cluster. ## Install and Configure the Datadog Integration You can start the configuration process either from the Cloud Portal or the Datadog dashboard. SingleStore recommends the following approach: 1. [Install the SingleStore Integration in Datadog](https://docs.singlestore.com/#section-idm460937000601283380534366196.md). 2. [Configure the Datadog Integration in the Cloud Portal](https://docs.singlestore.com/#section-idm4577563285104033805500032502.md). ## Install the SingleStore Integration in Datadog To install the SingleStore integration in Datadog, 1. On the [Datadog dashboard](https://app.datadoghq.com/), on the left navigation pane, select **Integrations**. 2. On the **Integrations** tab, search **SingleStore**, and then select **SingleStore** from the search results. 3. On the **SingleStore** dialog, select **Install Integration**. Once this integration is installed, proceed to the next step. 4. On the **Configure** tab, select **Connect Accounts**. This action takes you to the Cloud Portal. You only need to perform these steps once to connect your first cluster group with Datadog. Once the integration is installed and the accounts are connected, follow the steps specified under [Configure the Datadog Integration in the Cloud Portal](https://docs.singlestore.com/#section-idm4577563285104033805500032502.md) to connect consecutive cluster groups. ## Configure the Datadog Integration in the Cloud Portal To connect your SingleStore Helios cluster group with Datadog: 1. Sign in to the [Cloud Portal](https://portal.singlestore.com/). Select **Monitoring > Integrations** on the left navigation pane to access the page. 2. From the list of available integrations, select **+ Integration** for Datadog. 3. On the **Create Datadog Integration** dialog, from the **Cluster Group** list, select your cluster group. 4. Select **Create**. This action takes you to the Datadog sign-in page. After signing in to Datadog, proceed to the next step. 5. On the **Authorize access** screen, select the **Authorize** button. Upon successful authorization, you are taken to the **Integrations** page on the Cloud Portal. You can now monitor your SingleStore Helios databases using Datadog. ## Uninstall the Datadog Integration To uninstall/deactivate the Datadog integration, 1. **Uninstall the SingleStore integration on Datadog**: On the [Datadog dashboard](https://app.datadoghq.com/), select **Integrations > SingleStore > Configure > Uninstall Integration**. Once this integration is uninstalled, all previous authorizations are revoked. 2. **Remove the Datadog integration on the Cloud Portal**: On the [Cloud Portal](https://portal.singlestore.com/), go to **Monitoring > Integrations**. Select **Delete** for each Datadog configuration you want to remove. Additionally, remove all the API keys associated with this integration. To stop monitoring a specific cluster group (and not uninstall the integration), select **Delete** (**Cloud Portal > Monitoring > Integrations**) to remove the Datadog configuration for this cluster group. ## Metrics Collected Refer to [Data Collected](https://docs.datadoghq.com/integrations/singlestoredb_cloud/#data-collected) for a complete list of all the metrics that Datadog collects from the connected SingleStore Helios cluster group. *** Modified at: January 23, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-datadog/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-datadog/) (An index of the documentation is available at /llms.txt) --- # Configure Monitoring with New Relic You can monitor your SingleStore Helios clusters using New Relic via the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/). To visualize the SingleStore Prometheus metrics, use the [Prometheus receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) and send metrics data to New Relic. ## Prerequisites * A Kubernetes cluster and the `kubectl` command-line tool. * A New Relic account and a [license key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key) used for data ingest. * An active SingleStore Helios deployment. * A SingleStore [API key](https://docs.singlestore.com/cloud/reference/management-api/#section-idm4495199953840032756423936306.md). ## Configure OpenTelemetry and New Relic To ingest the SingleStore Prometheus metrics into New Relic using OpenTelemetry, 1. Clone the [New Relic OpenTelemetry Examples](https://github.com/newrelic/newrelic-opentelemetry-examples) GitHub repository. 2. Change directory to the `newrelic-opentelemetry-examples/other-examples/collector/singlestore/k8s/` directory. 3. Update the New Relic API (license) key and SingleStore API key values in the `secrets.yaml` file. ``` stringData: NEW_RELIC_API_KEY: SINGLESTORE_API_KEY: ``` 4. Update the SingleStore organization ID and cluster group ID in the `collector.yaml` file. To find these IDs, navigate to the [Cloud Portal](https://portal.singlestore.com): * In the URL, the UUID next to `/organizations/` is the organization ID. * Select Clusters in left navigation pane. Select the three dots under the **Actions** column for your cluster, and select **Settings** from the The UUID under **Group ID** is the cluster group ID. ``` - name: SINGLESTORE_ORG_ID value: - name: SINGLESTORE_WORKSPACE_GROUP_ID value: ``` 5. Run the application using the following command: ```shell kubectl apply -f k8s/ ``` To reset the configuration or clean resources, run the following command: ```shell kubectl delete -f k8s/ ``` ## View the Metrics To view the SingleStore metrics, query the New Relic data: ``` FROM Metric SELECT uniques(metricName) WHERE otel.library.name = 'otelcol/prometheusreceiver' AND metricName LIKE 'singlestore%' ``` Refer to [How to query your New Relic data](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/) for more information. To view the SingleStore dashboard, 1. Go to [one.newrelic.com](http://one.newrelic.com), and select **Integrations & Agents** on the left navigation pane. 2. On the **Integrations & Agents** page, search **SingleStore**, and select **SingleStore OpenTelemetry**. 3. On the **Installation plan** dialog, 1. Select an account. 2. [Configure OpenTelemetry and New Relic](https://docs.singlestore.com/#section-idm234728425434249.md) and select **Done**. Skip this step if the integration is already configured. 3. Select **See your data**. The SingleStore dashboard opens. ## References * [Monitoring SingleStore with OpenTelemetry Collector](https://github.com/newrelic/newrelic-opentelemetry-examples/tree/main/other-examples/collector/singlestore) * [How to query your New Relic data](https://docs.newrelic.com/docs/query-your-data/explore-query-data/get-started/introduction-querying-new-relic-data/) * [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) * [SingleStore OpenTelemetry](https://newrelic.com/instant-observability/singlestore) *** Modified at: July 24, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-new-relic/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/configure-monitoring-with-new-relic/) (An index of the documentation is available at /llms.txt) --- # Distributed Plancache > **📝 Note**: This is a Preview feature. The Distributed PlanCache (DPC) is a third layer of the plancache that supplements the [in-memory plancache and the on-disk persistent plancache (PPC)](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation/#UUID-0a745a7a-c630-4066-7532-d3ded280d626.md). The DPC allows cluster nodes to share compiled plans. Nodes can skip query optimization, [code generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md), and the LLVM compilation process if the plan has been compiled in a different node in the cluster. Thus, the load on the CPU is reduced and the first-time performance of queries that have been compiled on other nodes is improved. The DPC improves performance in the following scenarios: * **Fast scaling**: The DPC triggers plan synchronization during a node's reprovisioning phase. During reprovisioning, recently used query plans are downloaded to nodes’ PPCs. * **Clusters with multiple aggregator nodes**: Aggregators periodically sync the most recently used plans from other aggregators. ## Overview When the DPC is enabled, plans are synchronized automatically between nodes. That is, child aggregators and leaf nodes automatically download plans from the DPC into their local PPC. Once a plan is downloaded, the node can use that plan and avoid query optimization, code generation, and plan compilation in many cases. Similar to how the PPC functions, a plan downloaded to the DPC is usable except when: * A variable affecting the plan has changed in the node. * A table in a query has changed significantly (for example, the number of rows in the table has changed by a factor of two times or more) since the plan was generated. In addition to downloading plans from the DPC, nodes also upload plans to the DPC to make those plans available to other nodes. Similarly, plans can be deleted from the DPC to indicate to other nodes that those plans can be deleted. DPC operations do not interfere with regular cluster operations. While the DPC may consume CPU, memory, and network resources, all operations happen in the background and do not interfere with regular query execution. The DPC operates on a best-effort basis and does not guarantee that plan compilation will not occur on a new node. ## Remarks * DPC requires that the cluster have [Unlimited Storage](https://docs.singlestore.com/cloud/manage-data/#combining-unlimited-and-local-storage-databases.md) enabled. DPC utilizes unlimited storage to store plancache files. * DPC only stores plans generated during `MBC`, `LLVM`, and `INTERPRET_FIRST` interpreter modes. * Plan synchronization is not supported on the Master Aggregator. ## Enable and Manage the Distributed Plancache ## Enable the Distributed Plancache The `enable_distributed_plancache` engine variable controls the DPC. Use the following command to verify the DPC is enabled. ```sql SELECT @@enable_distributed_plancache; ``` ```output +--------------------------------+ | @@enable_distributed_plancache | +--------------------------------+ | 1 | +--------------------------------+ ``` ## Manage the Distributed Plancache When the DPC is enabled, nodes automatically download, upload, and delete plans from the DPC. Nodes download plans from the DPC to their local PPC when the cluster scales, is rebalanced, or a new node is added to the cluster. Nodes delete plans from the DPC when those plans are explicitly deleted from their local PPC with [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md). The download, upload, delete, and synchronization operations occur in the background and are managed by the DPC task queue. ## Aggregator Synchronization Child aggregators can be configured to automatically synchronize their local PPCs with the DPC. This process is called aggregator synchronization and is useful for clusters with multiple aggregators. When `enable_periodic_distributed_plancache_agg_sync` is set to `ON`, at the interval specified by `distributed_plancache_agg_sync_s`, each aggregator performs an aggregator synchronization operation and downloads the most recently used `distributed_plancache_max_download_plans` query plans from the DPC. ## Synchronous DPC Lookup With synchronous DPC lookup, query execution looks to the DPC before starting query optimization, and [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md). When a plan exists in the DPC, this synchronous lookup process is typically faster than local compilation, optimization, and code generation for large, complex queries. Syncronous DPC lookup is enabled when `enable_synchronous_dpc_lookup` is set to `ON` on aggregator nodes. ## Engine Variables The following engine variables are used to manage the DPC: | Name | Description | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `distributed_plancache_worker_threads` | Specifies the number of threads used to process tasks in the task queue for the DPC. | | `distributed_plancache_max_download_plans` | Specifies the maximum number of plans downloaded in a synchronization task for the DPC. | | `distributed_plancache_agg_sync_s` | Specifies the interval between aggregators' periodic synchronizations for the DPC. | | `enable_periodic_distributed_plancache_agg_sync` | Specifies if aggregators periodically synchronize their local PPC with the DPC. | | `enable_synchronous_dpc_lookup` | A session variable that specifies that nodes look for a plan in the DPC when a plan is not found in the node's local PPC. | Refer to [List of Engine Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables.md) for default values. ## Observe Distributed Plancache Statistics Connect to a node and use the [SHOW DISTRIBUTED\_PLANCACHE STATUS](https://docs.singlestore.com/cloud/reference/sql-reference/show-commands/show-distributed-plancache-status.md) command to observe statistics about the DPC on a specific node, as follows. ```sql SHOW DISTRIBUTED_PLANCACHE STATUS; ``` ```output +-------------------------------------------------------+-------+ | Stat | Value | +-------------------------------------------------------+-------+ | Successful Downloads Since Startup | 128 | | Skipped Downloads Since Startup | 110 | | Failed Downloads Since Startup | 0 | | Plans Uploaded Since Startup | 2 | | Plans Deleted Since Startup | 0 | | DB Synchronization Since Startup | 1 | | Successful Downloads From Periodic Sync Since Startup | 12 | | Skipped Downloads From Periodic Sync Since Startup | 107 | | Failed Downloads From Periodic Sync Since Startup | 0 | | Periodic Query Plan Syncs Since Startup | 1 | | Distributed Plancache Plans Used Since Startup | 90 | | Currently Queued Populate Download Tasks | 0 | | Currently Queued Download Tasks | 0 | | Currently Queued Upload Tasks | 0 | | Currently Queued Delete Tasks | 0 | | Avg Plan Download Latency (ms) | 13 | | Avg Plan Upload Latency (ms) | 38 | | Avg Plan Delete Latency (ms) | 0 | | Avg Duration For DB Plan Synchronization (ms) | 1710 | +-------------------------------------------------------+-------+ ``` **Note**: Failed Downloads indicate that the node tried to download a plan file from the DPC, but failed to do so typically because the plan did not exist in the DPC. Skipped Downloads indicate that the node tried to download the plan from the DPC, but the plan already existed in the local PPC. Use the following commands to view the number of tasks in the DPC task queue. ```sql SHOW STATUS LIKE 'Queued_DPC_Uploads'; SHOW STATUS LIKE 'Queued_DPC_Downloads'; SHOW STATUS LIKE 'Queued_DPC_PopulateDownloads'; SHOW STATUS LIKE 'Queued_DPC_Deletes'; ``` If the values of these metrics increase significantly and the number of compilations rises due to slow plan synchronization, SingleStore recommends considering an increase in the DPC worker thread pool size (`distributed_plancache_worker_threads`). ## Use DPC with a New Cluster Use the following steps to use the DPC for a specific database in a new cluster. Complete steps 1, 2, and 3 before creating the new cluster. 1. Enable the distributed plancache. This command enables the distributed plancache for the entire [cluster group](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/cluster-architecture.md). ```sql SET GLOBAL enable_distributed_plancache = ON; ``` 2. The plans that need to be downloaded to the new cluster must exist in the DPC, that is, those plans must be compiled locally in any of the nodes after `enable_distributed_plancache` was turned on. 1. This can be achieved by either removing and recompiling that plan, or letting the cluster run for a while so the nodes naturally recompile the plan over time. 3. Set `distributed_plancache_max_download_plans` to the number of plans that is expected for the database (default 1000). ```sql SET distributed_plancache_max_download_plans = ; ``` 4. Create a new cluster, attach the database to the new cluster. 5. The plans automatically sync after step 4 has been completed. 6. If more plans need to be synchronized explicitly or if the value set in step 3 was too small, run [\_SYNC\_PLANS](https://docs.singlestore.com/cloud/reference/sql-reference/code-generation-functions/sync-plans.md) `` on the aggregator node to download the most recently used query plans for that database again (plans that already exist in the DPC are skipped). *** Modified at: July 6, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/distributed-plancache/) (An index of the documentation is available at /llms.txt) --- # Historical Monitoring SingleStore Helios provides native monitoring which allows users to quickly and easily understand their application workloads and debug performance-related issues. Monitoring leverages the system information and statistics generated within SingleStore Helios, and retains and visualizes them using a dedicated and highly-available cloud monitoring infrastructure. Historical monitoring dashboards maintain a 30-day retention period, except for Query History, which retains data for 7 days. The following tables list the provided dashboards, their associated functionality, and how/when to use them to identify trends, troubleshoot and/or optimize workloads, and take action to remediate issues. ## View the Dashboards ## Cluster View | **Chart Name** | **What it shows** | **When to use it** | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CPU Utilization | The percentage of the host’s CPU that is being used:
  • max single-core load: The maximum CPU load across all of the available CPU cores
  • avg core load: The average CPU load across all of the available CPU cores
  • min single-core load: The minimum CPU load across all of the available CPU cores
| To understand CPU usage and host resource usage in general, or for a given workload. | | Memory Utilization | The percent of the host’s memory that is being used | To understand host memory usage for a given workload over time. | | Local Disk Utilization | The local disk utilization for theclusterTotal storage can be managed by dropping obsolete tables and/or purging older data in large tables. | To identify the amount of warm data so preventive actions can be taken to better handle the load, thereby ensuring stability and optimum performance. | | Read/Write Queries per Second | The number of reads/writes per second of the queries running on the system | To understand typical (“normal”) cluster activity to benchmark workloads and their query rate and identify anomalies in the read/write workload.If the number of rows read or written is very high or uneven, it could indicate that some queries or operations are taking longer to process than others. This can be due to poor indexing, inefficient queries, or database design issues. | | Failed Read/Write Queries per Second | The number of reads/writes failed per second of the queries running on the system | | | Rows Read or Written | The number of rows read/written | | | Execution Time per Read/Write Query | The elapsed time of read/write query | To identify changes in the pattern of execution time per read/write query from the historical norm. This may indicate an issue with an application or changes in your workload. | | Threads - Connected | The number of open connections (`threads_connected`) to the database relative to the maximum limit (`max_connections`) | To identify if the database is approaching the maximum allowed connections, which is indicated by a utilization near 100%. This can potentially lead to performance issues, as queries may need to wait in a queue until threads become available to process them. | | Threads - Running | The number of threads that are actively running queries (`threads_running`) relative to the maximum limit (`max_connection_threads`) | To identify if the system is approaching its capacity with regard to the number of queries that can be executed in parallel, which is indicated by a utilization near 100%. This can potentially lead to resource pressure, system unresponsiveness, latency spikes, and eventual failures. | ## Historical Workload Monitoring | **Chart Name** | **What it shows** | **When to use it** | | --------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Elapsed Time per Execution by Database | The elapsed time per query, grouped by database | To identify which databases incur the most long-running queries and observe changes in the pattern of execution time per query from the historical norm. This may indicate an issue with an application or changes in your workload. | | Execution Count | The number of queries executed in a given time | To perform capacity planning for workloads and identify if workloads in general, or workload spikes in particular, are putting theclusterat risk of running out of memory. | | CPU Time per Execution by Database | The CPU time spent per query activity, grouped by database | To identify which databases incur the most CPU usage.Note: A blank database indicates system activity that is not related to a user database. | | Memory Usage per Execution by Database | The memory bytes spent per query activity, grouped by database | To identify which databases incur the most memory usage.Note: A blank database indicates system activity that is not related to a user database. | | Disk Bytes per Execution by Database | The disk bytes spent per query activity, grouped by database | To identify which databases incur the most disk bytes.Note: A blank database indicates system activity that is not related to a user database. | | Network Bytes per Execution by Database | The network bytes spent per query activity, grouped by database | To identify which databases incur the most network bytes.Note: A blank database indicates system activity that is not related to a user database. | | Metrics by Query Plan | The queries executed and their relative resource consumption | To identify which queries are expensive, including how long queries are taking to complete, their CPU times, failure rates etc. | ## Memory Monitoring | **Chart Name** | **What it shows** | **When to use it** | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Memory Utilization | The percentage of a host’s memory that is being used | To understand host memory usage for a given workload over time. | | Memory Usage Breakdown | The memory in use categorized by Data, Query, Reserved & Other Internal Memory allocators compared to the total memory available | To perform capacity planning for memory and identify if the cluster is not performing optimally due to workloads in general, write workloads, or workload spikes in particular, and to discover where memory is allocated (table, query, etc.). | | Memory Used - Data | The data memory in use | To perform capacity planning for data memory and identify if given write workloads are putting the cluster at risk of running out of memory. | | Memory Used - Query | The query memory in use | To perform capacity planning for workloads and identify if workloads in general, or workload spikes in particular, are putting the cluster at risk of running out of memory. | | Memory Used - Other | The memory used bySingleStore’s memory allocators | To identify why memory allocations have increased, or are anomalously large, when there are no other indicators of increased memory use, such as workload or data, and to discover where memory is allocated (table, query, etc.). | ## Cache Monitoring | **Chart Name** | **What it shows** | **When to use it** | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Persistent Cache Utilization | Persistent cache utilization for thecluster.Total storage can be managed by dropping obsolete tables and/or purging older data in large tables. | To identify the quantity of warm data so preventive action can be taken to better handle the load, thereby ensuring stability and optimum performance. | | Distribution of Components Using Cache | Distribution of cache utilization by data, plancache, auditlogs, and tracelogs | To understand how the cache is being utilized.Analyzing cache usage can reveal if certain artifacts (such as data, plancache, audit logs, or trace logs) are consuming an excessive amount of space. This can either cause performance issues, or require additional resources to maintain optimal operation.Monitoring cache usage and activity can also help identify performance bottlenecks, which may require cache policies to be adjusted, additional resources to be allocated, and/or your workload to be optimized. | | Breakdown of Cache Utilization by Data | Cache consumption breakdown by "Data" category.Adding utilization across blobs, transaction logs, snapshots, temp blobs, etc. will be equal to the total cache utilized by "Data." | | | Distribution of Databases Using Cache | Distribution of cache utilization by databases.Adding utilization across databases will be equal to the total cache utilized by "Data." | | | Blob Cache Downloaded per Second (by Database) | Rate at which the blob cache is downloading files from remote storage. | To understand howSingleStore Heliosblob cache is performing.By understanding and monitoring the rate at which the blob cache is downloading files from remote storage, potential performance bottlenecks and/or issues related to blob cache activity can be identified.For example, if high download rates are observed relative to the size of your database and scale of your hardware, you may consider increasing the local cache size.Regularly reviewing this metric can help you make well-informed decisions for optimizing the performance ofSingleStore Helios. | | Blob Cache Evicted per Second (by Database) | Rate at which the blob cache is evicting files. | To understand howSingleStore Heliosblob cache is performing.By understanding and monitoring the rate at which the blob cache is evicting files, system resource utilization can be optimized based on your data management needs.A high eviction rate may indicate that the cache size is insufficient, or that your workload is imposing a high cache turnover. To improve overall cluster performance, reviewing data access patterns and adjusting the cache size is recommended.Regularly reviewing this metric can help you identify potential performance bottlenecks and make well-informed decisions for optimizing the performance ofSingleStore Helios. | ## Pipeline Dashboards ## Pipeline Summary | Chart Name | What it shows | When to use it | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | State Distribution | A high-level overview of all pipelines, including the number of pipelines in running, stopped, and error states, and the percentage of each | To identify potential issues by comparing the number of running pipelines to those that have either stopped or produced an error. | | Historical Pipeline State | The state of all pipelines over a period of time | To identify potential issues by examining how a pipeline behaves over time. | | Summary | The current state of all pipelines | To identify which pipelines are currently running, stopped, or in an errored state along with their associated database. | ## Pipeline Performance | Chart Name | What it shows | When to use it | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Execution Count | The total number of executions that have run in a pipeline (specifically, the queries that are run in the engine that ingest the data into tables) | To observe the workload from pipelines. | | Avg CPU Time per Execution | The average CPU time for each execution in a pipeline | To identify which pipelines are consuming excessive CPU cycles. | | Avg Elapsed Time per Execution | The average elapsed time for each execution in a pipeline | To identify which pipelines are experiencing degraded performance over time. | | Avg I/O per Execution | The average disk I/O (number of bytes thatSingleStoreread and written to the filesystem or the in-memory transaction log) per execution in a pipelineNote that this is the average value of[disk\_b](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-query-activities.md)from the total of`run_count`,`success_count`, and`failure_count`(from the[MV\_ACTIVITIES\_CUMULATIVE](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/mv-activities-cumulative.md)table). This is focused more on the load on the server than the data being ingested. | To identify if a pipeline is experiencing I/O-related performance issues (typically when this value is consistently high). | | Avg Memory Use per Execution | The average memory usage per execution in a pipeline | To identify which pipelines are exhibiting excessive memory use. | | Avg Network Bytes per Execution | The average network bytes per execution in a pipeline | To identify which pipelines are experiencing degraded performance due to network constraints. | | Pipeline Errors | Which pipelines have produced an error, including the pipeline name, error ID, error code, error message, and the time the error occurred | To identify and troubleshoot pipelines that have produced an error. | ## Pipeline Insights | Chart Name | What it shows | When to use it | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Rows Streamed | The number of rows streamed per pipeline over time, letting users track ingest throughput and data movement trends. | To monitor pipeline health, troubleshoot ingest issues, tune performance, audit or plan for capacity, and compare pipeline workloads. | | Data Streamed (in MBs) | The amount of data, in megabytes, streamed per pipeline over time, which allows users to monitor ingest volume and usage trends. | To assess pipeline ingest volume, detect anomalies or spikes, tune performance for large data loads, audit or plan capacity, and compare data movement across pipelines. | | # of Batches | The number of pipeline batches processed over time, which allows users to track ingest job frequency and execution patterns. | To monitor and troubleshoot pipeline activity, analyze batch processing rates, audit job execution, and compare workload patterns across pipelines. | | Time Spent per Batch | The average time taken to process each pipeline batch, which lets users track ingest speed and identify slow-running jobs. | To troubleshoot pipeline performance issues, spot latency bottlenecks, tune execution efficiency, and compare batch processing times across pipelines. | | Extractor Wait Time per Batch | The average time each pipeline batch spends waiting for extractor resources, which allows users to monitor resource contention and ingest delays. | To identify bottlenecks caused by extractor unavailability, troubleshoot pipeline latency, optimize resource allocation, and compare wait times across pipelines. | | Kafka Offsets Pending Ingest | The number of unread Kafka offsets waiting to be ingested per pipeline, which indicates the real-time backlog in streaming data pipelines. | To monitor ingest lag, detect pipeline bottlenecks, troubleshoot delays between source and target, and compare processing efficiency across Kafka-connected pipelines. | Refer to [Troubleshoot Pipeline Performance and Memory Usage](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-troubleshooting/troubleshoot-pipeline-performance-and-memory-usage.md) for more information on tracking pipeline resource usage through SQL queries using the pipeline's `activity_tracking_id` . ## Query History | Chart Name | What it shows | When to use it | | ------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Query History | Query runtimes, those queries that have succeeded, and those queries that have failed. | To view query runtimes over time, identify and resolve slow-running and failed queries, and view and optimize workloads in real time.Refer to[Query History](https://docs.singlestore.com/cloud/query-data/query-tuning/query-history.md)for additional information and examples. | ## Resource Pool Monitoring > **📝 Note**: This is a Preview feature.Please contact SingleStore [Support](https://support.singlestore.com) to enable this feature. | Chart Name | What it shows | When to use it | | --------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Finished Queries | The number of queries finished on a given resource pool | To perform capacity planning for workloads by resource pool and identify if workloads in general, or workload spikes in particular, are queueing the queries by current resource pool configurations. | | Killed Queries | The number of queries killed on a given resource pool | To understand how many queries are killed on a given resource pool. | | Queueing Queries | The number of queries queued for a given resource pool | To understand how many queries are queued over time and perform capacity planning to increase resource limits for a given pool as needed. | | Queue Time per Queued Query | The average queue time per query for a given resource pool | To understand how long queries are being queued; helps to perform capacity planning to increase resource limits for a given pool as needed. | *** Modified at: May 11, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/historical-monitoring/) (An index of the documentation is available at /llms.txt) --- # Managing Plancache Memory and Disk Usage Compiled query plan images, sometimes referred to as 'modules' or 'code modules', are stored in a plancache for later use. The plancache consists of both an in-memory plancache and an on-disk plancache, also known as the persistent plancache or PPC. Thus, the plancache incrementally consumes both memory and disk storage space. This topic addresses managing that memory and storage space. Refer to [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md) for more information on the plancache. Compiled query plan images may be removed from the in-memory plancache through expiration or eviction. A plan expires from the in-memory plancache (called expiration) after the time interval specified by the `plan_expiration_minutes` engine variable has elapsed. When a plan expires from the in-memory plan cache, the plan is removed fully from the in-memory plancache so when that plan is used again it must be fully reloaded from disk. Plans are evicted from the in-memory plan cache (called eviction) when the plancache memory usage goes over the thresholds specified by `compiled_images_eviction_memory_limit_mb` and `compiled_images_eviction_memory_limit_percent`, as described below. Once the plancache goes over the threshold, the engine evicts the least recently used query plan images until the memory use of the plancache falls below the threshold. Whether a plan expires or is evicted from the in-memory plancache, that plan remains in the on-disk plancache and is loaded back into memory the next time the query is run. Evicted plans will reload more quickly than expired plans, but in neither case will the plan be recompiled. Further, when a node restarts, the in-memory plancache starts off empty and plans are loaded back in from the on-disk plancache as queries are run. Query plans expire from the on-disk plancache as described in [Set Plan Expiration Limits](https://docs.singlestore.com/#UUID-d28dbae3-2336-1ac1-e97d-83832be8662a.md). When a plan expires from the on-disk plancache, it is completely removed from the plancache, and the query must be recompiled when it is run again. Query plans can be manually removed from the in-memory plancache as described in [Drop Plans from the Plancache](https://docs.singlestore.com/#UUID-fc252a59-15ea-15a6-9fe0-352241d148b0.md). All plans can be removed from the in-memory and on-disk plancache as described in [Drop All Plans from the Plancache](https://docs.singlestore.com/#section-idm234875781961294.md). ## Set Plan Expiration Limits Plancache expiration policy can be set through the `plan_expiration_minutes` and `disk_plan_expiration_minutes` engine variables. Expiration for the in-memory and on-disk plancaches is always enabled. Plans will expire from the on-disk plancache if the following conditions are met:  * The time limit specified by `disk_plan_expiration_minutes` is reached (assuming the plan has not been read from disk during that time), and * The plan is not in the in-memory plancache. This process also applies to temporary (temp) tables. SingleStore recommends reusing the same name for temporary tables to the degree possible. The following example shows how to set these variables to change the in-memory expiration limit to eight hours and set the on-disk expiration to one week. ```sql SHOW VARIABLES LIKE '%plan_expiration%'; ``` ```output +------------------------------+-------+ | Variable_name | Value | +------------------------------+-------+ | disk_plan_expiration_minutes | 20160 | | enable_disk_plan_expiration | ON | | plan_expiration_minutes | 720 | +------------------------------+-------+ 3 rows in set (0.08 sec) ``` ```sql SET GLOBAL plan_expiration_minutes = 480; ``` ```output Query OK, 0 rows affected (0.07 sec) ``` ```sql SET GLOBAL disk_plan_expiration_minutes = 10080; ``` ```output Query OK, 0 rows affected (0.09 sec) ``` ```sql SHOW VARIABLES LIKE '%plan_expiration%'; ``` ```output +------------------------------+-------+ | Variable_name | Value | +------------------------------+-------+ | disk_plan_expiration_minutes | 10080 | | enable_disk_plan_expiration | ON | | plan_expiration_minutes | 480 | +------------------------------+-------+ 3 rows in set (0.07 sec) ``` ## Manage Eviction of Compiled Query Plans > **📝 Note**: Plan eviction and the information in this section is applicable when `interpreter_mode` is set to `interpret_first` or `compile`. Compiled query plan images (sometimes referred to as 'modules' or 'code modules') are evicted using LRU (least-recently used) when the in-memory plancache reaches a set memory limit. The `compiled_images_eviction_memory_limit_percent` and `compiled_images_eviction_memory_limit_mb` variables are used to specify the memory limit available for caching query plan images. Query plan images are evicted when the `enable_compiled_images_eviction` engine variable is set to `ON`, which is the default setting. If the `compiled_images_eviction_memory_limit_percent` and `compiled_images_eviction_memory_limit_mb` variables are not set or are set to `0`, `compiled_images_eviction_memory_limit_percent` will be set as a percentage of `maximum_memory` as shown in the following table. | `maximum_memory`(GB) | `compiled_images_eviction_memory_limit_percent`Value | In-memory Plancache Memory Limit | | -------------------- | ---------------------------------------------------- | -------------------------------- | | 0-1 | Prohibited (`maximum_memory`minimum is 1GB) | | | 1-64 | `10` | 10% of`maximum_memory` | | 64+ | `5` | 5% of`maximum_memory` | If the `compiled_images_eviction_memory_limit_mb` or `compiled_images_eviction_memory_limit_percent` variables are set, the memory limit for the in-memory plancache is derived at runtime as shown in the following table. The notation **`_mb`** denotes `compiled_images_eviction_memory_limit_mb` and **`_percent`** denotes `compiled_images_eviction_memory_limit_percent`. | \_percent | \_mb | In-memory Plancache Memory Limit | | --------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0.0 | 0 | The value is derived in accordance with the default allocation rule specified in the table above. | | 0.0 | > 0 | If**`_mb`**exceeds the`maximum_memory`, a value derived from the above table is used. Otherwise, the value of**`_mb`**is used as is. | | > 0.0 | Any value | The value of**`_percent`**is used. | If the `compiled_images_eviction_memory_limit_percent` variable is set to a value other than 0.0, the value of `compiled_images_eviction_memory_limit_mb` is ignored. > **📝 Note**: Changes to `maximum_memory` may result in changes in the module cache limit, in accordance with the rules specified above. ## Drop Plans from the Plancache In addition to setting a retention policy for in-memory and on-disk plans, you may also choose to drop plans from the plancache to either compile a new query plan or reduce memory usage. For running nodes, the following three commands can drop a plan from the plancache: [DROP … FROM PLANCACHE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/drop-from-plancache.md) , [ALTER TABLE](https://docs.singlestore.com/cloud/reference/sql-reference/data-definition-language-ddl/alter-table.md), and [ANALYZE](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md). Successful completion of a new `ALTER TABLE` will invalidate the plancache for all queries that operate on the affected table, requiring their recompilation. The invalidation of the plancaches happens asynchronously. That is, new queries will compile and use new query plans, while currently running queries will use the old query plans. Once no queries are using the old query plans, they are marked as ready to be purged by the garbage collector. `ANALYZE` [invalidates stale plans](https://docs.singlestore.com/cloud/reference/sql-reference/operational-commands/analyze.md) from both the in-memory and the on-disk plancache, including plans which are on-disk but not in-memory. This invalidation ensures that all stale plans are discarded and that they are recompiled the next time the query is run. > **📝 Note**: The `ANALYZE` command invalidates plans by marking them as invalid so they won’t be used in the future, but it does not actually delete all the files in the on-disk plancache. The files will still be on disk, so it is not possible to reduce disk usage of the on-disk plancache with this command. ## Drop All Plans from the Plancache To remove all plans from the in-memory and on-disk plancaches, SingleStore recommends the practice of temporarily setting `plan_expiration_minutes` and `disk_plan_expiration_minutes` to `0`. Setting these engine variables to `0` will cause the plans in the plancaches to be marked as invalid and subsequently deleted by the garbage collector. To forcibly drop all plans in the in-memory and on-disk plancaches: Run `SHOW VARIABLES` and save the current values for the plan expiration minutes engine variables. ```sql SHOW VARIABLES LIKE '%plan_expiration_minutes'; /* be sure to save these values */ ``` Set the plan expiration minutes engine variables to `0` and invoke `SLEEP` to give the garbage collector time to delete the plans. A larger value for `SLEEP` may be needed depending on the size of the plancache and the speed of your disks. ```sql SET GLOBAL plan_expiration_minutes=0; SET GLOBAL disk_plan_expiration_minutes=0; SELECT SLEEP(300); ``` Reset the plan expiration minutes engine variables to their previous values. ```sql SET GLOBAL plan_expiration_minutes=; SET GLOBAL disk_plan_expiration_minutes=; ``` ## Related Topics * [Code Generation](https://docs.singlestore.com/cloud/query-data/advanced-query-topics/code-generation.md) *** Modified at: March 25, 2026 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/managing-plancache-memory-and-disk-usage/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/managing-plancache-memory-and-disk-usage/) (An index of the documentation is available at /llms.txt) --- # Workload Profiling ## Overview Starting in version 5.8, SingleStore Helios automatically exposes comprehensive metrics about cluster performance through a set of management views found in the `information_schema` database. At a high level, these views provide insight into workload bottlenecks, resource usage, and overall cluster performance. A database administrator can use this data to assess the health of the cluster, predict resource constraints, and diagnose issues. ## Background One of the core responsibilities of a database administrator is to monitor the performance of a database, and as necessary, take preventative or reactive steps to maintain its health. Complex workloads on a clustered database can make it especially difficult to prevent or fix problems when they arise. Third-party and native operating system tooling can help to diagnose issues, but some metrics are only available within the database itself. To help alleviate these difficulties, SingleStore Helios gathers statistics about the entire cluster, including all queries across all nodes, and exposes them in logical tables (views). These views contain data that can be used to help answer common questions, such as: * Which queries are using the highest proportion of system resources? * Is a query running, or is it waiting for available resources? * Which system resources are causing bottlenecks? Using these views in conjunction with built-in commands such as `PROFILE`, an administrator can better diagnose the root cause of pathological query performance, and determine which workload or resource improvements are necessary. ## Enable Advanced Counters You can enable the advanced counters by setting the `read_advanced_counters` global variable, which is discussed in the [Advanced Statistics](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/management-statistics-reference/#advanced-statistics.md) section of the [Management Statistics Reference](https://docs.singlestore.com/cloud/query-data/query-tuning/workload-profiling/management-statistics-reference.md) topic. These advanced counters are not required to use workload profiling in general, they are useful for adding more detailed stats for disk IO. ```sql SET GLOBAL read_advanced_counters = ON; ``` ```output Query OK, 0 rows affected (0.00 sec) ``` ## Concepts Each management view – and the type of data stored in it – is intended for performance analysis methodologies which seek to accomplish the following goals: * Finding high-latency queries * Finding the bottlenecks causing high-latency queries * Gathering statistics about all aspects of cluster performance to find issues beyond query execution Management views make it easier to achieve these goals by exposing the appropriate performance counters to measure each relevant attribute of the database, namely: * Node-level statistics, including what queries are running on both aggregators and leaves, or which background tasks are consuming resources * Database-level and partition-level statistics, including the specific database partition(s) associated with a query * Query-level statistics, including latency, throughput, saturation, and error metrics Together, these statistics provide insight into both high-level and low-level activities on the cluster. To learn more about the concepts and data exposed in each management view, see the [Management](https://docs.singlestore.com/cloud/reference/information-schema-reference/management.md) topic. *** Modified at: January 20, 2023 Source: [/cloud/user-and-cluster-administration/cluster-health-and-performance/workload-profiling/](https://docs.singlestore.com/cloud/user-and-cluster-administration/cluster-health-and-performance/workload-profiling/) (An index of the documentation is available at /llms.txt) --- # Manage Database Users ## Add a Database User Database users are created automatically for all SingleStore Helios users who are granted access to a cluster group. These users can access clusters in the Portal or the SQL Editor and can also be used for third-party SQL clients or development tools. Refer [Authenticate via Browser-based SSO using JWTs](https://docs.singlestore.com/cloud/security/database-access/authenticate-via-browser-based-sso-using-jwts.md) for more information about authentication from third-party clients. Database users can also be added by using the [`CREATE USER`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/create-user.md) command. The [`GRANT`](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/grant.md) command cannot be used to add a new user since auto user creation by using the `GRANT` command is deprecated and the `NO_AUTO_CREATE_USER` variable is enabled by default. If you have invited another SingleStore Helios user to join your [organization](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations.md), refrain from adding a database user with the same SingleStore Helios email address (i. e. the username) until after the invitation has been accepted. As the admin user is not always a cluster log-in option for all organization members, SingleStore recommends adding a separate database user for each organization member. ## Change a Database User Password The database `admin` password is configured when the cluster is first created. This is also referred to as the “Master Username” in the Cloud Portal, which is `admin` by default. To change this password, navigate to **Clusters** in the left navigation pane, and then select **Connect > CLI Client** for the cluster. Under **User Credentials**, select **Reset Password** next to the **Password** field and follow the provided instructions. Use the `SET PASSWORD` command to change a database user's password. A database user can also use this command to change their own password. ```sql SET PASSWORD FOR 'username'@'host' = PASSWORD('password'); ``` ## Remove a Database User To remove a database user, use the [DROP USER](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/drop-user.md) command. ```sql DROP USER ''@'' ``` ## Inspect Database User Permissions You can view grants and permissions by querying `information_schema.user_privileges`. You can also view grants for a user by running [SHOW GRANTS](https://docs.singlestore.com/cloud/reference/sql-reference/security-management-commands/show-grants.md): ```sql SHOW GRANTS FOR user@domain; ``` ## Set a Login Attempt Lockout Policy for a Database User You can specify the number of times a user can enter an incorrect password before they are locked out of the system. When a user reaches this limit, their account is locked for the specified number of seconds. This feature can be enabled per user or per role, in which case every user belonging to that role will be subject to failed login attempt lockout. ## Enable the Lockout Policy To enable the lockout policy: Set *both*`FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` for the user or role. `FAILED_LOGIN_ATTEMPTS` is the number of failed attempts before the account is locked, for example: `4`. `PASSWORD_LOCK_TIME` is the number of seconds a locked out account must wait before reattempting to log in. > **📝 Note**: You must set both `FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` to enable the feature. Enable the lockout feature at 4 failed attempts, with a lockout time of 4 hours (14400 seconds) when creating a user: ```sql CREATE USER user1 WITH FAILED_LOGIN_ATTEMPTS = 4 PASSWORD_LOCK_TIME = 14400; ``` Enabling the feature for a role: ```sql CREATE ROLE general WITH FAILED_LOGIN_ATTEMPTS = 4 PASSWORD_LOCK_TIME = 14400; ``` If a user is associated with more than one role with different password lock times, the larger `PASSWORD_LOCK_TIME` value is applied. If a user and a role the user is tied to have conflicting `FAILED_LOGIN_ATTEMPTS` settings, the lower value is applied. ## Update Lockout Settings If the `PASSWORD_LOCK_TIME` value is updated for a role or user, the new setting applies to currently locked accounts. For example, if a locked out user’s lockout time setting is 1 day, and `PASSWORD_LOCK_TIME` is then set to 4 hours, the new limit is enforced and the account will be unlocked 4 hours after it was locked. If a user’s lockout time setting is 4 hours, and the setting is increased to 1 day, the user will remain locked out for 1 day. If the `FAILED_LOGIN_ATTEMPTS` setting for a locked out user is updated to be higher than the current setting, the user is unlocked. If the new setting is lower than the current number of failed login attempts, and also higher than the user’s current number of failed login attempts, the new setting is ignored until the user successfully logs in. The user is still subject to the original `FAILED_LOGIN_ATTEMPTS` setting. ## Unlock a Locked Account To unlock a locked account: Use the `ALTER USER` command and specify `ACCOUNT UNLOCK`. ```sql ALTER USER user ACCOUNT UNLOCK; ``` If an account is locked either manually using the `ALTER USER … ACCOUNT LOCK` command or automatically has a result of exceeding the values specified for `FAILED_LOGIN_ATTEMPTS` and `PASSWORD_LOCK_TIME` arguments, restarting the cluster removes the lock and returns the account’s status to ONLINE. The account lock needs to be explicitly re-applied after the cluster is back online and/or automated as part of the customer’s operational procedures. Users can confirm the lock state before and after a restart by viewing the `ACCOUNT_STATUS` column in the information\_schema.USERS view. *** Modified at: April 30, 2026 Source: [/cloud/user-and-cluster-administration/manage-database-users/](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-database-users/) (An index of the documentation is available at /llms.txt) --- # Manage Organizations There are two types of Cloud users: 1. **Organization**: An organization (Allows shared access to clusters and related resources within a company or group.) user resides within the “control plane” of SingleStore Helios and can perform those actions that are available on the [Cloud Portal](https://portal.singlestore.com), such as manage organization users, manage clusters, and run SQL queries against databases data via the **SQL Editor** or **Kai Shell** (for SingleStore Kai enabled clusters). 2. **Database**: A database user resides within a cluster group. A database user’s lifetime is bound to that group such that, when a cluster group is deleted, all of the users, permissions, and groups are permanently removed as well. Database users can connect to the database via SQL client and run SQL queries against their data. Unlike organization users, database users must be [managed via SQL statements](https://docs.singlestore.com/cloud/security/administration/role-based-access-control-rbac-at-database-level.md). Note that a database user does not necessarily need to be an organization user to access a database created in an organization. These two types of uses work in concert to manage resources and data. ## About Organizations and Users SingleStore organizations allow shared access to deployments and resources within a company or group. A default organization is automatically created when a user registers on the [Cloud Portal](https://portal.singlestore.com). A user cannot create or rename the organization. You can view and manage an organization from the organization list at the top. It includes the following: * Current Organization * Billing & Usage * Users & Teams * Licenses * Organization Details * User Settings To view and manage the deployments within an organization, select **Clusters** on the left navigation pane. Select the three dots under the **Actions** column for your cluster, and then select an option from the list. Each organization must have a billing account to create clusters and databases. The billing account is used to pay for on-demand usage, or to pay for credits subscriptions and storage usage. Additional billing accounts can also be added, but only the default billing method will be charged for usage. A user is an Individual who logs into the Cloud Portal and can access portal services. Each user is associated with a default organization and automatically has access to all the resources within that organization. While most users will have one organization for their company, they can belong to, and be removed from, multiple organizations. An organization allows multiple users within a company to manage resources and databases affiliated with that organization. Users are common for all the databases within a cluster group. Privileges should be used to separate access. A separate admin user is created while provisioning a database. A password is specifically set for the admin user. Cloud Portal users can access the database only in the Cloud Portal **SQL Editor** and **Notebooks**. They cannot access the cluster via CLI or IDE tools. To access the database directly using the cluster endpoints use the admin user or create a separate database user. The `admin` user can log in using CLI or IDE tools. Each organization user is an administrator of that organization. A user that has admin permissions on an organization can: * Add users to, and remove users from, the organization * Manage billing and payment methods within the organization * Create resources (clusters, databases) within the organization * Manage/terminate resources (clusters, databases) within the organization Cloud Portal users and the admin user are automatically granted the following access. These permissions need not be explicitly granted. ```sql SHOW GRANTS ``` ```output GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, INDEX, ALTER, SHOW METADATA, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, ALTER VIEW, DROP VIEW, BACKUP, CREATE DATABASE, DROP DATABASE, CREATE PIPELINE, DROP PIPELINE, START PIPELINE, ALTER PIPELINE, SHOW PIPELINE, CREATE LINK, DROP LINK, SHOW LINK, DROP POOL, CREATE POOL ON *.* TO 'admin'@'%' IDENTIFIED BY PASSWORD WITH GRANT OPTION ``` Any user can manage billing accounts as needed. To add a billing account, select the organization name at the top, and then navigate to **\ > Billing & Usage**. Select the **+ Add Payment Method** button, and provide the required information. Alternatively, you can select **+ Add Payment Method** on the **Payment Methods** tab. A user’s organization is shown at the top. If a user is a member of multiple organizations, they may select the desired organization from the **Current Organization** list. ## In this section * [Manage Organization Users](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-organization-users.md) * [Manage Clusters](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-clusters.md) *** Modified at: May 22, 2026 Source: [/cloud/user-and-cluster-administration/manage-organizations/](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/) (An index of the documentation is available at /llms.txt) --- # Manage Clusters A cluster is owned by an organization, and members of the organization can create, scale, and terminate clusters. ## Create a Cluster To create a cluster, in the left navigation pane, select **Create New > Cluster**. Follow the provided instructions. A cluster cannot be renamed. Note that the cluster `admin` password is configured when the cluster is first created. SingleStore recommends that you save this password as you will need it to log into the database outside of the Cloud Portal, such as when using a SQL client. This is also referred to as the “Master Username” in the Cloud Portal, which is `admin` by default. ## Opt-in to Preview Features & Updates When enabling this option, the deployment receives the latest features and updates as soon as they are available. The configuration is suitable only for non-production deployments and cannot be changed after creation. Perform the following steps to get access to the Preview Features. 1. In the [Cloud Portal](http://portal.singlestore.com), select **Create New > Cluster**. 2. Under **Advanced Settings**, enable **Opt-in to Preview Features & Updates**. ## View Database Storage Usage and Cost To view a database’s storage usage and cost: * Navigate to the organization name in the left navigation pane: **(ORG:) > Billing & Usage > Usage Estimate > Storage Usage** to view storage used. * Navigate to the organization name at the top **(ORG:) > Billing & Usage > Usage Estimate > Billable > Storage Billed On-Demand (Average Monthly GB)** to view the monthly storage cost. * Note that the storage cost is billed in US dollars (`$`) and the storage price is dependent on the [deployment region](https://www.singlestore.com/managed-service-pricing/) that was selected when the database was created. ## Resize a Cluster Cluster size can be modified by simply selecting another size in the Cloud Portal. To resize a cluster,navigate to **Clusters**, select the three dots under the **Actions** column, and select **Resize & Scale** from the list. Resizing operations may require provisioning new instances from the cloud provider, and delays in availability can affect the overall completion time. Active workloads can further increase scaling duration. SingleStore recommends completing or minimizing active workloads before initiating resize operations. When reducing the size, ensure the new size can accommodate the existing workload. For example, when scaling down from S-4 to S-1, utilization must be below 25% to ensure a smooth scale-down process. Estimated times are not guaranteed and may vary depending on cloud provider provisioning and cluster workload. ## Suspend and Resume a Cluster Clusters do not need to be running when there is no active workload querying a database. To save cost, clusters can be suspended when inactive.  Databases will be retained as long as the Cluster Group containing the database exists. > **📝 Note**: Backup operations in SingleStore Helios fail if a cluster is in the process of resuming, suspending, or terminating. Starting a backup during these state transitions interrupts the operation and returns the following error:``` > ERROR 1317 ER_QUERY_INTERRUPTED: Leaf Error: Query execution was interrupted > ``` ## Suspend a Cluster To suspend the cluster, navigate to **Clusters**, select the three dots under the **Actions** column, and select **Suspend**. Users and Firewall rules will be retained. While suspending a cluster, the system does not forcefully terminate or block active workloads. To avoid delays, SingleStore recommends stopping or pausing all workloads before initiating the suspend operation. If workloads remain active, the suspended operation may take significantly longer to complete. ## Resume a Cluster To resume a suspended cluster, navigate to **Clusters** and select **Resume** under the **Actions** column for the cluster. Resume operations may take a few minutes to attach databases and come online. Resume operations may take a few minutes to attach databases and come online. Resume times can vary as cloud providers provision instances at different speeds. These provisioning delays occur within the cloud provider’s infrastructure layer and are not deterministically controlled by SingleStore. Therefore, consistent resume times are not guaranteed. ## Auto Suspend a Cluster Auto suspend allows you to configure a cluster to automatically suspend either after a set period of time or after a period of inactivity. It can be configured on an existing cluster, or set at the time of creation. There are two types of auto suspend:** Idle** and **Scheduled**. Only one type can be enabled at a time. * **Idle:** the cluster will get suspended after X seconds of engine inactivity. The time only takes effect from the point when the settings are enabled. For example, if you set it for 30 minutes but the cluster already has an engine idle time of one hour before the settings are enabled, then the cluster will get suspended only after 30 minutes from the time the settings are enabled. * **Scheduled:** the cluster will get suspended after X seconds have passed, irrespective of its activity. (*Note*: this type of auto-suspend is not available for Trial users.) The task that auto-suspends the clusters runs every 5 minutes, so there can be a maximum difference of five minutes between the time when the cluster should be suspended and when it will actually get suspended. Once a cluster is resumed, you can either keep the previous auto-suspend settings or disable them. The default behavior keeps the previous settings. The default auto-suspend settings at the time of cluster creation are: * For trial users, the settings are enabled by default to 20 minutes of idle time. * For standard users, the settings are disabled by default. > **📝 Note**: When a cluster resumes from auto-suspend, SingleStore Helios requests new compute capacity from the cloud provider. Capacity availability depends on the provider and may not be immediate, which can delay cluster resume operations. ## Terminate a Cluster To terminate a running cluster, navigate to **Clusters**, select the three dots under the **Actions** column, and select **Terminate** from the list. ## Data Retention in SingleStore Helios Service Post Termination Encrypted data from terminated deployments is automatically and securely purged from all systems on termination, and any administrative backup data is purged within 14 days. Hence, it is not possible to recover data from a terminated deployment. If you need to retain data post-termination, SingleStore recommends that you back up your data to an external object storage bucket. ## Cloud Workload Identity and Delegated Entities Cloud Workload Identity represents a set of Kubernetes and AWS resources and configurations that are provisioned for each cluster. It enables cluster workloads, such as pipelines, to use IAM Roles for Service Accounts (IRSA) to access cloud resources without long‑lived static credentials. Workloads can use this identity to authenticate to cloud provider services without storing or rotating static credentials. Refer to [Cloud Workload Identity and Delegated Entities](https://docs.singlestore.com/cloud/user-and-cluster-administration/cloud-workload-identity-and-delegated-entities.md) for more information. *** Modified at: May 12, 2026 Source: [/cloud/user-and-cluster-administration/manage-organizations/manage-clusters/](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-clusters/) (An index of the documentation is available at /llms.txt) --- # Manage Organization Users ## Create an Organization User (An organization user resides within the “control plane” of SingleStore Helios and can perform those actions that are available on the Cloud Portal, such as manage organization users, manage clusters, and run SQL queries against cluster data via the Cloud Portal SQL Editor.) A SingleStore account is automatically created for each user (A user that can log into the Cloud Portal and access portal services. Each Cloud Portal user is associated with a default organization and automatically has access to all of the clusters and related resources within it.) that registers on the [Cloud Portal](https://portal.singlestore.com). ## Add a User to an Organization Existing admin users can invite other users to join their organization. When adding a user to an organization, that user must already be registered on the Cloud Portal. A default organization is created for each user that self-registers on the Cloud Portal. However, a default organization is not created for a user that’s been invited to join an organization, but has not already self-registered on the Cloud Portal. To add a new user, navigate to the organization name at the top, **(ORG:) > Users \&Teams**, select the **Add Member** button, and follow the provided instructions. If a user is invited to register by an organization’s admin user, they are added to the admin’s organization upon accepting the invitation. ## Change an Organization User Password Please contact [SingleStore Support](https://support.singlestore.com) to change an organization user’s password. ## Remove a User from an Organization To remove a user from an organization, navigate to the Organization name at the top, **(ORG:) > Users & Teams**, select the ellipsis (three dots) in the **Actions** column next to the user to remove, and select **Remove Member**. ## Delete a SingleStore Account Please contact [SingleStore support](https://support.singlestore.com) to delete a SingleStore account. ## Add a Contact Email for Notifications In the **Organization Details** options, enter the **Organization Contact Email** in the **Organization Profile**. This email is used for automated notifications such as scheduled updates. SingleStore recommends configuring this as an internal distribution list of email contacts so that you can add and remove users as needed. *** Modified at: July 19, 2024 Source: [/cloud/user-and-cluster-administration/manage-organizations/manage-organization-users/](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-organizations/manage-organization-users/) (An index of the documentation is available at /llms.txt) --- # Manage SingleStore Helios using Terraform The [Terraform provider for SingleStore Helios](https://registry.terraform.io/providers/singlestore-labs/singlestoredb/latest) ("the provider") allows you to deploy and manage SingleStore Helios resources, such as compute clusters, with your Terraform workflow and configuration files. Here are a few operations that you can perform using this integration: | Category | Supported Operations | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Clustergroups |
  • Create, update, or delete cluster groups
  • Request information on cluster groups
  • Configure the deployment type
  • List the cluster groups accessible to the user
| | Clusters |
  • Create, resize, or delete clusters
  • Configure cluster size, cache configuration, and autoscaling
  • Scale and autosuspend clusters
  • Request information on clusters
  • List the clusters accessible to the user
| | Projects |
  • Create, update, or delete a project
  • List the projects available to the user
| | Private connections |
  • Configure a private connection to a cluster
  • Request information on available private connections
  • List the available private connections
| | Flow |
  • Create or delete a Flow instance
  • Request information on a Flow instance
  • List the Flow instances accessible to the user
| | Teams |
  • Create, update, or delete a team for the current organization
  • Add or remove users and teams
  • Request information on a team
  • List the teams
| | Users |
  • Invite a user to an organization
  • Remove a user
  • Request information on a user
  • List the users
| | Roles |
  • Create custom roles
  • Assign specific roles (or role grants) to a team or user
  • Request information on available roles
  • List the roles (and permissions) assigned to a team or user
  • List the roles available to a team or user
| | Invitations |
  • Get information on a specific user invitation
  • List all the invitations to the current organization
| | Regions |
  • List all the available regions that support clusters
| Refer to [SingleStore provider documentation](https://registry.terraform.io/providers/singlestore-labs/singlestoredb/latest/docs) for information on SingleStore Helios resources that you can deploy and manage using the provider. The provider uses the [Management API](https://docs.singlestore.com/cloud/reference/management-api.md) for managing SingleStore Helios instances. Refer to [Terraform provider for SingleStore Helios](https://github.com/singlestore-labs/terraform-provider-singlestoredb) GitHub repository for its source code and related information. ## Prerequisites * [Install Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) 0.13 or later. * [Generate an API key](https://docs.singlestore.com/cloud/reference/management-api/#section-idm4495199953840032756423936306.md) for your organization on the [Cloud Portal](https://portal.singlestore.com/). The provider uses this API key for authentication. ## Configure the Terraform Provider for SingleStore Helios To configure the provider to connect with SingleStore Helios, 1. Assign the API key generated earlier to the `SINGLESTOREDB_API_KEY` environment variable: ```shell export SINGLESTOREDB_API_KEY="" ``` 2. Add the `required_providers` block to your Terraform configuration file **main.tf**, and specify the SingleStore provider. You can copy the `required_providers` block from the Terraform Registry (go to [Terraform provider for SingleStore Helios](https://registry.terraform.io/providers/singlestore-labs/singlestoredb/latest)** > USE PROVIDER**). Here's a sample configuration: ```Terraform terraform { required_providers { singlestoredb = { source = "singlestore-labs/singlestoredb" version = "0.1.2" } } } provider "singlestoredb" { # Configuration options } ``` This `required_providers` block configures Terraform to use the provider for SingleStore. You can specify additional configuration options for the provider in the `provider "singlestoredb"` block. 3. Run the following command to initialize the current configuration and the SingleStore provider plugins: ```shell terraform init ``` ```output Terraform has been successfully initialized! ``` After a successful initialization, the Terraform provider is configured to connect with SingleStore Helios. You can now deploy and manage SingleStore Helios instances using Terraform. ## Example The following example demonstrates how to perform the following tasks using the provider: 1. Configure the provider. 2. Assign the API key. 3. Deploy a cluster group/cluster in the Cloud Portal. 4. Connect to the deployed cluster. 5. Display information about the deployed cluster group/cluster. 6. Terminate the cluster group and the cluster. Install Terraform before proceeding with the example. ## 1. Configure the Provider 1. Create a Terraform configuration file, **main.tf**, and then add the `required_providers` block from the [Terraform Registry](https://registry.terraform.io/providers/singlestore-labs/singlestoredb/latest). ```Terraform terraform { required_providers { singlestoredb = { source = "singlestore-labs/singlestoredb" version = "0.1.2" } } } provider "singlestoredb" { # Configuration options } ``` This configuration specifies the provider and its version. 2. Run the following command to initialize the configuration specified in the **main.tf** file: ```shell terraform init ``` ```output Terraform has been successfully initialized! ``` ## 2. Assign the API Key 1. On the Cloud Portal, select your organization from the Organization menu on the top. On the left navigation pane, select **API Keys > Create API Key**. 2. Enter a name for the API key, and set its expiration date. Select **Create API Key**. 3. Once the API key is created, copy and store it safely. The API key is displayed only once. 4. Assign the copied API key to the `SINGLESTOREDB_API_KEY` environment variable: ```shell export SINGLESTOREDB_API_KEY="xxxx" ``` ## 3. Deploy a Cluster Group/Cluster 1. Add the following code to the **main.tf** file (do not remove the provider configuration added earlier): ```Terraform data "singlestoredb_regions" "all" {} resource "singlestoredb_workspace_group" "example" { name = "testwsgroup" firewall_ranges = ["0.0.0.0/0"] // Ensure restrictive ranges for production environments. expires_at = "2222-01-01T00:00:00Z" region_id = data.singlestoredb_regions.all.regions.0.id // Prefer specifying the explicit region ID in production environments as the list of regions may vary. } resource "singlestoredb_workspace" "this" { name = "testworkspace" workspace_group_id = singlestoredb_workspace_group.example.id size = "S-00" suspended = false } output "endpoint" { value = singlestoredb_workspace.this.endpoint } output "admin_password" { value = singlestoredb_workspace_group.example.admin_password sensitive = true } ``` 2. Create a Terraform plan. Run the `terraform plan` command to display the actions the provider will take without performing them: ```shell terraform plan ``` ```output Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # singlestoredb_workspace.this will be created + resource "singlestoredb_workspace" "this" { + created_at = (known after apply) + endpoint = (known after apply) + id = (known after apply) + name = "testworkspace" + size = "S-00" + suspended = false + workspace_group_id = (known after apply) } # singlestoredb_workspace_group.example will be created + resource "singlestoredb_workspace_group" "example" { + admin_password = (sensitive value) + created_at = (known after apply) + expires_at = "2222-01-01T00:00:00Z" + firewall_ranges = [ + "0.0.0.0/0", ] + id = (known after apply) + name = "testwsgroup" + region_id = "04eb4250-5417-4300-9822-70cf4f114543" } Plan: 2 to add, 0 to change, 0 to destroy. Changes to Outputs: + admin_password = (sensitive value) + endpoint = (known after apply) ``` Note that this is only a preview of the actions that will be taken, and these actions are not yet applied. 3. Once you're ready, run the `terraform apply` command to deploy the configuration specified in the **main.tf** file (enter `yes` at the prompt): ```shell terraform apply ``` ```output Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # singlestoredb_workspace.this will be created + resource "singlestoredb_workspace" "this" { + created_at = (known after apply) + endpoint = (known after apply) + id = (known after apply) + name = "testworkspace" + size = "S-00" + suspended = false + workspace_group_id = (known after apply) } # singlestoredb_workspace_group.example will be created + resource "singlestoredb_workspace_group" "example" { + admin_password = (sensitive value) + created_at = (known after apply) + expires_at = "2222-01-01T00:00:00Z" + firewall_ranges = [ + "0.0.0.0/0", ] + id = (known after apply) + name = "testwsgroup" + region_id = "04eb4250-5417-4300-9822-70cf4f114543" } Plan: 2 to add, 0 to change, 0 to destroy. Changes to Outputs: + admin_password = (sensitive value) + endpoint = (known after apply) Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes ---- output omitted here --- Apply complete! Resources: 2 added, 0 changed, 0 destroyed. Outputs: admin_password = endpoint = "svc-XXXX-dml.aws-london-1.svc.singlestore.com" ``` You can also view the progress of the deployment process on the Cloud Portal. Once the deployment is complete, a cluster group named **testwsgroup** and a cluster named **testworkspace** is created. You can connect to the cluster using the endpoint returned in the output. The endpoint is also displayed in the output of the `terraform show` command. ## 4. Connect to the Cluster Use the endpoint from the output to connect to the cluster created in the previous step. The following example uses the endpoint of the cluster with the admin user's password and runs the `SHOW DATABASES` command on the cluster: ```shell export endpoint=$(terraform output -raw endpoint) export admin_password=$(terraform output -raw admin_password) singlestore -u admin -h $endpoint -P 3306 --default-auth=mysql_native_password --password=$admin_password -e 'SHOW DATABASES' ``` ```output +--------------------+ | Database | +--------------------+ | cluster | | information_schema | | memsql | +--------------------+ ``` ## 5. Display Information about the Deployment Run the `terraform show` command to get information about your deployed SingleStore Helios resources and display it: ```shell terraform show ``` ```output # data.singlestoredb_regions.all: data "singlestoredb_regions" "all" { id = "internal" regions = [ { id = "04eb4250-XXXX-70cf4f114543" provider = "AWS" region = "Europe West 2 (London)" }, --- output omitted here --- ] } # singlestoredb_workspace.this: resource "singlestoredb_workspace" "this" { created_at = "2023-07-06T11:55:41.391981Z" endpoint = "svc-XXXX-dml.aws-london-1.svc.singlestore.com" id = "e8704b30-XXXX-4388633ab0b4" name = "testworkspace" size = "S-00" suspended = false workspace_group_id = "959ca590-XXXX-3d94ae2a9570" } # singlestoredb_workspace_group.example: resource "singlestoredb_workspace_group" "example" { admin_password = (sensitive value) created_at = "2023-07-06T11:52:24.250554Z" expires_at = "2222-01-01T00:00:00Z" firewall_ranges = [ "0.0.0.0/0", ] id = "959ca590-XXXX-3d94ae2a9570" name = "testwsgroup" region_id = "04eb4250-XXXX-70cf4f114543" } Outputs: admin_password = (sensitive value) endpoint = "svc-XXXX-dml.aws-london-1.svc.singlestore.com" ``` This command displays a list of all the available regions and information on the deployed cluster group and the cluster attached to it. ## 6. Terminate the Cluster Group and the Cluster Run the `terraform destroy` command to terminate the cluster group created earlier (enter `yes` at the prompt): ```shell terraform destroy ``` ```output Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # singlestoredb_workspace.this will be destroyed - resource "singlestoredb_workspace" "this" { - created_at = "2023-07-06T11:55:41.391981Z" -> null - endpoint = "svc-XXXX-dml.aws-london-1.svc.singlestore.com" -> null - id = "e8704b30-XXXX-4388633ab0b4" -> null - name = "testworkspace" -> null - size = "S-00" -> null - suspended = false -> null - workspace_group_id = "959ca590-XXXX-3d94ae2a9570" -> null } # singlestoredb_workspace_group.example will be destroyed - resource "singlestoredb_workspace_group" "example" { - admin_password = (sensitive value) -> null - created_at = "2023-07-06T11:52:24.250554Z" -> null - expires_at = "2222-01-01T00:00:00Z" -> null - firewall_ranges = [ - "0.0.0.0/0", ] -> null - id = "959ca590-XXXX-3d94ae2a9570" -> null - name = "testwsgroup" -> null - region_id = "04eb4250-XXXX-70cf4f114543" -> null } Plan: 0 to add, 0 to change, 2 to destroy. Changes to Outputs: - admin_password = (sensitive value) -> null - endpoint = "svc-XXXX-dml.aws-london-1.svc.singlestore.com" -> null Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes singlestoredb_workspace.this: Destroying... [id=e8704b30-XXXX-4388633ab0b4] singlestoredb_workspace.this: Destruction complete after 3s singlestoredb_workspace_group.example: Destroying... [id=959ca590-XXXX-3d94ae2a9570] singlestoredb_workspace_group.example: Destruction complete after 0s ``` The cluster group and the cluster attached to it have now been terminated. ## Next Steps Refer to [Terraform Documentation for SingleStore provider](https://registry.terraform.io/providers/singlestore-labs/singlestoredb/latest/docs) for information on SingleStore Helios resources that you can deploy and manage using the provider. *** Modified at: May 20, 2026 Source: [/cloud/user-and-cluster-administration/manage-singlestore-helios-using-terraform/](https://docs.singlestore.com/cloud/user-and-cluster-administration/manage-singlestore-helios-using-terraform/) (An index of the documentation is available at /llms.txt) --- # Management API The SingleStore `Management` API provides a REST-based interface for provisioning, configuring, operating, and managing cloud resources within SingleStore Helios. The API is designed to support infrastructure automation, lifecycle management, and seamless integration with DevOps, CI/CD, and infrastructure-as-code workflows, enabling you to automate deployment pipelines, dynamically scale environments, enforce governance policies, manage operational workflows, and more. This RESTful API allows you to perform the following operations: * **Cluster groups**: Create, update, or delete cluster groups, request information on available cluster groups, list the cluster groups * **Clusters**: Create, update, suspend, resume, or delete clusters, request information on clusters, list the clusters, get the outbound allowlist for a cluster * **Projects**: List the projects accessible to a user * **Private connections**: Create, update, or delete private connections, request information on private connections * **Flow**: Create or terminate a Flow instance, request information on a Flow instance, list the Flow instances * **Invitations**: Invite a user, revoke an invitation, list the open invitations, request information on an invitation * **Teams**: Create, update, or delete a team for the current organisation, request information on a team, list the teams * **Users**: Add an existing user to the current organization, delete a user, request information on a user, list the users * **Roles**: Create, update, or delete a custom role, request information on a role, list the roles * **Jobs**: Create, update, or delete a job, request information on jobs, list the jobs * **Stage**: * Create, update, or delete a folder in a Stage, request information on a folder * Upload, download, modify, or delete a file in a Stage * **Files**: Create, upload, update, download, or delete a file in the personal, shared, or model space, list the files, request information on a file * **Secrets**: Create, update, or delete a secret, request information on a secret, list the secrets * **Audit logs**: Request Control Plane audit log entries * **Organizations**: Request information on current organization * **Billing**: Get raw compute and storage usage information for your organization * **Shared Edition** (Shared Tier): * Create or delete a Shared Edition deployment (starter cluster), request information on a starter cluster, list the starter clusters * Create, update, or delete a user in a Shared Edition deployment * **Cloud workload identity and delegated entities**: Request information on the cloud workload identity of a cluster, add or remove delegated entities, list the configured delegated entities * **Regions**: List the regions for a user The `Management` API supports [HTTP Bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/), which must be used in conjunction with SSL (HTTPS) for secure communication over the network. API calls made over HTTP will fail. The `Management` API authenticates requests using a unique, hex-encoded alphanumeric key, which can be generated on the [Cloud Portal](https://portal.singlestore.com/). For information on the `Management` API, refer to [Management API Overview](https://docs.singlestore.com/cloud/reference/management-api.md) and [Management API Reference](https://docs.singlestore.com/cloud/reference/management-api/reference.md). You can also interface with the `Management` API from [SingleStore Notebooks](https://docs.singlestore.com/cloud/container-services/notebooks.md) using the [Fusion SQL](https://docs.singlestore.com/cloud/reference/fusion-sql.md) commands. *** Modified at: May 26, 2026 Source: [/cloud/user-and-cluster-administration/management-api/](https://docs.singlestore.com/cloud/user-and-cluster-administration/management-api/) (An index of the documentation is available at /llms.txt) --- # Use the Workload Manager and Set Resource Limits ## In this section * [Use the Workload Manager](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md) * [Set Resource Limits](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits.md) * [FAQs](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/faqs.md) *** Modified at: March 2, 2026 Source: [/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/) (An index of the documentation is available at /llms.txt) --- # FAQs **What does MEMORY\_PERCENTAGE=50% in a resource pool mean?** The resource pool is given 50% of the available query memory on the node (aggregator and leaf). Note, this available query memory varies over time. The amount of memory on any node currently available for query execution can be calculated as (total\_server\_memory - alloc\_query\_execution - buffer\_manager\_cached\_memory). **In a cluster of 4 aggregators, 10 leaves, 148 db partitions, and workload\_management\_max\_connection\_per\_leaf=10K. The queries are on average using 300 connections. How many queries can run at a time?** 10K max connections per leaf for all aggregators = 10K/4 =2.5K max connections per aggregator. Assuming equal distribution of queries across all aggregators, WM evenly distributes the quotas across all aggregators. As per the [Queuing Policy](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md) there are two levels of queues in WM, each allocated 50% of resources. So one queue has 2.5/2 = 1.25K connections quota. Hence on any aggregator, only that number of queries can run such that they use at most 1.25K connections. Assuming queries are on average using 300 connections, about 4 queries can run at a time and rest will be queued. Note: In `INFORMATION_SCHEMA.MV_PROCESSLIST`, the column `REASON_FOR_QUEUEING` shows which component is responsible for queuing a query. **Why are queries using so many connections on a leaf?** The formulae for connections used by a query is: Number of leaf connections by a query = 1 + numBroadcasts \* numLeaves + numReshuffles \* numPartitions where numBroadcasts and numReshuffles depend on the type of query and query plan. You can get the number of broadcasts and reshuffles from ```sql SELECT QUERY_TEXT, OPTIMIZER_NOTES FROM INFORMATION_SCHEMA.PLANCACHE; ``` You can also get more information about query statistics from ```sql SELECT QUERY_TEXT, AVERAGE_MEMORY_USE, QUEUED_TIME, OPTIMIZER_NOTES FROM INFORMATION_SCHEMA.PLANCACHE; ``` **Is queuing by WM indicating a problem with the system?** It may or may not be. It may just be that WM is trying to ensure that leaf resources are not exhausted by queueing queries. Exhaustion of leaf resources can cause other issues like queries failing as nodes might not be able to open new network connections or spawn new threads. **The sum of a pool's CPU < 100% but > 95% . What could be a reason for an error while pool creation or alteration?** This occurs because system\_optimizer\_pool uses 5% CPU implicitly. *** Modified at: June 9, 2025 Source: [/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/faqs/](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/faqs/) (An index of the documentation is available at /llms.txt) --- # Set Resource Limits The **Resource Governance** (RG) feature restricts the resources that queries can use by grouping the queries into “pools” and then assigning quotas of resources to these pools. This allows you to prevent non-critical workloads from overloading the system. You can use resource pools to prevent a user’s query from making the system unusable for others connected to the database. Resource limits are for all databases in a cluster. It is not possible to set limits at the database level. The maximum number of resource pools allowed is 10. The resources controlled by resource pools are: * **Concurrency:** Limits the total queries that can run simultaneously in any given pool. For example, if you define a pool with a max concurrency of 25 and there are 4 aggregators in the cluster, each aggregator will only allow FLOOR(25/4)=6 queries to run at a time in that pool. The rest of the queries will be queued in FIFO order. Only queries are queued to limit concurrency. * **CPU:** Limits the percentage of available CPU capacity assigned to a pool. It includes both aggregator and leaf node CPUs. * On bare‑metal or VM‑based deployments, this corresponds to the host CPU capacity visible to the node. * On Kubernetes‑based deployments, the available CPU capacity is determined using the minimum of the host cores and the `cgroup` CPU limit for the container. Resource Governance then applies pool CPU percentages against this effective (cgroup‑aware) capacity. For example, if your leaf nodes have 100 cores, and Pool A has 20% as its CPU limit, 20 cores will be assigned to Pool A on each leaf node. * On a bare‑metal deployment, if your leaves have 100 cores and Pool A has 20% as its CPU limit, approximately 20 cores will be assigned to Pool A on each leaf node. * On a Kubernetes deployment where the host has 100 cores but the cgroup limit is equivalent to 40 cores, and Pool A has 20% as its CPU limit, Pool A can use up to 20% of 40 cores on each leaf node. All queries running in Pool A will share the same 20 cores between themselves. The OS will manage context switching between the queries. There are two variants: 2. **SOFT\_CPU\_LIMIT\_PERCENTAGE:** Pool B can temporarily use cores from Pool A when Pool A is underutilizing its CPU reservation. As a result, Pool A might temporarily have less than 20 cores at its disposal but there will be overall better performance. 3. **HARD\_CPU\_LIMIT\_PERCENTAGE:** Pool B can’t use cores from Pool A and hence the performance is more deterministic. Each pool can have both SOFT\_CPU\_LIMIT\_PERCENTAGE and HARD\_CPU\_LIMIT\_PERCENTAGE set. Which mode the resource governor is running in is determined by the variable `resource_governor_cpu_limit_mode`. To display it, execute: ```sql SELECT @@resource_governor_cpu_limit_mode; ```   All pools run in either the hard or soft mode, i.e. all pools have the same mode. * **Memory:** Limits the total leaf node memory that all queries running in a specific pool can together use. Which queries get killed due to lack of memory is determined by the values set for `QUERY_MEMORY_PERCENTAGE` and `MEMORY_PERCENTAGE` at the time of the resource pool creation. Any single query that uses more memory than is allocated via `QUERY_MEMORY_PERCENTAGE` will be killed. Further, the total memory of all queries running in the resource pool cannot exceed the value of `MEMORY_PERCENTAGE`. Any queries that exceed that limit will be killed. * **Query Timeout:** This can be specified to kill queries if they have been running or queued for a long period of time. For more information on the above settings, see [CREATE RESOURCE POOL](https://docs.singlestore.com/cloud/reference/sql-reference/resource-pool-commands/create-resource-pool.md). Each client connection is assigned to a resource pool that specifies some (or all) of these limits. > **📝 Note**: Resource limits also apply to any `LOAD DATA` queries, internal sampling queries, or stored procedures that are run by a user within a given resource pool. To see the status of queries running in pools run: ```sql SELECT * FROM INFORMATION_SCHEMA.MV_RESOURCE_POOL_STATUS; ``` To see the details about properties of pools run: ```sql SELECT * FROM INFORMATION_SCHEMA.RESOURCE_POOLS; ``` To see how much memory a query is using, check `MEMORY_BS` in `INFORMATION_SCHEMA.MV_ACTIVITIES`. To see the total memory allocated (not reserved) by a node currently use: ```sql SHOW STATUS EXTENDED LIKE '%TOTAL_SERVER_MEMORY%'; ``` Related to setting resource limits through governance is the concept of workload management. Workload management is a component of SingleStore Helios that automatically manages cluster workloads to ensure that they are matched with available system resources. For more information, see [Workload Management](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md). ## In this section * [Resource Pool Basics](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits/resource-pool-basics.md) * [Work with Resource Pool Engine Variables](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits/work-with-resource-pool-engine-variables.md) * [Administer Default Resource Pools](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits/administer-default-resource-pools.md) *** Modified at: March 2, 2026 Source: [/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits/](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits/) (An index of the documentation is available at /llms.txt) --- # Use the Workload Manager SingleStore Helios’s Workload Manager (WM) automatically manages cluster workloads by queuing and executing distributed queries based on available leaf connections, threads, and memory, so that queries only run when these resources are sufficiently available. It improves query execution efficiency and prevents workload surges from overwhelming the system. WM does not enforce CPU limits; CPU usage is controlled separately via resource pools (Resource Governance). WM is concerned with: * number of threads on leaves specified by the `workload_management_max_threads_per_leaf` variable. * number of connections on each leaf specified by the `workload_management_max_connections_per_leaf` variable. * leaf memory. **Definition of the Variables Used by WM** ` workload_management_max_threads_per_leaf ` The maximum number of threads to use per leaf. This number correlates with the `max_connection_threads` engine variable, and they should generally be set to the same value. This variable can sync to all aggregators. ` workload_management_max_connections_per_leaf ` The maximum number of connections to use per leaf node in the cluster. This variable can sync to all aggregators. ` max_connection_threads ` The maximum number of kernel threads for processing queries. For more, see [In-Depth Variable Definitions](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#in-depth-variable-definitions.md). WM divides leaf resources among multiple aggregators. Assume a system with one leaf node and two aggregators. The  `workload_management_max_threads_per_leaf` variable is set to 1000, implying the total number of threads on the leaf node at any point of time is limited to 1K. Queries running on both aggregators will try to use the threads on the leaf and the WM will attempt to split this limit of 1K threads between the two aggregators based on the requirement of the queries on those two aggregators. If Aggregator1 is heavily loaded, the WM may decide to give 900 threads quota to Aggregator1 and 100 threads quota to Aggregator2. To see these quotas, which change with time, on the specific node run: ``` SHOW WORKLOAD MANAGEMENT STATUS; ``` **`workload_management_min_sample_based_memory_estimation_runs`** Minimum number of runs for workload management to use sample based rather than static row size based memory estimation. When disabled (set to 0), the first time query run will not be managed by workload management based on its memory. In addition to splitting leaf resources among aggregators, WM also throttles the number of queries that can run on any aggregator. This is to ensure the current running queries do not exceed the quota on the aggregator. WM does this throttling by queuing queries. WM queues queries after Resource Governance. **Differences between Workload Manager and Resource Governance** * Related to workload management is the concept of resource governance. Unlike the Workload Manager which is built-in and enabled by default, you have to enable the resource governance. For more information, see [Set Resource Limits](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits.md). * In SingleStore’s query pipeline, if resource governance is enabled then resource governance given priority. WM queues queries after resource governance. * Resource governance and workload management are two mutually exclusive features. The Workload Manager does not have access to the resource pool configurations and the memory limits imposed by them. * The Workload Manager classifies a query based on estimated usage (connections, threads, memory) and determines if it can run or should be queued. * The Resource Pool checks `MAX_CONCURRENCY`, `MAX_QUEUE_DEPTH` to decide whether a query can run or should be queued. * Once a query starts executing, the Resource Pool enforces limits such as `MEMORY_PERCENTAGE`, `CPU_PERCENTAGE`, `QUERY_MEMORY_PERCENTAGE`, and `QUERY_TIMEOUT` to control actual resource usage and prevent overload. ## Query Lifecycle Flowchart This is a flowchart of how a query lifecycle is spent when it interacts with the aggregators, passes through the resource pool, and then comes under WM. ![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt5b8bef3d68a53259/6a7adbe977b03421603f3fea/Workload_query_lifecycle_flowchart-gtkvXo.png) ## Memory Management for Workload Manager The following three approaches are used to calculate the memory cost for a query and leverage the memory cost for workload management. * **Static row size based estimation**: For queries that do not have previous run stats, the memory cost is based on the estimated row count of each executor of the optimised query plan. This is multiplied by each executor’s projected row size and then summed up as the expected memory usage. The estimation can be found in the optimizer notes of the query’s corresponding plancache entry, for example: `"join_memory_costs_estimates":{"rows_in_memory": 1", bytes_in_memory": '8388616, "total_tables_saved_in_memory": 2}`. * **Sample based estimation**: For non-first run of non-profile queries, task stats are periodically collected from the leaf nodes plancache and these samples are used to calculate an expected memory cost. However, in this approach, if the leaf nodes or aggregator’s plancache entry get evicted, then the calculation will not be accurate. * **Profile based estimation**: For non-first run of profiled queries, task memory cost is collected during the profile stats collection phase and gathered by the aggregator after the query run finishes. Then the aggregated memory cost is used as the expected memory cost. Unlike the sample based estimation, this approach does not have the problem stated above, since the memory cost of only a specific run is used. ## Configure Workload Management The workload manager can be configured using engine variables, and the default settings are sufficient for most users regardless of cluster size. However, these engine variables can be tuned to fully utilize system resources for high concurrency workloads. These variables are discussed in the sections that follow. Before changing any of these values, ensure that you understand each of these variables and have evaluated their impact on your workload. All of the workload management engine variables are sync variables. To configure these variables, first read about [Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/sync-variables.md) to learn how to set them. Then, refer to [Workload Management Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists.md) for information on specific variables. ## Workload Management Queue Enhancements Incoming queries are filtered into queues based on their size (large, medium, or small). If a large number of queries are all filtered into the same queue, that queue could become oversaturated and this could lead to slower processing. SingleStore Helios has two engine variables `workload_management_queue_size_allow_upgrade` (specifies how many large queries can be waiting for execution in the LARGE queue when we want to move to that queue a MEDIUM query from the MEDIUM queue) and `workload_management_dynamic_resource_allocation` (enables/disables the dynamic reuse of WM queues feature) for dynamic query handling. These variables work together to dynamically move a query from a saturated queue to another queue if there are available resources. For example, if the medium queue is saturated and the large queue is not, the medium query will be moved into the large queue. To utilize dynamic query handling, set `workload_management_dynamic_resource_allocation variable` to ON. Detailed information about these engine variables may be obtained in the [Workload Management Sync Variables](https://docs.singlestore.com/cloud/reference/configuration-reference/engine-variables/list-of-engine-variables/#sync-variables-lists-2.md) section of the engine variable list. ## Query Classification The workload manager classifies queries by their size: small, medium, or large. Queries are classified to determine their [Queueing Policy](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md) described below. ## Small Queries Small queries do not exceed the `workload_management_memory_queue_threshold`, which is a small amount of memory relative the total amount of leaf memory available to queries. Small queries also do not use distributed joins. For this reason, they do not require threads or connections on the leaves. ## Medium Queries Medium queries are queries whose estimated resource usage per leaf (threads, connections, and memory) falls between a lower and an upper bound that Workload Management (WM) maintains internally: * Lower bound (small vs. medium) * Threads per leaf ≥ 1 * Connections per leaf ≥ 1 * Memory per leaf ≥ `total_leaf_query_memory` \* `workload_management_memory_queue_threshold` * Queries below lower bound limits are treated as small and are never queued by WM. * Upper bound (medium vs. large) * Threads per leaf ≤ 0.5 \* `workload_management_max_threads_per_leaf `/ `workload_management_expected_aggregators` * Connections per leaf ≤ 0.5 \* `workload_management_max_connections_per_leaf` / `workload_management_expected_aggregators` * Memory per leaf ≤ `total_leaf_query_memory` / `workload_management_expected_aggregators` Queries whose estimated usage is between these lower and upper bounds are classified as medium. Queries above the upper bound are classified as large and use the global queue. Thresholds are internal estimates per leaf, not numbers a user manually plugs in. Workload Management computes “threads used per leaf” and “connections used per leaf” internally from the query plan and runtime settings (for example `workload_management_max_threads_per_leaf`, flexible parallelism, join shape, etc.). It then uses them internally to decide whether a query is classified as small, medium, or large and if it must be queued. Hence in practice: * You should treat the thresholds conceptually (small / medium / large) rather than trying to hand-calculate the exact threads-per-leaf for every statement. * To understand why queries are being queued, you use: * `SHOW WORKLOAD MANAGEMENT STATUS` command or `MV_WORKLOAD_MANAGEMENT_STATUS` view to see aggregate running and queued queries and global thresholds. This gives an idea which queue is hot and which resource is tight. * `WORKLOAD_MANAGEMENT_QUEUE` information\_schema view to see which queries are queued and whether they are medium or large (`QUERY_TYPE`) and which resource (threads, connections, memory) is limiting them. ## Large queries Queries that exceed the upper threshold for medium queries are classified as large queries. ## Example: Checking Why Queries are Queued 1. Check overall WM status ```sql SHOW WORKLOAD MANAGEMENT STATUS; ``` Key details to observe: * **Running Threads Per Leaf** (from local queue) – total threads per leaf currently used by medium queries. * **Running Connections Per Leaf** (from local queue) – total connections per leaf for medium queries. * **Memory Threshold (MB) to Queue Locally** – memory level above which medium queries start queuing locally. * **Memory / Connections / Threads Threshold to Queue Globally** – limits above which queries are considered large and use the global queue. * **Queued Queries (from local queue) vs Queued Queries (from global queue)** – how many medium vs large queries are waiting. This helps find out whether you are hitting medium limits (local queue) or large limits (global queue). 2. Find out which specific queries are queued and why: ```sql SELECT QUERY, QUERY_TYPE, THREADS_PER_LEAF, THREADS_PER_LEAF_LIMIT, CONNECTIONS_PER_LEAF, CONNECTIONS_PER_LEAF_LIMIT, LEAF_MEMORY, LEAF_MEMORY_LIMIT FROM information_schema.WORKLOAD_MANAGEMENT_QUEUE; ``` Key details to observe: * `QUERY_TYPE` is Medium or Large as per WM’s classification. * For any queued row, at least one `*_USED` (internal) is close to or above its corresponding `*_LIMIT`: * If `THREADS_PER_LEAF` is close to `THREADS_PER_LEAF_LIMIT`, threads are the bottleneck. * If `CONNECTIONS_PER_LEAF` is close to `CONNECTIONS_PER_LEAF_LIMIT`, connections are limiting. * If `LEAF_MEMORY` is close to `LEAF_MEMORY_LIMIT`, memory is limiting. You can then correlate this with the PROCESSLIST view, if needed: ``` SELECT ID, USER, HOST, DB, INFO FROM information_schema.PROCESSLIST; ``` and match the text of `INFO` to the `QUERY` column from `WORKLOAD_MANAGEMENT_QUEUE`. ## Queueing Policy Queries are queued based on their size, as determined by the thresholds in the previous section, [Query Classification](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md). ## Small Queries Small queries are never queued. They use a negligible amount of leaf resources, which are not tracked. ## Medium Queries At regular intervals, the master aggregator collects statistics from the child aggregators. These statistics include the maximum resources used (threads per leaf, connections per leaf, and memory per leaf) during the interval and a rolling average of the maximum resources required during the previous intervals. At the end of each interval, the master aggregator uses the statistics gathered from the child aggregators to reallocate resources to each aggregator, if necessary. If the resource requirements for a new medium query exceed the total amount of resources allocated to all aggregators for all medium queries, the new query is queued in a local queue that is maintained by an individual aggregator. Each aggregator decides when to dequeue the queries in its local queue. Fifty percent of the total leaf resources are reserved for medium queries. ## Large Queries Large queries are ones whose estimated resource needs (threads, connections, or memory per leaf) exceed the upper thresholds for medium queries.Up to four large queries in the cluster will run concurrently. Any additional queries are queued in the global queue. Fifty percent of the total leaf resources are reserved for large queries. ## Local Queue vs Global Queue Workload Management uses two queues to control distributed queries. The local queue is a per query router that holds medium queries. The global queue is for large queries. Global queues ensure that a few heavy, fully distributed queries cannot monopolize leaf resources and starve the rest of the workload. It is a single cluster-wide queue that serializes large queries so that only a limited number run at once. ## Dynamic Reuse of WM Queues There are two queues created by Workload Manager: one for Medium queries and one for Large queries. WM dynamically moves queries from the Medium to the Large Queue when the Medium Queue is saturated and the Large Queue is not. There are two global variables to control this behavior: `workload_management_dynamic_resource_allocation` and `workload_management_queue_size_allow_upgrade` ## Errors and Warnings Queries return configurable errors if too many queries are queued or a query has been queued for too long. Queries also return configurable warnings when they were queued for a significant time relative to their actual execution time. This allows users to identify when their cluster resources are insufficient to meet the load of incoming queries. These errors and warnings are: `ER_TOO_MANY_QUEUED_QUERIES`: When the `workload_management_max_queue_depth` value is reached, SingleStore Helios will return this error instead of adding a new query to the queue. If you encounter this error, the maximum queue depth may need to be increased to accommodate the load. This error may also indicate that your cluster’s resources need to be increased or the load of incoming queries needs to be reduced. `ER_QUERY_QUEUE_TIMEOUT`: When a query has been queued longer than the `workload_management_queue_timeout` value, it will return this error and will be removed from the queue. This error indicates that your cluster’s resources or the load of incoming queries need to be adjusted to successfully process incoming queries. `ER_QUERY_QUEUED_WARNING`: When the ratio of time spent by a query in the queue versus the actual execution time of the query exceeds the `workload_management_queue_time_warning_ratio` value, the query will return this warning. The warning indicates the amount of time spent queued and the amount of time spent executing, to help you understand the sources of query latency. Encountering this warning is normal in many workloads, but if query latency is too high, your cluster’s resources may not be sufficient to process incoming queries in a timely manner. ## Observe the Status of Workload Management To see the current state of the workload management system, including the number of running and queued queries, and resource usage and thresholds, you can run the `SHOW WORKLOAD MANAGEMENT STATUS` command. You can also run the query `SELECT * FROM information_schema.MV_WORKLOAD_MANAGEMENT_STATUS;`, which will return the same information as the previous command, but per aggregator. The following example shows sample output from a small, lightly-loaded system. The units for the memory thresholds are in megabytes. ```sql SHOW WORKLOAD MANAGEMENT STATUS; ``` ```output +-------------------------------------------------------+-------+ | Stat | Value | +-------------------------------------------------------+-------+ | Queued Queries (from global queue) | 0 | | Queued Queries (from local queue) | 0 | | Total Queued Queries Since Startup | 0 | | Running Queries (from global queue) | 0 | | Running Queries (from local queue) | 0 | | Running Memory (MB) On Leaves (from global queue) | 0 | | Running Memory (MB) On Leaves (from local queue) | 0 | | Allocated Memory (MB) On Leaves (for local queue) | 6673 | | Required Memory (MB) On Leaves (from local queue) | 0 | | Avg Required Memory (MB) On Leaves (from local queue) | 0 | | Running Threads Per Leaf (from local queue) | 0 | | Allocated Threads Per Leaf (for local queue) | 2048 | | Required Threads Per Leaf (from local queue) | 0 | | Avg Required Threads Per Leaf (from local queue) | 0 | | Running Connections Per Leaf (from local queue) | 0 | | Allocated Connections Per Leaf (for local queue) | 2500 | | Required Connections Per Leaf (from local queue) | 0 | | Avg Required Connections Per Leaf (from local queue) | 0 | | Memory Threshold (MB) to Queue Locally | 66 | | Memory Threshold (MB) to Queue Globally | 3337 | | Connections Threshold to Queue Globally | 2500 | | Threads Threshold to Queue Globally | 2048 | +-------------------------------------------------------+-------+ ``` The following table provides a general description of some of the status fields. For information on the local queue and the global queue, see the [Query Classification](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager.md) section. | Status | Description | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Queued Queries | Total number of queries queued due to workload management on this aggregator | | Running Queries | Total number of queries in a queue, based on the number and resource intensity of other queries on this aggregator | | Running Memory On Leaves | Total estimated memory used per leaf from queries that are above/below“Memory Threshold to Queue Locally”/“Memory Threshold to Queue Globally”, which are run through this aggregator | | Running Threads Per Leaf | Total number of threads used per leaf from queries run through this aggregator | | Running Connections Per Leaf | Total number of connections established per leaf from queries, which are run through this aggregator | | Memory Threshold to Queue Locally | Estimated memory that a query needs to use before it is queued | | Memory Threshold to Queue Globally | Estimated memory that a query needs to use before it is queued and wait for multi-aggregator coordination to run | | Connections Threshold to Queue Globally | Total number of connections that a query takes before it is queued and wait for multi-aggregator coordination to run | | Threads Threshold to Queue Globally | Total number of threads that a query takes before it is queued and wait for multi-aggregator coordination to run | In addition, the `SHOW PROCESSLIST` command can be executed on any SingleStore node to obtain details about currently running queries. Also, `information_schema.processlist` or `information_schema.mv_processlist` can be queried to provide the same information but can be filtered for more focused results. The processlist provides information such as the user, host, database, command type, execution time, state, and additional information about each process. The information provided can help identify long-running or hanging queries, monitor query execution, and analyze query behavior. If you haven't received any of the errors listed above and you still have available memory or CPU, see the [Troubleshooting Poorly Performing Queries](https://docs.singlestore.com/cloud/query-data/query-tuning/troubleshooting-poorly-performing-queries.md) page for potential steps to take to resolve the issue. *** Modified at: April 16, 2026 Source: [/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager/](https://docs.singlestore.com/cloud/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/use-the-workload-manager/) (An index of the documentation is available at /llms.txt)