Important

The SingleStore 9.0 release candidate (RC) gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.

Flow on Helios Sizing Guidelines

Overview

Sizing guidelines help users make decisions on selecting the right Flow instance size on SingleStore Helios for their migration or data movement jobs.

Size a Workload

When selecting a Flow instance, evaluate your workload carefully. Consider the following key parameters:

Data Size
- The total volume of data that needs to be migrated or streamed.
- Includes both historical data and incremental changes.
- Larger datasets generally require larger Flow instances to achieve migration in a reasonable timeframe.
Number of Tables
- The count of tables involved in the job/pipeline.
- A higher number of tables increases concurrency requirements and can benefit from larger Flow instances.
Size of the Biggest Table
- The largest table in your workload may become a bottleneck.
- If the largest table is very large (hundreds of GBs or TBs), it may dictate the minimum Flow size you need.
Number of Threads on Source Database
- Determines the maximum parallelism you can achieve when reading from the source.
- If your source can’t provide enough threads, increasing Flow instance size may not help.
Estimated Time to Complete the Job
- The timeframe in which you want the migration to finish (e.g., overnight, within a maintenance window, or real-time).
- Tighter SLAs require more compute and memory resources.

Sizing Examples

Workload Type	Data Size	Number of Tables	Biggest Table Size	Threads Available	Completion Time Target	Suggested Flow Instance
Small	~50 GB	10	~8 GB	4	12 - 24 hours	f - 1
Medium	~500 GB	50 - 75	~100 GB	8 - 12	6 - 8 hours	f - 3
Large	~2 TB	200+	~400 GB	16	~6 hours	f - 6
Very Large/Enterprise	~10 TB	500+	~1 TB	32+	< 4 hours (tight SLA)	f - 8
Latency Sensitive/ Real-time Sync	~300 GB/day (ongoing stream)	20 - 30	~200 GB (growing daily)	12 - 16	Continuous / < 1 min lag	f - 4

Additional Notes

Thread Utilization on Source Database Server

Increasing the number of threads used on the source database server proportionally increases the load on system resources (CPU, memory, I/O). Proper sizing and monitoring are recommended to prevent performance degradation.
Completion Time Target

The Completion Time Target represents an estimated duration derived from internal performance testing. Actual completion times may vary depending on the source database server's ability to transmit records at the required rate.
Data Flow Behavior

During the initial data load, the system performs a bulk load operation in a sequential manner to ensure data consistency and reliability.
Handling Large Tables

For very large tables, SingleStore recommends using XL Ingest to optimize throughput and reduce load time.