Flow on Helios Sizing Guidelines

Overview

Sizing guidelines help users make decisions on selecting the right Flow instance size on SingleStore Helios for their migration or data movement jobs.

Size a Workload

When selecting a Flow instance, evaluate your workload carefully. Consider the following key parameters:

  1. Data Size

    • The total volume of data that needs to be migrated or streamed.

    • Includes both historical data and incremental changes.

    • Larger datasets generally require larger Flow instances to achieve migration in a reasonable timeframe.

  2. Number of Tables

    • The count of tables involved in the job/pipeline.

    • A higher number of tables increases concurrency requirements and can benefit from larger Flow instances.

  3. Size of the Biggest Table

    • The largest table in your workload may become a bottleneck.

    • If the largest table is very large (hundreds of GBs or TBs), it may dictate the minimum Flow size you need.

  4. Number of Threads on Source Database

    • Determines the maximum parallelism you can achieve when reading from the source.

    • If your source can’t provide enough threads, increasing Flow instance size may not help.

  5. Estimated Time to Complete the Job

    • The timeframe in which you want the migration to finish (e.g., overnight, within a maintenance window, or real-time).

    • Tighter SLAs require more compute and memory resources.

Sizing Examples

Workload Type

Data Size

Number of Tables

Biggest Table Size

Threads Available

Completion Time Target

Suggested Flow Instance

Small

~50 GB

10

~8 GB

4

12 - 24 hours

f - 1

Medium

~500 GB

50 - 75

~100 GB

8 - 12

6 - 8 hours

f - 3

Large

~2 TB

200+

~400 GB

16

~6 hours

f - 6

Very Large/Enterprise

~10 TB

500+

~1 TB

32+

< 4 hours (tight SLA)

f - 8

Latency Sensitive/ Real-time Sync

~300 GB/day (ongoing stream)

20 - 30

~200 GB (growing daily)

12 - 16

Continuous / < 1 min lag

f - 4

Additional Notes

  • Thread Utilization on Source Database Server

    Increasing the number of threads used on the source database server proportionally increases the load on system resources (CPU, memory, I/O). Proper sizing and monitoring are recommended to prevent performance degradation.

  • Completion Time Target

    The Completion Time Target represents an estimated duration derived from internal performance testing. Actual completion times may vary depending on the source database server's ability to transmit records at the required rate.

  • Data Flow Behavior

    During the initial data load, the system performs a bulk load operation in a sequential manner to ensure data consistency and reliability.

  • Handling Large Tables

    For very large tables, SingleStore recommends using XL Ingest to optimize throughput and reduce load time.

Last modified:

Was this article helpful?