Load Data with SingleStore Flow

Overview

SingleStore Flow (or, "Flow") is SingleStore's data migration and change data capture (CDC) solution which is comprised of two primary components:

SingleStore Ingest, also referred to as "Ingest," transfers the schema for all tables, moves table data for tables up to 10GB in size, and handles ongoing CDC for all tables into SingleStore.
SingleStore XL Ingest, also referred to as "XL Ingest," handles the initial transfer of large tables by breaking them into smaller logical partitions. It then copies multiple partitions from the source to the target in parallel. This helps transfer large tables efficiently, and is resilient to network outages and restarts of Flow, the source, or the destination.

Flow supports the following source databases for transfer to SingleStore:

Key benefits of using Flow:

No-code setup: Configure end-to-end migration via the Flow UI.
Automatic schema handling: Analyze the source database and automatically create the required schema in the target SingleStore database.
Real-time visibility: Monitor progress, view live logs, and troubleshoot quickly.
High throughput: Use XL Ingest to parallelize large table loads by chunking source data, significantly reducing migration time.

Flow is available in two deployment options:

Both provide the same data migration and CDC features, but differ in how they are deployed and used.

The following table outlines these two options:

Deployment Option	Self-Managed Flow Application	Flow on Helios
Description	Installed and operated in your own environment. Self-managed, installation required. Provides control over where the application runs and how it integrates with your system.	Runs in the SingleStore Helios cloud deployment. Fully managed by SingleStore, no installation required. A Flow instance is provisioned directly from the UI, and usage is billed only while the instance is running.
When to Use	When deployment placement or control over the environment matters the most, such as replicating large volumes of data from an on-premises source to SingleStore. This is because Flow performs best for high-volume transfers when its services run close to the data source.	When your data sources and destinations are in the same cloud region, along with your Flow on Helios instance, ensuring excellent performance along with convenience and robustness. When convenience and low operational overhead matter more than placement.

For more details on Flow on Helios, refer to Load Data with SingleStore Flow on Helios.