Moving Data Across Clusters

Move a Table or a Subset of Data (ETL via Object Storage)

This method involves the following steps:

1. Export from the source cluster group: From a cluster in the source cluster group, write the table to S3/GCS/Azure Blob using SELECT ... INTO. For example:

   SQL

   Copy

   SELECT * FROM src_db.src_table
     INTO S3 's3://my-bucket/my-prefix/'
     CONFIG '{...}'
     CREDENTIALS '{...}';

   Variants exist for INTO AZURE, INTO GCS, etc. for details refer SELECT

2. Re-create the schema in target cluster group: In a cluster in the destination cluster group, create the destination table with matching DDL.

3. Load into target cluster group: In the destination cluster group, use a Pipeline (recommended) or LOAD DATA from that bucket to load into the new table.

   SQL

   Copy

   CREATE PIPELINE p_move_table AS
     LOAD DATA S3 's3://my-bucket/my-prefix/'
     CONFIG '{...}'
     CREDENTIALS '{...}'
     INTO TABLE dest_db.dest_table;
   
   START PIPELINE p_move_table;

   For details on options available refer:

   CREATE PIPELINE

   Load Data with SingleStore Flow on Helios

   LOAD DATA

Move an Entire Database

If you want to move a whole database across cluster groups, you can:

Run BACKUP DATABASE from a cluster in the source cluster group to object storage.

Run RESTORE DATABASE from a cluster in the destination cluster group, pointing at that backup.