Load Data with Pipelines

This part of the tutorial shows how to ingest stock trades data from a public AWS S3 bucket into the SingleStore database using pipelines.

1. Run the following SQL commands to create the pipelines:

   SQL

   Copy

   USE trades;
   
   CREATE OR REPLACE PIPELINE company
   AS LOAD DATA S3 's3://singlestore-docs-example-datasets/trades/company.csv'
   CONFIG '{"region":"us-east-1"}'
   SKIP DUPLICATE KEY ERRORS
   INTO TABLE company;
   
   CREATE OR REPLACE PIPELINE trade
   AS LOAD DATA S3 's3://singlestore-docs-example-datasets/trades/trade.csv'
   CONFIG '{"region":"us-east-1"}'
   SKIP DUPLICATE KEY ERRORS
   INTO TABLE trade;

2. Run the following SQL commands to start the pipelines:

   SQL

   Copy

   USE trades;
   START ALL PIPELINES;

Once the Success message is returned for all the created pipelines, SingleStore starts ingesting the data from the S3 bucket.