# Pipeline Concepts

## Create a Pipeline

A core `CREATE PIPELINE` command contains:

* A **Pipeline Declaration** with the pipeline name.
* A **Data Source Specification** based on where the data is stored (AWS S3, Azure, Kafka, etc.).
* A **Data File Mapping** which maps the data file to a SingleStore table and which is based on the data file format.

Below is a simple pipeline that loads a CSV file from Amazon S3.

![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/bltb6a9c86624fcd231/6a2c4367dc08991c872f2408/pipeline_concepts_figure1-XATCgH.png)

In the pipeline above, the structure of the Data Source Specification is determined by the location of the data file, Amazon S3 in this example. The structure of the Data File Mapping is determined by the data file format, CSV in this example.

The following pipeline loads a Parquet file from Amazon S3.

![](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blt2281c5a407bd8bc3/6a2c42af464cc2c406852efc/pipeline_concepts_figure2-yts0BE.png)

In this pipeline, the Data Source Specification is the same as in the pipeline above (except for the file name), but the Data File Mapping is different as both the file format and the data file format are different (CSV in the prior example, Parquet in this example).

Refer to [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md) for a full description of the `CREATE PIPELINE` command.

## Supported Data Stores

Pipelines support the following data stores:

* Amazon S3 - [S3 Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-e9b729f5-e821-593f-4780-33e798409694.md)
* Google Cloud Services - [GCS Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-157713ba-2165-ec8f-6291-47e4c3e65737.md)
* HDFS - [HDFS Pipeline Syntax (Version 2.2.x or newer)](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a8df0412-4c5b-75f4-e6ce-753934798422.md)
* Kafka - [Kafka Pipeline Syntax (Version 0.8.2.2 or newer)](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-4605223c-017e-44d6-7b51-ec3b59137e07.md)
* Link - [Creating a Pipeline Using a Connection Link](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a3b0c2e9-827c-3e03-4fd6-ea2c57d954f1.md)
* Microsoft Azure - [Azure Blob Pipeline Syntax](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#UUID-a57594c8-138d-3bb1-d25b-aebef5185c17.md)
* MongoDB - [Replicate MongoDB® Collections using SQL](https://docs.singlestore.com/cloud/reference/singlestore-kai/replicate-mongodb-collections-to-singlestore/replicate-mongodb-collections-using-sql.md)
* MySQL - [Replicate Data from MySQL](https://docs.singlestore.com/cloud/load-data/data-sources/replicate-data-from-mysql.md)

## Supported Data File Formats

* Avro - [Kafka Pipeline Using Avro Format](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#section-idm234619121847915.md)
* CSV - [CREATE PIPELINE](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline.md)
* Iceberg - [Create an Iceberg Pipeline](https://docs.singlestore.com/cloud/load-data/data-sources/iceberg-ingest.md)
* JSON - [Kafka Pipeline Using JSON Format](https://docs.singlestore.com/cloud/reference/sql-reference/pipelines-commands/create-pipeline/#section-idm4529487879993633828365570841.md)
* Parquet - [Create a Parquet Pipeline](https://docs.singlestore.com/cloud/load-data/load-data-from-files/load-data-from-parquet-files/create-a-parquet-pipeline.md)

## In this section

* [The Lifecycle of a Pipeline](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/the-lifecycle-of-a-pipeline.md)
* [Pipelines Scheduling](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipelines-scheduling.md)
* [Parallelized Data Extraction with Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/parallelized-data-extraction-with-pipelines.md)
* [Data Shaping with Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/data-shaping-with-pipelines.md)
* [Aggregator Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/aggregator-pipelines.md)
* [Schema and Pipeline Inference](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/schema-and-pipeline-inference.md)
* [Pipeline Built-in Functions](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipeline-built-in-functions.md)
* [Pipeline Retry Options](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/pipeline-retry-options.md)
* [Writing Efficient Stored Procedures for Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/writing-efficient-stored-procedures-for-pipelines.md)
* [Best Practices for Pipelines](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/best-practices-for-pipelines.md)

***

Modified at: February 18, 2026

Source: [/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/](https://docs.singlestore.com/cloud/load-data/about-singlestore-pipelines/pipeline-concepts/)

(An index of the documentation is available at /llms.txt)