Pipeline Concepts

Create a Pipeline

A core CREATE PIPELINE command contains:

  • A Pipeline Declaration with the pipeline name.

  • A Data Source Specification based on where the data is stored (AWS S3, Azure, Kafka, etc.).

  • A Data File Mapping which maps the data file to a SingleStore table and which is based on the data file format.

Below is a simple pipeline that loads a CSV file from Amazon S3.

In the pipeline above, the structure of the Data Source Specification is determined by the location of the data file, Amazon S3 in this example. The structure of the Data File Mapping is determined by the data file format, CSV in this example.

The following pipeline loads a Parquet file from Amazon S3.

In this pipeline, the Data Source Specification is the same as in the pipeline above (except for the file name), but the Data File Mapping is different as both the file format and the data file format are different (CSV in the prior example, Parquet in this example).

Refer to CREATE PIPELINE for a full description of the CREATE PIPELINE command.

Supported Data Stores

Pipelines support the following data stores:

Supported Data File Formats

In this section

Last modified: May 8, 2025

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK