Data Ingestion

SingleStore can load data continuously or in bulk from a variety of sources. Popular loading sources include: files, a Kafka cluster, cloud repositories like Amazon S3, HDFS, or from other databases. As a distributed system, SingleStore ingests data streams using parallel loading to maximize throughput.

SingleStore Pipelines is an easy-to-use built-in capability that extracts, transforms, and loads external data using sources such as Kafka, S3, Azure Blob, and filesystems. It is ideal for scenarios where data from a supported source must be ingested and processed in real-time. This makes SingleStore Pipelines a good alternative to third-party middleware for basic ETL operations that must be executed as fast as possible, thus eliminating traditional long-running processes such as overnight batch jobs.

For bulk data load operations, SingleStore offers a LOAD DATA function that imports files in parallel to maximize performance.

For more information, see:

Last modified: November 22, 2022

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

(Optional) Run the following command to view the associated signature files.
Shell
```
curl undefined
```
Download the signature file from the SingleStore release server.
- Option 1: Click the Download Signature button next to the SingleStore file.
- Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.
- Option 3: Run the following command to download the signature file.
  Shell
```
curl -O undefined
```

After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

Shell

echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
      --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
      --bundle undefined \
      --new-bundle-format -

Verified OK