About SingleStore Pipelines

A SingleStore pipeline is a mechanism for continuously loading data into SingleStore from external sources including Apache Kafka, Amazon S3, Azure Blob Storage, Google Cloud Storage, and the local file system. Pipelines can extract, shape (modify), and load external data without the need for additional third-party tools. Pipelines are robust, scalable, highly performant, and support fully distributed workloads.

The following features make pipelines a powerful alternative to third-party ETL middleware in many scenarios:

• Easy continuous loading: Pipelines monitor their source folder or Kafka queue and, when new files or messages arrive, automatically load them. This simplifies the job of the application developer.

• Scalability: Pipelines inherently scale with SingleStore clusters as well as distributed data sources like Kafka and cloud data stores like Amazon S3.

• High performance: Pipelines data is loaded in parallel from the data source directly to the SingleStore leaf nodes, in most situations; this improves throughput by bypassing an aggregator. Additionally, pipelines have been optimized for low lock contention and concurrency.

• Exactly-once semantics: The architecture of pipelines ensures that transactions are processed exactly once, even in the event of failover.

• Debugging: Pipelines makes it easier to debug each step in the ETL process by storing exhaustive metadata about transactions, including stack traces and stderr messages.

• Concurrency: Multiple pipelines can insert data into a single table. This ability is similar to using multiple write queries. See Sync Variables Lists for more information.

• Backup: Database backups preserve the state of all pipelines (offsets, etc.) in that database. When a backup is restored, all pipelines in that database will revert to the state (offsets, etc.) they were in when the target backup was generated.

Pipelines support Avro, CSV, JSON, and Parquet.