Methods for Data Shaping with Pipelines

The details of each data shaping method are explained in the following table.

Data Shaping Method	Amount of Customization Logic Allowed	Ease of Use	Comments	Examples
In a `CREATE PIPELINE` statement	Low	Easiest	Pros: Generally, runs the fastest of the three data shaping methods; transactional guarantees. Cons: Limited to SQL WHERE CLAUSE to perform filtering and SQL expressions to perform other shaping logic; can only load data into one target table; cannot perform lookups in other tables; a single row in the data source cannot generate multiple rows in the target table.	`CREATE PIPELINE` uses the `LOAD DATA` clause to perform data shaping. See examples in the LOAD DATA topic.
Pipeline Stored Procedure	Medium	More Difficult	Pros: Transactional guarantees; cons of specifying data shaping logic directly in your `CREATE PIPELINE` statement don’t apply. Cons: Limited to the capabilities of the PSQL language; cannot use third-party libraries.	See examples in CREATE PIPELINE ... INTO PROCEDURE
Transform	High	Most Difficult	Pros: Can use any nearly any programming language and leverage third-party libraries. Cons: Not available in SingleStoreDB Cloud; runs slower than stored procedures for most use cases; deployment could be challenging; cannot insert into multiple tables; transactional guarantees apply to data written to `stdout`, only (refer to the guide in the next column for details).	See the guide Writing a Transform to Use with a Pipeline

Was this article helpful?