Data Shaping with Pipelines

After data is extracted from a SingleStore Pipeline’s data source, it can be optionally shaped (modified). After shaping, the pipeline loads the data into one or more destination tables in SingleStore.

Some common data shaping operations that can be performed are:

• Lookups from other SingleStore tables (in addition to the destination table(s))

• Normalizing data

• Denormalizing data

• Adding computed columns

• Filtering data (excluding specific columns or records)

• Mapping data values from the data source to new values

• Splitting records from the data source into multiple destination tables

• Adding surrogate keys

Data modifications made during shaping are not written back to the data source, unless done explicitly in a transform (SingleStoreDB Self-Managed only).

Ways to specify data shaping logic:

• In a CREATE PIPELINE statement.

• In a stored procedure that is called from the pipeline.