Data Shaping with Pipelines
After data is extracted from a SingleStore Pipeline’s data source, it can be optionally shaped (modified). After shaping, the pipeline loads the data into one or more destination tables in SingleStore.
Some common data shaping operations that can be performed are:
Lookups from other SingleStore tables (in addition to the destination table(s))
Normalizing data
Denormalizing data
Adding computed columns
Filtering data (excluding specific columns or records)
Mapping data values from the data source to new values
Splitting records from the data source into multiple destination tables
Adding surrogate keys
Data modifications made during shaping are not written back to the data source, unless done explicitly in a transform (SingleStoreDB Self-Managed only).
Ways to specify data shaping logic:
In a
CREATE PIPELINE
statement.In a stored procedure that is called from the pipeline.
In a transform that is called from the pipeline.