The Lifecycle of a Pipeline
The user creates a new pipeline using CREATE PIPELINE.
The user starts the pipeline using START PIPELINE.
Note
Steps 3 to 8 refer to a batch, which is subset of data that the pipeline extracts from its data source. These steps comprise one batch operation, which will succeed or fail completely. If any step fails, the batch operation rolls back.
The pipeline extracts a batch from its data source. The pipeline's offsets are updated to reflect the current position in the data source.
The pipeline optionally shapes (modifies) the batch, using one of three methods.
If the pipeline is able to successfully process the batch, the pipeline loads the batch into one or more SingleStore tables.
If an error occurs while a batch
b
is running, thenb
will fail andb
's transaction rolls back. Thenb
is retried at mostpipelines_max_retries_per_batch_partition
times. If all of the retries are unsuccessful andpipelines_stop_on_error
is set toON
, the pipeline stops. Otherwise, the pipeline continues and processes a new batchnb
,which processes the same files or objects thatb
attempted to process, excluding any files or objects that may have caused the error.For more information, see View and Handle Pipeline Errors.
The pipeline updates the
FILE_STATE
column in theinformation_schema.PIPELINES_FILES
table, as follows:Files and objects in the batch that the pipeline processed successfully are marked as
Loaded
.Files and objects in the batch that the pipeline did not process successfully, after all retries are unsuccessful (as described in step 6), are marked as
Skipped
.
A file or object that is marked as
Loaded
orSkipped
will not be processed again by the pipeline, unless ALTER PIPELINE ... DROP FILE ... is run.The pipeline does not delete files nor objects from the data source.
The pipeline checks if the data source contains new data. If it does, the pipeline processes another batch immediately by running steps 3 to 7 again. If the data source does not contain more data, the pipeline waits for
BATCH_INTERVAL
milliseconds (which is specified in theCREATE PIPELINE
statement) before checking the data source for new data. If the pipeline finds new data at this point, the pipeline runs steps 3 to 7 again.
Note
The user can stop a running pipeline using STOP PIPELINE. If this command is executed while a batch operation is executing, the batch operation completes before the pipeline stops.
During a pipeline's lifecycle, the pipeline updates the pipelines tables in the information schema, at different times. Other than the update of the information_schema.PIPELINES_FILES
table mentioned in step 7, all of the other updates are not discussed here.