Pipeline Errors
On this page
Troubleshoot Pipelines
Concepts
This topic requires an understanding of pipeline batches, which are explained in The Lifecycle of a Pipeline.
View pipeline errors
The Data Ingest provide information about pipeline errors that have occurred.
Query the information_ schema. PIPELINES_ ERRORS
table
You can run the following query to show all errors that have occurred, per database, per pipeline, per batch, and per partition.
SELECT DATABASE_NAME, PIPELINE_NAME, BATCH_ID, PARTITION, BATCH_SOURCE_PARTITION_ID,ERROR_KIND, ERROR_CODE, ERROR_MESSAGE, LOAD_DATA_LINE_NUMBER, LOAD_DATA_LINEFROM information_schema.PIPELINES_ERRORS;
Query files that were skipped
The query in the previous section does not show files that were skipped because they had errors.
SELECT * FROM information_schema.PIPELINES_FILES WHERE FILE_STATE = 'Skipped';
If you need additional information, such as the database, the partition, the error that was generated and the line of the error file or object that caused the issue, run the following query.
SELECT pe.DATABASE_NAME, pe.PIPELINE_NAME, pe.BATCH_ID, pe.PARTITION,pe.BATCH_SOURCE_PARTITION_ID, pe.ERROR_TYPE, pe.ERROR_KIND, pe.ERROR_CODE, pe.ERROR_MESSAGE,pe.LOAD_DATA_LINE_NUMBER, pe.LOAD_DATA_LINEFROM information_schema.PIPELINES_FILES pf, information_schema.PIPELINES_ERRORS peWHERE pe.BATCH_SOURCE_PARTITION_ID = pf.FILE_NAME and pf.FILE_STATE = 'Skipped';
Address specific errors
The following table lists errors that can occur when running a pipeline statement, such as CREATE PIPELINE
, and errors that can occur while a pipeline is extracting, shaping, and loading data.
Error |
Resolution |
---|---|
You get a syntax error when running |
Both |
You receive error |
The master aggregator can likely not connect to the pipeline's data source. |
|
The bucket name is case-sensitive. |
Error |
This error can occur when a pipeline attempts to run a transform. 1. 2. 3. |
|
An incorrect path to the transform was likely specified. |
Error: |
This error may occur when the default value (8MB) for the engine variable |
A parsing error occurs in your transform. |
To debug your transform, you can run |
Error: An error that isn't associated with any specific source partition of the pipeline occurred during the batch loading process. The whole batch will be failed.
Issue
The batch loading process was able to load the data from the source, but it failed to ingest the data into the SingleStore database.
Solution
Address the secondary error to solve the issue.
For example, consider the following error:
Error,2790,"An error that isn't associated with any specific source partition of the pipeline
occurred during the batch loading process. The whole batch will be failed.
Error 1205 : ""Leaf Error (svchost:3306): Lock wait timeout exceeded; try restarting transaction.
Unique key Row Value lock owned by connection id xxxx, query `open idle transaction`"""
In this case, the pipelines failed because the query was unable to acquire row locks for ingesting data.
Rename a table referenced by a pipeline
When trying to rename a table that is referenced by a pipeline the following error will result:
ERROR 1945 ER_CANNOT_DROP_REFERENCED_BY_PIPELINE: Cannot rename table because it is referenced by pipeline <pipeline_name>
The following sequence demonstrates how to rename a pipeline referenced table:
-
Save your pipeline settings:
SHOW CREATE PIPELINE <pipeline_name> EXTENDED; -
Stop the pipeline:
STOP PIPELINE <pipeline_name>; -
Drop the pipeline:
DROP PIPELINE <pipeline_name>; -
Change the name of the table:
ALTER TABLE <old_table_name> RENAME <new_table_name>; -
Recreate the pipeline with the settings obtain in step 1 and change the table name to reflect the new table name.
-
Start the pipeline:
START PIPELINE <pipeline_name>;
Pipeline errors that are handled automatically
Typical error handing scenario
In most situations, an error that occurs while a pipeline is running is handled in this way:
If an error occurs while a batch b
is running, then b
will fail and b
's transaction rolls back.b
is retried at most pipelines_
times.pipelines_
is set to ON
, the pipeline stops.nb
,which processes the same files or objects that b
attempted to process, excluding any files or objects that may have caused the error.
The following table lists events, which may or may not cause errors, and how the events are handled.
Event |
How the Event is Handled |
---|---|
The pipeline cannot access a file or object. |
The typical error handling scenario (mentioned earlier in this topic) applies.
|
The pipeline cannot read a file or object because it is corrupted. |
The typical error handling scenario (mentioned earlier in this topic) applies.
After fixing the issue with the corrupted file/object, you can have the pipeline reprocess the file/object by running |
A file or object is removed from the filesystem after the batch has started processing the file/object. |
The batch does not fail; the file or object is processed. |
A file is removed from the filesystem (or an object is removed from an object store) after the pipeline registers the file/object in |
The typical error handling scenario (mentioned earlier in this topic) applies.
|
The cluster restarts while the batch is being processed. |
The typical error handling scenario (mentioned earlier in this topic) applies. Once the cluster is online, |
A leaf node is unavailable before the pipeline starts. |
This does not cause the pipeline to fail. |
A leaf node fails while the pipeline is running. |
The batch fails. |
An aggregator fails while the pipeline is running |
The batch fails. |
The pipeline reaches the allocated storage space for errors. |
The pipeline pauses. |
Additional Information
Refer to the Pipeline Summary and Pipeline Performance dashboards for more information regarding troubleshooting pipeline errors and performance issues.
In this section
Last modified: July 29, 2024