Debugging Pipeline Errors

Viewing pipeline errors

The Data Ingest provide information about pipeline errors that have occurred. Some useful queries against these tables are provided in this section.

SELECT DATABASE_NAME, PIPELINE_NAME, BATCH_ID, PARTITION, BATCH_SOURCE_PARTITION_ID,
ERROR_KIND, ERROR_CODE, ERROR_MESSAGE, LOAD_DATA_LINE_NUMBER, LOAD_DATA_LINE
FROM information_schema.PIPELINES_ERRORS;

SELECT * FROM information_schema.PIPELINES_FILES WHERE FILE_STATE = 'Skipped';

SELECT pe.DATABASE_NAME, pe.PIPELINE_NAME, pe.BATCH_ID, pe.PARTITION,
pe.BATCH_SOURCE_PARTITION_ID, pe.ERROR_TYPE, pe.ERROR_KIND, pe.ERROR_CODE, pe.ERROR_MESSAGE,
pe.LOAD_DATA_LINE_NUMBER, pe.LOAD_DATA_LINE
FROM information_schema.PIPELINES_FILES pf, information_schema.PIPELINES_ERRORS pe
WHERE pe.BATCH_SOURCE_PARTITION_ID = pf.FILE_NAME and pf.FILE_STATE = 'Skipped';

Use CREATE PIPELINE… FORCE to Collect More Logs

When the CREATE PIPELINE statement fails, more information can be found with the SHOW WARNINGS command.

If SHOW WARNINGS does not reveal enough information, the pipeline can be created using the FORCE option and the errors can be fixed later. This will create a pipeline regardless of the connection error and store the errors in the information_schema.pipelines_errors table.

CREATE PIPELINE <pipeline_name> FORCE AS LOAD DATA <data_source>
  INTO TABLE <table_name>;

Use SELECT * to view the pipeline errors.

SELECT * FROM information_schema.pipelines_errors;

| DATABASE_NAME | PIPELINE_NAME | ERROR_UNIX_TIMESTAMP | ERROR_TYPE | ERROR_CODE | ERROR_MESSAGE|

Logging to Show More Information with Failures

Some data sources have debug logging availability which provides additional information along with the errors.

Use the following engine variable and command in the order they appear:

• SET GLOBAL pipelines_extractor_debug_logging=ON - this engine variable will set debug logging for pipelines if available.

• FLUSH EXTRACTOR POOLS - Flush extractor pools to remove any cached extractors.

Larger errors in the information_schema.pipelines_errors table will show more context about the cause of the failure.

Debug Running Pipelines

There are six tables in the information_schema database that relate to pipelines:

• PIPELINES - stores high-level information about any pipelines.

• PIPELINES_BATCHES_SUMMARY - contains high-level information about individual batches as they are loaded into the database.

• PIPELINES_BATCHES - contains detailed, low-level information about individual batches as they are loaded into a database.

• PIPELINES_ERRORS - contains detailed information about errors that occurred during extraction, transformation, or loading. Each row represents a single error event.

• PIPELINES_CURSORS - contains information about a pipeline’s offset ranges.

• PIPELINES_FILES - stores information about files that have been extracted from a file system-like data source, such as Amazon S3.