Verify Pipeline Success

The entire loading process takes around four minutes, but you do not need to wait for Pipelines to finish before querying. You can query the data as soon as you have started the loading process; this is part of the powerful functionality of Pipelines.

If you want to know whether the pipeline has finished loading, you can check the progress of a pipeline via the pipelines_files Information Schema table. For example, to see whether the "lineitem" pipeline has finished:

SELECT * FROM information_schema.pipelines_files WHERE pipeline_name = "tpch_100_lineitem";

You can also run your pipeline in the foreground, which makes it easy to verify that all the data has been loaded:

START PIPELINE <pipeline_name> FOREGROUND;

On the next page, we will show you how to get business-critical metrics.