EXTRACT PIPELINE … INTO OUTFILE
On this page
This command takes a sample of the data streaming into your pipeline and copies it into a file on disk.
EXTRACT PIPELINE pipe_line[FROM 'source_partition'[OFFSETS start_offset TO end_offset]]INTO OUTFILE 'file_name'
pipe_is the configured pipeline.
file_the output file containing your sample data.
source_is a source partition ID.
end_can be used to extract the exact range of sample data.
This command causes implicit commits.
See COMMIT for more information.
Refer to the Permission Matrix for the required permission.
You cannot run
EXTRACT PIPELINE when the pipeline is in a
A file containing transform data that can be used during debugging operations.
cat sample_output | python transform.py
The following saves random sample data.
EXTRACT PIPELINE p INTO OUTFILE 'transform_output';
The following is useful if there is a specific partition or file with a known problem.
EXTRACT PIPELINE p FROM '6' INTO OUTFILE 'transform_output';
The following extracts an exact range of data, which is useful if the problematic data is in a specifically known kafka region.
EXTRACT PIPELINE p FROM '10' OFFSETS 0 TO 6 INTO OUTFILE 'transform_output';
Last modified: April 6, 2023