Kafka Pipeline Using JSON Format

Kafka pipelines can use JSON formatted data for ingest. The minimum required syntax is below:

SQL

CREATE PIPELINE <pipeline_name> AS
LOAD DATA KAFKA '<host.example.com/my-topic>'
INTO TABLE <table_name>
FORMAT JSON;

To skip errors without skipping the entire batch, use the SKIP PARSER ERRORS clause.

SQL

CREATE PIPELINE <pipeline_name> AS
LOAD DATA KAFKA '<host.example.com/my-topic>'
INTO TABLE <table_name>
SKIP PARSER ERRORS 
FORMAT JSON;

Important

The use of SKIP PARSER ERRORS along with WITH TRANSFORM is not supported. Using both clauses will cause an error when creating or altering a pipeline.

If the SKIP PARSER ERRORS clause is added during create pipeline, the pipelines_stop_on_error configuration will be ignored.

The warning message will be logged in the pipelines_errors table, adding the Kafka message in the LOAD_DATA_LINE column.
The entire Kafka message which has the parse error will be logged.
Skip the message and move to the next Kafka message.

If the SKIP PARSER ERRORS clause is not added during create pipeline, the default behavior will remain the same.

Error messages will be logged in the pipelines_errors table, adding the Kafka message in the LOAD_DATA_LINE column.
The pipeline will stop or skip the current batch and move to the next batch based on whether the pipelines_stop_on_error variable is set to true or false.
The engine variable pipelines_parse_errors_threshold will stop or skip the current batch if the number of errors exceeds the threshold value. This depends on whether the pipelines_stop_on_error variable is set to true or false.

Note

See PIPELINES_BATCHES for more details about batches.

Kafka Pipeline Using JSON Format

Was this article helpful?

Was this article helpful?