Additional CREATE PIPELINE Examples

Pipelines may be created using the SKIP, IGNORE, or REPLACE clauses. These clauses can be used alone or in combination with each other.

SKIP

The SKIP ... ERRORS behavior lets you specify an error scenario that, when encountered, discards an offending row. See SKIP ALL ERRORS Example for more information.

Examples

CREATE PIPELINE <pipeline_name>  AS LOAD DATA <file_name> SKIP PARSER ERRORS INTO TABLE <table_name>;
CREATE PIPELINE <pipeline_name>  AS LOAD DATA <file_name> SKIP PARSER ERRORS REPLACE INTO TABLE<table_name>;
CREATE PIPELINE <pipeline_name>  AS LOAD DATA <file_name> SKIP CONSTRAINT ERRORS INTO TABLE <table_name>;
 
CREATE PIPELINE <pipeline_name>  AS LOAD DATA <file_name> SKIP ALL ERRORS INTO TABLE <table_name>;

IGNORE

Unlike SKIP ... ERRORS which discards offending rows, IGNORE may change the inserted row’s data to ensure that it adheres to the table schema. See IGNORE Error Handling

Examples

CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> IGNORE INTO TABLE <table_name>;
CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> IGNORE PARSER ERRORS INTO TABLE <table_name>;
CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> IGNORE PARSER ERRORS REPLACE INTO TABLE <table_name>;

REPLACE

If the REPLACE clause is specified, existing rows in the table are replaced with the ingested row for each matching PRIMARY KEY or UNIQUE index.

REPLACE may also be used with IGNORE and SKIP. However, REPLACE would come after the SKIP or IGNORE clause.

Examples

CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> REPLACE INTO TABLE <table_name>;
CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> SKIP ALL ERRORS REPLACE INTO TABLE <table_name>;
CREATE PIPELINE <pipeline_name> AS LOAD DATA <file_name> SKIP CONSTRAINT ERRORS INTO TABLE <table_name>;
CREATE PIPELINE p AS LOAD DATA <file_name> SKIP CONSTRAINT ERRORS REPLACE INTO <pipeline_name>;

Important

  • SKIP CONSTRAINT ERRORS  and  SKIP DUPLICATE KEY ERRORS  are unsupported with pipelines into stored procedures.

  • IGNORE, SKIP PARSER ERRORS, and  SKIP ALL ERRORS  are unsupported with non-CSV pipelines.

  • REPLACE, SKIP CONSTRAINT ERRORS, and SKIP DUPLICATE KEY ERRORS are supported with non-CSV pipelines.

Last modified: July 9, 2024

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK