PIPELINES_BATCHES_METADATA

DATABASE_NAME

The name of the database associated with the batch.

PIPELINE_ID

The internal unique identifier for the pipeline. Values of PIPELINE_ID are unique in this table.

PIPELINE_NAME

The name of the pipeline associated with the batch.

BATCH_ID

The internal unique identifier for the batch. Values of BATCH_ID are unique in this table.

BATCH_STATE

Specifies whether the batch transaction is successful. Possible values: Succeeded, In Progress, Failed, Queued, Cancelled, or No Data.

Succeeded: The batch is successfully loaded into the destination table.

In Progress: The batch is currently being loaded into the destination table.

Failed: The batch failed to be written. Batch failures are written as errors in the information_schema.PIPELINES_ERRORS table. Use the BATCH_ID to correlate a specific batch failure with more detailed error information.

Queued: The batch is waiting for sufficient resources in the workspace to be available before it runs.

Cancelled: The internal query of a pipeline batch is terminated. This occurs internally when STOP PIPELINE is run while a batch is running, or externally by pressing CTRL-C during START PIPELINE FOREGROUND.

No Data: The batch contains no data. Batches without data are automatically deleted as new batches are written.

BATCH_ROWS_WRITTEN

The number of rows affected in the destination table as a result of the batch transaction. When batches are written to the destination table, they consist of insert, replace, upsert, or delete operations. The BATCH_ROWS_WRITTEN value is the sum total of all the operations that occurred when the batch is written. This value may be NULL if the BATCH_STATE is any value other than Succeeded.

BATCH_ROWS_INSERTED

The total number of rows inserted in the batch. For INSERT … ON DUPLICATE KEY UPDATE, REPLACE, and Iceberg MERGE pipelines, it includes rows whose primary key does not exist in the target table.

BATCH_ROWS_UPDATED

The total number of rows updated in the batch. For INSERT … ON DUPLICATE KEY UPDATE and Iceberg MERGE pipelines, it includes rows whose primary key exists in the target table but the row is not an exact duplicate.

For REPLACE pipelines, rows with primary keys already present in the target table are included, even if the incoming row is an exact duplicate.

BATCH_ROWS_DELETED

The total number of rows deleted in the batch. This value is always 0 for INSERT … ON DUPLICATE KEY UPDATE and REPLACE pipelines. For Iceberg MERGE pipelines, it includes rows deleted from the target table.

BATCH_TIME

The elapsed time in seconds from the start of the batch transaction to its completion (either success or failure). This value reflects the total transaction time from the perspective of the Master Aggregator node. As a result, the elapsed time may be longer than a BATCH_PARTITION_TIME value for a given batch, as BATCH_PARTITION_TIME reflects timing from the perspective of the leaf node.

BATCH_START_UNIX_TIMESTAMP

The time that the batch transaction started in Unix timestamp format.

NUM_BATCH_PARTITIONS

The total number of batch partitions associated with the batch.