PIPELINES_BATCHES_SUMMARY

DATABASE_NAME

The name of the database associated with the batch.

PIPELINE_NAME

The name of the pipeline associated with the batch.

BATCH_ID

The internal unique identifier for the batch. Values of BATCH_ID are unique in this table.

ROWS_INSERTED

The total number of rows inserted in the batch. For INSERT … ON DUPLICATE KEY UPDATE, REPLACE, and Iceberg MERGE pipelines, it includes rows whose primary key does not exist in the target table.

ROWS_UPDATED

The total number of rows updated in the batch. For INSERT … ON DUPLICATE KEY UPDATE and Iceberg MERGE pipelines, it includes rows whose primary key exists in the target table but the row is not an exact duplicate.

For REPLACE pipelines, rows with primary keys already present in the target table are included, even if the incoming row is an exact duplicate.

ROWS_DELETED

The total number of rows deleted in the batch. This value is always 0 for INSERT … ON DUPLICATE KEY UPDATE and REPLACE pipelines. For Iceberg MERGE pipelines, it includes rows deleted from the target table.

BATCH_STATE

Specifies whether the batch transaction was successful. There are five possible values: Succeeded, In Progress, Failed, Queued, or Cancelled.

Succeeded: The batch was successfully loaded into the destination table.

In Progress: The batch is currently being loaded into the destination table.

Failed: The batch failed to be written. Batch failures are written as errors in the information_schema.PIPELINES_ERRORS table and you can use the BATCH_ID to correlate a specific batch failure with more detailed error information.

Queued: The batch is waiting for enough resources in the cluster to free up so it can run.

Cancelled: A pipeline batch's internal query is killed. That can be triggered internally, if you run STOP PIPELINE while a batch is running, or externally, by CTRL-C-ing START PIPELINE FOREGROUND.

START_TIME

The time that the batch transaction started in DATETIME format.

EXTRACTOR_WAIT_TIME

The amount of time, in seconds, that the pipeline spent waiting on the extractor to complete its operation.

TRANSFORM_WAIT_TIME

The amount of time, in seconds, that the pipeline spent waiting on the transform to complete its operation.

BATCH_TIME

The number of seconds the batch took to run, or if the batch is In Progress, the number of seconds it has been running for.

ROWS_PER_SEC

Equal to ROWS_STREAMED / BATCH_TIME.

ROWS_STREAMED

The number of rows extracted into SingleStore Helios. It can be higher or lower than the number of rows written, if, for instance, rows are filtered by a WHERE clause or SKIP clause, or if rows are modified by an upsert clause.

NUM_PARTITIONS

The total number of batch partitions for this batch.

NUM_PARTITIONS_FINISHED

The total number of finished batch partitions for this batch. A batch cannot finish until all batch partitions are finished.

MB_STREAMED

The number of megabytes extracted into SingleStore Helios.

MB_PER_SEC

Equal to MB_STREAMED / BATCH_TIME.