# PIPELINES\_SOURCE\_OFFSETS

This view contains JSON-based offset information for pipelines, particularly SingleStore Kafka Connect Pipelines. Each row represents a single offset entry for a specific task within a batch.

Unlike the integer-based offsets stored in `PIPELINES_CURSORS`, this view stores complex offset structures in JSON format to accommodate various Kafka Connect source connectors that require more sophisticated offset tracking.

> **📝 Note**: This view is primarily used for Kafka Connect Pipelines, where offset structures vary by connector type. For traditional pipelines (Kafka, S3, Azure, GCS, HDFS, FS), refer to [`PIPELINES_CURSORS`](https://docs.singlestore.com/db/v9.1/reference/information-schema-reference/data-ingest/pipelines-cursors.md).The `KEY` and `VALUE` columns contain JSON data whose structure depends on the specific Kafka Connect source connector being used. For example, Amazon Kinesis connectors use shard IDs and sequence numbers, while other connectors may use different offset structures.

## information\_schema.PIPELINES\_SOURCE\_OFFSETS Schema

| **Column Name** | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_NAME` | The name of the database that contains the pipeline.                                                                                                                                                                                                                                                                                                                                                                                                |
| `PIPELINE_NAME` | The name of the pipeline associated with these offsets.                                                                                                                                                                                                                                                                                                                                                                                             |
| `BATCH_ID`      | The unique identifier for the batch transaction. Batches are processed sequentially, and this value increases monotonically as new data is ingested. Use this column to track pipeline progress over time.                                                                                                                                                                                                                                          |
| `TASK_ID`       | The task identifier within the pipeline. Pipelines can create multiple tasks for parallel processing when the data source supports partitioning (for example, multiple Kinesis shards or Kafka partitions). The number of tasks is controlled by the`tasks.max`parameter in the pipeline`CONFIG`clause.                                                                                                                                             |
| `KEY`           | The offset key, in JSON format. The structure varies by connector type and identifies the specific partition or shard from which data is extracted. For Amazon Kinesis, this value typically contains the connector client name and shard identifier, such as`{"shardId": "shardId-XXXX"}`. For other connectors, the structure depends on how the connector tracks partitions.                                                                     |
| `VALUE`         | The offset value in JSON format. The structure varies by connector type and represents the position within the partition or shard up to which data has been successfully processed. For Amazon Kinesis, this value typically contains the sequence number, such as`{"sequenceNumber": "XXXX"}`. For other connectors, this value may include different position indicators, such as timestamps, log positions, or other connector-specific markers. |

***

Modified at: February 18, 2026

Source: [/db/v9.1/reference/information-schema-reference/data-ingest/pipelines-source-offsets/](https://docs.singlestore.com/db/v9.1/reference/information-schema-reference/data-ingest/pipelines-source-offsets/)

(An index of the documentation is available at /llms.txt)