Important

Helios features are now enabled during weekly update windows and are no longer directly tied to SingleStore engine releases. Refer to the release notes to view the latest features available in your Helios cluster.

PIPELINES_SOURCE_OFFSETS

This view contains JSON-based offset information for pipelines, particularly SingleStore Connector Pipelines. Each row represents a single offset entry for a specific task within a batch.

Unlike the integer-based offsets stored in PIPELINES_CURSORS, this view stores complex offset structures in JSON format to accommodate various Kafka Connect source connectors that require more sophisticated offset tracking.

Note

This view is primarily used for Connector Pipelines, where offset structures vary by connector type. For traditional pipelines (Kafka, S3, Azure, GCS, HDFS, FS), refer to PIPELINES_CURSORS.

The KEY and VALUE columns contain JSON data whose structure depends on the specific Kafka Connect source connector being used. For example, Amazon Kinesis connectors use shard IDs and sequence numbers, while other connectors may use different offset structures.

information_schema.PIPELINES_SOURCE_OFFSETS Schema

Column Name

Description

DATABASE_NAME

The name of the database that contains the pipeline.

PIPELINE_NAME

The name of the pipeline associated with these offsets.

BATCH_ID

The unique identifier for the batch transaction. Batches are processed sequentially, and this value increases monotonically as new data is ingested. Use this column to track pipeline progress over time.

TASK_ID

The task identifier within the pipeline. Pipelines can create multiple tasks for parallel processing when the data source supports partitioning (for example, multiple Kinesis shards or Kafka partitions). The number of tasks is controlled by the tasks.max parameter in the pipeline CONFIG clause.

KEY

The offset key, in JSON format. The structure varies by connector type and identifies the specific partition or shard from which data is extracted. For Amazon Kinesis, this value typically contains the connector client name and shard identifier, such as {"shardId": "shardId-XXXX"}. For other connectors, the structure depends on how the connector tracks partitions.

VALUE

The offset value in JSON format. The structure varies by connector type and represents the position within the partition or shard up to which data has been successfully processed. For Amazon Kinesis, this value typically contains the sequence number, such as {"sequenceNumber": "XXXX"}. For other connectors, this value may include different position indicators, such as timestamps, log positions, or other connector-specific markers.

Last modified:

Was this article helpful?