Sign inTry Free

Important

The SingleStore 9.1 release candidate (RC) is available now. It will be promoted to general availability (GA) soon and at that time it will be renamed to SingleStore 10.

In the interim, SingleStore 9.1 RC can be used to preview, evaluate, and provide feedback on the new and upcoming features in SingleStore 10, while SingleStore 9.0 is recommended for production workloads.

PIPELINES_SOURCE_OFFSETS

On this page

This view contains JSON-based offset information for pipelines, particularly SingleStore Kafka Connect Pipelines. Each row represents a single offset entry for a specific task within a batch.

Unlike the integer-based offsets stored in PIPELINES_CURSORS, this view stores complex offset structures in JSON format to accommodate various Kafka Connect source connectors that require more sophisticated offset tracking.

Note

This view is primarily used for Kafka Connect Pipelines, where offset structures vary by connector type. For traditional pipelines (Kafka, S3, Azure, GCS, HDFS, FS), refer to PIPELINES_CURSORS.

The KEY and VALUE columns contain JSON data whose structure depends on the specific Kafka Connect source connector being used. For example, Amazon Kinesis connectors use shard IDs and sequence numbers, while other connectors may use different offset structures.

information_schema.PIPELINES_SOURCE_OFFSETS Schema

Column Name

Description

DATABASE_NAME

The name of the database that contains the pipeline.

PIPELINE_NAME

The name of the pipeline associated with these offsets.

BATCH_ID

The unique identifier for the batch transaction. Batches are processed sequentially, and this value increases monotonically as new data is ingested. Use this column to track pipeline progress over time.

TASK_ID

The task identifier within the pipeline. Pipelines can create multiple tasks for parallel processing when the data source supports partitioning (for example, multiple Kinesis shards or Kafka partitions). The number of tasks is controlled by the tasks.max parameter in the pipeline CONFIG clause.

KEY

The offset key, in JSON format. The structure varies by connector type and identifies the specific partition or shard from which data is extracted. For Amazon Kinesis, this value typically contains the connector client name and shard identifier, such as {"shardId": "shardId-XXXX"}. For other connectors, the structure depends on how the connector tracks partitions.

VALUE

The offset value in JSON format. The structure varies by connector type and represents the position within the partition or shard up to which data has been successfully processed. For Amazon Kinesis, this value typically contains the sequence number, such as {"sequenceNumber": "XXXX"}. For other connectors, this value may include different position indicators, such as timestamps, log positions, or other connector-specific markers.

Last modified:

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In