Sign inTry Free

Important

Self-managed SingleStore will soon transition from version 9.1 RC to version 10. This new semantic versioning scheme will provide SingleStore with finer control over engine and feature releases that were not possible with the current versioning scheme.

In the interim, SingleStore 9.1 RC can be used to preview, evaluate, and provide feedback on the new and upcoming features in SingleStore 10 prior to its general availability. Ahead of this transition, SingleStore 9.0 is recommended for production workloads, which can later be upgraded to SingleStore 10.

PIPELINES_OFFSETS

On this page

This view contains information about a pipeline's offset ranges. Each row represents a single data source partition and tracks various offset positions throughout the pipeline's extraction and loading process.

Note

This view contains integer-based offsets for traditional pipelines (Kafka, S3, Azure, GCS, HDFS, FS). For Kafka Connect Pipelines that require JSON-based offset tracking, refer to PIPELINES_SOURCE_OFFSETS. The offset values in this view represent numeric positions or ranges within the data source partitions.

information_schema.PIPELINES_OFFSETS Schema

Column Name

Description

DATABASE_NAME

The name of the database that contains the pipeline.

PIPELINE_NAME

The name of the pipeline associated with the offsets.

SOURCE_TYPE

The pipeline data source type, such as KAFKA, S3, AZURE, GCS, HDFS, or FS(filesystem). This column identifies the external system from which data is extracted.

SOURCE_PARTITION_ID

The identifier of the source partition from which offsets are extracted.

For S3 pipelines, this value represents an object name in a bucket.

For Kafka pipelines, it represents the partition number. For file-based sources, it represents a file path.

EARLIEST_OFFSET

The starting offset of an offset range. Together with LATEST_OFFSET, this column defines the range of data that has been or will be processed. This value may be NULL if extraction has not started or is still in progress.

LATEST_OFFSET

The ending offset of an offset range (exclusive). Together with EARLIEST_OFFSET, this column defines the available data range in the source partition. This value may be NULL if extraction is in progress or has failed.

LATEST_LOADED_OFFSET

The most recent offset that the pipeline has extracted. This data may not yet be committed to the destination table. This value advances as the pipeline reads from the source.

LATEST_SUCCESSFUL_OFFSET

The most recent offset that has been successfully loaded and committed. If this value is lower than LATEST_LOADED_OFFSET, extracted data is still being processed or has encountered errors. This column represents the safe restart position for the pipeline.

LATEST_EXPECTED_OFFSET

The next offset at which the pipeline expects to resume loading. Under normal operation, this value closely follows LATEST_SUCCESSFUL_OFFSET. Large differences may indicate processing delays or partition rebalancing.

UPDATED_UNIX_TIMESTAMP

The time when the offset metadata was last updated, expressed as a Unix timestamp. Use this column to detect stalled pipelines or inactive partitions.

EXTRA_FIELDS

Additional extractor-specific metadata in JSON format. Some extractors include extra information, such as {"size": <bytes>} for S3 objects. For other sources, this column may contain connector-specific metadata, such as file timestamps, compression details, or partition information. This value may be NULL if no additional metadata is available.

Last modified:

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In