PIPELINES_CURSORS

DATABASE_NAME

The name of the database associated with the offsets.

PIPELINE_NAME

The name of the pipeline associated with the offsets.

SOURCE_TYPE

The name of the pipeline’s data source, such as KAFKA or S3.

SOURCE_PARTITION_ID

The identifier for a data source’s partition from which offsets are extracted. For an S3 pipeline, this value represents the name of a bucket’s object.

EARLIEST_OFFSET

The earliest offset for an offset range. This value indicates the starting offset and LATEST_OFFSET indicates the ending offset in an offset range. This value may be NULL if data extraction is still in progress or failed.

LATEST_OFFSET

The bounding offset for an offset range, exclusive. This value indicates the ending offset and EARLIEST_OFFSET indicates the starting offset in an offset range. This value may be NULL if data extraction is still in progress or failed.

CURSOR_OFFSET

The next offset at which the pipeline will start loading in this partition. This number represents the end of the range which has already been loaded by this pipeline, exclusive; all future considered offsets will be beyond this.

SUCCESSFUL_CURSOR_OFFSET

The offset cursor position in this source partition that has been successfully loaded. If this is lower than LATEST_OFFSET, it implies the intermediate offsets were skipped. Note: offsets will only be skipped if pipelines_stop_on_error is false.

UPDATED_UNIX_TIMESTAMP

The last updated time for an offset range in Unix timestamp format.

EXTRA_FIELDS

Some extractors output additional information when extracting data from the source. For S3 pipelines, you may encounter a value of {"size":<bytes>}, where <bytes> represents the size of an object in the source bucket.