PIPELINES
Warning
SingleStore 9.0 gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.
On this page
This view stores high-level information about any pipelines that have been created in the cluster.
information_ schema. PIPELINES Schema
Column Name |
Description |
---|---|
|
The name of the database associated with the pipeline. |
|
The name of the pipeline. |
|
The unique |
|
The pipeline's configuration in JSON format. |
|
The current state of the pipeline. |
|
The sum of the number of batches that have been skipped in the pipeline. |
|
The date/time when the pipeline was created or recreated by the |
|
The date/time when the pipeline was altered by an |
CONFIG_ JSON Schema
The CONFIG_
column in the information_
table contains a fixed set of read-only JSON key/value pairs.SHOW CREATE PIPELINE <pipeline-name>
statement.
Example CONFIG_ JSON
for Kafka Pipelines
{"name":"mypipeline","source_type":"KAFKA","connection_string":"172.17.0.2\/test-topic","batch_interval":0,"transform":["http:\/\/127.0.0.1:8000\/transform.py","",""],"dup_key_policy":null,"table":"messages","fields_terminated_by":"\t","fields_enclosed_by":"","fields_escaped_by":"\\","lines_terminated_by":"\n","lines_starting_by":"","extended_null":false,"column_list":null,"on_duplicate_key_update":null,"running":false}
Example CONFIG_ JSON
for S3 Pipelines
{"name": "my-s3-pipeline","source_type": "S3","connection_string": "my-s3-bucket-name","config": "{\"region\": \"us-west-1\"}","credentials": "{"aws_access_key_id": "your_access_key_id","aws_secret_access_key": "your_secret_access_key","aws_session_token": "your_temp_session_token","aws_web_identity_token": "your_aws_web_identiy_token","role_arn":"your_role_arn"}","batch_interval": 2500,"max_partitions_per_batch": -1,"transform": null,"load_error_policy": null,"dup_key_policy": null,"table": "my_table_name","fields_terminated_by": ",","fields_enclosed_by": "","fields_escaped_by": "\\","lines_terminated_by": "\n","lines_starting_by": "","extended_null": false,"column_list": null,"on_duplicate_key_update": null}
CONFIG_ JSON Schema Definition
Key Name |
Value Description |
---|---|
|
The name of the pipeline. |
|
The data source type for the pipeline. |
|
The connection string (information varies depending on the type of pipeline). |
|
The configuration information provided when creating a pipeline. |
|
A JSON object containing properties for the pipeline authentication/authorization. |
|
The time duration in milliseconds between batch extraction operations. |
|
The transform's URI, executable entry point, and arguments. |
|
The load error policy for the pipeline. |
|
The duplicate key policy that indicates how a row should be inserted if it contains a duplicate key value. |
|
The name of the table in which to insert data. |
|
The character that terminates a field. |
|
The character that encloses a field. |
|
The character that escapes a field. |
|
The character that terminates a line. |
|
The string prefix for a line. |
|
Specifies whether the non-quoted and case-insensitive string |
|
The column list to load data into. |
|
Specifies whether duplicate keys will be updated or ignored. |
|
Specifies whether the pipeline is currently running. |
Last modified: March 20, 2024