Warning
SingleStore 9.0 gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.
Pipelines Scheduling
On this page
If you create multiple pipelines in SingleStore, they will all run in parallel.
Variables |
Description |
---|---|
|
Allows you to specify the maximum number of partitions for each batch. |
|
Allows you to specify the maximum number of pipeline batch partitions that can run concurrently. |
|
Allows you to set the maximum number of pipelines running concurrently. |
Note
The number of partitions that a pipeline uses is dependent on its source.
For example, consider a SingleStore database with 10 partitions.
If the partition requirements, (as set via max_
) of any two pipelines exceed the total number of partitions, each pipeline will be run serially in a round robin fashion.
For example, consider a SingleStore database with 10 partitions, and 3 pipelines.
In this same scenario, if the pipelines_
variable was set to 5, and the max_
variable was not specified, then each pipeline P1, P2, and P3 will be run serially.
You can also use the pipelines_
variable in this scenario.
Pipelines and Resource Pools
Resource pools are used to group queries to prevent non-critical workloads from overburdening the system.MAX_
for a resource pool limits the number of SQL statements that can run simultaneously, thereby reducing the burden placed on a system.
Pipelines waiting in the queue use the thread pool slots reserved for background pipelines based on the settings of MAX_
and MAX_
.pipelines_
engine variable.
Last modified: May 8, 2025