# Pipelines Scheduling If you create multiple pipelines in SingleStore, they will all run in parallel. They will run in parallel until all SingleStore partitions have been saturated. You can use the variables in the following table to define the maximum number of partitions and the number of pipelines that can run at the same time. | Variables | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max_partitions_per_batch` | Allows you to specify the maximum number of partitions for each batch. This variable can be modified for each pipeline by using the[ALTER PIPELINE SET MAX\_PARTITIONS\_PER\_BATCH](https://docs.singlestore.com/db/v9.1/reference/sql-reference/pipelines-commands/alter-pipeline/#UUID-2be7ea6a-a0ff-014f-ec65-1facc5e11999.md)command. | | `pipelines_max_concurrent_batch_partitions` | Allows you to specify the maximum number of pipeline batch partitions that can run concurrently. This is a global variable. | | `pipelines_max_concurrent` | Allows you to set the maximum number of pipelines running concurrently. | > **📝 Note**: The number of partitions that a pipeline uses is dependent on its source. For example, for Kafka pipelines, the number of batch partitions that can run concurrently can not exceed the number of Kafka topic partitions. For example, consider a SingleStore database with 10 partitions. Without any constraints, it is possible to run 5 parallel pipelines using 2 partitions each, 2 pipelines using 5 partitions each, and so on. If the partition requirements, (as set via `max_partitions_per_batch`) of any two pipelines exceed the total number of partitions, each pipeline will be run serially in a round robin fashion. For example, consider a SingleStore database with 10 partitions, and 3 pipelines. Let's say the first batch of pipelines P1, P2, and P3 requires 4, 8, and 4 partitions, respectively. The pipelines are scheduled concurrently with the aim of saturating the partitions in a cluster. Hence, the scheduler will run pipelines P1 and P3 in parallel to process their first batch. And then, it will run pipeline P2 serially, because the sum of the number of partitions required by P2 and any other pipeline (P1 or P3) is greater than the number of partitions in the cluster (10 partitions). In this same scenario, if the `pipelines_max_concurrent_batch_partitions` variable was set to 5, and the `max_partitions_per_batch` variable was not specified, then each pipeline P1, P2, and P3 will be run serially. You can also use the `pipelines_max_concurrent` variable in this scenario. If the variable was set to 1, then each pipeline P1, P2, and P3 will be run serially as no two pipelines can run concurrently. ## Pipelines and Resource Pools Resource pools help isolate and control workloads by grouping queries. This prevents non-critical operations from overwhelming system resources. Use the `MAX_CONCURRENCY` setting on a resource pool to limit the number of SQL statements that can run simultaneously by reducing system load. Refer to [CREATE RESOURCE POOL](https://docs.singlestore.com/db/v9.1/reference/sql-reference/resource-pool-commands/create-resource-pool.md) and [Set Resource Limits](https://docs.singlestore.com/db/v9.1/user-and-cluster-administration/use-the-workload-manager-and-set-resource-limits/set-resource-limits.md) for more information. Pipelines waiting in the queue use the thread pool slots reserved for background pipelines based on the settings of `MAX_CONCURENCY` and `MAX_QUEUE_DEPTH`. Users may want to adjust the number of simultaneous running pipelines by changing the `pipelines_max_concurrent` engine variable. When running multiple pipelines in SingleStore, it's important to understand how global and local concurrency controls work together. * **Global Concurrency Controls**: * `pipelines_max_concurrent`: This engine variable sets the maximum number of pipeline batches that can run concurrently across the entire cluster. This is a global limit and applies to all pipelines, regardless of the resource pool to which they are assigned. Each pipeline batch can initiate multiple SQL queries. These queries are subject to the limits of their assigned resource pool. If a query is delayed by resource pool constraints, it can block the entire batch from completing. This can prolong its use of a background execution slot controlled by `pipelines_max_concurrent`. This can affect overall pipeline throughput. * **Resource Pools and SQL Query Limits**: * `MAX_CONCURRENCY`: This variable limits the number of SQL statements that run simultaneously in the pool, helping prevent non-critical workloads from overburdening the system. It does not control how many pipelines can run concurrently in a specific pool. * `MAX_QUEUE_DEPTH`: This variable determines how many SQL queries will be queued for the resource pool once `MAX_CONCURRENCY` is reached. It does not control how many pipelines can run concurrently in a specific pool. * **Pipeline Scheduling and Queuing**: * Pipelines that are ready to execute but waiting for available concurrency slots consume thread pool resources reserved for background pipelines. These threads are influenced by the `MAX_CONCURRENCY` and `MAX_QUEUE_DEPTH` settings of the resource pool. However, the actual number of concurrently running pipeline batches is controlled by the global `pipelines_max_concurrent` variable. For example, if `pipelines_max_concurrent` is set to 4 and you have two resource pools (A and B), no more than four pipeline batches will run at the same time, regardless of how many pipelines are assigned to pools A or B, or their individual `MAX_CONCURRENCY` settings. A pipeline, started via `START PIPELINE`, follows a repeating lifecycle: **execute batch > wait for BATCH\_INTERVAL (or skip wait) > execute batch > …** Pipelines interact with both `pipelines_max_concurrent` and the assigned resource pool only during the execute batch phase. *** Modified at: May 12, 2026 Source: [/db/v9.1/load-data/about-singlestore-pipelines/pipeline-concepts/pipelines-scheduling/](https://docs.singlestore.com/db/v9.1/load-data/about-singlestore-pipelines/pipeline-concepts/pipelines-scheduling/) (An index of the documentation is available at /llms.txt)