Data Loading for Filesystem Pipelines

When the master aggregator reads a directory’s contents, it schedules each file on a single SingleStoreDB partition. After each leaf partition across the cluster has finished extracting, transforming, and loading its file, a batch operation has been completed.