Combining Unlimited and Local Storage Databases
It is possible to create unlimited and local storage databases in the same cluster. However, unlimited storage databases tend to aggressively reserve space for the blob cache and fill it up, which can contribute to “Out of Disk” (OOD) errors. If you want to reserve a certain amount of space for local databases on the same cluster, then you must cap the unlimited storage database blob cache size by setting the maximum_blob_cache_size_mb variable. Set it to a value that leaves enough space for your local databases and any other requirements.
The maximum_blob_cache_size_mb variable is configured by default on startup, using heuristics. It is set to either 75% of total disk size or total disk size - 200GB, whichever is smaller. if the disk is smaller than 40GB, then the maximum blob cache size is set to the disk size. Users are likely to run into issues with disks smaller than 40GB, hence SingleStore also traces a warning on startup in this case.
To accommodate local databases you should manually set the maximum_blob_cache_size_mb variable to the initial default value minus the value you need for the local databases. This must be set on every leaf node manually since the available disk size can vary on each node. You will need to set it in the memsql.cnf file to preserve the value across restarts.
Using Unlimited Storage
When running in a self-hosted configuration, the unlimited storage databases feature requires the following to operate:
Amazon S3 or a 100% Amazon S3-compatible object store
Azure Blob storage
Google Cloud Storage
SingleStore has been tested extensively with S3 on AWS to confirm S3 compatibility. Please verify with your storage provider that the storage system is 100% S3-compatible before production use with unlimited storage databases. Before using an S3-compatible object store other than AWS S3 in production, customers should verify that their system functions and performs well in a production-scale test environment.
Functional S3 compatibility will allow SingleStore unlimited storage to run. However, be sure your object store has enough capacity and performance to support your workload.
You should run sdb-report calibrate-blob-storage as part of testing your S3-compatible object store, but do not substitute that for production-scale testing before putting your application in production.
Prior to creating an unlimited storage database, using the CREATE DATABASE command, the enable_bottomless engine variable must be set to ON (the default value).
If any storage-related error messages are generated when running sdb-report calibrate-blob-storage or CREATE DATABASE to create an unlimited storage database, then the storage system is not 100% S3-compatible and will not work with unlimited storage databases.
To use Ceph storage with unlimited storage databases, enable ceph_mode before creating the database.