Using Storage Repositories with Backups

Overview

For those Toolbox commands that support backup operations, the --repository option can be used to specify the data storage location where Toolbox reads data from and writes data to. For example, the sdb-admin create-backup command can write a database backup to the specified repository, whereas the sdb-admin list-backups command can read data from the specified repository to display the list of available backups.

SingleStore supports Amazon S3, Azure Blob Storage, Google Cloud Storage" (GCS) and Network File System (NFS) for backup storage options. Consequently, the --repository option can specify a location in an S3 bucket, Azure container, GCS or NFS mount.

The sdb-report calibrate-blob-storage command also supports the --repository option, using Amazon S3-compatible blob storage exclusively. An unlimited storage database is created in the specified repository which is then used to test blob storage performance.

The --repository option can be used with the following Toolbox commands:

• sdb-admin create-backup

• sdb-admin delete-backup

• sdb-admin list-backups

• sdb-admin restore-backup

• sdb-admin summarize-backups

• sdb-admin validate-backup

• sdb-report calibrate-blob-storage

Usage

The --repository option takes the following components as input: {protocol}://{path}[?{options}].

Where:

• protocol determines how the path to the storage location is interpreted, and which options are supported. Backup commands support s3, azure, GCS and file protocols for Amazon S3, Azure Blob Storage, GCS and NFS, respectively. The sdb-report calibrate-blob-storage command only supports s3.

• path defines the path to the storage location and restricts the scope to access specific resources within the specified storage. For example, when a backup command is run against s3://bucket-name/path/inside/bucket?region=us-east-1, only the backup files located under /path/inside/bucket of the bucket-name S3 bucket will be accessed. In this example, the s3://bucket-name/path/inside repository is different from, let’s say, s3://bucket-name/path/inside/bucket, which is another repository in the same S3 bucket.

• options specify additional configuration settings. For example, when using S3 for storage, you may specify all of the options in the BACKUP DATABASE command, including region, endpoint URL, and x-amz-server-side-encryption.

Note that the --repository option takes values in a URL format that corresponds to the storage protocol used. The following examples demonstrate the URL format of the --repository value for different storage options:

Amazon S3: s3://bucket-name/path/inside/bucket?region=us-east-1

Azure Blob Storage: azure://container/path/inside/container

GCS: gs://bucket/path_inside_bucket

NFS: file://host/some/directory