Sign inTry Free

Stage

On this page

Stage is a managed file storage service that helps you organize and manage local files for ingestion into your SingleStore database(s). Each workspace group has a Stage where you can create folders and upload files. Stage is also supported in the Shared Edition (Shared workspaces). Through Stage, you can also save query results into files.

Note

The workspace group must be running SingleStore version 8.1 or later.

Why Use Stage

Stage enables you to:

  • Manage files and folders using the Cloud Portal UI, Management API, Notebooks (via the SingleStore Python SDK), and Fusion SQL commands.

  • Load files from a Stage into tables using pipelines, LOAD DATA command, or Cloud Portal UI and notebooks.

  • Export query results into a file using the SELECT ... INTO STAGE statement.

Manage a Stage

You can manage files and folders in a Stage using any of the following:

Using the Cloud Portal UI

On the Cloud Portal, select Ingestion > Stage in the left navigation pane. Manage files and folders in a Stage from this page.

Upload a File

To upload a file in a Stage at the root path (/):

  1. Select Ingestion > Stage > Upload File(s).

  2. In the Upload File(s) dialog, perform the following:

    Select File(s)

    Either drag and drop file(s) to the dialog, or select Browse Files to select the files to upload.

    Workspace

    Select the workspace group associated with the Stage to which the files are uploaded.

    Folder

    Select one of the following options:

    • No folder: Uploads the file at the Stage root path (/).

    • Select folder: Uploads the file to an existing folder in the Stage; select a folder from the list.

    • Create folder: Uploads the file to a new folder in the Stage; enter a name for the new folder in the New folder name field.

    Stage

    (Read-only field) Specifies the cloud provider and region where the file is uploaded.

  3. Select Upload Files to upload the selected file(s) at the specified path in a Stage.

Alternatively, select a folder in Stage, and then select Upload File(s) to directly upload files in this (current) folder. The file is uploaded in the Stage associated with the parent folder.

Create a Folder

Folders help organize and manage files efficiently. To create a folder at the Stage root path (/):

  1. Select Ingestion > Stage > Create Folder.

  2. In the Create Folder dialog, perform the following:

    Folder Name

    Enter a name for the folder.

    Workspace

    Select the workspace group associated with the Stage where the folder is created.

    Stage

    (Read-only field) Specifies the cloud provider and region where the folder is created.

  3. Select Create Folder.

A folder is now added in the Stage associated with the selected workspace group.

Alternatively, to create a subfolder:

  1. Select an existing folder in Stage, and then select Create Folder.

  2. Enter a name for the folder in the Folder name field and select Create Folder. The new folder is created inside the existing folder (as a subfolder) in the Stage associated with the parent folder.

Manage Files and Folders

To perform a specific task, select the ellipsis (three dots) under the Actions column of the file or folder and select the corresponding option from the list.

Action

Instruction

Rename a file or folder

Select Rename, specify the file or folder name under the Name column, and select the Save rename (✔) icon. Select the Cancel rename (X) icon to cancel the rename operation. 

Download a file

Select Download to download the file.

Load to database

Select Load Data to ingest the file in a SingleStore database. This workflow generates a notebook that contains the commands to load the selected file. Refer to Ingest a File Using the Cloud Portal for more information.

Delete a file or folder

Select Delete, confirm the deletion and then select Delete again to remove the file or folder.

Warning

Deleting a folder removes the folder and all its contents, including any files and subfolders.

Using the Management API

Use the Stage path (/v1/stage endpoint) in the Management API to manage files and folders in a Stage. Refer to Management API and Management API Reference for more information.

For example, the following API call lists all the files and folders in the Stage attached to the workspace group with the specified ID:

curl -X 'GET' \
  'https://api.singlestore.com/v1/stage/68af2f46-0000-1000-9000-3f6f5365d878/fs/' \
  -H 'accept: application/json'

Using Notebooks or SingleStore Python Client

The SingleStore Python SDK supports the Stage object, which can be used to manage files and folders in a Stage. You can also use the Stage object (including other objects in the SingleStore Python SDK) in a notebook. Refer to the SingleStore Python Client and SingleStore Python SDK API Reference for more information.

For example, the following code snippet uploads a file named data.csv to a Stage attached to a workspace group named example-group:

from singlestoredb import manage_workspaces
 
mgr = manage_workspaces('access_key_token_for_the_Management_API')
wg = mgr.workspace_groups['example-group']
wg.stage.upload_file('/filepath/data.csv', '/data.csv')

Using Fusion SQL Commands

Refer to Fusion SQL for the supported operations.

Ingest a File using Stage

Ingest files from Stage into a database using the Cloud Portal, LOAD DATA command, or pipelines.

Using the Cloud Portal

Using the Cloud Portal UI and SingleStore Notebooks, you can ingest Stage files into a SingleStore database attached to the workspace associated with the Stage.

  1. Under Stage, select the ellipsis (three dots) in the Actions column of the file to upload, and then select Load To Database.

  2. In the Load Data dialog, perform the following:

    1. Choose Workspace: Select a cluster.

    2. Choose a database: Select a database attached to the workspace.

    3. Table: Select an existing table or enter a new table name.

  3. Select Generate Notebook. A notebook is created, which contains the instructions and queries required to load the data.

    Note

    You can edit the queries to modify table or column names, apply custom transformations, or make any other adjustments required before loading the data.

  4. Select Run > Run All Cells. Optionally, run each notebook cell individually, in sequence.

The Check that the data has loaded cell verifies if the data has been loaded.

Using the LOAD DATA command

Create a table with a structure that can store data from the file. Use the following LOAD DATA syntax to load a file from a stage:

LOAD DATA STAGE 'path_in_stage/filename.extension'
INTO TABLE <table_name>
[FORMAT {JSON | AVRO | CSV}];

Refer to LOAD DATA for complete syntax and related information.

The following example loads data from a CSV file from a Stage:

LOAD DATA STAGE 'simple.csv'
INTO TABLE simple_data
FIELDS TERMINATED BY ','
IGNORE 1 LINES;

Note

LOAD DATA STAGE command is not supported in the Shared Edition. The workspace group must be running SingleStore version 8.9 or later.

Using Pipelines

Create a table with a structure that can store the data from the file. Use the following CREATE PIPELINE syntax to load a file from a Stage:

CREATE PIPELINE <pipeline_name>
AS LOAD DATA STAGE <path_in_Stage/filename> { <pipeline_options> }
INTO TABLE <table_name>
{ <data_format_options> }

Once the table and pipeline are created, start the pipeline. Refer to CREATE PIPELINE for the complete syntax and related information.

Here's a sample CREATE PIPELINE statement that loads data from a CSV file:

CREATE PIPELINE dbTest.plTest
AS LOAD DATA STAGE 'data.csv'
BATCH_INTERVAL 2500
SKIP DUPLICATE KEY ERRORS
INTO TABLE t1
FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '\\'
LINES TERMINATED BY '\n' STARTING BY ''
FORMAT CSV;

Export SQL Results to a Stage

SQL results may be exported to a Stage as follows:

SELECT * FROM <table_name> GROUP BY 1 INTO STAGE '<table_results.csv>'
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n';

Use the GROUP BY 1 clause to avoid getting multiple files from each leaf node.

Supported Files

The Stage storage service supports the following file formats:

CSV

SQL

JSON

Parquet

GZ

Zstd

Snappy

Storage Limits

Each Stage can have up to 10GB of storage for free. Individual files must not exceed 5GB in size.

Last modified:

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In