Sign inTry Free

Install SingleStore Flow

On this page

This section contains instructions for installing Ingest and XL Ingest, and license information.

To use Ingest and XL Ingest, install the products and provide the product IDs to either team@singlestore.com or your account representative to obtain either a paid license or a Developer trial license.

Prerequisites

While the database prerequisites will vary based on the source, the following are the minimum requirements for using both Ingest and XL Ingest.

Component

Requirements

Virtual Machine (VM)

The sizing recommendation is based on the size of databases/tables being transferred.

For smaller databases sized 100GB or less, a minimum of 2 vCPUs, 8 GB RAM, and 100 GB of free disk space is recommended.

For large transfers, more disk space may be required. The required free space for Ingest is twice the size of the largest table to be transferred.

For very large tables, where XL Ingest must be used, the amount of local disk needed is 2 x (logical partition size) x (number of concurrent partition transfers).

More CPU cores and RAM is required for sources and destinations that contain a large amount of data and have an aggressive timeline in which to transfer it.

The VM must be close to the source database to minimize latency.

Refer to the Recommended Hardware Configuration section in the Ingest documentation for more information.

Browser

A Chromium-based browser, such as Google Chrome or Microsoft Edge, installed on the VM

Java

Amazon Corretto Java 21, a no-cost distribution of the Open Java Development Kit (OpenJDK), installed on the VM

Set Up the Encryption Key

Flow encrypts sensitive values in config/config.xml with AES-256-CBC. The Flow JVM reads the encryption key from the FLOW_ENCRYPTION_KEY environment variable. New installations must configure an encryption key before storing encrypted secrets.

Use the following key requirements:

  • Algorithm: AES-256. The decoded key must be exactly 32 bytes.

  • Encoding: Base64, with optional trailing = padding.

Flow validates the key at startup. If FLOW_ENCRYPTION_KEY is set but invalid, Flow does not start. If FLOW_ENCRYPTION_KEY is not set, new installations will not start, but upgraded installations may fall back to a legacy hardcoded key (insecure and deprecated). When Flow resolves the key successfully, it writes Info(CMG500) to the log together with a short SHA-256 fingerprint. Use this entry to confirm which key the process is using without exposing the key.

Note: SingleStore recommends rotating the encryption key every 90 to 180 days for security. Refer to Rotate the Encryption Key for the key rotation procedure.

Generate a Key on Windows

Run the following PowerShell commands to generate a Base64-encoded 32-byte key. The commands use a cryptographically secure random generator.

$bytes = New-Object byte[] 32
[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($bytes)[Convert]::ToBase64String($bytes)

If OpenSSL is available on the Windows host, run the following command:

openssl rand -base64 32

Store the generated value in a secret manager or another secure location. This value is required when configuring the FLOW_ENCRYPTION_KEY.

Generate a Key on Linux or macOS

Use OpenSSL to generate a Base64-encoded 32-byte key. OpenSSL is preinstalled on every supported Linux distribution.

Run the following command:

openssl rand -base64 32

Store the generated value in a secret manager or another secure location. This value is required when configuring the FLOW_ENCRYPTION_KEY.

Install Ingest

You can install Ingest in both Windows and non-Windows environments.

Install Ingest as a Windows Service

Note: Filenames may change from release to release.

  1. Download the following files to your host:

  2. Unzip the Ingest setup ZIP file. This creates an ingest directory.

  3. Move the ingest directory to your desired install location. For example, C:\SingleStore\ingest.

  4. Copy the downloaded ingest.jar file into the ingest directory so that all required files are in the same location.

  5. Change the directory to the ingest directory and open the ingest.xml file in a Notepad. Change the following lines as per your requirements.

    <id>SingleConnect-Ingest</id>

    This tag is used as a unique ID in Windows services.

    Each Ingest pipeline has its own ID.

    <name>SingleConnect Ingest</name>

    This tag is a display name in the Windows services.

    It must be brief but descriptive.

    <description>SingleConnect Ingest</description>

    This tag specifies a description of the Ingest service.

    Note: When a server hosts several Java versions, relying on the system’s default Java can result in version conflicts or unexpected failures. To guarantee Ingest uses the intended Java version, refer the full path to your required Java binary directly.

    For example, if Java 21 has been extracted to C:\java\java21, specify the absolute path to java.exe in your ingest.xml file:

    <executable>C:\java\java21\bin\java.exe</executable>

  6. In ingest.xml, uncomment the env line and replace the placeholder with the generated Base64 key. The service wrapper exports the key to the Java process automatically.

    <env name="FLOW_ENCRYPTION_KEY" value="YOUR_BASE64_KEY_HERE"/>

    To keep the secret out of ingest.xml, omit the env element and set FLOW_ENCRYPTION_KEY as a Windows system environment variable in System Properties > Advanced > Environment Variables. The Ingest service reads the updated value the next time it starts.

  7. Grant the following permissions to the user that runs the Ingest service, on the entire Ingest directory:

    • Read

    • Write

    • Delete

    Ensure that all contained files and subfolders get the same permissions.

  8. Install Ingest as a Windows service.

    1. Open a command prompt as an Administrator.

    2. Change the current directory to the ingest directory and run the Install-Service.bat file using the following command.

      Install-Service.bat

      This batch file runs commands to install the service. Once the command is completed, close the command prompt.

  9. The Ingest service must appear in the Windows services list like SingleConnect-Ingest, matching the ID set earlier.

  10. The Ingest service starts automatically upon installation and must be running. You can start, stop, or restart the Ingest service similar to other Windows services.

  11. To verify the encryption configuration, open the most recent log file under ingest\log\ and search for CMG500. Verify that the log contains the following line:

    Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>)

    If the log shows "...from legacy hardcoded fallback", Flow is using an insecure default key instead of FLOW_ENCRYPTION_KEY. This means the environment variable was not successfully passed to the JVM. Recheck the ingest.xml env entry, or the Windows system environment variable, and then restart the service.

  12. Open a Chromium-based browser at http://localhost:8081, or http://<host>:8081 if the application is installed on a remote server. You can change this port from the user interface (UI). The new port setting takes effect after the next run.

  13. Navigate to Settings > License and enter the license key.

Install Ingest in Non-Windows Environment

  1. Download the following files to your host:

  2. Unzip the Ingest setup ZIP file. This creates an ingest directory.

  3. Move the ingest directory to your desired install location. For example, /opt/singlestore/ingest.

  4. Copy the downloaded ingest.jar file into the ingest directory so that all required files are in the same location.

  5. Configure and start Ingest using ONE of the following methods:

    • Edit the launcher ingest.sh in the Ingest directory. Uncomment the FLOW_ENCRYPTION_KEY lines and replace the placeholder with the generated Base64 key.

      export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE
      java -cp ingest.jar:lib/*:. org.sirus.Run

      To start Ingest, run:

      ./ingest.sh

    • Leave ingest.sh unchanged, and export FLOW_ENCRYPTION_KEY in the shell that launches Ingest, then start Ingest.

      export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE
      java -jar ingest.jar java -jar ingest.jar

    • Pass the key as a Java system property.

      java -Dflow.encryption.key=YOUR_BASE64_KEY_HERE -jar ingest.jar

      Note: The flow.encryption.key Java system property takes precedence over the environment variable. This precedence is useful when multiple Flow instances run on the same host with different keys.

  6. To include external libraries, run the following command.

    Windows 

    java -cp "other_lib.jar;ingest.jar" org.springframework.boot.loader.JarLauncher

    Linux 

    java -cp "other_lib.jar:ingest.jar" org.springframework.boot.loader.JarLauncher

  7. Tail the most recent file under log/ and search for CMG500:

    grep -F 'Info(CMG500)' log/*.log | tail

    Expected output:

    Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>)

    If the log shows "..from legacy hardcoded fallback", the variable was not exported into the JVM environment. Confirm it in the same shell that launched Ingest.

    env | grep FLOW_ENCRYPTION_KEY

  8. Open a Chromium-based browser at http://localhost:8081, or http://<host>:8081 if the application is installed on a remote server. You can change this port from the user interface (UI). The new port setting takes effect after the next run.

  9. Navigate to Settings > License and enter the license key.

Install XL Ingest

You can install XL Ingest in both Windows and non-Windows environments.

Note: Filenames may change from release to release.

Install XL Ingest as a Windows Service

  1. Download the following files to your host:

  2. Unzip the XL Ingest setup ZIP file. This creates an xlingest directory.

  3. Move the xlingest directory to your desired install location. For example, C:\SingleStore\xlingest.

  4. Copy the downloaded xlingest.jar file into the xlingest directory so that all required files are in the same location.

  5. Change the directory to the xlingest directory and open xlingest.xml file in a Notepad. Change the following lines as per your requirements.

    <id>SingleConnect-XL-Ingest</id>

    This tag is used as a unique ID in Windows services.

    Each XL Ingest pipeline has its own ID.

    <name>SingleConnect XL Ingest</name>

    This tag is a display name in the Windows services.

    It must be brief and descriptive.

    <description>SingleConnect XL Ingest</description>

    This tag specifies a description of the XL Ingest service.

  6. In xlingest.xml, uncomment the env line and replace the placeholder with the generated Base64 key. The service wrapper exports the key to the Java process automatically.

    <env name="FLOW_ENCRYPTION_KEY" value="YOUR_BASE64_KEY_HERE"/>

    XL Ingest and Ingest use the same FLOW_ENCRYPTION_KEY value because they share the same configuration file. The XL Ingest service reads the updated value the next time it starts.

  7. Install XL Ingest as a Windows service.

    1. Open a command prompt as an Administrator.

    2. Navigate to xlingest directory and run the Install-SingleConnect-XL-Ingest-Service.bat file using the following command.

      Install-SingleConnect-XL-Ingest-Service.bat

      This batch file runs commands to install the service. Once the command is completed, close the command prompt.

  8. The XL Ingest service must appear in the Windows services list like SingleConnect-XL-Ingest, matching the ID you set earlier.

  9. The XL Ingest service starts automatically upon installation and must be running. You can start, stop, or restart the XL Ingest service like other Windows services.

  10. To verify the encryption configuration, open the most recent log file under xlingest\log\ and search for CMG500. Verify that the log contains the following line:

    Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>)

    If the log shows "...from legacy hardcoded fallback", Flow is using an insecure default key instead of FLOW_ENCRYPTION_KEY. This means the environment variable was not successfully passed to the JVM. Recheck the xlingest.xml env entry, or the Windows system environment variable, and then restart the service.

  11. Open a Chromium-based browser at http://localhost:8084. If the application is installed on a server, navigate to http://<server-ip-address>:8084 instead.

  12. Navigate to Configuration > License, and enter the license key.

Install XL Ingest in Non-Windows Environment

  1. Download the following files to your host:

  2. Unzip the XL Ingest setup ZIP file. This creates an xlingest directory.

  3. Move the xlingest directory to your desired install location. For example, /opt/singlestore/xlingest.

  4. Copy the downloaded xlingest.jar file into the xlingest directory so that all required files are in the same location.

  5. Configure and start XL Ingest using ONE of the following methods:

    • Edit the launcher xlingest.sh in the ingest directory. Uncomment the FLOW_ENCRYPTION_KEY lines and replace the placeholder with the generated Base64 key.

      export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE
      java -cp xlingest.jar:lib/*:. cirus.Run

      To start XL Ingest, run:

      ./xlingest.sh

    • Leave xlingest.sh unchanged, and export FLOW_ENCRYPTION_KEY in the shell that launches XL Ingest, then start XL Ingest.

      export FLOW_ENCRYPTION_KEY=YOUR_BASE64_KEY_HERE
      java -jar xlingest.jar
      java -jar ingest.jar

    • Pass the key as a Java system property.

      java -Dflow.encryption.key=YOUR_BASE64_KEY_HERE -jar xlingest.jar

      Note: The flow.encryption.key Java system property takes precedence over the environment variable. This precedence is useful when multiple Flow instances run on the same host with different keys.

  6. To include external libraries, use the following command.

    Windows 

    java -cp "other_lib.jar;xlingest.jar" cirus.Run

    Linux 

    java -cp "other_lib.jar:xlingest.jar" cirus.Run

  7. Tail the most recent file under log/ and search for CMG500:

    grep -F 'Info(CMG500)' log/*.log | tail

    Expected output:

    Info(CMG500): Resolved AES-256 encryption key from FLOW_ENCRYPTION_KEY env var (fingerprint <16-hex-chars>)

    If the log shows "..from legacy hardcoded fallback", the variable was not exported into the JVM environment. Confirm it in the same shell that launched XL Ingest.

    env | grep FLOW_ENCRYPTION_KEY

  8. Open a Chromium-based browser at http://localhost:8084. If the application is installed on a server, navigate to http://<server-ip-address>:8084 instead.

  9. Navigate to Configuration > License, and enter the license key.

Apply the License Key

The Flow suite of products comes with a validity period specific to each customer's agreement. You must obtain valid license keys to begin using the software. For learning and evaluation purposes, a free license with usage limits is available, refer to Free Trial Access to Flow for more details.

Licenses specify a maximum capacity in units. To transfer data to a SingleStore destination, the capacity of the Ingest license must be at least equal to the number of capacity units in the destination.

Each software product has its own unique product ID when set up individually. To obtain a valid license, contact SingleStore Sales with the Product ID.

For Ingest, go to Settings > Licence. In Licence Key, enter the license key, and then select Apply to save the settings.

To retrieve the Product ID, go to your Ingest instance, go to Settings > Licence, and copy the ID from the 'Product Id'.

For XL Ingest, go to Configuration. In Licence Key, enter the license key, and then select Save to save the settings.

To retrieve the Product ID, go to your XL Ingest instance, go to Configuration > Licence, and copy the ID from the 'Product Id'.

Last modified:

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In