Sign inTry Free

SingleStore Ingest Operational FAQ

On this page

This document describes some of the most frequent operational user queries.

Stop Ingest

Perform the following steps to stop Ingest.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Open Windows Services and stop the Ingest Windows service.

  3. If the Ingest Windows service stops successfully but the Ingest page is still loading, you may need to terminate the non-responding processes.

  4. To terminate the non-responding Ingest process, follow these steps.

    1. Open the command prompt as Administrator.

    2. Run the following command:

      netstat -ano | find "8081"

      Replace 8081 with the port number on which Ingest is running.

      For example, Ingest is running on port 9090, and the process ID for Ingest is 4044,.

    3. Open Windows Task Manager, navigate to the Details tab, and find the process with PID 4044.

    4. Right-click on the process and select End Task. This terminates the runaway Ingest process.

Restart Ingest

Perform the following steps to restart Ingest.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete. Then, stop the Ingest Windows service.

  2. Restart the Ingest Windows service and wait for the Ingest dashboard to appear, refreshing the page as needed.

  3. Navigate to the Dashboard tab, under Schedule, and enable the scheduler.

  4. Ingest picks up the correct processing sequence. Confirm that the last sequence number loaded correctly and that the new sequence number started correctly. If needed, perform a rollback to the correct sequence number for processing.

Perform a Full Extract

Use the following steps to perform a full extract for an initial sync of all tables in a pipeline.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to Dashboard > Tables.

    1. Select the tables (one or more) for the full extract.

    2. Ensure that the primary key (PKey) column(s) are applied.

    3. Enable Delta Extract (With History).

    4. Select Apply.

  3. Navigate to Dashboard > Operations and select Full Extract.

  4. Navigate to the Logs tab. Check the logs to confirm that the selected tables have started the full extract.

Skip an Initial Extract

Use the following steps to perform deltas without an initial sync, or when using XL Ingest.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to Dashboard > Tables.

    1. Select the table(s) to perform deltas without an initial sync of the data.

    2. Ensure that the primary key (PKey) column(s) are applied.

    3. Enable Delta Extract (With History).

    4. Enable Skip Initial Extract.

    5. Select Apply.

  3. Navigate to Dashboard > Operations and select Full Extract.

  4. Navigate to the Logs tab. Check the logs to confirm that the selected table(s) skipped the full extract.

Redo an Initial Extract

Use the following steps to redo an initial extract for existing tables.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to Dashboard > Tables.

    1. Select the table(s) for which to redo the initial extract.

    2. Ensure that the primary key (PKey) column(s) are applied.

    3. Enable Delta Extract (With History).

    4. Enable Redo Initial Extract.

    5. Select Apply.

  3. Navigate to Dashboard > Operations and select Sync New Tables.

  4. Navigate to the Logs tab. Check the logs to confirm that the selected table(s) have started the full extract.

Perform a Rollback

Use the following steps to perform a rollback in Ingest.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to the Operations tab, select the gear icon, and then select Rollback.

  3. In the pop-up window, select the sequence number from which you want to replay the log files.

    Confirm that the log files are available on the source server.

  4. Select Select.

  5. Navigate to the Dashboard tab. Under Schedule, enable the scheduler.

Add New Tables to an Existing Pipeline

Use the following steps to add only new tables to existing pipelines.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to Dashboard > Tables:

    1. Select one or more new tables as required.

    2. Ensure the primary key (PKey) column(s) are applied.

    3. Enable Delta Extract (With History).

    4. Select Apply.

  3. Navigate to Dashboard > Operations and select Sync New Tables.

  4. Navigate to the Logs tab. Confirm that the existing tables were ignored and the new table(s) started to sync.

Create a New Pipeline

Use the following steps to create an additional Ingest pipeline.

  1. Unzip the Ingest ZIP file into a folder, such as C:\singlestore-flow\pipeline\.

  2. Edit the ingest.xml file located in the C:\singlestore-flow\pipeline\ folder.

  3. Change the <id>Bryteflow-Ingest</id> and <name>BryteFlow Ingest</name> tags to reference the new pipeline names.

  4. Check that port 8081 is free and not being used by other processes.

  5. Open a command prompt as Administrator, navigate to the new pipeline folder, run Install-Service.bat, and then close the command prompt.

  6. Open the Windows Services console and start the newly-created service.

  7. Launch the pipeline at localhost:8081.

  8. To avoid using port 8081 for future pipelines, SingleStore recommends changing the port to another preferred Web port by navigating to Settings > Instance Details > Web Port.

  9. Restart the Windows service and launch the Ingest application on the new Web port, such as localhost:8082.

  10. Ensure you have a valid license key to get started.

Enable Windows Authentication for SQL Server

Use the following steps to enable Windows authentication in Ingest.

  1. Stop the Ingest service.

  2. Download and copy the sqljdbc_auth.dll file to the bin directory of your Java installation. You can download the DLL from here or from Microsoft.

  3. Start the Ingest service and add the following JDBC option in the Source Database connection settings.

    integratedSecurity=true

Latency Calculation

Latency is calculated based on the time from when the record is committed to the source until it is available at the destination.

Prerequisites for Oracle Source

The Flow installation guide provides details on how to enable supplemental logging for source table(s) for delta replication. Refer to Install SingleStore Flow for more information.

Note:

  • If database supplemental logging is enabled, only inserts and deletes are captured, and updates are missed.

  • If table-level supplemental logging is enabled, only updates are captured, and inserts and deletes are missed.

Therefore, both database and table-level supplemental logging are required.

Synchronize Schema Changes Using Sync Struct

Use the following steps to perform a Sync Struct for existing tables when the source structure changes.

  1. Navigate to the Dashboard tab. Under Schedule, disable the scheduler and wait for the load to complete.

  2. Navigate to Dashboard > Operations and select Sync Struct.

  3. Navigate to the Logs tab to check the logs. Ingest compares the structures and synchronizes any differences.

Last modified: January 29, 2025

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK