Configuration of XL Ingest

Warning

SingleStore 9.0 gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.

Dashboard

Navigate to the Dashboard tab. The dashboard provides a summary of the data being managed and the work completed in the last 24 hours. It includes a pie chart showing task statuses and a bar chart indicating the amount of data transferred over the last 30 days.

Configuration

Navigate to the Configuration tab. Configure the source and destination database in Ingest. Navigate to http://localhost:8084. Although the setup is automatic, you must follow certain steps to transfer a large table to the desired destination. Set up XL Ingest by entering the following configuration details.

Location of CDC Installation	Enter the path for Ingest.
Web Port	Enter the port of XL Ingest (`8084`)
No. of source threads	Enter the number of source threads.
No. of destination threads	Enter the number of destination threads.
Product Id	By default, the product ID is the field.
Licence Key	Enter the license key.
Expiry	Displays the expiration date of your license.
Delete on Sync	Enable or disable as per your requirements.

Select Save to save the settings.

Table Setup

Table setup allows you to configure the tables.

Navigate to the Table Setup tab.
Select the tables from the list on the left to be processed in XL Ingest.
In the Table Settings page, choose the slicing column (typically the primary key). For partitioned tables, slicing is automatically handled by XL Ingest.
Select the Auto-slice button to automatically generate slices for large tables. Select the Schedule Count button to retrieve the number of records in the table, then enter the following auto-slice parameters:
1. No. of Slices: Number of slices to split the table into.
2. Round-off: Number of trailing significant digits to ignore when estimating the slice size. Simplifies the calculation by focusing only on the most significant digits of the total record count.
The Schedule Slice button is enabled only after the count is determined.

Table Status

After selecting and configuring the tables, the tables appear in the Table Status page.

Navigate to the Table Status tab.

It displays the status of each table and a bar graph indicating the amount of data transferred.
Select a table to view its slicing status and progress.

Table Slicing Status

The Table Slicing Status page displays the list of all slices or partitions of the table.

The following are the buttons for each slice to schedule:
1. Compare: Schedules a comparison of the slice between the source and destination database.
2. Force Compare: If compared earlier, forces a comparison of the slice between the source and destination databases.
3. Sync. Schedules the transfer of data between source and destination databases.
4. Force Sync: If the data has been synced previously, forces the transfer of the data between the source and destination databases.
The color of each button indicates the state of the respective task as follows:
1. Dark blue: Task can be scheduled and is recommended.
2. Light blue: Task can be scheduled but no urgent action is required.
3. Red: Task is scheduled and can be canceled before it runs, as required.
4. Grey: Task is in progress.

Schedule Tasks

XL Ingest tasks are long-running and cannot be executed immediately. They must be scheduled, and the progress is available after execution. Perform the following tasks:

Select Schedule Compare for the slice to start the slice comparison between the source and destination databases.
Select Schedule Sync for the slice to start the synchronization of data between the source and destination databases.
View the status of scheduled tasks in the Task Status page.

Task Status

The Task Status page displays the list of the tasks in the last 24 hours.

Navigate to the Task Status tab.
Tasks are categorized as follows:
1. Completed
2. Scheduled
3. Running
4. Failed
Tasks can be filtered and monitored for progress. You can cancel scheduled tasks before they start, as required.