Operations

On/Off toggle: Turns the scheduler on or off. When the scheduler is disabled, Ingest stops looking for changes in the source database.

If the scheduler state changes either manually or automatically (e.g., after a service restart or user change to the toggle), a message appears next to the it explaining the reason for the state change (e.g., "Manual on", "Restore on startup").

Same as the On/Off toggle in the Schedule section of the dashboard.
Full Extract: Trigger the initial bulk load for all the selected tables, except those marked as Skip Initial Extract. Every Ingest pipeline must undergo a full extract at least once to begin with.
Sync New Tables: Trigger the initial bulk load for tables marked as Redo Initial Extract and newly added tables in an ongoing replication. This operation cannot replace a full extract.
Sync Struct: Compare and fix the table structure in the destination database. Load fails if the schema changes. The operator identifies when the load started failing and rolls back to a point just before the initial failure. During this operation, Ingest compares and updates the structure of the destination table without losing data.
Rollback: Replay logs from a specific point to reapply changes to the destination. If the primary key columns are correctly selected, there should be no duplicates in the destination table. Rollback affects all tables in the selected list.
Manual Delta : Triggers a manual delta extract to process incremental changes.

Rollback

In the event of unexpected issues, such as intermittent source database outages or network connectivity problems, you can rollback Ingest to a point in time before the issue occurred and replay the changes.

To perform the rollback:

Navigate to the Schedule tab.
Select Rollback.
The rollback screen appears, displaying a list of available rollback points in descending order, depending on the source database log retention policy.
Select the desired date (radio button) and Select.
Select Rollback to initiate the rollback.
Ingest automatically catches up from the selected point in time to the current time that replays all log entries and applies them to the destination.

Sync Struct

Sync Struct compares and updates the table structure in the destination database. When schema changes are made to the source database, the replication may fail due to mismatches between the source and destination. Sync Struct resolves this by aligning the destination schema with the source, without losing data.

Note: Basic changes, such as adding a column, are generally handled successfully. However, significant changes (for example, altering a column type or size) may not be supported. In such cases, you may need to drop the destination table and reload all data using a Full Extract operation before resuming replication.

Planned Schema Upgrade with Sync Struct (Recommended)

To minimize risk of data loss during schema upgrades, use Sync Struct in a controlled manner. Plan the change when application updates are paused.

Follow these steps:

Pause updates from your application running on the source database.
Turn off the scheduler to prevent load failures while the schema is being modified.
Apply the necessary schema changes to the source database.
After the changes are complete, go to the Operations tab and select Sync Struct.
This synchronizes schema changes to the destination.
Wait for the Sync Struct process to complete successfully, which is indicated by the log message:
```
Delta key for <X> rolled back from <Y>.
```
Flow automatically rolls back to the last successful load file number and ensures that all changes are applied correctly, no manual rollback is required. Sync Struct resumes the scheduler when it completes.
Optionally, validate that the source and destination schema are in sync by running a SELECT COUNT (*) query in both databases, before and after the sync.
Resume updates from your application.

Example: Adding a column to a MySQL Table

Consider a table named employees being replicated to the destination.

Turn off the scheduler.

Alter the table on the source:

SQL

ALTER TABLE employees ADD COLUMN salary DECIMAL(10,2);

On the Operations tab, select Sync Struct. (This automatically turns the scheduler back on.)

Destination Schema Comparison

The following queries show the destination schema before and after the sync for comparison.

Before Sync Struct

SQL

SHOW CREATE TABLE employees;

+---------------------------------+-------+--------+-------------+-----------------+
|CREATE TABLE `employees` (                                                        |
|  `employeeNumber` int(11) NOT NULL DEFAULT '0',                                  |
| `lastName` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,   |
|  `firstName` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, |
|  `extension_` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,|
|  `email` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,    |
| `officeCode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, |
|  `reportsTo` int(11) DEFAULT NULL,                                               |
|  `jobTitle` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,  |
|  PRIMARY KEY (`employeeNumber`),                                                 |
|  SHARD KEY `__SHARDKEY` (`employeeNumber`),                                      |
|  SORT KEY `__UNORDERED` ()                                                       |
| )                                                                                |
+---------------------------------+-------+--------+-------------+-----------------+

After Sync Struct

The newly added column salary is present in the following output:

SQL

SHOW CREATE TABLE employees;

+---------------------------------+-------+--------+-------------+-----------------+
|CREATE TABLE `employees` (                                                        |
|  `employeeNumber` int(11) NOT NULL DEFAULT '0',                                  |
|  `lastName` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,  |
|  `firstName` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, |
|  `extension_` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,|
|  `email` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,    |
|  `officeCode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,|
|  `reportsTo` int(11) DEFAULT NULL,                                               |
|  `jobTitle` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,  |
|  `salary` decimal(10,2) DEFAULT NULL,                                            |
|  PRIMARY KEY (`employeeNumber`),                                                 |
|  SHARD KEY `__SHARDKEY` (`employeeNumber`),                                      |
| SORT KEY `__UNORDERED` ()                                                        |
)                                                                                  |
+---------------------------------+-------+--------+-------------+-----------------+

Handling Schema Changes during CDC

If the source schema is changed while CDC is ongoing, the load may fail due to the schema mismatch. Flow retries the load a few times, but if the issue persists, the scheduler is automatically turned off to prevent further failures.

If you encounter errors, follow these steps:

If the scheduler is not turned off, turn it off and wait for the load to complete.
On the Operations tab, select Sync Struct and wait for the process to complete successfully. Sync Struct automatically rolls back the CDC by one extract file (current - 1).
If errors persist, manually roll back CDC two or three files before the failed extract file:
1. Identify the failed extract file number by navigating to the log folder in your Ingest installation directory and opening sirus-YYYY-MM.log.
2. On the Operations tab, select Rollback, then select the file number to rollback to (usually n - 2 or n - 3 where n is the failed extract file).
3. Start the rollback.
If a significant amount of time has passed or the log file is unavailable, trigger a Redo Initial Extract to sync the table.