# Source Database Setup ## Network Configuration for Source Databases To enable Flow on Helios to access your source database: 1. Identify the outbound IP addresses of your SingleStore Helios workspace. 2. Add these IPs to the inbound firewall rules or allowed IP list of your source database. 3. Test the connection from Flow to your source database to confirm successful access. * *Oracle* Flow supports on-premises Oracle database and Amazon Oracle RDS database as source databases.## Preparing On-Premises Oracle**Supported Oracle Versions** - Oracle 9i and above. - Oracle RAC 11g.**Enable Change Tracking on an On-Premises Oracle DB** 1) Ensure the Oracle database is running in `ARCHIVELOG` mode. 2) Enable supplemental logging at the database level to ensure additional details are logged in the archive logs.  To enable full supplemental logging, run: ```sql ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ``` Alternatively, to enable minimal database supplemental logging, run: ```sql ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; ALTER DATABASE FORCE LOGGING; ``` 3) Enable supplemental logging at the table level. Run the following command for each relevant table: ```sql ALTER TABLE . ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ```## Preparing Oracle on Amazon RDS**Enable Change Tracking for Amazon Oracle RDS** 1) Enable supplemental logging at the database level to ensure additional details are logged in the archive logs.  To enable supplemental logging at the database level, run: ```shell exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD', 'ALL'); ``` 2) To retain archived redo logs on your DB instance (e.g., for 24 hours), run: ```shell exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours', 24); ``` 3) To enable supplemental logging at the table level, run the following command for each relevant table: ```sql ALTER TABLE . ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ```## Grants Required for Oracle Source ConnectorThe Oracle user running Ingest must have the following privileges:- `SELECT` access on all tables to be replicated. - `SELECT` access on `V$ARCHIVED_LOG.`The following command must return records:```sql SELECT * FROM V$ARCHIVED_LOG; ```The user must also have the following security permissions::- `CREATE SESSION`  - `SELECT` access on `V_$LOGMNR_CONTENTS` - `SELECT` access on `V_$LOGMNR_LOGS` - `SELECT` access on `ANY TRANSACTION` - `SELECT` access on `DBA_OBJECTS` - `EXECUTE` access on `DBMS_LOGMNR`Run the following grant commands for the user:```sql GRANT SELECT ON V_$ARCHIVED_LOG TO ; GRANT SELECT ON V_$LOGMNR_CONTENTS TO ; GRANT EXECUTE ON DBMS_LOGMNR TO ; GRANT SELECT ON V_$LOGMNR_LOGS TO ; GRANT SELECT ANY TRANSACTION TO ; GRANT SELECT ON DBA_OBJECTS TO ; GRANT execute_catalog_role TO ; GRANT LOGMINING TO ; -- Required for Oracle 12c and higher ```For Oracle 10g, create a public synonym for `DBMS_LOGMNR`:```sql CREATE PUBLIC SYNONYM DBMS_LOGMNR FOR SYS.DBMS_LOGMNR; ```If using Oracle 19c with continuous log mining, grant additional permissions:```sql GRANT SELECT ON V_$DATABASE TO ; GRANT SELECT ON V_$LOG TO ; GRANT SELECT ON V_$LOGFILE TO ; ```For Oracle Pluggable Databases (PDBs), ensure that all the above grants are run on the root database.## Verification of Oracle Source SetupTo verify that Oracle is correctly set up for change tracking, run the following commands and check the results.| **Condition to be checked** | **SQL Command** | **Expected Result** | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | Is ArchiveLog mode enabled? | `SELECT log_mode FROM V$DATABASE;` | `ARCHIVELOG` | | Is Supplemental logging enabled at the database level? | `SELECT supplemental_log_data_min FROM V$DATABASE;` | `YES` | | Is Supplemental Logging enabled at the table level? | `SELECT log_group_name, table_name, always, log_group_type FROM dba_log_groups;` | `, , ALWAYS, ALL COLUMN LOGGING` |## Data Types in OracleIngest supports most Oracle data types. The following table lists the supported data types:| BINARY\_DOUBLE | BINARY\_FLOAT | CHAR | | -------------- | ------------------------------ | -------- | | DATE | INTERVAL DAY TO SECOND | LONG | | LONG RAW | NCHAR | NUMBER | | NVARCHAR | RAW | REF | | TIMESTAMP | TIMESTAMP WITH LOCAL TIME ZONE | VARCHAR2 | * *Microsoft SQL Server* The SQL Server setup depends on the selected replication option: **Change Tracking** or **Change Data Capture**.Follow the recommended steps to set up your SQL Server source connector.## Enable Change Tracking on SQL ServerTo enable change tracking on a SQL Server database version that is later than 2008, perform the following steps.1) Install the Microsoft [bcp](https://learn.microsoft.com/en-us/sql/tools/bcp-utility) utility and the following drivers. 1. [VC++ 2017 64 bit](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170)  2. [ODBC drivers 18 64 bit](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16)  3. [SQL CMD version 15 64 bit](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver15)  > **📝 Note**: BCP may not work on Linux. If BCP is not available or cannot be used, use JDBC.To disable BCP, see workaround in [SingleStore Operational FAQ](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/#section-idm234981959696914.md). 2) To enable change tracking at the database level, run the following command for each database. ```sql ALTER DATABASE SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 7 DAYS, AUTO_CLEANUP = ON); ``` 3) To enable change tracking at the table level, run the following command for each table. ```sql ALTER TABLE ENABLE CHANGE_TRACKING WITH (TRACK_COLUMNS_UPDATED = ON); ``` 4) Ensure that change tracking is enabled for all databases and tables to be replicated. 5) The Ingest database replication login user must have appropriate permissions to view change-tracking information. To grant the `VIEW` permission to event viewer logs, run: ```sql GRANT VIEW SERVER STATE TO ; ``` To allow the user to view change-tracking information at the table level, run: ```sql GRANT VIEW CHANGE TRACKING ON . TO ; ```**Note**: If you are configuring Ingest for a completely new SQL Server database, ensure that at least one transaction is performed on the database to generate the log sequence number. This is necessary for Ingest to start tracking the changes.## Enable Change Data Capture on SQL ServerTo enable change data capture on a SQL Server database version that is later than 2008, perform the following steps.1) Install the Microsoft [bcp](https://learn.microsoft.com/en-us/sql/tools/bcp-utility) utility and the following drivers. 1. [VC++ 2017 64 bit](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170)  2. [ODBC drivers 18 64 bit](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16)  3. [SQL CMD version 15 64 bit](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver15)  > **📝 Note**: BCP may not work on Linux. If BCP is not available or cannot be used, use JDBC.To disable BCP, see workaround in [SingleStore Operational FAQ](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/#section-idm234981959696914.md). 2) To enable change data capture at the database level, run the following for each database. ```sql USE ; EXEC sys.sp_cdc_enable_db; ``` 3) To enable change data capture at the table level, run the following command for each table. ```sql USE GO EXEC sys.sp_cdc_enable_table @source_schema = ‘’, @source_name = ‘’, @role_name = null, @supports_net_changes = 0; ``` 4) By default, the data retention period for SQL Server CDC is three days, where the history of the changed data is only retained for three days. Anything over the three-day retention period is cleaned (deleted). To modify the default retention period, run: ```sql --Update minutes in the CDC job USE GO EXECUTE sys.sp_cdc_change_job @job_type = N'cleanup', @retention = @New_retention_period_in_minutes; GO ``` 5) Ensure that change data capture is enabled for all databases and tables to be replicated. 6) Use the Configuration Manager to turn on the SQL Server agent and confirm that the agent is running.## Grants Required for SQL ServerThe Ingest database replication login user must have `VIEW CHANGE TRACKING` permission to view the Change Tracking information.To review all change tracking tables (enabled or disabled), run:```sql SELECT * FROM sys.all_objects WHERE object_id IN (SELECT object_id FROM sys.change_tracking_tables WHERE is_track_columns_updated_on = 1); ```## Verification of SQL Server SourceTo verify if Change Tracking is already enabled on the database, run:```sql --Review all change tracking tables that are = 1 enabled, or = 0 disabled SELECT * FROM sys.change_tracking_databases WHERE database_id = DB_ID('databasename'); ```The following SQL command lists all tables with Change Tracking enabled for the selected database.```sql USE databasename; SELECT sys.schemas.name AS schema_name, sys.tables.name AS table_name FROM sys.change_tracking_tables JOIN sys.tables ON sys.tables.object_id = sys.change_tracking_tables.object_id JOIN sys.schemas ON sys.schemas.schema_id = sys.tables.schema_id; ```## Data Types in SQL ServerIngest supports most SQL Server data types. The following is a list of supported data types:| BIGINT | REAL | VARCHAR (max) | | ------------- | -------------- | ----------------- | | BIT | FLOAT | NCHAR | | DECIMAL | DATETIME | NVARCHAR (length) | | INT | DATETIME2 | NVARCHAR (max) | | MONEY | SMALLDATETIME | BINARY | | NUMERIC (p,s) | DATE | VARBINARY | | SMALLINT | TIME | VARBINARY (max) | | SMALLMONEY | DATETIMEOFFSET | TIMESTAMP | | TINYINT | CHAR | UNIQUEIDENTIFIER | | VARCHAR | HIERARCHYID | XML | * *MySQL* Flow supports on-premises MySQL and Amazon RDS MySQL databases as source databases.## Preparing On-Premises MySQL## Enable Change Tracking on an on-premises MySQL1) To prepare MySQL for change tracking, enable binary logging. Configure the following parameters in the MySQL configuration file: 1. On MySQL for Windows: `my.ini` 2. On MySQL for Linux: `my.cnf` 2) Add or modify the following parameters: | **Parameter** | **Value** | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `server_id` | Any value starting from 1.E.g.`server_id`= 1 | | `log_bin = ` | Path to the binary log file.E.g. for Windows:`log_bin = D:\MySQLLogs\BinLog`E.g. for Linux:`log_bin = /var/lib/mysql/binlog` | | `binlog_format` | Set to row for row-based logging.E.g.`binlog_format = row` | | `expire_logs_days` | Number of days after which the binary log can be automatically removed.To avoid disk space issues it is strongly recommended not to use the default value (0).E.g.`expire_log_days = 4`**Note:**`expire_logs_days`is deprecated in MySQL 8.0 and removed in 8.2.0. | | `binlog_expire_logs_seconds` | Sets the binary log expiration period in seconds.E.g.`binlog_expire_logs_seconds = 604800`(for 7 days or 7 \* 24 \* 60 \* 60 seconds).**Note**: Use this for MySQL 8.2.0 and later. Replaces`expire_logs_days`. | | `binlog_checksum` | Set to none to disable checksums.E.g.`binlog_checksum = none`SingleStoreFlowsupports CRC32 too. | | `binlog_row_image` | Set to full for capturing all changes.E.g.`binlog_row_image = full` |## Preparing MySQL on Amazon RDS## Enabling Change Tracking on Amazon RDS MySQL1) In the AWS Management Console, create a new DB parameter group for MySQL. 2) Configure the following parameters. | **Parameter** | **Value** | | ----------------- | ----------------------------------------------------- | | `binlog_format` | `binlog_format=row` | | `binlog_checksum` | Set to`binlog_checksum=none`or`binlog_checksum=CRC32` | 3) Apply the new parameter group to the MySQL RDS DB instance. 4) Ensure that binary logging is enabled on the RDS instance for the change tracking setup to work properly.## Enabling Change Tracking on AWS Aurora MySQL Serverless v2## PrerequisitesEnsure the `mysqlbinlog` utility is available in your system’s `PATH` and matches your MySQL server version.## Steps1) In the AWS Management Console, create a new custom DB cluster parameter group: 1. Parameter group name - Enter a name for the group. 2. Description - Enter a description for the group. 3. Engine type - Select `Aurora MySQL` from the dropdown. 4. Parameter group family - Select `Aurora-mysql-8.0`. 5. Type - Select `DB cluster parameter group.` 2) Configure the following parameters in the newly created parameter group: | **Parameter** | **Value** | | ------------------ | ----------------------- | | `binlog_format` | `binlog_format=row` | | `binlog_checksum` | `binlog_checksum=CRC32` | | `binlog_row_image` | `binlog_row_image=full` | 3) Apply the parameter group to the Regional cluster and Writer cluster. 4) Restart the cluster for the changes to take effect.To check and validate the configuration:- Verify the version of Aurora and MySQL: ``` SELECT @@aurora_version, VERSION() ``` - Show binary logs created by the Aurora cluster: ``` SHOW BINARY LOGS; ``` - Verify the following variables: ``` SHOW VARIABLES LIKE 'log_bin'; SHOW VARIABLES LIKE 'binlog_format'; SHOW VARIABLES LIKE 'binlog_checksum'; SHOW VARIABLES LIKE 'binlog_row_image'; ``` - To rotate binary logs if needed: ``` FLUSH BINARY LOGS; ```If there are errors connecting to the source, CDC may fail. In such cases, restart the Flow service and perform a rollback if required. Set the scheduler time to a safe value such as one minute to minimize any issues.**Note**: These settings depend on AWS internal implementation. Any changes introduced by AWS may affect replication in the future.## Grants Required for MySQL Source ConnectorThe Ingest user must have the following privileges:- `REPLICATION CLIENT`  - `REPLICATION SLAVE`  - `SELECT` privileges on the source tables designated for replication.To grant the necessary permissions to a MySQL user, run the following commands:```sql CREATE USER 'bflow_ingest_user' IDENTIFIED BY ''; ``````sql GRANT SELECT, REPLICATION CLIENT, SHOW DATABASES ON *.* TO bflow_ingest_user; ``````sql GRANT SELECT, REPLICATION SLAVE, SHOW DATABASES ON *.* TO bflow_ingest_user; ```## Install MySQL Binary Log Utility (mysqlbinlog) for Change Data Capture (CDC)Flow requires access to the `mysqlbinlog` utility to perform CDC.**Installation Steps** 1) Download and install `mysqlbinlog`.  Download the MySQL Server package and extract the zip file to a directory on the virtual machine (VM) where Flow is deployed. 2) Add `mysqlbinlog` to the system path.                                                                           Once installed, ensure the directory containing `mysqlbinlog` (typically the `bin` folder inside your MySQL installation path) is added to the system's `PATH` environment variable. Depending on your operating system, you may need to take additional steps to make the path setting persistent (e.g. updating .`bashrc` on Linux or system environment variables on Windows).To verify the installation, open a terminal or command prompt and run:``` mysqlbinlog -v ```This should print the version information. * *PostgreSQL* ## Preparing PostgreSQLTo ensure integration between Flow and PostgreSQL database, perform the following steps:1) Use a PostgreSQL database version 9.4.x or later. 2) Configure access permissions by adding the IP address of the Ingest machine to the `pg_hba.conf` configuration file, using the `replication` keyword in the database field. For example: `host replication all 189.452.1.212/32 trust`  Ensure that the IP address is allowed to access the database for replication purposes. 3) Set the following parameters and values in the `postgresql.conf` configuration file: 1. `wal_level = logical`  2. `max_replication_slots`: Set this to a value greater than 1.  `max_replication_slots` defines the maximum number of replication slots available for logical replication. The value must match the number of tasks you want to run. For example, to run four tasks, set this to at least 4. Slots open automatically as soon as a task starts and remain open even when the task ends. You need to manually delete unused slots. 3. `max_wal_senders`: Set this to a value greater than 1. This parameter controls the number of concurrent replication tasks. 4. `wal_sender_timeout = 0 `: This parameter terminates replication connections that remain inactive for more than the specified time. Although the default is 60 seconds, SingleStore recommends setting this to 0 to disable the timeout mechanism. 4) Restart PostgreSQL for the changes to take effect. 5) Once the PostgreSQL configuration is updated, create a logical replication slot for the database to sync with Flow. Run the following command:  ```sql SELECT pg_create_logical_replication_slot('bflow_replication_slot', 'test_decoding'); ``` 6) Grant superuser permissions to the Flow user. Superuser privileges are necessary to access replication-specific functions. 7) Grant Replication role to the Flow user. 8) Grant ** Read-only** access for all the replicating tables to Flow user. * *Snowflake* ## Preparing SnowflakeThe Snowflake JDBC connector is generally available (GA) for production use, while all other JDBC connector sources are in a non-production preview\.While full extracts are supported, CDC is not available.To ensure integration between Flow and Snowflake, the source database user must have `SELECT` privileges on all tables that are to be migrated or replicated.To grant the necessary privileges, run the following command for each required table:```sql GRANT SELECT ON TABLE TO [USER | ROLE] ; ```where `` is the Ingest user configured to connect to Snowflake. * *Any JDBC-compliant Database* All JDBC-compliant databases except Snowflake are in a non-production preview. They are still available for testing - feel free to reach out to [SingleStore Support](https://support.singlestore.com/hc/en-us) with any observations. * *MongoDB* MongoDB is currently in private preview. Reach out to [SingleStore Support](https://support.singlestore.com/hc/en-us) for the setup. *** Modified at: December 12, 2025 Source: [/db/v9.1/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/source-database-setup/](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow-on-helios/singlestore-ingest/source-database-setup/) (An index of the documentation is available at /llms.txt)