# Source Database Setup * *Oracle* Flow supports on-premises Oracle database and Amazon Oracle RDS database as source databases.## Preparing On-Premises Oracle**Supported Oracle Versions**- Oracle 9i and above. - Oracle RAC 11g.**Enable Change Tracking on an On-Premises Oracle DB**1) Ensure the Oracle database is running in `ARCHIVELOG` mode. 2) Enable supplemental logging at the database level to ensure additional details are logged in the archive logs.  To enable full supplemental logging, run: ```sql ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ``` Alternatively, to enable minimal database supplemental logging, run: ```sql ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; ALTER DATABASE FORCE LOGGING; ``` 3) Enable supplemental logging at the table level. Run the following command for each relevant table: ```sql ALTER TABLE . ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ```## Preparing Oracle on Amazon RDS**Enable Change Tracking for Amazon Oracle RDS**1) Enable supplemental logging at the database level to ensure additional details are logged in the archive logs.  To enable supplemental logging at the database level, run: ```shell exec rdsadmin.rdsadmin_util.alter_supplemental_logging('ADD', 'ALL'); ``` 2) To retain archived redo logs on your DB instance (e.g., for 24 hours), run: ```shell exec rdsadmin.rdsadmin_util.set_configuration('archivelog retention hours', 24); ``` 3) To enable supplemental logging at the table level, run the following command for each relevant table: ```sql ALTER TABLE . ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ```## Grants Required for Oracle Source ConnectorThe Oracle user running Ingest must have the following privileges:- `SELECT` access on all tables to be replicated. - `SELECT` access on `V$ARCHIVED_LOG.`The following command must return records:```sql SELECT * FROM V$ARCHIVED_LOG; ```The user must also have the following security permissions::- `CREATE SESSION` - `SELECT` access on `V_$LOGMNR_CONTENTS` - `SELECT` access on `V_$LOGMNR_LOGS` - `SELECT` access on `ANY TRANSACTION` - `SELECT` access on `DBA_OBJECTS` - `EXECUTE` access on `DBMS_LOGMNR`Run the following grant commands for the user:```sql GRANT SELECT ON V_$ARCHIVED_LOG TO ; GRANT SELECT ON V_$LOGMNR_CONTENTS TO ; GRANT EXECUTE ON DBMS_LOGMNR TO ; GRANT SELECT ON V_$LOGMNR_LOGS TO ; GRANT SELECT ANY TRANSACTION TO ; GRANT SELECT ON DBA_OBJECTS TO ; GRANT execute_catalog_role TO ; GRANT LOGMINING TO ; -- Required for Oracle 12c and higher ```For Oracle 10g, create a public synonym for `DBMS_LOGMNR`:```sql CREATE PUBLIC SYNONYM DBMS_LOGMNR FOR SYS.DBMS_LOGMNR; ```If using Oracle 19c with continuous log mining, grant additional permissions:```sql GRANT SELECT ON V_$DATABASE TO ; GRANT SELECT ON V_$LOG TO ; GRANT SELECT ON V_$LOGFILE TO ; ```For Oracle Pluggable Databases (PDBs), ensure that all the above grants are run on the root database.## Verification of Oracle Source SetupTo verify that Oracle is correctly set up for change tracking, run the following commands and check the results.| **Condition to be checked** | **SQL Command** | **Expected Result** | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | Is ArchiveLog mode enabled? | `SELECT log_mode FROM V$DATABASE;` | `ARCHIVELOG` | | Is Supplemental logging enabled at the database level? | `SELECT supplemental_log_data_min FROM V$DATABASE;` | `YES` | | Is Supplemental Logging enabled at the table level? | `SELECT log_group_name, table_name, always, log_group_type FROM dba_log_groups;` | `, , ALWAYS, ALL COLUMN LOGGING` |## Data Types in OracleIngest supports most Oracle data types. The following table lists the supported data types:| BINARY\_DOUBLE | BINARY\_FLOAT | CHAR | | -------------- | ------------------------------ | -------- | | DATE | INTERVAL DAY TO SECOND | LONG | | LONG RAW | NCHAR | NUMBER | | NVARCHAR | RAW | REF | | TIMESTAMP | TIMESTAMP WITH LOCAL TIME ZONE | VARCHAR2 | * *Microsoft SQL Server* The SQL Server setup depends on the selected replication option: **Change Tracking** or **Change Data Capture**.Follow the recommended steps to set up your SQL Server source connector.## Enable Change Tracking on SQL ServerTo enable change tracking on a SQL Server database version that is later than 2008, perform the following steps.1) Install the Microsoft [bcp](https://learn.microsoft.com/en-us/sql/tools/bcp-utility) utility and the following drivers. 1. [VC++ 2017 64 bit](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) 2. [ODBC drivers 18 64 bit](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16) 3. [SQL CMD version 15 64 bit](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver15) > **📝 Note**: BCP may not work on Linux. If BCP is not available or cannot be used, use JDBC.To disable BCP, see workaround in [SingleStore Operational FAQ](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/#section-idm234981959696914.md). 2) To enable change tracking at the database level, run the following command for each database. ```sql ALTER DATABASE SET CHANGE_TRACKING = ON (CHANGE_RETENTION = 7 DAYS, AUTO_CLEANUP = ON); ``` 3) To enable change tracking at the table level, run the following command for each table. ```sql ALTER TABLE ENABLE CHANGE_TRACKING WITH (TRACK_COLUMNS_UPDATED = ON); ``` 4) Ensure that change tracking is enabled for all databases and tables to be replicated. 5) The Ingest database replication login user must have appropriate permissions to view change-tracking information. To grant the `VIEW` permission to event viewer logs, run: ```sql GRANT VIEW SERVER STATE TO ; ``` To allow the user to view change-tracking information at the table level, run: ```sql GRANT VIEW CHANGE TRACKING ON . TO ; ```**Note**: If you are configuring Ingest for a completely new SQL Server database, ensure that at least one transaction is performed on the database to generate the log sequence number. This is necessary for Ingest to start tracking the changes.## Enable Change Data Capture on SQL ServerTo enable change data capture on a SQL Server database version that is later than 2008, perform the following steps.1) Install the Microsoft [bcp](https://learn.microsoft.com/en-us/sql/tools/bcp-utility) utility and the following drivers. 1. [VC++ 2017 64 bit](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) 2. [ODBC drivers 18 64 bit](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16) 3. [SQL CMD version 15 64 bit](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver15) > **📝 Note**: BCP may not work on Linux. If BCP is not available or cannot be used, use JDBC.To disable BCP, see workaround in [SingleStore Operational FAQ](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest-operational-faq/#section-idm234981959696914.md). 2) To enable change data capture at the database level, run the following for each database. ```sql USE ; EXEC sys.sp_cdc_enable_db; ``` 3) To enable change data capture at the table level, run the following command for each table. ```sql USE GO EXEC sys.sp_cdc_enable_table @source_schema = ‘’, @source_name = ‘’, @role_name = null, @supports_net_changes = 0; ``` 4) By default, the data retention period for SQL Server CDC is three days, where the history of the changed data is only retained for three days. Anything over the three-day retention period is cleaned (deleted). To modify the default retention period, run: ```sql --Update minutes in the CDC job USE GO EXECUTE sys.sp_cdc_change_job @job_type = N'cleanup', @retention = @New_retention_period_in_minutes; GO ``` 5) Ensure that change data capture is enabled for all databases and tables to be replicated. 6) Use the Configuration Manager to turn on the SQL Server agent and confirm that the agent is running.## Grants Required for SQL ServerThe Ingest database replication login user must have `VIEW CHANGE TRACKING` permission to view the Change Tracking information.To review all change tracking tables (enabled or disabled), run:```sql SELECT * FROM sys.all_objects WHERE object_id IN (SELECT object_id FROM sys.change_tracking_tables WHERE is_track_columns_updated_on = 1); ```## Verification of SQL Server SourceTo verify if Change Tracking is already enabled on the database, run:```sql --Review all change tracking tables that are = 1 enabled, or = 0 disabled SELECT * FROM sys.change_tracking_databases WHERE database_id = DB_ID('databasename'); ```The following SQL command lists all tables with Change Tracking enabled for the selected database.```sql USE databasename; SELECT sys.schemas.name AS schema_name, sys.tables.name AS table_name FROM sys.change_tracking_tables JOIN sys.tables ON sys.tables.object_id = sys.change_tracking_tables.object_id JOIN sys.schemas ON sys.schemas.schema_id = sys.tables.schema_id; ```## Data Types in SQL ServerIngest supports most SQL Server data types. The following is a list of supported data types:| BIGINT | REAL | VARCHAR (max) | | ------------- | -------------- | ----------------- | | BIT | FLOAT | NCHAR | | DECIMAL | DATETIME | NVARCHAR (length) | | INT | DATETIME2 | NVARCHAR (max) | | MONEY | SMALLDATETIME | BINARY | | NUMERIC (p,s) | DATE | VARBINARY | | SMALLINT | TIME | VARBINARY (max) | | SMALLMONEY | DATETIMEOFFSET | TIMESTAMP | | TINYINT | CHAR | UNIQUEIDENTIFIER | | VARCHAR | HIERARCHYID | XML | * *MySQL* Flow supports on-premises MySQL and Amazon RDS MySQL databases as source databases.## Preparing On-Premises MySQL**Enable Change Tracking on an on-premises MySQL**1) To prepare MySQL for change tracking, enable binary logging. Configure the following parameters in the MySQL configuration file: 1. On MySQL for Windows: `my.ini` 2. On MySQL for Linux: `my.cnf` 2) Add or modify the following parameters:| **Parameter** | **Value** | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `server_id` | Any value starting from 1.E.g.`server_id`= 1 | | `log_bin = ` | Path to the binary log file.E.g. for Windows:`log_bin = D:\MySQLLogs\BinLog`E.g. for Linux:`log_bin = /var/lib/mysql/binlog` | | `binlog_format` | Set to row for row-based logging.E.g.`binlog_format = row` | | `expire_logs_days` | Number of days after which the binary log can be automatically removed.To avoid disk space issues it is strongly recommended not to use the default value (0).E.g.`expire_log_days = 4`**Note:**`expire_logs_days`is deprecated in MySQL 8.0 and removed in 8.2.0. | | `binlog_expire_logs_seconds` | Sets the binary log expiration period in seconds.E.g.`binlog_expire_logs_seconds = 604800`(for 7 days or 7 \* 24 \* 60 \* 60 seconds).**Note**: Use this for MySQL 8.2.0 and later. Replaces`expire_logs_days`. | | `binlog_checksum` | Set to none to disable checksums.E.g.`binlog_checksum = none`SingleStoreFlowsupports CRC32 too. | | `binlog_row_image` | Set to full for capturing all changes.E.g.`binlog_row_image = full` |## Preparing MySQL on Amazon RDS## Enabling Change Tracking on Amazon RDS MySQL1) In the AWS Management Console, create a new DB parameter group for MySQL. 2) Configure the following parameters. | **Parameter** | **Value** | | ----------------- | ----------------------------------------------------- | | `binlog_format` | `binlog_format=row` | | `binlog_checksum` | Set to`binlog_checksum=none`or`binlog_checksum=CRC32` | 3) Apply the new parameter group to the MySQL RDS DB instance. 4) Ensure that binary logging is enabled on the RDS instance for the change tracking setup to work properly.## Enabling Change Tracking on AWS Aurora MySQL Serverless v2## PrerequisitesEnsure the `mysqlbinlog` utility is available in your system’s `PATH` and matches your MySQL server version.## Steps1) In the AWS Management Console, create a new custom DB cluster parameter group: 1. Parameter group name - Enter a name for the group. 2. Description - Enter a description for the group. 3. Engine type - Select `Aurora MySQL` from the dropdown. 4. Parameter group family - Select `Aurora-mysql-8.0`. 5. Type - Select `DB cluster parameter group.` 2) Configure the following parameters in the newly created parameter group: | **Parameter** | **Value** | | ------------------ | ----------------------- | | `binlog_format` | `binlog_format=row` | | `binlog_checksum` | `binlog_checksum=CRC32` | | `binlog_row_image` | `binlog_row_image=full` | 3) Apply the parameter group to the Regional cluster and Writer cluster. 4) Restart the cluster for the changes to take effect.To check and validate the configuration:- Verify the version of Aurora and MySQL: ```sql SELECT @@aurora_version, VERSION() ``` - Show binary logs created by the Aurora cluster: ```sql SHOW BINARY LOGS; ``` - Verify the following variables: ```sql SHOW VARIABLES LIKE 'log_bin'; SHOW VARIABLES LIKE 'binlog_format'; SHOW VARIABLES LIKE 'binlog_checksum'; SHOW VARIABLES LIKE 'binlog_row_image'; ``` - To rotate binary logs if needed: ```sql FLUSH BINARY LOGS; ```If there are errors connecting to the source, CDC may fail. In such cases, restart the Flow service and perform a rollback if required. Set the scheduler time to a safe value such as one minute to minimize any issues.**Note**: These settings depend on AWS internal implementation. Any changes introduced by AWS may affect replication in the future.## Grants Required for MySQL Source ConnectorThe Ingest user must have the following privileges:- `REPLICATION CLIENT` - `REPLICATION SLAVE` - `SELECT` privileges on the source tables designated for replication.To grant the necessary permissions to a MySQL user, run the following commands:```sql CREATE USER 'bflow_ingest_user' IDENTIFIED BY ''; ``````sql GRANT SELECT, REPLICATION CLIENT, SHOW DATABASES ON *.* TO bflow_ingest_user; ``````sql GRANT SELECT, REPLICATION SLAVE, SHOW DATABASES ON *.* TO bflow_ingest_user; ```## Install MySQL Binary Log Utility (mysqlbinlog) for Change Data Capture (CDC)Flow requires access to the `mysqlbinlog` utility to perform CDC.**Installation Steps**1) Download and install `mysqlbinlog`.  Download the MySQL Server package and extract the zip file to a directory on the virtual machine (VM) where Flow is deployed. 2) Add `mysqlbinlog` to the system path.                                                                           Once installed, ensure the directory containing `mysqlbinlog` (typically the `bin` folder inside your MySQL installation path) is added to the system's `PATH` environment variable. Depending on your operating system, you may need to take additional steps to make the path setting persistent (e.g. updating .`bashrc` on Linux or system environment variables on Windows).To verify the installation, open a terminal or command prompt and run:``` mysqlbinlog -v ```This should print the version information.## Handling invalid DATE and DATETIME valuesWhen migrating from a MySQL source database, some rows may contain non-standard or invalid DATE or DATETIME values that are not supported by SingleStore. Flow automatically maps certain known invalid values into valid SingleStore values so that they can be loaded without failing the pipeline. For the exact mappings, refer to [MySQL to SingleStore Data Type Mapping.](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/data-type-mapping-in-singlestore-flow/mysql-to-singlestore-data-type-mapping/#section-id235431689525864.md) However, other invalid or out-of-range date and time values can still cause errors such as:```sql Error(PGT54): Table core_data.`mage_order` stage 3 error java.sql.SQLException: (conn=142441) Leaf Error (node-83201ac7-c278-4902-8df7-ab7902e5adb3-leaf-ag2-0.svc-83201ac7-c278-4902-a1f2-600f50d9b580:3306): Invalid DATE/TIME in type conversion for column 'col_datetime' ```SingleStore provides a global variable called `data_conversion_compatibility_level` that controls how strictly the database validates data type conversions. Lowering this setting results in less stringent validation and allows more values to be accepted during conversion.For example, the compatibility level can be temporarily set to an earlier version such as `6.5`:```sql SET GLOBAL data_conversion_compatibility_level = '6.5'; ```The following then executes successfully:```sql SELECT @@data_conversion_compatibility_level; DROP TABLE IF EXISTS date_tbl; CREATE TABLE date_tbl ( id INT PRIMARY KEY NOT NULL, d DATETIME ); INSERT date_tbl VALUES (1, '0001-01-01 07:07:24.000000'); SELECT * FROM date_tbl; ```For more details on `data_conversion_compatibility_level,` refer to [Data Type Conversion.](https://docs.singlestore.com/db/v9.1/reference/sql-reference/data-types/data-type-conversion.md)> **❗ Important**: By default, SingleStore recommends using the highest available `data_conversion_compatibility_level` to enforce strict validation. Lowering it can allow non‑standard or invalid data to be inserted across the entire cluster. Use this workaround only in exceptional cases (for example, short-term Flow migrations that must preserve these values), and restore the highest supported compatibility level as soon as possible. * *PostgreSQL* ## Preparing PostgreSQLTo ensure integration between Flow and PostgreSQL database, perform the following steps:1) Use a PostgreSQL database version 9.4.x or later. 2) Configure access permissions by adding the IP address of the Ingest machine to the `pg_hba.conf` configuration file, using the `replication` keyword in the database field. For example: `host replication all 189.452.1.212/32 trust` Ensure that the IP address is allowed to access the database for replication purposes. 3) Set the following parameters and values in the `postgresql.conf` configuration file: 1. `wal_level = logical` 2. `max_replication_slots`: Set this to a value greater than 1.  `max_replication_slots` defines the maximum number of replication slots available for logical replication. The value must match the number of tasks you want to run. For example, to run four tasks, set this to at least 4. Slots open automatically as soon as a task starts and remain open even when the task ends. You need to manually delete unused slots. 3. `max_wal_senders`: Set this to a value greater than 1. This parameter controls the number of concurrent replication tasks. 4. `wal_sender_timeout = 0 `: This parameter terminates replication connections that remain inactive for more than the specified time. Although the default is 60 seconds, SingleStore recommends setting this to 0 to disable the timeout mechanism. 4) Restart PostgreSQL for the changes to take effect. 5) Once the PostgreSQL configuration is updated, create a logical replication slot for the database to sync with Flow. Run the following command:  ```sql SELECT pg_create_logical_replication_slot('bflow_replication_slot', 'test_decoding'); ``` 6) Grant superuser permissions to the Flow user. Superuser privileges are necessary to access replication-specific functions. 7) Grant Replication role to the Flow user. 8) Grant ** Read-only** access for all the replicating tables to Flow user. * *Snowflake* ## Preparing SnowflakeThe Snowflake JDBC connector is generally available (GA) for production use, while all other JDBC connector sources are in a non-production preview\.While full extracts are supported, CDC is not available.To ensure integration between Flow and Snowflake, the source database user must have `SELECT` privileges on all tables that are to be migrated or replicated.To grant the necessary privileges, run the following command for each required table:```sql GRANT SELECT ON TABLE TO [USER | ROLE] ; ```where `` is the Ingest user configured to connect to Snowflake. * *Any JDBC-compliant Database* All other JDBC-compliant databases except Snowflake are in a non-production preview. They are still available for testing - feel free to reach out to [SingleStore Support](https://support.singlestore.com/hc/en-us) with any observations.## Using External Databases via a Custom JDBC DriverUse the following steps to set up an external database that does not have a dedicated source connector built into Flow:1) Download the JDBC driver JAR file from your database vendor.  2) Stop the Flow service if it's currently running, then restart with the JDBC driver in the classpath. Use the following command, replacing `` with your JDBC driver.   Note that JAR files are separated by a colon `(:)` on Linux and by a semicolon `(;)` on Windows. On Linux: ```shell java -Xss10M -cp ":ingest.jar" org.springframework.boot.loader.JarLauncher ``` On Windows: ```shell java -Xss10M -cp ";ingest.jar" org.springframework.boot.loader.JarLauncher ``` Example (Vertica): ```shell # Linux java -Xss10M -cp "vertica-jdbc-25.3.0-0.jar:ingest.jar" org.springframework.boot.loader.JarLauncher ``` ```shell # Windows  java -Xss10M -cp "vertica-jdbc-25.3.0-0.jar;ingest.jar" org.springframework.boot.loader.JarLauncher ```Refer to [Any Database Configuration](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/configure-source-database/#section-id235293551650634.md) for instructions on how to configure the database on the dashboard. * *MongoDB* MongoDB is currently in private preview. Reach out to [SingleStore Support](https://support.singlestore.com/hc/en-us) for the setup. * *Salesforce* > **📝 Note**: This is a Public Preview feature.To set up Salesforce as a source, create a **Connected App** in your org using the following steps:1) Log in to your Salesforce org. 2) Click the **Setup** gear icon in the top right. 3) In the **Quick Find** box, search for **App Manager**, open it, and select **New Connected App**. 4) Enter the following basic details: 1. **Connected App Name**: SingleStore Ingest App (or your preferred name) 2. **API Name**: Auto-populated 3. **Contact Email**: Your email address 5) Check **Enable OAuth Settings** and set the **Callback URL** as follows: * Production:  * Sandbox:  6) Select the following OAuth scopes: * Access and manage your data (api) * Perform requests on your behalf at any time (refresh\_token, offline\_access) * Full access (full) 7) Move the selected scopes from **Available OAuth Scopes** to **Selected OAuth Scopes**. 8) In additional settings, enable: * Require Secret for Web Server Flow * Require Secret for Refresh Token Flow * Enable Client Credentials Flow (optional, for server-to-server authentication)  9) Click **Save**, then **Continue**, and wait 2-10 minutes for the app to become active.## Get the Security TokenA security token is needed to configure the Salesforce source in the Flow UI.1) Go to **Setup** > **Personal Information** > **Reset My Security Token** 2) Security token will be sent to your email. Copy and save this token securely. 3) Append the token to your Salesforce password and use it as the password when configuring the source in the Flow UI.## Get the Consumer Key and Consumer SecretA consumer key and consumer secret is needed to configure the Salesforce source in the Flow UI.1) Go to **Setup** > **App Manager**.  2) Find your Connected App. 3) Click the dropdown arrow on the right and select **View**. 4) Click **Manage Consumer Details**. 5) Verify your identity by entering the verification code sent to your email. 6) Once verified, you will be able to view the **Consumer Key** and **Consumer Secret**. 7) Copy and save these values for use when configuring the source in the Flow UI. *** Modified at: March 4, 2026 Source: [/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/source-database-setup/](https://docs.singlestore.com/db/v9.1/load-data/load-data-with-singlestore-flow/singlestore-ingest/source-database-setup/) (An index of the documentation is available at /llms.txt)