SingleStore supports loading data from MongoDB® using CDC-in pipelines via AWS PrivateLink.

To replicate your MongoDB® databases using private links:

1. Request the VPC ID and Subnet ID for the AWS instance that hosts your SingleStore Helios deployment.

   Contact SingleStore Support and provide the following information:

   • Workspace ID and name.

   • In the support ticket, specify that the request is for VPC ID and Subnet ID for the AWS instance that hosts this workspace.

2. Log in to MongoDB® Atlas and create a private endpoint service, select Network Access > Private Endpoint > Add Private Endpoint. Refer to Configure Private Endpoints for more information.

   While creating a private endpoint service, on the Create a Private Endpoint dialog, select or enter the following:

   1. Cloud Provider: Select AWS. This feature is only supported for SingleStore Helios workspaces deployed using AWS, and it only supports connections to AWS PrivateLink.

   2. Atlas Region: Select the availability region of your SingleStore Helios deployment.

      Note

      Your deployment and private endpoint must be in the same region.

   3. VPC ID and Subnet ID: Once the MongoDB® Atlas endpoint is in Available state, enter the VPC ID and Subnet IDs received from SingleStore in the VPC ID and Subnet IDs fields, respectively.

3. Select Network Access > Private Endpoint, and copy the Atlas Endpoint Service Name of your private endpoint service.

4. Log in to the Cloud Portal, and create an outbound connection from SingleStore.

   While configuring the outbound connection, on the Create Connection dialog, enter or select the following information:

   1. Endpoint: Select SingleStore Endpoint.

   2. Service Name: Enter your MongoDB® Atlas Endpoint Service Name copied earlier.

   Even though data is ingested into SingleStore, an outbound connection is used because the ingestion process is initiated by connecting from SingleStore Helios to the AWS instance/resource.

5. Ensure that the Prerequisites are met.

6. Copy the VPC Endpoint ID of your outbound connection from the Cloud Portal.

   1. Select <your_workspace_group> > Firewall > Private Links.

   2. Under the Actions column of your private connection, select the three dots, and select View Connection.

   3. From the View Connection dialog, copy the first 22 characters of the VPC Endpoint Service Name. For example, if the VPC Endpoint Service name is vpce-xxxxxxxxxxxxxxxxx-vxxxxxx.vpce-svc-XXXX.vpce.amazonaws.com, copy vpce-xxxxxxxxxxxxxxxxx. This is the VPC Endpoint ID of your outbound connection..

7. On MongoDB® Atlas:

   1. Select <your_endpoint> > Edit > Finalize Endpoint Connection.

   2. Under Your VPC Endpoint ID, enter the VPC Endpoint ID copied in the previous step, and then select Create.

   3. Select Private Endpoint > Proceed to connect, and copy the connection string available for mongosh or Compass.

8. In SingleStore, create a link to the private endpoint. Specify the connection string copied earlier using the mongodb.connection.string parameter in the CREATE LINK or CREATE {TABLE | TABLES} AS INFER PIPELINE command. For example:

   SQL

   Copy

   CREATE LINK <link_name> AS MONGODB
     CONFIG '{
       "mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>"
       "collection.include.list": "<source_db.source_collection>"
       "mongodb.ssl.enabled":"true",
       "mongodb.authsource":"admin"}' 
     CREDENTIALS 
       '{"mongodb.password": "<password>", "mongodb.user": "<user>"}';

   SQL

   Copy

   CREATE TABLE IF NOT EXISTS <table_name> 
   AS INFER PIPELINE AS LOAD DATA 
   MONGODB "source_db.source_collection"
   CONFIG '{
       "mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>",
       "collection.include.list": "source_db.source_collection" }'
   CREDENTIALS '{
       "mongodb.user":"xxxx",
       "mongodb.password":"xxxx"}'
   FORMAT AVRO;

9. Create the required tables, pipelines, and stored procedures.

   Note

   If you are ingesting all the databases and collections, ensure that no two databases have collections with the same name in MongoDB®.

   The following example uses the LINK clause. Refer to Replicate MongoDB® Collections using SQL for more information.

   SQL

   Copy

   CREATE TABLES AS INFER PIPELINE AS LOAD DATA 
     LINK <link_name> "*" FORMAT AVRO;

10. Verify that the pipelines are created, and then start the pipelines to start the ingestion.

    SQL

    Copy

    -- Verify that the pipelines are created --
    SHOW PIPELINES;

    SQL

    Copy

    -- Start the pipelines --
    START PIPELINE <pipeline_name>;

    To start all the pipelines, run the following command:

    SQL

    Copy

    START ALL PIPELINES;