Replicate MongoDB® Collections via AWS PrivateLink

On this page

SingleStore supports loading data from MongoDB® using CDC-in pipelines via AWS PrivateLink.

To replicate your MongoDB® databases using private links:

  1. Request the VPC ID and Subnet ID for the AWS instance that hosts your SingleStore Helios deployment.

    Contact SingleStore Support and provide the following information:

    • Workspace ID and name.

    • In the support ticket, specify that the request is for VPC ID and Subnet ID for the AWS instance that hosts this workspace.

  2. Log in to MongoDB® Atlas and create a private endpoint service, select Network Access > Private Endpoint > Add Private Endpoint. Refer to Configure Private Endpoints for more information.

    While creating a private endpoint service, on the Create a Private Endpoint dialog, select or enter the following:

    1. Cloud Provider: Select AWS. This feature is only supported for SingleStore Helios workspaces deployed using AWS, and it only supports connections to AWS PrivateLink.

    2. Atlas Region: Select the availability region of your SingleStore Helios deployment.

      Note

      Your deployment and private endpoint must be in the same region.

    3. VPC ID and Subnet ID: Once the MongoDB® Atlas endpoint is in Available state, enter the VPC ID and Subnet IDs received from SingleStore in the VPC ID and Subnet IDs fields, respectively.

  3. Select Network Access > Private Endpoint, and copy the Atlas Endpoint Service Name of your private endpoint service.

  4. Log in to the Cloud Portal, and create an outbound connection from SingleStore.

    While configuring the outbound connection, on the Create Connection dialog, enter or select the following information:

    1. Endpoint: Select SingleStore Endpoint.

    2. Service Name: Enter your MongoDB® Atlas Endpoint Service Name copied earlier.

    Even though data is ingested into SingleStore, an outbound connection is used because the ingestion process is initiated by connecting from SingleStore Helios to the AWS instance/resource.

  5. Ensure that the Prerequisites are met.

  6. Copy the VPC Endpoint ID of your outbound connection from the Cloud Portal.

    1. Select <your_workspace_group> > Firewall > Private Links.

    2. Under the Actions column of your private connection, select the three dots, and select View Connection.

    3. From the View Connection dialog, copy the first 22 characters of the VPC Endpoint Service Name. For example, if the VPC Endpoint Service name is vpce-xxxxxxxxxxxxxxxxx-vxxxxxx.vpce-svc-XXXX.vpce.amazonaws.com, copy vpce-xxxxxxxxxxxxxxxxx. This is the VPC Endpoint ID of your outbound connection..

  7. On MongoDB® Atlas:

    1. Select <your_endpoint> > Edit > Finalize Endpoint Connection.

    2. Under Your VPC Endpoint ID, enter the VPC Endpoint ID copied in the previous step, and then select Create.

    3. Select Private Endpoint > Proceed to connect, and copy the connection string available for mongosh or Compass.

  8. In SingleStore, create a link to the private endpoint. Specify the connection string copied earlier using the mongodb.connection.string parameter in the CREATE LINK or CREATE {TABLE | TABLES} AS INFER PIPELINE command. For example:

    CREATE LINK <link_name> AS MONGODB
    CONFIG '{
    "mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>"
    "collection.include.list": "<source_db.source_collection>"
    "mongodb.ssl.enabled":"true",
    "mongodb.authsource":"admin"}'
    CREDENTIALS
    '{"mongodb.password": "<password>", "mongodb.user": "<user>"}';
    CREATE TABLE IF NOT EXISTS <table_name>
    AS INFER PIPELINE AS LOAD DATA
    MONGODB "source_db.source_collection"
    CONFIG '{
    "mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>",
    "collection.include.list": "source_db.source_collection" }'
    CREDENTIALS '{
    "mongodb.user":"xxxx",
    "mongodb.password":"xxxx"}'
    FORMAT AVRO;
  9. Create the required tables, pipelines, and stored procedures.

    Note

    If you are ingesting all the databases and collections, ensure that no two databases have collections with the same name in MongoDB®.

    The following example uses the LINK clause. Refer to Replicate MongoDB® Collections using SQL for more information.

    CREATE TABLES AS INFER PIPELINE AS LOAD DATA
    LINK <link_name> "*" FORMAT AVRO;
  10. Verify that the pipelines are created, and then start the pipelines to start the ingestion.

    -- Verify that the pipelines are created --
    SHOW PIPELINES;
    -- Start the pipelines --
    START PIPELINE <pipeline_name>;

    To start all the pipelines, run the following command:

    START ALL PIPELINES;

Last modified: May 12, 2025

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK