Replicate MongoDB® Collections via AWS PrivateLink
On this page
SingleStore supports loading data from MongoDB® using CDC-in pipelines via AWS PrivateLink.
To replicate your MongoDB® databases using private links:
-
Request the VPC ID and Subnet ID for the AWS instance that hosts your SingleStore Helios deployment.
Contact SingleStore Support and provide the following information:
-
Workspace ID and name.
-
In the support ticket, specify that the request is for VPC ID and Subnet ID for the AWS instance that hosts this workspace.
-
-
Log in to MongoDB® Atlas and create a private endpoint service, select Network Access > Private Endpoint > Add Private Endpoint.
Refer to Configure Private Endpoints for more information. While creating a private endpoint service, on the Create a Private Endpoint dialog, select or enter the following:
-
Cloud Provider: Select AWS.
This feature is only supported for SingleStore Helios workspaces deployed using AWS, and it only supports connections to AWS PrivateLink. -
Atlas Region: Select the availability region of your SingleStore Helios deployment.
Note
Your deployment and private endpoint must be in the same region.
-
VPC ID and Subnet ID: Once the MongoDB® Atlas endpoint is in
Available
state, enter the VPC ID and Subnet IDs received from SingleStore in the VPC ID and Subnet IDs fields, respectively.
-
-
Select Network Access > Private Endpoint, and copy the Atlas Endpoint Service Name of your private endpoint service.
-
Log in to the Cloud Portal, and create an outbound connection from SingleStore.
While configuring the outbound connection, on the Create Connection dialog, enter or select the following information:
-
Endpoint: Select SingleStore Endpoint.
-
Service Name: Enter your MongoDB® Atlas Endpoint Service Name copied earlier.
Even though data is ingested into SingleStore, an outbound connection is used because the ingestion process is initiated by connecting from SingleStore Helios to the AWS instance/resource.
-
-
Ensure that the Prerequisites are met.
-
Copy the VPC Endpoint ID of your outbound connection from the Cloud Portal.
-
Select <your_
workspace_ group> > Firewall > Private Links. -
Under the Actions column of your private connection, select the three dots, and select View Connection.
-
From the View Connection dialog, copy the first 22 characters of the VPC Endpoint Service Name.
For example, if the VPC Endpoint Service name is vpce-xxxxxxxxxxxxxxxxx-vxxxxxx.
, copyvpce-svc-XXXX. vpce. amazonaws. com vpce-xxxxxxxxxxxxxxxxx
.This is the VPC Endpoint ID of your outbound connection. .
-
-
On MongoDB® Atlas:
-
Select <your_
endpoint> > Edit > Finalize Endpoint Connection. -
Under Your VPC Endpoint ID, enter the VPC Endpoint ID copied in the previous step, and then select Create.
-
Select Private Endpoint > Proceed to connect, and copy the connection string available for mongosh or Compass.
-
-
In SingleStore, create a link to the private endpoint.
Specify the connection string copied earlier using the mongodb.
parameter in the CREATE LINK or CREATE {TABLE | TABLES} AS INFER PIPELINE command.connection. string For example: CREATE LINK <link_name> AS MONGODBCONFIG '{"mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>""collection.include.list": "<source_db.source_collection>""mongodb.ssl.enabled":"true","mongodb.authsource":"admin"}'CREDENTIALS'{"mongodb.password": "<password>", "mongodb.user": "<user>"}';CREATE TABLE IF NOT EXISTS <table_name>AS INFER PIPELINE AS LOAD DATAMONGODB "source_db.source_collection"CONFIG '{"mongodb.connection.string": "<connection_string_in_the_mongodb+srv://cluster0-pl-0.xxxxx.mongodb.net/_format>","collection.include.list": "source_db.source_collection" }'CREDENTIALS '{"mongodb.user":"xxxx","mongodb.password":"xxxx"}'FORMAT AVRO; -
Create the required tables, pipelines, and stored procedures.
Note
If you are ingesting all the databases and collections, ensure that no two databases have collections with the same name in MongoDB®.
The following example uses the
LINK
clause.Refer to Replicate MongoDB® Collections using SQL for more information. CREATE TABLES AS INFER PIPELINE AS LOAD DATALINK <link_name> "*" FORMAT AVRO; -
Verify that the pipelines are created, and then start the pipelines to start the ingestion.
-- Verify that the pipelines are created --SHOW PIPELINES;-- Start the pipelines --START PIPELINE <pipeline_name>;To start all the pipelines, run the following command:
START ALL PIPELINES;
Related Topics
Last modified: May 12, 2025