Single PrivateLink and NLB

This configuration uses a single Network Load Balancer (NLB) with multiple listeners to forward traffic to each Kafka broker. Perform the following steps:

1. Create one target group per broker. Each target group should:

   1. Forward traffic to the broker’s IP address.

   2. Use the appropriate port (e.g., 6001, 6002, or 6003) based on the security protocol in use.

2. Create a single Network Load Balancer. Only one NLB is required for this setup, and it will be used to route traffic to all Kafka brokers.

   

3. For each target group, create a listener on a unique port (e.g., 6001, 6002, 6003) that forwards to the target group.

   

4. Create an endpoint service pointing to the load balancer as shown above.

   

5. Whitelist the AWS SingleStore account shared by SingleStore to the allowed principals for the endpoint service.

   

Once the PrivateLink service has been created, use it to configure the outbound connection in SingleStore.

1. Create an Outbound Private Endpoint

   1. In the SingleStore Cloud Portal, navigate to the Firewall tab in your Helios workspace.

      

   2. Create an Outbound Private Endpoint using the PrivateLink service created in the previous step.

2. Create the Pipeline

   1. After the endpoint is active, configure your Kafka pipeline to use it.

   2. Use the appropriate port in the LOAD DATA KAFKA command based on the security protocol. For SASL/SCRAM, use port 9096.

   3. In the spoof.dns section, map each broker’s hostname and port to the outbound private endpoint and corresponding NLB listener port (e.g., 6001, 6002, 6003).

      SQL

      Copy

      CREATE OR REPLACE PIPELINE kafka_munis_poc_pipeline AS
      LOAD DATA KAFKA '<broker-1>:9096,<broker-2>:9096,<broker-3>:9096/<kafka-topic>'
      CONFIG '{
        "spoof.dns": {
          "<broker-1>:9096": "<vpc-endpoint>:6001",
          "<broker-2>:9096": "<vpc-endpoint>:6002",
          "<broker-3>:9096": "<vpc-endpoint>:6003"
        },
        "sasl.username": "<REDACTED>",
        "sasl.mechanism": "SCRAM-SHA-512",
        "security.protocol": "SASL_SSL",
        "ssl.ca.location": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"
      }'
      CREDENTIALS '{"sasl.password": "<REDACTED>"}'
      DISABLE OUT_OF_ORDER OPTIMIZATION
      SKIP DUPLICATE KEY ERRORS
      INTO TABLE <table_name>;

   4. The broker hostnames and ports in the KAFKA URI and the spoof.dns mapping must match exactly. You may use any port, as long as it is consistent across both settings.

Separate PrivateLink per Kafka Broker

This setup creates one NLB and one outbound endpoint per Kafka broker. Perform the following steps:

1. Create one target group per broker. Each target group must:

   1. Forward traffic to the broker’s IP address.

   2. Use the appropriate port (e.g., 6001, 6002, or 6003) based on the security protocol in use.

2. For each Kafka broker, create an NLB associated with its corresponding target group.

   1. Ensure that the NLB is deployed in the same availability zone (AZ) as the broker and its target group.

   2. Configure a listener on the NLB using the appropriate port for your Kafka connection protocol (e.g., 6001, 6002, or 6003).

   3. Each listener should forward traffic to the target group that routes to the specific broker’s IP address.

   

3. Create an endpoint service pointing to the load balancer as shown above.-

   

4. Whitelist the AWS SingleStore account shared by SingleStore to the allowed principals for each endpoint service.

   

5. While outbound private endpoints can typically be created directly in the SingleStore Cloud Portal, the portal currently allows only one outbound endpoint per workspace. Since this configuration requires multiple endpoints, open a Support ticket to have SingleStore create additional outbound endpoints. Provide the following:

   1. Endpoint Service ARNs

   2. AWS network IDs (e.g., use1-az1, use1-az2) of the subnets hosting the Kafka brokers

6. Create the pipeline using the provisioned endpoints. SingleStore will provide the outbound private endpoints associated with each Kafka broker. Once these are available, configure the pipeline using the spoof.dns setting as shown below:

   SQL

   Copy

   CREATE OR REPLACE PIPELINE kafka_munis_poc_pipeline AS
   LOAD DATA KAFKA '<broker-1>:9096,<broker-2>:9096,<broker-3>:9096/<kafka-topic>'
   CONFIG '{
     "spoof.dns": {
       "<broker-1>:9096": "<vpc-endpoint 1>:9096",
       "<broker-2>:9096": "<vpc-endpoint 2>:9096",
       "<broker-3>:9096": "<vpc-endpoint 3>:9096"
     },
     "sasl.username": "<REDACTED>",
     "sasl.mechanism": "SCRAM-SHA-512",
     "security.protocol": "SASL_SSL",
     "ssl.ca.location": "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"
   }'
   CREDENTIALS '{"sasl.password": "<REDACTED>"}'
   DISABLE OUT_OF_ORDER OPTIMIZATION
   SKIP DUPLICATE KEY ERRORS

Prerequisites

Refer to the following Confluent documentation for information on instructions:

• AWS: Use AWS PrivateLink for Dedicated Clusters on Confluent Cloud

• Azure: Use Azure Private Link for Dedicated Clusters on Confluent Cloud

• GCP: Use Google Cloud Private Service Connect for Dedicated Clusters on Confluent Cloud

Set Up Private DNS Access

Confluent Kafka can only be integrated using private DNS from SingleStore to resolve the Confluent Kafka domain. This requires a custom domain to be configured in Confluent, which will be used to create the private DNS zone in SingleStore.

To enable a custom domain, you must provision a dedicated Confluent cluster. Shared or enterprise-tier clusters are not supported, as they do not allow custom domain configurations.

Open a Support ticket with SingleStore. Provide the following:

1. The custom domain name used in Confluent.

2. PrivateLink service(s) created in Confluent

3. Availability zones where the PrivateLink endpoints are hosted.

SingleStore will create a Private DNS Zone. This maps the Confluent bootstrap domain to the private service. Once DNS is configured, you can reference the bootstrap domain directly by creating the pipeline