Load Data from Estuary Flow

Prerequisites

• An Estuary Flow collection

• An active SingleStore deployment

Connect to SingleStore from Estuary Flow

To configure SingleStore as a destination:

1. Log in to Estuary Flow.

2. Create a new materialization.

   1. On the Dashboard, select Destinations from the left navigation pane.

   2. On the Destinations page, select NEW MATERIALIZATION.

   3. On the Create Materialization page, from the list of connectors, search and select the SingleStore connector.

   4. Enter or select the following information:

      1. Name: Enter a name for the materialization.

      2. Data Plane: Select a data plane to use.

      3. Auth Token: Specify an authentication token. This token is used to authenticate Kafka consumers to this materialization task.

   5. Link a capture (collection) to this task.

   6. Select NEXT > TEST to test the materialization.

   7. Select SAVE AND PUBLISH to publish this task.

3. Log in to your SingleStore deployment.

4. Create a table to store the ingested data. For example:

   SQL

   Copy

   CREATE TABLE estuaryFlow
   (id INT, created DATETIME(6), product_ids JSON);

5. Create a pipeline to ingest the data from Estuary Flow. For example:

   SQL

   Copy

   CREATE PIPELINE estuaryIngest AS
           LOAD DATA KAFKA "dekaf.estuary-data.com:9092/<collection_name>"
           CONFIG '{
               "security.protocol":"SASL_SSL",
               "sasl.mechanism":"PLAIN",
               "sasl.username":"{<materialization_name>}",
               "broker.address.family": "v4",
               "schema.registry.username": "{<materialization_name>}",
               "fetch.wait.max.ms": "2000"
           }'
           CREDENTIALS '{
               "sasl.password": "<auth_token>",
               "schema.registry.password": "<auth_token>"
           }'
           INTO TABLE estuaryFlow
           FORMAT AVRO SCHEMA REGISTRY 'https://dekaf.estuary-data.com'
           ( id <- id, created <- created, product_ids <- product_ids );

   where,

   • collection_name: Name of the collection linked to the materialization task.

   • materialization_name: Name of the Estuary Flow materialization.

   • auth_token: Authentication token specified for the materialization.

6. Start the pipeline. For example:

   SQL

   Copy

   START PIPELINE estuaryIngest;

The pipeline starts ingesting data from the Estuary Flow collection to the SingleStore table. Run the SHOW PIPELINES command to view the status of the pipeline.

Refer to SingleStore | Estuary Flow for more information, for example, configuration properties.