Load Data from Estuary

Estuary is a managed data integration platform that supports change data capture (CDC), streaming, and batch data movement. It enables you to continuously materialize data from operational databases, applications, and other supported sources into SingleStore for real-time analytics. Ingest data from Estuary into SingleStore using materializations.

SingleStore supports the following methods to integrate with Estuary:

Prerequisites

  • An Estuary collection

  • An active SingleStore cluster

Direct SingleStore Materialization

Use the Estuary-provided SingleStore materialization connector to configure a database connection directly in Estuary. This method enables you to ingest data from Estuary collections into SingleStore without configuring a SingleStore pipeline.

To configure the SingleStore materialization:

  1. Log in to Estuary.

  2. On the Dashboard, select Destinations from the left navigation pane.

  3. On the Destinations page, select New Materialization.

  4. On the Create Materialization page, from the list of connectors, search for and select the SingleStore materialization (not SingleStore Dekaf).

  5. Under Materialization Details, enter or select the following information:

    1. Name: Enter a name for the materialization.

    2. Data Plane: Select a data plane to use.

  6. Under Endpoint Config, enter the following:

    Field

    Description

    Address

    IP address or hostname of the SingleStore cluster in the host:port format.

    User

    Username of the SingleStore database user.

    Password

    Password for the SingleStore database user.

    Database

    Name of the SingleStore database to connect with.

  7. Under Advanced Options, configure SSL/TLS for the connection. For example:

    1. Download the singlestore_bundle.pem certificate file from the Cloud Portal.

    2. Under SSL Mode (in Estuary dashboard), select required.

    3. Under SSL Server CA, select Use secret from file and then upload singlestore_bundle.pem.

    Refer to Connect to SingleStore using TLS/SSL for related information.

  8. Under Source Collections, select and configure the collection(s) to materialize in SingleStore.

  9. Select Next.

  10. Select Test to validate the materialization.

  11. Select Save and publish to publish this materialization.

The SingleStore materialization is now configured and ready to use. Refer to SingleStore Materialization Connector for related information.

Kafka-Compatible Dekaf Connector

Use the SingleStore variant of the Kafka-compatible Dekaf connector to configure SingleStore as a destination in Estuary. After publishing the materialization, create a Kafka pipeline using the CREATE PIPELINE command to ingest data. Perform the following tasks:

  1. Log in to Estuary.

  2. On the Dashboard, select Destinations from the left navigation pane.

  3. On the Destinations page, select New Materialization.

  4. On the Create Materialization page, from the list of connectors, search for and select the SingleStore Dekaf materialization (not SingleStore).

  5. Under Materialization Details, enter or select the following information:

    1. Name: Enter a name for the materialization.

    2. Data Plane: Select a data plane to use.

  6. Under Auth Token, specify an authentication token. Use this token to authenticate Kafka consumers to this materialization task.

  7. Under Source Collections, select and configure the collection(s) to materialize in SingleStore.

  8. Select Next.

  9. Select Test to validate the materialization.

  10. Select Save and publish to publish this materialization.

  11. Copy the full materialization name (org-name/prefix/materialization-name). Use it as the username value (sasl.username and schema.registry.username) in the pipeline configuration.

  12. Connect to your SingleStore cluster.

  13. Create a table to store the ingested data. For example:

    CREATE TABLE s2estuary
    (id INT, created DATETIME(6), product_ids JSON);
  14. Create a pipeline to ingest the data from Estuary. For example:

    CREATE PIPELINE estuaryIngest AS
    LOAD DATA KAFKA "dekaf.estuary-data.com:9092/<topic_name>"
    CONFIG '{
    "security.protocol":"SASL_SSL",
    "sasl.mechanism":"PLAIN",
    "sasl.username":"{<materialization_name>}",
    "broker.address.family": "v4",
    "schema.registry.username": "{<materialization_name>}",
    "fetch.wait.max.ms": "2000"
    }'
    CREDENTIALS '{
    "sasl.password": "<auth_token>",
    "schema.registry.password": "<auth_token>"
    }'
    INTO TABLE s2estuary
    FORMAT AVRO SCHEMA REGISTRY 'https://dekaf.estuary-data.com'
    ( id <- id, created <- created, product_ids <- product_ids );

    where:

    • topic_name: Name of the Kafka topic published by the Dekaf binding. It may differ from the collection name.

    • materialization_name: Full Estuary materialization name.

    • auth_token: Authentication token specified for the Dekaf materialization.

  15. Start the pipeline. For example:

    START PIPELINE estuaryIngest;

The pipeline starts ingesting data from the Estuary collection to the SingleStore table. Run the SHOW PIPELINES command to view the status of the pipeline.

Refer to SingleStore | Estuary for more information, including configuration properties.

Last modified:

Was this article helpful?