Load Data from Estuary
On this page
Estuary is a managed data integration platform that supports change data capture (CDC), streaming, and batch data movement.
SingleStore supports the following methods to integrate with Estuary:
Prerequisites
-
An Estuary collection
-
An active SingleStore cluster
Direct SingleStore Materialization
Use the Estuary-provided SingleStore materialization connector to configure a database connection directly in Estuary.
To configure the SingleStore materialization:
-
Log in to Estuary.
-
On the Dashboard, select Destinations from the left navigation pane.
-
On the Destinations page, select New Materialization.
-
On the Create Materialization page, from the list of connectors, search for and select the SingleStore materialization (not SingleStore Dekaf).
-
Under Materialization Details, enter or select the following information:
-
Name: Enter a name for the materialization.
-
Data Plane: Select a data plane to use.
-
-
Under Endpoint Config, enter the following:
Field
Description
Address
IP address or hostname of the SingleStore cluster in the
host:portformat.User
Username of the SingleStore database user.
Password
Password for the SingleStore database user.
Database
Name of the SingleStore database to connect with.
-
Under Advanced Options, configure SSL/TLS for the connection.
For example: -
Download the singlestore_
bundle. pem certificate file from the Cloud Portal. -
Under
SSL Mode(in Estuary dashboard), select required. -
Under SSL Server CA, select Use secret from file and then upload singlestore_
bundle. pem.
Refer to Connect to SingleStore using TLS/SSL for related information.
-
-
Under Source Collections, select and configure the collection(s) to materialize in SingleStore.
-
Select Next.
-
Select Test to validate the materialization.
-
Select Save and publish to publish this materialization.
The SingleStore materialization is now configured and ready to use.
Kafka-Compatible Dekaf Connector
Use the SingleStore variant of the Kafka-compatible Dekaf connector to configure SingleStore as a destination in Estuary.
-
Log in to Estuary.
-
On the Dashboard, select Destinations from the left navigation pane.
-
On the Destinations page, select New Materialization.
-
On the Create Materialization page, from the list of connectors, search for and select the SingleStore Dekaf materialization (not SingleStore).
-
Under Materialization Details, enter or select the following information:
-
Name: Enter a name for the materialization.
-
Data Plane: Select a data plane to use.
-
-
Under Auth Token, specify an authentication token.
Use this token to authenticate Kafka consumers to this materialization task. -
Under Source Collections, select and configure the collection(s) to materialize in SingleStore.
-
Select Next.
-
Select Test to validate the materialization.
-
Select Save and publish to publish this materialization.
-
Copy the full materialization name (org-name/prefix/materialization-name).
Use it as the username value ( sasl.andusername schema.) in the pipeline configuration.registry. username -
Connect to your SingleStore cluster.
-
Create a table to store the ingested data.
For example: CREATE TABLE s2estuary(id INT, created DATETIME(6), product_ids JSON); -
Create a pipeline to ingest the data from Estuary.
For example: CREATE PIPELINE estuaryIngest ASLOAD DATA KAFKA "dekaf.estuary-data.com:9092/<topic_name>"CONFIG '{"security.protocol":"SASL_SSL","sasl.mechanism":"PLAIN","sasl.username":"{<materialization_name>}","broker.address.family": "v4","schema.registry.username": "{<materialization_name>}","fetch.wait.max.ms": "2000"}'CREDENTIALS '{"sasl.password": "<auth_token>","schema.registry.password": "<auth_token>"}'INTO TABLE s2estuaryFORMAT AVRO SCHEMA REGISTRY 'https://dekaf.estuary-data.com'( id <- id, created <- created, product_ids <- product_ids );where:
-
topic_: Name of the Kafka topic published by the Dekaf binding.name It may differ from the collection name. -
materialization_: Full Estuary materialization name.name -
auth_: Authentication token specified for the Dekaf materialization.token
-
-
Start the pipeline.
For example: START PIPELINE estuaryIngest;
The pipeline starts ingesting data from the Estuary collection to the SingleStore table.SHOW PIPELINES command to view the status of the pipeline.
Refer to SingleStore | Estuary for more information, including configuration properties.
Last modified: