Important
Helios features are now enabled during weekly update windows and are no longer directly tied to SingleStore engine releases. Refer to the release notes to view the latest features available in your Helios cluster.
Load Data from Estuary
On this page
Estuary is a managed data integration platform that supports change data capture (CDC), streaming, and batch data movement.
SingleStore supports the following methods to integrate with Estuary:
Prerequisites
-
An Estuary collection
-
An active SingleStore cluster
Direct SingleStore Materialization
Use the Estuary-provided SingleStore materialization connector to configure a database connection directly in Estuary.
To configure the SingleStore materialization:
-
Log in to Estuary.
-
On the Dashboard, select Destinations from the left navigation pane.
-
On the Destinations page, select New Materialization.
-
On the Create Materialization page, from the list of connectors, search for and select the SingleStore materialization (not SingleStore Dekaf).
-
Under Materialization Details, enter or select the following information:
-
Name: Enter a name for the materialization.
-
Data Plane: Select a data plane to use.
-
-
Under Endpoint Config, enter the following:
Field
Description
Address
IP address or hostname of the SingleStore cluster in the
host:portformat.Refer to SingleStore Helios Endpoints to determine the endpoint (
host:port) of your cluster.User
Username of the SingleStore database user.
Password
Password for the SingleStore database user.
Database
Name of the SingleStore database to connect with.
-
Under Advanced Options, configure SSL/TLS for the connection.
For example: -
Download the singlestore_
bundle. pem certificate file from the Cloud Portal. -
Under
SSL Mode(in Estuary dashboard), select required. -
Under SSL Server CA, select Use secret from file and then upload singlestore_
bundle. pem.
Refer to Connect to SingleStore Helios using TLS/SSL for related information.
-
-
Under Source Collections, select and configure the collection(s) to materialize in SingleStore.
-
Select Next.
-
Select Test to validate the materialization.
-
Select Save and publish to publish this materialization.
The SingleStore materialization is now configured and ready to use.
Kafka-Compatible Dekaf Connector
Use the SingleStore variant of the Kafka-compatible Dekaf connector to configure SingleStore as a destination in Estuary.
-
Log in to Estuary.
-
On the Dashboard, select Destinations from the left navigation pane.
-
On the Destinations page, select New Materialization.
-
On the Create Materialization page, from the list of connectors, search for and select the SingleStore Dekaf materialization (not SingleStore).
-
Under Materialization Details, enter or select the following information:
-
Name: Enter a name for the materialization.
-
Data Plane: Select a data plane to use.
-
-
Under Auth Token, specify an authentication token.
Use this token to authenticate Kafka consumers to this materialization task. -
Under Source Collections, select and configure the collection(s) to materialize in SingleStore.
-
Select Next.
-
Select Test to validate the materialization.
-
Select Save and publish to publish this materialization.
-
Copy the full materialization name (org-name/prefix/materialization-name).
Use it as the username value ( sasl.andusername schema.) in the pipeline configuration.registry. username -
Connect to your SingleStore cluster.
-
Create a table to store the ingested data.
For example: CREATE TABLE s2estuary(id INT, created DATETIME(6), product_ids JSON); -
Create a pipeline to ingest the data from Estuary.
For example: CREATE PIPELINE estuaryIngest ASLOAD DATA KAFKA "dekaf.estuary-data.com:9092/<topic_name>"CONFIG '{"security.protocol":"SASL_SSL","sasl.mechanism":"PLAIN","sasl.username":"{<materialization_name>}","broker.address.family": "v4","schema.registry.username": "{<materialization_name>}","fetch.wait.max.ms": "2000"}'CREDENTIALS '{"sasl.password": "<auth_token>","schema.registry.password": "<auth_token>"}'INTO TABLE s2estuaryFORMAT AVRO SCHEMA REGISTRY 'https://dekaf.estuary-data.com'( id <- id, created <- created, product_ids <- product_ids );where:
-
topic_: Name of the Kafka topic published by the Dekaf binding.name It may differ from the collection name. -
materialization_: Full Estuary materialization name.name -
auth_: Authentication token specified for the Dekaf materialization.token
-
-
Start the pipeline.
For example: START PIPELINE estuaryIngest;
The pipeline starts ingesting data from the Estuary collection to the SingleStore table.SHOW PIPELINES command to view the status of the pipeline.
Refer to SingleStore | Estuary for more information, including configuration properties.
Last modified: