Load Data from Estuary Flow

You can stream data from Estuary Flow to SingleStore using the Kafka-compatible Dekaf API via pipelines.

To load data into SingleStore:

  1. Log in to the Estuary Dashboard.

  2. Generate a Refresh Token.

    1. On the left navigation pane, select Admin.

    2. On the CLI-API tab, under the Refresh Token section, select Generate Token.

    3. On the Generate Refresh Token dialog, enter a description for the token, and then select GENERATE TOKEN.

  3. Copy and securely store the token. This token is used in the CREDENTIALS clause of the CREATE PIPELINE statement to authenticate the connection.

    Note: The token is displayed only once.

  4. Create a table in SingleStore to store the data to be ingested.

  5. Create a pipeline to stream data from Estuary Flow.

  6. Start the pipelines.

Here's an example:

CREATE TABLE estExample (id INT, title VARCHAR(255));
CREATE PIPELINE estPipeline AS
LOAD DATA KAFKA "<endpoint/database>"
CONFIG '{
"security.protocol":"SASL_SSL",
"sasl.mechanism":"PLAIN",
"sasl.username":"{}",
"broker.address.family": "v4",
"schema.registry.username": "{}",
"fetch.wait.max.ms": "2000"
}'
CREDENTIALS '{
"sasl.password": "<ESTUARY_ACCESS_TOKEN>",
"schema.registry.password": "<ESTUARY_ACCESS_TOKEN>"
}'
INTO table estExample
FORMAT AVRO SCHEMA REGISTRY '<schema_registry>'
( id <- id, title <- title );
START PIPELINE estPipeline;

Last modified: December 19, 2024

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined
  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined
  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
    --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
    --bundle undefined \
    --new-bundle-format -
    Verified OK