Create Wasm UDFs

You can create Wasm UDFs using the CREATE FUNCTION statement.

Syntax

CREATE [OR REPLACE] FUNCTION function_name
    { <explicitTypedPred> | <implicitTypedPred> }

explicitTypedPred: 
        <typeSpec> 
        AS WASM 
        (ABI <abiSpec>)
        (GROW TO <numBytes>) 
        FROM <contentSrc> 

implicitTypedPred: 
        AS WASM 
        (ABI <abiSpec>)
        (GROW TO <numBytes>)
        FROM <contentSrc> 
        (WITH WIT FROM <contentSrc>)


function_name: <standard function name>
typeSpec: '(' <paramList>? ')' RETURNS <result>
abiSpec: BASIC | CANONICAL
numBytes: [0-9]+
paramList: <param> (',' <param>)*
param: <varName> <dataType>
result: TABLE (<paramList>) | <dataType>
varName: <standard column name>

contentSrc: BASE64 '"' <base64> '"                |
            HTTP '"' <url> '"' <cred>? <config>?  |
            LINK <linkName> <connStr>             |
            S3 <s3_config>?                       |
            AZURE <azure_config>?                 |
            GCS <gcs_config>?

linkName: <link connection name>
connStr: <provider-specific path to a resource>
base64: <base64-encoded string>
url: <standard URL form>
cred: CREDENTIALS '"' <credSpec> '"'
credSpec: <json with provider-specific credentials>
config: CONFIG '"' <configSpec> '"'
configSpec: <json with provider-specific configuration>

s3_config:
    S3 { '<bucket-name>' | '<bucket-name/path>' }
      [CONFIG '<configuration_json>']
      CREDENTIALS '<credentials_json>'

azure_config:
    AZURE { '<container-name>' | '<container-name/object-name>' | '<container-name/prefix/object-name>' }
      CREDENTIALS '<credentials_json>'
      [CONFIG '<configuration_json>']

gcs_config:
   GCS { '<bucket-name>' | '<bucket-name/path>' }
     CREDENTIALS '<credentials_json>'
     [CONFIG '<configuration_json>']

Here's a sample of the syntax for an explicitly-typed Wasm UDF:

CREATE FUNCTION <name> <typeSpec> 
    AS WASM
    (ABI <abiSpec>)
    (GROW TO <numBytes>)
    <contentSrc> 

Here's a sample of the syntax for an implicitly-typed Wasm UDF using Canonical ABI:

CREATE FUNCTION <name> 
    AS WASM
    (ABI CANONICAL)
    (GROW TO <numBytes>)
    <contentSrc> 
    (WITH WIT <contentSrc>)

Here's a sample of the syntax for an implicitly-typed Wasm UDF using Basic ABI:

CREATE FUNCTION <name> 
    AS WASM
    ABI BASIC
    (GROW TO <numBytes>)
    <contentSrc> 

Remarks

  • When using the Canonical ABI, an implicitly-typed Wasm UDF automatically infers the data type information from the WIT specification.

  • When using the Basic ABI, an implicitly-typed Wasm UDF infers the data type information from the function's signature.

  • The explicitly-typed Wasm UDF allows you to specify your own data types and maps them to the corresponding type supported by the current ABI. See Wasm Data Type Coercions for more information.

    Warning

    Using inappropriate data types may cause the UDF to return unpredictable results or an error.

  • The implicitly-typed Wasm UDF defaults to the Canonical ABI if the ABI clause is not specified in the CREATE FUNCTION statement.

  • The explicitly-typed Wasm UDF does not support WIT document specification.

  • The Basic ABI does not support WIT document specification.

  • The implicitly-typed Wasm UDF does not support the specification of parameters or return types in a UDF.

  • The WIT document specification must be specified in an implicitly-typed Wasm UDF that uses Canonical ABI.

  • The GROW TO clause is optional for both the Canonical ABI and Basic ABI. The default size is 16 MB (256 Wasm pages).

Wasm UDF Sources

SingleStoreDB supports Wasm UDFs created from the following sources in the CREATE FUNCTION statement:

  • Base-64 encoded string: You can pass the contents of a Wasm module directly to the CREATE FUNCTION statement as an inline base-64 encoded string, see Examples. The maximum size of the CREATE FUNCTION statement can be modified using the max_allowed_packet global variable.

  • Remote URL: To download the content of a Wasm module from a remote endpoint, you can specify the URL of the endpoint using the HTTP clause. Here's an example:

    CREATE FUNCTION wasm_udf AS WASM FROM HTTP 'https://url/to/udf.wasm' ...

    Optionally, you can specify the CONFIG and CREDENTIALS clauses for the URL. The LINK object is not supported with HTTP, and the credential configuration must be provided inline in the CREATE FUNCTION command.

  • Cloud Provider: You can download the content of a Wasm module from the following cloud providers: S3, Azure, and GCS. You also can specify the URL and connection credentials using the LINK clause, see Examples.

Wasm UDF Naming Convention

  • When using a WIT file, the name of the Wasm UDF must match the name of the corresponding Wasm function as declared in the WIT IDL, after substituting hyphens with underscores.

  • When you are not using a WIT file, the name of the Wasm UDF must match the name of the function exported from the module. This is usually the same function name declared in the source language.

Examples

Example 1 - Create a Wasm UDF Using Rust

The following example creates a Wasm UDF that computes the expression x^y in Rust and builds the project in Visual Studio Code. This example imports the Wasm module from an S3 bucket

Setup VS Code

  1. Install the Remote - Containers extension in VS Code.

  2. Clone the wasm-udf-tutorial GitHub repository. This repository contains a VS Code Development Container with all the required dependencies.

    git clone git@github.com:singlestore-labs/wasm-udf-tutorial.git
  3. Open the Command Palette, and select Remote Containers: Open Folder in Container.

  4. Navigate to the directory where you cloned the GitHub repository, and select Open.

After VS Code completes building the project, proceed with the example.

Create a Wasm UDF

  1. Open a new directory in VS Code, and initialize a Cargo package in this directory using the following command:

    cargo init --vcs none --lib
  2. Update the Cargo.toml file with the following configuration:

    [package]
    name = "power"
    version = "0.1.0"
    edition = "2018"
    
    [dependencies]
    wit-bindgen-rust = { git = "https://github.com/bytecodealliance/wit-bindgen.git" }
    
    [lib]
    crate-type = ["cdylib"]
  3. Create a .wit file with the WIT specification, for example power.wit.

    power-of: func(base: s32, exp: s32) -> s32

    This specifies that the power-of function accepts two 32-bit integers and returns a single 32-bit integer.

  4. Add the following code to the src/lib.rs file:

    wit_bindgen_rust::export!("power.wit");
    struct Power;
    impl power::Power for Power {
    
        fn power_of(base: i32, exp: i32) -> i32 {
            let mut res = 1;
            for _i in 0..exp {
                res *= base;
            }
            res
        }
    }
  5. Compile the program into a Wasm module using the following command:

    cargo build --target wasm32-wasi

    For this example we use the wasm32-wasi target. The power.wasm file is created in the target/wasm32-wasi/debug/ directory.

    Note

    When your code is debugged and ready for production, you can add the --release flag to the cargo build command instead to enable further optimizations and remove the debug info. For example,

    cargo build --release

    The resulting power.wasm file is created in the target/wasm32-wasi/release/ directory.

  6. Upload the power.wasm and power.wit files to your cloud storage account, say an S3 bucket named wasm-modules.

Load the Wasm UDF

Execute the following command in the SingleStoreDB command-line:

CREATE DATABASE wasm_test;
USE wasm_test;
CREATE FUNCTION power_of AS WASM FROM S3 'wasm-modules/power.wasm' 
        CREDENTIALS '{
                "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", 
                "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]"
        }' 
        CONFIG '{"region": "us-east-1"}'
WITH WIT FROM S3 'wasm-modules/power.wit'
        CREDENTIALS '{
                "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", 
                "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]"
        }' 
        CONFIG '{"region": "us-east-1"}';

This command imports the power.wit and power.wasm files created earlier in this example. The Wasm UDF is now ready to use. For example,

SELECT power_of(2,4) AS Result;
****
+-----------+
| Result    |
+-----------+ 
| 16        |
+-----------+

Example 2 - Create a Wasm UDF Using C

The following example creates a Wasm UDF that computes the expression x^y using the C language. See Setup VS Code in the previous example to build this example in VS Code and include the required dependencies, for example wit-bindgen.

  1. Create a .wit file with the WIT specification, for example power.wit.

    power-of: func(base: s32, exp: s32) -> s32

    This specifies that the power-of function accepts two 32-bit integers as arguments and returns a single 32-bit integer.

  2. Run wit-bindgen on your IDL to generate Wasm bindings.

    wit-bindgen c --export power.wit
  3. Add the following code to the source file (power.c):

    int32_t power_of(int32_t base, int32_t exp)
    {
        int32_t res = 1;
        for (int32_t i = 0; i < exp; ++i)
        {
            res *= base;
        }
        return res;
    }
    
  4. Compile the program into a Wasm module using the following command:

    clang                            \
        --target=wasm32-unknown-wasi \
        -mexec-model=reactor         \
        -s                           \
        -I.                          \
        -o power.wasm                \
        power.c

    This command generates the power.wasm file that contains the Wasm module.

  5. Upload the power.wasm and power.wit files to your cloud storage account, say an S3 bucket named wasm-modules.

  6. Run the following command in SingleStoreDB command-line to load the Wasm module:

    CREATE FUNCTION power_of AS WASM FROM S3 'wasm-modules/power.wasm' 
            CREDENTIALS '{
                    "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", 
                    "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]"
            }' 
            CONFIG '{"region": "us-east-1"}'
    WITH WIT FROM S3 'wasm-modules/power.wit'
            CREDENTIALS '{
                    "aws_access_key_id": "ASIAZPIKLSJ3HM7FKAUB", 
                    "aws_secret_access_key": FwoGZXIvYXdzEL3fv [...]"
            }' 
            CONFIG '{"region": "us-east-1"}';

The Wasm module is now ready to use. For example,

SELECT power_of(4,2) AS Result;
****
+-----------+
| Result    |
+-----------+ 
| 16        |
+-----------+

Example 3 - Load Wasm Using a Base-64 encoded String

The following example shows how to load a Wasm module created in C, which is encoded to a base-64 encoded string:

  1. Add the following code to the source file, say mult.c:

    int mult(int a,int b) {
      return a * b;
      }
  2. Compile the program into a Wasm module using the following command:

    clang                            \ 
        --target=wasm32-unknown-wasi \
        -mexec-model=reactor         \
        -Wl,--export-all             \
        -o mult.wasm mult.c
  3. To encode a Wasm file to a base-64 encoded string, run the following command (in Linux):

    base64 -w 0 mult.wasm
    ****
    AGFzbQEAAAABCgJgAABgAn9/AX8DBAMAAAEEBQFwAQEBBQMBAAIGKwd/AUGAiAQLfwBBgAgLfwBBgAgLfwBBgAgLfwBBgIgEC38AQQALfwBBAQsHqAELBm1lbW9yeQIAEV9fd2FzbV9jYWxsX2N0b3JzAAALX2luaXRpYWxpemUAARlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQAEbXVsdAACDF9fZHNvX2hhbmRsZQMBCl9fZGF0YV9lbmQDAg1fX2dsb2JhbF9iYXNlAwMLX19oZWFwX2Jhc2UDBA1fX21lbW9yeV9iYXNlAwUMX190YWJsZV9iYXNlAwYKSwMCAAsIABCAgICAAAs9AQZ/I4CAgIAAIQJBECEDIAIgA2shBCAEIAA2AgwgBCABNgIIIAQoAgwhBSAEKAIIIQYgBSAGbCEHIAcPCw==

    Copy the base-64 encoded from the standard output (stdout).

  4. Run the following command to load the Wasm module:

    CREATE FUNCTION mult AS WASM ABI BASIC FROM BASE64 'AGFzbQEAAAABCgJgAABgAn9/AX8DBAMAAAEEBQFwAQEBBQMBAAIGKwd/AUGAiAQLfwBBgAgLfwBBgAgLfwBBgAgLfwBBgIgEC38AQQALfwBBAQsHqAELBm1lbW9yeQIAEV9fd2FzbV9jYWxsX2N0b3JzAAALX2luaXRpYWxpemUAARlfX2luZGlyZWN0X2Z1bmN0aW9uX3RhYmxlAQAEbXVsdAACDF9fZHNvX2hhbmRsZQMBCl9fZGF0YV9lbmQDAg1fX2dsb2JhbF9iYXNlAwMLX19oZWFwX2Jhc2UDBA1fX21lbW9yeV9iYXNlAwUMX190YWJsZV9iYXNlAwYKSwMCAAsIABCAgICAAAs9AQZ/I4CAgIAAIQJBECEDIAIgA2shBCAEIAA2AgwgBCABNgIIIAQoAgwhBSAEKAIIIQYgBSAGbCEHIAcPCw==';

The Wasm module is now ready to use. For example,

SELECT mult(4,2) AS Result;
****
+-----------+
| Result    |
+-----------+ 
| 8        |
+-----------+
 

Example References

For more examples, see SingleStore Wasm Toolkit. Here are some references:

Troubleshooting

  • If a Wasm UDF returns a backtrace indicating an out-of-memory exception, increase the amount of linear memory available. To increase the amount of linear memory available:

    • Re-create the UDF with a higher value for GROW TO, and/or

    • Increase the value of wasm_max_linear_memory_size global variable.

  • To list all the Wasm UDFs for the current database, use the SHOW FUNCTIONS command. For Wasm UDFs, the Function Type is User Defined Function and the Runtime Type field is Wasm.

  • To view the SQL command used to create a Wasm UDF, use the SHOW CREATE FUNCTION command. Since the body of the Wasm module is binary and large, it is omitted in the output. For example,

    CREATE OR REPLACE FUNCTION mywasm(input int(11) NOT NULL) RETURNS int(11) NOT NULL AS WASM ABI CANONICAL GROW TO 16777216;
  • To view detailed resource metrics on a command using Wasm UDFs, use the PROFILE command.

  • SingleStore provides a Rust crate that can act as a debugger. For more information see Wasm Remote Debugger Service.

  • If a Wasm code generates an exception, use the SHOW WASM BACKTRACE command to view the complete backtrace from the error message. For example,

    SHOW WASM BACKTRACE \G
    ************************* 1. row *************************
    Backtrace: wasm trap: wasm `unreachable` instruction executed
    wasm backtrace:
    0: 0xe00d - <unknown>!abort
    1: 0x7aaa - std::sys::wasi::abort_internal::hb387f3c4e8f42ceb
    at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys/wasi/mod.rs:85:14
    2: 0x9be4 - std::process::abort::hdfad6720ede82433
    at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/process.rs:2059:5
    3: 0xa0b9 - rust_oom
    at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/alloc.rs:341:5
    -------- output truncated --------