Sign inTry Free

Important

New database features are no longer connected to engine versions; features are now enabled independently during scheduled update windows, and “major” and “minor” releases no longer exist in Helios. Please visit the release notes to view the latest features available in your database clusters.

Data Type Conversions

On this page

When saving a DataFrame to SingleStore, the DataFrame column is converted to the following SingleStore type:

Spark Type

SingleStore Type

LongType

BIGINT

IntegerType

TINYINT

ShortType

SMALLINT

FloatType

FLOAT

DoubleType

DOUBLE

ByteType

TINYINT

StringType

TEXT

BinaryType

BLOB

DecimalType

DECIMAL

BooleanType

TINYINT

TimeStampType

TIMESTAMP(6)

DateType

DATE

When reading a SingleStore table as a Spark DataFrame, the SingleStore column type is converted to the following Spark type:

SingleStore Type

SparkType

TINYINT

ShortType

SMALLINT

ShortType

INT

IntegerType

BIGINT

LongType

DOUBLE

DoubleType

FLOAT

FloatType

DECIMAL

DecimalType

TIMESTAMP

TimeStampType

TIMESTAMP(6)

TimeStampType

DATE

DateType

TEXT

StringType

JSON

StringType

TIME

TimeStampType

BIT

BinaryType

BLOB

BinaryType

Data Type Conversion Remarks

  • When using the onDuplicateKeySQL option, the connector will error when writing a null-terminated StringType (i.e., \0).

  • DECIMAL in SingleStore and DecimalType in Spark have different maximum scales and precisions. An error will occur if you perform a read or write from a table or DataFrame with unsupported precision or scale. SingleStore’s maximum scale for the DECIMAL data type is 30, while Spark’s maximum scale for DecimalType is 38. Similarly, SingleStore’s maximum precision is 65, while Spark’s maximum precision is 38.

  • TIMESTAMP in SingleStore supports values from 1000 to 2147483647999. SingleStore treats a null value in the TIMESTAMP column as the current time.

  • The Avro format does not support writing TIMESTAMP and DATE types. As a result, the SingleStore Spark Connector currently does not support these types with Avro serialization.

Last modified:

Was this article helpful?

On this page

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

  1. (Optional) Run the following command to view the associated signature files.

    curl undefined

  2. Download the signature file from the SingleStore release server.

    • Option 1: Click the Download Signature button next to the SingleStore file.

    • Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.

    • Option 3: Run the following command to download the signature file.

      curl -O undefined

  3. After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

    echo -n undefined |
        cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
          --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
          --bundle undefined \
          --new-bundle-format -
    Verified OK

Try Out This Notebook to See What’s Possible in SingleStore

Get access to other groundbreaking datasets and engage with our community for expert advice.

Sign UpLog In