Data Type Conversions

On this page

When saving a DataFrame to SingleStore, the DataFrame column is converted to the following SingleStore type:

Spark Type	SingleStore Type
LongType	BIGINT
IntegerType	TINYINT
ShortType	SMALLINT
FloatType	FLOAT
DoubleType	DOUBLE
ByteType	TINYINT
StringType	TEXT
BinaryType	BLOB
DecimalType	DECIMAL
BooleanType	TINYINT
TimeStampType	TIMESTAMP(6)
DateType	DATE

When reading a SingleStore table as a Spark DataFrame, the SingleStore column type is converted to the following Spark type:

SingleStore Type	SparkType
TINYINT	ShortType
SMALLINT	ShortType
INT	IntegerType
BIGINT	LongType
DOUBLE	DoubleType
FLOAT	FloatType
DECIMAL	DecimalType
TIMESTAMP	TimeStampType
TIMESTAMP(6)	TimeStampType
DATE	DateType
TEXT	StringType
JSON	StringType
TIME	TimeStampType
BIT	BinaryType
BLOB	BinaryType

Data Type Conversion Remarks

When using the onDuplicateKeySQL option, the connector will error when writing a null-terminated StringType (i.e., \0).
DECIMAL in SingleStore and DecimalType in Spark have different maximum scales and precisions. An error will occur if you perform a read or write from a table or DataFrame with unsupported precision or scale. SingleStore’s maximum scale for the DECIMAL data type is 30, while Spark’s maximum scale for DecimalType is 38. Similarly, SingleStore’s maximum precision is 65, while Spark’s maximum precision is 38.
TIMESTAMP in SingleStore supports values from 1000 to 2147483647999. SingleStore treats a null value in the TIMESTAMP column as the current time.
The Avro format does not support writing TIMESTAMP and DATE types. As a result, the SingleStore Spark Connector currently does not support these types with Avro serialization.

Last modified: July 28, 2022

Was this article helpful?

Verification instructions

Note: You must install cosign to verify the authenticity of the SingleStore file.

Use the following steps to verify the authenticity of singlestoredb-server, singlestoredb-toolbox, singlestoredb-studio, and singlestore-client SingleStore files that have been downloaded.

You may perform the following steps on any computer that can run cosign, such as the main deployment host of the cluster.

(Optional) Run the following command to view the associated signature files.
Shell
```
curl undefined
```
Download the signature file from the SingleStore release server.
- Option 1: Click the Download Signature button next to the SingleStore file.
- Option 2: Copy and paste the following URL into the address bar of your browser and save the signature file.
- Option 3: Run the following command to download the signature file.
  Shell
```
curl -O undefined
```

After the signature file has been downloaded, run the following command to verify the authenticity of the SingleStore file.

Shell

echo -n undefined |
    cosign verify-blob --certificate-oidc-issuer https://oidc.eks.us-east-1.amazonaws.com/id/CCDCDBA1379A5596AB5B2E46DCA385BC \
      --certificate-identity https://kubernetes.io/namespaces/freya-production/serviceaccounts/job-worker \
      --bundle undefined \
      --new-bundle-format -

Verified OK