Data Type Conversions
When saving a DataFrame to SingleStore, the DataFrame column is converted to the following SingleStoreDB type:
Spark Type | SingleStoreDB Type |
|---|---|
LongType | BIGINT |
IntegerType | TINYINT |
ShortType | SMALLINT |
FloatType | FLOAT |
DoubleType | DOUBLE |
ByteType | TINYINT |
StringType | TEXT |
BinaryType | BLOB |
DecimalType | DECIMAL |
BooleanType | TINYINT |
TimeStampType | TIMESTAMP(6) |
DateType | DATE |
When reading a SingleStoreDB table as a Spark DataFrame, the SingleStoreDB column type is converted to the following Spark type:
SingleStoreDB Type | SparkType |
|---|---|
TINYINT | ShortType |
SMALLINT | ShortType |
INT | IntegerType |
BIGINT | LongType |
DOUBLE | DoubleType |
FLOAT | FloatType |
DECIMAL | DecimalType |
TIMESTAMP | TimeStampType |
TIMESTAMP(6) | TimeStampType |
DATE | DateType |
TEXT | StringType |
JSON | StringType |
TIME | TimeStampType |
BIT | BinaryType |
BLOB | BinaryType |
Data Type Conversion Remarks
When using the
onDuplicateKeySQLoption, the connector will error when writing a null-terminated StringType (i.e., \0).DECIMALin SingleStoreDB andDecimalTypein Spark have different maximum scales and precisions. An error will occur if you perform a read or write from a table or DataFrame with unsupported precision or scale. SingleStoreDB’s maximum scale for theDECIMALdata type is 30, while Spark’s maximum scale forDecimalTypeis 38. Similarly, SingleStoreDB’s maximum precision is 65, while Spark’s maximum precision is 38.TIMESTAMPin SingleStoreDB supports values from 1000 to 2147483647999. SingleStore treats a null value in theTIMESTAMPcolumn as the current time.The Avro format does not support writing
TIMESTAMPandDATEtypes. As a result, the SingleStore Spark Connector currently does not support these types with Avro serialization.