On this page
singlestore-spark-connector has extensive support for rewriting Spark SQL and DataFrame operation query plans into standalone SingleStoreDB queries.
SQL Pushdown is either enabled or disabled on the entire Spark Session.
disablePushdown values, run them on separate Spark Sessions.
We currently support most of the primary Logical Plan nodes in Spark SQL including
We also support most Spark SQL expressions.
The best place to look for examples of fully supported queries is in the tests.
UnixTimestamphandle only time less than
2038-01-19 03:14:08, if they get
TimestampTypeas a first argument.
FromUnixTimewith default format (
yyyy-MM-dd HH:mm:ss) handle only time less than
DecimalTypeis truncated on overflow (by default, Spark either throws an exception or returns
nullif at least one argument is
null(in Spark these functions skip nulls).
When a value can not be converted to a numeric or fractional type, SingleStoreDB returns 0 (Spark returns
Atanh(x), for x ∈ (-∞, -1] ∪ [1, ∞), returns null (Spark returns
When a string is cast to a numeric type, SingleStoreDB accepts its prefix if it is numeric (Spark returns
nullif the whole string is not numeric).
When a numeric type is cast to a smaller one (in size), SingleStoreDB truncates it.
For example, 500 cast to the
Bytewill be 127.
Note: Spark optimizer can optimize casts for literals and then the behaviour for literals matches custom Spark behaviour.
When a fractional type is cast to an integral type, SingleStoreDB rounds it to the nearest integer.
Roundrounds down if the number to be rounded is followed by 5 and it is
DECIMALis rounded up).
Convworks differently if the number contains non-alphanumeric characters.
ShiftRightUnsignedconvert the value to an
UNSIGNED BIGINTand then produce the shift.
In case of an overflow, it returns
0 (1 << 64 = 0 and 10>>20 = 0).
BitwiseGetreturns 0 when the bit position is negative or exceeds the bit upper limit.
Initcapdefines a letter as the beginning of a word even if it is enclosed in quotation marks, brackets, etc.
For example "dear sir/madam (miss)" is converted to "Dear Sir/Madam (Miss)".
Skewness(x), in Spark 3.
STD(x) = 0returns
Last modified: July 27, 2022