Migrate between SingleStore Spark Connector Versions

Note

The SingleStore Spark Connector versions 2.0 and 3.0 are no longer supported.

The following sections describe the differences in configuration and functionality between the SingleStore Spark Connector versions.

Version 3.0 > 3.1

  • The SingleStore Spark Connector is rebranded from memsql-spark-connector to singlestore-spark-connector.

  • The Spark data source format is changed from memsql to singlestore.

  • The configuration prefix is changed from spark.datasource.memsql.<config_name> to spark.datasource.singlestore.<config_name>.

    Note

    You can still use the previous variants of the configuration prefix, but the names of all the internal classes are updated. If you use these classes, you must update them in your code.

  • The connector uses the MariaDB JDBC driver.

Version 3.1 > 3.2

  • The SingleStore Spark Connector now reads in parallel from aggregators, and added support for following parallel read features: readFromAggregators, readFromLeaves, and readFromAggregatorsMaterialized. See Parallel Read Support for more information.

Version 3.2 > 4.0

  • Update the JDBC driver to use the SingleStore JDBC driver instead of MariaDB JDBC driver. See The SingleStore JDBC Driver for more information.