Configuration Settings
Warning
SingleStore 9.0 gives you the opportunity to preview, evaluate, and provide feedback on new and upcoming features prior to their general availability. In the interim, SingleStore 8.9 is recommended for production workloads, which can later be upgraded to SingleStore 9.0.
On this page
The SingleStore Spark Connector leverages Spark SQL’s Data Sources API.
The singlestore-spark-connector
is configurable globally via Spark options and locally when constructing a DataFrame.spark.
The connection to SingleStore relies on the following Spark configuration options:
Basic Options
Option |
Description |
Default Value |
---|---|---|
|
The hostname or IP address of the SingleStore Master Aggregator in the |
|
|
The hostname or IP address of SingleStore Aggregator nodes to run queries against in the |
|
|
SingleStore username. |
|
|
SingleStore password. |
|
|
The query to run (mutually exclusive with |
|
|
The table to query (mutually exclusive with query). |
|
|
If set, all connections use this database by default. |
Read Options
Option |
Description |
Default Value |
---|---|---|
|
Disables SQL Pushdown when running queries. |
|
|
Enables reading data in parallel for some query shapes. |
|
|
Specifies a comma separated list of parallel read features that are tried in the order they are listed. |
|
|
Specifies the amount of time (in |
|
|
Specifies the amount of time (in |
|
|
Specifies the maximum number of partitions in the resulting DataFrame. |
|
|
Repartitions data before reading. |
|
|
Specifies a comma separated list of columns that are used for repartitioning (when |
Write Options
Option |
Description |
Default Value |
---|---|---|
|
Specifies the behavior during |
|
|
This option is deprecated, please use |
|
|
Compresses data on load. |
|
|
Serializes data on load. |
|
|
Specifies additional keys to add to tables created by the connector. |
|
|
If this option is specified and a new row with duplicate |
|
|
Specifies the size of the batch for row insertion. |
10000 |
|
The maximum number of errors in a single |
|
|
If enabled, the connector creates a rowstore table. |
|
Connection Pool Options
Option |
Description |
Default Value |
---|---|---|
|
Enables the use of connection pool on the driver. |
|
|
The maximum number of active connections with the same options that can be allocated from the driver pool at the same time. |
-1 |
|
The maximum number of connections with the same options that can remain idle in the driver pool without extra ones being released. |
8 |
|
The minimum amount of time (in ms) an object may sit idle in the driver pool before it is eligible for eviction by the idle object evictor (if any). |
30000 (30 sec) |
|
The number of milliseconds to sleep between runs of the idle object evictor thread on the driver. |
1000 (1 sec) |
|
The maximum number of milliseconds that the driver pool waits (when there are no available connections) for a connection to be returned before throwing an exception. |
-1 |
|
The maximum lifetime of the connector (in ms) after which the connection fails the next activation, passivation or validation test. |
-1 |
|
Enables the use of connection pool on executors. |
|
|
The maximum number of active connections with the same options that can be allocated from the executor pool at the same time. |
-1 |
|
The maximum number of connections with the same options that can remain idle in the executor pool, without extra ones being released. |
8 |
|
The minimum amount of time an object may sit idle in the executor pool before it is eligible for eviction by the idle object evictor (if any). |
2000 (2 sec) |
|
The number of milliseconds to sleep between runs of the idle object evictor thread on the executor. |
1000 (1 sec) |
|
The maximum number of milliseconds that the executor pool waits (when there are no available connections) for a connection to be returned before throwing an exception. |
-1 |
|
The maximum lifetime of the connector (in ms) after which the connection fails the next activation, passivation or validation test. |
-1 |
Last modified: February 23, 2024