SparkConf is used to specify the configuration of your Spark application. This is used to set Spark application parameters as key-value pairs. For instance, if you are creating a new Spark application, you can specify certain parameters as follows: val conf = new SparkConf() .setMaster(“”local[2]””)

What is the use of Spark conf?

Class SparkConf. Configuration for a Spark application. Used to set various Spark parameters as key-value pairs. Most of the time, you would create a SparkConf object with new SparkConf() , which will load values from any spark.

Where is Spark config?

The default Apache Spark configuration directory is $SPARK_HOME/conf. In accordance with the Filesystem Hierarchy Standard (FHS), this task creates a new configuration directory under /etc.