TDD or Test Driven Development has become one of the cornerstones of agile programming methodologies in the last decade or so. Nowadays most developers are writing test cases and test suites based on a complete set of mock objects before they are actually getting to the real programming tasks at hand.

TDD provides several advantages: producing highly testable and high quality code, providing great code coverage, higher customer satisfaction, and better code modularity.

While I was doing some Apache Spark coding , I was missing a small but vital functionality of testing my Scala code just as easily as I do my Java code using JUnit. Here I share steps I took to enable Scala application testing using JUnit.

Step 1: Enable JUnit 4.0 integration with your Spark project in Intellij

Create a proxy Java class in the Intellij Java src/java directory structure (as presented by the image “listing 01” below) called This proxy class can be later deleted, as after we add the support of JUnit to the project we do not need it anymore. We could utilize it for building up our tests later. It is up to you.

sample java project structure
Listing 01: Sample Java project structure

When created, click the class name in the editor and go into the “Intention Actions” dialog, the yellow bulb image “listing 02”
Intention Actions.
Listing 02: Intention Actions

Once this was achieved, add the support for JUnit 4 to your project’s support setup (listing 03).
adding JUnit 4.0 support to the project setup
Listing 03: Adding JUnit 4.0 support to the project setup

Step 2: Code your RDD class (if required)


public class MyAwsomeClass implements Serializable {

private String a ;
private String b ;
private String c ;

public MyAwsomeClass(String a,String b,String c){
this.a = a ;
this.b = b ;
this.c = c ;

Step 3: Code your Scala application

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkContext, SparkConf}
import org.junit.Test

class Controller {

def Controller(): Unit ={

def doSomething(): Unit = {

//use this to load a single file
val subsetFile = “C:\\input\\data\\A1Benchmark\\real_1.csv”

val conf = new SparkConf().setAppName(“Controller1”).setMaster(“local”)
val sc = new SparkContext(conf)

val timesSeriesRdd: RDD[MyAwesomeClass] = sc.textFile(subsetFile).map { line =>
val fields = line.split(“,”)
new MyAwesomeClass(fields(0), fields(1), fields(2))
// print out the number of lines within the RDD

Step4: Code your JUnit test class

The code should be created under the test/java directory , see “listing 04” below.

Test directory structure
listing 04: JUnit directory structure

import org.junit.Test;

public class MyAwesomeTest {

public void doControllerTest(){

Controller controller = new Controller();


Step 5: Running the JUnit test

Run the JUnit as a standalone test and then verify the expected results (example can be found in “listing 05” below ).

JUnit Test Results
Listing 05: JUnit Test Results

If everything goes well thus far, you can now test your Scala/Spark code using JUnit. That is great progress. But what is next? Now we can go further to create a JUnit test suite. Test suites are a very convenient method of aggregating multiple scenarios into a single logical bundle.

Step 6: Coding the JUnit suite

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

public class TestSuite {


After coding the suite, we can run all of the classes inside the test suite as a single logical bundle.

More information on setting up Intellij with Apache Spark support can be found here: Setting up intellij with Apache Spark.

Join The Discussion

Your email address will not be published. Required fields are marked *