Apache Spark continues to be one of the most prominent names in the world of Big Data technologies. Spark adds lots of features to the basic Scala language, but the 2 most prominent ones are RDD’s (Resilient Distributed Dataset) and the Spark Shell (REPL).

If you have ever tried programming in Scala to run on Apache Spark, you would likely get to a point that you would say to yourself “the shell is a nice tool for debugging , but i need more than this if i want to be more productive.” Modern IDE’s provide a large variety of tools to support enhanced productivity, tools like syntax highlighting, auto completion, debugging, and so on.

Intellij IDEA Community Edition provides a very effective IDE to work with Apache Spark source code, and here are some examples.

Syntax highlighting

Syntax highlighting on Intellij community edition

Apache Spark debugging

Apache Spark debugging on intellij community edition

Running standalone Spark applications on your local machine

Running a standalone application on Intelij community edition

If you find some of these features applicable to your needs, you might want to configure your machine to work with Intellij IDEA Community Edition. Here are the instructions of how to do so — Download the full article here.

IBM BigInsights 4.1, IBM distribution for Hadoop, supports Apache Spark 1.4.1 out of the box. To work with a Spark cluster configured using BigInsights 4.1, you simply configure the same version of Spark on your desktop machine where IntelliJ is installed.

Join The Discussion

Your email address will not be published. Required fields are marked *