Brian Burns is a developer on the EclairJS project.
Apache’s Spark is one such engine—although it is unique in that it is actually optimized for several different types of compute tasks. In particular, Apache Spark has:
- An SQL engine
- A component for querying and analyzing streaming data
- A rich set of Machine Learning (ML) algorithms
- An engine for querying and manipulating graph representations of data such as social networks
Moreover, these capabilities are integrated so that you can, for example, run queries against streams of data. Last but not least, Spark is fast because it has parallelized operators which work on data kept in memory whenever possible, and it is massively scalable from your laptop to thousands of nodes in distributed clusters.
EclairJS is particularly suited to applications that are “chatty”, such as user-facing and interactive applications that use input from users to drive analyses or other operations that are performed in Spark. Of course, the ultimate case of “chatty” applications are those that handle streams of data, such as applications that continuously update visualizations like maps or graphs. EclairJS uses web sockets to communicate between Node applications and Spark, and these are ideally suited to handle streaming data.
We hope you find EclairJS useful, and we are very keen on recruiting new contributors and learning about new use cases!
Why should I contribute?
Spark is one of the fastest growing projects at Apache, and it is quickly making inroads into many businesses and enterprises. By becoming involved in the EclairJS project now, you can get a jump on this new technology, and be prepared to make use of its wide range of functionalities from within your existing Node.js web app development environment.
In addition, EclairJS strives to mirror as much of the Spark API as possible. Admittedly, the project’s existing contributors have their own biases and will prioritize those parts of the API that are most important to them. So, if there is a particular Spark operation or set of operations that would be useful for your project, it may be quicker to contribute them to the project now rather than wait for today’s contributors to implement them.
What technology problem will I help solve?
Spark has a number of different language interfaces already built-in, namely Scala, Java, Python plus some support for R. However, it is missing any built-in capability to support typical web app development platforms such as Node.js.
How will EclairJS help my business?
EclairJS blog posts
See how to run multiple Spark jobs from one node instance.
Get step-by-step instructions for how to write your first Node.js Express app with EclairJS.