2021 Call for Code Awards: Live from New York, with SNL’s Colin Jost! Learn more

Serverless Spark using Knative

Building and deploying an analytic service on Cloud is a challenge. A bigger challenge is to maintain the service. In a world where users are gravitating towards a model where cluster instances are to provisioned on the fly, to be used for analytics and so on, and then to have these cluster instance shut down when the jobs get done, the relevance of containers and container orchestration is more important than ever.

In short Customers are looking for Serverless Spark Clusters. As a service provider we run 1000+ node Cluster in production and running at this scale presents its own challenges. At at this scale, many system components fail, including etcd, Docker image pulls, network, KubeDNS.

Knative is an open source community project which adds components for deploying, running, and managing serverless, cloud-native applications to Kubernetes. This talk covers the challenges involved in providing Serverless Spark Clusters and shares the specific issues one can encounter when running large number of Spark workloads using Knative.