IBM Spectrum Conductor with Spark addresses challenges that organizations face in deploying Apache Spark. These challenges include integrating Spark into existing environments, managing Spark lifecycles in the face of frequent updates to open-source Spark distributions, and managing numerous Spark deployments within the organization.

IBM Spectrum Conductor with Spark (known previously as IBM Platform Conductor for Spark) is an end-to-end enterprise-grade multitenant solution for Apache Spark. It allows organizations to deploy Apache Spark efficiently and effectively, supporting multiple deployments of Spark, increasing performance and scale, and maximizing resource usage.

Benefits

IBM Spectrum Conductor with Spark eliminates resource silos that are tied to multiple Apache Spark implementations, and provides the following additional benefits:

  • Cuts costs by using a service orchestration framework for distributed Apache Spark workloads to maximize resource utilization
  • Improves performance and efficiency with Spectrum Computing’s resource manager, a proven solution for granular and dynamic resource allocation
  • Simplifies application management by providing a consolidated framework for Apache Spark deployment, monitoring, and reporting

IBM Spectrum Conductor with Spark supports Spark multitenancy, allowing you to run multiple instances of Spark, including different versions, in a shared environment. Different groups can run their own instances of Spark and it is not necessary for all Spark instances to be upgraded in lockstep.

Components

Apache Spark requires a separate resource manager and can interface with various distributed storage models. IBM Spectrum Conductor with Spark addresses both of these requirements:

  • It incorporates Spectrum Computing’s resource manager, a proven technology for granular and dynamic resource management.
  • It supports IBM Spectrum Scale technology for storage management. IBM Spectrum Scale enables high-performance access to a common data to support a scale-out solution and provide a high-availability platform.

IBM Spectrum Conductor with Spark also includes a Spark distribution, making it a complete end-to-end solution for organizations looking to deploy Spark both for exploratory projects and in production environments.

Join the IBM Spectrum Conductor with Spark technical community to connect with other members and collaborate with experts by accessing blogs, forums, and wikis.

Collaborate

Download

Download a free trial

Service requests and PMRs

Request for enhancement

Choose a request type:

Software Lifecycle

Social

IBM Conductor with Spark on Slack IBM Spectrum Conductor with Spark on Twitter IBM Spectrum Conductor with Spark on Facebook IBM Spectrum Conductor with Spark on LinkedIn IBM Spectrum Conductor with Spark on YouTube IBM Spectrum Conductor with Spark on Instagram