Yonggangby Yonggang Hu, Chief Architect – Application Middleware, Platform Computing

Are you up to speed on Spark? Want to integrate it with an existing environment, or wondering how to get started? IBM® Platform™ Symphony can help you run and manage Spark more efficiently and more flexibly in a cloud environment. Some of the key management capabilities of Platform Symphony will be in the IBM Big Insights Enterprise Management package.

Spark is a powerful open-source processing engine for big data. It offers unique value and features such as a built-in DAG scheduler, in-memory computation, and a unified programming model. However, there are challenges in creating and managing Spark applications in an enterprise data center to meet business demands.

We have developed a design and implementation that uses IBM Platform Symphony to address some of these challenges and to provide a true multi-tenancy Spark cloud environment. It allows users to dynamically create, add, and remove Spark instances and to manage Spark application SLAs. It also demonstrates the ability to run and manage multiple instances and multiple versions of Spark applications (including development, UAT, and production code) from different users, groups, and lines of business, both on premise and in the cloud (a hybrid cloud environment).

The Platform Symphony EGO resource manager not only supports fine-grained resource allocation on demand, but also provides the cloud administrator with a policy-based resource plan to control application SLAs and maximize infrastructure efficiency.

A notebook-based interface is introduced to provide an easy-to-use, interactive user experience. A Spark tenant (context, executor) can be shared across different users, while at the same time different tenants can be configured to run with application isolation and security.

Finally, the IBM Application Service Controller for Platform Symphony allows the end user to run and manage other application components (e.g., Storm, MapReduce, etc.) in the same way as the Spark application in order to compose more complex business applications.

What is IBM Platform Symphony? And what is IBM Application Service Controller for Platform Symphony?

IBM Platform Symphony is an application and infrastructure middleware solution that delivers powerful enterprise-class management for running a wide variety of distributed applications and big data analytics on a scalable, shared grid. It manages and accelerates large numbers of parallel applications, for faster results and better utilization of all available resources. For more information on Platform Symphony, please go on this site: IBM Platform Computing

The IBM Application Service Controller for Platform Symphony add-on extends the IBM Platform Symphony grid to provide a shared-service backbone for a broad portfolio of distributed software frameworks. By enabling a wide variety of applications to share resources and coexist on the same infrastructure, the IBM Application Service Controller for Platform Symphony helps organizations reduce cost, simplify management, increase efficiency and improve performance.

Are you interested in learning more about how IBM Platform Symphony can help solve some of your critical customer challenges of running Spark on Hadoop? Come and see the demo of running a multi-tenant Spark environment either on premise or in the cloud.

To learn more about IBM Platform Symphony and Spark, join us at Strata + Hadoop World in San Jose IBM’s booth (booth #1115) and see these technologies in action. Live demonstrations will be conducted at the IBM booth Thursday Feb 19th at 12:00 PM and Friday Feb 20th at 12:00 PM. #Strataconf

Please join the on-going conversation on Twitter using #HadoopNext

Join The Discussion

Your email address will not be published. Required fields are marked *