While the IBM Power platform has proven a valuable asset in tackling machine and deep learning challenges (MLDL), installing software necessary for such tasks has been quite challenging on any architecture even with many groups providing support. Popular tools such as Tensorflow and PyTorch depend on many python libraries and the use of finicky installation tools such as Bazel to help bring them to life. To make matters worse, high performance settings call for the use of certain combinations of tool versions, introducing even more compatibility issues. This means that the process of installation on any platform would involve going down many dependency rabbit holes, many of which led to dead ends. Many groups have turned toward Anaconda (or Miniconda) to help reduce the burden of installing tools and increase usability. Unfortunately, many times these conda based packages are not architecture-aware and cannot fully take advantage of the system accelerators. To get full speed processing out of your hardware, many groups are back looking at how to compile their software stacks.

However, this all changed when IBM introduced their Watson Machine Learning Community Edition (WML CE) under the standard Anaconda system. The IBM WML CE provides MLDL framework to both x86 and PPC64LE architectures, allowing groups to run a heterogeneous environment without having to worry about the code base. With the new conda packaging installing any popular MLDL tool on an IBM Power system machine is as simple as running one install command. The WML CE framework provided is fully optimized for all x86 or PPC64LE architectures and removes any version compatibility issues. This has lowered the labor of installing MLDL software from many hours to a 15 minute job.

The access and availability of WML CE resource has greatly impacted the IBM Power platforms’ usability for MLDL applications. This new pathway for accessing standard MLDL tools has synchronized the software stacks between architectures, increasing the scope of work we can accomplish. For many groups, the difficulty of installing these tools have forced IT staff to not wade into the world of MLDL for fear of having to manage the ever changing software stack. With the conda packaging for MLDL with full architecture acceleration, the hurdle of install and management has been eliminated and tools such as Caffe, PyTorch, and Tensorflow are now much more accessible on all machines.

https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/


Authors:

Michaela Buchanan
Lead DL Computational Undergraduate
Center for Genome Research and Biocomputing
Oregon State University

Christopher M. Sullivan
Assistant Director for Biocomputing
Center for Genome Research and Biocomputing
Oregon State University

Join The Discussion

Your email address will not be published. Required fields are marked *