The latest version of PowerAI is here with version 1.6.0! We’ve been hard at work transforming the packaging and delivery model, updating the versions of the included frameworks and packages and adding new features. Let’s walk through the major changes in 1.6.0.
Conda Packaging and the PowerAI conda channel
Previous releases of PowerAI introduced interoperability with Anaconda’s python distribution. This provided a consistent and optimized source for python package dependencies and add-on packages typically needed in data science and Artificial Intelligence workloads. PowerAI was still mostly provided using Linux distribution specific packaging, however. With version 1.6.0, PowerAI is moving completely to the conda environment. This transition provides a tremendous set of benefits.
Natural package dependency resolutions
Dependencies are now coded directly in each of the installation packages. Conda offers a full featured packaging solution and dependencies are determined and installed at package installation time. In addition, conda packages are built with a toolchain that is independent from the operating system, which allows for OS independent packages. The result is no more separate .rpm or .deb packages, targeted at the different distros, just a single set of conda packages for everywhere.
Open and continuous delivery
Now that PowerAI 1.6.0 is completely converted to conda packages, it allows us to simplify (and amplify!) the delivery model. PowerAI will now have an official online conda channel:
This should help in not only obtaining the PowerAI packages, but also with downstream continuous integration environments. Fixes, updates, and “ship when ready” additions are possible with this delivery model, which are exciting new capabilities for PowerAI.
Simplified installation and system management
Another exciting change in PowerAI 1.6.0 is that NVIDIA’s CUDA, cuDNN and NCCL are provided as conda packages and no longer need to be installed system-wide. Note that the NVIDIA device driver itself is not included and will still need to be installed separately. Check the updated IBM Knowledge Center for the full details on installation.
PowerAI 1.6.0 includes CUDA version 10.1, cuDNN 7.5 and NCCL 2.4.2. All GPU enabled packages have been built against these versions. Providing these important dependencies as packages in the conda channel allows them to be installed, depended upon, updated and managed like any other. Additionally, conda’s conda env features allows for package sets to be installed only to certain specified and insulated conda environments, if desired.
Meta package control
PowerAI 1.6.0 also has some additional controls for the package set. The powerai and powerai-release meta packages have been introduced so that it is possible to install the 1.6.0 package versions even when newer packages are introduced in the future. The powerai meta package will install the entire set of base PowerAI packages. It’s versioned to allow installation of particular PowerAI release. For example, to install the PowerAI package set released with version 1.6.0:
conda install powerai=1.6.0
The powerai-release meta package offers even more flexibility by allowing the installation of individual packages the correspond to a particular PowerAI release. For example:
conda install tensorflow-gpu powerai-release=1.6.0
That will install Tensorflow from PowerAI 1.6.0 without unrelated packages.
So what else is new?
As is typical for a new PowerAI release, we’ve included the latest versions of our included frameworks as well as some new additions. Operating System support has been updated to RHEL 7.6 and Ubuntu 18.04.
The Tensorflow set of packages have been updated and include Tensorflow (1.13.1), Tensorboard (1.13), Tensorflow-Probability (0.6), Tensorflow-Estimator(1.13). Tensorflow Large Model Support (TFLMS) version 1 is still included and built in to the Tensorflow package, but we’ve also included a brand new version 2 of TFLMS as a separate add-on package. Stay tuned for more information about this package in a future blog post!
Pytorch has reached 1.0.1 and also includes an LMS feature of its own. Caffe (1.0), Caffe2 (1.0.1) and ONNX (1.3.0) also receive version bumps. (again check the Knowledge Center for all of the details)
IBM Distributed Deep Learning (DDL) moves to version 1.3.0 and picks up improved support for IBM Platform LSF as well as Tensorflow’s eager mode.
IBM pai4sk moves to version 1.3.0 and adds integration with Rapids cuML (0.2.0) and cuDF (0.2.0). pai4sk now supports MPI based distribution of data-loading, training, prediction, and metrics for SnapML solvers. SnapML includes CPU accelerated versions of Decision Tree and Random Forest algorithms, along with multi-class classification for Spark.
Docker image updates
Separate Images for Individual Frameworks
Last but not least, we’ve updated the docker images hosted on Dockerhub.com. The images are now based on Ubuntu 18.04. In addition to default image that includes all of PowerAI, we’ve also included Docker images for each individual framework. Separate images are included for Tensorflow, Pytorch, Caffe, and SnapML. There is also a base image available that has none of the PowerAI packages installed, but does have the channel enabled. This can be used as a foundation for customized images.
More information can be found on Dockerhub.com at:
With PowerAI 1.6.0, we aim to reduce the system management burden, increase flexibility and as a result, the velocity of your development pipelines. We’re excited about these changes and hope to hear from you so we can make PowerAI even better.