If you are trying to develop machine or deep learning models, chances are that you have used many open source Python libraries. Python packages are commonly found popular open source package repositories such as PyPI and Anaconda Distribution. Some Python packages e.g. Tensorflow, include native hardware and operating system specific libraries. Read on to see how you can help the Conda Forge open source community build more POWER systems hardware architecture (ppc64le) specific packages.

Conda is a popular open source package management system that is often used for building python packages in a platform agnostic way (e.g. for Windows, MacOS, and Linux) and for installing multiple package versions in the same machine. Conda can now be used to build Python packages for the POWER systems architecture (ppc64le) and for the ARM architecture. This article will describe some tools and give you the steps to build a Python conda package for POWER and enable the conda-forge community to publish the package from Anaconda Cloud’s conda-forge channel in the POWER/linux-ppc64le platform specific channel.

The conda-forge open source community maintains thousands of popular open source packages that can be built for multiple platforms. Once the recipe is updated in a github conda feedstock repo, community maintained continuous integration (CI) services automatically build, test and publish these packages to the conda-forge channel for installation (conda install <package> -c conda-forge).

Let me take the example of how I enabled the community to publish a power pygithub conda package using the pygithub feedstock. First find a POWER VM and install Docker in it. Here are some free publicly POWER system available resources for the open source community for requesting a POWER VM. Run the conda forge linux-anvil-ppc6le container (Dockerfile).

root@cit1000:~/ip# docker run -it condaforge/linux-anvil-ppc64le
[conda@c9040d0b9b58 ~]$
[conda@c9040d0b9b58 ~]$ conda activate base
(base) [conda@c9040d0b9b58 ~]$

Install the conda-smithy package:

(base) [conda@c9040d0b9b58 ~]$ conda install conda-smithy

Clone the pygithub feedstock repo:

(base) [conda@c9040d0b9b58 ~]$ git clone https://github.com/conda-forge/pygithub-feedstock.git
...
(base) [root@c9040d0b9b58 conda]# cd pygithub-feedstock/
(base) [root@c9040d0b9b58 pygithub-feedstock]# ls
azure-pipelines.yml  conda-forge.yml  LICENSE.txt  README.md  recipe

Add a line to conda-forge.yml configure the CI system for automatically building the recipe on both the POWER and ARM system architectures:

provider: {linux_aarch64: default, linux_ppc64le: default}

Run the conda smithy tool to update the build scripts under the .ci_support directory:

(base) [root@c9040d0b9b58 pygithub-feedstock]# conda smithy rerender
...
Re-rendered with conda-build 3.17.8, conda-smithy 3.3.4, and conda-forge-pinning 2019.05.11
You can commit the changes with:

    git commit -m "MNT: Re-rendered with conda-build 3.17.8, conda-smithy 3.3.4, and conda-forge-pinning 2019.05.11"

These changes need to be pushed to github!

If you get this error:

RuntimeError: conda-forge-pinning version in root env (2019.05.06) is out-of-date (2019.05.11). Exiting. 

Run:

conda update conda-forge-pinning

Run conda build and conda verify to verify that the package indeed builds properly on POWER:

(base) [root@c9040d0b9b58 pygithub-feedstock]# conda build .
...
Successfully built PyGithub
Installing collected packages: PyGithub
Successfully installed PyGithub-1.43.6
...

Now you can make a pull request:

(base) [root@c9040d0b9b58 pygithub-feedstock]# git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

    modified:   .azure-pipelines/azure-pipelines-linux.yml
    modified:   .azure-pipelines/run_docker_build.sh
    new file:   .ci_support/linux_aarch64_python2.7.yaml
    new file:   .ci_support/linux_aarch64_python3.6.yaml
    new file:   .ci_support/linux_aarch64_python3.7.yaml
    new file:   .ci_support/linux_ppc64le_python2.7.yaml
    new file:   .ci_support/linux_ppc64le_python3.6.yaml
    new file:   .ci_support/linux_ppc64le_python3.7.yaml
    new file:   .travis.yml
    new file:   .travis/build_steps.sh
    new file:   .travis/run_docker_build.sh
    modified:   README.md

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   conda-forge.yml

(base) [root@c9040d0b9b58 pygithub-feedstock]# git add .
(base) [root@c9040d0b9b58 pygithub-feedstock]# git commit -m "MNT: Re-rendered with conda-build 3.17.8, conda-smithy 3.3.4, and conda-forge-pinning 2019.05.11"
[master c5671ce] MNT: Re-rendered with conda-build 3.17.8, conda-smithy 3.3.4, and conda-forge-pinning 2019.05.11
 13 files changed, 310 insertions(+), 4 deletions(-)
 create mode 100644 .ci_support/linux_aarch64_python2.7.yaml
 create mode 100644 .ci_support/linux_aarch64_python3.6.yaml
 create mode 100644 .ci_support/linux_aarch64_python3.7.yaml
 create mode 100644 .ci_support/linux_ppc64le_python2.7.yaml
 create mode 100644 .ci_support/linux_ppc64le_python3.6.yaml
 create mode 100644 .ci_support/linux_ppc64le_python3.7.yaml
 create mode 100644 .travis.yml
 create mode 100755 .travis/build_steps.sh
 create mode 100755 .travis/run_docker_build.sh

Create a fork of the recipe repo and push your changes to your fork:

(base) [root@c9040d0b9b58 pygithub-feedstock]# git remote add ipremote https://github.com/ipoddar-ibm/pygithub-feedstock.git
(base) [root@c9040d0b9b58 pygithub-feedstock]# git push ipremote

Make a pull request to start automated CI builds on all platforms e.g. https://github.com/conda-forge/pygithub-feedstock/pull/36. Upon submission of the pull request, CI builds are automatically kicked off and can be tracked from the pull request. The POWER architecture (ppc64le) builds are done in Travis CI and can be accessed through the Details link as shown below (e.g. here are the build logs for Travis CI job for the pygithub recipe on ppc64le).

Issues can be discussed in the conda-forge-ppc64le gitter-channel.

The author would like to thank community leads and contributors (Isuru Fernando, Marius Van Niekerk, Jonathan Helmus, Jason Furmanek, Kamalochana Subbaiah) for enabling support for the POWER architecture.

Resources

  1. Conda
  2. Community supported conda-forge channel
  3. POWER/linux-ppc64le platform specific conda channel
  4. conda-smithy
  5. Conda-forge-ppc64le gitter-channel

Join The Discussion

Your email address will not be published. Required fields are marked *