Evaluate your open source application for porting to IBM Power Systems

This tutorial is part of the Learning path: Port your open source applications to Linux on Power series.

Topic Type
Introduction to Linux on IBM Power Systems Article
Evaluate your open source application for porting to Linux on Power Tutorial
Linux on IBM Power Systems application porting and tuning guide Tutorial
15 Porting and tuning tools for Linux on Power Tutorial
Porting to Linux on Power: 5 tips that could turn a good port into a great port Article
Port RStudio Server on IBM Power Systems (coming soon) Tutorial
Build MongoDB on IBM POWER9 (coming soon) Tutorial

Applications used by enterprises are mostly quite complex, being composed of a wide variety of commands, tools, and libraries. Often there is a mix of open source and closed source code. In this tutorial, we describe a series of steps, specifically focused on open source components that can be followed to evaluate the complexity and understand the requirements for porting the application suite to the Linux on Power (ppc64le) platform.

This tutorial describes a four-step evaluation process using an example, Jitsi, which is a collection of open source voice, video conferencing, and instant messaging applications.

Step 1: Locate the source code and licenses, initiate an early community interaction

There may be some portions of the application code which are proprietary, and you will have to determine if you can make necessary changes when porting the application to ppc64le. It is therefore important to:

  • Locate the source code needed to build the application, ensuring that it resides in a repository (for example, in GitHub) where you can ultimately raise pull requests or submit patches. If a new project is available as a .tar file, your ability to make code contributions is hampered.
  • Review project issues, mailing lists, and other communication channels used by the community and look at past contributions to understand the community’s openness towards accepting code from external contributors.
  • Identify the licenses and Contributor License Agreement (CLA). Most repositories contain a LICENSE file while CLA requirements are generally captured in CONTRIBUTING.md.
  • Interact with the community users early using mailing lists and feature requests, or project issues to gauge their interest (for example, https://community.jitsi.org/t/support-for-ibm-power-linux-le-ppc64le-platform/75463).

These steps should be relatively quick and simple. For instance, finding a GitHub repository can be as simple as searching the web for GitHub along with the package name. There may be multiple repositories to look through for some applications – usually you want to find the most official version. Reviewing GitHub issues and pull requests should tell you if other people are submitting patches and if those patches are getting accepted. Projects use different methods of communication. Sometimes, mailing lists and Slack channels can be used to communicate with the community users to see if they are open to receive patches for ppc64le. You may want to look at the code and the closed pull requests to see if the community supports another architecture, such as ARM, basically, to understand if they have previously accepted architecture-specific code.

Licenses such as BSD, MIT, Apache 2.0, and so on are generally considered quite friendly for both contributors and users of the project. CLAs are a little more complex. There are some very standard CLAs, (for example, Linux Foundation, Apache Foundation, and so on) which have both corporate and individual CLAs. CLAs generally apply to significant contributions, and may not come into play for minor contributions, but do make sure that you are familiar with the project’s license and CLA. Consult a legal adviser if unsure.

With experience, the above steps might take as little as a few minutes to an hour on a typical project. More complex projects may take longer to understand the community and the scope of the open source project.

Identifying and recursively evaluating other open source dependencies on which the main project depends is also crucial in understanding the overall complexity of the port upfront. This can be accomplished by the following ways:

  • Building instructions and developer guides such as dev-guide-web provide a good starting point.
  • Reviewing the contents of the Dockerfile as in the case of docker-jitsi-meet can unearth more cases, such as use of x86_64-specific binaries
  • Checking to see if the community is using and maintaining their own repositories of prebuilt architecture-specific binary files of the various application components. And in some cases, checking even their dependencies (like the one at download.jitsi.org) early may give you additional insights. Larger applications with several interlinked components generally tend to adopt this approach and it usually means some additional work and collaboration with the community to ensure that the ppc64le binary files are also available from an equivalent location after the porting process is complete.

Step 2: Become familiar with the application’s architecture and how the components fit together

The application architecture gives you an initial idea of the overall size of the application. In general, more number of components would imply the need for more in-depth understanding of the application and its domain, for integration and validation of all the ported pieces.

The following steps can get you started on this:

  • Review the project documentation and architecture to understand the components that it is made up of.
  • Be aware of your use case and requirements to understand what needs to be ported to Power.
  • A quick, high-level review of the entire source tree like jitsi is also helpful in mapping key components to their sources.

As with Step 1, with experience, the review of the documentation, architecture, and source tree as stated above could be fairly quick for small projects. However, the larger the project and the more complex its components, the more time this could take. One option in the case of large projects is to first ensure that top of tree currently builds on x86_64. Checking recent Continuous Integration (CI) logs, trying the build yourself, or checking with the community could be some ways to do this. Another option is to identify a release of the project close to top of tree that is stable, and then attempting to run the full build scripts on the Power server. Even if that build has major failures, it helps you to focus on the areas that you may need to review more thoroughly from a community and documentation point of view.

Note: Although it is preferable to contribute with fixes than report problems, some communities are amenable to a simple GitHub issue that requests that they support ppc64le and they may have someone willing to do the work. Usually, their first question is where they can get ppc64le hardware, and luckily there are a few places from where they can be provided access to virtual machines in the cloud for porting. In any case, if you have a build failure, you can interact in the community on how to fix it and where to get the information from, and get help with the most relevant documentation. Also, if you are able to successfully interact with the members in the community and they know your intentions, they can help guide your porting work and be more likely to accept your changes when you raise them through a pull request.

Step 3: Know your target distribution and delivery mechanism

Communities typically work with and support one or two distributions which can be determined by:

  • Reviewing their public CI (for example, Travis CI, CircleCI, Jenkins, and so on) jobs
  • Looking at their home, downloads (download.jitsi.org), or release pages
  • Checking base/Dockerfile
  • Reviewing the list of officially or community supported hub.docker.com images

Obviously, you must know what your intended porting target is. Keep in mind that most communities don’t have Red Hat Enterprise Linux (RHEL) licenses and are more likely to support distributions such as Ubuntu, Fedora, or CentOS. The more aligned you are with their supported distributions, the easier your porting process will be. Because the contents of distributions vary substantially, the underlying dependencies for porting to, say, RHEL, are often more complex then porting to Ubuntu.

Step 4: Evaluate dependencies

Perform the following tasks to evaluate dependencies:

  • Check for availability.
    • OSPAT should be your starting point for checking if any of the dependencies are already available on Power. There are currently thousands of packages that are either available as part of the distribution (that is, have been ported and build instructions available through ppc64le/build-scripts) or in the form of containers at https://hub.docker.com/u/ibmcom. In case of Jitsi, you can find prosody on OSPAT.
    • conda-forge can be used to check the availability of Python and R language packages.
  • Identify the source code language.
    • Java™ based components such as jibri , jigasi, and jitsi-videobridge build without issues on Power. In general, you will find that most Java applications will build fine on Power, unless they have calls to native C or C++ code, in which case the native code needs to be ported and the resulting shared library file must be generated and used appropriately. An example of this is jicofo.
    • go packages (for example, frep) also mostly build without issues on Power unless they have embedded assembly code. Simple changes ensuring appropriate use of GOOS and GOARCH may sometimes be needed to ensure that the Makefile build artifacts such as binary files or Docker images are correctly generated for each architecture. Running automated tests and basic functional validation should ensure that there are no issues.
    • JavaScript based packages (for example, jitsi-meet and etherpad-lite) should mostly work using the npm install command. Although rare, there may be instances where such packages depend on libraries or other packages that may first need to be ported to Power.
    • Python packages such as certbot are usually installable and work using the pip install command.
    • C/C++ packages on the other hand, have embedded assembly or Intel®-specific optimizations and generally need more attention and porting work. Refer to the Introduction to the SDK for Linux on Power article to identify the tools to help porting, tuning, and optimizing C and C++ code.

Closing the loop

In many cases, porting packages of any complexity is an iterative process. Sometimes an initial build that generates thousands of lines of failures may be simplified by identifying a handful of library dependencies that need to be included in a build file. After those are identified and resolved, you may find the output from the next build containing only a handful of failures that you can focus on.

You may also find that the package you are porting also has standard builds for Microsoft® Windows®, MacOS, or other operating systems or environments. You may have to look carefully at those builds to identify dependencies that may not matter on Power, for instance deep linkage into Windows or .NET applications, or functionality that links into specific MacOS management. Not all installation scripts will be appropriate for your use. So, as with step 1, make sure to identify not just the dependencies that you need to identify and locate, but also those that you explicitly don’t need to build.

You may want to think about how the project will be consumed on ppc64le: if it is a container and needs a Dockerfile created or updated, or a binary file created and published on the project’s website, or just build scripts that are needed (for example, in ppc64le/build-scripts).

Finally, as with any port, it isn’t always useful if your port provides single point in time builds. As developers make new changes to the project, you may want to ensure that those are also tested, and feedback goes back to those communities about whether the project still builds and passes its tests on ppc64le. For that, you might want to consider integrating with the project’s existing CI environment. For Power, there are several ways to do this, the easiest in general probably being travis-ci.com which (through the use of a .travis.yml file in the project) allows the project to build on ppc64le hardware for every change going forward.