Best practices for porting open source applications to Linux on IBM Power

Overview

Enterprise applications are typically quite complex, composed of a wide variety of commands, tools, and libraries. Often there is a mix of open source and closed source code in any major application. This tutorial describes a series of steps (specifically focused on open source components) that can be followed to evaluate the complexity and understand the requirements for porting the application suite to the Linux on Power (ppc64le) platform. Most of these steps would also apply to proprietary source code which needs to be ported to the Linux on Power platform as well.

Learning objectives

This tutorial describes a four-step evaluation process using an example, Jitsi, which is a collection of open source voice, video conferencing, and instant messaging applications.

Prerequisite knowledge

Following are the prerequisites that you need to fulfill before you begin with the porting process.

• Basic Power architecture and porting
• Docker concepts
• Basic open source concepts

Skill level

Beginner

Estimated time to complete

It takes about 2-4 hours to complete the porting process.

Porting open source applications to Linux on Power

Complete the following set of comprehensive steps to port your open source application to Linux on Power.

Step 0: Check if the application is already ported to Linux on Power

Many applications are already ported to Linux on IBM Power and may not need any additional work. The simplest ways to try this are:

• Check out the indexes of ported applications (OSPAT) and see if the application is available somewhere already.
• If it is not listed in the indexes, import or copy the source code to your Power server, follow the normal build instructions, and see if it works. It might just be that easy!
• Finally, check our repository to see if there is a build script for the application there.

You may be lucky to find that you have no porting work to do! However, if these do not show the results you are hoping for, you can continue with the following steps to port the software.

Step 1: Locate the source code and licenses and initiate an early community interaction

Occasionally, you will be the first person who wants an application or project to be available on Linux on IBM Power. These steps will show you how to do a proper port, assuming that you are planning to submit any changes you performed in the project.

1. Review license: In an open source world, the usual expectation is that the broad community, including you, contribute to the development and porting of all applications. While it is possible in some cases to not contribute your changes back upstream (to the official source code repository) you will have to review the license to see if it is permitted to withhold the changes and still make your final application available for use based on the terms of the licenses. As a result, some of the steps we advocate here could technically be skipped, for example, in the case of an evaluation, you can opt to skip the generation of dependencies for an application.
2. Evaluate code before submission: Many projects can be ported through simple updates to a makefile or a Dockerfile in the case of containers (for example, pom.xml for maven-based Java projects, CMakeLists.txt when we have cmake as the build system). Other files like package.json for node.js projects, and setup.py and requirements.txt for Python projects typically don't need any changes at all. In fact, the vast majority of open source projects we have seen could be ported this way in a day or less, if you are familiar with the Linux distributions, packaging, build environments, and so on. However, the initial open source evaluation is important if you are planning to submit your code upstream, or use the resulting ported software in your application.

3. Consult with legal department: Open source code is licensed under a moderately sized set of standard licenses. Identifying safe licenses is beyond the scope of this learning path, but web sites like Open Invention Network (OIN), the SPDX have a lot of guidance and general information. You need to determine if you can make the necessary changes when porting the application to ppc64le, and if you work for a corporation, you may need to review your open source work with your legal department. Refer to the following steps to be prepared for that discussion, or simply to ensure you understand your rights as a software developer working on this project, and to set the groundwork: have a lot of guidance and general information. You need to determine if you can make the necessary changes when porting the application to ppc64le, and if you work for a corporation, you may need to review your open source work with your legal department. Refer to the following steps to be prepared for that discussion, or simply to ensure you understand your rights as a software developer working on this project, and to set the groundwork:

   • Locate the source code needed to build the application, and ensure that it resides in a repository (for example, in GitHub) where you can ultimately raise pull requests or submit patches. Your ability to contribute code to a new project is hampered if it is only available as a .tar file.
   • Review project issues, mailing lists, and other communication channels used by the community and look at past contributions to understand the community's openness towards accepting code from external contributors.
   • Identify the licenses and Contributor License Agreement (CLA). Most repositories contain a LICENSE file while CLA requirements are generally captured in CONTRIBUTING.md.
   • To gauge the interest of community users, interact and engage earlier using mailing lists, feature requests, and project issues.

4. Find a GitHub repository: Finding a GitHub repository can be as simple as searching the web for GitHub + package name. There may be multiple repositories to look through for some applications – usually, you want to find the most recent official version.

5. Review GitHub issues and pull requests: Reviewing GitHub issues and pull requests should tell you if other people are submitting patches and if those patches are getting accepted. However, some communities have broader communication mechanisms than just GitHub. You may need to look for references to project-specific mailing lists and Slack channels. You may pose informal questions and find quick answers to any questions. Typical questions might include to check if the community is open to receiving patches, if/how they would do testing on Power servers (Oregon State University Lab), and how they would do continuous integration and continuous deployment (CI/CD) if needed. This may also be the place for technical questions about the porting applications.

   

6. Study closed pull requests: You may want to look at the code and the closed pull requests to see if the community supports another architecture, such as ARM, to understand if they have previously accepted architecture-specific code. You can also tell how active the community is, how many contributors are available, how recently issues were raised or code was merged, and so on. You can also see which continuous integration (CI) environment the community is using for regular testing. The following image highlights the aspects to be observed and understood.

   View larger image

7. Evaluate licenses: When reviewing the community, as previously mentioned, you will want to evaluate the licenses based on your intended uses. Licenses such as Berkeley Source Distribution (BSD), Massachusetts Institute of Technology (MIT), Apache 2.0, and so on are generally considered quite friendly for both contributors and users of the project. Contributor License Agreements (CLAs), are a little more complex. There are some very standard CLAs, (for example, Linux Foundation, Apache Foundation, and so on) that have both corporate and individual CLAs. CLAs generally apply to significant contributions, and may not come into play for minor contributions, but do make sure that you are familiar with the project's license and CLA. Consult a legal adviser if unsure.

With experience, the above steps might take as little as a few minutes to an hour on a typical project. More complex projects may take longer to understand the community and the scope of the open source project.

Identifying and recursively evaluating other open source dependencies on which the main project depends is also crucial in understanding the overall complexity of the porting process up front. This can be accomplished in the following ways:

• Review the contents of the Dockerfile, for example, docker-jitsi-meet to unearth more cases, such as the use of x86_64-specific binaries.
  • An x86_64 binary file is likely to be used when the application code has not been ported and supported on other architectures. This can be easily handled by replacing the hard-coded references to a particular architecture (in this case, amd64) by determining the system architecture at build time. This can be achieved by adding conditional code in the appropriate build files and by using –build-arg to pass architecture as an argument to the docker build command. This is easy to implement, especially when the equivalent IBM Power binary files, such as s6-overlay-ppc64le.tar.gz, are already available.
  • However, in some cases, when the equivalent ppc64le binary files are not present, more effort is required in terms of porting and working with the community to make them available. Here, the ppc64le support for frep will first need to be enabled using a pull request and then interaction with the frep community is needed to make the binary file for ppc64le available officially.
• Find if the community is using and maintaining their repositories of prebuilt architecture-specific binary files of the various application components. And in some cases, checking their dependencies (like the one at download.jitsi.org) at an early stage may give you additional insights. Larger applications with several interlinked components generally tend to adopt this approach and it usually means some additional work and collaboration with the community to ensure that the ppc64le binary files are also available from an equivalent location after the porting process is complete.

Step 2: Become familiar with the application's architecture and how the components fit together

The application architecture gives you an initial idea of the overall size of the application. In general, a greater number of components would imply the need for a more in-depth understanding of the application and its domain, for integration and validation of all the ported pieces.

The following guidelines can get you started on this:

• Review the project documentation and architecture to understand the components that it consists.

• Be aware of your use case and requirements to understand what needs to be ported to Power.

• Perform a quick, high-level review of the entire source tree like jitsi to map key components to their sources.

As mentioned in step 1, the architecture and the source tree as stated above could be fairly quick for small projects. However, larger the project and more complex its components, the longer it may take. One option in the case of large projects is to first ensure that the top of the tree is currently built on x86_64. Checking recent CI logs, trying the build yourself, or checking with the community could be other ways to do this. Another option is to identify a release of the project close to the top of the tree that is stable, and then attempt to run the full build scripts on the Power server. Even if that build has major failures, it helps you focus on the areas you may need to review more thoroughly from a community and documentation point of view.

Note: Most communities prefer patches and pull requests to GitHub issues, but sometimes you can make a general request for the community to support a ppc64le release of the project. Typically, their first response to this request is where they can get ppc64le hardware, and luckily there are a few places from where they can be provided access to virtual machines in the cloud for porting.. In any case, if you have a build failure, you can interact in the community to find a fix and seek guidance to get the most relevant documentation. Also, if you can successfully interact with the members of the community and they know your intentions, they can help guide your porting work and be more likely to accept your changes when you raise them through a pull request.

Step 3: Know your target distribution and delivery mechanism

Communities, typically work with and support one or two distributions that can be determined by:

• Reviewing their public CI (for example, Travis CI, CircleCI, Jenkins, and so on) jobs
• Looking at their home, downloads (download.jitsi.org), or release pages
• Checking base/Dockerfile
• Reviewing the list of officially or community-supported hub.docker.com images

You must know what your intended porting target is. If the community is typically building on Ubuntu and you need an RHEL port, your work may be a bit more difficult. Most communities don't have Red Hat Enterprise Linux (RHEL) licenses and are more likely to support distributions such as Ubuntu, Fedora, or CentOS. The more aligned you are with their supported distributions, the easier your porting process will be. Because the contents of distributions vary substantially, the underlying dependencies for porting to (for example, RHEL) are often more complex than porting to Ubuntu.

Step 4: Evaluate dependencies

Perform the following tasks to evaluate dependencies:

• Check for availability.
  • The OSPAT tool should be your starting point for checking if any of the dependencies are already available on Power. There are currently thousands of packages that are either available as part of the distribution (that is, have been ported and build instructions available through ppc64le/build-scripts) or available in the form of containers at https://hub.docker.com/u/ibmcom. In the case of Jitsi, you can find prosody on OSPAT.
  • conda-forge can be used to check the availability of Python and R language packages.
• Identify the source code language.
  • Java-based components such as jibri, jigasi, and jitsi-videobridge build without issues on Power. In general, you will find that most Java applications will build fine on Power unless they have calls to native C or C++ code, in which case the native code needs to be ported and the resulting shared library file must be generated and used appropriately. An example of this is jicofo.
  • go packages (for example, frep) also mostly build without issues on Power unless they have embedded assembly code. Simple changes ensuring appropriate use of GOOS and GOARCH may sometimes be needed to ensure that the makefile build artifacts such as binary files or Docker images are correctly generated for each architecture. Running automated tests and basic functional validation should ensure that there are no issues.
  • JavaScript-based packages (for example, jitsi-meet) should mostly work using the npm install command. Although rare, there may be instances where such packages depend on libraries or other packages that may first need to be ported to Power.
  • Python packages such as certbot are usually installable and work using the pip install command.
  • C/C++ packages, on the other hand, have embedded assembly or Intel -specific optimizations and generally need more attention and porting work.

Closing the loop

In many cases, porting packages of any complexity is an iterative process. Sometimes an initial build that generates thousands of lines of failure information may be simplified by identifying a handful of library dependencies that need to be included in a build file. After those are identified and resolved, you may find the output from the next build containing only a handful of failures that you can focus on.

You may also find that the package you are porting also has standard builds for Microsoft Windows, MacOS, or other operating systems or environments. You may have to look carefully at those builds to identify dependencies that may not matter on Power, for instance, deep linkage into Windows or .NET applications or functionality that links into specific MacOS management. Not all installation scripts will be appropriate for your use. So, as mentioned in step 1, make sure to identify not just the dependencies that you need to identify and locate, but also those that you explicitly don't need to build.

You may want to think about how the project will be consumed on ppc64le: if it is a container and needs a Dockerfile created or updated, or a binary file created and published on the project's website, or just build scripts that are needed (for example, in ppc64le/build-scripts).

Finally, as in any porting process, it isn't always useful if your port operation provides single point-in-time builds. As developers make new changes to the project, you may want to ensure that those are also tested, and feedback goes back to those communities specifying whether the project still builds and passes its tests on ppc64le. For that, you might want to consider integrating IBM Power builds with the project's existing CI environment. For Power, there are several ways to do this, and the easiest is travis-ci.com, which (through the use of a .travis.yml file in the project) allows the project to build on ppc64le hardware for every change going forward.

This tutorial is intended to be a live document and suggestions and recommendation on additional topics to cover are welcomed and can be raised within the IBM Power Developer eXchange community. Our team has extensive experience in porting and can help with other tips and suggestions to make porting as simple as possible to Linux on Power. Thanks for reading and we look forward to your comments!