About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Introduction
I had the good fortune to be selected to present at the OpenPOWER Summit US 2018 on the topic of tools for porting and tuning for Linux on Power. The time slots for the presentations were fairly short (maximum of 30 minutes), but there was a lot I wanted to cover. So, I did my best to highlight the tools I felt had the most value, and perhaps lacked general awareness. A video of the presentation can be found at https://www.youtube.com/watch?v=PJwnfDSHOLI. (There are lots of great presentations from the OpenPOWER Summit US 2018 appearing on the OpenPOWER Foundation's YouTube channel at: https://www.youtube.com/channel/UCNVcHm09eXVbvUzZkQs0_Sg.)
Instead of listening to me drone on for 30+ minutes, what follows is the basic content of the presentation in textual form, not verbatim and slightly more detailed - a better reference.
As a general note, I classify the use-cases for the tools in the following three ways:
: The user has the source code available and is willing and able to change the source code for portability and performance advantage; the code can be recompiled and relinked; there is a representative performance scenario which can be run for analysis.
: The user has the source code, but may not be willing or able to change it; the code can be recompiled or relinked; there is a representative performance scenario which can be run for analysis.
: Source code is not required; neither recompiling nor relinking is required; there is a representative performance scenario which can be run for analysis.
Advance Toolchain
The IBM® Advance Toolchain is a software suite containing the latest releases of compilers, various libraries, and various tools related to application porting, tuning, and debugging. Recent releases of these components include support for the latest features and the latest optimizations for OpenPOWER and IBM Power® processor-based platforms. The purpose of the Advance Toolchain is to make these more modern software components available on distributions which only provide significantly older releases. The distributions, justifiably, are reluctant to change major components of the operating system like compilers and system libraries as the risk to stability is not worth the opportunity for better performance. Some distributions have made strides in providing developers with much more recent components as a developer toolset that provides a later compiler and its prerequisites. The Advance Toolchain goes farther in providing not only the latest compilers but also the latest releases of many system libraries. In addition, those system libraries are built with the latest compilers. In this way, applications built with the Advance Toolchain benefit not only from new optimizations in the latest software, but also by having that software compiled with the new compiler.
Further, the Advance Toolchain is where new compatibility features may appear first. For example, there is an ongoing effort under the auspices of the GCC project to provide compatible implementations of the Intel vector intrinsics. Those would not have appeared in major Linux distributions for a year or more, but have already appeared in the Advance Toolchain.
The Advance Toolchain is supportable through IBM Support Line. Updates with bug fixes and security-related fixes are released often. It is available for free download, and is entirely open source at https://github.com/advancetoolchain/advance-toolchain.
One caveat is that when an application is built with the Advance Toolchain, it then has a dependency on the Advance Toolchain runtime. So, if that application is to be deployed elsewhere, the Advance Toolchain runtime package must be installed there as well. Because the runtime is free, this is not of significant concern, but something of which to be aware.
Best practices:
- Better: At a minimum, use the latest of any distribution-provided developer toolset to get a recent release of the compiler.
- Best: Use the Advance Toolchain to get the latest release of the compiler, libraries, and tools; plus, those libraries built with the latest release of the compiler!
XL compilers
IBM XL compilers are IBM's flagship proprietary compiler suite, used for reporting SPEC benchmark results on IBM AIX®, IBM z/OS®, and Linux on Power. The IBM XL compiler development team works closely with the IBM Research team to incorporate the very best optimization techniques for performance advantage. Recently, the IBM XL C/C++ compiler switched to use a source code parser (front end) based on Clang, allowing the IBM XL C/C++ compiler to significantly improve source code compatibility with GCC and LLVM. Also, most common command-line options for GCC are also supported by the XL C/C++ compilers.
There are two variants of IBM XL C/C++ for Linux:
- Fully licensed
- Community Edition: A free, full function offering without support
The IBM XL compilers can very well make use of the Advance Toolchain libraries and tools. In a sense, you can get the best of both worlds by using IBM's flagship proprietary compiler with the latest fully optimized libraries. (Note that this will impose a dependency on the Advance Toolchain runtime.)
Listing 1. XL compilation using IBM Advance Toolchain libraries
Beyond standard compilation, the IBM XL compilers also offer several advanced features which can be used to performance advantage, one of which falls in the gray box category: automatic parallelization. If this is enabled using a command line option, code can be generated to automatically use the multithreading capabilities of the Power system for performance advantage.
The IBM XL compilers also include the following advanced features that can be used to further exploit the capabilities of the Power systems with source code changes:
- Transparent exploitation of GPU resources by taking advantage of the OpenMP 4.5 support
- High performance optimized math libraries (ESSL, BLAS)
- Optimization reports that can indicate areas of the code in which optimization opportunities could be increased with changes
Best practice:
- Try using the IBM XL C/C++ for Linux Community Edition. It is free, and its compatibility with GCC should make it a drop-in replacement simply by changing the PATH. If the performance advantage is significant, consider adopting the fully licensed and supported version for integration into a production build environment.
ma (Migration Advisor)
The command-line Migration Advisor scans the source code looking for likely portability issues. As of this writing, the list of checkers is slightly fewer for the command-line version:
#ifdef x86- Non-portable system calls, APIs, built-ins, assembly
long double, Float128- Non-portable hardware transactional memory use
chardefault signedness
Usage is very simple:
Listing 5. ma results
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/migration-advisor.
cpi (CPI Breakdown)
The command-line CPI Breakdown tool will profile a representative performance scenario and report a hierarchical set of information about where the program is spending its time. Using the command-line CPI Breakdown tool is a two-step process:
record: Profile the performance scenario and record relevant hardware events.display: Display the collated results in the form of a hierarchical layout of events, metrics, and their respective relative contribution to overall CPI measurement.
Use of the command-line CPI Breakdown tool is simple. The first step is to record the hardware event counts.
Listing 7. cpi record
Note that the scenario ("./load" in this example) will be run several times in succession in order to collect all relevant hardware performance events, as only a handful are collected during each run.
The second step is to display the CPI breakdown.
Listing 8. cpi display results
There is a lot of information displayed. For convenience, there are command flags which can limit the output.
Listing 9. cpi display with filters
Because the ultimate goal is to narrow down where in the code adverse events are happening, there is a further convenience function that can drill down on the most frequently occurring events. New profiling runs are launched in which those specific events are recorded and the profiling information, including source, line, and potentially instruction are included in the command output.
Listing 10. cpi drilldown results
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/cpi-breakdown
Note: Recent versions of the perf command can generate detailed CPI breakdown information:
$ perf stat --metrics cpi_breakdown ./command
The required hardware events are multiplexed during the run, so only a single run of the command is required, unlike with the cpi command. The output from perf is not as readable as with the cpi command:
curt
There is a tool called curt on AIX that displays statistics related to system utilization. A new tool for Linux, also called curt is inspired by the AIX tool (but is otherwise unrelated).
Statistics reported by the Linux curt tool include:
- Per-task-per-CPU user, system, interrupt, hypervisor, and idle time
- Per-task, per-process, and system-wide user, system, interrupt, hypervisor, and idle time
- Per-task, per-process, and system-wide utlization percentage, and migration counts
- Per-task-per-syscall invocation counts, elapsed time, average time, minimum time, and maximum time
- Per-task-per-HCALL invocation counts, elapsed time, average time, minimum time, and maximum time
- Per-task-pre-interrupt counts, elapsed time, average time, minimum time, and maximum time
Use of the tool is a two-step process:
Use
perf recordto generate a recording of relevant events:Use the curt script to process the data recorded by the perf command:
$ ./curt.py perf.data
With the most recent version of curt, both recording and reporting can be done in a single, simple step: $ ./curt.py --record all command
Sample output (heavily edited for brevity and clarity):
Listing 11. curt results
The project is written in Python, is open source, and can be found at: https://github.com/open-power-sdk/curt.
Power Functional Simulator
, , and
The Power Functional Simulator is a full-system simulator for Power. This very powerful tool provides a complete Power environment when a Power processor-based system is otherwise unavailable or impractical. Given an image of an installed file system, it will boot through firmware and operating system to a login prompt for a Power processor-based development environment.
The Power Functional Simulator can be found at: https://developer.ibm.com/tutorials/download-and-install-sdk-for-lop/.
Convenient wrapper scripts, which greatly simplify getting an environment established, can be found at: https://github.com/open-power-sdk/power-simulator.
Listing 12. Power Functional Simulator boot
The environment provided by the Power Functional Simulator is single core, single thread. There are simulators available for Power8, Power9, and Power10 processors. A simulator is a great way to begin getting experience with Linux on Power, beginning a porting effort, and even for establishing a robust cross-compilation and runtime environment.
Performance Simulator
The Performance Simulator is a cycle-accurate Power instruction stream reporting tool. It transforms a Power instruction trace into a report where the various stages of each cycle of every instruction's lifetime is reported. The resulting reports can be viewed with one of the viewers which is included with the Performance Simulator package.
Instruction traces can be captured using the itrace function of the Valgrind tool suite. Valgrind itrace is not included with the Valgrind that comes with Linux distributions. However, itrace is available with the Valgrind that comes with the IBM Advance Toolchain (see above).
Using the Performance Simulator is a three-step process:
Record instruction trace (
.vgifile):$ valgrind --tool=itrace --binary-outfile=tracefile.vgi --num-K-insns-to-collect=100 --demangle=no commandCreate a
.qtformat file from the.vgifile:$ vgi2qt -f tracefile.vgi -o tracefile.qtRun the Performance Simulator timer (
.pipefile):$ /opt/ibm/sim_ppc/sim_p10/bin/run_timer tracefile.qt 100000 10000 1 tracefile -scroll_pipe 1 -scroll_begin 1 -scroll_end 100000
You can view the resulting instruction timing report with the jviewer.
jviewer
The jviewer tool displays the instruction cycle report.
Figure 10. jviewer display
Each phase of each instruction’s lifetime is shown visually in the main pane, lower left. The instruction disassembly is in the rightmost pane. Hovering over any of the character mnemonics in the main pane will display more details about it. In the example, the cursor is over a ‘u’ near the center of the pane. The corresponding explanation is “cannot issue unit not free”. This instruction is currently waiting for the branch unit to be made available from the processing of a previously executed instruction.
pipestat
The pipestat tool operates on the output of the Performance Simulator cycle-accurate timer, and performs detailed analysis, reporting the following information:
- Most executed loops, including misaligned short loops
- Most executed blocks with long latency instructions, redundant loads
- Most executed incorrectly hinted branches
- Most executed mispredicted and frequently mispredicted branches
- Most executed code paths that have a store followed soon after by a load of the same address
- Most load-hit-store related events on a particular instruction address
- Most executed instructions where the result is used a small number of instructions later but takes a large number of cycles before the dependent instruction starts
The pipestat tool produces a lot of useful output. Extracted below are a few snippets.
Listing 13. pipestat results
The information displayed may be a bit cryptic, but there is comprehensive documentation that comes with pipestat. The pipestat tool can be found at https://developer.ibm.com/tutorials/download-and-install-sdk-for-lop/.
pveclib
No Instruction Set Architecture (ISA) can have every possible useful vector instruction and an associated compiler built-in. The pveclib project provides some well-crafted implementations of useful vector functions which are not part of the Power ISA. A sampling of the functions includes:
udiv_qrnndfxu_bcdadd, fxu_bcd_subvec_BCD2DFP, vec_DFP2BCDvec_bcdadd, vec_bcdsub, vec_bcdmul, vec_bcddivvec_shift_leftdovec_isalpha, vec_isalnum, vec_isdigitvec_toupper, vec_tolowervec_absdubvec_revq, vec_revd, vec_revw, vec_revhvec_clzq, vec_popcntqvec_sldq, vec_srqi, vec_srq, vec_slqi, vec_slqvec_pastedvec_mulouw, vec_muleuw, vec_mulosw, vec_muleswvec_adduqm, …- …
The pveclib project is open source and can be found at https://github.com/open-power-sdk/pveclib.
SPHDE
The Shared Persistent Heap Data Environment (SPHDE) provides some advanced, high-performance, cross-platform implementations of functionality useful in a multiprocess or multithreaded application environment:
- Shared Address Space (SAS): A shared-memory implementation in which the virtual addresses of data are common, so interprocess communications can freely pass pointers to data
- Shared Persistent Heap: A multiprocess environment can allocate memory dynamically in the Shared Address Space
- Lockless Logger: A multithreaded or multiprocess environment can safely and locklessly use an in-memory logging capability for recording events with minimal application performance impact
- Lockless producer-consumer queue: Message passing between multiple processes can be fast and efficient, avoiding the use of locks and requiring zero copying
- Fast timestamps: Instead of using expensive system calls or somewhat less expensive virtual dynamic shared objects (VDSOs), a single instruction can be used to access a system-wide synchronized timebase register, which is significantly faster.
SPHDE is open source and can be found at https://github.com/sphde/sphde.
LPCPU
The Linux Performance Customer Profiling Utility (LPCPU) is a free download which can be used to collect system information and system performance-related data. Use of LPCPU is a three-step process:
System-wide data is collected for a specified period of time. The information collected is gathered into a compressed .tar file.
The .tar file can then be offloaded to another system for post-processing and analysis.
Point a browser at the resulting summary.html file.
Data collected includes output from the following common tools:
iostatmpstatvmstatperfor OProfilememinfotopsar/proc/interruptstcpdump- Kernel trace
- Hardware performance counters
netstat...
The post-processing steps create numerous charts and interactive graphs.
Figure 11. LPCPU CPU utilization graph 1
Figure 12. LPCPU CPU utilization graph 2
Figure 13. LPCPU Interrupts graph
Figure 14. LPCPU Dirty Memory graph
The LPCPU tool is open source and can be freely downloaded at https://github.com/open-power-sdk/lpcpu.
Support
Formal support is available for the Advance Toolchain and the XL compilers.
The IBM Power Developer eXchange Community is a hub for all sorts of information about hardware and software components in the Linux on Power ecosystem. Refer to: https://community.ibm.com/community/user/powerdeveloper/home.
Community support options are also available at the following well-known websites:
- https://stackoverflow.com
- https://serverfault.com
- https://stackexchange.com
- And other similar websites
Best practices
Given all the nifty tools mentioned, where should one start when embarking on an effort to port code to Linux on Power?
For interpreted code, including Java, Python, Perl, and shell scripts, just move the code to Linux on Power, and it should run without modification.
For compiled code, with little to no source code analysis:
- Start with ma (Migration Advisor). It will flag portability issues in the code being ported.
- Build with the Advance Toolchain. Measure performance.
Build with XL compilers, using the Advance Toolchain libraries. Measure performance.
Additionally, for compiled code:
Use common performance analysis tools such as perf to look for hot spots in the code for careful analysis, and see if any can be explained by architectural differences between prior platforms and Power. See "Porting to Linux on Power: 5 tips that could turn a good port into a great port" at https://developer.ibm.com/articles/l-port-x86-lop/. Apply mitigation if possible.
Consider using the cpi-breakdown tool for deeper architecture-specific analysis. Drill-down on higher-frequency hazards. Apply mitigation if possible.
For very deep analysis, consider using the pipestat command for fine-grained instruction-level analysis.
As always, consider alternative approaches to common, important functionality that might be provided by SPHDE or pveclib.
For system-wide performance analysis, use LPCPU and/or curt.
Learn more
- Porting to Linux on Power: 5 tips that could turn a good port into a great port
- IBM Software Development Kit for Linux on Power (SDK)
- GitHub - open-power-sdk/power-simulator
- Port your x86 Linux applications to Linux on IBM Power
- IBM Redbook - Performance Optimization and Tuning Techniques for IBM Power Systems Processors Including IBM Power8
- IBM Redbook - Implementing an IBM High-Performance Computing Solution on IBM Power8
- OpenPOWER Foundation
Conclusion
Hopefully, the collection of tools explained in this article provide what's needed to get the best results as quickly as possible. If there is a gap in a tool's functionality or documentation, or a new tool would help, or reality isn't meeting expectations, ask in one of the support channels! We're here to help!
Take the next step
Join the Power Developer eXchange Community (PDeX). PDeX is a place for anyone interested in developing open source apps on IBM Power. Whether you're new to Power or a seasoned expert, we invite you to join and begin exchanging ideas, sharing experiences, and collaborating with other members today!