IBM Developer Blog

Follow the latest happenings with IBM Developer and stay in the know.

A look into how Linux x86 compares to LinuxONE

When I get excited about new technology, it is really easy to expect that everyone knows what I am talking about. Whether I’m describing the latest book I’m reading or Netflix series I’m watching, I will often just assume everyone knows what I’m talking about. How could you not think Jessica on “Love is Blind” was a bit crazy? Wait, you haven’t seen it? Yeah, okay. Now you see what I mean. Before I go off on something else, let me step back for a minute and introduce to you a really cool box I’ve been working on.

I’d like you to meet LinuxONE III.

LinuxONE platform architecture

LinuxONE III is a beautifully engineered server that runs enterprise Linux distributions like SLES, RHEL, and Ubuntu. It’s not just any other piece of hardware, though. It has a few tricks up its sleeve that make it different from other boxes. LinuxONE leverages years of hardware innovation that originated with IBM’s mainframe platform, IBM Z. The processors that are available on IBM Z get additional microcode that makes them Linux specific just for LinuxONE. The box design is unique in that LinuxONE utilizes the z/Architecture Linux kernel to communicate with the underlying cores, memory, and I/O devices. The z/Architecture instruction set for Linux is more commonly known and seen as s390x. You may have seen some packages available for download marked as s390x, along with x86, arm, and other types of architectures.

What else makes LinuxONE different from Linux on x86?

The Linux operating system is broken into the user space and the kernel space. The kernel space is where hardware differences are handled. If you’re new to Linux, check out the article Anatomy of a Linux kernel for more background.

LinuxONE III processor

Both x86 and s390x are CISC instruction sets. The architecture-dependent kernel code in Linux interacts with the processor’s instruction set to communicate with the hardware throughout the box. It is in this small space (the green box in the above picture) that we enable Linux to interact with the hardware in LinuxONE. You’ll find that the experience in the user space is the same across hardware platforms. The commands, tooling, and packages also remain largely the same.

The beautiful engineering in the LinuxONE hardware is what really sets it apart from x86 and other architectures. Let’s take a look behind the doors. There is a lot we could look at but my favorites are:

  • System Assist Processors (SAP)
  • Processor Unit (PU)
  • Processor Resource/System Manager (PR/SM)
  • Flexible Support Processor (FSP)

System Assist Processors (SAP)

I mentioned above that the processors on LinuxONE are different the mainframe processors because of microcode that enables them to run Linux. There is also a specialty processor called a System Assist Processor (SAP). The SAP is dedicated to running I/O workloads. Traditionally when a processor receives an I/O instruction, it will block and wait for the I/O instruction to complete. Because I/O instructions usually involve reading and writing to disk, they can occupy the processor for more cycles than other instructions. With the intelligence built into the hardware that allows the SAP to handle the I/O instructions, the main processors remain free to run other work and gain operational efficiencies. This means that Linux processors can focus on your programs and handling actual workloads while the SAPs handle all of the data I/O in tandem.

Processor Unit (PU)

Each of the processors mentioned — general processor and SAP — are cores. Each core is a superscalar and out-of-order processor. This allows for 10 concurrent issues to execution units in a single CPU cycle. For the LinuxONE III processor, there are 12 cores that reside on the Processor Unit (PU). There’s even on-chip compression in the LinuxONE III, so data doesn’t have to travel as far to be processed by industry-standard compression algorithms.

Processor unit

Each PU has 3 levels of cache. Level 1 and Level 2 cache exist in the core while Level 3 cache is on the PU. The L3 cache size is 256MB and is shared across the cores. The cache sizes in most typical desktop and server processors are less than 50MB. This is a design decision that helps the platform process data-intensive, multi-threaded applications faster. Even more amazingly, each core has its own crypto co-processor that can be used for cryptographic algorithms that use clear or protected keys, not to mention the built-in data compression acceleration that is also now on the PU. There is a lot of technology packed into this small space!

Processor Resource/System Manager (PR/SM)

Processor Resource/System Manager (PR/SM) is the wizard behind the curtain. It is a hypervisor firmware that allows for virtualization of the LinuxONE hardware. It can separate or share physical resources such as cores, I/O channels, and LAN interfaces across multiple logical partitions (LPARs). LPARs look and act like separate machines even though they reside on the same hardware. Within the LPARs, you can run another layer of virtualization like KVM to run multiple Linux guests. You can even use PR/SM to over-allocate your resources across your environments and get the most out of your box. With the integration at the firmware layer, PR/SM deals directly with the system resources resulting in greater speed and efficiency.

Flexible Support Processor

Don’t worry, this isn’t another specialisation of the cores. This is completely different. The Flexible Support Processor (FSP) is based on the PowerPC microprocessor technology and is used in multiple ways within LinuxONE III.

Each of the Processor Units (up to 4 at a time) reside in a drawer called a Central Processor Complex (CPC). In the LinuxONE III, the CPC has been engineered to include two FSPs. Within the CPC, the FSPs communicate with the support element (SE) in the machine. The SE acts as a single point of control. It will communicate with the FSP to control things like the power supply.

Overhead view of processor drawer

Front view of processor drawer

Two FSPs are also installed in the I/O drawers included in the LinuxONE III. Their job is to make the I/O channels fast and effective at moving I/O data to and from memory.

If you’re looking for one of the most highly rated security modules (HSM) on the market that is certified at FIPS 140-2 Level 4, then you can use two or more of the FSPs to act as the HSM.

Final thoughts

It’s hard to succinctly describe how all of this fits together. This video does just that: In just under two and a half minutes, it describes how all of this works together to function inside a single box.

The many layers of engineering in LinuxONE really set it apart from other servers built on top of the x86 architecture. So many applications gain benefits just by running on the hardware. For applications that work with a lot of data, who wouldn’t love the boost of a SAP?

I’ve barely skimmed the surface of the layers of technology that exist in our cores and on the PUs. The cores rate at 4.5 GHz — and while that’s an impressive rating, you really need to see applications or multiple environments run all located within LinuxONE thanks to the PR/SM technology to really grasp just how much compute power exists between all of the cores all over the box. Check out the IBM LinuxONE Community Cloud to see for yourself how Linux looks and feels just as you would expect it to anywhere else, while gaining all of the performance advantages that come from the normal Linux cores, SAPs, and FSPs all working together to drive every efficiency possible in hardware.

Additional resources