IBM’s approach to open technology
Find out how we're investing in the communities and projects that matter most to the enterprise
For several years, IBM has been recognized by many within the open source community as a leader in open source. However, that leadership is less well known and appreciated outside of the open source communities in which IBM engages.
From our work with Linux, Apache, and Eclipse to our current work across all layers of the cloud stack, application development, blockchain, artificial intelligence, quantum computing, and machine learning, IBM has demonstrated a sustained commitment to driving innovation in open source, delivering a broad portfolio of offerings based on open source, and helping to build sustainable, thriving communities and ecosystems around those open source projects about which we care the most.
In fact, IBM is one of the most prolific companies making contributions of open source code to GitHub organizations and repositories.
At IBM, we believe that our leadership in open source is a differentiating value for our clients—so much so that we even came up with a catch phrase that we had trademarked to express that value: IBM is Open by Design.
In this article, learn how IBM has played, and continues to play, a leading role in the evolution of open source—an approach to software development that has nearly completely displaced proprietary software—and how we leverage that leadership to deliver world-class offerings and solutions that benefit our customers.
A long history
IBM has a long history of working in open source. In fact, in many ways, IBM is partly responsible for the open source movement’s success. IBM’s strong support for Linux—with patent pledges and investment of significant technical resources in the face of some of the legal uncertainty surrounding its use in the late 1990s—brought about a change in posture toward open source for many enterprises. Under the right circumstances, open source such as Linux could be a compelling and trusted adjunct to proprietary software for a growing number of enterprises.
The successful deep collaboration with other industry leaders on Linux led IBM to again work with others to establish both the Apache Software Foundation and the Eclipse Foundation. We went even further with Eclipse by seeding the new foundation with a significant initial code contribution and dedicated technical resources, and in both cases providing legal leadership in authoring the licenses for those communities. On a broad scale, IBM’s participation in these and well over a thousand other projects and communities set the tone for open source adoption among enterprises.
While it might come as a surprise to some, one of our key platforms, WebSphere Application Server, is more than 70% open source, comprising more than 700 open source components. Of course, WebSphere is not alone. Many of IBM’s technology offerings and solutions leverage open source—especially those comprising our most strategic initiatives including cloud, big data and analytics, blockchain, IoT, machine learning, and AI.
In addition to consuming open source and contributing to a vast array of projects, IBM has also made many major contributions to open source, including:
- our Java runtime, J9, contributed to the Eclipse Foundation as the Eclipse OpenJ9 incubator
- our OpenLiberty runtime for Java EE and MicroProfile applications, contributed to openliberty.io
- our open blockchain project to Hyperledger as Hyperledger Fabric
- our serverless platform to Apache OpenWhisk
- our quantum computing API Qiskit
- our AI Fairness 360 Toolkit (AIF360) and AI Robustness Toolbox (ART)
- analytics projects such as the code that became Apache Toree and Apache SystemML
- and over 100 others in just the past 3 years
The case for open governance
One thing IBM has learned through all of this is that those communities that strive for inclusiveness and open governance tend to attract the largest ecosystems and most expansive markets. However, not all open source is created equal, and not all communities thrive.
There is a broad range of open source, and much of it is not truly open. Many open source projects are run by a single individual (or vendor) and are quite closed in their governance, severely limiting contributions from others. Other projects are more welcoming to outside contributions but are still closed when it comes to setting technical strategy and direction.
The key point is that once a project reaches a certain level of success, it usually reaches a tipping point. Without open governance, users realize there is a greater risk of vendor lock-in, or even project abandonment. Users and contributors alike want a voice in the decisions, and if they feel that their voice is not being heard, projects have been known to fail or fork. This typically has a detrimental effect on the ecosystem, which increases risk to users should the community around it collapse.
“We have worked hard over the years to establish a solid and respected reputation in open source circles, and especially in those communities where we invest strategically.
The reality is that open technology projects managed under open governance—the sort of open governance found in organizations such as Apache, Eclipse, Mozilla, and Linux—are demonstrably more successful (by an order of magnitude), have a longer life, and are less risky than those projects that are controlled by a single vendor, or are more restrictive in their governance.
While IBM often contributes to and consumes from open source projects that are not under open governance, when our clients or our offering teams feel that a project is important enough, we often work with the individual or vendor that controls the project to help them see the value of open governance, and to see the potential for even greater success. If we can effectively bring the project to open governance, we increase our investment considerably to help ensure that project’s success, and work to grow the community and ecosystem.
For the good of the ecosystem
IBM knows that a rising tide floats all boats. It isn’t enough that IBM succeeds—we need to ensure that many can succeed to ensure a vibrant ecosystem. This reduces the risk that comes with embracing open source for ourselves and, more importantly, our users. Other vendors see this as well, as participants in the renaissance of open source understand that competing on plumbing or commodity is not where the opportunity lies.
We believe that clients who leverage open source should establish a process for managing its use in their enterprise such that they consider not only technical merit and license, but also the evaluation of the community and ecosystem.
With the majority of enterprises charting a course toward a preference for open source over proprietary offerings, the first step is quite often to attempt to integrate your own open source stack. Eventually, once the scope of the challenge is better appreciated through experience, the pattern often shifts to partnering with others who have the requisite deep skills and experience, with a thorough understanding of the relevant communities.
We also believe that clients who embark on a journey to integrate open source should have a partner or partners who can help them engage and influence the community. These partners should be committed to the same enterprise interests and leverage the accrued goodwill and technical eminence in the community to advance a technical agenda commensurate with their interests.
IBM’s commitment and contribution to open source is unrivaled in the industry. We have worked hard over the years to establish a solid and respected reputation in open source circles, and especially in those communities where we invest strategically. IBMers serve on a number of open source foundation boards, including Linux, Eclipse, Apache, CNCF, Node.js, Hyperledger, and many others, and we have tens of thousands of IBMers using and contributing to open source. We value and work toward open governance because we feel that this is the best way to ensure the long-term success and viability of open source projects. IBM developers are working every day in open source projects that matter, making thousands of contributions to hundreds of open source projects each month.
Not all open source is created equal
Many open source projects on GitHub and elsewhere could be described as “furniture left by the curbside for anyone to take.” The code is published with an open source license (or not!) but isn’t being actively maintained. Even in those cases where the code is actively maintained, it is likely that a single individual or company handles it. This is all well and good until that individual (or company) decides to pursue something more interesting or meets an unfortunate circumstance.
There are countless examples of the consequences of investing in open source projects with a single owner. A while back, Facebook declared that it was going to discontinue Parse (a popular mobile development platform), leaving thousands of developers stranded. There are many more cases where a developer released some cool capability as open source, and then, for whatever reason, abandoned or neglected it.
Now, some open source purists might say, “But you still have the software,” which is true. But here’s the rub: Are you really going to maintain a mobile app development platform or NoSQL database by yourself?
IBM evaluates open source projects by looking closely at five aspects of the project:
- Responsible licensing—Obviously, we look to understand the open source license that is associated with the technology.
- Accessible commit process—We seek to ensure that there is a clearly defined process for making contributions that welcomes outside contributors.
- Diverse ecosystem—We confirm that there are multiple vendors and ISVs that are delivering offerings based on the technology.
- Participative community—We require that there be a process for contributors to grow their technical eminence in the community.
- Open governance—We evaluate the governance model to determine whether it is truly open.
Of course, we also look at the technology and assess whether there’s an architectural fit, but the technology can usually be fixed and improved over time. The key is whether we believe that there are enough positives to warrant the investment to help bring the project to true open governance that will benefit all.
IBM’s journey begins with Apache, Eclipse, and Linux
IBM’s journey in open governance for open source started with our leadership in the formation of the Apache Software Foundation in 1999, which was created to provide open governance for the development of the Apache HTTP server. IBM was one of the founding sponsors of the foundation, helped shape the license and governance, and contributed to numerous projects. In addition, IBMers have been serving in leadership roles within the organization and on the ASF board since its launch.
Eighteen years later, there are nearly 200 projects on ASF, many of which have nothing to do with the original HTTP server project. This includes web technologies, XML, web services, document processing, mobile, cloud, big data and analytics, serverless, and messaging. Clearly, we have helped to create a safe place to collaborate and innovate in the open.
In 2001, IBM worked with others to create the Eclipse Foundation with an initial grant of the Eclipse Java IDE framework. We had similar goals for the Eclipse Foundation as we did with Apache: to create a safe place to collaborate and innovate under open governance. Seventeen years later, there are over 360 projects at Eclipse with equivalent diversity of domain. Again, the open governance provides a venue that attracts open source developers and provides a crucible in which to collaborate and innovate in the open.
IBM was an early adopter and contributor to Java reaching back to the earliest days. We were instrumental in helping shape the Java language and runtime, and J2EE specifications along with Sun Microsystems. Over the years, we have helped drive Java to open source that resulted in OpenJDK becoming the premier open source Java.
More recently, IBM open sourced its J9 runtime, a high-performance, low-memory footprint Java Virtual Machine (JVM) optimized for the cloud, as well as the Liberty runtime for Java EE and MicroProfile applications, which provides the open basis for WebSphere. We continue to help lead and contribute to establishing the Eclipse Foundation Specification Process that is to replace the former Java Community Process (JCP) for Jakarta EE.
In 2007, IBM collaborated with other key industry leaders to establish the Linux Foundation as a founding Platinum sponsor. Of course, our investment goes well beyond this sponsorship. We have been, and continue to be, a leader in the Linux community by investing hundreds of engineering resources over the years in the Linux kernel and in many of the now more than 80 collaborative projects under the Linux Foundation—some in which IBM played an instrumental role in launching.
In July 2015, following close on the heels of the OCI launch, IBM, Google, Docker, Weaveworks, Red Hat, and many others launched the Cloud Native Compute Foundation with the intention of providing an open governance model for Google’s Kubernetes project. This is a key component of IBM’s technical strategy for cloud and other technologies relating to cloud native applications.
Since that time, the organization has thrived under the leadership of the CNCF chair, IBM’s Todd Moore, growing to a total of 18 projects including Kubernetes, etcd, rkt, fluentd, containerd (see below), and gRPC. IBM is growing its investment and contributions to CNCF technologies because of their relevance to our cloud strategy, most notably Kubernetes. IBM continues to increase its investment in Kubernetes, etcd, and containerd and as our investment grows, we will likely expand the set of projects.
Our partnership with Google on Docker, Kubernetes, and CNCF has also born additional fruit. IBM and Google joined forces with Lyft to collaborate on a merger of IBM’s Amalgam8, Lyft’s Envoy, and Google’s Service Control. The result was the Istio project, a first-class abstraction for routing and policy management for cloud native microservices. Our goal is to eventually move Istio to CNCF to ensure open governance for this important and increasingly popular project.
IBM has been one of the top contributors to Docker over the last few years. Three of our developers earned the respect of their peers at Docker, Inc., and were named maintainers. As the use and popularity of Docker grew, and as the number of non-Docker Inc. employee contributors grew, the pressure to bring Docker under open governance also grew.
This community pressure initially led Docker to establish the Open Container Initiative, with IBM as one of the founding sponsors, in June of 2015. Docker contributed libcontainer and the Docker image and transfer format specification to seed the newly created initiative. IBM has been one of the top contributors to the OCI since its formation. Recently, one of our top engineers was recognized for his contribution and leadership by being elected to the OCI Technical Oversight Board.
Of course, that is only the first aspect of Docker to be brought under open governance. In December 2016, Docker contributed containerd to the Cloud Native Compute Foundation (CNCF). containerd is a core container runtime component that can manage the complete container lifecycle for containers on its host system. Two IBMers have earned maintainer status on the containerd project; again, a testament to IBM’s leadership in open source.
Recently, Google announced the Knative project, developed in close partnership with IBM and a number of other key vendors in the serverless and platform as a service domain. Knative provides the building blocks to enable serverless capabilities to Kubernetes. We believe that this will be a key technology and are working closely in the community to evolve platforms such as Cloud Foundry and OpenWhisk to be based on Knative.
AI and machine learning
IBM recently open sourced some key technologies for AI, including:
- The AI Fairness 360 toolkit (AIF360), an open source software toolkit that can help detect and remove bias in machine learning models
- The Adversarial Robustness Toolbox for rapid crafting and analysis of attack and defense methods for machine learning models
- Fabric for Deep Learning (FfDL, pronounced fiddle), a deep learning platform offering TensorFlow, Caffe, PyTorch etc. as a Service on Kubernetes
This space is as hot as they come, so look for more open source innovation from IBM in the months to come.
In 2015, IBM recognized the incredible potential of blockchain technology, the same technology that underlies Bitcoin. Our research in this space led us to conclude that none of the existing blockchain technology platforms was really suitable for the enterprise. So, IBM set forth to build a new blockchain platform with enterprise requirements in mind—one that could be used in highly regulated environments.
As with the Apache HTTP server and the Linux kernel, we recognized that a technology this important should not be controlled by any single vendor. It would never be successful if it was just available from IBM. So we worked with the Linux Foundation to help establish Hyperledger, the fastest-growing project at the LF, ever. IBM contributed 44 thousand lines of code and established the first of the Hyperledger projects, Hyperledger Fabric, under open governance. Since then, there have been nine additional projects incubated at Hyperledger.
Hyperledger Fabric was the first to be incubated, the first to graduate to “Active” status, and the first to reach a 1.0.0 release – in June 2017. There have been nearly 300 engineers representing 40 companies that have contributed to the four releases of Fabric. This is a testament to the value of developing open source under open governance.
When Amazon introduced AWS Lambda in 2014, it signaled a potentially game-changing direction for cloud—function as a service (FaaS), or serverless computing. Many companies started to explore this space including Google, Microsoft, and many others. IBM was no exception. In early 2015, IBM Research began work on development of an implementation of a robust serverless capability for the IBM Cloud.
As with our blockchain efforts that led to the creation of Hyperledger Fabric, we recognized that in order for our serverless efforts to be seen as a viable alternative to the proprietary AWS Lambda offering, our implementation needed to be open sourced under open governance so that we could grow a vibrant community and ecosystem around that open source project. In February 2016, we open sourced our serverless platform implementation and named it OpenWhisk. As interest in OpenWhisk grew, we then worked with partners like Adobe and Red Hat to establish Apache OpenWhisk as an incubating project in November 2016.
IBM worked with both factions and convinced them that the way to resolve the situation was to bring node.js development under open governance. IBM helped the other key stakeholders establish the Node.js Foundation under the Linux Foundation and worked to heal the schism. The io.js fork was merged back into node.js and the project is now enjoying wild levels of success and an increased maturity thanks to IBM leadership.
And much, much more
These are just a few of the examples of IBM’s leadership and investment in open source. In just this past year, IBM also helped drive the formation of the OpenStack Foundation, Cloud Foundry Foundation, Node.js Foundation, the Open API Initiatve, and ODPI. We were also behind the creation of a number of Apache projects aside from Apache OpenWhisk, Apache Toree and Apache SystemML. This past year, we initiated more than 50 new open source projects and published countless tutorials and sample applications as open source.
We are also encouraging more and more IBM-developed projects to be published as open source from the IBM development and research labs. In the past three years of the program, we have published hundreds of new open source projects through this program, and many of those projects have garnered sufficient community interest to graduate to open governance projects at Apache, Eclipse, and the Linux Foundation.
IBM has invested significant resources in adapting key open technologies to our Open Power and Z platforms, including MariaDB, Go (which is the language used by many of the emerging open source projects targeted at the system level), Docker, runC, Linux, OpenStack, Cloud Foundry, Kubernetes, Swarm, and many more. In fact, much of IBM’s open source influence emanates from the Systems division.
Focus on the enterprise
We invest heavily in areas such as security, scalability, robustness, live upgrade, globalization, documentation, continuous integration, and delivery in the projects that we consider strategic. Of course, we also invest in the functional aspects of projects that bring IBM innovation into the open. We contribute extensively in other important aspects, including marketing, evangelism, and various board-level committee activities. Finally, we often lead the efforts to define interoperability and portability. We do this because interoperability and portability are critical to the success of any open technology endeavor. After all, that is what the consumer is expecting.
In all of these cases (and more), IBM has invested close to $1 billion and dedicated hundreds of open source development, marketing, and evangelism resources over the past five years. We have initiated many of these projects and worked tirelessly to help the organizations and the projects they hosted to define and achieve success. We do this because the value that IBM derives from these projects and organizations extends beyond the open source software itself. Much of the benefit comes from the vibrant communities and thriving ecosystems that develop around these centers of gravity for open technology. Our offerings succeed in proportion to the success of the projects in which we invest.
We focus on driving interoperability, portability, and many of the “-ilities” that matter most to the enterprise. We also focus on contributing IBM innovation upstream, and prefer that to keeping it in closed source when it pertains to these strategic projects such as OpenStack. Yes, there is IBM value-add in our offerings, but we don’t invest in interoperability and portability so that we can then throw those attributes out the window when we deliver our offerings based on those open technologies. Our strategy is based on leveraging a solid foundation of open technology and ensuring that the interfaces (APIs and SPIs) defined by these technologies are fully exposed and not hidden or inaccessible behind some proprietary veneer of value-add.
We strive hard not to fork the community code by creating an “IBM Hyperledger Fabric” or “IBM Kubernetes.” The Hyperledger Fabric in IBM Blockchain Platform is the same Hyperledger fabric that is released by the Hyperledger organization. The Kubernetes that we integrate into IBM Cloud is the same code that is released by the Cloud Native Computing Foundation. The Docker included in IBM Container Service is the same Docker that that community releases. The Cloud Foundry is the same code that is released by the Cloud Foundry Foundation. The IBM value-add is that we integrate all these capabilities developed in open source to realize the IBM Cloud.
We invest in the community code for these strategic technologies and ensure that fixes and new features are up-streamed rather than adding extra complexity and effort on IBM’s part to maintain an independent, differentiating version. Where we desire to add extensibility that can leverage IBM (or others’) differentiating capabilities, we work within the community to create the necessary API or SPI. We also invest in making sure that those extension points are not abused to create a potential for lock-in.
As you can see, IBM takes open source seriously. When we engage in a project, we bring a focus on those aspects that matter most to the enterprise: interoperability, portability, security, scalability, and accessibility. We do this by investing in the community and helping to shape programs that can deliver those characteristics that matter to our clients. We value open governance because it ensures the long-term success and viability of those projects that form the foundation of our enterprise offerings and solutions.