The history of IBM’s contributions to Cloud Foundry, part 1
What we can learn from the evolution of the Cloud Foundry platform
In the summer of 2018, before the start of the Cloud Foundry summit in Boston, I co-wrote an article pointing out some common misconceptions and facts about Cloud Foundry. This article was written with the context of a growing container movement in the tech industry.
While Cloud Foundry used containers, orchestrated them, and made them easy for mere mortals, it hides so much of that complexity. Because of that fact, it seems that users failed to realize that the emerging Docker and Kubernetes were simply exploring a cross section of the vast set of features that Cloud Foundry addressed.
“Cloud Foundry under the hood” session with Nima Kaviani
Last year’s article was popular, but the enthusiasm around exposing the primitive features that Cloud Foundry embedded within its core was not curbed. Indeed, Docker – and in particular Kubernetes – reached peak excitement, with increased adoptions that are eclipsing all other platforms in less than one year.
Perhaps now it’s time to contemplate another introspective post on the subject. But instead of framing it as enumerating Cloud Foundry’s features and values, maybe it’s time to retrace the history of this great platform and draw some lessons learned that could be useful to this community and beyond.
As an integral member of the Cloud Foundry community with years of IT industry experience, and having participated in various (new and mature) open-source software (OSS) communities, it is even a duty that I should be compelled to fulfill.
Developers at Cloud Foundry Summit 2016 in Santa Clara
Naturally, as a warning to the reader, it’s important to disclose that my viewpoint and opinions are colored with the lenses of my experiences, my immediate colleagues, my collaborators, and my employer: IBM Cloud.
Instead of hiding these biases, I will expose them where needed. So read on if you want to get one perspective on the history of Cloud Foundry from the point of view of an IBMer.
What I hope to achieve with this three-part series is twofold. First, I want to create a record for posterity, including both IBM’s contributions and the evolution of the Cloud Foundry platform.
And second (and perhaps the most useful part of this series), I want to extract lessons learned that could be important to both Cloud Foundry community members and other communities in the same space, as we move forward.
The beginnings: circa 2012
Cloud Foundry was created at VMware by a group of ex-Google employees who looked to duplicate the in-house platform that Google uses to manage its many services and applications: Borg. However, instead of creating another proprietary platform that would benefit only a small few, the leaders of the early Cloud Foundry (including VMware’s management) decided to make the platform open source.
IBM Cloud Foundry Day in London in 2018
I remember the meetup where Derek Collison announced the open sourcing of Cloud Foundry in late 2010, and wide availability of the source code, at the offices of VMware in Palo Alto. I recall this well because I was working on a research project to create a cloud platform as a service (PaaS) called Altocumulus. It was not suprising that VMware was also working on a similar project.
After all, cloud was the new hit thing in Silicon Valley. While many were focusing on creating the necessary software to compete with the booming AWS EC2 platform, many were also looking at what was next, and PaaS seemed to be the right direction. However, what was interesting about releasing this new PaaS was the open source part.
The IBM Cloud Foundry team gathered for dinner in Basel, Switzerland, during Cloud Foundry Summit 2017
The IBM software team became interested immediately, and various folks at IBM started playing with the first Cloud Foundry release, which initially was completely packaged into a USB key. After multiple trials internally to set up and run Cloud Foundry, the OSS project captured the attention of the nascent OSS technology team led by Dr. Angel Diaz.
The early efforts of Doug Davis and Chris Ferris paid off as Angel and his group decided to form a team around Cloud Foundry (under the leadership of Alex Tarpinian) and engage the new Cloud Foundry steward company: Pivotal.
At the same time this was going on, a new cloud enterprise player was emerging. Under the leadership of Rob Mee, Pivotal, the agile software consulting firm based in San Francisco, inherited the Cloud Foundry code base. This was a great fit, because Pivotal had a large team of Ruby developers and had entered into a partnership with VMware.
Alex Tarpinian and Michael Fraenkel, two of the initial leaders for IBM’s Cloud Foundry team
Naturally, this new stewardship also meant great change for the Cloud Foundry code base, because Pivotal engineers started to include their opinions. However, the biggest change at the time was that the Pivotal senior management team insisted on using a strong agile process to guide the directions of Cloud Foundry. Meanwhile, IBM and other big players were ready to invest millions and dedicate dozens of engineers to the project. Was this a culture clash waiting to happen?
Pairing and processes: 2014 to present
Perhaps no aspect of the process that Pivotal used for Cloud Foundry has had more contention than pairing. Pair programming is a long-time agile value that dictates that all code (development or test) should be done in pairs. While interesting and novel to some, it has been long researched and explored in the academic communities and even used extensively in various small start-ups. The consequences of pairing are probably greater than either how hard or how useful it is in practice.
First, pair programming requires that both engineers be present at the same time and the same location, or, when remote pairing, tethered to the same audio and video line. Besides the practical consequence of tying up two resources to work on the same code, the logistical requirements for pairing became a point of contention when more international companies, with developers spread across time zones around the world, joined the Cloud Foundry efforts.
A pairing session
To address the concerns and complaints around their process, the Pivotal management team gave unique opportunities to train engineers from all companies by allowing them to join and learn Pivotal’s agile approach to software engineering. And so the dojo program, which came to define how engineers get up to speed and help contribute to Cloud Foundry, was born. Early IBM engineers in the Cloud Foundry project, such as Michael Fraenkel, Matt Sykes, Julian Friedman, and I, participated in part of this program. The time commitment was sometimes four to eight weeks.
Besides the huge time commitment and potential costs, one of the controversial aspects of the pairing process at the time was that dojo developers needed to participate in person in San Francisco, and they had to pass a test to be invited. Known as the RPI (Rob’s Pairing Interview), the test was a Pivotal trade-secret programming exercise that Rob Mee and the trained management team used to determine whether potential employees (or in this case Cloud Foundry OSS engineers) had the skills and would fit the Pivotal agile culture.
Modern developer, Nima Kaviani of IBM, pairing in the age of mobile workers
Over time, as more engineers participated in the dojo process, the initial negative reactions or push-back dwindled, whether about the costs, commitment, or what some perceived as unnecessary barriers to contribute. The IBM team presented what they learned from their experiences and bore witness that the commitment and costs were overall worth it. Indeed, the process, while not perfect, had its merits – for example, quick ramp-up time, sustaining quality, empathy toward all developers, no one owner, and many others.
While the early dojo program went a long way to bring Cloud Foundry beyond the realm of just Pivotal engineers, it did not help accelerate the community. Perhaps not accelerating contributions was a good thing for Cloud Foundary in general at the time. Internally, the project was going to a near re-write on a popular, new (but appropriate for cloud software) language called Golang.
This post recalled the beginnings of Cloud Foundry, from an ambitious open source project created at VMware by ex-Googlers to the move to a hot new agile company called Pivotal. I summarized early years under Pivotal stewardship and the early growth with participation from larger industry companies such as IBM, SAP, GE, and others. I also looked at the unique (for open source) agile process that Pivotal insisted on using and the ramifications of this process (both good and bad).
Cloud Foundry Summit 2017 sponsors with Cloud Foundry Foundation Executive Director Abby Kearns
Part 2 of this series discusses the creation of a foundation and summarizes the history and evolution of the major components of the Cloud Foundry project, for example, BOSH, Runtime/Diego, Extensions, and the recent raprochement with other open-source projects like Kubernetes and Istio. For each of those areas, I look at how IBM and other participants were able to contribute and help Cloud Foundry enjoy the success that it has had.