The history of IBM’s contributions to Cloud Foundry, part 3
The Cloud Foundry CLI, services, and the CF-Extensions Project Management Committee
In part 2 of this series, I exposed a thorough overview of the evolution of the key components of Cloud Foundry. I also recalled the creation of the Cloud Foundry Foundation and how that organization helped give structure, fairness, and support to all members and contributors to the Cloud Foundry code base.
In the last part of this series, I explore a few more areas that IBM has contributed to strongly over the years. In doing so, I summarize their history, as per my recollection, including the Cloud Foundry CLI, services APIs, and the bits service. I also summarize the CF-Extensions Project Management Committee history, because I am tightly intertwined with its creation and evolution.
The IBM booth at the Cloud Foundry Summit 2018 in Basel
To conclude, I extract three lessons learned from my experiences helping and witnessing the evolution of this large open source project. These lessons are not mean to be normative nor pointing fingers at failures. Rather, I hope they can be useful to other open source software projects (and even to the Cloud Foundry community itself), as we continue the cloud revolution in IT from the past decade into the future.
When a company as large as IBM participates in an open source project, we tend to contribute to many parts. This means engineers work on all aspects of the code and documentation. It also means sponsoring events, meetups, hackathons, and related activities. And finally, it means having a massive advocacy plan to help promote the technology in question through tutorials, blog posts, journeys, and related collateral.
Chris Clark of the Cloud Foundry Foundation helping run one of the many hackathons that IBM supports and sponsors
With a project as large as Cloud Foundry, we sponsored and participated in the Cloud Foundry Conference in 2013 and every Cloud Foundry summit since 2014. We sponsored two hackathons and countless Cloud Foundry Days and webinars, as well as created a massive trove of advocacy materials that you can find and use on the IBM Cloud. However, in addition to the projects I mentioned in part 2, what other groups of projects and smaller items have we contributed to? The following sections provide a good sampling of miscellaneous contributions that I have not yet highlighted.
As mentioned earlier in this series, When Cloud Foundry was first released, most of the components were written in Ruby, including the command-line interface (CLI), which is probably the most used component in the platform. It constitutes the interface for all developers and platform managers. The CLI is how they interact with the platform to deploy and manage their applications.
Anand Gaitonde of Pivotal, a longtime former anchor for the Cloud Foundry CLI project
I am not exaggerating in saying that the Cloud Foundry CLI exposes “the experience” that everyone has with Cloud Foundry as developers. It’s “the language” that is used to communicate with the platform. Its importance cannot be denied. Early versions, while good, suffered from two important issues (among many challenges). First, the CLI was slow to start, and second, its installation was suboptimal. Both of these issues could directly be blamed on the language and runtime choice for the CLI.
An early prototype of writing the CLI in Golang showed so much success and benefits that the CLI was the first component to be completely rewritten and delivered. The reimplementation was not only fast, but required simply downloading one binary file! All was good, and the CLI team at Pivotal was a favorite place to rotate into for those wanting to learn Golang.
Simon Lueng, former anchor and engineer for Cloud Foundry CLI
As IBMers began participating in the dojo program, the Open Technology team from our Silicon Valley Lab increasingly got more involved in the CLI. For a period of a year and a half (in 2016-2017) the duo of Simon Lueng and Jonathan Berkhahn constituted the majority of the CLI team engineers (with Simon as the engineering lead), and helped the product management team from Fujitsu and Pivotal.
Finally, another worthy mention was the internationalization (i18n) of the command-line interface. Since the CLI was the primary interaction to Cloud Foundry, and as IBM started to plan its roll out of Bluemix (the previous name for IBM Cloud) across the world (particularly in Asia), there was a need for support for many languages. Developers interacted a lot with the CLI, so it needed to be translated. The problem was that Golang did not support such features at the time.
Jonathan Berkhahn, former engineer for the Cloud Foundry CLI
To address this shortcoming, I developed a tool that allows any Golang program to be converted (re-written semi-automatically) in new code that can be easily internationalized. Using the i18n4go code, I was able to demonstrate how to convert the entire CLI code base quickly. Working with Pivotal CLI engineers and IBM translators in China, we refined the i18n4go tool, while also converting the entire CLI to one that supports all the major languages for every release since 2017 (10 at the current count).
One of the crown jewels of the Cloud Foundry architecture is its simple and elegant approach to how it exposes, integrates, and helps discover services. You can think of services as any piece of code or data that is external to a user’s application. This view is a broad concept but one that is useful for how Cloud Foundry users conceptualize and design their applications.
In addition to being flexible about what is a service, Cloud Foundry defines the lifecycle around how services are both integrated – and more importantly – discovered. The Cloud Foundry Service Broker architecture defines the interactions and data that all services and service brokers need to expose. Briefly, services needed to have some metadata when used and needed to expose an endpoint.
David Stevenson, Director of Engineering at Pivotal, lead for the Services APIs team
To discover and create service instances, Cloud Foundry users need to go a service broker that exposes a marketplace of different services. It’s not uncommon for Cloud Foundry environments, such as the IBM Cloud Foundry Enterprise Environment to have dozens to hundreds of services exposed from a variety of service brokers. The architecture was so successful that early on, the folks adding new features (such as container services) to the IBM Cloud wanted to participate in the Cloud Foundry service architecture.
Because Cloud Foundry is an all-encompassing platform as a service (PaaS), the service architecture had three limitations that needed to be addressed in order to make it more broadly useful. First, the architecture assumed from the start that services would be discovered, selected, and bound to applications in a short time frame. For services that required provisioning or some post-binding process, there was no way to easily tell the application when the service was ready or when it was deleted.
David Sabeti, Director of Engineering at Pivotal and former anchor for the Services APIs team
The IBM Watson team quickly encountered this limitation when they tried to integrate their complex analytics services and expose them in the IBM Cloud marketplace. We needed a means to allow services to have asynchronous lifecycles. Working with Pivotal, we got this work started, which led to the creation of a special Services APIs team to look into other aspects of the service broker architecture that needed improvements.
The need to externalize access to the service instances was the second improvement to the architecture that the IBM team stumbled upon and Pivotal had also discovered. In particular, as the IBM Cloud added container services support, it became clear that customers wanted container-based applications to also integrate the growing service catalog that Cloud Foundry exposed.
We expanded the IBM team working on services to also include two team members from our IBM China Development Lab to spin up the work to add service keys. Working closely with the Pivotal Services APIs product manager Shannon Coen, we were able to manage a distributed team of engineers from China to add an important new feature to the Cloud Foundry Service Broker architecture.
Shannon Coen (Pivotal) and Doug Davis (IBM), two of the leaders for the Open Service Broker APIs extraction
Finally, as Kubernetes grew in popularity, Cloud Foundry looked to the Cloud Native Computing Foundation (CNCF) to simplify and streamline its architecture. We also looked for Cloud Foundry components that could be exposed as common components for other projects. The Cloud Foundry Service Broker was a clear example of a component that could serve this purpose, like it did on the IBM Cloud.
Working with Pivotal, Google, and others in the Cloud Foundry Foundation and the CNCF, Doug Davis (IBM) and Shannon Coen (Pivotal), collaborating from others, led the creation of the Open Service Broker API. It now stands as a common means for integrating services into any PaaS or container application service framework.
As previously discussed, the Cloud Foundry architecture includes all the components needed for a full platform. That architecture is one of its charms and its main appeal. However, in some cases this architecture means reduced flexibility in implementint a component. For example, most platforms need registries for large blobs of bits. Registries are needed to not only store the images that are used to spin up containers, but also to store the software that is installed in these containers.
Simon Moser (IBM), leader for the IBM Cloud Foundry team in Germany, and his spouse
Cloud Foundry implemented registries by using a blobstore that had some flexibility to be backed by or switched to external cloud services, such as AWS S3. While a sensible approach, the blobstore was not as useful for the machinery that the Cloud Foundry runtime runs when building and layering images. A more focused component was needed that could be fast and tailored to its tasks.
The IBM Germany team, under the leadership of Simon Moser, took the opportunity to contribute a large component to Cloud Foundry by taking on the task to design and implement a Bit Service for Cloud Foundry. After a few iterations, the team delivered a working Bit Service in early 2018 that is now being shipped as part of every Cloud Foundry deployment release.
As Cloud Foundry expanded to include more CNCF projects into its core and looked how to expose some of its components so that they could be used, the common thread that wove across community discussions was about enabling extensions. Goals were to allow the platform to be extended in specific plug points but also to allow an open process for adding new components to the platform. This cross-cutting concern was realized by Sam Ramji when he asked me to take the leadership of one of the three Project Management Committees that now constitute Cloud Foundry, called: CF-Extensions.
After an opportunity to present my plans for the new CF-Extensions Project Management Committee to the Cloud Foundry Foundation board at the Cloud Foundry Summit in Basel at the end of 2017, I started hosting monthly calls. Using the Community Advisory Board (CAB) calls (which I also host) as a springboard, the CF-Extensions quickly attracted dozens of project submissions ranging from UI, to CLI plugins, new components, and various improvements to the platform.
Experimentation and Extensions track at CF Summit 2018 in Basel
As we had predicted, opening the platform to extensions allowed “1000s flowers to bloom.” However, what are some early successes? What projects have moved from incubation to widely accepted and core components? Well, we can point to at least 10 projects that constitute the CF-Extensions core, but I’ll highlight two that have had particularly strong and successful receptions.
First, the IBM-led App AutoScaler was created by Michael Fraenkel and Bo Yang as an extraction to the IBM Cloud auto-scaling service. Using information from the runtime, the App AutoScaler service exposes a simple means (through the service broker architecture) that applications can specify scaling policies and have the scaling service automatically apply when selected criteria are met. After being deployed in production by IBM, SAP, and tried by other Cloud Foundry Foundation members, the App AutoScaler has now graduated to a core project in 2019.
Project Stratos UI is another great example of widely accepted extensions, a success story in the CF-Extensions Project Management Committee. Led by Neil MacDougall of SUSE, this complete UI for Cloud Foundry allows all users with a browser (desktop or mobile) to intuitively manage and view the entire list of their Cloud Foundry environments at a glance.
Neil MacDougall of SUSE, leader of the Stratos UI extensions project
The UI takes care of slicing the Cloud Foundry data into manageable and intuitive tabs. You can aggregate multiple environments from various vendors. It is also functional, allowing you to create, delete, and modify your applications and services. It’s a full-blown, integrated web console for Cloud Foundry.
Additionally, since becoming an extension, the Stratos team has had three major releases, adding the ability for the UI to use skins (matching the colors and style of whoever is adopting it) and to be customized with new functionality, among other features. For example, working with the App AutoScaler team, a newer version of the Stratos UI will include auto scaling data and functionality as an extension.
There are many lessons I could draw from my experiences with Cloud Foundry (as I am sure anyone would), but I want to extract lessons that I believe can be generalized to other communities. Hopefully, these lessons can resonate with other members of the Cloud Foundry community and be useful to other open source software groups as well.
Lesson 1: help strengthen community with monthly calls
Every community needs a drumbeat to keep it going strong. For many open source communities, regular emails or conversations in social channels can keep the community going. Maybe it’s regularly a gathering at an event, for example the Open Source Summits that the Linux Foundation puts on a bi-yearly basis.
For Cloud Foundry, in addition to similar summits, we found that a monthly community advisory board (CAB) meeting worked well. This call has evolved from a status meeting call where different project teams would give statuses on their new features and releases to what I believe is now a sustainable format.
Eric Malm of Pivotal, Project Management Committee lead for Cloud Foundry Runtime
Instead of status calls, we use the CAB meetings as a means to showcase highlights from the key Project Management Committees (PMCs) governing the Cloud Foundry Foundation and the community. This means that PMC leads (Dieu Cao and Eric Malm of Pivotal, Marco Voelz of SAP, and myself) get a chance to highlight what new features or releases they would like to communicate.
Importantly, we found that the one hour call is best used if these highlights are kept to a minimum. For Cloud Foundry, we use the first 20 minutes of each call for PMC highlights, including from the Cloud Foundry Foundation itself. We learned to use the rest of the call for two important purposes.
Dieu Cao of Pivotal, Cloud Foundry PMC lead
First, we allow freeform Q & A. We give the community participants an opportunity to ask questions of PMCs and Cloud Foundry Foundation leaders and of the general community. And second, we use the remaining time to give community participants an opportunity to present and demo a project, an idea, a use case, or a concern that they have in mind.
These CAB presentations, typically limited to 20 minutes, are triaged, curated, and scheduled before the call. The CAB leader (like I have been for the past two years) sends an email to announce the call to the community, to request a proposal for presentations, and to directly invite leads of interesting projects to present. The CAB call is conducted through Zoom video conference and recorded and posted on YouTube with an accompanying CAB blog post that Cloud Foundry Foundation member Altoros has created regularly.
Roger Stukrof (background) of Altoros taking notes at the CAB call live during the Cloud Foundry Summit in Boston
The resulting format has received positive feedback from the community and as a boost allowed various external members of Cloud Foundry to present ideas for extending the platform through CF-Extensions. Two end-of-year anonymous surveys in 2017 and 2018 gathered results showing that this format works and has contributed to a stronger community overall.
Lesson 2: lower barriers to entry
A common saying in Silicon Valley is that the “smartest engineers don’t work for you.” It is a constant reminder that, while every organization and community can be proud of its talents, you should remember that more talent exist elsewhere. Therefore, a sensible stance is to always attract talent to grow your organization or your community.
Attracting talent is a complex topic that would require its own dedicated blog post and focus. However, we can all agree that one way to discourage talent is to create unnecessary barriers to entry into the organization or the community. On the other hand, adding some barriers is also necessary, because you also want to prevent distractions for the community.
Cloud Foundry Summit participants lining up at the summit in Santa Clara in 2016
So how can an open source software community attract talent while keeping some safe guards? For Cloud Foundry, the approach was to test for skills, agile knowledge (basic), and empathy. Pivotal proposed to achieve this with the RPI test and the dojo program.
As I previously mentioned, the dojo process was great and a boost to anyone who could participate, it also was a barrier that likely distracted and prevented numerous developers, who otherwise would have, from participating. While it is also true that pull-requests have always been a way to participate in Cloud Foundry, the need for dojo graduation was a requirement (early on) to be able to become a committer.
Alex Tarpinian, part of the IBM Open Technology team
The lesson learned here is to be proactive in reviewing and trying to strike a balance between what requirements and safe guards you put around a community and how open the community is (and is perceived to be). Generally, the more open, with fewer restrictions, the better. There is no hard science here, and a tactful approach that revisits decisions and looks at activity and participation data, is perhaps a reasonable approach.
Lesson 3: embrace change
While Cloud Foundry used an open agile process for its development, the actual architecture of the platform was rather opinionated and comprehensively baked. Initially the platform left little place for changes or extensions by design, which in many ways violates the basic principle of agility: embrace change.
Ideally, in retrospect, most platforms should be designed to have extensions and be pluggable. Designing a useful platform that is extensible and solves the problems of customers (right now) and allows changes to solve other future problems is a daunting task. Therefore, the initial Cloud Foundry designers chose to be opinionated instead (and that spirit continued).
Of course, the problem is that there is one constant in technology: change will occur. Your assumptions today will most likely not remain the same, so you need to adjust and change. You need to be agile. Software not only needs to be malleable (in the sense of change through agile methodologies), but also extensible.
Nimesh Bhatia and Carl Swanson of IBM, leaders of the IBM Cloud Foundry Enterprise Environment team
It’s costly to keep changing software, even when change is possible. Approaches such as test-driven development are beneficial to the development process, because they alleviate and contain costs. However, another additional useful approach is to design for changes. Make sure that your system contains clear areas where you can add extensions and experiments.
The Eclipse platform is an extreme example of a large open source software system that is designed for changes. Almost everything is a plug-in and can be changed (most of them dynamically). Such design points are useful but perhaps too extreme. A better balance is what the Kubernetes designers have achieved. The system comes built with all the right knobs but also contains a set of key primitives that can be extended (for example, custom resource definition).
The lesson learned here is to design for extensions, to allow and encourage experimentations, and to also use an agile approach that itself minimizes the cost of actual code changes. This approach strikes the right balance of allowing external innovation while providing a core that is complete but can also change and evolve with some efforts.
Cloud Foundry is now a mature PaaS. The history of its technical evolution and flourishing community hold many lessons that other communities can learn as they evolve their own platforms. While the presentation here uses the perspective of one company and of one of its employees, I believe that with small changes, these lessons can also be applied to other companies and other communities.
I don’t have a crystal ball prediction for the future of the Cloud Foundry community, except that I believe it will keep being important for a group of customers. It’s not clear if the customer base will continue to increase, nor if the community itself will keep growing. These are complex, forward-looking questions to answer. I chose not to opine.
However, one thing for me is for sure, and it is that Cloud Foundry’s place in the history of cloud and cloud platforms is cemented. And for a class of customers, Cloud Foundry offers everything they need and more, so why would you change or break up with a good platform? That would not be rational and would be suboptimal.
Writing any piece that retraces the history of a complex human group or work product is difficult and requires the peer review and assistance of many. For me, this blog series was no different. I thank IBM colleagues and dojo graduates Nima Kaviani and Swetha Repakula for their diligent feedback on early drafts. I also want to acknowledge Alex Tarpinian and the entire Open Technology team at IBM for being great colleagues and supporting this series.
Also, a grateful and warm thank you to colleagues across the Cloud Foundry community from the Cloud Foundry Foundation to various member organizations and dojo graduates and participants.
And thanks to the Cloud Foundry colleagues at Pivotal, particularly in San Francisco, who under the leadership of Rob Mee, have had a disproportionately large contribution to Cloud Foundry. The platform would not be what it is today without Pivotal.
Finally, I firmly believe that I am a better engineer and a better person today for having met and built lasting relationships with the various Cloud Foundry Pivots I have worked with, encountered, and met over the years. Thank you.