Consuming open source at scale The IBM Watson Workspace team shares how they successfully balance development and compliance by automating every step of the process in a continuous delivery environment. Meet the IBM Watson Workspace team consuming open source at scale Watson Workspace redefines how users collaborate. Too many conversations and too many tools result in a lot of noise and interruptions – which means you spend a lot time and energy pivoting back and forth. You don’t really know where to focus because every unread mark and notification is screaming for your attention and every conversation is weighted the same in the list. This means it’s far too easy to lose where you were, miss what’s important or what’s even worth catching up on. With Watson, we leverage cognitive intelligence in meaningful ways to improve how you and your team work. We summarize conversations, prioritize what’s important to you, understand business dialect, and connect actionable language to next steps so you can focus on what matters and get more done. DevOps culture and open source “Widespread adoption of DevOps culture combined with the explosion in Cloud-Native open source technologies has created an unprecedented opportunity for developers to deliver amazing products to their end-users, whether there are 100 or 100 million of them.” Brendan Arthurs – Architect, IBM Watson Work Process automation is key Beyond making things better for our end-users, Watson Workspace has redefined how we build, test and deliver software in IBM. We started from the premise that the cloud-first principles which revolutionized the consumer software market can and should apply just as well when building enterprise collaboration tools: Horizontal scalability to support millions of users, true continuous delivery with builds deployed to production within hours of code being committed, security at the foundation of everything we do and a delivery model where every step of the process, from testing to deployment is automated. Open source management and approval process “IBM’s review and approval process for continuous delivery enables teams to confidently adopt the latest technologies by streamlining the review process which was previously seen as an inhibitor to using the most appropriate open source code for continuous delivery offerings.” David Lau – Release Manager, IBM Watson Work Built on a microservice architecture, components in Watson Workspace are updated more than 50 times every working day. Open source is at the core of everything we build. From the Spring framework for lightweight services and Hystrix for resilient network calls, to Apache Kafka for reliable scalable asynchronous messaging, we build our product on a rock solid foundation. This foundation in turn relies on countless additional battle-hardened open source libraries. When the full tree of dependencies is expanded out, we use over 600 open source packages. This keeps growing daily as developers continue to add new features to the solution. Each of these needs to be reviewed for origin and license compatibility, and regularly updated as security vulnerabilities are identified and fixed. The importance of vulnerability management in dependent packages is hard to overstate, as a glance at recent news headlines shows. Rather than attempting to track every published vulnerability in every package we use, we have put automation in place to ensure we always use the latest supported version of each dependency, pulling in the latest security fixes in the process. All of which leads to a huge volume of open source package versions flowing through our pipeline. In order to stay true to our continuous delivery principles where changes are deployed to production within hours of being committed to source control, we have built an elaborate automated open source clearance pipeline which identifies every dependency referenced and checks its status against a GitHub repo where we track all approvals. Anything new is processed by calling IBM’s in-house tooling to identify the license type and determine whether additional review is required. If not, the package is included in a bulk request and cleared for release without the need for human intervention. When combined with previously approved packages, this covers over 91% of all references to open source packages added to Watson Workspace. Security and open source “Automatically updating to the latest published version of our open source dependencies means security fixes are always applied.” Olgierd Pieczul – Security Architect, IBM Watson Work Adopting a micro-service architecture on a container runtime is necessary but not sufficient to achieve continuous availability while innovating at speed. To maintain confidence, one must constantly operate with failure as an expected element. To address this concern – our process and culture has adopted fail-small best practices where everything gets automated, configured as code, and released in small increments that we can canary test and roll-back with ease. That applies to not only our core product but all down stream tools and processes we rely on, including the handling of legal and security aspects of open source. Through close partnership with IBM’s Open Technology Group, IBM Watson Workspace has delivered on an integrated process that has shown in practice we can protect IBM’s intellectual property while also maintaining developer velocity and a secure runtime. In fact, the net result has made it easier than ever to empower developers to choose the best tool for the job and for our operations to maintain a secure environment for our customers. Without IBM’s approach to open source, none of this would be possible. At a glance “Over 90% of open source packages are automatically approved without human intervention” Quick stats Process automation reaps big benefits for the Watson Workspace team. 100+ Developer initiated Pull Requests per week 50+ Microservice deployments per day 1900+ Open source packages automatically cleared Team bios David Brooks ( @brooksda ) David Brooks is an IBM Distinguished Engineer and the CTO for Watson Work, where he is responsible for leading the architecture and development of Watson Work Services, Watson Workspace and Box Relay. He is focused on the use of cognitive technology to understand business conversations to allow users to prioritize what’s important and act in-context on what matters. David believes that building software starts with an effective team culture. When he started Watson Work, he formed a team that was structured to achieve continuous delivery, to fail fast and fail small, to own the problem end-to-end, and to build a foundation from the ground-up using a hyperscale micro-service architecture under active management with container orchestration and an effective tool chain to support build, test, and deployment pipelines. His passion lies in leading a team capable of delivering on key engineering principles to achieve a cloud native architecture that can rise to any challenge through quick iterative delivery. Moving fast and failing-small best describe David’s approach to software development. Prior to starting Watson Work, David was responsible for creating and leading the development of IBM Connections, the market leading enterprise social software platform. David has over 15 years experience building large scale enterprise and cloud applications with a proven track record for driving innovation through challenging the status quo. Brendan Arthurs Brendan Arthurs is the DevOps and Infrastructure Architect for Watson Work and has been part of the project’s core technical leadership since its inception. He is passionate about breaking down barriers to efficient software development and delivery. In defining Watson Work’s pipeline and runtime model, he has removed the bottlenecks associated with traditional development, prioritising fully automated delivery of small, high quality changes on a continuous basis. Brendan has a long history of delivery in cloud in his 14 years at IBM, from the first ICS Software as a Service offerings in the mid 2000s, through to more recent work on realtime meetings and collaboration. Throughout this time he has constantly sought to bridge the perceived gaps between Design, Dev, Test and Ops, allowing teams to align on a common objective of delivering amazing products to their end-users. He sees the advent of DevOps culture across the industry as the culmination of this journey. When combined with the recent explosion in cloud-native opensource technologies, Brendan sees the opportunities available to development teams today as unprecedented. Josef Harte Josef Harte is the lead for Watson Work’s core pipeline squad. A member of the team since the early days when the project was just an idea, he was part of the core team that laid the foundations of the Watson Work platform. With an eye for quality, he enjoys “getting his hands dirty” building technical solutions that can meet the demands of a vibrant development culture. Working closely with both Watson Work’s technical leaders and key stakeholders in areas such as release management, he has helped drive conceptual ideas through design and implementation to achieve success at continuous delivery, one of the project’s foremost goals. A relative newcomer compared to some of Watson Work’s more senior engineers, Josef joined IBM back in early 2013 after completing an MSc in computer science. Originally graduating with a degree in physics, Josef was drawn to software engineering due to a fascination with building and understanding complex systems. During his time at IBM, where he first started working on an on-premise collaboration solution with a more rigid development environment, he has experienced first-hand the transition to DevOps culture. He welcomes the sea-change in development practices and sees embracing the open source community as critical to success in modern software development.