How to use Waterfall and Agile practices on your next project

When you first begin working on a project, figuring out which methodology or way of working can seem daunting. How do you know which approach will produce the best results? What if your project requires more than just one practice or way of working? Could using more than one methodology create better outcomes?

When tasked with delivering a new API functionality to comply with new government regulations, with set industry-wide specifications to meet, we had to decide whether to use a Waterfall or an Agile method. For a brush up on these different methods, see our articles, “The Waterfall Model: Advantages, disadvantages, and when you should use it” and “The Agile Method: Everything you need to know”.

In this article, you’ll learn about our first-hand experience of implementing the Waterfall Model as well as Agile methodologies in the context of delivering an API Platform and how a hybrid approach may be better than using just one, single practice.

API Connect

For this project, IBM API Connect was chosen as the strategic product. API Connect is a platform to create, manage, and securely socialize information via APIs. This centralized platform allows multiple API providers to deliver services and portals to their customers both inside and outside the organization. API Connect is made up of:

  1. An API manager for managing API creation, publishing, communities, subscriptions, and exposure.

  2. An API gateway for managing and enforcing API traffic, rate limits, throttling, and security.

  3. A developer portal where consumers can explore APIs, subscribe through free or paid-for memberships, create applications and generate credentials, and learn more about the APIs that they are using and consuming.

The Waterfall Proof of Concept

After choosing the correct product, it was important to set goals. Here’s what the organization came up with:

  • Completion by a set deadline, which was determined by an external entity with sanctions for being late.

  • A set number of intermediaries have to be able to enroll and use the system on behalf of end users.

  • Specific, well-defined APIs have to be available for subscriptions and callable by the set date.

  • The system can have no unplanned downtime without financial cost after the go-live date.

  • The systems must pass certain pre-defined tests agreed to by the industry.

An on time, in budget delivery

When using the Waterfall Methodology, there are three key outputs from projects that must be satisfied: scope, cost, and time. Depending on your project requirements, you and your team may need to come to an agreement to reduce output on two of those three key outputs. For example, this particular project has a clearly defined and inflexible scope created well in advance of the start time, with requirements and timelines already agreed upon. If the scope is not delivered, there will be financial recriminations, meaning an increase in overall cost.

The cost of this project is also already set and defined, with most of the external resources on fixed price contracts to an arranged end date. Because of this, resources are focused on delivering a minimum regulatory requirement as quickly and as efficiently as possible.

By choosing to focus on time versus scope and cost, we are able to deliver without delay, in budget, and with general satisfaction from regulators and the organization.

Teams working in tandem

Throughout the delivery of this project, teams build the key parts of their solution or product semi-independently. APIs are built by one team, the API Connect infrastructure is built by its own subject matter experts, and a support team builds a QRadar and then a Splunk system for exporting logs. Other teams deal with load-balancing and higher-level traffic-balancing. There is even a dedicated DataPower team.

Each team creates and implements their own design. When integration is required between any two systems, the two teams have informal conversations around how it can work. In the event that a decision can’t be made, the project lead is consulted. Each team describes the pros and cons, allowing the project lead to make the final decision. This is also how changes in design are resolved. Again, the project lead makes the decision, with the outcome documented in a live deployment summary that is updated for each major resolution.

Controlling traffic and governance while lowering risk

The Waterfall approach works particularly well for the initial build of the API Connect infrastructure. By being the central point of the traffic flow, the majority of decisions can be made at the infrastructure layer. The control of traffic and governance throughout for consumer applications and subscriptions can be defined at the onboarding level and are thus passed on to downstream teams.

When limitations or organization hardlines surface from downstream services, it is easily resolved, as API Connect allows a lot of traffic alteration using policies, DataPower extensions, and gateway scripts. This flexibility of the platform is highlighted throughout different phases of development and the relative ease of customization of the software platform provides some de-risking of the project.

Becoming Agile

The transition from a Waterfall project to an Agile project in an existing system is somewhat chaotic. It takes a while to understand the team dynamics and move to a new way of working, as many teams are still insular (a symptom of drastic changes within the working environment).

Having created a Minimum Viable Product (MVP), a solution that met the regulators’ minimum requirements, the organization decides it’s time to make the project more stable and reduce technical debt. They’d like to make it more compatible for quick and easy future adoption. It’s also important to align the practices of the project with the organization defined and enforced production-level governance and control.

To do this, a series of high-level goals are created:

  • Eliminate technical debt and defects.

  • Onboard 8 new API Providers’ teams onto the hub by the end of the year.

  • The system must have no unplanned downtime, and planned downtime should be kept to a minimum.

  • The systems must pass stringent pre-defined tests agreed upon by the industry over the next year.

  • Integrate lower middleware layers (used elsewhere in the organization) with the MVP API Connect layer.

To complete these goals, several teams are created with scrum leads and project managers. Additionally, a team of architects has been consulted and added into scrums (daily meetings) when required. Scrums provide an increased understanding on how systems work.

Clearly defined expectations

Following tickets as part of a fixed iteration (sprint) allows for better time management and additional workloads can be pushed back on. Enforced is a policy of only doing work that has a clearly defined ticket. This allows a greater understanding of what is being performed and what is expected in the fortnightly sprints, adding a lot of administration time. The ticket system provides a prioritized backlog for teams to work against. This is also true for sprint planning or ticket management, which feels at times like an Agile ticket-driven-development methodology, which basically means if there’s no ticket, there’s no development.

Successful retrospectives

A great part of the Agile approach is the ability to look back at how the team is working, what can be improved, and what went well. This means the team is able to look at whether delivery targets are always met, if there are shortfalls in knowledge and experience, and when dependencies are mismatched or unaccounted for. Regularly reflecting on past sprints and including changes where necessary is vital. As a result, there is better communication, greater delivery week-on-week, and a realignment of expectations as to what could feasibly be achieved in a given time period across a full stack system.

So, which methodology is better?

API Connect as a product can work across multiple methodologies, whether it be Agile, Waterfall, or something else. The teams that manage and support API Connect need to be flexible to change their approach for each project and its required goals.

As you reach different phases of your project, some instances may seem as if they are better suited for an Agile way of working, while others may require a linear or sequential approach such as the Waterfall Method. For example, infrastructure, often being split across different teams for different clouds, with slower build times than a typical iteration, benefits from the Waterfall Method. However, the application improvement and development are much better suited to often feature changing requirements and thus lends itself to the Agile approach.

For these reasons, we think a hybrid approach that uses fundamentals from both Agile and Waterfall Methodologies may be the best. Elements from both methodologies give you the perfect recipe for success.

Aiden Gallagher
Jack Dunleavy
Peter Reeves