A debate has been raging for the past few years about the right way to build software products. There are essentially two schools of thought and the two are pitted against each other with as much conviction and animosity as a presidential election. And much like a political or religious debate, neither side seems to be much interested in influence from the other. Despite what you’ll hear from zealots on either side however, they both have their strengths and weaknesses, and each has circumstances in which it is a better choice. The goal of this blog is to compare the differences and guide decision for what might be best for your business.
Stage-gated processes such as Waterfall emphasize completion of one task before proceeding to the next. For example, a business stakeholder must perform due diligence and complete requirements documentation before passing the project on to the technology leads to perform technical analysis and the designers to implement a graphic design. Upon completion of the technical analysis and graphic design artifacts, the project is ready to be implemented by programmers.
This approach has some benefits. The business team is forced to gather all of their requirements before production begins and that can be a fantastic incentive to get the business stakeholders to prioritize the effort and complete the document. Otherwise, business stakeholders might be inclined to do a half effort, pass along a requirements document lacking a lot of details and expect technologists to begin designing and building the product. Everyone will be frustrated in the end, when the product is late, and doesn’t look or act as was desired. Waterfall is very good at solving this type of problem.
Waterfall can also be a significant advantage from the perspective of budgeting. When working with external vendors or even service units within a larger enterprise which are driven by strict annual budgeting, there can be little tolerance for wasting money. By practicing a Waterfall methodology, business stakeholders get everything in place to be maximally efficient with the use of technology and design resources before engaging them, thereby reducing capital waste. And, the vendor can also deliver on a fixed-cost or fixed-timeline, which helps overcome trust issues that can be a problem otherwise working with external vendors who bill by the hour.
Iterative Methodology (Agile & Lean)
Iterative methods such as Scrum, Extreme Programming and the Rational Unified Process (RUP) rose to prominence in the 1990s, in direct response to the perceived short-comings of Waterfall and other Stage-Gated approaches. In 2001, the Agile Manifesto was written to unify these efforts and as a result the Agile Method was born. The key issue they collectively seek to address, is the waste of building the wrong product or features. Whereas Waterfall in particular is great for reducing the cost of production, it is slow to produce results (since planning must be 100% complete before implementation begins) and outright terrible at discovery and responding to user feedback. Proponents of iterative methods would argue that while stage gating methods such as Waterfall make efficient user of capital and resources to develop a product, it is only efficient in creating something that nobody wants! This is a real problem for startups and new products, in particular.
The Agile method focuses on software development and includes several sub-processes such as Scrum, Extreme Programming, and Adaptive. Collectively, they provide tactical guidance and frameworks for implementing iterative development process within an organization. The Scrum sub-process for example emphasizes face-to-face communication and requires a daily meeting between team members to provide updates and social accountability, as well as align everyone around the common goals and challenges. It also recommends that all members of the team work together in a single office without walls (called a “bull pen”) to facilitate open and organic communication. The idea is that the team can be more reactive and ultimately agile and responsive to changing business requirements this way.
Agile also mandates that delegates from each business unit be an active part of the process, every step of the way. Each iteration is typically 1-4 week time-boxes and while developers are implementing the design for the current iteration, designers and business stakeholders are creating the requirements for the next iteration, to be completed at the same time as implementation of the current. When the current implementation is completed, it goes to QA to test, while development codes the next iteration, and so forth.
In this way, design and functional requirements are being created just shortly before they are implemented. This allows for a tight feedback loop in which product designers quickly realize flaws in implementation and business stakeholders can quickly perform user testing and correct any flawed assumptions as for what is actually being built.
The Lean method is similar to the Agile method as it is iterative in nature, though the focus is product-market fit and less upon product implementation. Whereas Agile provides a feedback loop for internal discovery among business units, Lean focuses more upon customer feedback, discovering product-market fit and optimization details such as conversion rate and landing page optimization.
So which is a better option for your organization? It depends. Waterfall has been beat up a lot by those who reject it in favor of iteration, but it is actually a better fit when working with external vendors, since cost-control is often a key issue. Agile can be a great choice internally within organizations that are product focused and which foster an open enough environment to allow for communication and inter-departmental cooperation. A lot of organizations realistically do not have the culture to succeed with Agile however. I use to work at one organization in particular which had a known policy that business stakeholders were not even allowed to enter the Engineering wing!
The nature of the project is also important. If you are working on creative or discovery-driven projects such as analytics-driven optimization or are simply charged with increasing revenue by whatever means, iteration is the only option, since you need to make changes, test, and respond systematically. Conversely, if you are performing a systems integration project with very specific requirements that are not going to change and require no discovery, Waterfall will likely bring more success.
Another consideration is the composition of your team. Agile is often preferred among the more big-picture and senior members of a team, who have the competence and visibility of overall business objectives to be effective. And in fact, it is good to engage such people and allow them the opportunity to work with Agile and Lean methods, since they will likely get bored and not be challenged by having every detail spelled out to them in a Waterfall style requirements document. Conversely though, junior team members often need the structure and oversight of a more structured environment and might not perform well on an Agile team. Agile has also been implicated as non-effective on larger teams. If you have more than 20 people on your project team, the overhead of managing rapid-succession iterations might overwhelm your effort.
Finally, consider stepping back from all of the details of what Stage Gate and Iterative methodologies are, and simply ask this question: does your business benefit more from a large-batch or small-batch approach? Stage Gating is much like an assembly line and assumes you have all of the answers and are not making any strategy mistakes. If the answers are known already or discovery of the answers within your organization is not going to be tolerated, then large-batch is the way to go. If you are an early-stage company or are intro ducting new products or trying to optimize user engagement or monetization of those products however, small batch is the right choice, since you need to build a little bit, test, get feedback, and take the next steps based upon feedback. If retaining top intellectual talent is core to your business, Agile provides more latitude and will allow you to keep your best minds engaged.
The following lists provide a quick comparison of waterfall and iterative pros and cons.
» Creative and Discovery-driven projects
» Transactional Projects (black & white requirements)