In the software development life cycle there are two predominant methodologies, Waterfall and Agile. In this post we will talk about whether you should switch from Waterfall to an Agile process, specifically Scrum, and what benefits your project receives by choosing a particular methodology. To understand when we should use a particular technique we must first understand that technique.
Going Off the Waterfall in a Barrel
Waterfall is a software development methodology most notably describe in a 1970 paper, [Managing the Development of Large Software Systems|http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf], written by Winston Royce. In the paper Dr. Royce describes a practice for developing software that seems intuitive.
Base Premise of Waterfall
Before software can be developed it must first have defined requirements and a user interface design must be created. Both the Requirements and the Design must be completed and approved before implementation by developers can begin. This all makes sense and I don’t think anyone would disagree. You simply can have developers writing code for a system that has not been designed. Dr. Royce quickly points out the flaws in this naive approach to software development.
Firstly, when a project starts the Project Managers and Business Analysts know very little about the physical and technological limitations that will be encountered. To address this problem Dr. Royce explains that a feedback cycle must be assumed. The feedback cycle should, if at all possible, be constrained to successive steps. For instance an issue discovered in Design will only go back to Analysis and an issue discovered in Development will only go back to Design.
Unfortunately, the wishful solution doesn’t seem to work in practice. Issues found in Development or Testing all too often travel back up the stream all the way to Analysis or even base requirement changes for the system. Dr. Royce acknowledges this limitation and simply states that he feels the process is sound and the issues are manageable. For the most part I would say that Dr. Royce is not incorrect.
Further practices encouraged by Waterfall include heavily documenting decisions and updating the documents when changes occur. This process generally becomes tedious and seems to be the first thing to go when a deadline is approaching.
If you haven’t read the Waterfall paper, I strongly encourage you to do so. It is informative and will give insight to the thinking of your peers and other individuals involved in the software development process.
Benefits of Waterfall:
- Very prescriptive
- Well defined
Detriments of Waterfall:
- Very lengthy feedback cycle
- Costly delays of delivery
- Restricts communication
Scrum to the Rescue
Scrum is an Agile methodology first described in a 1986 paper, [New New Product Development Game|https://cb.hbsp.harvard.edu/cbmp/product/86116-PDF-ENG], written by Hirotaka Takeuchi and Ikujiro Nonaka. In the opening volley of the paper, Takeuchi & Nonaka state that the status quo of a relay-race approach to product development may no longer be adequate and instead suggest a rugby approach.
Rugby? Yes, that’s right rugby. If you are not familiar with the game it has a single action that has inspired Takeuchi & Nonaka, the scrum. A scrum in rugby is similar to a huddle, where all the players are grouped together with their heads down, with one very important difference. A huddle is a planning strategy in most sports where the plan for the next move is discussed. In Rugby, a scrum is used instead to move the ball forward towards a scoring opportunity.
So, how does a rugby scrum apply to software development? The idea is to combine the traditional phases of software development so that they overlap as much as is possible. In theory the requirements would only be created enough so that design can begin. Design would only go as far as needed for development to begin. Development would only progress far enough for testing to begin. Naturally the phases of a software development project would not simply stop when the next began but instead would continue in a semi-constant stream, constantly staying just far enough ahead of the adjoining stage to provide more requirements, designs, and/or working software.
Key Factors of Scrum
In the paper, Scrum consists of six major factors. Built-in Instability, Self-organizing project teams, Overlapping development phases, multilearning, subtle control, and organizational transfer of learning.
Built-in Instability begins with broad business goals and/or a general strategic direction. The team is simply tasked with achieving the stated goal or moving in the stated direction. In achieving the objective the team is likely to make many mistakes and take many risks. The team must constantly evaluate the goals it has been given or even defined and make course corrections to stay true.
Self-Organizing Project Teams
Self-organizing project teams arise from this instability by adding needed members and/or positions and changing project direction to meet goals. Team members can change when the objective changes or a pivot occurs to more adequately produce the desired result. A self-organizing team is given the trust, ability, and responsibility to make the decisions that could lead to success or failure of the project. According to the paper a self-organizing team exhibits three conditions: Autonomy, Self-transcendence, and Cross-fertilization.
Overlapping Development Phases
Overlapping development phases, as discussed earlier, is about taking the traditional stages of software development and beginning the next phase as early as possible before the completion of the previous or current phase. The benefits from this overlapping of the development phases go beyond compressing the development cycle and an extremely important side effect. The sooner a phase is able to begin the sooner the team can gain feedback generated in that phase and the sooner the earlier phases can adjust and make course corrections in the project. With a tighter feedback cycle a team is able to reduce overall costs and deliver software much earlier and cheaper than with a traditional approach.
Multilearning, is a term created by Takeuchi & Nonaka to best describe an important aspect of successful teams. The teams practiced both multilevel learning and multi-functional learning.
Multilevel is the concept that teams must learn at individual, group, and corporate levels. The authors suggest that different companies have accomplished this learning by utilizing different techniques. In today’s society we can see individual learning encouraged by companies like google by providing 20% time. During this time employees are essentially forbidden from working on work related projects and are instead encouraged to work on something, anything, that interests them. Other forms of individual learning might come from peer pressure. I myself, for instance, create a lunch meeting and invite the entire development group to learn something that will improve their knowledge of development. The frequency of these meetings varies based on participation and reception. At the group level, companies can encourage learning by hosting events or sending team members to conferences. Group learning can be vary from industry knowledge to team building exercises. The last learning level is corporate. Corporate learning can be encouraged by creating a movement or corporate wide program that is intended to improve transparency and highlight opportunities for improvement. At Toyota, employees are encouraged to voice any suggestions they have that might increase productivity. These suggestions do not fall on deaf ears but instead are carefully considered and often tested at production facilities to measure effectiveness.
Multi-functional learning is about gaining experience in fields outside your own. One practice that has stuck with me since my Army days is the practice of knowing the job of the person above you and the person below you. Translated to software development, a developer should know how to QA test and write requirements. A QA tester should know logic flows and how to write requirements. Finally, a Business Analyst should know logic flow and how to QA test. It takes years to master any of the mentioned disciplines, but there is no reason why an individual can’t have a passing knowledge of all three. This is not to say that the business analyst can fill in for QA, or that QA can fill in for the developer, or even that the developer can fill in for the business analyst. We should all understand the basics of the others jobs though.
Subtle control is the enforcement of self-control and light control from the business. We don’t want strict or heavily enforced process from the corporation as that will deaden our ability to react to change and adjust course. On the other hand, we don’t want chaos either. To achieve the right balance we require the ability to self-control the project and team along with gentle oversight from the corporation just to keep everyone sane.
Transfer of Learning
Transfer of learning is simply that. It is the responsibility of everyone on the team to share what they have discovered with members of the team and outside the group. Only by sharing our knowledge and practices can we improve.
Benefits of Scrum:
- Deliver faster
- Ability to course correct as feedback is received
- Encourages an atmosphere of learning and sharing
Detriments of Scrum:
- Requires extraordinary effort from everyone involved
- The process if not prescriptive at all
- Requires buy-in from entire company
Scrum, therefore, is really only the way to go if your project can be given a certain level of autonomy and everyone involved is fully committed to the team. Waterfall is likely a better process for your team if nearly everything about your project can be or is already known before development is intended to start. Scrum is also not effective if your company is unwilling to release software incrementally. The feedback gained from incremental releases is invaluable in many of the key areas of the Scrum fundamentals.
In this article we talked about what Waterfall and Scrum were intended to be. In the next article we will discuss what Scrum has become and define all the current roles, meetings, and practices associated with Scrum.