One of my teams been involved in a bit of a discussion as to whether or not defects in the product backlog should be sized using Story Points, just as we size stories. Mike Cohn has a very good blog about “Story Points for Bug Fixing.” As I read his post, I believe it is specific to the scenario where a team has inherited a set of defects from a prior release or a legacy product that they are going to attempt to work through in the midst of delivering new features. Mike’s recommendation is that defects be sized, much like stories would, and that the resulting completed work would count toward the team’s velocity. I agree with that approach. I believe the scenario that we’ve encountered is a different one, and thus a different approach may be more suitable. The scenario I would like to explore is one in which defects found in the current release’s earlier iterations. Should those defects be sized and be included in the team’s velocity?
Consider a scenario where the features set to be delivered consists of 100 points of work, and that the team’s average velocity is 20 point per iteration. The figure below shows what a release burndown might look like under these conditions. Prior to iteration 1, there are 100 points of work to do. At the end of iteration 1, we have completed 20 points of work with high quality and there are 80 points remaining. Finally, at the end of iteration 5 the backlog has been reduced to zero*. The burndown in this scenario is “Backlog Points Remaining after Iteration – Scenario 1”
This burndown shows the points remaining after each iteration is complete. Each iteration, the team completes 20 points of work on stories that are in the backlog.
In another scenario, we identify defects from earlier iterations, size them, and include them in the iteration’s velocity. So, in scenario 2, each iteration still shows a velocity of 20, but the burndown does not reach zero after the five iterations are complete.
This burndown shows the points remaining after each iteration is complete. Again, the team completes 20 points of work each iteration.
Without the proper visibility to the information feeding this chart, it appears that the product manager is growing the backlog. However, if you were to decompose the velocity into its constituent points (illustrated below), you see that in the second scenario the contribution of defects to the team’s velocity grows throughout the release cycle. The assumption that the effort put toward defects will grow is reasonable; if there are defects being introduced early on in the release and nothing changes to address the root cause, the defects will continue to be introduced and their contribution will become more and more significant. Without transparency to what made up the velocity, the team can feel that they have executed with their ‘expected velocity’ and the product owner can feel like she is getting asked to make unreasonable concessions due to quality problems. Do you think it is reasonable that the product owner would remove scope because the quality of earlier deliverables was not high?
This view shows the points that were dedicated to stories in the backlog compared to points assigned to defects.
So, what to do?
I do agree that bugs from prior releases should be sized by the team and prioritized by the product owner, much like a new story. In that case, defects from prior releases would be included in the velocity, and the product owner could determine what new features could be completed if work toward resolving legacy issues was suspended.
If defects introduced in earlier iterations are going to be fixed in subsequent iterations, their contribution to the team’s velocity should be made big and visible. Only if the information is visible will it get the proper attention so the team and product management can work to figure out the root cause of the defects. If the team is not going to make the effort put toward resolving defects visible, I would suggest that it would be best to not size the defects and to merely let the ongoing reduction in team velocity be the canary in the coal mine. Unlike the scenario that Mike spoke about, the team cannot reasonably suspend defect resolution to deliver additional features.
Lastly, if the team’s anticipated velocity includes an allocation to fixing defects that are yet to be introduced, this needs to be accounted for as part of release planning, as well.
Ultimately, it is critical to identify the root cause of the defects and to address the root cause. Without addressing the root cause, we are left with an ever-increasing allocation of time to resolving defects that we are introducing as we build new features.
Conclusion: Sizing defects that are introduced as part of the release that you are currently working on can produce a misleading velocity. If you do want to size defects and have them contribute toward the velocity, segregate the points from Defects and Stories so that the effort toward each becomes plainly visible.
Footnote:
*For the sake of illustration, I have simplified the scenario. Indeed, there is often some variability in velocity from one iteration to the next. This scenario also assumes that the product backlog is constant throughout the five iterations.
Another way of stating this might be that teams are “double dipping” as far as getting credit for a story. If a 5 point story is deemed ready in iteration 1 but requires 2 points of defect fixes in iteration 2, the team should not get 7 points of credit. More accurately, the team actually did NOT complete that 5 point story in the first iteration and in reality their velocity in iteration 1 should have been 5 points lower.
Matt,
I don’t think getting credit matters. This is in the same category as trying to game the velocity numbers — it’s a losing proposition. If I purposefully overestimate a story to make my velocity number look better, I’m going to be hurting next iteration when I will be expected to sign up for yesterday’s weather.
Likewise, if I put points on something that shouldn’t have points, be it defects or overhead or meetings, in order to get some extra credit for stuff I do, then I’m forever going to be looking for that extra point in order to keep my velocity ever increasing. Most developers aren’t after such credit and if they are there is likely an unhealthy focus on increasing the velocity metric in the organization and they will soon realize the futility of it.
I see value in your argument that the team in your example did not truly complete that 5 point story, but logistically, there is no point in revising iteration 1’s velocity an iteration or two later when you discover the defect (unless you are using an average velocity from multiple iterations).
Matt, your point is valid. A situation like that points to some difficulty in knowing that a story is done with high quality, and not declaring victory before the work is really complete.
I don’t buy that “The assumption that the effort put toward defects will grow is reasonable…” I contend that a process introduces errors at a certain rate which is generally proportional to the velocity. If you do fix them as fast as they come in (just an iteration or two later), the effort put toward fixing them will remain generally level with the level rate at which they are being introduced.
Is that the case in your team? Is the team spending a greater and greater amount of each iteration fixing defects? Are they introducing a greater number of defects in each iteration than in the iteration before?
Andrew thanks for commenting.
The premise of that comment is that if we don’t understand the root cause of the defect (e.g., misunderstood test, met an unexpected condition, impact from another feature, etc.), and do something to address the root cause, it is going to be difficult to increase the quality of our code. In Software craftsmanship: the new imperative, Pete McBreen references the “old programmers’ drinking song, ‘Ninety-nine bugs in a program, fix one bug, now there are 100 bugs in the program.'” Of course, individual experience will vary.
I would suggest that if there are enough defects that there is a discussion about sizing them, there is probably cause to dig in on root causes. If the rate of introduction were such that the defect list is cleared each iteration, estimating them becomes waste and I would suggest again that perhaps the velocity without defects having points is the best measure of the team’s progress.
Right — if a team could clear the defect list each iteration, then estimating them is indeed waste, waste, waste.
Dan,
Why should the pointing / not-pointing of defects be any different just because you crossed a release boundary? What if we just finished iteration 13, released the product, found a bug in the field while we were working on iteration 14, entered the bug in the backlog, and know that we need to fix it before this release ships, but haven’t decided when to fix it yet? Would you point it? Based on your post, I think you would. But if you shifted all that right one iteration (introduced the defect in I14), you wouldn’t point it.
I prefer the policy of fixing defects as soon as you find them, regardless of whether they were introduced in the current iteration’s work, last iteration’s work, or last release’s work. But there are times when defects can’t be fixed in the current iteration: it’s too close to the end of the iteration, the fix is too hard to do right now, there is too much other WIP for committed work in the iteration, there were too many new defects all of the sudden (a very rare occurrence) due to some rare situation, etc.
In this situation, when putting defects in the backlog happens but is not the preferred approach (dare I say not the normal approach), then there is a known quantity of identifiable work in the backlog that should not be growing, at least not steadily. In that situation, it’s just work to me, just like defects left over from prior releases. And in that situation, I want to know how big the full backlog is (defects and stories) and how long it will take to knock it out given our current velocity.
I hope you don’t mind all my replies, but I love this topic and I think it’s very important.
Let me give an illustration: Suppose we have the “fix it right away” / “zero defect” policy that I describe in my previous reply. And suppose that in, say, iteration 7 something goes haywire and we discover 40 defects. We try our darnedest to clean them up in iteration 7 and even some in iteration 8. Our velocities take a hit because we didn’t estimate that stuff because of our fix it right away policy. But we’re left with a wad of defects still that need to be retested to see if they are duplicates or already fixed or not recreatable or just plain invalid — or genuine problems still in the product. Time is marching on while that analysis is done. Next thing you know these defects are aging in the backlog. If we would agree to stick every last one of them in the current iteration and knock them out, then don’t point them. That’s the policy. But if, for whatever reason, if we’re going to do none in this iteration, some next iteration, some more not until after the soft launch, and I don’t know when for the rest, then in that case I want to estimate that work.
Great discussion here. Lots of good thinking going on.
@Dan Neumann
“Do you think it is reasonable that the product owner would remove scope because the quality of earlier deliverables was not high?”
I don’t think it’s a question of negotiating scope. The purpose of tracking velocity is to provide a leading indicator of either (a) the approximate date when the known scope is likely to be completed, or (b) the approximate amount of scope that can be delivered by a given date. It’s an empirical observation of reality, in the nature of “the sun rises in the east.” You can’t make the sun rise in the west just because it’s “in plan.”
It’s good that the product owner has an early warning that things aren’t progressing as well as everyone expected. The corrective action should be based on business considerations. (Of course, that assumes “velocity” is actually being used as intended.)
“…it is critical to identify the root cause of the defects and to address the root cause. Without addressing the root cause, we are left with an ever-increasing allocation of time to resolving defects…”
I agree with the first part of that statement, and I see the second part as a red flag.
I think it’s useful to track scope due to recently-introduced defects because it provides information to support continuous improvement efforts. If a team is using techniques such as iterations and measures such as velocity, it is usually the case that the team is also using (or /trying/ to use) other techniques commonly associated with “agile” development. Some of those techniques – notably TDD/BDD, pairing, automated testing at multiple levels of abstraction, continuous integration, clear definition of done, and frequent customer feedback – tend to minimize the number of new defects as the product evolves. If instead the defect rate climbs as the codebase grows, then the situation suggests there might be room for improvement in the team’s practices.
With that in mind, I think that allocating time to fix the “inevitable” defects is a process smell. It’s a workaround and not a solution. The solution is to minimize defect production. Some questions that come immediately to mind are:
1. Is the definition of “done” adequate?
2. How are stories reviewed before being deemed “accepted”?
3. Are the defects “discovered requirements” (good: one of the benefits of the agile approach), “mis-communicated requirements” (bad: needs improvement), or actual “bugs” – programming errors (bad: WTF? Using TDD, pairing, etc.? Probably not!).
4. What is the state of the automated test suite? Quality of unit tests, functional tests, integration tests, etc.? Edge cases adequately covered? Test cases yielding false positives? Other issues?
@Andrew Fuqua
“…teams are “double dipping” as far as getting credit for a story.”
This is a process smell. It causes me to ask how management measures employees’ work. If team members are measured on how many points they deliver, then they have an incentive to game the velocity metric. It’s possible the project manager and possibly other managers in the hierarchy need coaching.
“I prefer the policy of fixing defects as soon as you find them, regardless of whether they were introduced in the current iteration’s work, last iteration’s work, or last release’s work.”
In my experience, this policy makes sense in some circumstances and not in others. In the typical “agile” project to build a business application, it’s often the case that the product owner wants to make a tactical choice to leave a minor defect alone in order to get a valuable new feature delivered earlier. It is a business decision.