Software Estimation is a Losing Game

https://rclayton.silvrback.com/software-estimation-is-a-losing-game

This is an argument, and like all arguments, it's supports a specific position. I don't try to defend software estimation; I leave it to my colleagues to present that case. I write about this issue because I believe the software community needs to have an internal debate about whether software estimation in its current form is appropriate for projects.

Software estimation is the process of predicting the time and effort required to accomplish development or maintenance tasks. Estimates are generally used to help customers (this could be your own organization) make decisions. Customers often want a general idea of how long it will take to accomplish tasks and since the customer is always right, we engineers have always agreed to participate in a practice that we frankly despise. We estimate because it's easier to sell a software solution with a fixed delivery time than to actually admit that the customer is simply going to get whatever product your team can complete in the allotted time. The estimate becomes our commitment to delivering a piece of software under some arbitrary deadline.
Sadly, we commit to these deadlines, despite the likelihood of missing them, and in full acknowledgement of the consequences of missing our estimates. These consequences include, but are not limited to:

• Loss of credibility with your customer and possibly your own team.
• Internal team friction, especially when management over-promises on a delivery.
• Poor morale, particularly when the team has to work excessive hours to meet a delivery.
• Low quality software if developers rush to meet deadlines and skip quality control practices like testing.
• Project Failure when estimates are consistently off and the customer has completely lost faith in your team's ability to deliver.

Over the last six years, I've participated in a number of software projects where poor software estimation has led to very bad outcomes. As a result, I've spent many hours pondering, and discussing with friends, strategies that would have led these projects to greater success. Given the difficulty of estimating effort on a software project, not to mention the near-certain guarantee of getting it wrong, I've come to a radical question...

Do we need software estimation?

I've posed this question a couple of times to project managers and corporate types, particularly when I'm asked to provide some crazy estimate for a customer. "Well of course we do!" is typically the answer I get back. "A customer's not going to give us a contract without some guarantee of delivery." And the sad thing is, this response is fair. It's fair, because we have allowed it to be fair, even though we know this is not how software projects work.
In reality, software teams can only provide a set of capabilities proportional to the amount of time and effort we are allotted. In that sense, we are very much like the hospitality industry...we can't guarantee a guest they will be fully refreshed when they leave our resort, but we can assert that the longer the guest stays, the more likely they will achieve this outcome.

The cult of software estimation.

Despite our inherent understanding of the difficulties of software estimation, we continue to believe that it is the correct course of action for helping us plan and manage projects. As an industry, we've invented a host of methodologies to help us manage the uncertainty of estimation (planning poker, t-shirt sizes, etc.) and written countless books on the topic. In fact, many people have made software estimation itself a career, coaching teams and project managers on the art of predicting uncertainty. Yet, we are still just as incompetent at the process as we were three decades ago (maybe even more so).
The belief that software estimation is critical is so pervasive that it has been ingrained in project management philosophies like Agile, giving the impression that they are inseparable. This highlights what I call the “cult of software estimation”, which is literally that the belief that the importance of software estimation is above reproach [1].

The uncertainty of software estimation.

Software estimation is not an inherently bad process. The problem is, we're really bad at it. It's literally the equivalent of making a weather forecast. You wouldn't commit a forecaster to a prediction for rain a month from now (well, if you do, you're naive). Like any prediction, it's only really accurate, and useful, when making near-term forecasts.
There are a number of factors that lead to skewed software estimates. From my own experience, I've seen all of the following (but I'm sure there's more):

• Tasks not granularly specified - a task is too ambiguous leading to a lot of thrashing on how it should be implemented.
• Task creates new tasks for the same sprint - common with research tasks whose outcomes include some sort of solution to be built in the sprint.
• Estimates made for unfamiliar technologies - trap for developers who inaccurately estimate the time to completion based on past experience with disparate platforms.
• Working on unrelated tasks - i.e. context-switching; the tasks assigned to a developer are unrelated and force them to "spin up" for each one.
• Tasks with in-sprint dependencies - developers become immediately blocked because precursor tasks are unfinished. This is especially problematic if the dependencies need to be finished by multiple people.
• Research tasks with non-tangible outcomes - I've seen people take weeks to research/prototype some solution. If you don't constrain ambiguous tasks, you're inviting the task-taker to waste time.
• One person estimating for another - common when developer is absent during planning and another does the estimation for them.
• Peer pressure - engineer(s) press another to accomplish a task faster than they believe they can. This generally happens in high-stress environments when a stronger personality bullies a developer into working harder.
• Poor morale: teams under a lot of pressure tend to deliver slower.
• External pressure: pressure by management to estimate less/more time-to-completion because of political and financial agenda.

None of these factors should surprise you. In fact, our software estimation processes are designed to deal with them. Our solution is to apply psychology and mathematical models to develop better estimates. In fact, I feel like we waste more effort (per capita) than any other industry analyzing our own development practices seeking better efficiencies.

The resource cost of software estimation.

When you think about it, even for small teams, there is generally someone dedicated to studying things like "burn down" or normalizing software estimates by person, over time. Traditionally, software development teams are really small (2-10 people). So when you think about all the time and effort that one person has dedicated in studying the work habits of their team, it's possible you're losing a significant percentage of your resources to overhead.
More importantly, creating software estimates tends to be a group activity, typically occurring at the beginning of a sprint during some form of planning meeting. If you are dealing with a fairly complex system, or have a lot of people on the team, you'll spend anywhere from 2-8 hours planning a sprint. Consider the fact that you may be wasting entire man-days (number of people estimating * time spent estimating) of development time just coming up with estimates.
When you think about it, that's a lot of time and effort lost to come up with estimates that are going to generally be wrong.

What do we gain from software estimation?

• A better appreciation for how much a team can accomplish in a set amount of time.
• A roadmap that can be used to determine the time-to-completion of a project.
• A nice dashboard with charts that can show individual developer accountability to management.

What do we lose from software estimation?

• If the estimation is bad, the ability to maintain a realistic pace of development. You’re either burning team out or looking stupid because you finished too early.
• Flexibility in planning and execution, particularly if you are planning too far out and commit to a set of tasks. What happens when your approach changes and you are going back to the customer to explain a task that was planned to take two weeks will now take 4 weeks?
• Lost resources if you don’t have a significantly long project, the overhead of having someone study and normalize estimations will burden you.
• Efficiency, particularly when estimation is combined with time-boxed development cycles like a sprint.

Software Estimation and Time-Boxed Development Cycles.

I would like to expand that last bullet. We typically use software estimation as a tool to help us plan our sprints. How else are we to know how many tasks we should take on in a given period of time? I contend that this practice really just promotes a host of inefficiencies in your development process:

• Lost effort due to context-switching - Engineers only take as many tasks that fit within their window (say 2 weeks). This usually means the developer takes a couple of large tasks and several unrelated ones (hell they may all be unrelated). Working on unrelated tasks causes context-switching, which reduces overall efficiency since the developer has to “retool” for the new tasks.
• Lost time due to fragmentation - Just like blocks in a hard drive, you can’t perfectly fit tasks into time slots (3-hour task into a half-day slot). There’s always going to be some amount of lost time. Of course, you could always reduce the “block size”, but is that even realistic (i.e. planning tasks down to the minute!)?
• Discourages maximum productivity - What happens when you accomplish all of your tasks in a sprint? Some engineers may choose to pull more tasks off the backlog; others might gold plate. I’ll tell you what I might do…take a nap. Especially, if I just spent the last four months working my ass off to get a system up and running!

Why estimate?

So I ask again, why should we estimate? The only benefits rendered to our managers and customers is a prediction that really can't be trusted. The consequences, however, tend to range from reduced development efficiency, morale issues, low quality software, and potential loss of work.

What is the alternative to not estimating?

I don't have an exact methodology. Maybe it's something like Kanban, though I don't know enough about Kanban to say one way or the other. I can tell you what this paradigm might look like.
Let's call it, Budget-Driven Development, or BuDD for short. In BuDD, development teams are given a set of resources and develop what they can until the budget is gone. I’m sure I just gave someone a heart attack, but when you think about it, it’s not too unreasonable. Many contracts have fixed time and costs. You make a guarantee that the customer will get a specific number of man-hours of development. Given a prioritized backlog, developers work on the three most important items. I imagine it would work similar Pandora's prioritization model, but not be constrained by time.
You guarantee quality to your customer by:

1. Delivering capability often.
2. Allowing the customer to prioritize development.

Isn't that just Agile? - Yes! Without arbitrary estimates!
The benefit to the customer is that you’re making a guarantee that you will (hopefully) maximize their value by not delivering early and pocketing the rest of their investment. Your team will also discouraged from remaining idle because it will be immediately apparent by the substance of your deliveries.
Not estimating does not mean your are not managing your team. In fact, effective management (the kind that should be happening in the first place) becomes more important:

1. Team is constantly collaborating. Not estimating does not negate planning meetings, more sync’s or scrums, etc.
2. Backlog is managed. This includes managing dependencies and ordering and prioritizing tasks, etc.
3. Tasks are sized (as opposed to estimating delivery time). You should still be able to say, "this task is too big for one delivery."
4. Development cycles are scoped to tangible deliveries and you should favor smaller deliveries over larger ones.

Hopefully, I have convinced you (on some level) to rethink the use of software estimation. The next section discusses principals you can employ if you decide to move from estimation.

Moving towards a no-estimate methodology.

Whatever strategy you choose to employ to manage your team, I recommend at least adopting the following principals for managing your team:

• Prefer granular specifications - The more granular the specification, the more accurate you can gauge the scope of a task. This means, don't settle on stories as a unit of work. Instead, break that story down into technical tasks. Use schema and contract specification technologies that explicitly define interfaces. Draw mockups. Essentially, remove the ambiguity from the task.
• Choose task-oriented development cycles instead of time-based ones. This strategy minimizes context-switching and time fragmentation.
• Group related tasks; prevent context switching and attempt to maximize developer flow.
• Opt for smaller development cycles, particularly if more flexibility is needed. More importantly, this allows you to demonstrate your work more often and receive feedback.
• Stagger your team’s responsibilities and efforts. Have the tech-lead/architect specify tasks a sprint ahead of when they are planned to be done. Test and QA should happen after a development cycle. Don't try to cram dependent tasks into the same cycle.
• Choose the three most important tasks and work on them until completion. Don't waste your time with filler tasks just because you need to fill a time slot.
• Stop and regroup - don’t be afraid to stop the sprint if you feel it's necessary. This might occur if you feel tasks aren’t specified well enough or if developers are thrashing (not getting work done efficiently). Take time to organize the team and then carry on.
• Pivot - if things aren't working, or you feel like the team needs to go another direction, do so. The worst thing you can do in Agile development is not be agile.

Conclusion.

Whether you buy the argument that software estimation is unnecessary, or are still an ardently supporting the estimation camp, I think we can agree on a couple of ideas:

1. Software estimation is difficult.
2. Many teams fail to accurately make estimates.
3. Trying to meet unrealistic estimates can destroy a team's morale.
4. Failing to meet an estimate hurts a team's credibility.

The question is, if this is such a big problem, what should we do about it?

Comments, Notes, and Further Explanation

1. While you read this, you are probably thinking that I believe software estimation seriously flawed. I actually think software estimation can work. But to do it right, it takes a highly-cohesive team working in a well-known problem domain, with significant experience in the technologies being used. Even then, there's still the potential for estimation failure. For instance, think about the effects of holidays, vacation, and illness. What about significant weather or internal or external political issues. This might not happen often, but it can still cause problems. What I really believe is that it only works for about 5% of teams.

Richard Clayton –

Unrepentant Thoughts on Software and Management.