Software estimation for great fun and profit

Have you seen a well-oiled agile team go about their work? It is a pleasure to watch. They have a rhythm for software delivery ‐ the programmers work in a sustainable pace and they meet business goals. I think this is because agile project management does two things right: story points for relative sizing and predicting future performance by using past averages (velocity and volatility). These two methods account for the fact that individual estimates are fallible and work around it.

But not all teams are agile. One reason is that customers don't understand how unpredictable software development can be.

Thus begins a long conversation, the first, in a series of long conversations.

The other reason for projects to not use common-sense project management is that they simply can't. They may be in constant flux to allow any rhythm to take hold and their timelines too short to even start calculating a reliable average velocity. Real world, things hairy.

We have to up our game to work well in those environments. Without the luxury of sensible agile practices, we will have to get better at estimating individual tasks. The challenge then is to deliver software predictably without a working process to do so. It is an interesting challenge, and can even come with a few benefits. In this post, I'll argue that done right, estimating individual tasks well is a great productivity booster and makes software development a bit more fun.

But first, we have to understand what we are up against.

Estimates are commitments to ourselves

Is there a wall-poster which says estimates are not commitments? We should get one. We should read it in the mornings and evenings, before food and after food. Because the first casualties of an estimate are us, the developers.

You are working on a story that you had estimated to be a 1-pointer. It typically takes you a day to finish a point, but this one is taking longer. Anxiety creeps in, but you don't recognize it yet: this story is not supposed to take this long, ergo, I'm too slow.

We become wedded to the estimate, an arbitrary number ‐ a guess made from incomplete data, and it worries us when reality doesn't match up. This is "anchoring".

The anxiety induced by anchoring is not an articulated thought. It is a nagging thread in the back of the mind chipping into the joy of our work.

But it is a silly game that the mind plays and we can eradicate it by simply being aware of it. That is both easy and hard. Easy because once recognized, the worry is disposed off easily, and hard because mindfulness require an awful lot of practice.

We could have produced an accurate estimate in the first place and avoided this anxiety. But before we get to that, we have to look at what estimates do to our customers.

Estimates are commitments to our customers

Let me illustrate. You are asked for an estimate on when a big feature can be released, and the following happens:

The moral of the story is that no matter how many ifs, buts and other caveats we insert, the customer will treat our estimates with an inappropriately high level of certainty.

There are a few things we could have improved on though. At least next time, we'll:

• Re-iterate that estimates are not commitments.
• Ask what our estimates are going to be used for. We should always know the full implication of the numbers we provide.
• Be careful to not amp up expectation in others because today's good weather does not translate into tomorrow's. But we respect our business owners and understand that they are capable of dealing with reality. We keep them aware of slips in the plan as they happen. We remain stoic in the face of both progress and setback.
• Not work on weekends. We worked on weekends, and it didn't help. It never does, in the long run. Don't work on weekends, and don't give optimistic estimates thinking you would.
• Relax and reflect. When the going gets tough, we forget to think. The stress creates adrenaline, and it clouds our judgement. The datatable filtering was not a show-stopper. It could have waited and we could have shipped on time, only if we had the presence of mind to consider the possibility.

Our estimates are going to come back and bite us ‐ that is the nature of the beast ‐ we ourselves become bound to it through anchoring and our customers become bound to it because they need a number to plan things around.

Casual estimates

Casual conversations are the minefield when it comes to dangerously wrong estimates:

Stop right there and take note:

• The customer did your job for you and supplied an estimate in the question itself. In fact they started from the lowest possible number, putting pressure on you to anchor on it. That is negotiation. Estimation should never be a negotiation.
• We thought this was a casual question and gave a casual (aka wrong) answer. But when you get around to actually estimating this, telling the customer that the earlier estimate was wrong is akin to backtracking on a commitment. It wouldn't matter that it was a ballpark number, that detail will be forgotten by both the customer and you. Casual promises are promises nonetheless, so don't make them.
• Have conversations based on time instead of story points. Story points are a useful abstraction, try to stick to it if you can. (Why use story points instead of hours for estimating).

Now the same conversation would go like this:

This is obvious but overlooked ‐ the accuracy of an estimate is proportional to the time & effort you spend on it. The more you invest, the better the estimate. And no matter how quick and approximate an estimate your customer needs, you cannot afford to be lazy and put out a number that you don't have maximum confidence in.

Estimation is a high-risk activity and should be treated with the gravity it deserves. Casual estimation underplays the risk involved and the time it takes to come up with a reliable estimate. It makes us rush through and produce unreliable numbers which ultimately leaves everyone worse off.

We should explicitly allocate time for estimation. This conveys the time & cognitive effort required to do it and ensures that stakeholders think through what they really need before they ask us to spend time on estimating them.

Now the question becomes what do we do with all that time we allocated for estimation? To answer which, I have to first hate on fuzziness.

Fuzziness

Fuzziness is a knot in the stomach. I feel it when I estimate a story without thinking it through. It is my body telling me that I'm painting myself into a corner. But fuzzy is easy; we see a story and put a number to it without expending the mental energy to articulate it.

We say this story is a 1-pointer or a 2-pointer based on a hazy intuitive notion. This is fuzzy, it doesn't work well when you have a short-term delivery horizon. We tend to underestimate the task at hand and overestimate our abilities, and no matter how many times we have been wrong, we fool ourselves that we can do better this time. We are also people pleasers, we hate giving bad news and tend to default on the most optimistic numbers.

Getting better at estimation involves replacing fuzzy estimates with concrete data. It is hard to defend estimates made based on gut feeling, but data can speak for itself. Data doesn't people-please ‐ it gives us the conviction needed to communicate well and set the right expectations with our customers.

We defeat fuzziness by concretizing, aka, planning a task to its actionable items.

Plan, don't estimate.

We have to know what we are estimating to be able to estimate it. Sounds obvious? It is not. We often don't know the specifics of a story when we estimate it; we simply do a high-level sizing and throw a relative number. We keep aside the figuring out of what needs to be done for the later implementation stage.

The reason we default to such fuzzy estimation has a lot to do with the economics of energy of the prefrontal cortex. This is the part of our brains that is responsible for decision making, prioritization and deliberate thought.

Deliberate thinking is a formidable productivity booster (duh!). But generating thought is energy intensive. So we resist it and rely on gut feeling, which is simply the brain cruising on auto-pilot. It is not an effective strategy. Here, look at what some upfront planning did to Rachel Aaron, a writer (emphasis hers):

She managed to increase her daily output from 2000 words to 10,000 words a day, mostly thanks to planning. Programming is also in its basic form, just writing. There are many lessons to be learnt from that craft, and this one is a case in point. Instead of wallowing in the code for hours ‐ going where the code takes us, ending up in blind alleys and backtracking, and getting lost in the trees and losing sight of the forest, a little bit of thought & planning can do wonders for our days as programmers.

The earlier we plan, the better. In fact, going into as much detail of a story as we possibly can before we estimate is the simplest way to increase the accuracy of our estimates. It will also bring out any hidden assumptions before we give out invalid estimates, or worse still, get neck deep into implementing the wrong thing.

Implementation becomes a breeze once we have a solid plan to follow. It is a beautiful day of writing code when that happens.

Estimate as a by-product of a plan

So what has planning got to do with estimation? It obviously makes for a more enjoyable programming experience. But it also brings our estimates to ninja precision. That is the whole point of this post.

Estimate is a by-product of a plan. It can't help but jump out at us when we look at a well thought out, written plan. At a sufficiently fine level of granularity, we would have the very implementation steps in the plan, and to come up with an accurate estimate, all we need to answer is how hard is it to convert this into code.

Drilling into this level of granularity might not be practical in many large agile environments. There will be too many people and stories for us to go into depth for each of them. But remember we are not talking about those environments. They have their own strategies which we cannot afford to have.

However, it is easy to make mistakes even in the relative sizing of stories as is done in a pure agile practice. Some of the stories turn out to be much larger than we anticipated. That is fine, the process accounts for such errors. But the feedback that the story was not estimated correctly is a lot more useful when we have a plan backing the estimate. Without a plan, the estimate is a one dimensional number, and that is all we are left with to evaluate any feedback on. But with a plan, the feedback is more useful ‐ it is easier to find out what went wrong in our assumptions and account for that when estimating similar stories in the future.

To conclude, plan early, plan often, plan deep. And good things will follow.