Overconfidence to accuracy

(written by Lawrence Krubner, however indented passages are often quotes)

Taryn East writes:

An eye-opening article by Robin Hanson, called Who Cares About Forecast Accuracy?, describes how and why corporations spend heaps of money on making predictions… but almost none on testing whether those predictions actually worked out.

The article discusses the fact that actually recording the accuracy of pie-in-the-sky forecasts means that it’s much harder to hide a poor track-record for leadership. That a lot of forecasting is more about signaling affiliation to the company (“yeah, we’re doing great, I predict we’ll be done by tomorrow!”) than actual accuracy. Quite possibly just another effect of bosses preferring overconfidence to accuracy.

From my own experience – I find that many “leaders” don’t want you to say that the project might be in trouble… and the article discusses how this can happen because the middle-managers don’t want somebody to go on record as having questioned their vision… because it’ll reflect badly on them down the line should something go wrong. It’s much harder to sweep losses under the carpet if nobody officially takes notice of how often it happens.

I think there’s also another reason, not mentioned in the article. That being the long-term vs short-term payoffs. The strategic echelons of an organisation need your predictions so that they can make their own overall predictions of how long a plan will take to implement. But they don’t want to bother with the long-term cost of implementing a full system with all the process-change and downtime that it will take. I’ve personally found it extremely difficult to “sell” both management and fellow-programmers on changing to to a methodology which will allow for recording and feedback of prediction accuracy. These are methodologies that aren’t particularly heavy or difficult to implement (compared, say, with six-sigma), but which provide a quick and easy way to get better at providing an accurate estimate of your times.

The problem seems to be, that it takes more time and effort to implement, and people don’t want to do the “work”. They want to make their guesses *and* make t more accurate… but not to have it impact the “bottom ilne” in any way. This is a bit like saying “we’d like to buy faster computers, but won’t give you any budget for their increased cost”. You somehow have to just improve anyway.

What normally ends up happening is that you don’t properly implement any system of accuracy, so you end up simply guessing and just hope for the best. Actually spending the effort to make those guesses meaningful is too much like hard work, and nobody wants to add to their workload. This is understandable… but not exactly helpful (or realistic). Instead of improving over time, we continually wallow in a world of wild guesstimates and “fudge-factors” that never seem to quite cover the extra time it always seems to take.

The follow up work, keeping track of how accurate a prediction was, is suppose to be one of the things that Toyota is good at. I’ve read that a difference between Toyota and General Motors is that at General Motors, a manager needs to explain why a given improvement produced less of a gain than was predicted, whereas those improvements that produce more gain than was predicted are treated as successes, but at Toyota they see it as a problem either way: if a given improvement produced more or less gain than was predicted, they investigate to find out why it departed from the prediction.

If we restrain the conversation to just software, I understand that Pivotal Tracker is pretty good about getting people to make estimates and then seeing how accurate those estimates were.

In general, I think the problem is one of accountability. At least at the companies where I’ve worked, top managers are never held accountable to whatever predictions they might make. There is a vague accountability — the department should do well or you are fired. But that is punitive and does not develop the internal intelligence of the processes that the company uses to move forward.