Sign up for a discount for the webinar “How to Measure Anything in Project Management” presented in person by Doug Hubbard. Just take a project management survey, and you will also get the summary report of the survey findings. See details at the end of this article.
Of all the project-management-related measurements a firm could make, there are three that are particularly critical and yet almost never measured. Each of these are measurements my team have made many times across many types of projects even though some might consider these to be “immeasurable.”
None of these measurements are new. Mature quantitative methods exist and have been applied to for each of these measurement problems. Also, each of these methods can be done using nothing more than the statistical functions available in Excel spreadsheets and the methods are simple enough that we cover them in a training course that takes one day.
This is not a comprehensive list of commonly ignored, critical metrics in project management (we cover more in our training) but I think these three are particularly interesting. They are: the chance of cancellation, the total cost of scope creep and the performance if project management itself.
The Chance of Cancellation
Project management is, to a significant degree, project cancellation management. By cancellation I don’t just mean project “failure.” Failure covers a wider variety of outcomes including not finishing on time or on budget or not meeting critical objectives. Cancellation is a much more conclusive type of failure. It refers only to the outcome where time, effort and money was spent on a project but then the work was terminated with nothing to show for it.
Almost everyone who has been in project management for a while has seen projects get canceled but almost nobody sees “chance of cancellation” included in a business case of a project along with costs and benefits. Estimates of average project cancellation reported in some studies (10% to 20%) are high enough to constitute a major risk. Software projects taking two years or longer have cancellation rates closer to 30%. Bonds that have default rates that high are considered high-risk junk bonds. Some more pessimistic studies (with varying degrees of credibility) estimate the chance of project cancellation in software project to be even higher. Taking this risk into account can have a significant effect on project approval and even how a project is planned.
For these reasons, the value of measuring the chance of cancellation is high. When my team conducts a probabilistic analysis of risks in projects, we calculate the “Value of Information” on every uncertain variable using well-known methods from Decision Theory. We find that measuring the chance of cancellation is both the highest value measurement (on average) and the least measured. This is one of the clearest examples of what we call the Measurement Inversion – the tendency to spend little or no time measuring the most critical variables while spending much time measuring variables with little chance of improving a decision.
Some of the factors that indicate project cancellation can be influenced by the project manager and some cannot. One key factor that can be influenced is project duration. There is almost a linear relationship between the chance of a project being canceled and its duration. Each month of duration of a software project, for example, increases the chance of cancellation by 1.2%. Beyond two years, each additional month increases the chance of cancellation by 0.5%. This is an average over many firms and industries – this risk may be lower or higher in individual firms. We also know that about half of projects which are canceled are terminated after they exceeded their initial estimated budget. So, when project managers are doing what they can to finish on time, they are also helping to reduce the chance of cancellation.
Of course, factors which are seemingly outside of the control of project management have bearing on cancellations. The actions of competitors, changes in regulations, discovery of new technology, or a change in the economy which reduces demand for a particular product could all make the value of a current project go negative and may force the termination of a project. However, even though the project manager doesn’t directly control these factors, tracking just some of the key exogenous risks can allow a firm to change the direction of a project in an intelligent manner. A project scope can be reduced to make delivery faster in order to experience some benefits – even if only temporary – or to focus on different features that now have a higher return.
Finally, if a project must be cancelled, then at least it can be canceled early to minimize losses. This requires that some regular tracking of the original business case should be conducted so that the effort can be terminated soon after an analysis shows the benefits are unlikely to justify the costs. Pharmaceutical development projects, for example, have about an 8% chance of making it through to approval by the FDA but many drug companies have a formal “stage gate” process for discontinuing a new drug early when it looks like the chance of success will not justify further effort. Many other types of projects, however, will go on for weeks, months, or even years past the point when analysis would have shown that the project benefits no longer justified the effort. It might seem defeatist to actively monitor the conditions under which a project should be canceled. But it is actually the healthy alternative to letting unprofitable projects suck up the air from what could be higher-payoff efforts.
The Cost of Scope Creep
When the business requests that a new feature be added to software or a new factory design, the cost of that change is often not fully communicated. At best, the cost of that change in near-term project effort will be estimated but in some types of projects, like software development, this may not be communicated at all. Even if this cost of the change is estimated, it may be excluding other major costs of increasing scope.
We once estimated the cost of scope creep from a newly requested feature that would have extended the delivery date of a major software project by a month. The additional development and testing effort made up only about one quarter of the total cost of the scope creep. Another quarter of the total cost of the scope creep came from increases in future maintenance effort. As many software development professionals know, there is a non-linear relationship between the size of a software project and future maintenance – e.g., doubling the lines of code may more than triple future maintenance costs. Almost all of the remaining costs of adding this new feature – about half of the total – were the deferred benefits of all of the other features of the software. That is, the other previously-planned features of this software had benefits of their own and those benefits would be deferred by a month.
In this particular case, the total cost of adding that new, previously-unanticipated feature was about four times the immediate development costs of adding the new feature. Perhaps one of the reasons scope creep is such an issue is because the temptation to add new features mid-project are not informed by a full accounting of the costs of adding that feature. Whenever the full costs of any action are not known, decisions will be made less economically. Wasteful behaviors will continue because a cost is “hidden” to the decision maker. Perhaps when the true cost of scope creep is presented, decisions to increase scope will be less frequent.
The Effectiveness of The Project Management Method Itself
If you are counting on a methodology or a standardized “body of knowledge” to help you with project management, then you need to ask “How do I know this works?” This is an important question because other research I cite in the books shows the existence of a kind of “analysis placebo effect.” That is, the use of some “formal” method may make you feel more confident in decisions and estimates even though performance does not improve – and even when the method decreases performance.
This measurement may seem hard but in many cases it has already been done for you. This is just a matter of understanding the available research on certain key components of methods used in project management. I have reviewed this research in a fair amount of detail in my books and it is clear that some methods in estimation and decision-making measurably outperform others. Unfortunately, influential sources of project risk management “best practices” still promote methods which have shown no measurable benefit – especially in regards to risk analysis. For example, research about the use of ordinal scales (such as those used in various “risk scores”), risk matrices (also called probability and impact matrices in project management) shows that these methods are not only an improvement but may do more harm than good.
The good news is that there is also research showing that some methods consistently outperform the unaided intuition of experts in a variety of estimation and decision-making tasks. Methods based on statistical models and simulations are shown to outperform intuition in many areas that apply to project management such as estimating cost, duration, and the probabilities of various events. This doesn’t mean the expert is entirely replaced. On the contrary, research also shows that experts can be trained to overcome certain biases and inconsistencies in their judgment and decision making. In this way, the expert becomes a type of “calibrated measurement instrument” where we have measured the performance of the expert themselves in making estimates and decisions of certain types. This allows the experts to improve so that we can use their judgement directly in quantitative decision models.
As I stated at the beginning, this is by no means a complete list. There are other measurements in project management which – while nearly always ignored – could greatly improve decisions and the outcomes of projects. Most project managers and their firms would benefit greatly from learning about other less-obvious measurements and the statistical methods behind estimating them.
For a 20% discount on any “How to Measure Anything in Project Management” webinar, just share this post on LinkedIn, and take a survey here. Once you take the survey and share this post we will give you a coupon code for 20% discount on the cost of the seminar. You will also receive the summary report of the findings from the survey.
