by Douglas Hubbard | Feb 18, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Wednesday, February 18, 2009 4:54:25 PM, by RossShott.
“Hello Mr. Hubbard,
Your book “How to Measure Anything” is remarkable!
I am currently completing my Master of Technology in Futures Studies degree by working on my thesis project. The project entails fleshing out a Foresight Maturity Model (similar to a CMM or CMMI) recently developed by Social Technologies. Meaning that I have been given the task of figuring out how to measure foresight within companies in such a way that those companies can be evaluated by a Futurist consulting firm using the FMM tool or the company could apply the FMM tool to their own organization for foresight improvement efforts. How do you measure foresight capability I wondered? I wondered even more about how do you measure the value of improved foresight? (By foresight I mean having a better ability to forecast alternative futures for robust decision-making and putting in place practices which will enable you to better prepare for changes and unexpected events in the future).
I searched the library and the internet before purchasing a handful of books on measurement and experiment design. Your book is a diamond among a sea of broken glass. “How to Measure Anything” is so well written I could barely put it down. I read it straight through (including doing the calibration exercises) in two nights after work and began reading it a second time with highlighter in hand.
I can’t thank you enough for the incredible insights and inspiration you have given me! Your book with go to the top of the stack as I begin my Ph.D. in this Fall.
Best regards,
Ross Shott
Graduate Student, University of Houston”
Ross,
Thanks for the kind endorsement and thanks for bringing this conversation over from the email conversation that we started.
My first question for you is the same one with which I begin all measurement problems: “What do you mean by foresight, exactly?” If you can think about this question and those below, I’ll get closer to solving your problem.
But while you consider that, let me propose a possible meaning and you can modify it further. I suppose you might mean the ability to predict particular events. Specifically, foresight might have more to do with forecasting trends like “What is the next big bio-tech revolution?” instead of “Will it rain tomorrow?”. But correct me if you mean something else.
For now, lets suppose you mean what I suggested and consider what its value might be. Presumably, foresight has value because of the effect it would have on your decisions and behaviors. As I ask in the book, what would you imagine you would be doing differently if you had more foresight? Imagine a list of specific events and ask what would be the value of acting on it.
Thinking about these specific events will help us focus on the right question. Tell me as much as you can about this application and let generate some specific examples of “improved foresight” and, then, some specific examples of what you would be able to do better if you had that foresight. Once we have that, let’s follow the path as shown in the book:
1) Decompose it – Think of a list of specific events you would have foresight of, a list of specific actions you might take if you had such foresight, and the specific advantages of those actions. Think of how all these might come together into a formula that computes the value of foresight.
2) State your current uncertainty – for each of the decomposed items in step #1, state what you know now about it. Estimate how often these events might occur. Estimate the possible actions and estimate the impacts of good and bad outcomes.
3) Compute the value of information for each of the variables – this requires that you think of the costs and benefits of some particular alternative you might take differently if you had more information about the value of foresight. Are you considering a specific software purchase that would allegedly improve foresight? Then that is the decision this information would help with.
4) Design the empirical method – for the variable(s) with the highest information value, think about how you would observe it. Then, based on the possible observation, design a systematic observation based, say, on random sampling or controlled experiments.
This sounds rather vague right now, but we can get more specific if you provide more specifics about the problem.
Thanks,
Doug Hubbard
by Douglas Hubbard | Dec 22, 2008 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Monday, December 22, 2008 4:23:05 PM, by Dynera.
“Hi Doug,
I have your book on order and I was wondering if you cover the measurement of preventation effectivenss. For instance, I was involved with a software architecture team who’s purpose was to vet architecture designs. We recieved several feedback forms saying that our input was useful but besides that we really didn’t see any other way to demonstrate our effectiveness.
Any feedback is appreciated.
Thanks.
Paul”
Paul,
First, my apologies for the long delay in my response. Between the holidays and site problems, it’s been quite a while since I’ve looked at the Forums.
Vetting architectural designs should have measures similar to any quality control process. Since you are a software architect, I would suggest the following:
1) Ask for more specifics in your feedback forms. Ask if any of the suggestions actually changed a design and how. Ask which suggestions, specifically, changed the design.
2) Count up these”potential cost savings identified” and count these up for each design you review.
3) Periodically go into more detail with a random sample of clients and your suggestions to them. For this smaller sample, take a suggestion that was identfied as one that resulted in a specific change (as in point #1 above) and get more details about what would have changed. Was an error avoided that otherwise would have been costly and, if so, how costly? Some existing research or the calibrated estimates of your team might suffice to put a range on the costs of different problems if they had not been avoided.
4) You can estimate the percentage of total errors that your process finds. Periodically use independent reviewers who develop separately assess the same design and compare their findings. I explain a method in Chapter 14 for using the findings from two different vetters to determine the number of errors that neither found. In short, if both vetters routinely find all of the same errors, then you probably find most of them. If each vetter finds lots of errors that the other does not find, then then are probably a lot that neither find. You can also periodically check the process by the same method used to measure proofreaders – simply add a small set of errors yourself and see how many are found by the vetter.
This way, you can determine whether you are catching, say, 40% to 60% of the errors instead of 92% to 98% of the errors. And of the errors you find, you might determine that 12% to 30% would have caused rework or other problems in excess of 1 person-month of effort had it not been found. Subjective customer responses like “very satisfied” can be useful, but the measures I just described should be much more informative.
Thanks for your interest.
Doug Hubbard
by Douglas Hubbard | Dec 8, 2008 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted at http://www.howtomeasureanything.com, on Monday, December 08, 2008 9:32:43 PM, by Amichai.
“The challenge is to come up with a simple, clearly defined number (or a few numbers) that reflect how well the organization is executing towards the goals from the CEO level to the line worker level. It would also be good to have an insight into what misalignment occurs between departments.
Most importantly, we should be able to track our alignment score overtime and track the impact of our improvements and corrective actions.
Any advice would be greatly appreciated!”
Thanks for your question. I mention strategic alignment just briefly in the book. As with all measurement questions, we start with What do you mean? . In this case, you offered perhaps part of the meaning – how well the organization is executing towards the goals from the CEO level to the line worker level. I would think that the rate it is approaching certain defined goals is itself measurable, especially if the goal is measurable. But perhaps some concrete examples could help us zero in more. Does alignment mean that the activities of each of these levels are contributing to the probability of reaching a goal at a stated time? Does it mean these activities are contributing toward reaching the goal faster? Perhaps the observable object is just the rate of completion of the goal itself. Isn’t alignment just another way to say performance given the way you are using it? If not, what is different?
If you are not just concerned about a % complete measure toward a particular goal then it seems to me that, in effect, you are trying to correlate certain observable input to the probability or rate of completion of stated goals – which, again, sounds a bit like performance. So what you are really trying to measure is whether each of some list of observable activities is improving the rate or probability of completion of a specific goal. Another issue you will have to consider is the tradeoff among multiple possibly competing goals. If you have more than one goal, you will have to collapse these goals into a single value using the utility curve approach I describe in the book or possibly simply monetizing the various goals. This way you can tell if your overall alignment has improved if you have increased satisfaction of one goal but slightly decreased another. Ultimately, it seems like you have a type of forecasting problem. You are asking Based on what I have seen so far (of some list of observations) are we getting closer to or further away from meeting Goal X? The score you are looking for should correlate with rate or probability of goal completion.
In addition to the questions I asked above, here are some more I have for you:
* Is you definition of alignment close to what I just talked about?
* What are some of the goals you are talking about?
* What will be the use of this measure, specifically?
* How much do you know now? What have you seen so far that causes you to think that some departments/individuals are more aligned than others?
If you can answer these questions then I think I can give you some very specific ideas.
Thanks,
Doug Hubbard
