by Douglas Hubbard | Feb 18, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Wednesday, February 18, 2009 4:54:25 PM, by RossShott.
“Hello Mr. Hubbard,
Your book “How to Measure Anything” is remarkable!
I am currently completing my Master of Technology in Futures Studies degree by working on my thesis project. The project entails fleshing out a Foresight Maturity Model (similar to a CMM or CMMI) recently developed by Social Technologies. Meaning that I have been given the task of figuring out how to measure foresight within companies in such a way that those companies can be evaluated by a Futurist consulting firm using the FMM tool or the company could apply the FMM tool to their own organization for foresight improvement efforts. How do you measure foresight capability I wondered? I wondered even more about how do you measure the value of improved foresight? (By foresight I mean having a better ability to forecast alternative futures for robust decision-making and putting in place practices which will enable you to better prepare for changes and unexpected events in the future).
I searched the library and the internet before purchasing a handful of books on measurement and experiment design. Your book is a diamond among a sea of broken glass. “How to Measure Anything” is so well written I could barely put it down. I read it straight through (including doing the calibration exercises) in two nights after work and began reading it a second time with highlighter in hand.
I can’t thank you enough for the incredible insights and inspiration you have given me! Your book with go to the top of the stack as I begin my Ph.D. in this Fall.
Best regards,
Ross Shott
Graduate Student, University of Houston”
Ross,
Thanks for the kind endorsement and thanks for bringing this conversation over from the email conversation that we started.
My first question for you is the same one with which I begin all measurement problems: “What do you mean by foresight, exactly?” If you can think about this question and those below, I’ll get closer to solving your problem.
But while you consider that, let me propose a possible meaning and you can modify it further. I suppose you might mean the ability to predict particular events. Specifically, foresight might have more to do with forecasting trends like “What is the next big bio-tech revolution?” instead of “Will it rain tomorrow?”. But correct me if you mean something else.
For now, lets suppose you mean what I suggested and consider what its value might be. Presumably, foresight has value because of the effect it would have on your decisions and behaviors. As I ask in the book, what would you imagine you would be doing differently if you had more foresight? Imagine a list of specific events and ask what would be the value of acting on it.
Thinking about these specific events will help us focus on the right question. Tell me as much as you can about this application and let generate some specific examples of “improved foresight” and, then, some specific examples of what you would be able to do better if you had that foresight. Once we have that, let’s follow the path as shown in the book:
1) Decompose it – Think of a list of specific events you would have foresight of, a list of specific actions you might take if you had such foresight, and the specific advantages of those actions. Think of how all these might come together into a formula that computes the value of foresight.
2) State your current uncertainty – for each of the decomposed items in step #1, state what you know now about it. Estimate how often these events might occur. Estimate the possible actions and estimate the impacts of good and bad outcomes.
3) Compute the value of information for each of the variables – this requires that you think of the costs and benefits of some particular alternative you might take differently if you had more information about the value of foresight. Are you considering a specific software purchase that would allegedly improve foresight? Then that is the decision this information would help with.
4) Design the empirical method – for the variable(s) with the highest information value, think about how you would observe it. Then, based on the possible observation, design a systematic observation based, say, on random sampling or controlled experiments.
This sounds rather vague right now, but we can get more specific if you provide more specifics about the problem.
Thanks,
Doug Hubbard
by Douglas Hubbard | Feb 16, 2009 | Facilitating Calibrated Estimates, How To Measure Anything Blogs, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Monday, February 16, 2009 11:32:17 AM.
“I am looking for some material (articles or books) on the subject of Calibration. I want to be expert in Calibration.
Thakur”
I certainly support your goal for becoming and expert in this topic. It is a well-studied topic but is still far too obscure in practical applications. Beyond my book, the most important sources are the purely academic literature…which I would definitely recommend for anyone who wants to be an expert. My next book The Failure of Risk Management, will cover this topic with a slightly different emphasis and, in some cases, in more detail. In both books, I resort to several academic studies, including the following.
A key source is Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press, 1982. It is a compilation of several research papers on the topic. You can buy this book on Amazon.
Here are several more articles:
A.H. Murphy and R. L. Winker, ‘‘Can Weather Forecasters Formulate Reliable Probability Forecasts of Precipitation and Temperature?,’’ National Weather Digest 2, 1977, 2–9.
D. Kahneman and A. Tversky, ‘‘Subjective Probability: A Judgment of Representativeness,’’ Cognitive Psychology 3, 1972, 430–454.
G.S. Tune, ‘‘Response Preferences: A Review of Some Relevant Literature,’’ Psychological Bulletin 61, 1964, 286–302.
E. Johnson, ‘‘Framing, Probability Distortions and Insurance Decisions,’’ Journal of Risk and Uncertainty 7, 1993, 35.
D. Kahneman and A. Tversky, ‘‘Subjective Probability: A Judgment of Representativeness,’’ Cognitive Psychology 4, 1972, 430–454.
D. Kahneman and A. Tversky, ‘‘On the Psychology of Prediction,’’ Psychological Review 80, 1973, 237–251.
A. Tversky and D. Kahneman, ‘‘The Belief in the ‘Law of Small Numbers,’’’ Psychological Bulletin, 1971.
A. Koriat, S. Lichtenstein, and B. Fischhoff, ‘‘Reasons for Confidence,’’ Journal of Experimental Psychology: Human Learning and Memory 6, 1980, 107–118
by Douglas Hubbard | Feb 11, 2009 | Computing Value of Information, How To Measure Anything Blogs, News
Originally posted at http://www.howtomeasureanything.com, on Wednesday, February 11, 2009 2:16:38 PM, by andrey.
“Hello Douglas,
First of all let me say I have thoroughly enjoyed reading your book. I have a technical background (software engineering) and have always been surprised at how “irresponsible” some business-level decision making can be – based on gut instincts and who-knows-what. This ‘intuitive’ approach is plagued with biases and heuristics, the effects of such approach has been widely publicized (for example here). This is one of many reasons I found your book very simulating and the AIE approach as a whole very promising.
However I have reservations about a few points you make. Please forgive me my ignorance if my questions are silly, my math has become rusty with the years passing by.
One of my concerns is the validity of the assumption that you make when explaining ‘The Rule of Five’, 90% CI and especially when using Monte Carlo simulation. I can believe (although it would’ve been great to see the sources) that ‘there is a 93% chance that the median of a population is between the smallest and largest values in any random sample of five from that population’. But when you are applying this to the Monte Carlo simulations, you assume that the mean (which is also the median for symmetric probability distributions) is exactly in the middle of the confidence interval. Which, I think, makes a big difference to the outcome because of the shape of the normal distribution function. If you assume the the median is for example very close to the lower or upper bound of the confidence interval by putting a different value into the =norminv(rand(),A, B) formula the results would be different.
I am still working through your book (second reading), trying to ‘digest’ and internalise it properly. I would be very grateful if you could explain this to me.
Thank you very much,
Andrey”
Thanks for your comment.
I don’t show a source (I’m the one who coined the term “Rule of Five” in this context) but I show the simple calculation, which is easily verifiable. The chance of randomly picking one sample with a parameter value above the true population median for that parameter is, by definition, 50%. We ask “what is the probability that I could happen to randomly choose five samples in a row that are all above the true population median?” It is the same chance as flipping five coins and getting all heads. The answer is 1/2^5 or 3.125% Likewise, the probability that we could have just picked 5 in a row that were all below the true population median is 3.125%. That means that there is a 93.75% chance that some were above and some were below – in other words, that the median is really between the min and max values in the sample. It’s not an “assumption” at all – it is a logically necessary conclusion from the meaning of the word “median”.
You can verify this experimentally as well. Try generating any large set of continuous values you like using any distribution you like (or just define a distribution function for such a set). Determine the median for the set. Then randomly select 5 from the set and see if the known population median is between the min and max values of those 5 samples. Repeat this a large number of times. You will find that 93.75% of the time the known median will be between the min and max values of the sample of 5.
I believe I also made it clear that this only applies to the median and not the mean. I further stated that if, on the other hand, you were able to make the determination that the distribution is symmetrical then, of course, it applies to the mean as well. Often, you may have reason to do this and this is no different than the assumption in any application of a t-stat or z-stat based calculation of a CI (which are always assumed to be symmetrical).
Furthermore, you certainly should not use a function that generates a normally distributed random number if you know it not to be normally distributed and I don’t believe I recommended otherwise. If you know the median and the mean of the distribution you intend to generate are not the same, then you can’t count on Excel’s normdist function to be a good approximation. For the sake of simplicity, I gave a very limited set of distributions functions for random values in Excel. But we can certainly add a lot more (triangular, beta, lognormal, etc.)
My approach is not to assume anything you don’t have to. If you don’t know that the distribution isn’t lopsided, you can simulate that uncertainty, too. Is it possible that the real distribution could actually be lognormal? Then put a probability on that and generate accordingly. Why “assume” something is true if we can explicitly model that we are uncertain about something?
Thanks,
Doug
by Douglas Hubbard | Feb 5, 2009 | How To Measure Anything Blogs, News
Hello, [I have] short question – where i can find the book in Russian? On the site it is written that the translation exists.
I`m writing postgraduate research on the efficiency of IT – it is quite interesting to take a deep look into your methodology, but the translation can significantly increase the speed of this process))
[Thanks] in advance,
Anton
by Douglas Hubbard | Feb 3, 2009 | How To Measure Anything Blogs, News
Originally posted at http://www.howtomeasureanything.com, on Tuesday, February 03, 2009 9:20:28 AM, by lascar.
“Hello.
I’ve enjoyed the book and am trying to apply AIE to some of our IT decision making. Unfortunately I don’t have a statistician’s background and my college days are quite far away. So, [I’m] trying to catch up on some basics. Wikipedia is amazingly helpful in this sense.
I have two questions. If they are two basic for this forum, I’d appreciate anyone at least directing me to some resources which might help me answer them. Direct answers of cause are even more welcomed.
1. In Monte Carlo example in the book, the assumption is that most of the variables have Normal distribution. And if not, there are 2 more distributions mentioned – Uniform and Binary. I guess these are most common? My question is: how does one quickly evaluate what type of distribution is fitting for a variable? I’d guess it is quite straight forward with binary distribution. However from this article (http://en.wikipedia.org/wiki/List_of_probability_distributions), it seams there is quite a choice of distributions.
The MC scenario I’m running is to evaluate performance of a software package. I also realize that a quick proof of concept (running software and collecting metrics) might shed more light on distribution of some metric/variable. However that requires acquiring an expensive license first. So the decision I’m trying to facilitate is to prove that we need a POC and I need to calculate the value of improving on these measurements with POC’s help – alas – that cost of POC is worth lowering the uncertainty of measurements.
2. In the same Monte Carlo example standard deviation of 3.29 is used and the statement is that it is for 90% CI. However I’ve stumbled on this article (http://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data) and it seams the standard deviation for 90% is 1.645. 3.49 is closer to 99% CI. Can someone clarify, please?
Thank you.”
Thanks for your interest. First, yes there are quite a few distributions to choose from. I included the three simplest. The normal distribution is a very specific type of “bell curve”. I won’t go into how this bell curve is different from other bell curves, but the difference between this distribution and a uniform or binary is simple. The normal distribution is a range of values that are more likely in the middle but go out in both directions forever, albeit the odds are diminishingly small at the tails. The formula I gave for converting the bounds to normal distribution allows for values outside of your bounds – there is a 5% chance it could be higher than the upper bound and a 5% chance it is lower than the lower bound.
In a uniform distribution the values can only possible between the upper and lower bounds. Unlike the normal distribution, there is no chance the value could be outside of the bounds. Also, unlike the normal distribution, values are not more likely in the middle. Any value between the bounds of a uniform distribution is equally likely to any other value between the bounds. Use uniform distributions when you know that a variable can’t possibly be outside of the bounds. For example, if I know an uncertain variable about a productivity improvement from a new technology can’t be less than 0% and can’t be more than 10% (perhaps that’s the maximum amount of time spent on the activity being automated) then I would use a uniform distribution. However, if I’m not certain of those bounds but I think values around 5% are more likely than other values, then I might make it a normal distribution. Note that the normal, then, would allow the productivity improvement to be a negative value or greater than 10% even though values in the middle are more likely.
Binary is a simple one. It applies to events that either happen or do not. For example, if you are building a Monte Carlo simulation of a construction project and you want to model the chance of a labor strike, then you need to show that it will happen (or not) with a given probability. If there is a 10% chance of a strike then there is a 90% chance of no strike. The only values generated are either 1 or 0 and nothing in between. Of course, if there is a strike, you might want to use a normal or uniform to simulate the duration of the strike.
Regarding your second question, there is no inconsistency between the two values you mention. There are 3.29 standard deviations in a 90% CI if you subtract the upper bound from the lower bound, as I describe in the book. But there are 1.645 standard deviations from the middle of the range to either bound – which is half the distance between the bounds. (1.645 x 2 = 3.29) When you use 1.645, it is because you are starting with the middle value and computing the 90% CI. In the situation in the book, we start with a 90% CI and need to compute the standard deviation so we can simulate it.
