How to Measure Performance

by Douglas Hubbard | Mar 20, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News

Originally posted at http://www.howtomeasureanything.com, on Friday, March 20, 2009 9:14:48 PM, by jerry.

“Greetings,

I loved your book. Thanks for sharing such valuable information. Now I’m trying to apply it.

I am leading a project of training developers and instructional designers and am attempting to put together a meaningful way to measure their performance. I have come up with some parameters that seem evident to me, such as time to complete a lesson, number of edits recommended (to the designer), type of edits recommended (order, strategies, completeness of content), edit recommendation trends (is the number of recommended edits going up or going down).

Is there a particular part of your book I should re-read that would help me frame a thorough performance evaluation measuring framework? Or can you suggest anything that would help expand the framework or make it a more reliable measure of performance?

Thank you in advance for any direction you can point me in or for any suggestions you can provide.

Jerry”

Thanks for reading my book. I think you might find part of what you are looking for in Chapter 11 on measuring preferences and attitudes. On page 197 I show how different performance measures of a software developer could be combined into a single metric by quantifying the acceptable tradeoffs.

You might also consider more of an “end result” metric of some kind. Isn’t the ultimate success of the instructional material measured by the performance of students? Obviously, many things affect the performance of students but among those should be the design of the material. Individual students will vary but if one set of material consistently results in better student performance than another set, then I think it’s fair to attribute some of that to the material designer.

Thanks,

Doug Hubbard

Lens Model Example – Chapter 12

by Douglas Hubbard | Mar 1, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News

Originally posted at http://www.howtomeasureanything.com, on Sunday, March 01, 2009 1:30:45 PM, by Paddy.

“Could you please clarify what scenarios the can Lens Model can remove human inconsistency in decision making (i.e., problems that are well defined/repeatable or unstructured)? Would like to apply Lens Model to evaluate computer interfaces.

Also, could you please clarify the variables in step 6 of the Lens Model Procedure – Perform regression analysis. For example, could you please clarify independent and dependent variables in step 6 and the end output in step 7. Diagram was great, example would be better.

Thanks,

Amran”

Originally posted at http://www.howtomeasureanything.com, on Friday, April 17, 2009 9:21:44 AM, by Paddy.

“Any help with an example would be much appreciated.

Thank you”

Chapter 6

by Douglas Hubbard | Feb 19, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News

Originally posted on http://www.howtomeasureanything.com/forums/ on Thursday, February 19, 2009 1:41:48 PM, by Thakur.

“I enjoyed reading Chapter 6 (Measuring Risk: Introduction to the Monte Carlo Simulation). It was very informative. After reading it I tried to do the following using Excel. But I failed.

1). Simulating the Monty Hall Problem.

2). Simulating Birthdays

3). Genetics: Simulating Population Control

Can You please help me and guide me.

Thanks

Thakur”

You are asking for a lot! But how about I answer a bit at a time? First, lets do Monty Hall.

For those of you who might not have heard of this problem, its based on a classic probability theory example. Imagine that you are on the 70’s game show “Let’s Make a Deal” hosted by Monty Hall. You are a contestant and you are given three doors to choose from. Behind one of the doors is a brand new car! If you choose the door with the car behind it, you get to drive it away.

You choose a door. But then Monty Hall shows you what is behind one of the other doors to reveal one of the “joke prizes” (e.g. a donkey). Then he asks you if you would like to keep the door you first chose or switch to the other remaining door. People often think that the odds of winning would be the same whether you switched or not. But they would be wrong.

To demonstrate why switching doors would be better, let’s set up a spreadsheet simulation where we define columns for the prize door, the chosen door, and the revealed door. One more column will be used as a flag to indicate whether we would have won if we stayed with the first door we chose or if we should have switched doors. Then we will copy down the first row of these columns to a few thousand rows to see the outcome.

Column 1, The Prize Door: This is the door the prize is really behind. As a contestant, you wouldn’t know this information, but we need it for the simulation. Write “The Prize Door” in cell A1. In cell A2 write =int(rand()*3+1). This will randomly generate the value of 1, 2 or 3.

Column 2, The Chosen Door: This is the door the contestant chose. In B1, write “The Chosen Door” and in B2 write the same formula you wrote in A2; =int(rand()*3+1). Again, this will randomly generate the value of 1, 2 or 3.

Column 3, The Revealed Door: This is the door Monty Hall reveals. Monty will always reveal a door you didn’t choose and it will always be a door that doesn’t have a prize behind it. In cell C1 write “The Revealed Door” and in C2 write =if(and(a2=1,b2=1),int(rand()*2+2),if(and(a2=1,b2=2),3,if(and(a2=1,b2=3),2,if(and(a2=2,b2=1),3,if(and(a2=2,b2=2),int(rand()+.5)*2+1,if(and(a2=2,b2=3),1,if(and(a2=3,b2=1),2,if(and(a2=3,b2=2),1,int(rand()*2+1))))))))) This seems clumsy, but its visually easier to decompose and understand than some approaches I might have taken. This will generate values according to the following table:

Prize Door……Chosen Door……Revealed Door
1…………………..1………………….2 or 3
1…………………..2………………….3
1…………………..3………………….2
2…………………..1………………….3
2…………………..2………………….1 or 3
2…………………..3………………….1
3…………………..1………………….2
3…………………..2………………….1
3…………………..3………………….1 or 2

Column 4, Winning Strategy; This cell tells you what the winning strategy would have been. Either you stick with the door you first chose or you switch doors. In D1 write “Winning Strategy” and in D2 write =if(A2=b2,0,1). This will generate a 0 if the winning strategy would have been to stick with the door you have and a 1 if you were better off switching.

Now copy down row 2 a thousand rows and take the average of the values in column 4 (remember not to average in the text in D1). One way to do this is write =average(D2:D1001) in cell E1. If you were just as well off sticking with the first chosen door as switching, then this average would be .5. But you will find that the average will be about .667. In other words, two thirds of the time the winning strategy was switching doors. The reason this works is that when Monty Hall reveals one of the other doors, he gives you additional information you didn’t have before. He reveals ONLY a door that doesn’t have a prize and ONLY a door you didn’t choose. When you first choose a door, you have a 2/3 chance of not winning (the prize is behind one of the other two doors). Once he reveals which of the other 2 doors is not a winner, then the remaining door has a 2/3 chance of winning.

Check back for my responses to your other questions. For clarification, when you talk about birthdays do you mean simulating the problem where you find minimum number of people before there is equal odds that at least 2 people have the same birthday?

Thanks for your question
Doug Hubbard

Value of Foresight

by Douglas Hubbard | Feb 18, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News

Originally posted on http://www.howtomeasureanything.com/forums/ on Wednesday, February 18, 2009 4:54:25 PM, by RossShott.

“Hello Mr. Hubbard,

Your book “How to Measure Anything” is remarkable!

I am currently completing my Master of Technology in Futures Studies degree by working on my thesis project. The project entails fleshing out a Foresight Maturity Model (similar to a CMM or CMMI) recently developed by Social Technologies. Meaning that I have been given the task of figuring out how to measure foresight within companies in such a way that those companies can be evaluated by a Futurist consulting firm using the FMM tool or the company could apply the FMM tool to their own organization for foresight improvement efforts. How do you measure foresight capability I wondered? I wondered even more about how do you measure the value of improved foresight? (By foresight I mean having a better ability to forecast alternative futures for robust decision-making and putting in place practices which will enable you to better prepare for changes and unexpected events in the future).

I searched the library and the internet before purchasing a handful of books on measurement and experiment design. Your book is a diamond among a sea of broken glass. “How to Measure Anything” is so well written I could barely put it down. I read it straight through (including doing the calibration exercises) in two nights after work and began reading it a second time with highlighter in hand.

I can’t thank you enough for the incredible insights and inspiration you have given me! Your book with go to the top of the stack as I begin my Ph.D. in this Fall.

Best regards,

Ross Shott
Graduate Student, University of Houston”

Ross,

Thanks for the kind endorsement and thanks for bringing this conversation over from the email conversation that we started.

My first question for you is the same one with which I begin all measurement problems: “What do you mean by foresight, exactly?” If you can think about this question and those below, I’ll get closer to solving your problem.

But while you consider that, let me propose a possible meaning and you can modify it further. I suppose you might mean the ability to predict particular events. Specifically, foresight might have more to do with forecasting trends like “What is the next big bio-tech revolution?” instead of “Will it rain tomorrow?”. But correct me if you mean something else.

For now, lets suppose you mean what I suggested and consider what its value might be. Presumably, foresight has value because of the effect it would have on your decisions and behaviors. As I ask in the book, what would you imagine you would be doing differently if you had more foresight? Imagine a list of specific events and ask what would be the value of acting on it.

Thinking about these specific events will help us focus on the right question. Tell me as much as you can about this application and let generate some specific examples of “improved foresight” and, then, some specific examples of what you would be able to do better if you had that foresight. Once we have that, let’s follow the path as shown in the book:

1) Decompose it – Think of a list of specific events you would have foresight of, a list of specific actions you might take if you had such foresight, and the specific advantages of those actions. Think of how all these might come together into a formula that computes the value of foresight.
2) State your current uncertainty – for each of the decomposed items in step #1, state what you know now about it. Estimate how often these events might occur. Estimate the possible actions and estimate the impacts of good and bad outcomes.
3) Compute the value of information for each of the variables – this requires that you think of the costs and benefits of some particular alternative you might take differently if you had more information about the value of foresight. Are you considering a specific software purchase that would allegedly improve foresight? Then that is the decision this information would help with.
4) Design the empirical method – for the variable(s) with the highest information value, think about how you would observe it. Then, based on the possible observation, design a systematic observation based, say, on random sampling or controlled experiments.

This sounds rather vague right now, but we can get more specific if you provide more specifics about the problem.

Thanks,

Doug Hubbard