by Douglas Hubbard | Mar 20, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted at http://www.howtomeasureanything.com, on Friday, March 20, 2009 9:14:48 PM, by jerry.
“Greetings,
I loved your book. Thanks for sharing such valuable information. Now I’m trying to apply it.
I am leading a project of training developers and instructional designers and am attempting to put together a meaningful way to measure their performance. I have come up with some parameters that seem evident to me, such as time to complete a lesson, number of edits recommended (to the designer), type of edits recommended (order, strategies, completeness of content), edit recommendation trends (is the number of recommended edits going up or going down).
Is there a particular part of your book I should re-read that would help me frame a thorough performance evaluation measuring framework? Or can you suggest anything that would help expand the framework or make it a more reliable measure of performance?
Thank you in advance for any direction you can point me in or for any suggestions you can provide.
Jerry”
Thanks for reading my book. I think you might find part of what you are looking for in Chapter 11 on measuring preferences and attitudes. On page 197 I show how different performance measures of a software developer could be combined into a single metric by quantifying the acceptable tradeoffs.
You might also consider more of an “end result” metric of some kind. Isn’t the ultimate success of the instructional material measured by the performance of students? Obviously, many things affect the performance of students but among those should be the design of the material. Individual students will vary but if one set of material consistently results in better student performance than another set, then I think it’s fair to attribute some of that to the material designer.
Thanks,
Doug Hubbard
by Douglas Hubbard | Mar 9, 2009 | Facilitating Calibrated Estimates, How To Measure Anything Blogs, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Monday, March 09, 2009 9:14:11 AM.
“I just read your book and found it fascinating. Thanks.
On calibrated estimates, once experts are calibrated, do they stay calibrated?
Or do you repeat every time that you are beginning a project or making an estimate.
I’m just thinking in a corporate setting – do you just do it once for a group of people that you may want estimates for or would you do it before each project. Do it annually?
What has been your experience on how long people stay calibrated?
Thanks,
Praveen”
by Douglas Hubbard | Mar 5, 2009 | General Topics, News
Originally posted at http://www.howtomeasureanything.com on Thursday, March 05, 2009 7:30:54 PM, by Paddy.
“Hi Douglas
Firstly let me give you a huge wrap for the book – I would say its invaluable but that would be wrong, because as I am learning “Everything can be measured”!
I am interested in what is traditionally referred to as a ‘soft’ area, where good measurements are hard to come by – Leadership and [Organizational] Development. Seeing as you asked to be stumped/challenged, Ill throw my biggest fish at you first…
How can you measure leadership?
To help direct the discussion that I hope will flow from this, lets talk about to specific examples: in a sporting context (ie impact of player/coach leadership on scores) and in a corporate context (impact of management/employee leadership on profits)
Lets see if this stumps you…
Paddy”
Thanks for your question. That is something I’ve been asked more than once. As with all measurement questions, I start out with “What do you mean…leadership?”. Then ask, how do you observe examples of leadership? If someone says “Leadership is better here than there” what observations are they basing that on?
Sometimes people define the observations for leadership as being some measure of performance of an organization. In that case, what they really want to measure is the performance. But sometimes they want to ask if particular “leadership styles” result in the improved performance. In that case, they should think about correlating surveys of staff about leadership (to determine the type of leadership style) and correlate that result with observed performance.
Perhaps they are asking about some ill-defined sense of charisma separate from the performance f the organization. In that sense, a survey of subordinates should suffice. But in that case, we want to be careful of some other effects that might get confused with charisma or leadership but most definitions of leadership would not include. For example, physical attractiveness and even being tall are often associated with subjective perceptions of leadership. US Presidents, for example, are almost always significantly taller than average. Tom Malloy in the 1980’s studied how attire affects perceptions of charisma, competence and authority. That’s the problem with the subjective sense of leadership they way it is often used. People can’t help but to let things affect our assessment of leaders even thought we know they shouldn’t.
Perhaps leadership is defined by examples of particularly inspirational ideas such as President Kennedy’s decision to go to the moon. Perhaps Joan of Arc leading the charge is leadership because it so inspired her troops. If the these cases are what you mean, then perhaps you should think about survey people about how inspired they are.
Personally, I think all of this is sort of meaningless if it doesn’t lead to performance. So, as I mention in the book, you need to ask why you want to know this. Are you evaluating prospective executive staff? Are you evaluating who will run a new division better? If you can zero in on why you care, you will probably find that measuring actual leadership (or whatever that means) is not your real concern. If you are trying to predict performance, I suggest that past performance is important. Would Kennedy or Joan of Arc have been that inspirational if they failed? Does a subjective perception of leadership by subordinates matter if the leader doesn’t meat objectives? Probably not.
Thanks,
Doug Hubbard
by Douglas Hubbard | Mar 1, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted at http://www.howtomeasureanything.com, on Sunday, March 01, 2009 1:30:45 PM, by Paddy.
“Could you please clarify what scenarios the can Lens Model can remove human inconsistency in decision making (i.e., problems that are well defined/repeatable or unstructured)? Would like to apply Lens Model to evaluate computer interfaces.
Also, could you please clarify the variables in step 6 of the Lens Model Procedure – Perform regression analysis. For example, could you please clarify independent and dependent variables in step 6 and the end output in step 7. Diagram was great, example would be better.
Thanks,
Amran”
Originally posted at http://www.howtomeasureanything.com, on Friday, April 17, 2009 9:21:44 AM, by Paddy.
“Any help with an example would be much appreciated.
Thank you”
by Douglas Hubbard | Feb 19, 2009 | How To Measure Anything Blogs, Measurement Questions From Readers, News
Originally posted on http://www.howtomeasureanything.com/forums/ on Thursday, February 19, 2009 1:41:48 PM, by Thakur.
“I enjoyed reading Chapter 6 (Measuring Risk: Introduction to the Monte Carlo Simulation). It was very informative. After reading it I tried to do the following using Excel. But I failed.
1). Simulating the Monty Hall Problem.
2). Simulating Birthdays
3). Genetics: Simulating Population Control
Can You please help me and guide me.
Thanks
Thakur”
You are asking for a lot! But how about I answer a bit at a time? First, lets do Monty Hall.
For those of you who might not have heard of this problem, its based on a classic probability theory example. Imagine that you are on the 70’s game show “Let’s Make a Deal” hosted by Monty Hall. You are a contestant and you are given three doors to choose from. Behind one of the doors is a brand new car! If you choose the door with the car behind it, you get to drive it away.
You choose a door. But then Monty Hall shows you what is behind one of the other doors to reveal one of the “joke prizes” (e.g. a donkey). Then he asks you if you would like to keep the door you first chose or switch to the other remaining door. People often think that the odds of winning would be the same whether you switched or not. But they would be wrong.
To demonstrate why switching doors would be better, let’s set up a spreadsheet simulation where we define columns for the prize door, the chosen door, and the revealed door. One more column will be used as a flag to indicate whether we would have won if we stayed with the first door we chose or if we should have switched doors. Then we will copy down the first row of these columns to a few thousand rows to see the outcome.
Column 1, The Prize Door: This is the door the prize is really behind. As a contestant, you wouldn’t know this information, but we need it for the simulation. Write “The Prize Door” in cell A1. In cell A2 write =int(rand()*3+1). This will randomly generate the value of 1, 2 or 3.
Column 2, The Chosen Door: This is the door the contestant chose. In B1, write “The Chosen Door” and in B2 write the same formula you wrote in A2; =int(rand()*3+1). Again, this will randomly generate the value of 1, 2 or 3.
Column 3, The Revealed Door: This is the door Monty Hall reveals. Monty will always reveal a door you didn’t choose and it will always be a door that doesn’t have a prize behind it. In cell C1 write “The Revealed Door” and in C2 write =if(and(a2=1,b2=1),int(rand()*2+2),if(and(a2=1,b2=2),3,if(and(a2=1,b2=3),2,if(and(a2=2,b2=1),3,if(and(a2=2,b2=2),int(rand()+.5)*2+1,if(and(a2=2,b2=3),1,if(and(a2=3,b2=1),2,if(and(a2=3,b2=2),1,int(rand()*2+1))))))))) This seems clumsy, but its visually easier to decompose and understand than some approaches I might have taken. This will generate values according to the following table:
Prize Door……Chosen Door……Revealed Door
1…………………..1………………….2 or 3
1…………………..2………………….3
1…………………..3………………….2
2…………………..1………………….3
2…………………..2………………….1 or 3
2…………………..3………………….1
3…………………..1………………….2
3…………………..2………………….1
3…………………..3………………….1 or 2
Column 4, Winning Strategy; This cell tells you what the winning strategy would have been. Either you stick with the door you first chose or you switch doors. In D1 write “Winning Strategy” and in D2 write =if(A2=b2,0,1). This will generate a 0 if the winning strategy would have been to stick with the door you have and a 1 if you were better off switching.
Now copy down row 2 a thousand rows and take the average of the values in column 4 (remember not to average in the text in D1). One way to do this is write =average(D2:D1001) in cell E1. If you were just as well off sticking with the first chosen door as switching, then this average would be .5. But you will find that the average will be about .667. In other words, two thirds of the time the winning strategy was switching doors. The reason this works is that when Monty Hall reveals one of the other doors, he gives you additional information you didn’t have before. He reveals ONLY a door that doesn’t have a prize and ONLY a door you didn’t choose. When you first choose a door, you have a 2/3 chance of not winning (the prize is behind one of the other two doors). Once he reveals which of the other 2 doors is not a winner, then the remaining door has a 2/3 chance of winning.
Check back for my responses to your other questions. For clarification, when you talk about birthdays do you mean simulating the problem where you find minimum number of people before there is equal odds that at least 2 people have the same birthday?
Thanks for your question
Doug Hubbard
