Originally posted at http://www.howtomeasureanything.com, on Friday, March 20, 2009 9:14:48 PM, by jerry.
“Greetings,
I loved your book. Thanks for sharing such valuable information. Now I’m trying to apply it.
I am leading a project of training developers and instructional designers and am attempting to put together a meaningful way to measure their performance. I have come up with some parameters that seem evident to me, such as time to complete a lesson, number of edits recommended (to the designer), type of edits recommended (order, strategies, completeness of content), edit recommendation trends (is the number of recommended edits going up or going down).
Is there a particular part of your book I should re-read that would help me frame a thorough performance evaluation measuring framework? Or can you suggest anything that would help expand the framework or make it a more reliable measure of performance?
Thank you in advance for any direction you can point me in or for any suggestions you can provide.
Jerry”
Thanks for reading my book. I think you might find part of what you are looking for in Chapter 11 on measuring preferences and attitudes. On page 197 I show how different performance measures of a software developer could be combined into a single metric by quantifying the acceptable tradeoffs.
You might also consider more of an “end result” metric of some kind. Isn’t the ultimate success of the instructional material measured by the performance of students? Obviously, many things affect the performance of students but among those should be the design of the material. Individual students will vary but if one set of material consistently results in better student performance than another set, then I think it’s fair to attribute some of that to the material designer.
Thanks,
Doug Hubbard
Hello Doug,
Your example of performance measurements for software developer (p.197) is very interesting for me – I would like to use it in practice. I tried to come up with a single number as you suggest in the book (similar to QALY) but I can’t figure out how to practically do that. When QALY is used, it seems they assume a simple utility function that has shape such as 1 year of the perfect quality life (“1” on the quality scale) would equal 2 years on the half perfect life (“0.5” on the quality scale) and so on. That way, for example, it is obvious that a medicine that “gives” 5 QALYs is better than an alternative with the value of 4 QALYs. However my utility function is different although may be visually similar in shape. Please I would like to use a single metric to evaluate the relative performance of software developers based on this custom utility function, similar to the example in your book. The performance will be evaluated between the individuals and between the past and the present for each individual. For example I’d like to say that a particular developer scored 50 points in the previous quarter, and 55 in the current quarter therefore his/her performance has increased.
Or may be a broader question is better: how would I practically come up with a metric using *any* utility function that is constructed as a result of a research on stakeholder preference?
Thanks very much,
Andrey
The utility function is not assumed to be linear – in fact I would be surprised if it was. The reason why we draw the curve is precisely because we do not necessarilly expect that one year of “perfect quality life” would equal two years of “half quality life”.
Likewise, if you are using a utility function to combine different measures of software developer performance, then I would not assume they would be linear. They are whatever values you would say they have to your organization. But once you define the utility curve for, say, “error-free code rate” vs. “code per month”, you will be able to express any combination of these two values as a single “quality adjusted value”. Remember, the utility curve is a curve where all the points on the curve are considered – by you or your firm – to be equally desirable to any other point on the same curve. As I explain in that same chapter, just follow a given curve until you come to some pre-defined quality. That way, all quality and quantity combinations can be stated as a single number where they are all adjusted for to the same quality.
This is something that I believe I explain in the book. Perhaps if we discuss this further I can think of where the confusion arises and I might be able to think of a different way of explaining it. I’m writing the second edition now so it would be helpful if we could find a better explaination. Let me know if what I just explained helped. If not, perhaps you should give me a call and we can discuss it in person.
Thanks,
Doug