I’m reintroducing the Measurement Challenge for the blog. I ran it for a couple of years on the old site and had some very interesting posts.
Use this thread to post comments about the most difficult – or even apparently “impossible” – measurements you can imagine. I am looking for truly difficult problems that might take more than a couple of rounds of query/response to resolve. Give it your best shot!
Doug Hubbard
Hi Doug,
I am a professional investor and a fan of Warren Buffett. My question is: At the time Buffett started his first partnership in 1956, how would a potential investor have measured Buffett’s value? We can measure the value of Buffett to his investors in retrospect by looking at the returns that Buffett earned for his investors. Is this just a case of where measurement could take decades, at which point the opportunity has been missed? How would I apply your principles if I was looking for the next Warren Buffett? Or to phrase this another way: How would I measure the value to me of investing in a fund run by someone who claims to be the next Buffett, but does not have a long track record?
Many thanks,
Tom
Tom,
If what you mean is how can we predict the next Warren Buffett, I don’t think we can, exactly. But if your question is “How do we – even slightly – improve the odds of selecting successful investors compared to my unaided judgment?”, then I think we can do that.
Paul Meehl shows a large volume of work comparing simple statistical models against human intuition. He collected research comparing the judgments of humans to statistical models in such as predicting the outcomes of football games, predicting business failures, and the prognosis of liver disease over a large number of trials. The results were conclusive. In a wide variety of domains and even in areas where it was assumed the human expert was essential, simple statistical models outperformed the humans at predicting outcomes.
So, to your question, can you collect data about characteristics of various investors and track them over time? You would have to be sure to test properly for how much variance can be accounted for by luck, of course. Then start comparing your own estimates to that of the regression model and see which does better. Remember, you don’t have to collect data on ALL investors. That’s what random sampling is for. Also, in my book, I describe something called the Lens method that seems to work even for the “data challenged”. The Lens method works by statistically the smoothing your own judgments – without any historical data. Past research shows that our own judgments are so inconsistent that even a model that just removes our inconsistencies is a significant improvement.
Doug Hubbard
I have been tasked with creating a value-based pricing model for my employer’s products so that we can maximize profitability. This means that instead of using our cost-plus approach, we would know how different customer sectors value our products. We would price products at a premium that captures some but probably not all of this value. This usually involves educating customers on how paying more for our product actually saves money over the near/long term verses paying a lower price for our competitors’ products. My employer’s products are not very distinguishable from our competitors’ products but our on-time delivery, lead-time, financial stability, global reach and other features are definitely distinguishable and hence, measurable. My employer sells industrial products in a business-to-business model supported by field salesmen. Buyers representing our customers are quite keen on hiding their value of my employer’s products and services in order to cut the best deal for themselves so I have little confidence that I can take direct measurements from the customer; however, I feel confident that I can get some kind of value measures on the aforementioned attributes from our outside sales team. Current value based pricing models rely on something called a fair value graph where a product’s price is plotted on the vertical axis and its dimensionless value coefficient is plotted on the horizontal axis. The value coefficient is calculated with a series of rankings on various product attributes using 1 to 10 scales. A competitor’s product is ranked the same way and depending on how the points fall on the graph, one can determined if their product is overvalued or undervalued relative to the competition. After reading your book, I have little confidence that such a method has credibility because it does not dollarize value, it merely provides a dimensionless rank relative to chosen competitors using subjective ordinal rankings that have little substance. For example, what is “7” on-time delivery? My challenge to you is how do I use your methods to dollarize my employer’s value? I can picture calibrating salespeople, getting 90% confidence ranges on the value of on-time delivery, lead-time, etc. but after that is where I get stuck. Do I perform a Monte Carlo analysis and if yes, what would that look like? Your book example uses a $400,000 investment that has a variety of productivity gains/losses and shows a 14% chance of not breaking even. I cannot conceptualize a result like “there is a 14% chance customer group A will reject our value-based price of $200 for product X”. Thank you, Tom B.
Tom,
Thanks for your question.
Since you read my book, you know my position on these ordinal “1 to 10” scales. They are never really necessary once someone figures out what the real problem is. They simply gloss over the problem making managers feel like it was solved in some way.
In the book example you cite, the 14% does not indicate a discrete “all or nothing” outcome. I wouldn’t model a discrete, binary chance that an entire customer group would reject a given price. A more realistic model is that some uncertain percentage of that group will not purchase the product at a given price. For example “At $200, our 90% CI is that 10% to 35% of customers will decline to purchase product X”. This is, in fact, what all price optimization models are doing directly or indirectly.
But recall another one of the maxims I mention in the book: no matter what you are measuring, assume it has been measured before. This certainly turns out to be true in this case. Not only is there a large body of academic work on estimating price elasticity and then computing optimal prices, there are well-established and proven tools available to you on the market now.
Is your business in the B2B sales area? If so, one of my current clients is Zilliant and your problem is exactly the sort their software addresses. Zilliant has a large number of very able “price scientists” who have developed the algorithms for price optimization for customers in many B2B situations. Their customers include many of the largest manufacturers and distributors you can think of. I would start by giving them a call (go to Zilliant.com). Then you can do real price science and drop the whole “1 to 10” activity.
Thanks again,
Doug Hubbard
How do you provide metrics around “innovation” and its impact on a team? What metric(s) would show the benefit of focusing on innovation? What metric(s) would show the negative impacts of not focusing on innovation? How do you quantify that “innovation” matters to a team & what they do?
I mention this issue in the book. The biggest problem people have with measuring innovation is the second of the three obstacles to measurement (i.e. the Concept, Object, Method). They often don’t have a clear idea about what innovation even means, what they see when they see more of it, and what decision would be influenced by such a measurement.
Words like Innovation are really usually umbrella terms that stand for a big set of concepts – each of which themselves is ill-defined. So start by asking yourself what you mean by innovation, what you see when you see it, and what decision do you expect to be influenced by it. If you know there is such a thing as innovation it is only because you must at least have observed examples of it. Great. So name the example. If you think it means more than that example keep adding to the list. But everything you include has to be a clearly defined and observable phenomenon. Does innovation mean development times for new products is reduced? Does it mean that new ideas are more likely to succeed in the market (remember, Ogilve said “If it doesn’t sell, it isn’t creative” Great point about getting pragmatic with a fluffy concept). Does it mean at least that more ideas are generated as proposals so that idea selection comes from a better set of choices. Or does it mean that that you won an Oscar or Nobel for your work? Whatever it is (even if it is many or all of these) it must be observable.
Now ask about the purpose of the measurement. Why do you care? What decision do you think you might make differently if I told you right now what your “innovation level” was? Do you have an idea for initiatives to improve innovation? What evidence is there that those could work? Is it a way to evaluate project leaders so you become more likely to select project leads that produce useful solutions? Again, all the outcomes are observable. You may find, in the end, that once you figure out what innovation means and why you want to know it, that you end up not needing the original vague term anymore.
Thanks for your question!
Doug Hubbard