What do you think?
Rate this book
The Tyranny of Metrics
How the obsession with quantifying human performance threatens our schools, medical care, businesses, and government
Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we've gone from measuring performance to fixating on measuring itself. The result is a tyranny of metrics that threatens the quality of our lives and most important institutions. In this timely and powerful book, Jerry Muller uncovers the damage our obsession with metrics is causing--and shows how we can begin to fix the problem.
Filled with examples from education, medicine, business and finance, government, the police and military, and philanthropy and foreign aid, this brief and accessible book explains why the seemingly irresistible pressure to quantify performance distorts and distracts, whether by encouraging "gaming the stats" or "teaching to the test." That's because what can and does get measured is not always worth measuring, may not be what we really want to know, and may draw effort away from the things we care about. Along the way, we learn why paying for measured performance doesn't work, why surgical scorecards may increase deaths, and much more. But metrics can be good when used as a complement to--rather than a replacement for--judgment based on personal experience, and Muller also gives examples of when metrics have been beneficial.
Complete with a checklist of when and how to use metrics, The Tyranny of Metrics is an essential corrective to a rarely questioned trend that increasingly affects us all.
Today, organizations of all kinds are ruled by the belief that the path to success is quantifying human performance, publicizing the results, and dividing up the rewards based on the numbers. But in our zeal to instill the evaluation process with scientific rigor, we've gone from measuring performance to fixating on measuring itself. The result is a tyranny of metrics that threatens the quality of our lives and most important institutions. In this timely and powerful book, Jerry Muller uncovers the damage our obsession with metrics is causing--and shows how we can begin to fix the problem.
Filled with examples from education, medicine, business and finance, government, the police and military, and philanthropy and foreign aid, this brief and accessible book explains why the seemingly irresistible pressure to quantify performance distorts and distracts, whether by encouraging "gaming the stats" or "teaching to the test." That's because what can and does get measured is not always worth measuring, may not be what we really want to know, and may draw effort away from the things we care about. Along the way, we learn why paying for measured performance doesn't work, why surgical scorecards may increase deaths, and much more. But metrics can be good when used as a complement to--rather than a replacement for--judgment based on personal experience, and Muller also gives examples of when metrics have been beneficial.
Complete with a checklist of when and how to use metrics, The Tyranny of Metrics is an essential corrective to a rarely questioned trend that increasingly affects us all.
240 pages, Hardcover
First published January 1, 2017
About the author
Jerry Z. Muller
13 books36 followersJerry Z. Muller is professor of history at the Catholic University of America in Washington, DC.
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 30 of 233 reviews
November 10, 2018
Ayn Rand Lives
This is an important book about an important subject. It’s primary importance lies in the fact that it is completely wrong about what’s at issue and how to fix it. It is so wrong that it makes the case for its antithesis. This too is wrong, if only slightly less 0bviously so.
Here’s the thesis: “We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through ‘transparency.’ But the identification of accountability with metrics and with transparency is deceptive. Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement,... The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement.”
Here’s the problem: what Muller means by standardized metrics is what some group of people - managers, politicians, scientists - have agreed among themselves as to what constitutes ‘success’. The entire book is then devoted to examples of how this sort of agreement results in stupid actions and consequences. My experience, like his, is that many metrics are misconceived, obsolete, counter-productive, and well... stupid. But rather than investigate how to improve the way these metrics are arrived at by any group, Muller plumps for replacing them with a vague concept of ‘individual experience.’ But what is the content of this individual experience? It can only be another more or less (usually less) articulate metric, some different criterion of correct choice, or it’s nothing but fantasy. And if this criterion has not been accepted by the group as superior to the existing ‘standardized’ criterion, how has ‘accountability’ been served?
An answer might be that the decision on the criterion used by an individual in defiance of the established metric is the precise action for which he or she is to be held accountable. Is the metric of individual experience better than the established metric? This would be quite reasonable - disobeying orders for a good reason should be acceptable when the orders are obviously destructive, illegal, or have unexpected collateral effects. But this answer would require some elaboration about the process by which an individual’s experience, when in conflict with established criteria of decision, can be reconciled with, incorporated into, or modify the established view of correct action. A military courts-martial for example is usually not convened to determine whether insubordination has taken place but whether it is justified in the circumstances.
But Muller doesn’t provide the least hint how the experience of an individual, or new experiences at all, should be used to modify the ‘standing orders’ of established metrics. He seems to believe that there are real but inherently inarticulate criteria living in the nervous systems of decision-makers which, if forced into some level of operational literacy, would lose their real import. He also believes that in general terms these inarticulate instincts are superior to any metric formulated by a group. If the individual is in conflict with the group, the group is simply wrong. Whatever else he has to say about accountability is therefore nonsense. There is none. Disagreement about ‘what counts’ is down to some sort of genetically-instilled spiritual preferences (or perhaps hedonistic utility) that can’t be questioned lest it inhibit the exercise of individual sovereignty.
I understand Muller’s frustration with the idiotic metrics which are employed in business and social policy. I have my own thesaurus of war stories about how such metrics have destroyed organisations and damaged individuals. And Muller is fighting against a real intellectual enemy - the idea that measurement of anything is an activity devoid of political judgment. This ideology, and it is just that, is the antithesis to Muller’s thesis. It seeks to establish that what we measure, principally in our choice of a metric of success, as some objective property of the thing measured. This is the foundation stone of scientism which the belief that individuals must submit to the rationality of reality, a reality determined by a consensus of experts. These experts may include a diverse set of skills - accountancy, economics, sociology, management or finance for example - but their aim is the same: to impose a standard, universal metric of success wherever they ply their influence. And Muller is right to resist this sort of intellectual totalitarianism.
But he is not right to substitute the arbitrariness of some group of experts with the arbitrariness of some individual’s experience. This is just another ideology. It strikes me that Muller takes the ideas of economic neo-liberalism to their philosophical outer limits. Margret Thatcher and Ronald Reagan didn’t believe that society existed; only individuals interacting did. Muller extends this to the realm of practical ideas from moral calculus to commercial decision-making. Discussion, mutual discovery, agreement and group learning are not things he can even consider in his ideological cage. How conflicts about what is important and therefore what should be measured in business, politics, and any cooperative effort are simply not his concern. What he calls ‘metric fixation’ is no different from ‘social awareness’. I am quite sure that Ayn Rand is alive and well in Muller’s garden shed.
Postscript: Muller implicitly presumes that social groups are fictional and should be given no status within his ideology of measurement. The opposing ideology, that only social conventions have meaning in measurement, is also prevalent. See here for a typical example: https://www.goodreads.com/review/show.... There are interesting philosophical implications of both views. For example ‘measurement socialists’ believe that the measurements they make are objective properties of the thing measured. While ‘measurement individualists’ like Muller believe that what is measured is entirely in the head of the one doing the measuring. Debates between the two groups are rarely edifying. For an alternative theory which avoids both the Scylla of collectivism and the Charybdis of individualism in measurement, see here: https://www.goodreads.com/review/show...
This is an important book about an important subject. It’s primary importance lies in the fact that it is completely wrong about what’s at issue and how to fix it. It is so wrong that it makes the case for its antithesis. This too is wrong, if only slightly less 0bviously so.
Here’s the thesis: “We live in the age of measured accountability, of reward for measured performance, and belief in the virtues of publicizing those metrics through ‘transparency.’ But the identification of accountability with metrics and with transparency is deceptive. Accountability ought to mean being held responsible for one’s actions. But by a sort of linguistic sleight of hand, accountability has come to mean demonstrating success through standardized measurement,... The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement.”
Here’s the problem: what Muller means by standardized metrics is what some group of people - managers, politicians, scientists - have agreed among themselves as to what constitutes ‘success’. The entire book is then devoted to examples of how this sort of agreement results in stupid actions and consequences. My experience, like his, is that many metrics are misconceived, obsolete, counter-productive, and well... stupid. But rather than investigate how to improve the way these metrics are arrived at by any group, Muller plumps for replacing them with a vague concept of ‘individual experience.’ But what is the content of this individual experience? It can only be another more or less (usually less) articulate metric, some different criterion of correct choice, or it’s nothing but fantasy. And if this criterion has not been accepted by the group as superior to the existing ‘standardized’ criterion, how has ‘accountability’ been served?
An answer might be that the decision on the criterion used by an individual in defiance of the established metric is the precise action for which he or she is to be held accountable. Is the metric of individual experience better than the established metric? This would be quite reasonable - disobeying orders for a good reason should be acceptable when the orders are obviously destructive, illegal, or have unexpected collateral effects. But this answer would require some elaboration about the process by which an individual’s experience, when in conflict with established criteria of decision, can be reconciled with, incorporated into, or modify the established view of correct action. A military courts-martial for example is usually not convened to determine whether insubordination has taken place but whether it is justified in the circumstances.
But Muller doesn’t provide the least hint how the experience of an individual, or new experiences at all, should be used to modify the ‘standing orders’ of established metrics. He seems to believe that there are real but inherently inarticulate criteria living in the nervous systems of decision-makers which, if forced into some level of operational literacy, would lose their real import. He also believes that in general terms these inarticulate instincts are superior to any metric formulated by a group. If the individual is in conflict with the group, the group is simply wrong. Whatever else he has to say about accountability is therefore nonsense. There is none. Disagreement about ‘what counts’ is down to some sort of genetically-instilled spiritual preferences (or perhaps hedonistic utility) that can’t be questioned lest it inhibit the exercise of individual sovereignty.
I understand Muller’s frustration with the idiotic metrics which are employed in business and social policy. I have my own thesaurus of war stories about how such metrics have destroyed organisations and damaged individuals. And Muller is fighting against a real intellectual enemy - the idea that measurement of anything is an activity devoid of political judgment. This ideology, and it is just that, is the antithesis to Muller’s thesis. It seeks to establish that what we measure, principally in our choice of a metric of success, as some objective property of the thing measured. This is the foundation stone of scientism which the belief that individuals must submit to the rationality of reality, a reality determined by a consensus of experts. These experts may include a diverse set of skills - accountancy, economics, sociology, management or finance for example - but their aim is the same: to impose a standard, universal metric of success wherever they ply their influence. And Muller is right to resist this sort of intellectual totalitarianism.
But he is not right to substitute the arbitrariness of some group of experts with the arbitrariness of some individual’s experience. This is just another ideology. It strikes me that Muller takes the ideas of economic neo-liberalism to their philosophical outer limits. Margret Thatcher and Ronald Reagan didn’t believe that society existed; only individuals interacting did. Muller extends this to the realm of practical ideas from moral calculus to commercial decision-making. Discussion, mutual discovery, agreement and group learning are not things he can even consider in his ideological cage. How conflicts about what is important and therefore what should be measured in business, politics, and any cooperative effort are simply not his concern. What he calls ‘metric fixation’ is no different from ‘social awareness’. I am quite sure that Ayn Rand is alive and well in Muller’s garden shed.
Postscript: Muller implicitly presumes that social groups are fictional and should be given no status within his ideology of measurement. The opposing ideology, that only social conventions have meaning in measurement, is also prevalent. See here for a typical example: https://www.goodreads.com/review/show.... There are interesting philosophical implications of both views. For example ‘measurement socialists’ believe that the measurements they make are objective properties of the thing measured. While ‘measurement individualists’ like Muller believe that what is measured is entirely in the head of the one doing the measuring. Debates between the two groups are rarely edifying. For an alternative theory which avoids both the Scylla of collectivism and the Charybdis of individualism in measurement, see here: https://www.goodreads.com/review/show...
January 2, 2019
As I was reading the midsection of this book, I was thinking it was a chore--where "chore" means taking some notes--but an absolutely necessary one, given the revelatory nature. Then, the ending hit me with an emotional wallop. That could not have happened if I hadn't read on through the material.
The wallop was this: we have made an idol out of transparency. And this: our whole society is like a bad marriage. We don't know how to fix it, but we can't just divorce everybody.
Before I read this book, I had thought it was going to be pitting two writers I've valued against each other. For doesn't Daniel Kahneman talk about the usefulness of checklists and algorithms in slow thinking and good decisions? From the title alone (and not knowing what "metrics" signifies), I had thought Muller was going to be disagreeing. But no; metrics in this sense means measurement of outcome to render judgment of institutional productivity and success, as in school testing, crime reporting, and medical statistics.
The book is not a long, complicated one. Minus the notes, it's less than 200 pages. Instead of being a history, like the other Muller books I've read, it's an analysis and would-be remedy through better understanding that focuses on what usually can't be attended to since outside the spotlight of our attention (in part due to its interdisciplinary nature). Muller says his own experience as chair of his department at a private university is what drove him to write it. The book is for the frustrated people who work in all our institutions--universities, schools, the medical system, law enforcement, the military, and (even) business and finance. It's also for those who have been judged--those of us who've been deemed resistant to progress when we protested the disregard of our own expertise. And it's for all of us who are served by and need our institutions.
I read another Goodreads review saying Tyranny is less interesting than another related book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. But see: right in that title is "Big Data," the villain. That surely grabs the attention, like the polemics in scripture where whatever actually happened has been doctored into a tale of good and evil that meets the needs and aspirations of a target group. But Muller doesn't write like that.
We need a light shone on the problem, that is, a light of another sort than metrics furnishes, and not more good vs. bad guys.
Measurement in and of itself isn't bad; it's the uses to which it's put. Metrics has become so popular because we live in interesting times.
Society is in flux, and we haven't figured out pluralism. We Americans used to pride ourselves on having done so. We--and Western Europe--would look down on other societies whom we deemed less successful at being liberal democracies, but we underestimated the extent to which our success was based on our being more more homogeneous, or at any rate, or at any rate run by a homogeneous group that was characterized by shared values and mutual trust.
Now that we are more diverse along multiple dimensions, we have tried to be "more objective" by substituting metrics for missing levels of trust. In business, management became cut off from workers--I think here of the kinds of changes in companies portrayed in American Pastoral--and owners/shareholders from management. The application of metrics is a covert way of saying "We don't trust you" and of asserting control. But to do so meant reducing evaluation to the lowest common denominator and, often, changing the rules on what the purpose of the company is. Is it producing a fine-quality product? Or is it producing numbers that will tranquilize managers and pacify shareholders?
Then we take economic models that don't even work so well in business and apply them to the other systems lacking in the information conveyed by economic indicators like price or supply and demand.
Since the '60s, the Left was distrustful of society. Subsequently the Right wanted to see the bottom line. The Left became susceptible to pressures to justify government expenditures. Costs of education and medicine exploded relative to other parts of the economy, adding to the growing distrust of professionals. Now everyone wanted transparency. Everyone wants to see the bottom line, and that's what metrics promised.
According to Muller, sometimes the various goals at which we're aiming are mutually exclusive. If one goal for a college is to be more accessible, that means admitting students who are less well prepared. Then, in consequence, the graduation rate is likely to drop, not rise, nor are incomes for graduates going to be higher. Thus, demanding that the college simultaneously graduate more students and attain higher incomes for graduates isn't consistent with accessibility. Also, goals can have unexpected consequences. If, for example, we make job income of graduates a measure of success, we put pressure on institutions of higher learning to produce more investment bankers, consultants, and high-end attorneys. Do we need more? We're inadvertently measuring education only in terms of money-earning potential.
Notes from Muller's discussion of the public schools:
A coalition of the political left, right, business, and civil rights workers all wanted measurement. It was thought that the achievement gap was due to teachers' lack of professional ability. No Child Behind showed no results after ten years, and essentially none since the '70s. Unintended consequences: the problem isn't that the tests can't measure or diagnose children, but that they themselves became the criteria for judging schools. Test-taking skills became what was taught, rather than the subject that was to be measured. Under the pressure, creaming--eliminating the poor-functioning from the results--and cheating occurred, as well as teaching to the test, destroying the predictive ability of the tests. Race to the Top under Obama judged teachers as well as schools. Incentives--pay for performance (P4P) doesn't work, but schools went all in. These measures reward extrinsic motivation, not intrinsic, meaning the reason teachers became teachers--the desire to teach children. Paradox: better education widens, not shrinks, the achievement gap since educability is based on human capital. The catchy names signify pious hopes, not achievable goals. The emphasis on measuring English and math detracts from overall education and demoralizes teachers. The power to effect the wished-for change is not in them, but they are blamed.
The continuation of expensive metrics in the absence of achievable goals becomes virtue signalling.
Here's a pertinent article from the education columnist of the Atlanta Journal-Constitution. Date written in blog is July 9, 2018; published in print edition on July 23. https://www.myajc.com/blog/get-school...
Notes from Muller's discussion of medicine:
Nowhere are metrics more in vogue than in medicine. Metrics are helpful for information and diagnosis. They are also touted for cost control and effectiveness of services. Muller gives three examples of successful programs that were developed from inside by the professionals themselves and rewarded intrinsic motivation. Those successes, however, were applied punitively to other institutions. The institutions that developed the programs were already high-functioning systems. Theoretically, applications of such measures will make patients, doctors, and insurance companies act like consumers, but that hypothesis is not supported by the results of studies (done, moreover, by professionals who have a stake in supporting the hypotheses, so that their findings are all the more significant). Again P4P does not work, and represents the triumph of hope over experience. P4P has unintended consequences, such as neglect for conditions not being evaluated. Creaming and "treating to the test" occur. Medicine is treated as (only) a profit-making venture.
Do these results show the venality of human beings? I think what they show is the adaptive ability of human beings. We want to figure out what's going on; predict and control. If the rules change we try to adapt. If the measurement of one condition indicates its relative importance, we treat it rather than the condition that's not being measured. If readmissions are used against a hospital, outpatient services are expanded so that the "readmission" is now an "outpatient" service. What's going on often isn't blatant cheating but, rather, adjusting to the new rules, and, in that case, can you fault the institution? The unintended consequence of the form of measurement is to create new rules, to which we can expect adjustment. Or, as Muller puts it, measurement changes behavior.
Measurement also is costly and time-consuming, and detracts from doing anything--from the missions of institutions. Measurement decreases creativity and risk-taking, e.g., doctors who avoid operating on the worst-off patients. Measurement will get at the low-hanging fruit, that is, surgeons whose patients are suffering disproportionately poor outcomes quit, but after that, little systemic improvement occurs despite the outlay of money and time.
Charities:
When we get into charities, measurement has become all about overhead vs. money spent on the mission, to the point of being counterproductive. Those who run charities should learn from the military's advances in measurement in the area of counterintelligence. Measure, not input--what the charity does--but outcome, based on observation and knowledge of local conditions, not a generalized and uninformative overview. Outcome is the bottom line, not what the charity does.
A negative example of measurement from the military: Vietnam-era body counts. American soldiers died trying to retrieve bodies to back up those body counts.
Even in business, where you might think some of these measures work since the goals are economic after all, P4P only works for rote types of tasks that lack intrinsic rewards. Otherwise, people want wider recognition and reward, not just for their numbers.
Muller ends with a list of predictable negative outcomes and a checklist on how to use metrics. He gave that info at the outset, and once I'd gone through the book, I understood it.
I found his "Excursis" on "When Transparency is the Enemy of Performance" to be particularly significant. It's very brief, and I was reading between the lines and tying it in with other ideas. I was reminded of the extensive 2016 Atlantic article on "How American Politics Went Insane"--same sort of thing. He's saying (like Daniel Kahneman) that the outcomes we desire also have downsides, and those we wish to avoid aren't without positive aspects. He's explaining why perfect transparency would be counterproductive in intimate relationships, politics, diplomacy, and intelligence work. The latter two, diplomacy and intelligence work, are obvious, I think; The Atlantic article is a long explanation regarding politics: https://www.theatlantic.com/magazine/...
As to marriage/relationships, imagine what it would be like if our every thought scrolled along our foreheads, as with closed-circuit TV.
WUMO comic strip for 12/30/2018
The book's been an important one for me. I hope it will remind me not to precipitously accept metrics-based judgments.
External reviews:
https://blogs.sciencemag.org/books/20...
http://timharford.com/2018/02/review-...
https://seekingalpha.com/article/4201...
Author interview (via email):
https://www.insidehighered.com/news/2...
The wallop was this: we have made an idol out of transparency. And this: our whole society is like a bad marriage. We don't know how to fix it, but we can't just divorce everybody.
Before I read this book, I had thought it was going to be pitting two writers I've valued against each other. For doesn't Daniel Kahneman talk about the usefulness of checklists and algorithms in slow thinking and good decisions? From the title alone (and not knowing what "metrics" signifies), I had thought Muller was going to be disagreeing. But no; metrics in this sense means measurement of outcome to render judgment of institutional productivity and success, as in school testing, crime reporting, and medical statistics.
The book is not a long, complicated one. Minus the notes, it's less than 200 pages. Instead of being a history, like the other Muller books I've read, it's an analysis and would-be remedy through better understanding that focuses on what usually can't be attended to since outside the spotlight of our attention (in part due to its interdisciplinary nature). Muller says his own experience as chair of his department at a private university is what drove him to write it. The book is for the frustrated people who work in all our institutions--universities, schools, the medical system, law enforcement, the military, and (even) business and finance. It's also for those who have been judged--those of us who've been deemed resistant to progress when we protested the disregard of our own expertise. And it's for all of us who are served by and need our institutions.
I read another Goodreads review saying Tyranny is less interesting than another related book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. But see: right in that title is "Big Data," the villain. That surely grabs the attention, like the polemics in scripture where whatever actually happened has been doctored into a tale of good and evil that meets the needs and aspirations of a target group. But Muller doesn't write like that.
We need a light shone on the problem, that is, a light of another sort than metrics furnishes, and not more good vs. bad guys.
Measurement in and of itself isn't bad; it's the uses to which it's put. Metrics has become so popular because we live in interesting times.
Society is in flux, and we haven't figured out pluralism. We Americans used to pride ourselves on having done so. We--and Western Europe--would look down on other societies whom we deemed less successful at being liberal democracies, but we underestimated the extent to which our success was based on our being more more homogeneous, or at any rate, or at any rate run by a homogeneous group that was characterized by shared values and mutual trust.
Now that we are more diverse along multiple dimensions, we have tried to be "more objective" by substituting metrics for missing levels of trust. In business, management became cut off from workers--I think here of the kinds of changes in companies portrayed in American Pastoral--and owners/shareholders from management. The application of metrics is a covert way of saying "We don't trust you" and of asserting control. But to do so meant reducing evaluation to the lowest common denominator and, often, changing the rules on what the purpose of the company is. Is it producing a fine-quality product? Or is it producing numbers that will tranquilize managers and pacify shareholders?
Then we take economic models that don't even work so well in business and apply them to the other systems lacking in the information conveyed by economic indicators like price or supply and demand.
Since the '60s, the Left was distrustful of society. Subsequently the Right wanted to see the bottom line. The Left became susceptible to pressures to justify government expenditures. Costs of education and medicine exploded relative to other parts of the economy, adding to the growing distrust of professionals. Now everyone wanted transparency. Everyone wants to see the bottom line, and that's what metrics promised.
According to Muller, sometimes the various goals at which we're aiming are mutually exclusive. If one goal for a college is to be more accessible, that means admitting students who are less well prepared. Then, in consequence, the graduation rate is likely to drop, not rise, nor are incomes for graduates going to be higher. Thus, demanding that the college simultaneously graduate more students and attain higher incomes for graduates isn't consistent with accessibility. Also, goals can have unexpected consequences. If, for example, we make job income of graduates a measure of success, we put pressure on institutions of higher learning to produce more investment bankers, consultants, and high-end attorneys. Do we need more? We're inadvertently measuring education only in terms of money-earning potential.
Notes from Muller's discussion of the public schools:
A coalition of the political left, right, business, and civil rights workers all wanted measurement. It was thought that the achievement gap was due to teachers' lack of professional ability. No Child Behind showed no results after ten years, and essentially none since the '70s. Unintended consequences: the problem isn't that the tests can't measure or diagnose children, but that they themselves became the criteria for judging schools. Test-taking skills became what was taught, rather than the subject that was to be measured. Under the pressure, creaming--eliminating the poor-functioning from the results--and cheating occurred, as well as teaching to the test, destroying the predictive ability of the tests. Race to the Top under Obama judged teachers as well as schools. Incentives--pay for performance (P4P) doesn't work, but schools went all in. These measures reward extrinsic motivation, not intrinsic, meaning the reason teachers became teachers--the desire to teach children. Paradox: better education widens, not shrinks, the achievement gap since educability is based on human capital. The catchy names signify pious hopes, not achievable goals. The emphasis on measuring English and math detracts from overall education and demoralizes teachers. The power to effect the wished-for change is not in them, but they are blamed.
The continuation of expensive metrics in the absence of achievable goals becomes virtue signalling.
Here's a pertinent article from the education columnist of the Atlanta Journal-Constitution. Date written in blog is July 9, 2018; published in print edition on July 23. https://www.myajc.com/blog/get-school...
Notes from Muller's discussion of medicine:
Nowhere are metrics more in vogue than in medicine. Metrics are helpful for information and diagnosis. They are also touted for cost control and effectiveness of services. Muller gives three examples of successful programs that were developed from inside by the professionals themselves and rewarded intrinsic motivation. Those successes, however, were applied punitively to other institutions. The institutions that developed the programs were already high-functioning systems. Theoretically, applications of such measures will make patients, doctors, and insurance companies act like consumers, but that hypothesis is not supported by the results of studies (done, moreover, by professionals who have a stake in supporting the hypotheses, so that their findings are all the more significant). Again P4P does not work, and represents the triumph of hope over experience. P4P has unintended consequences, such as neglect for conditions not being evaluated. Creaming and "treating to the test" occur. Medicine is treated as (only) a profit-making venture.
Do these results show the venality of human beings? I think what they show is the adaptive ability of human beings. We want to figure out what's going on; predict and control. If the rules change we try to adapt. If the measurement of one condition indicates its relative importance, we treat it rather than the condition that's not being measured. If readmissions are used against a hospital, outpatient services are expanded so that the "readmission" is now an "outpatient" service. What's going on often isn't blatant cheating but, rather, adjusting to the new rules, and, in that case, can you fault the institution? The unintended consequence of the form of measurement is to create new rules, to which we can expect adjustment. Or, as Muller puts it, measurement changes behavior.
Measurement also is costly and time-consuming, and detracts from doing anything--from the missions of institutions. Measurement decreases creativity and risk-taking, e.g., doctors who avoid operating on the worst-off patients. Measurement will get at the low-hanging fruit, that is, surgeons whose patients are suffering disproportionately poor outcomes quit, but after that, little systemic improvement occurs despite the outlay of money and time.
Charities:
When we get into charities, measurement has become all about overhead vs. money spent on the mission, to the point of being counterproductive. Those who run charities should learn from the military's advances in measurement in the area of counterintelligence. Measure, not input--what the charity does--but outcome, based on observation and knowledge of local conditions, not a generalized and uninformative overview. Outcome is the bottom line, not what the charity does.
A negative example of measurement from the military: Vietnam-era body counts. American soldiers died trying to retrieve bodies to back up those body counts.
Even in business, where you might think some of these measures work since the goals are economic after all, P4P only works for rote types of tasks that lack intrinsic rewards. Otherwise, people want wider recognition and reward, not just for their numbers.
Muller ends with a list of predictable negative outcomes and a checklist on how to use metrics. He gave that info at the outset, and once I'd gone through the book, I understood it.
I found his "Excursis" on "When Transparency is the Enemy of Performance" to be particularly significant. It's very brief, and I was reading between the lines and tying it in with other ideas. I was reminded of the extensive 2016 Atlantic article on "How American Politics Went Insane"--same sort of thing. He's saying (like Daniel Kahneman) that the outcomes we desire also have downsides, and those we wish to avoid aren't without positive aspects. He's explaining why perfect transparency would be counterproductive in intimate relationships, politics, diplomacy, and intelligence work. The latter two, diplomacy and intelligence work, are obvious, I think; The Atlantic article is a long explanation regarding politics: https://www.theatlantic.com/magazine/...
As to marriage/relationships, imagine what it would be like if our every thought scrolled along our foreheads, as with closed-circuit TV.
WUMO comic strip for 12/30/2018
The book's been an important one for me. I hope it will remind me not to precipitously accept metrics-based judgments.
External reviews:
https://blogs.sciencemag.org/books/20...
http://timharford.com/2018/02/review-...
https://seekingalpha.com/article/4201...
Author interview (via email):
https://www.insidehighered.com/news/2...
April 18, 2018
I was not surprised by historian Jerry Z. Muller's comments about "metric fixation" in his illuminating new book, The Tyranny of Metrics.
Some years ago the chairman and CEO of a Fortune 500 company remarked to me that nobody, not even Jack Welch, the then-idolized leader of General Electric, could possibly turn in solid revenue and profit increases steadily, quarter-after-quarter, year-after-year, through ups and downs in the market. It was clear to him that somebody was cooking the books at GE. I had been in business long enough then to understand how easy it is to shift sales and revenue from one quarter or year to the next and how other steps could be taken to fudge the numbers. (I was also aware of Welch's reputation as "Neutron Jack" for his ruthless practice of "downsizing" to increase profits, a practice that can also be timed to give the appearance of steadily increasing profits.) It had been clear to me for many years that numbers can lie.
In his new book, Muller tackles our society's obsession with metrics and accountability. However, "[t]his book is not about the evils of measuring," he writes. "It is about the unintended consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics but metric fixation." In just 200 pages, Muller assesses the use and misuse of metrics through case studies drawn from a wide range of fields: colleges and universities, K-12 education, medicine, policing, the military, business and finance, and philanthropy and foreign aid. The research he cites, and the examples he chooses, are compelling.
I've had my own intimate experience with the misuse of metrics in both philanthropy and foreign aid. More than thirty years ago, when I was running a consulting agency that raised money for nonprofit organizations, I wrote and spoke to whoever would listen about the absurdity of measuring nonprofit performance on the basis of its fundraising costs. Years later, when I became involved in consulting with NGOs in developing countries, I saw for myself the folly of the metrics obsession that had seized hold of the international development community after Bill Gates began proselytizing on the subject. That fixation on the numbers forced far too many charities and government agencies to funnel money toward easily measured but trivial or even irrelevant programs while ignoring others that might actually have some positive impact in the field.
Muller sums up the problem nicely. "There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive." Amen.
Some of the examples Muller cites are familiar to the public at large. "Teaching to the test," for example. The tendency of some surgeons to decline to operate in difficult cases because failure would lower their success ratios. And the tendency of police under pressure from politicians to make the numbers look better by classifying felonies as misdemeanors or altogether refusing to write up crimes. These are just a few of the many sad ways that our obsession with accountability distorts our understanding of the world around us.
Among the many unintended consequences of the misuse of metrics that Muller cites are the following:
** Inducing people whose performance is measured to divert their efforts to what gets measured;
** Promoting short-termism (as in Wall Street's obsessive preoccupation with quarterly earnings reports at the expense of companies' long-term health);
** Discouraging innovation and risk-taking;
** Sidetracking nonprofit staff members (or corporate employees, for that matter) from focusing on the mission that motivates them; and
** Forcing employees to spend time logging data instead of doing their jobs (a requirement that was a major factor in convincing my brother to close his psychiatric practice many years ago).
Muller concludes The Tyranny of Metrics with a useful checklist of ten questions that any manager should ask when considering the application of metrics at work.
Some years ago the chairman and CEO of a Fortune 500 company remarked to me that nobody, not even Jack Welch, the then-idolized leader of General Electric, could possibly turn in solid revenue and profit increases steadily, quarter-after-quarter, year-after-year, through ups and downs in the market. It was clear to him that somebody was cooking the books at GE. I had been in business long enough then to understand how easy it is to shift sales and revenue from one quarter or year to the next and how other steps could be taken to fudge the numbers. (I was also aware of Welch's reputation as "Neutron Jack" for his ruthless practice of "downsizing" to increase profits, a practice that can also be timed to give the appearance of steadily increasing profits.) It had been clear to me for many years that numbers can lie.
In his new book, Muller tackles our society's obsession with metrics and accountability. However, "[t]his book is not about the evils of measuring," he writes. "It is about the unintended consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics but metric fixation." In just 200 pages, Muller assesses the use and misuse of metrics through case studies drawn from a wide range of fields: colleges and universities, K-12 education, medicine, policing, the military, business and finance, and philanthropy and foreign aid. The research he cites, and the examples he chooses, are compelling.
I've had my own intimate experience with the misuse of metrics in both philanthropy and foreign aid. More than thirty years ago, when I was running a consulting agency that raised money for nonprofit organizations, I wrote and spoke to whoever would listen about the absurdity of measuring nonprofit performance on the basis of its fundraising costs. Years later, when I became involved in consulting with NGOs in developing countries, I saw for myself the folly of the metrics obsession that had seized hold of the international development community after Bill Gates began proselytizing on the subject. That fixation on the numbers forced far too many charities and government agencies to funnel money toward easily measured but trivial or even irrelevant programs while ignoring others that might actually have some positive impact in the field.
Muller sums up the problem nicely. "There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring; what gets measured may have no relationship to what we really want to know. The costs of measuring may be greater than the benefits. The things that get measured may draw effort away from the things we really care about. And measurement may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive." Amen.
Some of the examples Muller cites are familiar to the public at large. "Teaching to the test," for example. The tendency of some surgeons to decline to operate in difficult cases because failure would lower their success ratios. And the tendency of police under pressure from politicians to make the numbers look better by classifying felonies as misdemeanors or altogether refusing to write up crimes. These are just a few of the many sad ways that our obsession with accountability distorts our understanding of the world around us.
Among the many unintended consequences of the misuse of metrics that Muller cites are the following:
** Inducing people whose performance is measured to divert their efforts to what gets measured;
** Promoting short-termism (as in Wall Street's obsessive preoccupation with quarterly earnings reports at the expense of companies' long-term health);
** Discouraging innovation and risk-taking;
** Sidetracking nonprofit staff members (or corporate employees, for that matter) from focusing on the mission that motivates them; and
** Forcing employees to spend time logging data instead of doing their jobs (a requirement that was a major factor in convincing my brother to close his psychiatric practice many years ago).
Muller concludes The Tyranny of Metrics with a useful checklist of ten questions that any manager should ask when considering the application of metrics at work.
May 23, 2018
It's good for what it is and I think it's right, but there is nothing new here. Read Cathy O'Neills Weapons of Math Destruction, which is better and more interesting.
October 4, 2018
A sustained critique of what Muller calls "metric fixation" is necessary and long-overdue. Muller captures how the overuse and dependency on metrics and quantification, particularly for evaluating performance, is hollowing out and damaging key social, cultural, political, commercial, and philanthropic institutions. Spelling this out in a succinct and accessible manner for an audience broader than academia is vital. For that, Muller should be commended.
However, the book is not as hard-hitting as it needs to be, it reads as a rushed effort, and its broader analysis is superficial, at least in terms of its contextualisation. Worse, it overlooks or downplays key structural and ideological factors. Nowhere, for instance, is neoliberalism mentioned or really even alluded to. Muller is right to highlight that there's no single cause, but some causes are more significant and more influential than others. Economic transformations and the social and political dominance of the fiscal right should be given much more weight than they are. Moreover, the critique of transparency is wrongheaded and the examples used are lazy. It is here that Muller's ideological convictions start to become abundantly clear. The problem, according to Muller, isn't neoliberal capitalism but an obsession with transparency coming from both the left and the right. This isn't just an aside - this is the foundation from which the entire argument is built.
So while the book nicely captures some of the consequences of "metric fixation", it fails to adequately account for its underlying causes. True, Muller suggests that we can redress our overuse of metrics and offers the reader a guide on how and when to use them appropriately. But without pointing readers to the broader structures that are actually encouraging this fixation in the first place, the tyranny will persist. Again, a sustained critique is needed more than ever and it is great that work has begun. Regardless, we need much more than a conservative exposition of a problem stemming from conservatism.
For more critical insights, read Mark Fisher's Capitalist Realism, especially Chapter Six ('All that is solid melts into PR') and Annie McClanahan's critique of credit scoring in chapter two of Dead Pledges: Debt, Crisis, and Twenty-First Century Culture.
However, the book is not as hard-hitting as it needs to be, it reads as a rushed effort, and its broader analysis is superficial, at least in terms of its contextualisation. Worse, it overlooks or downplays key structural and ideological factors. Nowhere, for instance, is neoliberalism mentioned or really even alluded to. Muller is right to highlight that there's no single cause, but some causes are more significant and more influential than others. Economic transformations and the social and political dominance of the fiscal right should be given much more weight than they are. Moreover, the critique of transparency is wrongheaded and the examples used are lazy. It is here that Muller's ideological convictions start to become abundantly clear. The problem, according to Muller, isn't neoliberal capitalism but an obsession with transparency coming from both the left and the right. This isn't just an aside - this is the foundation from which the entire argument is built.
So while the book nicely captures some of the consequences of "metric fixation", it fails to adequately account for its underlying causes. True, Muller suggests that we can redress our overuse of metrics and offers the reader a guide on how and when to use them appropriately. But without pointing readers to the broader structures that are actually encouraging this fixation in the first place, the tyranny will persist. Again, a sustained critique is needed more than ever and it is great that work has begun. Regardless, we need much more than a conservative exposition of a problem stemming from conservatism.
For more critical insights, read Mark Fisher's Capitalist Realism, especially Chapter Six ('All that is solid melts into PR') and Annie McClanahan's critique of credit scoring in chapter two of Dead Pledges: Debt, Crisis, and Twenty-First Century Culture.
January 21, 2018
Too many metrics are killing productivity
During the great famine in China, local officials yanked plants out of the ground and raced ahead of Mao’s itinerary to plant them at his next stop, thus fooling the chairman into thinking bumper crops were everywhere. They kept their jobs and their heads, but as a result, Mao declared there was no famine and refused to release government stores of rice to the starving. Fifty million died. This is the poster child for metrics.
The Tyranny of Metrics is an anecdote-filled examination of how industry, services and government have gone overboard with self-defeating metrics. Muller comes at it from a personal angle as well as a piece of research. He is a history professor, bogged down in the mindless world of forms and stats. It is crippling his own research and networking time, and does nothing to further his field, his career or his own satisfaction.
He says there are three components to the metrics fixation:
-replacing judgment with numerical values
-publicizing numbers to make institutions transparent and accountable
-managing people is to give them numerical targets and evaluations
Ironically, despite all this measurement, productivity is lagging. Make that because of.
This is the Soviet system, where the central authority set goals for factories, services and bureaucracies, and in which workers find ways to achieve those goals at the cost of quality, service, risk-taking and innovation. Examples are students who can pass tests but have little knowledge, subprime mortgages on false applications, police who hide crimes, and banks that open phony accounts for existing customers to meet quotas. Even the federal Government Accounting Office (GAO) got renamed Government Accountability Office, (which managed to prevent confusion by recycling its initials). There is no better way to see this in action than Ken Loach’s Palme D’Or-winning film I Daniel Blake, which shows the UK’s performance-obsessed unemployment office in action. The objective is to spend as little time as possible with customers, and refuse as many as possible for the slightest misstep.
Muller proves his points with perverse outcomes galore. Hospitals keep dying patients alive for 30 days because that’s the measure of treatment success. Surgeons decline to operate on iffy cases for fear of them dying too soon. Creative teachers quit and move to private schools where teaching to the test and abandoning real learning are not the main activities. Universities cripple research in favor of reporting on every aspect of education, right up to how much each graduate is earning ten years later. All their efforts are focused on moving up the rankings in the various, competing lists. The cheating by teachers, administrators and politicians on No Child Left Behind are legendary.
Muller ends with ten points to ponder when you find yourself in an evaluation situation. A simple rule of thumb, he says, is that if the object of your measurements can be by altered by the process of measurement, the results can be less than accurate.
A 2006 survey of HR managers found that metrics and rankings “resulted in lower productivity, inequity, skepticism, decreased employee engagement, reduced collaboration, damage to morale and mistrust in leadership.” So more and more firms do it every year.
David Wineberg
During the great famine in China, local officials yanked plants out of the ground and raced ahead of Mao’s itinerary to plant them at his next stop, thus fooling the chairman into thinking bumper crops were everywhere. They kept their jobs and their heads, but as a result, Mao declared there was no famine and refused to release government stores of rice to the starving. Fifty million died. This is the poster child for metrics.
The Tyranny of Metrics is an anecdote-filled examination of how industry, services and government have gone overboard with self-defeating metrics. Muller comes at it from a personal angle as well as a piece of research. He is a history professor, bogged down in the mindless world of forms and stats. It is crippling his own research and networking time, and does nothing to further his field, his career or his own satisfaction.
He says there are three components to the metrics fixation:
-replacing judgment with numerical values
-publicizing numbers to make institutions transparent and accountable
-managing people is to give them numerical targets and evaluations
Ironically, despite all this measurement, productivity is lagging. Make that because of.
This is the Soviet system, where the central authority set goals for factories, services and bureaucracies, and in which workers find ways to achieve those goals at the cost of quality, service, risk-taking and innovation. Examples are students who can pass tests but have little knowledge, subprime mortgages on false applications, police who hide crimes, and banks that open phony accounts for existing customers to meet quotas. Even the federal Government Accounting Office (GAO) got renamed Government Accountability Office, (which managed to prevent confusion by recycling its initials). There is no better way to see this in action than Ken Loach’s Palme D’Or-winning film I Daniel Blake, which shows the UK’s performance-obsessed unemployment office in action. The objective is to spend as little time as possible with customers, and refuse as many as possible for the slightest misstep.
Muller proves his points with perverse outcomes galore. Hospitals keep dying patients alive for 30 days because that’s the measure of treatment success. Surgeons decline to operate on iffy cases for fear of them dying too soon. Creative teachers quit and move to private schools where teaching to the test and abandoning real learning are not the main activities. Universities cripple research in favor of reporting on every aspect of education, right up to how much each graduate is earning ten years later. All their efforts are focused on moving up the rankings in the various, competing lists. The cheating by teachers, administrators and politicians on No Child Left Behind are legendary.
Muller ends with ten points to ponder when you find yourself in an evaluation situation. A simple rule of thumb, he says, is that if the object of your measurements can be by altered by the process of measurement, the results can be less than accurate.
A 2006 survey of HR managers found that metrics and rankings “resulted in lower productivity, inequity, skepticism, decreased employee engagement, reduced collaboration, damage to morale and mistrust in leadership.” So more and more firms do it every year.
David Wineberg
December 22, 2021
I failed to love this book, for several reasons:
- the language is very dry, feels like some weasel-consulting presentation
- once the author started starting, he couldn't stop ;P literally, ~20% of the book is its intro
- the content is very repetitive
- yes, they are a few interesting examples (primarily in the education area), but fewer than one could expect
- I hoped for some more structured approach to metrics
- Is there anything I liked about this book? Well, the chapter about transparency was quite interesting
If you're looking for a good book about metrics and how (not) to use them, you should probably look somewhere else ;(
- the language is very dry, feels like some weasel-consulting presentation
- once the author started starting, he couldn't stop ;P literally, ~20% of the book is its intro
- the content is very repetitive
- yes, they are a few interesting examples (primarily in the education area), but fewer than one could expect
- I hoped for some more structured approach to metrics
- Is there anything I liked about this book? Well, the chapter about transparency was quite interesting
If you're looking for a good book about metrics and how (not) to use them, you should probably look somewhere else ;(
January 14, 2023
3.5/5
O título “The Tyranny of Metrics” do professor Jerry Z. Muller é indissociável do título “The Tyranny of Merit” do imensamente mais conhecido professor Michael J. Sandel. Mas em defesa de Muller, o seu livro é de 2018, e o de Sandel de 2020. Mas a aproximação não se fica pelos títulos, vai ao fundo dos dois tópicos eleitos: mérito e métricas. Não as colocando lado a lado, mas antes em lados opostos, diga-se lados políticos. Porque se o “mérito” é o santo graal da esquerda, o motor da crença messiânica de que todos podemos ser tudo e fazer tudo desde que nos esforcemos. As métricas são o Santo Graal da direita, em que tudo tem de ser medido para que tudo possa ser transparente, porque só quando ajustado pela medida objetiva se pode eliminar qualquer vestígio de viés humano.
continuar a ler no blog:
https://virtual-illusion.blogspot.com...
O título “The Tyranny of Metrics” do professor Jerry Z. Muller é indissociável do título “The Tyranny of Merit” do imensamente mais conhecido professor Michael J. Sandel. Mas em defesa de Muller, o seu livro é de 2018, e o de Sandel de 2020. Mas a aproximação não se fica pelos títulos, vai ao fundo dos dois tópicos eleitos: mérito e métricas. Não as colocando lado a lado, mas antes em lados opostos, diga-se lados políticos. Porque se o “mérito” é o santo graal da esquerda, o motor da crença messiânica de que todos podemos ser tudo e fazer tudo desde que nos esforcemos. As métricas são o Santo Graal da direita, em que tudo tem de ser medido para que tudo possa ser transparente, porque só quando ajustado pela medida objetiva se pode eliminar qualquer vestígio de viés humano.
continuar a ler no blog:
https://virtual-illusion.blogspot.com...
April 3, 2018
I really enjoyed Muller's 'The Tyranny of Metrics,' and felt it delivered exactly the type of read I thought it would. In a world that is becoming more digitized and people are pushing for increased transparency, Muller writes about instances where it appears we've become too reliant on blindly following the data behind metrics without understanding the harm that may come.
To be clear, Muller is not against metrics but he lays out very good arguments for why they may not always be appropriate, how they get gamed or manipulated, and why they are useless unless they are able to be judged subjectively by people with intimate knowledge of the field and where the shortcoming of those metrics might be. Stretching across areas like schools, police departments, militaries, governments, hospitals, charities and more, Muller gives great examples that are directly applicable and always thought provoking. If anything, I do wish there were a bit more meat on the bones of some of his examples with expanded discussion-- but perhaps there will be a follow up!
The only part that seemed a bit off was towards the end when he spent a short chapter equating modern society's growing need for transparency and metrics to the need for politicians, governments etc. to exercise more privacy and reduce transparency. The argument he was making was about how if every piece of email of a public official is made public, then they will be less likely to speak to their colleagues/opponents with candor and less able to reach the optimal compromises because they will have to worry about these negotiations being made public. While some of his arguments felt valid, it made me squirm in my seat a bit the more he pushed for the right of government to operate with reduced transparency.
All in all, though, I really enjoyed this book and would recommend it to anyone who lives in the world of data and metrics like I do. Muller provides a good mirror with which to look at how you treat metrics and consider whether you need to incorporate more of the 'art' into the 'science' of data and metrics.
To be clear, Muller is not against metrics but he lays out very good arguments for why they may not always be appropriate, how they get gamed or manipulated, and why they are useless unless they are able to be judged subjectively by people with intimate knowledge of the field and where the shortcoming of those metrics might be. Stretching across areas like schools, police departments, militaries, governments, hospitals, charities and more, Muller gives great examples that are directly applicable and always thought provoking. If anything, I do wish there were a bit more meat on the bones of some of his examples with expanded discussion-- but perhaps there will be a follow up!
The only part that seemed a bit off was towards the end when he spent a short chapter equating modern society's growing need for transparency and metrics to the need for politicians, governments etc. to exercise more privacy and reduce transparency. The argument he was making was about how if every piece of email of a public official is made public, then they will be less likely to speak to their colleagues/opponents with candor and less able to reach the optimal compromises because they will have to worry about these negotiations being made public. While some of his arguments felt valid, it made me squirm in my seat a bit the more he pushed for the right of government to operate with reduced transparency.
All in all, though, I really enjoyed this book and would recommend it to anyone who lives in the world of data and metrics like I do. Muller provides a good mirror with which to look at how you treat metrics and consider whether you need to incorporate more of the 'art' into the 'science' of data and metrics.
June 14, 2019
Oh, the metrics you will use! Mueller pointed out the problems of over-using metrics in society, especially when it is used for compensation purposes, and when assigned top-down without consultation with people who are actually involved in the work.
1. What is measurable may not be important, and what is important may not be measurable.
2. Metrics focus people’s attention to what is measured, to the neglect of everything that is not.
3. Metrics used for internal reporting is generally fine, but when used by other agencies for the purpose of ‘improvement’ is often fraught with problems. This leads to gaming and sometimes outright fraud.
4. Metrics avoid the really hard problems and focus on easily measurable ones.
5. Metrics encourage short-termism, and discourage risk taking, for new products and services are by definition not on the metrics.
6. Metrics are always backward looking and are useless in highly fluid situations such as counter-insurgency.
7. Some metrics penalise schools and hospitals that are helping the most needy and desperate and thus make the problem worse.
8. Sometimes problems are beyond the workers’ control and it can be unfair to penalise them for poor outcomes.
He shared some examples:
1. Medicine: when 30-day mortality of hospitals is tracked, some hospitals artificially prolong the life of patients to 31 days, and surgeons start to turn away the sickest patients. When re-admission rates are tracked, some will put patients who return in the emergency ward.
2. Schools: schools will take the best students, and ask the weaker ones to leave.
3. Crime: when crime rates need to come down, police can reclassify cases to less serious ones.
4. Public companies: when earnings per share is tracked and executives rewarded according to stock price, CEOs cut staff (even if it causes the core support to collapse), and raise prices (such as the infamous Epipen).
5. Banks: when impossible targets are set, some staff would commit fraud, like Wells Fargo.
Solution:
1. Realise there are unintended consequences of any metrics used.
2. Acknowledge that not everything can be improved and some intervention makes things worse.
3. Involve the people who are actually on the ground doing work when devising metrics.
A solid 5 star book because I know what he describes to happen in the real world.
1. What is measurable may not be important, and what is important may not be measurable.
2. Metrics focus people’s attention to what is measured, to the neglect of everything that is not.
3. Metrics used for internal reporting is generally fine, but when used by other agencies for the purpose of ‘improvement’ is often fraught with problems. This leads to gaming and sometimes outright fraud.
4. Metrics avoid the really hard problems and focus on easily measurable ones.
5. Metrics encourage short-termism, and discourage risk taking, for new products and services are by definition not on the metrics.
6. Metrics are always backward looking and are useless in highly fluid situations such as counter-insurgency.
7. Some metrics penalise schools and hospitals that are helping the most needy and desperate and thus make the problem worse.
8. Sometimes problems are beyond the workers’ control and it can be unfair to penalise them for poor outcomes.
He shared some examples:
1. Medicine: when 30-day mortality of hospitals is tracked, some hospitals artificially prolong the life of patients to 31 days, and surgeons start to turn away the sickest patients. When re-admission rates are tracked, some will put patients who return in the emergency ward.
2. Schools: schools will take the best students, and ask the weaker ones to leave.
3. Crime: when crime rates need to come down, police can reclassify cases to less serious ones.
4. Public companies: when earnings per share is tracked and executives rewarded according to stock price, CEOs cut staff (even if it causes the core support to collapse), and raise prices (such as the infamous Epipen).
5. Banks: when impossible targets are set, some staff would commit fraud, like Wells Fargo.
Solution:
1. Realise there are unintended consequences of any metrics used.
2. Acknowledge that not everything can be improved and some intervention makes things worse.
3. Involve the people who are actually on the ground doing work when devising metrics.
A solid 5 star book because I know what he describes to happen in the real world.
May 26, 2018
The Tyranny of Metrics is a clear expression of mainly the deleterious effects of using metrics in the workplace - where counting shouldn't be counted on.
"Dan Cable and Freek Vermeulen of the London Business School recall many of the problems we have explored: the depressive effect of performance pay on creativity; the propensity to cook the books; the inevitable imperfections of the measurement instruments; the difficulty of defining long-term performance; and the tendency for extrinsic motivation to crowd out intrinsic motivation" (138).
In other words, what you measure is what you get - so be sure you know what you want.
"Dan Cable and Freek Vermeulen of the London Business School recall many of the problems we have explored: the depressive effect of performance pay on creativity; the propensity to cook the books; the inevitable imperfections of the measurement instruments; the difficulty of defining long-term performance; and the tendency for extrinsic motivation to crowd out intrinsic motivation" (138).
In other words, what you measure is what you get - so be sure you know what you want.
September 21, 2022
A sensible enumeration of the ways in which organizations chase bad metrics, with a checklist at the end for evaluating whether a metric is a good idea or not. The main takeaway is that metrics are helpful when they’re used for understanding but not for goal-setting or rewards (e.g., metrics can identify surgical procedures that may be more useful but surgeons shouldn’t be paid based on the number of surgeries they perform or the success rate, which can lead to them turning away sicker patients). Too often a metric is a poor proxy for the thing it intends to measure (e.g., body count for battlefield progress), and organizations spend too much time on the data collection and documentation of proliferating metrics. Metrics that are overly simple divert resources from potentially more worthwhile but less countable efforts (e.g., teaching to the test, rather than teaching students how to love learning).
As a data scientist, I’ve been trapped in metric hell more than once, imploring leads to focus on triangulating the more nuanced findings from both qualitative and quantitative research instead, but being steered toward goals that are easier to count. Muller makes a great point for how demoralizing this can be for workers who believe strongly in the fundamental goals of an organization and want to see it help people.
This book was clearly written by a historian; many of the examples are from 30-50 years ago, but they’re still relevant. The book lacked gender and cultural diversity; the vast majority of experts cited were white men. The author also deadnamed Chelsea Manning, despite referencing her (correctly named) Wikipedia page, though perhaps that was an editorial oversight. Despite these issues, the book is worthwhile reading for anyone trying to measure the success of an organization.
As a data scientist, I’ve been trapped in metric hell more than once, imploring leads to focus on triangulating the more nuanced findings from both qualitative and quantitative research instead, but being steered toward goals that are easier to count. Muller makes a great point for how demoralizing this can be for workers who believe strongly in the fundamental goals of an organization and want to see it help people.
This book was clearly written by a historian; many of the examples are from 30-50 years ago, but they’re still relevant. The book lacked gender and cultural diversity; the vast majority of experts cited were white men. The author also deadnamed Chelsea Manning, despite referencing her (correctly named) Wikipedia page, though perhaps that was an editorial oversight. Despite these issues, the book is worthwhile reading for anyone trying to measure the success of an organization.
May 19, 2018
The title of this book suggests that it could be a polemic of sorts against measurement, more specifically "metrics" – something dismissed early in the piece by the author. Muller presents the idea, backed up by research and case studies, that the use of metrics to guide management of everything from management to education, the military to charities often achieves the opposite of its stated aim.
He begins with his own experience as a dean in a university faculty and the growing pressures to justify everything according to numbers and algorithms to such an extend that the actual purpose of the faculty – educating people – gets lost in an avalanche of form-filling and spreadsheets, results in the employment of more and more administrative staff to cater for the numerical information generated, assesses academic staff according to criteria that have little if anything to do with scholarship, sanctioning or firing those that don't meet these standards.
He notes that these processes also result in the system being gamed, so that employees meet the standards and so their jobs. Examples are given from several sectors: hospitals where surgeons to the easy cases, not the more difficult where someone might die or reclassifying outpatients and the like, as well as various strategies in the military and education as well as in business.
In the local news this morning, there's a piece about how employees of a bank which had a scheme to have primary school children put money in those accounts (their own or the bank's) which had no funds in them 30 days after being opened. They did this because otherwise they would not receive a particular reward related to the bank's incentive system. The bank found out about this and sanctioned no-one, nor did they notify the regulators. A currently-running Royal Commission into the finance industry has repeatedly exposed all sorts of gaming the syatem in this way.
A presumption of this approach is that people can only be extrinsically motivated i.e. by money and that intrinsic motivation, such as that you would hope to see in education, medicine, government and the military, as well as financial institutions. If you've been unemployed in Australia or the UK in particular, then this method has been applied to you, and those you have to consult, including outsourced "employment" entities, or the privatised education system where training is being offered.
Muller's book has a continuing theme of pay for performance, which some here want to apply to teachers as a motivator, and he demonstrates that this idea really achieves the opposite, from low-level jobs to CEOs and senior executives, because the process takes those engaged in it away from the aims of the organisation or their profession. I must admit that this doesn't surprise me at all,and is painfully obvious, from the time I encountered it in my workplace in the late 1980s (and no, it didn't apply to me: I was just asked to work longer hours without being paid).
As an historian, Muller provides background to this idea, which has come and gone over the years under the rubric of "scientific management" a downside of which included the decline of interest in the work put in front of you, because the individual, their knowledge, experience and expertise isn't valued as input or perhaps in principle.
Muller is an excellent writer, particularly in the first part, which he describes as more theoretical, the second part being given over to examples from various fields. It doesn't take long to read, and because it's not a polemic (he does provide some pages of remedial suggestions) the prose is also matter-of-fact and so largely unobtrusive. I'd never heard of him before, and I'm tempted to look for his earlier books, to see what I can find out from them and have a relaxing read as well.
He begins with his own experience as a dean in a university faculty and the growing pressures to justify everything according to numbers and algorithms to such an extend that the actual purpose of the faculty – educating people – gets lost in an avalanche of form-filling and spreadsheets, results in the employment of more and more administrative staff to cater for the numerical information generated, assesses academic staff according to criteria that have little if anything to do with scholarship, sanctioning or firing those that don't meet these standards.
He notes that these processes also result in the system being gamed, so that employees meet the standards and so their jobs. Examples are given from several sectors: hospitals where surgeons to the easy cases, not the more difficult where someone might die or reclassifying outpatients and the like, as well as various strategies in the military and education as well as in business.
In the local news this morning, there's a piece about how employees of a bank which had a scheme to have primary school children put money in those accounts (their own or the bank's) which had no funds in them 30 days after being opened. They did this because otherwise they would not receive a particular reward related to the bank's incentive system. The bank found out about this and sanctioned no-one, nor did they notify the regulators. A currently-running Royal Commission into the finance industry has repeatedly exposed all sorts of gaming the syatem in this way.
A presumption of this approach is that people can only be extrinsically motivated i.e. by money and that intrinsic motivation, such as that you would hope to see in education, medicine, government and the military, as well as financial institutions. If you've been unemployed in Australia or the UK in particular, then this method has been applied to you, and those you have to consult, including outsourced "employment" entities, or the privatised education system where training is being offered.
Muller's book has a continuing theme of pay for performance, which some here want to apply to teachers as a motivator, and he demonstrates that this idea really achieves the opposite, from low-level jobs to CEOs and senior executives, because the process takes those engaged in it away from the aims of the organisation or their profession. I must admit that this doesn't surprise me at all,and is painfully obvious, from the time I encountered it in my workplace in the late 1980s (and no, it didn't apply to me: I was just asked to work longer hours without being paid).
As an historian, Muller provides background to this idea, which has come and gone over the years under the rubric of "scientific management" a downside of which included the decline of interest in the work put in front of you, because the individual, their knowledge, experience and expertise isn't valued as input or perhaps in principle.
Muller is an excellent writer, particularly in the first part, which he describes as more theoretical, the second part being given over to examples from various fields. It doesn't take long to read, and because it's not a polemic (he does provide some pages of remedial suggestions) the prose is also matter-of-fact and so largely unobtrusive. I'd never heard of him before, and I'm tempted to look for his earlier books, to see what I can find out from them and have a relaxing read as well.
October 18, 2019
I would consider myself fairly bullish on metrics, tracking and organization in general, so I took the time to listen to Muller’s counter-argument. Most of it boils down to Goodheart’s Law ("When a measure becomes a target, it ceases to be a good measure”). What really surprised me and I deeply appreciated was his final section arguing against transparency, which is very much a core value I strongly believe in. He makes a very persuasive case of the vital role privacy makes in our lives, as well as the drawbacks of transparency in organizations and government. Very very thought-provoking, and written really well while still being very short. One of the top quotes at the end summarizes the book best: "Remember that sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are soluble, and even fewer are soluble by metrics. It’s not true that everything can be improved by measurement, or that everything that can be measured can be improved. Nor is making a problem more transparent necessarily a step to its solution. Transparency may make a troubling situation more salient, without making it more soluble."
February 18, 2018
4 stars, but one more for novelty. At times conservative and at times liberal, a hard look at what the obsession with metrics is doing to modern life, across education, business, military, government, foreign aid, and more. The author makes a highly persuasive case using succinct arguments to deconstruct what the downsides are to the obsession with numbers. And it’s quite alarming.
September 8, 2018
"The point isn't to survey the field but to highlight a few case studies." Would have been cooler if metrics were brought down with really solid stats, but "judgement" wielded by the author was interesting too.
September 20, 2021
20 yılı aşkın süredir #veri alanında bir #kariyer sahibiyim.
Bu süreçte şirketlerin veriye dayalı daha etkin yöntemler kullanması ve altyapılar oluşturması için binlerce kişiye eğitim verdim, yüzlerce projeyi aktif olarak yürüttüm.
#Veritabanı tasarımından #performansyönetimi'ne kadar, #işzekası'ndan #makineöğrenmesi'ne, #büyükveri'den #yapayzeka'ya kadar pek çok alanda çalışmalarım oldu ve olmaya devam ediyor.
Ama işin bir de diğer yönü var. Ne yazık ki veri alanında çalışan insanların çoğunlukla işin teknik boyutunda kaldıklarını, belki biraz iş birimlerinin alanlarına kafa yorduklarını ama çoğunlukla #felsefe'den uzak kaldıklarını da gördüm.
Oysa felsefe, yaptığınız işin felsefesi çok kritiktir. Olumlu ve olumsuz yönlerini değerlendirecek bir bakışa sahip olmanız çok önemlidir.
Bu kitap (#SayılarınDiktatörlüğü) kendini verinin her şeyi belirleyen ana unsur olduğu yönünde kesin bir inanca kaptırmış insanlar için ufuk açıcı. Veri alanında kariyer yapanların, orta kademe #yönetici'lerin, üst kademe yöneticilerin kesinlikle okuması gerekli.
Not: Biraz eleştirel bir bakışla okumanızda fayda var. Yazar çok yanlı kalmamaya çalışmış olsa da kendi hipotezine doğal olarak fazlaca inançlı. Verdiği örnekler çok aydınlatıcı ama tersi bakış açısıyla tersini doğrulayan çok sayıda örnek de bulunabilir.
İkinci not: Kitabın ana mesajı: Sadece ölçmeye bel bağlamak, geleneksel değerlendirme süreçlerini göz ardı etmek pek çok durumda yaşamsal önemde bir hata olur.
#kitap #okumaönerisi
#neokudum #nedenokumalı
#mustafaacungil
Bu süreçte şirketlerin veriye dayalı daha etkin yöntemler kullanması ve altyapılar oluşturması için binlerce kişiye eğitim verdim, yüzlerce projeyi aktif olarak yürüttüm.
#Veritabanı tasarımından #performansyönetimi'ne kadar, #işzekası'ndan #makineöğrenmesi'ne, #büyükveri'den #yapayzeka'ya kadar pek çok alanda çalışmalarım oldu ve olmaya devam ediyor.
Ama işin bir de diğer yönü var. Ne yazık ki veri alanında çalışan insanların çoğunlukla işin teknik boyutunda kaldıklarını, belki biraz iş birimlerinin alanlarına kafa yorduklarını ama çoğunlukla #felsefe'den uzak kaldıklarını da gördüm.
Oysa felsefe, yaptığınız işin felsefesi çok kritiktir. Olumlu ve olumsuz yönlerini değerlendirecek bir bakışa sahip olmanız çok önemlidir.
Bu kitap (#SayılarınDiktatörlüğü) kendini verinin her şeyi belirleyen ana unsur olduğu yönünde kesin bir inanca kaptırmış insanlar için ufuk açıcı. Veri alanında kariyer yapanların, orta kademe #yönetici'lerin, üst kademe yöneticilerin kesinlikle okuması gerekli.
Not: Biraz eleştirel bir bakışla okumanızda fayda var. Yazar çok yanlı kalmamaya çalışmış olsa da kendi hipotezine doğal olarak fazlaca inançlı. Verdiği örnekler çok aydınlatıcı ama tersi bakış açısıyla tersini doğrulayan çok sayıda örnek de bulunabilir.
İkinci not: Kitabın ana mesajı: Sadece ölçmeye bel bağlamak, geleneksel değerlendirme süreçlerini göz ardı etmek pek çok durumda yaşamsal önemde bir hata olur.
#kitap #okumaönerisi
#neokudum #nedenokumalı
#mustafaacungil
March 30, 2018
If your work is anything like mine, the bosses have become acolytes of metrics, the pseudo-science of numerically evaluating everything that everyone does as a way of tracking and improving service. On the surface, this makes a great deal of sense. How can you know that you have employed a mediocre teacher or an incompetent nurse without some data to demonstrate their ineffectiveness? How can you compare one hospital or college with another without knowing what their outcomes are? But, as anyone who has ever been subject to the blandishments of metric-driven management, things are not always as they seem.
Let me give you one excellent example. There are national statistics published yearly about hospitals. I work in a hospital with mortality and morbidity statistics that are appalling in comparison to most of the other hospitals in the area. We must be doing something wrong, right? Far from it. Because we are a Level One trauma center, we take on the most challenging of cases, the sickest of the sick and the most broken of the injured. More of them die; more of them are very ill. Now, the statisticians claim to take into account all of these factors with risk scores, yet we still do very poorly on these scales. A patient looking at those statistics without any background information will think that they must avoid our hospital like the plague, when it could well be that we may be just the place they need to be. (Fortunately, very few patients actually do this, but that's another story).
I am a middle manager and at every meeting a table of metrics is marched out that purports to measure just how well we are doing our jobs. Fortunately, I do not work in the kind of place that rewards good metrics with pay differentials or much other than praise, but even this publication of statistics (in the name of transparency) is misleading, demoralizing, and shaming. As the author of this fine book points out, those are some of the unintended consequences of a dependence on metrics.
The central thesis of this book is one with which I wholeheartedly agree: that which can be measured may not be the most important indicator of quality work. And (the corollary), those things which are most important may not be amenable to measurement. I can give you another excellent example from my own experience. Our clinic does very poorly when it comes to female patients who get their mammograms done. We encourage the providers to order them, even present them with report cards on how well they are doing (more metrics), yet the numbers rarely budge. As we dug a little deeper, it became evident that there were some serious systemic problems preventing some women from getting mammograms, particularly those at or around the poverty line. Our institution has no mammogram machine, and the most commonly recommended place to get this done does not accept the kind of insurance that most of these women have; transportation to a location that does accept their insurance is prohibitively expensive. And our clinic cares for more women who fit into this low income category than others. In other words, we can refer to our heart's content, but these mammograms are never going to get done until we pull down the institutional barriers to care. But none of this shows up in the metrics.
Now, I will admit (as does the author) that some professions respond well to metric measures of their work. In the financial sector, there really is no other valid way to evaluate someone's work other than the amount of money they bring in (though he also points out that this does not take into account the mentoring and teaching in which every true professional should engage). In professions where there is little in the way of personal motivation other than money, metrics may also be quite useful. But these professions are the exception rather than the rule.
One of the distinctions that Muller astutely points out is that between intrinsic and extrinsic motivation. In those jobs taken on solely for pay, extrinsic motivation (money, threat of demotion or termination) may be the sole reason a person comes to work at all. But for most of us, there is a great deal of intrinsic motivation; the feeling of being of use, pride in doing something well, appreciation of our company and its mission. For many professionals (professors and nurses come to mind), because there are many struggles when practicing those professions, there is almost always a great deal of intrinsic motivation (otherwise we would go do something else). The reason this distinction is so important is because the use of metrics can be so deeply demoralizing, reducing one's profession to a series of numbers while completely ignoring what is truly meaningful to the employee.
There are many other very important aspects of this craze for metrics in this book and I encourage you to give it a read if you are being judged on the basis of them. It does feel as if this is a once over lightly treatment of the topic; though he has many sections, several of them are only a few pages long and do not even begin to speak to all the implications of the use of metrics. Still, as an introduction to the hazards and pitfalls, as well as the legitimate uses of metrics, it is a fine read.
Let me give you one excellent example. There are national statistics published yearly about hospitals. I work in a hospital with mortality and morbidity statistics that are appalling in comparison to most of the other hospitals in the area. We must be doing something wrong, right? Far from it. Because we are a Level One trauma center, we take on the most challenging of cases, the sickest of the sick and the most broken of the injured. More of them die; more of them are very ill. Now, the statisticians claim to take into account all of these factors with risk scores, yet we still do very poorly on these scales. A patient looking at those statistics without any background information will think that they must avoid our hospital like the plague, when it could well be that we may be just the place they need to be. (Fortunately, very few patients actually do this, but that's another story).
I am a middle manager and at every meeting a table of metrics is marched out that purports to measure just how well we are doing our jobs. Fortunately, I do not work in the kind of place that rewards good metrics with pay differentials or much other than praise, but even this publication of statistics (in the name of transparency) is misleading, demoralizing, and shaming. As the author of this fine book points out, those are some of the unintended consequences of a dependence on metrics.
The central thesis of this book is one with which I wholeheartedly agree: that which can be measured may not be the most important indicator of quality work. And (the corollary), those things which are most important may not be amenable to measurement. I can give you another excellent example from my own experience. Our clinic does very poorly when it comes to female patients who get their mammograms done. We encourage the providers to order them, even present them with report cards on how well they are doing (more metrics), yet the numbers rarely budge. As we dug a little deeper, it became evident that there were some serious systemic problems preventing some women from getting mammograms, particularly those at or around the poverty line. Our institution has no mammogram machine, and the most commonly recommended place to get this done does not accept the kind of insurance that most of these women have; transportation to a location that does accept their insurance is prohibitively expensive. And our clinic cares for more women who fit into this low income category than others. In other words, we can refer to our heart's content, but these mammograms are never going to get done until we pull down the institutional barriers to care. But none of this shows up in the metrics.
Now, I will admit (as does the author) that some professions respond well to metric measures of their work. In the financial sector, there really is no other valid way to evaluate someone's work other than the amount of money they bring in (though he also points out that this does not take into account the mentoring and teaching in which every true professional should engage). In professions where there is little in the way of personal motivation other than money, metrics may also be quite useful. But these professions are the exception rather than the rule.
One of the distinctions that Muller astutely points out is that between intrinsic and extrinsic motivation. In those jobs taken on solely for pay, extrinsic motivation (money, threat of demotion or termination) may be the sole reason a person comes to work at all. But for most of us, there is a great deal of intrinsic motivation; the feeling of being of use, pride in doing something well, appreciation of our company and its mission. For many professionals (professors and nurses come to mind), because there are many struggles when practicing those professions, there is almost always a great deal of intrinsic motivation (otherwise we would go do something else). The reason this distinction is so important is because the use of metrics can be so deeply demoralizing, reducing one's profession to a series of numbers while completely ignoring what is truly meaningful to the employee.
There are many other very important aspects of this craze for metrics in this book and I encourage you to give it a read if you are being judged on the basis of them. It does feel as if this is a once over lightly treatment of the topic; though he has many sections, several of them are only a few pages long and do not even begin to speak to all the implications of the use of metrics. Still, as an introduction to the hazards and pitfalls, as well as the legitimate uses of metrics, it is a fine read.
February 5, 2018
Metrics is a word that can convey many feelings and emotions. It can be the means of beneficial analysis and quantification, as well as being something one is a slave to, recording unrealistic or unrequired data ‘just because’. For many, it is no longer just measuring the performance of something but the measurement of measuring itself that is in focus in the real world.
This book considers, therefore, the real tyranny of metrics and the obsession that many hold with this management elixir and seeks to discover if the problem can be changed.
It was an interesting, thought-provoking and engaging read. Surprisingly so, if you consider the subject is hardly going to set the world on fire. The author may be swimming against the accepted tide and holds up metric-mania for possible ridicule. This is not a bad thing when the real motivation of metrics lost. There is a precedent – everyone in working life will have experienced where a description or measurement of a process can be, for some, more important than the underlying process itself.
Not everything is bad with metrics, far from it, but it is the application and reasoning that is often skewed. The author does not just pour cold water on the idea of metrics though – he seeks to show an applicable use that can be beneficial in the process. Yet reprogramming years of possibly wrongly accumulated knowledge may be a challenge.
It is definitely a book worth considering, whether you are already on-side with the author’s views or are open to a possible contrarian argument. The book may appear overly academic for the casual reader, but it is a manageable read with a modicum of focus and it can service several audiences well. Its readability is aided by relatively short chapters and good internal signposting.
This book considers, therefore, the real tyranny of metrics and the obsession that many hold with this management elixir and seeks to discover if the problem can be changed.
It was an interesting, thought-provoking and engaging read. Surprisingly so, if you consider the subject is hardly going to set the world on fire. The author may be swimming against the accepted tide and holds up metric-mania for possible ridicule. This is not a bad thing when the real motivation of metrics lost. There is a precedent – everyone in working life will have experienced where a description or measurement of a process can be, for some, more important than the underlying process itself.
Not everything is bad with metrics, far from it, but it is the application and reasoning that is often skewed. The author does not just pour cold water on the idea of metrics though – he seeks to show an applicable use that can be beneficial in the process. Yet reprogramming years of possibly wrongly accumulated knowledge may be a challenge.
It is definitely a book worth considering, whether you are already on-side with the author’s views or are open to a possible contrarian argument. The book may appear overly academic for the casual reader, but it is a manageable read with a modicum of focus and it can service several audiences well. Its readability is aided by relatively short chapters and good internal signposting.
April 15, 2018
The Tyranny of Metrics (2018) by Jerry Z Muller is an interesting look at problems caused by misusing metrics. It's impressively short and Muller has read widely and pondered the problems caused by over relying on poor metrics.
Muller outlines why metrics have been used. It looks at increasing costs and people wanting, wisely, to improve productivity. Metrics were also seen as a way of resolving the principal / agent problem. They were also seen as a way of doing something objective to assess outcomes. Looking at numbers and working on productivity also helped manufacturing improve. Applying similar ideas to services, wars, education, health, police and other areas seemed like a good idea.
In practice, however, using metrics has led to many perverse outcomes. In education, just getting more people to go to University and get good grades has led to more people just going, many of whom don't gain much from their education and also to US 'grade inflation'. Their professors, meanwhile pump out more papers and join paper citation circles to pump their 'impact factor'. In war 'body counts' as a measure of success in Vietnam contributed to the US failure there. In medicine it's led to doctors avoiding difficult cases. Optimising for short term profits has even harmed many businesses.
Muller concludes by providing a guideline for when and how to use metrics.
Muller isn't against all metrics, but he is making a very good case that they are often misused and that people need to be very careful about how and where they are used. The book is very much worth a read for anyone interested in policy or economics.
Muller outlines why metrics have been used. It looks at increasing costs and people wanting, wisely, to improve productivity. Metrics were also seen as a way of resolving the principal / agent problem. They were also seen as a way of doing something objective to assess outcomes. Looking at numbers and working on productivity also helped manufacturing improve. Applying similar ideas to services, wars, education, health, police and other areas seemed like a good idea.
In practice, however, using metrics has led to many perverse outcomes. In education, just getting more people to go to University and get good grades has led to more people just going, many of whom don't gain much from their education and also to US 'grade inflation'. Their professors, meanwhile pump out more papers and join paper citation circles to pump their 'impact factor'. In war 'body counts' as a measure of success in Vietnam contributed to the US failure there. In medicine it's led to doctors avoiding difficult cases. Optimising for short term profits has even harmed many businesses.
Muller concludes by providing a guideline for when and how to use metrics.
Muller isn't against all metrics, but he is making a very good case that they are often misused and that people need to be very careful about how and where they are used. The book is very much worth a read for anyone interested in policy or economics.
July 9, 2022
The topic or what Muller calls "metrics fixation" is a super timely and important subject. So I applaud the idea of diving into it. Yet, the execution, so to say, is disappointing at the very least. The book is very dry and repetitive. It also seems like a rushed effort to say something smart about a topic you don't know a lot about, so you start to read hastily as much as you can. But then, not having enough time to synthesize all the info consumed, you pour over your readers this half-baked overview of your sources with a few (a bit banal for social scientist) thoughts of your own. I bet if this book wasn't written by a tenured professor, if wouldn't be published. And for those who really look into learning something about the social processes beyond the construction of numbers, I cannot recommend enough the work of Sally Engle Merry.
September 14, 2018
Very good introduction to some of the flaws of metrics. If you are already familiar with some of the literature - or are wary of metrics - this probably isn’t new information, but it is all put together and summarized very nicely. I wished he had engaged with the anthropology and STS work on this topic.
July 7, 2018
Muller explains the dangers of trusting metrics for anything, with examples from government, business and charities. It’s a good reminder of what could go wrong, and shows that judgment based on knowledge and experience trumps judgment based on metrics.
May 2, 2019
found it difficult to focus on.
metrics can be gamed and once known and the basis for preferential treatment most certainly will be.
metrics can be gamed and once known and the basis for preferential treatment most certainly will be.
May 17, 2020
I'm extremely glad I read this book, despite having read the quite similar (and also good) book, Weapons of Math Destruction. As a data scientist and government consultant who specializes in performance management, I am certainly guilty of the metric fixation discussed. Indeed, even when I think I am self-aware and push to incorporate qualitative information, etc, it seems it still perpetuates the problem.
The main point that I took away from the Tyranny of Metrics is that we need to trust and encourage professional judgment. Aka the subjective. Supplementing professional judgment with standardized, objective numbers is a good thing when it is low-stakes (i.e. not tied to pay or job security) and has context, but must not come at the expense of the work itself (by draining resources or intrinsic motivation).
An example it made me think of on a personal level is how I set a goal each year for a number of books to read, and track that goal using Goodreads. I understand that the goal is a means, not an end. The bigger picture is that I'd like to expand my mind, build/maintain the discipline of sustained attention, and do more of something I enjoy. If I read some particularly long books one year or if a book inspires me to write more and I don't hit my goal, it is perfectly alright. When the goal metric becomes higher-stakes than that, you create an incentive to game the system and undermine the bigger picture.
Along the same lines, the book got me thinking about perhaps the most prevalent and destructive example of metric fixation, which is using money/GDP to represent well-being, or GDP growth to represent societal progress. Sacred Economics is a fabulous book on this subject and there are myriad other articles and talks.
This is the line I keep thinking of from The Tyranny of Metrics
"Management practices based on economic models may dampen or even destroy non-economic realities such as intrinsic motivation and social relations."
Examples of those non-economic realities include neighbors looking after one another's children rather than paying for childcare, volunteering, or offering things for free-- what is sometimes called "the gift economy." I have been seeing many examples recently during the pandemic, but when leaders do not see these realities because they don't show up in the GDP metric and they act accordingly (such as framing Congress' economic relief package as "fiscal stimulus"), they push us more into a society where everything is a transaction and a commodity. Lots of food for thought.
The main point that I took away from the Tyranny of Metrics is that we need to trust and encourage professional judgment. Aka the subjective. Supplementing professional judgment with standardized, objective numbers is a good thing when it is low-stakes (i.e. not tied to pay or job security) and has context, but must not come at the expense of the work itself (by draining resources or intrinsic motivation).
An example it made me think of on a personal level is how I set a goal each year for a number of books to read, and track that goal using Goodreads. I understand that the goal is a means, not an end. The bigger picture is that I'd like to expand my mind, build/maintain the discipline of sustained attention, and do more of something I enjoy. If I read some particularly long books one year or if a book inspires me to write more and I don't hit my goal, it is perfectly alright. When the goal metric becomes higher-stakes than that, you create an incentive to game the system and undermine the bigger picture.
Along the same lines, the book got me thinking about perhaps the most prevalent and destructive example of metric fixation, which is using money/GDP to represent well-being, or GDP growth to represent societal progress. Sacred Economics is a fabulous book on this subject and there are myriad other articles and talks.
This is the line I keep thinking of from The Tyranny of Metrics
"Management practices based on economic models may dampen or even destroy non-economic realities such as intrinsic motivation and social relations."
Examples of those non-economic realities include neighbors looking after one another's children rather than paying for childcare, volunteering, or offering things for free-- what is sometimes called "the gift economy." I have been seeing many examples recently during the pandemic, but when leaders do not see these realities because they don't show up in the GDP metric and they act accordingly (such as framing Congress' economic relief package as "fiscal stimulus"), they push us more into a society where everything is a transaction and a commodity. Lots of food for thought.
Shelved as 'dnf'
July 18, 2023I only read a couple sections.
I did a search to vet the book. One review mentioned he cites Scott Atlas, who is now known for spreading misinformation about COVID response and discouraging preventative measures and testing.
I then searched for that section, and found that the author promotes odd or incomplete criticisms of metrics. He is not a math person and not well versed in the areas he writes about, and it shows. He cites Atlas's criticism of WHO metrics that rank US Healthcare as poorer than other developed nations. The criticism being that these metrics include externalities such as, is there equal access to health care? Oddly, considering externalities is good. Metrics often consider only one small thing, and causes are usually a lot more than one factor. We care about health outcomes for everyone, not just if medicine is effective in a vacuum. However, lumping different aspects of health into one score can be unhelpful, if the actors don't see these multiple aspects of an issue. Unfortunately, the author doesn't address the actual metric approach of weighting. He thinks we should treat medicine as a vacuum, I guess. A very defensive view, in my opinion.
He then makes a statement about racial differences in health in the US and completely ignores the reality of racism in healthcare. This is well known and researched. In fact, he cites "possibly genetic factors". Note, that is for health outcomes for people of color only seen in the US - racism is a huge cause of health disparity and some is external and some is internal to the practice of medicine.
Racism isn't mentioned in the book, and African Americans are mentioned only to say they have worse health outcomes and lower school achievements without any context for the historical and current role of racism and violence. Why even mention these things? Usually, such statements are made only to victim blame. Literally, their homes were blown up. Their land has been stolen. Their rights have been denied. They are judged more harshly and punished in schools by teachers and not taken seriously in hospitals. We have data on these unnatural causes.
From the two sections that I read about education and health care:
I think the author doesn't have the requisite math or domain knowledge. I agree that metrics are poorly used and there is a very important need to explore usefulness of metrics from a human behavior standpoint. But, this book fails in that regard. He doesn't seem to understand systems and measurement and monitoring of systems. I think many people don't and that's an issue. Finally, many people create metrics from intuition, so I don't think intuition is the answer. We have a lot of research showing the gaps and bias of intuiton. Data should complement knowledge and expand it.
I did a search to vet the book. One review mentioned he cites Scott Atlas, who is now known for spreading misinformation about COVID response and discouraging preventative measures and testing.
I then searched for that section, and found that the author promotes odd or incomplete criticisms of metrics. He is not a math person and not well versed in the areas he writes about, and it shows. He cites Atlas's criticism of WHO metrics that rank US Healthcare as poorer than other developed nations. The criticism being that these metrics include externalities such as, is there equal access to health care? Oddly, considering externalities is good. Metrics often consider only one small thing, and causes are usually a lot more than one factor. We care about health outcomes for everyone, not just if medicine is effective in a vacuum. However, lumping different aspects of health into one score can be unhelpful, if the actors don't see these multiple aspects of an issue. Unfortunately, the author doesn't address the actual metric approach of weighting. He thinks we should treat medicine as a vacuum, I guess. A very defensive view, in my opinion.
He then makes a statement about racial differences in health in the US and completely ignores the reality of racism in healthcare. This is well known and researched. In fact, he cites "possibly genetic factors". Note, that is for health outcomes for people of color only seen in the US - racism is a huge cause of health disparity and some is external and some is internal to the practice of medicine.
Racism isn't mentioned in the book, and African Americans are mentioned only to say they have worse health outcomes and lower school achievements without any context for the historical and current role of racism and violence. Why even mention these things? Usually, such statements are made only to victim blame. Literally, their homes were blown up. Their land has been stolen. Their rights have been denied. They are judged more harshly and punished in schools by teachers and not taken seriously in hospitals. We have data on these unnatural causes.
From the two sections that I read about education and health care:
I think the author doesn't have the requisite math or domain knowledge. I agree that metrics are poorly used and there is a very important need to explore usefulness of metrics from a human behavior standpoint. But, this book fails in that regard. He doesn't seem to understand systems and measurement and monitoring of systems. I think many people don't and that's an issue. Finally, many people create metrics from intuition, so I don't think intuition is the answer. We have a lot of research showing the gaps and bias of intuiton. Data should complement knowledge and expand it.
September 30, 2018
In the Tyranny of Metrics, Jerry Muller presents a clear case against the current, and growing, over-reliance on KPIs and other performance metrics for assessing people's and organizations' work. One of the more common admonitions in AI is to be careful what you ask for, or, as Russell & Norvig put it in their incredibly popular textbook on AI, "what you ask for is what you get." If your institutions set KPIs rewarding citations, for example, then you're likely to end up with illicit citation rings, with pals citing each other pointlessly -- pointless except for the KPI reward, of course. KPIs have a very strong tendency to replace the real objectives of an organization -- education, good governance, public health, common welfare -- with much lower quality ersatz objectives -- student popularity, volume of memoranda, quick hospital discharges, numbers of arrests for petty crimes.
Another common problem is using absolute metrics when only relativized metrics make sense. For example, insisting that education funding reward schools whose students perform better on standardized tests can make for a good sound-bite, but has always been understood to produce "teaching to the test" -- that is, a narrow educational focus that leads to ex-students poorly prepared to deal with a complex world with wide-ranging problems. But importantly it also leads to schools taking a low-risk approach to their education. Instead of seeking out and supporting students with disabilities or minority socioeconomic backgrounds, they will narrow their admissions to those already likely to perform well on standardized tests. That can really pay off for their budgets, but it's also letting society down. The major blame should fall on the politicians who force the performance metrics on the schools in the first place. Instead of an absolute test-performance standard, a relativized standard, comparing outcome performance to initial performance, would eliminate that particular distortion of educational goals. (While doing nothing about teaching to the test, of course, which requires some other response.)
Muller reviews many of the ways in which metrics can go wrong, and he specifically considers them in some of the more socially important domains in which they do go wrong: education (secondary and tertiary), policing, medicine, finance, the military, foreign aid. His book is an excellent starting point for thinking about the general subject of work performance measurement or its particular consideration in any of these domains.
I can't give Muller five stars here, however. One area in which he goes seriously astray is the issue of transparency in our institutions. Muller quite rightly points out that in many processes we require confidentiality and that the demand for transparency may well have gone too far. In diplomacy, privacy or secrecy can be essential to achieving a reasonable outcome. Diplomacy requires compromise, and compromise requires giving away something that you'd prefer not to. If the thing given way is made public too soon, then the diplomatic transaction may well implode before any compromise can be agreed. People have secrets for good reason. Similarly, the demand for honesty in politicians (generally an unsuccessful demand to be sure, but commonly loudly made in the media nonetheless) can turn into a fetish, where politicians who legitimately change their minds are excoriated for having no position or those who bend the truth to achieve a greater good are pummelled for dishonesty. The whole point of politics is for our politicians to achieve greater goods, not lesser goods, so these kinds of admonition by Muller are well taken.
Yet Muller goes far too far in this. Wikileaks' revelation of war crimes in Iraq was accompanied by the revelation of identities of intelligence agents around the world. Julian Assange didn't care about the safety of those agents or the future ability of, say, the CIA to recruit other agents. That's a failure of over-zealous transparency, for sure. But Muller gives no credit at all to the other side. For example, he fails to accept that the whistleblowing revelation of the war crimes itself was a good thing and should be legally protected. He fails to acknowledge any benefit from Freedom of Information laws. Worse still, he castigates Edward Snowden for revealing many of the secrets of the NSA. While the United States has the right to have a "no such agency", it doesn't have the right to have spying and anti-encryption programs of the depth and breadth of the NSA. What Snowden exposed was, and is, criminal activity endangering democracy (see, e.g., https://www.bbc.com/news/uk-45510662). I think it absurd that Muller can't put in even one word of support for such transparency. I wish for treatments of these subjects that show better balance and judgment than Muller's.
Another common problem is using absolute metrics when only relativized metrics make sense. For example, insisting that education funding reward schools whose students perform better on standardized tests can make for a good sound-bite, but has always been understood to produce "teaching to the test" -- that is, a narrow educational focus that leads to ex-students poorly prepared to deal with a complex world with wide-ranging problems. But importantly it also leads to schools taking a low-risk approach to their education. Instead of seeking out and supporting students with disabilities or minority socioeconomic backgrounds, they will narrow their admissions to those already likely to perform well on standardized tests. That can really pay off for their budgets, but it's also letting society down. The major blame should fall on the politicians who force the performance metrics on the schools in the first place. Instead of an absolute test-performance standard, a relativized standard, comparing outcome performance to initial performance, would eliminate that particular distortion of educational goals. (While doing nothing about teaching to the test, of course, which requires some other response.)
Muller reviews many of the ways in which metrics can go wrong, and he specifically considers them in some of the more socially important domains in which they do go wrong: education (secondary and tertiary), policing, medicine, finance, the military, foreign aid. His book is an excellent starting point for thinking about the general subject of work performance measurement or its particular consideration in any of these domains.
I can't give Muller five stars here, however. One area in which he goes seriously astray is the issue of transparency in our institutions. Muller quite rightly points out that in many processes we require confidentiality and that the demand for transparency may well have gone too far. In diplomacy, privacy or secrecy can be essential to achieving a reasonable outcome. Diplomacy requires compromise, and compromise requires giving away something that you'd prefer not to. If the thing given way is made public too soon, then the diplomatic transaction may well implode before any compromise can be agreed. People have secrets for good reason. Similarly, the demand for honesty in politicians (generally an unsuccessful demand to be sure, but commonly loudly made in the media nonetheless) can turn into a fetish, where politicians who legitimately change their minds are excoriated for having no position or those who bend the truth to achieve a greater good are pummelled for dishonesty. The whole point of politics is for our politicians to achieve greater goods, not lesser goods, so these kinds of admonition by Muller are well taken.
Yet Muller goes far too far in this. Wikileaks' revelation of war crimes in Iraq was accompanied by the revelation of identities of intelligence agents around the world. Julian Assange didn't care about the safety of those agents or the future ability of, say, the CIA to recruit other agents. That's a failure of over-zealous transparency, for sure. But Muller gives no credit at all to the other side. For example, he fails to accept that the whistleblowing revelation of the war crimes itself was a good thing and should be legally protected. He fails to acknowledge any benefit from Freedom of Information laws. Worse still, he castigates Edward Snowden for revealing many of the secrets of the NSA. While the United States has the right to have a "no such agency", it doesn't have the right to have spying and anti-encryption programs of the depth and breadth of the NSA. What Snowden exposed was, and is, criminal activity endangering democracy (see, e.g., https://www.bbc.com/news/uk-45510662). I think it absurd that Muller can't put in even one word of support for such transparency. I wish for treatments of these subjects that show better balance and judgment than Muller's.
January 23, 2022
My gracious.
The whole “we need to return to trusting the seasoned judgment of wise professionals” argument would be so much more interesting if the author had bothered to acknowledge the possibility that prejudice, avarice, and intellectual inertia might have characterized “wise professionals’” “seasoned judgment” in the golden past to which he implicitly refers.
Evidence-based medicine is a technical, scientific, and moral improvement on what came before it, and failing to acknowledge this is a fatal flaw in the author’s argument.
However, there is one important theme in the book, though one which is probably better theorized and documented elsewhere: that incentivizing performance metrics also always inevitably and ineluctably incentivizes cheating on the metric.
The whole “we need to return to trusting the seasoned judgment of wise professionals” argument would be so much more interesting if the author had bothered to acknowledge the possibility that prejudice, avarice, and intellectual inertia might have characterized “wise professionals’” “seasoned judgment” in the golden past to which he implicitly refers.
Evidence-based medicine is a technical, scientific, and moral improvement on what came before it, and failing to acknowledge this is a fatal flaw in the author’s argument.
However, there is one important theme in the book, though one which is probably better theorized and documented elsewhere: that incentivizing performance metrics also always inevitably and ineluctably incentivizes cheating on the metric.
June 19, 2020
Metrics are becoming a bigger part of my working life these days, and I was excited for a deeper dive into an epistemological critique. This book did an ok job at that, covering a lot of the downsides of metric fixation that becomes obvious to anyone working with them. Also appreciated the discussion of transparency, as it is a common call in all sorts of reform. Not always a good thing. Book overall struggled to offer any new, ground-breaking analysis, instead spending a lot of time going through examples from the last decade or so, mostly in education.
Displaying 1 - 30 of 233 reviews