Medical Kaizen
In response to the post below about Atul Gawande's article, several readers have sent links to these posts. Without any prompting from me, Professor Postrel emailed the following to Soxblogger James Frederick Dwight, thereby saving me the trouble of composing a response (and giving Dynamist readers the benefit of his statistical training and years of following the quality revolution in manufacturing):
Mr. Dwight: I have read with interest your posts on the New Yorker article by Dr. Gawande. It is undoubtedly correct that a rigorous analysis would require some sort of multiple regression approach to control for patient differences, assuming that genetic data at the patient level are available across the entire sample. Then coefficients on the dummy variables for each clinic would presumably capture the effect of each clinic (subject to the usual caveats about potential omitted variables).
The vehement tone of your criticism is, however, unwarranted, because it seems very unlikely that the superior performance of the clinic in Minneapolis is entirely or largely due to random assignment of genetic types. I conclude this on the basis of the following factors:
1) The types of CF that result in rapid death will constitute a vanishingly small percentage of the population at any one time, for the obvious reason that these unfortunate patients die off quicky. If, as I suspect from the crudity of measures that you highlight, the data have not been corrected for this bias, the hardest cases will have almost no effect on the performance differences found.
2) You provide no links or citations for your claims about genetic variants and their impact on longevity. Perusal of the CF Foundation website provides no such information. Taking your word for it, however, the key datum we need to evaluate Gawande's claim is NOT how many genetic variations there are but how big the range of severity is across those variants (weighted for their persistence in the clinic patient pools).
(Beyond evaluating Gawande's claims, it might be interesting to see if dummy variables on each genetic variation, both independently and in interaction with the clinic dummies, explain much of the variation. This would get at whether certain clinics were especially good at handling particular genetic variants. I don't know if we would have enough data points to do this though--degrees of freedom could be an issue.)
3) Your confidence about the low variance of performance among physicians is unwarranted. Even conditional upon being in the upper tail of a distribution of attribute X, there may still be substantial variation within that upper tail. More importantly, if attribute X is imperfectly correlated with what we really care about, attribute Y call it, then a group from the upper tail of the X distribution may well look like a bell curve in the Y dimension. Let X be the attributes that get people through medical licensing and let Y be proficiency in treating CF. It would not be shocking if professional training captured only some of the factors that ON THE MARGIN make one practitioner more effective than another.
4) It seems cosmically unlikely that the best clinic in the sample just happened to be the one run by the pioneer in comparative methods, the one who used demonstrably different techniques for treatment, quality assurance, and patient compliance. The idea that Minneapolis just happened to get a phenomenally favorable genetic draw AND was an innovator in treatment (doing things very differently) seems far-fetched.
5) The Gawande article claims that not only is there a bell curve in performance, but the leaders are IMPROVING faster than the average and below-average clinics. Unless patient turnover is negligible, that militates strongly against the suggestion that it is all the luck of the genetic draw. If patient turnover is negligible, then at best one could argue that there are genetic types that are not only easier to treat but that have initially hidden pathways to improvement not present in other types. This seems strained as well. More likely, we have an example of organizations exhibiting what the Japanese call "kaizen" or "continuous improvement through incremental refinement."
6) There is a vast body of empirical evidence in fields ranging from computer programming to automobile manufacturing that performance variations among similar units are large and persistent. To the extent that these are differences at the individual level (e.g. programming) no known intervention exist, to my knowledge. To the extent that they exist at the organizational level (e.g. auto factories), they can be addressed with a variety of management practices, many of which can help but none of which promise immediate performance convergence. For example, the Wall Street Journal [actually the NYT--vp] has just run an article on how getting hospitals to follow established standards of care in a few selected disease categories can have huge impacts on mortality. Failure to follow these standards of care is attributed primarily to physicians' inability to remember all the things that should be done given the stress and information overload they face and/or their cultural unwillingness to follow codified protocols consistently.
7) The CF Foundation website makes laudatory mention of the Gawande article and registers no objection whatsoever to its use of the data.
For all of these reasons, your tone of accusation and conclusive dismissal of Gawande's thesis is premature. Ideally, we would have peer-reviewed econometric studies to settle these matter conclusively. Absent those, however, your objections fall short of discrediting Gawande or the New Yorker.
Mr. Dwight has not had a chance to respond yet. I'm posting this because I thought readers would be interested in the counterargument.