[FoRK] Lancet, shmancet
eugen at leitl.org
Mon Nov 1 12:43:33 PST 2004
November 01, 2004
Talking rubbish about epidemiology
Posted by Daniel
As Chris said, with respect to the Lancet study on excess Iraqi deaths,
âI can predict with certainty that there will be numerous posts on
weblogs supporting the war attacking the studyâ. Score several Cassandra
points for Chris, they werenât slow in coming. You can have the
know-nothing rightwing flack variety or the handwringing liberal variety. And
to be honest, the standard of critique is enough to make you weep.
Taking the complaints that seem to have been raised about this study:
âThat is, a one in twenty chance that the effect simply does not
existâ (from Tech Central Station). The author of the TCS piece appears
to believe that because the Lancet study published a 95% confidence interval,
there is a 5% chance that there was no effect. The problem with this critique
is that it is not true.
âa relative risk ratio of anything less than three is regarded as
statistically insignificantâ. This is also from TCS, and also, simply,
not true. Interesting to note that TCS appear to have upped the ante on this
piece of bogus epidemiology; historically when they have been talking about
passive smoking, the threshold for relative risk ratios has been two. Which
is also bollocks. The TCS author appears to have a very shaky grasp of the
statistical concepts he is using.
âThis isnât an estimate. Itâs a dart boardâ. The critique
here, from Slate, is that the 95% confidence interval for the estimate of
excess deaths (8,000 to 200,000) is so wide that itâs meaningless.
Itâs wrong. Although there are a lot of numbers between 8,000 and
200,000, one of the ones that isnât is a little number called zero.
Thatâs quite startling. One might have hoped that there was at least some
chance that the Iraq war might have had a positive effect on death rates in
Iraq. But the confidence interval from this piece of work suggests that there
would be only a 2.5% chance of getting this sort of result from the sample if
the true effect of the invasion had been favourable. A curious basis for a
humanitarian intervention; âwe must invade, because Saddam is killing
thousands of his citizens every year, and we will kill only 8,000 moreâ.
The estimate of prewar mortality is too low. The idea here is that the sample
chosen for the survey had a mortality rate of about 5 per 1000 in the two
years before the invasion. And, because the death rate for the period 1985-90
was 6.8 per 1000 according to UN figures, this in some way suggests that the
estimates are at fault.
This critique is more interesting, but hardly devastating. For one thing, the
contention that the Iraqi death rate did not fall from 6.8 to around 5 during
the 1990s is based on âit must have doneâ rather than on hard
numbers. Since the 6.8 number includes (as far as I can tell) atrocities
committed by Saddam during the period which were not repeated in 2000-03, I
am less convinced than the Slate author that the discrepancy strikes such a
huge blow to the studyâs credibility. In any case, since the study
compares own-averages of the clusters before and after the invasion, anyone
wanting to make this critique needs to come up with a convincing explanation
of why it is that the study had a lower death-rate than the national average
before the invasion and not after the invasion.
âvarious bog standard methodological quibbles are really really
devastatingâ. This line of attack is usually associated with Steven
Milloy, so I will nickname it the âdevastating critiqueâ. The example
I found was here. The modus operandi is to take a decent piece of statistical
research carried out by someone who got his hands dirty with the data, point
out a few areas in which it differs from the Platonic Form of the
Epidemiological Study (if youâre dealing with a really good study, it
does your work for you here by alerting you to the specific difficulties),
and then say something like âsheeeesh, how did this ever get
published?!?!â. Iâve done it myself a few times, but thatâs
hardly a recommendation.
The Chicago Boyz blog post is an excellent example of the âDevastating
Critiqueâ. Surprise surprise, estimating civilian casualties is a
difficult business. Thatâs why the confidence interval is so wide. They
donât actually raise any principled reasons why the confidence interval
ought to be wider than the one published, and therefore they arenât
raising any questions which would make us think that this confidence interval
should include zero.
It gives a different number from Iraq Body Count. so it must be wrong. This
critique is also fairly stupid. The IBC numbers are compiled from
well-sourced English language press reports. They therefore represent a lower
bound on any credible estimate of casualties, not a definitive number.
Thousands of people die in the UK every day; how many of them make it into
the papers? How may into the Arabic language press?
One can score extra points for intellectual dishonesty on this count by
citing Oxblog to try to imply that IBC is in some way an overestimate (and
therefore, of course, to push that confidence interval in the direction of
zero). As the link Iâve provided shows, the Oxblog critique (which I
donât agree with) refers in the main to whether documented casualties can
be blamed on the Americans; there is no well-founded challenge to suggest
that the people IBC lists as dead are in fact consuming oxygen.
There is something intrinsically suspect about accelerated peer review. As
John pointed out not so long ago, the time taken for peer review is
determined by academic procrastination above all other factors. Every
academic paper could complete its peer review very quickly if the reviewers
got their finger out because they thought it was important. The suggestion
that people are trying to make here is that reviewers for the Lancet usually
spend six months humming and hawing over the data, to the exclusion of all
other activity, and that this process was short-circuited by politically
motivated editors wanting to rush something into print without anyone having
a proper look at it. No such six month scrutiny ever takes place, and this
objection is also Simply Not True.
The 100,000 figure should not have been headlined. Another staple critique of
epidemiological studies one doesnât like. It is true of more or less any
study you hear of, since you never hear of studies that donât have
interesting headlines. In all honesty, I donât like these extrapolated
numbers, never have and never will. I donât like linear models and I
donât like extrapolation. However, itâs a venial sin rather than a
mortal one, and I have never, ever, at all, heard of anyone criticising it in
a study that they otherwise liked. (Simple thought experiment; if the results
of the study had been talking about 100,000 fewer deaths, would this critique
have been made by the same people? Like hell).
The important thing as far as Iâm concerned is the position of zero in
the confidence interval; it seems very unlikely indeed that the process
described could have given this sample if it was not the case that the
invasion had made the death rate in Iraq worse rather than better. And this
conclusion of the study is basically unchallenged. In fact, itâs in a
better position than âunchallengedâ; itâs been challenged so
weakly and on such spurious grounds that my Bayesian assessment has been
updated in its favour, on the basis that if those who disliked the
studyâs conclusion had any real ammunition against it, the published
critiques would not have been so weak.
Eugen* Leitl <a href="http://leitl.org">leitl</a>
ICBM: 48.07078, 11.61144 http://www.leitl.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
More information about the FoRK