[FoRK] Lancet, shmancet

Eugen Leitl eugen at leitl.org
Mon Nov 1 12:43:33 PST 2004


November 01, 2004
Talking rubbish about epidemiology
Posted by Daniel

As Chris said, with respect to the Lancet study on excess Iraqi deaths,
“I can predict with certainty that there will be numerous posts on
weblogs supporting the war attacking the study”. Score several Cassandra
points for Chris, they weren’t slow in coming. You can have the
know-nothing rightwing flack variety or the handwringing liberal variety. And
to be honest, the standard of critique is enough to make you weep.

Taking the complaints that seem to have been raised about this study:

“That is, a one in twenty chance that the effect simply does not
exist” (from Tech Central Station). The author of the TCS piece appears
to believe that because the Lancet study published a 95% confidence interval,
there is a 5% chance that there was no effect. The problem with this critique
is that it is not true.

“a relative risk ratio of anything less than three is regarded as
statistically insignificant”. This is also from TCS, and also, simply,
not true. Interesting to note that TCS appear to have upped the ante on this
piece of bogus epidemiology; historically when they have been talking about
passive smoking, the threshold for relative risk ratios has been two. Which
is also bollocks. The TCS author appears to have a very shaky grasp of the
statistical concepts he is using.

“This isn’t an estimate. It’s a dart board”. The critique
here, from Slate, is that the 95% confidence interval for the estimate of
excess deaths (8,000 to 200,000) is so wide that it’s meaningless.
It’s wrong. Although there are a lot of numbers between 8,000 and
200,000, one of the ones that isn’t is a little number called zero.
That’s quite startling. One might have hoped that there was at least some
chance that the Iraq war might have had a positive effect on death rates in
Iraq. But the confidence interval from this piece of work suggests that there
would be only a 2.5% chance of getting this sort of result from the sample if
the true effect of the invasion had been favourable. A curious basis for a
humanitarian intervention; “we must invade, because Saddam is killing
thousands of his citizens every year, and we will kill only 8,000 more”.

The estimate of prewar mortality is too low. The idea here is that the sample
chosen for the survey had a mortality rate of about 5 per 1000 in the two
years before the invasion. And, because the death rate for the period 1985-90
was 6.8 per 1000 according to UN figures, this in some way suggests that the
estimates are at fault.

This critique is more interesting, but hardly devastating. For one thing, the
contention that the Iraqi death rate did not fall from 6.8 to around 5 during
the 1990s is based on “it must have done” rather than on hard
numbers. Since the 6.8 number includes (as far as I can tell) atrocities
committed by Saddam during the period which were not repeated in 2000-03, I
am less convinced than the Slate author that the discrepancy strikes such a
huge blow to the study’s credibility. In any case, since the study
compares own-averages of the clusters before and after the invasion, anyone
wanting to make this critique needs to come up with a convincing explanation
of why it is that the study had a lower death-rate than the national average
before the invasion and not after the invasion.

“various bog standard methodological quibbles are really really
devastating”. This line of attack is usually associated with Steven
Milloy, so I will nickname it the “devastating critique”. The example
I found was here. The modus operandi is to take a decent piece of statistical
research carried out by someone who got his hands dirty with the data, point
out a few areas in which it differs from the Platonic Form of the
Epidemiological Study (if you’re dealing with a really good study, it
does your work for you here by alerting you to the specific difficulties),
and then say something like “sheeeesh, how did this ever get
published?!?!”. I’ve done it myself a few times, but that’s
hardly a recommendation.

The Chicago Boyz blog post is an excellent example of the “Devastating
Critique”. Surprise surprise, estimating civilian casualties is a
difficult business. That’s why the confidence interval is so wide. They
don’t actually raise any principled reasons why the confidence interval
ought to be wider than the one published, and therefore they aren’t
raising any questions which would make us think that this confidence interval
should include zero.

It gives a different number from Iraq Body Count. so it must be wrong. This
critique is also fairly stupid. The IBC numbers are compiled from
well-sourced English language press reports. They therefore represent a lower
bound on any credible estimate of casualties, not a definitive number.
Thousands of people die in the UK every day; how many of them make it into
the papers? How may into the Arabic language press?

One can score extra points for intellectual dishonesty on this count by
citing Oxblog to try to imply that IBC is in some way an overestimate (and
therefore, of course, to push that confidence interval in the direction of
zero). As the link I’ve provided shows, the Oxblog critique (which I
don’t agree with) refers in the main to whether documented casualties can
be blamed on the Americans; there is no well-founded challenge to suggest
that the people IBC lists as dead are in fact consuming oxygen.

There is something intrinsically suspect about accelerated peer review. As
John pointed out not so long ago, the time taken for peer review is
determined by academic procrastination above all other factors. Every
academic paper could complete its peer review very quickly if the reviewers
got their finger out because they thought it was important. The suggestion
that people are trying to make here is that reviewers for the Lancet usually
spend six months humming and hawing over the data, to the exclusion of all
other activity, and that this process was short-circuited by politically
motivated editors wanting to rush something into print without anyone having
a proper look at it. No such six month scrutiny ever takes place, and this
objection is also Simply Not True.

The 100,000 figure should not have been headlined. Another staple critique of
epidemiological studies one doesn’t like. It is true of more or less any
study you hear of, since you never hear of studies that don’t have
interesting headlines. In all honesty, I don’t like these extrapolated
numbers, never have and never will. I don’t like linear models and I
don’t like extrapolation. However, it’s a venial sin rather than a
mortal one, and I have never, ever, at all, heard of anyone criticising it in
a study that they otherwise liked. (Simple thought experiment; if the results
of the study had been talking about 100,000 fewer deaths, would this critique
have been made by the same people? Like hell).

The important thing as far as I’m concerned is the position of zero in
the confidence interval; it seems very unlikely indeed that the process
described could have given this sample if it was not the case that the
invasion had made the death rate in Iraq worse rather than better. And this
conclusion of the study is basically unchallenged. In fact, it’s in a
better position than “unchallenged”; it’s been challenged so
weakly and on such spurious grounds that my Bayesian assessment has been
updated in its favour, on the basis that if those who disliked the
study’s conclusion had any real ammunition against it, the published
critiques would not have been so weak.

Eugen* Leitl <a href="http://leitl.org">leitl</a>
ICBM: 48.07078, 11.61144            http://www.leitl.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
http://moleculardevices.org         http://nanomachines.net

More information about the FoRK mailing list