Burying The Lancet - Part 1

[Guest guest]

Mon, 5 Sep 2005 13:51:33 UT

" Medialens Media Alerts " <noreply

Burying The Lancet - Part 1

 

 

 

 

 

 

MEDIA LENS: Correcting for the distorted vision of the corporate media

 

September 5, 2005

 

 

MEDIA ALERT: BURYING THE LANCET - PART 1

 

An Exchange Between The Independent's Mary Dejevsky And Lancet Author

Les Roberts

 

 

" It is odd that the logic of epidemiology embraced by the press every

day regarding new drugs or health risks somehow changes when the

mechanism of death is their armed forces. " (Les Roberts, Johns Hopkins

School

of Public Health)

 

 

 

As a test of the independence and honesty of the mass media, few tasks

are more revealing than that of reporting our own government's

responsibility for the killing of innocents abroad. In an age of

'converged'

political parties and globalised corporate influence, few establishment

groups have any interest in seeing such horrors exposed, while many have

much to lose. Corporate journalists are therefore subject to two very

real, competing pressures:

 

1) the moral, human pressure of reporting honestly our responsibility

for mass killing, and

 

2) state-corporate pressure and flak punishing dissent and rewarding

servility to power.

 

The results tell us much about the moral and political health of our

media and our democracy.

 

On July 20 an article by Terry Kirby and Elizabeth Davies in the

Independent noted that a November 2004 report in the Lancet had estimated

Iraqi civilian deaths at nearly 100,000, but that the methodology " was

subsequently criticised " . (Kirby and Davies, `Iraq conflict claims 34

civilians lives each day as " anarchy " beckons,' The Independent, July 20,

2005)

 

The report in question was produced by some of the world's leading

research organisations - the Johns Hopkins Bloomberg School of Public

Health in Baltimore, Columbia University, and Baghdad's Al-Mustansiriya

University - and was published in one of the world's most prestigious

science journals - The Lancet. We were therefore keen to know which

criticisms Kirby and Davies had in mind. We wrote to the Independent

and Kirby

replied on July 22:

 

" So far as I am aware, the Lancet's report was criticised by the

Foreign Office. " (Kirby to David Edwards, July 22, 2005)

 

Also on July 20, an Independent editorial claimed that the Lancet

findings had been reached " by extrapolating from a small sample... While

never completely discredited, those figures were widely doubted " .

(Leader,

`The true measure of the US and British failure,' The Independent, July

20, 2005)

 

We challenged the Independent's Mary Dejevsky, senior leader writer on

foreign affairs:

 

" What is the basis for the claim that the sample was `small`? The

report authors told me that the sample was standard for research of this

kind, so that `we have the scientific strength to say what we have said

with great certainty. I doubt any Lancet paper has gotten as much close

inspection in recent years as this one has!' " (David Edwards to Mary

Dejevsky, July 21, 2005)

 

Dejevsky responded on August 10:

 

" personally, i think there was a problem with the extrapolation

technique, because - while the sample may have been standard for that

sort of

thing - it seemed small from a lay perspective (i remember at the time)

for the conclusions being drawn and there seemed too little account

taken of the different levels of unrest in different regions. my main

point, though, was less based on my impression than on the fact that this

technique exposed the authors to the criticisms/dismissal that the govt

duly made, and they had little to counter those criticisms with, bar

the defence that their methods were standard for those sort of surveys.

regards, mary " (August 10, 2005)

 

We responded on August 18:

 

" Thanks, Mary. You say that `personally' you `think there was a problem

with the extrapolation technique' because while the sample was standard

it was `small from a lay perspective`. Your argument then is that the

problem with the extrapolation technique was that people like you had a

problem with it because the sample seemed too small. That's a deeply

shocking response from a senior journalist writing in a serious newspaper

about such an important report. We are talking about +our+

responsibility for the mass death of civilians, after all.

 

" Should the methodology not be judged by the standards of science and

reason rather than some ill-informed `lay perspective`? Why on earth

would we judge anything of importance by the standards of an ill-informed

view?

 

" Your claim that the authors had little with which to counter criticism

is flatly false. I can send you many powerful replies provided to us by

the report authors in response to a range of (mostly trivial)

criticisms we found in the media.

 

Best wishes

 

David Edwards "

 

Dejevsky replied the same day:

 

" thanks - i obviously sounded more off-hand than i intended. i just

feel that extrapolation may be entirely sound when you can project over

relatively uniform areas (subject, geographical whatever), but that -

common sense suggests - it will be less reliable when the situation is so

uneven, as in iraq. this may be unjust and ill-informed, and maybe the

arguments from the report's authors were not sufficiently aired because

they were - in effect - suppressed. if you have some of the counter

arguments i would be interested to see them (beyond the defence that the

methodology is standard, tried and tested etc).

 

" incidentally, i think it is absolutely legitimate, and right, for

journalists to apply a common sense standard to scientific arguments and

methods. we should have been far more exacting over the intelligence

methodology that gave us saddam's wmd, for instance. all the best, mary "

(August 18, 2005)

 

This was a challenge we had to accept. We were disturbed by Dejevsky's

response and were keen to know what the team behind the Lancet report

would make of it. We contacted Les Roberts, a world renowned

epidemiologist and lead author of the report. Roberts responded on

August 22 with

an email which he asked us to forward to the Independent:

 

" Dear Mr. Kirby and Ms. Dejevsky,

 

" I was disappointed to hear that you felt our study was in some way

dismissed by Jack Straw's anemic response to our report in the Lancet

last

November. Serious reviews of our work and the criticisms of it were run

in the Financial Times, the Economist, the Chronicle of Higher

Education (attached above) and the WSJ [Wall Street Journal] Online on

August

5th. Closer to home, John Rentoul of the Independent solicited a

response to the Jack Straw letter last Nov. 21st and we responded with

the

attached letter [Not provided here]. I am told that it was printed by

your

paper.

 

" Many people, like Ms. Dejevsky, have used the word extrapolation to

describe what we did. When I hear people use that word they mean what is

described in my Webster's Unabridged: `1. Statistics. to estimate the

value of a variable outside its tabulated or observed range.' By this

definition and the one I hear used by everyone on this side of the

Atlantic, we did not extrapolate. We did sample. We drew conclusions from

within the confines of that universe from which we sampled. Aside from a

few homeless and transient households that did not appear in the 2002

Ministry of Health figures or households who had been dissolved or killed

since, every existing household in Iraq had an equal chance that we

would visit them through our randomization process.

 

" I understand that you feel that the sample was small: this is most

puzzling. 142 post-invasion deaths in 988 households is a lot of deaths,

and for the setting, a lot of interviews. There is no statistical doubt

mortality is up, no doubt that violence is the main cause, and no doubt

that the coalition forces have caused far more of these violent deaths

than the insurgents (p<.0000001).

 

" In essence this is an outbreak investigation. If your readers hear

about a sample with 10 cases of mad cow disease in 1000 British citizens

randomly tested, I am sure they would have no doubt there was an

outbreak. In 1993, when the US Centers for Disease Control randomly

called 613

households in Milwaukee and concluded that 403,000 people had developed

Cryptosporidium in the largest outbreak ever recorded in the developed

world, no one said that 613 households was not a big enough sample. It

is odd that the logic of epidemiology embraced by the press every day

regarding new drugs or health risks somehow changes when the mechanism

of death is their armed forces.

 

" The comments of Ms. Dejevsky regarding representativeness `(it seemed

small from a lay perspective (i remember at the time) for the

conclusions being drawn and there seemed too little account taken of the

different levels of unrest in different regions. my main point,

though, was

less based on my impression than on the fact that this technique exposed

the authors to the criticisms/dismissal that the govt duly made, and

they had little to counter those criticisms with, bar the defence that

their methods were standard for those sort of surveys.)' are also cause

for concern because she seems to have not understood that this was a

random sample.

 

" By picking random neighborhoods proportional to population, we are

likely to account for the natural variability of ethnicity, income, and

violence. Her words above strongly suggest that the Falluja numbers

should be included, rather than being used to temper the results from the

other 32 neighborhoods. Please understand how extremely conservative we

were: we did a survey estimating that ~285,000 people have died due to

the first 18 months of invasion and occupation and we reported it as at

least ~100,000.

 

" Finally, there are now at least 8 independent estimates of the number

or rate of deaths induced by the invasion of Iraq. The source most

favored by the war proponents (Iraqbodycount.org) is the lowest. Our

estimate is the third from highest. Four of the estimates place the death

toll above 100,000. The studies measure different things. Some are

surveys, some are based on surveillance which is always incomplete in

times of

war. The three lowest estimates are surveillance based.

 

" The key issues are supported by all the estimates that attribute

deaths to the various causes: violence is way up post-invasion and the

Coalition is responsible for many times more deaths than are the

insurgents.

The exact number is less important that these two indisputable facts

which helps us to understand why things are going badly and how to fix

them.

I hope these thoughts are helpful.

Sincerely,

Les Roberts "

 

Perhaps most damning in Roberts' reply - in light of media criticism of

the Lancet`s alleged exaggeration of civilian deaths - was his

refutation of the claim that the uneven levels of violent unrest in Iraq

compromised the accuracy of the figures. In fact the study not only

accounted

for this variability, it erred on the side of caution by excluding data

from Fallujah where deaths were unusually high. Moreover, other violent

hotspots - such as Ramadi, Tallafar and Najaf - were all passed over in

the sample by random chance. This suggests that the actual total of

civilian deaths is likely to be higher than 100,000. Indeed, it would

make

far more sense for the media to be criticising the report authors for

under-estimating the number of deaths.

 

We wrote to Dejevsky asking if she had received Roberts' response. She

replied on September 1:

 

" yes, and i understand the arguments. but i stick to my position that

extrapolation, however scientific and well-thought through is no

substitute for real figures. i know that the 'real' figures here do

not exist,

but i still think that extrapolation has obvious drawbacks which lay

the resulting figures open to question - and therefore vulnerable to govt

spokesmen who seek to discredit them. incidentally, my view on

extrapolation is really neither here nor there. my chief objection to

it is, as

i have just said, that it lays the figures themselves open to question

by those who have an interest in discrediting them.

all the best, mary "

 

Edward Herman, co-author with Noam Chomsky of the classic media study,

Manufacturing Consent, commented on this latest response:

 

" Massive incompetence in support of a war-apologetic agenda. Dejevsky

objects to the figures because they are vulnerable to discrediting for

reasons that make no sense. I wonder if she finds sampling discreditable

in all cases. " (Email to Media Lens, September 1, 2005)

 

This is something we were keen to find out by examining media responses

to other cases of sampling (see below and Part 2).

 

 

The Puzzled Epidemiologist

 

It is understandable that Roberts was puzzled by Kirby's and Dejevsky`s

responses. After all, in 2000 Roberts began the first of three surveys

in Congo for the International Rescue Committee (IRC) in which he used

methods akin to those of the Iraq study. Roberts' first survey

estimated that an astonishing 1.7 million people had died in Congo

over 22

months of armed conflict - on average 2,600 people were dying every day.

The IRC's president, Reynold Levy, put the figures in perspective:

 

" It's as if the entire population of Houston was wiped off the face of

the Earth in a matter of months. " (Hrvoje Hranjski and Victoria

Brittain, `2,600 a day dying in Congolese war,' The Guardian, June 10,

2000)

 

As Roberts says, the reaction could not have been more different:

 

" Tony Blair and Colin Powell quoted those results time and time again

without any question as to the precision or validity. " (Quoted, Lila

Guterman, `Researchers Who Rushed Into Print a Study of Iraqi Civilian

Deaths Now Wonder Why It Was Ignored,' The Chronicle Of Higher Education,

January 27, 2005; http://chronicle.com/free/2005/01/2005012701n.htm)

 

Indeed, within a month of Roberts' IRC report being published, the UN

Security Council passed a resolution that all foreign armies must leave

Congo, and later that year, the United Nations called for $140 million

in aid to the country, more than doubling its previous annual request.

Citing the study, the US State Department announced an additional $10

million for emergency programmes in Congo.

 

In his October 2001 speech to the Labour party conference, Tony Blair

said the international community could resolve many of the world's worst

conflicts:

 

" It could, with our help, sort out the blight that is the continuing

conflict in the Democratic Republic of the Congo, where three million

people have died through war or famine in the last decade. " (`Part one of

the speech by prime minister, Tony Blair, at the Labour Party

conference,' The Guardian, October 2, 2001)

 

The three million figure was produced by Roberts' study using

essentially the same methodology employed in Iraq. And yet, in

rejecting the

Lancet report out of hand, Blair told parliament:

 

" Figures from the Iraqi Ministry of Health, which are a survey from the

hospitals there, are in our view the most accurate survey there is. "

(David Hughes, `No inquiry into Iraq death toll, says Blair,' Daily Mail,

December 9, 2004)

 

Foreign secretary Jack Straw said the Government would examine the

Lancet figures " with very great care, " adding, " it is, however, an

estimate

that is not based on standard methodology for assessing casualties " .

(`This week's big issues: New attack on Blair's Iraq policy,' The

Independent, December 5, 2004)

 

Like so much that Straw says, this was simply untrue.

 

Blair's press spokesman said the government had a number of " concerns

and difficulties " about the methodology used, Patrick Wintour and

Richard Norton-Taylor reported in the Guardian:

 

" `The findings were based on extrapolation and treating Iraq as if it

were all the same in terms of the level of the conflict,` he said of the

study published in the Lancet. `This is not the case.' (Patrick Wintour

and Richard Norton-Taylor, `No 10 challenges civilian death toll,' The

Guardian, October 30, 2004)

 

Then, by way of a classic example of media propaganda, Wintour and

Norton-Taylor presented the government`s concocted `controversy` as

genuine:

 

" The controversy about the study largely turns on whether the sample

size of 7,800 people used by the team of US and Iraqi academics was

sufficiently large, and whether the 33 neighbourhoods chosen were

representative of the rest of the country. "

 

This, again, was false. In reality, there was and is no real

controversy about the size of the sample among scientists and serious

commentators. Michael J. Toole, head of the Center for International

Health at the

Burnet Institute, an Australian research organisation, said:

 

" That's a classical sample size. " Researchers typically conduct surveys

in 30 neighbourhoods, so the Iraq study's total of 33 strengthens its

conclusions. " I just don't see any evidence of significant

exaggeration, " Toole added. (Cited, Guterman, op. cit)

 

David R. Meddings, a medical officer with the Department of Injuries

and Violence Prevention at the World Health Organization, said surveys of

this kind always have uncertainty because of sampling and the

possibility that people gave incorrect information about deaths in their

households. However, Meddings added:

 

" I don't think the authors ignored that or understated. Those cautions

I don't believe should be applied any more or any less stringently to a

study that looks at a politically sensitive conflict than to a study

that looks at a pill for heart disease. " (Ibid)

 

The Independent helped fuel the myth of a controversially small sample:

 

" The Lancet said the research was based on a sample of fewer than 1,000

Iraqi households but said the findings were convincing. " (Colin Brown,

`Blair petitioned to set up inquiry into Iraqi war dead,' The

Independent, December 8, 2004)

 

The media also made much of a comment printed in the Washington Post by

Marc E. Garlasco, a senior military analyst at Human Rights Watch, who

said of Roberts` figures: " These numbers seem to be inflated. "

(Guterman, op. cit)

 

This was reported in the British media. Unreported anywhere, as far as

we can tell, is the fact that Garlasco has since admitted that he had

not read the Lancet paper at the time and calls his quote in the Post

" really unfortunate " . Garlasco says he told the reporter:

 

" I haven't read it. I haven't seen it. I don't know anything about it,

so I shouldn't comment on it. " But " like any good journalist, he got me

to. " (Ibid)

 

The large gap between the Lancet estimate and that of Iraq Body Count -

a constant feature of press coverage - is also not controversial. John

Sloboda, a professor of psychology at the University of Keele, and a

co-founder of Iraq Body Count, says his team's efforts will inevitably

lead to a count smaller than the actual figure because not every death is

reported in the news media.

 

Dr. Woodruff said, " Les [Roberts] has the most valid estimate. " (Ibid)

 

Dr. Toole agreed: " If anything, the deaths may have been higher [than

the Lancet study's estimate] because what they are unable to do is

survey families where everyone has died. " (Ibid)

 

Journalists, however, know better. Roger Alton, editor of the Observer

gave us his view of the Lancet report:

 

" I find the methodology a bit doubtful... " (Email to Media Lens,

November 1, 2004)

 

David Aaronovitch, then of the Guardian, told us:

 

" I have a feeling (and I could be wrong) that the report may be a dud. "

(Email to Media Lens, October 30, 2004)

 

Perhaps Aaronovitch's " feeling " is a close relation of Dejevsky's when

she writes " I just feel " the " extrapolation technique " is unsuited to a

situation as " uneven " as Iraq.

 

 

Part 2, comparing media responses to Roberts' work on Congo and Iraq,

will follow shortly.

 

 

Write to us at: editor

 

This is a free service but we need all the support we can get - please

consider donating to Media Lens: http://www.medialens.org/donate.html

 

A printer-friendly version of this alert can be found here for

approximately one week after the date at the top:

http://www.medialens.org/alerts/index.php

 

and then, thereafter, in our archive at:

http://www.medialens.org/alerts/archive.php

 

 

Visit the Media Lens website: www.medialens.org