By Steven E. Moore

After doing survey research in Iraq for nearly two years, I was surprised to read that a study by a group from Johns Hopkins University claims that 655,000 Iraqis have died as a result of the war. Don't get me wrong, there have been far too many deaths in Iraq by anyone's measure; some of them have been friends of mine. But the Johns Hopkins tally is wildly at odds with any numbers I have seen in that country. Survey results frequently have a margin of error of plus or minus 3% or 5% -- not 1200%.

The group -- associated with the Johns Hopkins Bloomberg School of Public Health -- employed cluster sampling for in-person interviews, which is the methodology that I and most researchers use in developing countries. Here, in the U.S., opinion surveys often use telephone polls, selecting individuals at random. But for a country lacking in telephone penetration, door-to-door interviews are required: Neighborhoods are selected at random, and then individuals are selected at random in "clusters" within each neighborhood for door-to-door interviews. Without cluster sampling, the expense and time associated with travel would make in-person interviewing virtually impossible.

However, the key to the validity of cluster sampling is to use enough cluster points. In their 2006 report, "Mortality after the 2003 invasion of Iraq: a cross-sectional sample survey," the Johns Hopkins team says it used 47 cluster points for their sample of 1,849 interviews. This is astonishing: I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points.

Neither would anyone else. For its 2004 survey of Iraq, the United Nations Development Program (UNDP) used 2,200 cluster points of 10 interviews each for a total sample of 21,688. True, interviews are expensive and not everyone has the U.N.'s bank account. However, even for a similarly sized sample, that is an extraordinarily small number of cluster points. A 2005 survey conducted by ABC News, Time magazine, the BBC, NHK and Der Spiegel used 135 cluster points with a sample size of 1,711 -- almost three times that of the Johns Hopkins team for 93% of the sample size.

What happens when you don't use enough cluster points in a survey? You get crazy results when compared to a known quantity, or a survey with more cluster points. There was a perfect example of this two years ago. The UNDP's survey, in April and May 2004, estimated between 18,000 and 29,000 Iraqi civilian deaths due to the war. This survey was conducted four months prior to another, earlier study by the Johns Hopkins team, which used 33 cluster points and estimated between 69,000 and 155,000 civilian deaths -- four to five times as high as the UNDP survey, which used 66 times the cluster points.

The 2004 survey by the Johns Hopkins group was itself methodologically suspect -- and the one they just published even more so.

Curious about the kind of people who would have the chutzpah to claim to a national audience that this kind of research was methodologically sound, I contacted Johns Hopkins University and was referred to Les Roberts, one of the primary authors of the study. Dr. Roberts defended his 47 cluster points, saying that this was standard. I'm not sure whose standards these are.

Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million.

When I pointed out these numbers to Dr. Roberts, he said that the appendices were written by a student and should be ignored. Which led me to wonder what other sections of the survey should be ignored.

With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq. However, there is a definitive method of establishing if it is. Recording the gender, age, e

I was in Iraq for nine months myself, and your analysis of a Westerner's life resonates. I was confined to my hotel for much of the time, and felt my freedoms erode as I became increasingly hunted.

However, I was in Iraq doing public opinion research. While I was increasingly unable to travel, I was consistently getting info on what Iraqis are thinking, and still get frequent polls.

51% of Iraqis think that Iraq is on the right track. Compare this, for instance, to Michigan where Governor Granholm inspires a 34% "right track" response on polling in her state.

More than 60% of Iraqis think that the Iraqi government is doing a good job.

Iraq is a tough place to be a Westerner, but the Iraqis are optimistic, statistically speaking.

20,000 to 30,000 insurgents, many from outside Iraq, are trying to oppress the 20 million or so Iraqis who say on polls that they want democracy, rather than a theocracy or a baathist style dictatorship.

In Baghdad, thousands of bodies have been pulled from the Tigris, but the deaths aren't reported. How bad is the violence?

Still, even these figures don't tell the whole story. For that, a visit to Medical City is in order. The Ministry of Health has instituted a strict policy for journalists, requiring them to seek permission before visiting the facility. Those allowed in get only a truly sanitized tour; more often than not reporters are barred from entering. But at the gate, guards who have worked at the facility tell a chilling tale. "Last year, I saw maybe 1,000 bodies a month coming into the morgue," says one man who, fearing for his life, requested his name not be published. "Now we're getting nearly 1,000 a week."

All, he says, are victims of sectarian violence, both Sunni and Shia, but the officials at the morgue inside Medical City will not tell you that. "The officials don't want us talking to the media," says another guard, also requesting anonymity. "I've heard them telling people that most of the deaths are because of terrorists, but I've also seen the bodies myself and I can tell you that most of them were executed by death squads."