Unhealthful News 9 – recognizing the limits of research as simple as counting

If you followed the news today about the shooting in Tucson of a member of congress, a federal judge, and several others, you will have noticed that the "several" changed a few times.  (I will leave it to others [update: e.g. Krugman] to analyze the implications of this event for the American culture of violence (and to comment on the overlooked painful irony that one of the victims that was killed was born on 9-11 2001).  Those on the political right are falling over themselves trying to explain why this is completely unrelated to their practices of trafficking in violent imagery and rhetoric toward their political opponents, and of supporting unfettered access to weaponry.  Yeah, right.  Who could have predicted that such incitement and access would reach people who are inclined to violence?)

Here is the bit that is relevant for health reporting:  News reporters seem to think it is possible to report exact measures, such as the number of people hurt or otherwise affected by something, down to the last person.  Sometimes this error is caused by researchers or government agencies who make absurd claims.  But the reporters contribute, and they should know better from just following the news, since something as simple as the count of people shot in the Tucson incident changed over the course of a day.  This is not unexpected by anyone other than the reporters who for many hours insisted that it was exactly 18 before changing the reports to say it is exactly 26 (those were the numbers that CNN used, others varied, and it seems to be converging on 20 as I write this). 

Updating estimates when better information become available is fine, of course.  It is the only way to function intelligently.  But presenting them as if they are exactly right in the first place (not "we have reports of…" or "it is currently believed that…" but "the number is…") is not, and it is downright embarrassing to insist that you have the number exactly right, change it substantially, and then resume insisting that you have it exactly right (because you got it wrong the first time, and were sure you were exactly right, you must be right now??).  You might recall that during the first day or two following the 9-11 attacks the news was reporting the number of deaths, down to the last digit (something like 8456), as if it were known, until revising this number down by half.  You probably do not recall that months later the official count (still being reported down to the last digit) was again revised, changing by several tens.  This means that not only was the last digit wrong, but so was the third digit.

I emphasize digits because they offer an easy method for reporting numbers that are not known exactly:  rounding.  This is taught in science classes.  If you are not sure of your number within plus or minus 10, round the units digit to zero (8460, not 8456).  If you are extremely unsure, as is often the case, reporting only one figure ("current estimates put the toll at 8000") is best.  It is a rough and not precisions defined method, and there are better more formal ways to do it, but this simple method goes a long way. 

Of course, someone claiming greater precision than can possibly exist does have its advantages.  It is a pretty good clue to the reader that they really do not understand the limits of the evidence, and thus probably do not really understand the science they are writing about.  When someone does a study of 88 people and reports that something was true for 68.2% of them (rather than rounding to 68% in the research report and recognizing further uncertainty by saying "about 70%" or "about two-thirds" when summarizing it), you can be sure that they are trying to convince you that the research is more useful than it really is.

There is more to say about both precision and rounding, so I will come back to it.  I will just mention that people naturally round when they cannot give a precise estimate (like when answering the surveys used by researchers) which is then often misinterpreted by researchers.  There is a good joke that gets it exactly backwards:  A natural history museum employee is asked how old a particular fossil is, and responds "twenty million and four years".  When the museum patron expresses wonder that anyone could know such a thing, the employee responds "well, when I took this job, they told me it was twenty million years old, and I have been here four years."  What makes it funny, of course, is that no one would ever make such a mistake, except that supposed experts almost always do when reporting health research.