Unhealthful News 30 - Figuring out who to believe (part 1)

The challenge that probably interests me more than anything else in my intellectual career is how to recognize when someone in a debate is clearly right without having to become expert in the subject matter, and the closely related problem of how to make it clear that you are right to outside observers.  The specific situation that most interests me is one where the observer of the debate is intelligent, generally well-versed in similar subject matter (science, politics), and genuinely interested in figuring out the truth but does not have any particular expertise in the specific subject matter and is not likely to acquire it.  I know this does not describe most situations where you might be trying to persuade someone, but it is a particularly important case (it describes many cases of trying to win over opinion leaders) and one that seems to present a surmountable challenge.

It is not so easy though.  I have tried doing it on several topics over the years, most recently tobacco harm reduction.  It is clear to me -- as someone who has made an extensive study of the epidemiology, economics (that is, what people like), politics, and ethics of the matter -- that there is no legitimate case to be made against encouraging THR unless someone accepts some very odd goals.  I am fairly certain I have identified the motives of those who oppose THR, and it is clear to me that if they openly admitted their real goals and preferences they would face opposition from the vast majority of the population.  They apparently agree with that assessment, since they hide their real motives beyond pseudo-scientific claims and rhetoric. 

That is what is clear to me.  But I know that to most observers it is not clear that the opponents of THR are trafficking in dishonest nonsense and misdirection.  They know how to use the vocabulary of science and make "sciencey" arguments (i.e., things that sound like they ought to be scientific claims, but really are not, in the spirit of Colbert's "truthiness").  To the completely uninitiated, it sounds like there is a scientific argument going on about health risks, when there is no legitimate debate on those points whatsoever.  To those who know a bit more, it seems like there is a legitimate debate going on about ethics and behavior, though there is barely more of a case against THR from those quarters than there is from the health science.  I know from experience that if I can sit down and talk with a member of my target audience (in particular, someone who is genuinely interested in learning the truth), I can almost always convince him or her of the truth. 

On the other hand, in such circumstances I generally have the advantage that my listener knows me, and thus knows that there is no chance I am the one spouting utter nonsense and simply lying about the science when I point out that the other side is doing just that.  So perhaps I have not quite achieved my goal of figuring out how to communicate the material to someone who wants to know the truth but does not know, going in, that I am the one that should be believed.  I think I have some insight into the topic, and would like to try to communicate some pointers, and at the same time try to better figure out how to do it myself.

I will explore that theme and goal periodically (maybe most every week) in this series because it is critical to what I am trying to do.  Eventually I will challenge a health news claim that you (some particular one of you) was inclined to believe.  Perhaps you will believe me because I have built up enough credibility through my other analyses, but maybe you will want me to make a case for why I am right that does not require you to start by assuming I am right.  For example, I suspect some readers must be asking (if they have read this series, particularly what I wrote yesterday), "why should we believe you, the iconoclast, rather than the icons of epidemiology in academia and government; if your calls for methodologic reform are right, why is almost no one adopting them."

In short, I want to explore what I can write and what you can realize that would lead you to believe me?

To start exploring "why should you believe me?", I would like to invoke the work of someone who I consider to be very talented at making a good case for why we should believe him.  Many of my readers follow Chris Snowdon's Velvet Glove Iron Fist blog, but may not be as familiar with his other book and blog The Spirit Level Delusion.  (If you are somewhat familiar you might want to check back, he added a lot of new material last week.)  This is his response to the book "The Spirit Level", by two epidemiologists, Richard Wilkinson and Kate Pickett (W&P), which claims that wealth or income inequality in a society (not the well-known problems of poverty, but inequality per se) causes all manner of health and social problems.  W&P's book apparently has a big following among British lefty pundits – those who are predisposed to support the policies that would be recommended were the book correct.  It has received much less attention in the U.S. (perhaps due to a dumb choice of title, which sounds New Age-ish to the ears of those of us who refer to that tool as just a "level", "bubble level", or perhaps "carpenter's level" and had never before heard the term "spirit level"), though it has been picked up by a few lefty pundits like Nicholas Kristof (which I commented on with dismay since I like Kristof's non-naive analyses). 

Snowdon's book (and associated interviews and blog posts) do a thorough job of debunking W&P and showing that it is utter junk science.  I am confident that no serious reader who was genuinely interested in learning the truth could read what he wrote and still believe that there was a legitimate debate about whether W&P's analysis was legitimate.  He could not easily win a fight by simply presenting his own assertions that were counter to theirs, hoping readers would choose to believe him.  Why would they choose to believe a journalist who is not backed by a major publisher over two university researchers?  (Most readers of this blog perhaps realize that a sharp scientifically literate journalist is probably a better scientific thinker than most people who publish epidemiology, but the average reader would not know this.)  There is a lot to mine from his presentation, and I can only touch on the answer today (more later).

The key to Snowdon's methods is pointing out, in ways that any sensible reader can see without an expertise in the subject matter, fundamental flaws in W&P's arguments.  The reader is then forced to either believe the critique or believe that Snowdon is fabricating gross out-and-out lies.  For example, in the first of his recent posts, Snowdon addresses W&P's implication that the many studies on the subject of inequality that came before them all supported their claim.  He first points out that if you read carefully, W&P only state that there were 200 papers that tested the relationship between income inequality and health.  They overlook the fact that quite a few of them conducted that test and concluded that there was nothing there.

(I am reminded of a Colbert episode from last week where he was joking at length about Taco Bell being accused of putting "beef" in its food that did not actually meet the U.S. Department of Agriculture legal definition of beef.  A Taco Bell spokesman responded to the accusations by pointing out that all of their beef was USDA inspected.  Colbert noted that "inspected" is not the same thing as "approved".  This further reminds me of a word that you may see in epidemiologic survey research, "validated", which basically is more like "inspected" though authors try to make it seem more like "approved".  I expect I will take up that point sometime in this series.)

Snowdon then went on to produce a series of quotes from previous researchers about findings that disagree with W&P's claim.  His key observation here is not that the evidence that W&P were wrong is more compelling than the evidence they were right.  That argument would require the reader to have expertise in the field to sort out the conflicting claims, to know whether all relevant studies were being cited, to know what exactly the quoted study results mean, etc.  But Snowdon's key point was a different one:

Those with a healthy scepticism will have noticed that I have only quoted studies that support one side of the debate. It’s a slippery and misleading trick and it is exactly what Wilkinson and Pickett do throughout The Spirit Level. The difference is that I made it clear from the outset of this book that there are many conflicting studies. Readers of The Spirit Level would be hard-pressed to guess that there was any debate at all.

So Snowdon has successfully pointed out to the reader that whatever the weight of the evidence might show, the evidence does not resemble what W&P claim it is.  To doubt that point would require believing that Snowdon was making up the quotes he wrote, something that would undoubtedly be picked up on by those on the other side and that would destroy his credibility, and thus is vanishingly unlikely.  (Also the interested reader could check it himself.)  He then redoubles the point by showing that a study that W&P cited as being the exemplary support for their thesis was actually quite equivocal.  Roughly speaking that one translates into, "if that's all you got, why did you even show up?"

(Aside:  This is also is support for a criticism I make about the way reference citations are used in health science.  Far too many authors, reviewers, and editors seem to think that it is appropriate to make a sweeping statement and then cite a single supporting study following it.  But all this does is create an illusion of increased credibility – finding a single quote or citation to support a particular claim is almost completely uninformative because there is some support for all but the most hopeless claims.  Authors need to either implicitly say "this broad claim is true; we assert this based on our expertise about the entire body of evidence and you will have to trust us", provide a complete review of the evidence, or direct the reader to further analysis of the point (a legitimate use of a citation, and should be used more often).  Citing a single piece of support as if it justifies a sweeping claim is just a way of trying to mislead readers.)

While pointing out that W&P are trying to misrepresent the weight of the evidence is not sufficient to deny any particular claim they make (Snowdon debunks many of their points in detail using other argument), is should be enough to make the open-minded reader seriously doubt everything that W&P claimed.  The general lesson is:  If authors can be shown to be denying the existence of opposing evidence and conclusions – not disagreeing, challenging its validity, or saying that it is overwhelmed by the evidence on the other side, but simply pretending it does not exist – this is pretty good evidence that they are not honest analysts and, moreover, do not think their case can stand on its merits.

Of course, W&P made it easy for Snowdon to shatter their credibility by making it so brittle.  They put the reader in the position of either believing they have unequivocal evidence for a "new theory of everything" (to quote from Snowdon's snarky subtitle), or concluding that they were just pulling a sales-job on the reader.  If they had behaved like scientists – recognizing the best contrary evidence and being properly equivocal – rather than peddlers or evangelists, it would have been necessary to explore the merits of their argument to challenge their claims and credibility.

Still, it is useful to figure out how to debunk as easy a target as is The Spirit Level.  We need to start with the challenge of winning one-sided debates before we can take on arguments that have some credibility.