Unhealthful News 14 - three modern myths: Ophiuchus, miracles, and RCTs

I wonder which one of these myths from this morning's news attracts more blind devotion.  Worldwide, it is probably catholic church miracles, but among people who read the health research, I suspect it is RCTs.

It is a bit of a thematic stretch, but I just have to report on the fact that according to a proposed change in the rules of astrological signs (which seems to be creating a schism in the faith), I am now an "Ophiuchus".  As a member of the 1/13th of the population to have the excitement of being assigned the new unpronounceable addition to the "Pisces" and "Sagittarius" series, I am supposed to be "honest, intellectual, sexually magnetic, prone to change and of course, jealous" and to "require special attention to understand".  Well, I guess I like some of that (though why is jealous proceeded by "of course"??).  As for the "understand" bit, I try hard to make my writing accessible to everyone, but should I worry that they are talking about my blog?  (Which, I suppose would be much more "egocentric" than "jealous".)  Be sure to give me feedback if that is the case.

As you might expect, the news reports on the astrological realignment were mostly whimsical, and full of "people who believe this kind of thing say…" caveats if they took a more serious tone.  By contrast, reports on the late pope, John Paul II, being on the fast track to sainthood reported that he had carried out a miracle (required for sainthood – it is not enough to just have a worldly resume that the committee is impressed by) as if it were just a simple fact.  It seems that someone who prayed to him sometime after his death reported that she was cured of a usually incurable disease.  Fortunately for the sainthood committee, it is sufficient to do a recall-based study of a single subject with vaguely-defined outcomes and no comparison group.  It is still probably better quality than most of the research done on environmental tobacco smoke, but that is not high praise.

And speaking of health science issues that are hot enough to provoke a fistfight, everyone is reporting on a new review of health outcomes from Britain (good news for the people: available as a real (open access) publication) that recommends against exclusively breast feeding past four months in places like Europe where babies are unlikely to get infections from other food sources.  This was primarily based on concerns about some babies not getting enough nutrients from exclusive breastfeeding during months five and six.  I might address the substance of the analysis later, but I just want to focus on one statement in the article right now: "Apart from two randomised trials in Honduras, the studies were observational, precluding proof of causation for the outcomes examined…."  I will not repeat what I have written about the annoying failure of non-scientists (this was written by medics) to understand that you can never prove causation; I wrote a post intended to be a permanent footnote on that point, so will just reference it.

I will address a more practical point here:  This is a case where it should be pretty obvious that randomized trials not only do not "prove" causation, but they actually introduce some serious problems when studying something like this.  Randomized trials have the advantage that they randomize the confounding (which is to say that characteristics of people, apart from the intervention being researched, that might affect the outcome are randomized with respect to the intervention), so it can be modeled as statistical error rather than being systematically associated with the intervention (which is to say, it avoids the situation where those characteristics affect whether someone would have chosen the intervention if given free choice, thereby mix together the effects of the intervention and the effects of those characteristics that caused someone to choose the intervention).  That is a good thing.  But it comes at the cost of forcing people to behave in funny ways that may not resemble real life.

In this case, mothers' choices about whether to exclusively breastfeed in months five and six will normally be affected by such things as whether the baby seems to be getting enough nourishment and whether breastfeeding is making mother and baby happy or miserable.  The resulting choices are probably associated with how healthy the baby is, presumably in a sensible way (i.e., those mothers who think or are told their baby is not getting enough nutrients, perhaps because he acts like he is starving all the time, will supplement his feeding).  So a randomized trial avoids the "problem" of women trying to figure out what is best for the baby and acting on it, and therefore could measure what happens if everyone is forced to stop exclusive breastfeeding at 4 months or forced to continue to 6 months.  But that is not a very interesting question.  The much more interesting question is what is better for those who are unsure about which of those choices seems better.  Observational studies may not be great at getting at that answer either, but a randomized trial is pretty much guaranteed to make a hash of it, averaging together results for people who had really good reasons to not make a particular choice with others who were indifferent.

Consider this analogy:  Are authors better at writing research papers while listening to music or in silence?  An observational study that compared writing speed, focus, etc. of those who choose to listen to music to those who do not would have some confounding:  Perhaps younger people like music and have different work skills (which we could control for) or perhaps people who work in silence are just naturally more focused (which we would have a very hard time controlling for).  But imagine the randomized trial:  Those who were assigned to listen to music would include many who simply cannot work with music playing (or perhaps the music would just be chosen badly – trials are also bad at adjusting the intervention so that it is optimal for different people).  Since there are relatively few people who simply cannot work in silence, this would bias the study to almost inevitably "showing" that listening to music makes you work less well.  That is true for some people, and they drag down the average - but they would have never done it in real life, so the implication ("turn off the music and you will work better") is wrong.  Similarly if women who are pretty sure their baby needs more food were stuck in an "exclusively breast feed until six months" group (and did not have the self-confidence to just depart from their assigned role), the health of the babies in that group would suffer on average.

Furthermore, presumably those two randomized trials did not compare exactly the old recommendation with the new one, and they certainly did not compare women in a rich, well-fed Western population (the target of the recommendations).  Thus, they were an unconfounded test of something different from what we are actually trying to figure out.  Not so good after all.  Observational studies, with a more realistic range of variation in behaviors, more often allow exactly the comparison of interest.

So, just to make the point clear:  Randomized trials are not a useful study design for answering some types of questions, and a particular randomized trial is likely to be even less useful if it was not designed to answer the exact specific question of interest.  Moreover, even when a randomized trial is indeed better than the alternatives, it almost never measures exactly what we want to know (except for very simple either-or choices that are actually made in controlled clinical settings).  This says nothing about the new breastfeeding advice, of course, though it does point out that the authors' scientific knowledge is… well, on par with that of most people writing about health research.


Now I wonder which of the three myths I mused about today will generate the first bit of hate mail.