Unhealthful News 47 - Two stories of the problems with randomized trials

One simple lesson I have focused on in a few UNs is to not fall into the trap of believing that randomized trials (RCTs, experiments on people) are always the best way of gathering information.  The usual mantra is "you cannot always do them" or "they would be unethical for some things" and so they are not always possible but, the claim goes, "they are always best if you can do them".

A news item a few days ago emphasizes the ethical problem with RCTs, even when you can do them.  To try to determine (as they eventually confirmed) whether prenatal surgery for spina bifida really is better than surgery shortly after birth (it was generally believed that the former has better outcomes, but creates some risks of triggering premature birth and apparently they were wondering if it was enough better), the small number of American surgical centers that are capable of doing the former basically conspired to take away the option except as part of the RCT they were doing.  The article pointed out the heartbreak of mother with a fetus who had been diagnosed with spina bifida entering the trial and finding out she was randomized to the believed-inferior option.

Now this is not quite as nefarious as it sounds.  A conspiracy to deny a treatment to everyone except those in a trial is roughly equivalent to drug regulators allowing the use of a new drug only as part of a trial.  However, this was a case where the new and presumed better treatment had been available for two decades and was then taken away – and there is something significant that it was taken away by a conspiracy of private actors rather than a government agency that, for better or worse, we have granted police powers to.  (There is also the fact that a baby being born into a sufficiently rich family would presumably be able to get the better treatment outside the U.S., while those who rely on insurance or public funding are forced into the experiment; for the case of a new drug, the need to divert a controlled substance raises the bar for buying-out, at least somewhat.) 

There is one of those "yeah, sure – you just keep believing that if it lets you get through the day"-type myths, that if something is being studied in a trial then everyone's beliefs are supposed to be in a state of "equipoise" about the options, a fancy way of saying they have no idea which is better.  But those who claim this are generally taking refuge in the error of equating "we are not as sure as we would like about which is better" (true, or we would not be doing the experiment) with "we have no strong belief about which is better (almost always false, especially since the experiments are often motivated by the belief that a new therapy is better and we want to confirm that).  Typically the "not always ethical to do a RCT" claim refers to cases where one of the exposures is actively bad for people, as it would be if we were studying the effects of smoking.  But the quasi-ethicists who govern these matters seem to not understand that assigning people to a believed-inferior treatment (even if both treatments are better than nothing) is functionally no different from assigning something harmful.  This story exemplifies that. 

It is still possible to argue that, in effect, forcing some people to take the apparently inferior treatment for a while, in order to become sufficiently sure which is better, is an acceptable cost.  I suspect that most of us would agree with that characterization much of the time, that the value of the knowledge justifies using a group of people who have the bad luck to need the treatment as study subjects.  Nevertheless, the ethical concern about hurting some people for the greater good is not limited to those cases that are considered ethically unacceptable – virtually all RCTs suffer from it.

A more nuts-and-bolts problem, of the type that I have tried to explain, is reported in this story from this morning.  A review of studies found that many of them excluded older people, even though older people use far more medical care.  A closer look at the study, Examining the Evidence: A Systematic Review of the Inclusion and Analysis of Older Adults in Randomized Controlled Trials, reveals that the authors (the senior of whom, Rod Hayward, was one of my mentors and taught me much about evidence-based medicine and is one of the best thinkers in the field) actually seem to be more concerned about the general lack of research protocols that provide information that is optimal for making decisions about older people. These include failure to research diseases and outcomes that might be more relevant to older people, and not doing age-specific breakdowns to see if study results might only reflect the experience of the younger majority among the study subjects.  It appears that the news reporter focused on just the exclusions as either a hook or as a way to dumb-down the story.

All of the concerns seem valid.  The lack of breaking out subgroups is a problem with a lot of medical research (it is not so common with observational epidemiology); there is a strange obsession about reporting results by race, but amazingly little attention to the obviously more important characteristic, age.  As for lack of research on the right topics, there is much to complain about in terms of what research is prioritized.  Both of these are made worse for RCTs, where cost and logistics limit the number of studies that can be done (thus many topics are not covered) and the number of participants in each study (thus there subgroup analysis is less informative because the subgroups get so small).

The point about trials excluding older people, either explicitly or because requirements that subjects have good health and are free living are likely to disproportionately exclude them, is a problem more specific to experiments.  Some people are easier to experiment on than others, because they are less likely to suffer as badly from side effects, because they are more similar to one another, or because they are easier to deal with.  Such  problems do not haunt observational research quite so much:  Subjects do not have to be as cooperative or functional, and any side effects would have happened anyway, so you do not have to endeavor to exclude people because of them.  Also, you can usually collect more data and so having a homogeneous population, as a substitute for having enough data to analyze a mixed population, is not so necessary.

The bottom line remains that RCTs have advantages and disadvantages.  Perhaps observational evidence on spina bifida outcomes was so hopelessly confounded that it genuinely was difficult to be confident about what was better.  When I worked on similar issues (research about both birth outcomes and high-tech, high-skill, rare medical interventions) we saw lots of situations where this was the case.  Among the biggest problems (creating bias to unknown degrees) are that the highest-skilled clinicians tend to gravitate to the most exciting new technologies, and the sickest people often get the more aggressive treatment.  On the other hand, a lot of trials that are done end up telling us nothing much of value compared to the observational evidence we already have.  Sometimes they simply provide too little data, but often they provide a nice solid answer to a question we were not actually very interested in, like "does this treatment for a disease that is usually geriatric provide better outcomes for the rare 50-year-olds that get the disease?"