Unhealthful News 125 - Predicting usefulness of statins: a calculation is not a recommendation

The recent news produced a great example of what happens when health reporters, used to being told what to say by touts overstating the implications of their research, encounter a technically valid analysis whose simplistic interpretation is misleading.  A new calculation by Nicholas J. Wald, Mark Simmonds, and Joan K. Morris assessed the current British method of calculating whether someone should receive statin drugs as a prophylactic against future cardiovascular disease event (CVD; i.e., a heart attack or stroke).  They found that a rule that says "just start using statins when someone turns 55" is almost as good as the current rule based that considers a function of age, sex, smoking status, diabetic status, serum cholesterol, and blood pressure.

(I am loathe to cite anything from the journal in which it appeared, PlosOne, and am not providing a link, because its publisher has an explicit policy of censoring anything that is pro-THR.  But the story is too good and the authors cannot be blamed for the behavior of the journal that is presumably outside of their control or knowledge, any more than chocolate consumers can be blamed for the child labor that supports their consumption.  And, yes, they do strike me as similar.)

What the authors did is a fun mathematical exercise which is really easier to understand than the logistic regressions that produce most epidemiologic reports.  The problem is that while reporters think they understand what regression results mean, the language here is completely foreign to them.  Basically, it consists of looking at predictors (age, etc.) to figure out what a combination of them, used optimally, tell you about how to sort people who could be protected from a CVD event by statins and those whom the statins would just be wasted on.  They then observed that age alone was almost as good as the current multi-factorial combination.

The most technical part is what "almost as good" means.  It means that the area under the ROC curve (don't worry about what that means if you do not know) is almost as large.  Equally accurate and easier to understand, it means that for any given level of false positives (e.g., wasting statins on only 20% of the population that would not, even without statins, have a CVD event in the time period) you get almost as low a false negative rate (e.g., failing to give statins to 10% of those who would have a CVD event if no one got statins).  The other way works too:  Almost as good can be defined as if you pick a percentage such that you do not want to miss more than that, and rate of giving statins of no value is not much higher.  For those not familiar with this way of thinking, keep in mind that you never know for sure about anyone, so every time you add someone to the treated group, there is a chance you are benefiting them and a chance that your effort is wasted.  The goal is to identify the ones with the highest risk so the chance the effort is wasted is lower, but it is never zero.

For putting numbers like these in perspective (note that the 20% and 10% are about the range the study is talking about), it is important to realize that 20% false positives is a lot less impressive than it sounds, since the vast majority of the population are at trivial risk for a CVD event.  Well over half the population are clearly at such low risk, due to youth, that it would make no sense to give them statins.  Thus the 20% roughly means that if you exclude the part of the population that it would be absurd to worry about, then you are giving the drug to most of the rest who get no benefit from it.  But this sounds like a worse idea than it really is, for the reason I conclude with:  this is actually probably way too small a percentage of the population to be wasting statins on.

The news reports picked up the result with some sense of amazement that age alone could be so useful.  But that is perhaps that simplest fact of epidemiology: age is very effective predictor of almost every disease, better than most any other information, excepting directed diagnostic tests for specific ticking timebombs.  What the new calculation found, that once you know someone's age, you know most of what you can possibly learn from other questions and simple tests about CVD risk, is not shocking.  Since age is very cheap and easy to detect, more so even than common cheap tests like cholesterol screening, if it tells us most of what we need, it might make sense to stop with that information.  That is what the calculation showed.  Unfortunately, the news reports took away: 

-The UK policy should just be to give statins to everyone over 55.
-Using age alone is as good a rule as any.

It might just be the usual tendency of the health press to exaggerate every finding, but I cannot help but wonder if the UK press got overly excited about the first author having the title "Sir".  To a non-Brit the references to a research "Sir Nicholas" rather than "Wald" are kind of odd.  They make you expect him to be confronting Monty Python's killer rabbit rather than Monte Carlo statistical simulations.  More to the point, the attempt to be deferential was misguided because these researchers never attempted to figure out the best answer.  They made the claim that given just the two choices, current methods and age only, the latter does almost as well at predicting, and it may not be worth the extra cost to collect all the data.

But the press conclusions were incorrect.  First, using age alone is not quite as good a predictor as starting with age and adding further information, despite the implications of some reports (I trust this is obvious – other information can never hurt, and will obviously tells us something).  Moreover, sex is just as easy a predictor variable to work with as age, and offers some additional predictive value; I am quite sure that a rule of "start men on statins at 54 and women at 56" is better in every way than "start everyone at 55", since men are at higher CVD risk earlier.  (Presumably these would not be the exact best ages, but they would undoubtedly better than keeping them both the same.)  There would be benefit with basically no cost, since men and women reading the newspaper are quite capable of understanding the message that they should start on different birthdays.  Once in a clinical setting, where someone can look up more details, why not include smoking status, blood pressure, etc.?  Perhaps it would turn out that cholesterol tests offered so little further information that they were not worth the expense and blood draw, but all the other information is basically free.  These further refinements would not be difficult to calculate (someone else could easily replicate the analysis if the authors for some reason did not want to add this).

But before anyone gets too excited about that, there is the need for some serious thinking about whether either of these standards, or any minor variation, is way too conservative.  The authors used a measure for assessing cost-effectiveness that is a little bit cryptic:  CVD-free years of life gained.  That is a combination of years of potential life lost (not dying early of CVD) and years living after a CVD event, which can have severely diminished in quality, but not necessarily.  But even if many of those years are quite healthy, some of the loss is quite severe.  Thus, using rules like either of those being discussed, with costs in the low five figure (varies depending on which model) range in pounds per CVD-free year gained, is extremely stingy.  It seems like we should definitely be willing to pay more for that, which basically means wasting medical care on more false positives to avoid a few more false negatives – i.e., to save more healthy life years for a mere few thousand pounds each. 

The health reporters seemed completely oblivious to the fact that the recommendation for where to draw the line hinges on this much more crucially then which predictive variables are used.  The difference between the predictive models was trivial if we consider that both of them impose a cost-effectiveness cutoff that is easily too conservative by half.  A spokesperson for the British Heart Foundation objected to the age-only proposal in the BBC story, but she seemed only worried that the recommendation would result in doing fewer tests.  Though she was ostensibly worried about treatment not being aggressive enough, she voiced no objection to the conservative cutoff (presumably she did not understand the study either).  A statement by the National Health Service in the same news report hinted at a similar failure to understand on the part of the government.  Wald himself apparently did nothing to help anyone understand the model when he was interviewed.  It is not all that surprising that the reporters did not understand, but these other people really had no excuse for getting excited about the calculations and ignoring the real question.  Except for Wald, maybe, who might have been distracted by a lurking killer bunny.