One of the reasons that there is so much junk published in health science, particularly of the kind that make for junk news, is that there are no repercussions for declaring a dramatic, telegenic conclusion that turns out to be absurdly wrong. Every real science allows for the possibility that a particular study, done a particular way on a particular day, might produce a result that is "wrong" in the sense of running contrary to what is (or later becomes) agreed upon wisdom on the point. That is the nature of science, of random error, of unexpected effects of methodologic choices, etc. What makes a mess of it is if someone doing a little faulty study, whose results are what they are, thinks he is writing Principia Mathematica, or at least "The mortality of doctors in relation to their smoking habits". In most fields, one of the key lessons taught in graduate school is that you – each individual student – know very little compared to the extent of human knowledge in and around your subject of study. This produces an epistemic modesty that makes for better science.
This message seems to be lost on health researchers. Part of the problem is that a lot of researchers are trained only as physicians, not as scientists, and clinical training usually includes the message that you are supposed to act like a god and never admit to your ignorance. Actually, that is not entirely fair – it is perhaps more a matter that medical training causes people to become unaware of their ignorance, a trick of mind control that is utterly baffling and might be of interest to the psychologists working at Guantanamo. When this god complex spills over into research, it creates a tendency to say "my little lame study showed X, and therefore X is True and the world should be changed in the following way based on that…."
Of course, this does not explain the behavior of health researchers who studied science and got research degrees rather than professional degrees, though maybe some of it is a spillover effect. Also, epidemiologists, toxicologist, etc. also get to play god sometimes, influencing or even controlling decisions that are important to how people live their lives.
But the bigger problem, I believe, is that no one in health science is ever asked to say they are sorry for a faulty conclusion they adamantly declared. Changing your beliefs based on evidence is the mark of a scientific or otherwise intelligent mind (though political pundits like to call it "flip flopping"). But failing to recant the old conclusions, refusing to admit that you made them, and never explaining what made you wrong before and makes you correct now are dishonest behaviors that warrant embarrassment and public criticism. In a world of such ethics, having to change your conclusions means either admitting you were wrong or being justly criticized for failing to do so. In that world you have a lot of incentive to not over-conclude. Again, this does not mean there is any embarrassment in saying "my research did Y and the result was X", even though this turns out to contradict better evidence, so long as you stop there. But if you make a press release and adamant declarations about X being true, you deserve a reputation as a bad scientist and someone whose opinions should not be trusted.
That is not the world of health research and publishing.
I was thinking about this because here at Vapefest this weekend, several people have mentioned to me the new research by Thomas Eissenberg and his research group, who notoriously reported – and aggressively touted to the media – that e-cigarettes deliver no nicotine to users. Basically, he did a badly designed study (a minor error) and then implied he had created a Great Work for the Ages (a very major error). That group's new study (described here) discovered what, oh, maybe a million people already knew from personal experience: E-cigarettes do deliver nicotine after all.
Surprise!
From what has been reported, the study might well exhibit the same kind of naivety that got the researchers into trouble in the first place: Doing one tiny study of an extremely heterogeneous phenomenon and making a big deal about the quantitative results. At least this time the results are rather less absurd than "zero", but they are still quantitatively meaningless. If you told me what levels of nicotine absorption you wanted to get from a study of three (yes, just three) vapers, I am sure I could design a study of three people that within a few tries would give you those numbers. What remains of the scientific integrity of research on nicotine and tobacco can only hope that they do not imply the specific quantitative results matter when they publish their results, let alone that they make policy recommendations based on them. We should not be too optimistic, though, based on this.
The rumor here (though I could not find any documentation of this) is that Eissenberg's message then switched from "these things are bad because they do not give smokers the nicotine they need" to "these things are bad because they deliver so much nicotine that users might get too much". This might not be true – it might be that he is becoming a "friend of the cause" of vaping, as some users speculated.
But either way, the major crime against science and common sense was the original conclusion. If you "discover" something based on one lousy study, you should not do an interview about it on CNN. And if your "discovery" is contrary to what lots of people with better evidence than your own are pretty sure is true, then you are an utter fool for making declarations on national television.
Oh, but wait, maybe not. You would be an idiot if there were any repercussions for staking your scientific reputation on such a claim. But if you publish in public health, and especially if you want to be an influential pundit in public health, then making an incorrect over-the-top claim to the press makes you a shrewd politician not a bad scientist. Much like it is in Hollywood, not so much like it is in science, any publicity is good publicity. Consider what Eissenberg and company have contributed to the discussion and science about e-cigarettes. Then realize that his name comes up in discussions of experts on the subject (and he self-represents as one). Welcome to a world where Charlie Sheen is on television giving advice about relationships and healthy living thanks to his widely-reported contributions to these areas.
This message seems to be lost on health researchers. Part of the problem is that a lot of researchers are trained only as physicians, not as scientists, and clinical training usually includes the message that you are supposed to act like a god and never admit to your ignorance. Actually, that is not entirely fair – it is perhaps more a matter that medical training causes people to become unaware of their ignorance, a trick of mind control that is utterly baffling and might be of interest to the psychologists working at Guantanamo. When this god complex spills over into research, it creates a tendency to say "my little lame study showed X, and therefore X is True and the world should be changed in the following way based on that…."
Of course, this does not explain the behavior of health researchers who studied science and got research degrees rather than professional degrees, though maybe some of it is a spillover effect. Also, epidemiologists, toxicologist, etc. also get to play god sometimes, influencing or even controlling decisions that are important to how people live their lives.
But the bigger problem, I believe, is that no one in health science is ever asked to say they are sorry for a faulty conclusion they adamantly declared. Changing your beliefs based on evidence is the mark of a scientific or otherwise intelligent mind (though political pundits like to call it "flip flopping"). But failing to recant the old conclusions, refusing to admit that you made them, and never explaining what made you wrong before and makes you correct now are dishonest behaviors that warrant embarrassment and public criticism. In a world of such ethics, having to change your conclusions means either admitting you were wrong or being justly criticized for failing to do so. In that world you have a lot of incentive to not over-conclude. Again, this does not mean there is any embarrassment in saying "my research did Y and the result was X", even though this turns out to contradict better evidence, so long as you stop there. But if you make a press release and adamant declarations about X being true, you deserve a reputation as a bad scientist and someone whose opinions should not be trusted.
That is not the world of health research and publishing.
I was thinking about this because here at Vapefest this weekend, several people have mentioned to me the new research by Thomas Eissenberg and his research group, who notoriously reported – and aggressively touted to the media – that e-cigarettes deliver no nicotine to users. Basically, he did a badly designed study (a minor error) and then implied he had created a Great Work for the Ages (a very major error). That group's new study (described here) discovered what, oh, maybe a million people already knew from personal experience: E-cigarettes do deliver nicotine after all.
Surprise!
From what has been reported, the study might well exhibit the same kind of naivety that got the researchers into trouble in the first place: Doing one tiny study of an extremely heterogeneous phenomenon and making a big deal about the quantitative results. At least this time the results are rather less absurd than "zero", but they are still quantitatively meaningless. If you told me what levels of nicotine absorption you wanted to get from a study of three (yes, just three) vapers, I am sure I could design a study of three people that within a few tries would give you those numbers. What remains of the scientific integrity of research on nicotine and tobacco can only hope that they do not imply the specific quantitative results matter when they publish their results, let alone that they make policy recommendations based on them. We should not be too optimistic, though, based on this.
The rumor here (though I could not find any documentation of this) is that Eissenberg's message then switched from "these things are bad because they do not give smokers the nicotine they need" to "these things are bad because they deliver so much nicotine that users might get too much". This might not be true – it might be that he is becoming a "friend of the cause" of vaping, as some users speculated.
But either way, the major crime against science and common sense was the original conclusion. If you "discover" something based on one lousy study, you should not do an interview about it on CNN. And if your "discovery" is contrary to what lots of people with better evidence than your own are pretty sure is true, then you are an utter fool for making declarations on national television.
Oh, but wait, maybe not. You would be an idiot if there were any repercussions for staking your scientific reputation on such a claim. But if you publish in public health, and especially if you want to be an influential pundit in public health, then making an incorrect over-the-top claim to the press makes you a shrewd politician not a bad scientist. Much like it is in Hollywood, not so much like it is in science, any publicity is good publicity. Consider what Eissenberg and company have contributed to the discussion and science about e-cigarettes. Then realize that his name comes up in discussions of experts on the subject (and he self-represents as one). Welcome to a world where Charlie Sheen is on television giving advice about relationships and healthy living thanks to his widely-reported contributions to these areas.
Very funny closing statement and I do find it fascinating that science is one of the industries where one is not held responsible for incorrect statements and the consequences for such statements are nil.
ReplyDeleteThanks.
ReplyDeleteJust to clarify, my indictment is not aimed at all of science, just public health and medical sciences. It is undoubtedly true in some other areas too (see my forthcoming post -- probably UN107 relating to economics), but I know it is not true in some sciences. The difference might come down to sciences where you can make a living impressing the pundits, creating a cycle of "experts" who are really hacks and pundits, and those where you are being judged by serious scientists because there are no casual pundits.