Survey research involves no magic opinio-meters that plug into people's heads. Everyone knows that. All of us have been survey subjects. And yet we seem all too willing to believe claims about survey results that could not possibly have been measured well by asking questions, even in the best case. The best you can hope for is to measure actions and characteristics of people that they are capable of meaningfully and accurately reporting. Moreover, since almost all surveys are based on checking a box, what is measured has to be measurable that way (i.e., it cannot require the conversation or free-text description that it takes to really communicate the details of someone's preferences, experiences, and motives).
GlaxoSmithKline released the results of a survey of smokers they commissioned regarding the potential ban of menthol in cigarettes. I am not going to address the underlying policy discussion, because I have already covered that. Rather, I would like to point out what readers (and those news outlets that basically just printed the content of that press release as news) should have noticed about some of the claims. Some observers might suggest that the main reason to question the study results is that it was paid for by someone who had a stake in the matter at hand (if a mandatory reduction in the quality of cigarettes causes people to try to quit, some of them will buy GSK's products that many people believe aid quitting). It is certainly true that the fact that they released the results (which they could have chosen to not do, unlike with, say, all scholarly research funded by tobacco companies that I know of, where the funder cannot suppress the result if they do not like it) tells us something about the results: GSK does not think that the results offer any competitive advantage as marketing information, but does think that they could influence the political debate in a direction they prefer.
The real reasons for doubting the results are not a lazy ad hominem criticism, though, but a scientific criticism of the claimed results, one that anyone can understand. The most reported result was, "if the FDA were to ban menthol cigarettes, four out of five menthol smokers (82 percent) say they are likely to try quitting." I saw this reported as "82 said they would quit", which obviously misinterprets the press release. But just consider the actual claim: What question was asked to get those responses? We do not know. But since I suspect that if you designed the right series of questions, you could get 3/4 of smokers to say they are likely to try quitting next month if the month contains a Thursday. "Try to quit" is a phrase that can mean very little effort or volition, but elicits the impression of something aggressive and likely to succeed.
Consider also, "almost 40 percent [of menthol smokers] say that menthol flavoring is the only reason they smoke." Again, by asking the right questions, it is possible to get many smokers to attribute their behavior entirely to social factors, daily patterns, the aesthetics, etc., rather than the drug delivery. We did some focus group research of smokers, and quite often no one would mention nicotine as part of their motivation. And, yes, it is practically mandatory to acknowledge that it is not all about nicotine. But it does not play any role in the motives of more than 40% of this subpopulation of smokers? Come on!
But, gee, maybe it is true: "The survey shows that menthol smokers feel "twice-addicted" – both to the menthol and to the tobacco – and most are attracted by the taste and feel of menthol cigarettes." Um, but wait a minute, how does a survey show that someone is addicted to menthol? It is pretty sketchy to even claim that someone is addicted to smoking at all, since addiction is not well-defined, so you either ask about addiction and get an answer based on each individual's idiosyncratic interpretation of the term, or you ask well-posed questions and idiosyncratically decide for yourself which of those represent addiction. But how can you possibly figure out whether menthol smokers are addicted to tobacco (presumably that means nicotine) and menthol independently?
Only those rare individuals who had been stuck able to buy only non-menthol cigarettes for a while could realistically assess how they would feel about smoking non-menthols, while measuring an independent "addiction" to smoking menthol, apart from tobacco, would require that someone had experienced… well, I have no idea what. Maybe vaping nicotine-free menthol e-cigarettes, which has probably been experienced by approximately zero of the respondents. And that says nothing about how it can be that 40% of the respondents smoke only for the menthol, but they are apparently also addicted to tobacco. Go figure.
The point is that you should go figure. These results are so absurd that no one should take them seriously. And everyone should learn enough from the most absurd claims that they do not take any of the other survey results seriously either. Whatever you might think of GSK and the ridiculously self-serving balance of the press release – about how wonderful their barely-functional products are and how lousy other options for quitting smoking are – it is not difficult to be able to see the absurdity of their conclusions about the survey results. Perhaps if they told us what they actually asked we could make something useful from the data, though I suspect that any survey that attributes 40% of smoking entirely to something other than nicotine is pretty much doomed.
Oh, but good news, the reader has no idea what the survey questions actually were and how they reached their conclusions, but they do make the effort to report, "For analysis, sample data were statistically weighted by race, gender, income, and menthol versus non-menthol smoking to accurately reflect the current population of adult smokers on each of these dimensions." The humor of that might be lost on many readers, so to offer an analogy, imagine someone making a salad of Miracle Whip and iceberg lettuce, and serving it over Jello, but making sure to sprinkle it only with organic sea salt -- or, if you want a less colorful metaphor, call it polishing the brass on the Titanic.
GlaxoSmithKline released the results of a survey of smokers they commissioned regarding the potential ban of menthol in cigarettes. I am not going to address the underlying policy discussion, because I have already covered that. Rather, I would like to point out what readers (and those news outlets that basically just printed the content of that press release as news) should have noticed about some of the claims. Some observers might suggest that the main reason to question the study results is that it was paid for by someone who had a stake in the matter at hand (if a mandatory reduction in the quality of cigarettes causes people to try to quit, some of them will buy GSK's products that many people believe aid quitting). It is certainly true that the fact that they released the results (which they could have chosen to not do, unlike with, say, all scholarly research funded by tobacco companies that I know of, where the funder cannot suppress the result if they do not like it) tells us something about the results: GSK does not think that the results offer any competitive advantage as marketing information, but does think that they could influence the political debate in a direction they prefer.
The real reasons for doubting the results are not a lazy ad hominem criticism, though, but a scientific criticism of the claimed results, one that anyone can understand. The most reported result was, "if the FDA were to ban menthol cigarettes, four out of five menthol smokers (82 percent) say they are likely to try quitting." I saw this reported as "82 said they would quit", which obviously misinterprets the press release. But just consider the actual claim: What question was asked to get those responses? We do not know. But since I suspect that if you designed the right series of questions, you could get 3/4 of smokers to say they are likely to try quitting next month if the month contains a Thursday. "Try to quit" is a phrase that can mean very little effort or volition, but elicits the impression of something aggressive and likely to succeed.
Consider also, "almost 40 percent [of menthol smokers] say that menthol flavoring is the only reason they smoke." Again, by asking the right questions, it is possible to get many smokers to attribute their behavior entirely to social factors, daily patterns, the aesthetics, etc., rather than the drug delivery. We did some focus group research of smokers, and quite often no one would mention nicotine as part of their motivation. And, yes, it is practically mandatory to acknowledge that it is not all about nicotine. But it does not play any role in the motives of more than 40% of this subpopulation of smokers? Come on!
But, gee, maybe it is true: "The survey shows that menthol smokers feel "twice-addicted" – both to the menthol and to the tobacco – and most are attracted by the taste and feel of menthol cigarettes." Um, but wait a minute, how does a survey show that someone is addicted to menthol? It is pretty sketchy to even claim that someone is addicted to smoking at all, since addiction is not well-defined, so you either ask about addiction and get an answer based on each individual's idiosyncratic interpretation of the term, or you ask well-posed questions and idiosyncratically decide for yourself which of those represent addiction. But how can you possibly figure out whether menthol smokers are addicted to tobacco (presumably that means nicotine) and menthol independently?
Only those rare individuals who had been stuck able to buy only non-menthol cigarettes for a while could realistically assess how they would feel about smoking non-menthols, while measuring an independent "addiction" to smoking menthol, apart from tobacco, would require that someone had experienced… well, I have no idea what. Maybe vaping nicotine-free menthol e-cigarettes, which has probably been experienced by approximately zero of the respondents. And that says nothing about how it can be that 40% of the respondents smoke only for the menthol, but they are apparently also addicted to tobacco. Go figure.
The point is that you should go figure. These results are so absurd that no one should take them seriously. And everyone should learn enough from the most absurd claims that they do not take any of the other survey results seriously either. Whatever you might think of GSK and the ridiculously self-serving balance of the press release – about how wonderful their barely-functional products are and how lousy other options for quitting smoking are – it is not difficult to be able to see the absurdity of their conclusions about the survey results. Perhaps if they told us what they actually asked we could make something useful from the data, though I suspect that any survey that attributes 40% of smoking entirely to something other than nicotine is pretty much doomed.
Oh, but good news, the reader has no idea what the survey questions actually were and how they reached their conclusions, but they do make the effort to report, "For analysis, sample data were statistically weighted by race, gender, income, and menthol versus non-menthol smoking to accurately reflect the current population of adult smokers on each of these dimensions." The humor of that might be lost on many readers, so to offer an analogy, imagine someone making a salad of Miracle Whip and iceberg lettuce, and serving it over Jello, but making sure to sprinkle it only with organic sea salt -- or, if you want a less colorful metaphor, call it polishing the brass on the Titanic.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.