Two days ago, I wrote a primer on confounding. A couple of times this week I discussed how controlling for confounding can be done wrong. But that might have been somewhat of a misleading emphasis. Sometimes when you are too deep into something you forget that the real problems are more basic than the ones you usually bother to struggle with. In the case of epidemiology, the bigger problem then controlling for confounders badly is pretending that confounders do not exist at all. It is really a huge problem when there is literally no way to control for the worst of them.
This week a study was reported – as naively as can be imagined – that purported to show that old people suffer much more impairment when trying to multitask, in particular to talk on the phone while crossing the street, as compared to college kids. This was based on some simulator studies and a wonderful example of uncontrolled confounding. As an aside, I am greatly amused that the reports all specified that the tasks were "talking on a cell phone while crossing the street" as if (a) a cell phone is some novel technology that needs to be specified, (b) that "cell" is the standard phrasing (elsewhere in the anglophone world it is a "mobile"), and most important (c) there is any other kind of phone you can talk on while crossing the street. (Look out for that cord running across the crosswalk up ahead!)
As is the case with so many psychology studies, the main conclusion is almost certainly correct, the problem being that it does not really follow from the study. The biggest problem is something that epidemiologists are quite familiar with, but that psychologists may not be, the age-period-cohort issue. That is, if we study current 70-year-olds (a particular age), right now (a particular period), then you inevitably have people who are entirely from one cohort (born about 1941). If it is conceivable that an outcome is caused by either age (older people have trouble multitasking compared to people with young intensely-alert brains) or cohort (people born in 1941 only learned to walk while talking on the phone at age fifty-something, if ever, while current college students have been walking and phoning since childhood), or both, then the study makes it impossible to sort out the effects. Assuming we are interested in the effect of being old, and not the effect of having come of age without phones that can cross the street, we are stuck. Since there is perfect collinearity of the two, there is no variable than can be used to control for the effect of the second because there is nothing that can distinguish them. We would need a study that separated age from cohort (i.e., we would have to wait until we could study people of the same age from a relevantly different cohort, one that learned to cross streets when kids already had cell phones), which will require waiting about 50 years.
There were other related problems. The participants were not actually crossing a street, but were walking on a treadmill with display screens around them creating a virtual reality. In other words, they were playing a first-person video game (with their bodies as the controller), something that college kids are almost all comfortable with, but that those aged 60-80 are much less familiar with than they are with actually crossing the street. Thus, part of the measured effect was neither the age nor the cohort effect associated with being on the phone, but the cohort effect of not being used to video games. The researchers tried to make a big deal about how the study shows that walking is not as natural as we think it is; they are no doubt right that it is not natural when "walking outside" involves a treadmill and synchronized video screens in a dark room.
If we really wanted to solve some of the study problems sooner, we would not really have to wait 50 years or get old people run over. There are some clever tricks, like finding some young people who were not used to having a mobile phone, or comparing older people who have more and less experience with them (and even with video games). That introduces other more subtle confounding, but at least it gets some traction on the biggest confounding problem. Notice the subtext lesson here: One way to get rid of confounding is to control for covariates but an even better way is to design a study that avoids it.
Of course, we do not really need to do that at all, because we already are quite sure that old people have a harder time crossing the street while talking on the phone. The study told us nothing, but the news reporters just did not get that. Unfortunately, the way they reported it actually could create a public health danger. Another way of saying "older people have more trouble", one that even the average health news reporter ought to be able to figure out, is "younger people have less trouble". I am sure that is not lost on younger readers. But 70-year-olds are seldom found talking on the phone while crossing the street, while 20-year-olds do it all the time, and thus are at much higher total risk. But if they read the headlines that the authors of this study created (via use of their press release and the credulous health press) young people will think they have been told they are good to go. Look out at the crosswalk!, indeed.
This reminds me of another study that told us nothing despite having a clearly correct conclusion, and that is a rather telling indictment of the quality of epidemiology (not just pickin' on psych today). Some researchers in public health at University of Alberta did a study which they claimed showed that allowing body checking in youth hockey increases injuries. It seems that a rule change moved 11-year-olds in youth league hockey from playing with the 10-year-olds in a league that did not allow checking to the league with 12-year-olds that did allow checking. So the researchers compared the injury rate from the year before the change to the year after and, sure enough, injuries increased. They got so breathlessly excited about the fact that they had a "natural experiment" that they seem to have overlooked the fact that it was a very badly designed experiment, another one with perfect (and thus non-controllable) confounding.
Before reading on, see if you can see the problem and prove yourself smarter than a University of Alberta public health faculty member (a lower standard than you can imagine).
Got it? If you said "wait a minute, the 11-year-olds went from being the bigger kids, surrounded by smaller and thus less damaging bodies, to being the smaller kids among a group of larger and stronger bodies," you win. There was perfect collinearity between becoming exposed to checking and becoming exposed to older opponents. It is safe to say, without any serious doubt, that each of those increases injuries. But when we are only measuring their combined effect, we learn very little. One thing that we do not learn, despite the claim of the authors, is the effect of exposure to checking alone.
But I have not even gotten to the worst part yet. One of the authors presented their results and conclusions at a departmental seminar when I was there (back when it was much more difficult to be smarter than a University of Alberta public health faculty member ;-), and I pointed out this fatal flaw that they had not even recognized. What did they do? They put in a few-word caveat in the discussion that it was maybe possible that there might be a wee confounding problem because of this, though that does not really matter; they did not change their interpretation of their results or their conclusions at all. And a journal published it anyway and, what is worse, the editors and reviewers no longer had the excuse that they might not have figured out the fatal problem because it was mentioned (though downplayed) in the paper.
There is always a bright side, though. It is fun to have a perfect teaching example of horribly bad epidemiology that you can use in classes at the same institution that produced the article. Well, fun for the teacher and students anyway.
This week a study was reported – as naively as can be imagined – that purported to show that old people suffer much more impairment when trying to multitask, in particular to talk on the phone while crossing the street, as compared to college kids. This was based on some simulator studies and a wonderful example of uncontrolled confounding. As an aside, I am greatly amused that the reports all specified that the tasks were "talking on a cell phone while crossing the street" as if (a) a cell phone is some novel technology that needs to be specified, (b) that "cell" is the standard phrasing (elsewhere in the anglophone world it is a "mobile"), and most important (c) there is any other kind of phone you can talk on while crossing the street. (Look out for that cord running across the crosswalk up ahead!)
As is the case with so many psychology studies, the main conclusion is almost certainly correct, the problem being that it does not really follow from the study. The biggest problem is something that epidemiologists are quite familiar with, but that psychologists may not be, the age-period-cohort issue. That is, if we study current 70-year-olds (a particular age), right now (a particular period), then you inevitably have people who are entirely from one cohort (born about 1941). If it is conceivable that an outcome is caused by either age (older people have trouble multitasking compared to people with young intensely-alert brains) or cohort (people born in 1941 only learned to walk while talking on the phone at age fifty-something, if ever, while current college students have been walking and phoning since childhood), or both, then the study makes it impossible to sort out the effects. Assuming we are interested in the effect of being old, and not the effect of having come of age without phones that can cross the street, we are stuck. Since there is perfect collinearity of the two, there is no variable than can be used to control for the effect of the second because there is nothing that can distinguish them. We would need a study that separated age from cohort (i.e., we would have to wait until we could study people of the same age from a relevantly different cohort, one that learned to cross streets when kids already had cell phones), which will require waiting about 50 years.
There were other related problems. The participants were not actually crossing a street, but were walking on a treadmill with display screens around them creating a virtual reality. In other words, they were playing a first-person video game (with their bodies as the controller), something that college kids are almost all comfortable with, but that those aged 60-80 are much less familiar with than they are with actually crossing the street. Thus, part of the measured effect was neither the age nor the cohort effect associated with being on the phone, but the cohort effect of not being used to video games. The researchers tried to make a big deal about how the study shows that walking is not as natural as we think it is; they are no doubt right that it is not natural when "walking outside" involves a treadmill and synchronized video screens in a dark room.
If we really wanted to solve some of the study problems sooner, we would not really have to wait 50 years or get old people run over. There are some clever tricks, like finding some young people who were not used to having a mobile phone, or comparing older people who have more and less experience with them (and even with video games). That introduces other more subtle confounding, but at least it gets some traction on the biggest confounding problem. Notice the subtext lesson here: One way to get rid of confounding is to control for covariates but an even better way is to design a study that avoids it.
Of course, we do not really need to do that at all, because we already are quite sure that old people have a harder time crossing the street while talking on the phone. The study told us nothing, but the news reporters just did not get that. Unfortunately, the way they reported it actually could create a public health danger. Another way of saying "older people have more trouble", one that even the average health news reporter ought to be able to figure out, is "younger people have less trouble". I am sure that is not lost on younger readers. But 70-year-olds are seldom found talking on the phone while crossing the street, while 20-year-olds do it all the time, and thus are at much higher total risk. But if they read the headlines that the authors of this study created (via use of their press release and the credulous health press) young people will think they have been told they are good to go. Look out at the crosswalk!, indeed.
This reminds me of another study that told us nothing despite having a clearly correct conclusion, and that is a rather telling indictment of the quality of epidemiology (not just pickin' on psych today). Some researchers in public health at University of Alberta did a study which they claimed showed that allowing body checking in youth hockey increases injuries. It seems that a rule change moved 11-year-olds in youth league hockey from playing with the 10-year-olds in a league that did not allow checking to the league with 12-year-olds that did allow checking. So the researchers compared the injury rate from the year before the change to the year after and, sure enough, injuries increased. They got so breathlessly excited about the fact that they had a "natural experiment" that they seem to have overlooked the fact that it was a very badly designed experiment, another one with perfect (and thus non-controllable) confounding.
Before reading on, see if you can see the problem and prove yourself smarter than a University of Alberta public health faculty member (a lower standard than you can imagine).
Got it? If you said "wait a minute, the 11-year-olds went from being the bigger kids, surrounded by smaller and thus less damaging bodies, to being the smaller kids among a group of larger and stronger bodies," you win. There was perfect collinearity between becoming exposed to checking and becoming exposed to older opponents. It is safe to say, without any serious doubt, that each of those increases injuries. But when we are only measuring their combined effect, we learn very little. One thing that we do not learn, despite the claim of the authors, is the effect of exposure to checking alone.
But I have not even gotten to the worst part yet. One of the authors presented their results and conclusions at a departmental seminar when I was there (back when it was much more difficult to be smarter than a University of Alberta public health faculty member ;-), and I pointed out this fatal flaw that they had not even recognized. What did they do? They put in a few-word caveat in the discussion that it was maybe possible that there might be a wee confounding problem because of this, though that does not really matter; they did not change their interpretation of their results or their conclusions at all. And a journal published it anyway and, what is worse, the editors and reviewers no longer had the excuse that they might not have figured out the fatal problem because it was mentioned (though downplayed) in the paper.
There is always a bright side, though. It is fun to have a perfect teaching example of horribly bad epidemiology that you can use in classes at the same institution that produced the article. Well, fun for the teacher and students anyway.
Nice post and a lovely font ;-)
ReplyDelete