One of the great frustrations I have, trying to exist in the borderlands between science and science-related policy and popular discourse about scientific topics, is the naive view of what it means for a paper having "peer reviewed journal article". The failure to understand how little peer review means is shared by some lay people, many policy makers, and even a lot of people who write scientific papers or have titles in that category (being a scientist is a way of thinking, not a credential). I have a collection of motivating observations, details, and other bits which I will cover in some follow-up posts. Which to facilitate reading I will come back and catalog in the following space with updates:
[
NOTHING HERE YET
]
But for today, for #UnhealthfulNews 200, I wanted to skip the news and explore an extended metaphor that might help clarify what I am trying to explain.
My father and his contemporaries, who were on the road in middle America a lot, recall the appearance of hundreds of McDonalds restaurants along the highways as a great thing. Before McDonalds, there were diners and burger joints along the road, of course. But you always had to worry about quality, including in particular whether the food might make you sick. It was not necessarily quick or easy to find someplace to eat, the menu had to be navigated and second-guessed, and there was always the possibility that "your type" was not welcome in that particular establishment. Readers who have never experienced such a world have probably heard related rules-of-thumb, like eat where the truckers are stopping because they have better data on those issues than you do (though I don't think the word "data" appears in the cliche version of the saying).
Readers without such travel experience probably wonder what is so great about homogenized roadside fast food places, especially those who would never eat at McDonalds, or perhaps even despise the very existence if it and its ilk. Figuring out the local dining is part of the fun of travel, isn't it.? Indeed, it sometimes is, but sometimes you are just trying to make time on the highway. Fast food is as beneficial for that as being able to drive at 60 miles per hour. From that perspective, the invention of McDonalds (and a few of its contemporary early chains) was a technological breakthrough at the level of having a GPS and search engine on your mobile phone.
 |
| Pier Review |
Convenience: McDonalds has roadside billboards directing you to the next location and easily spotted buildings. Journals have indexes. Whatever someone might claim about really liking the fries (believing in the superiority of the content), this is undoubtedly the biggest draw for a non-local (non-expert) venturing into unknown territory and needing to rapidly get nourished (very roughly educated about a scientific topic).
Safety: McDonalds has amazing quality control; the chance of getting at least a little bit sick from a random roadside eatery might be as high as 10%, while at McDonalds (assuming the food is something you can eat) it is about 0% . The journal review process lowers the chance that what you are reading suffers from utterly incompetent analysis or gross bias. Unfortunately, in public health it does so only by something like the same 10 percentage points. In other areas of health science it does rather better, but certainly does not eliminate those problems, though few non-experts probably understand the limits of this benefit.
Familiarity: The same quality control means that McDonalds food tastes the same everywhere in the country, and the fare is the same too so for those who do not want to read a menu no thinking is required. Journal articles almost always follow the same familiar format, and the abstract tells unsophisticated readers what they are supposed to think so for those who do not want to read the study, no thinking is required.
Some of the preceding praise is, of course, not exactly high praise. Still, I would not want to imply there is no value in either institution. (Though, I will note for the record that even though I liked it as a child, I have not eaten at McDonalds in 25 years; exception: I just had to try it in India where the beef is replaced with veggie burgers.) For centuries, what passed for scientific communication in health consisted mostly of the half-assed opinions of physicians, reporting from the basis of their "professional experience". The current journal system is obviously as much as improvement on that as McDonalds is over a rat-infested diner that lacks a working sink in the employee restroom.
But then there is the problem of the downsides that exist, even beyond recognizing that the benefits are more modest than many people think:
Homogenization: Even as standardization and strict recipes create some benefits, they also eliminate the variety of what can be eaten (analyzed) and force everything into the same mold -- perhaps appropriate for fast food, but clearly not good for making truly great food (studies that require something other than the standard recipe). So, for example, a paper that uses atypical methodology that is clearly better than the standard practice is very likely to get rejected. It might seem like this is not really about peer review as practiced, and it is not entirely, but that is part of the problem. Assessing what is better requires a lot more effort and specific skill; but the current review process creates a McDonalds-like assembly line, where it is assumed that any vaguely skilled person can be made into a cook (reviewer) and the incentives are for only speed and adhering to standard practices. Indeed, because so many papers need to be peer reviewed and the qualification for being a reviewer is basically "submitted a paper vaguely related to the topic", the average quality of the people who review papers in health science is necessarily the about the same as the average quality of those who write them. They are no more capable of judging what is better (or even good) than McDonalds cooks are of becoming chefs; with a few exceptions, they are both technicians, with no plan, and perhaps no potential, to become chefs (serious scientific thinkers). Assembly lines fill stomachs and journals with amazing efficiency, but do nothing to create either cuisine or good science.
Frozen in time: Science thrives on creativity, as does dining (as opposed to mere feeding). The tyranny of the health science journal publishing system has stifled advances in thinking and methodology for longer than I have been in the field. If you were to Copy a research report on a topic that is still of interest from moderately decent journal from 1980 and submit it today, and chances are none of the reviews would suggest the methods are archaic. This definitely differs from many sciences, of course, where the editor would immediately reject, perhaps asking "have you been in a coma for the last few decades?" Yes, the McDonalds menu has changed, with much fanfare, by about one item a year since the 1970s, and health science journal articles have seen a similar level of evolution, has but science should be a bit more dynamic.
Unhealthfulness exclusivity: Some would argue that eating any McDonalds at all is actively harmful for your health. That does not fit my metaphor -- there is obviously no harm in consuming journal articles, and indeed there would be terrible harm from ignoring them all (as opposed to never eating at McDonalds, which would be harmless). So we shall acknowledge the departure and set it aside, and instead continue the metaphor with a "Supersize Me" diet of just McDonalds. Eliminating from your diet anything that is not served at McDonalds and its ilk would obviously be a terrible idea. We would probably agree that anyone who insisted on doing that was seriously misguided. We should realize that the same applies for anyone who claims to be expert on a public health topic, but whose scientific knowledge is limited to the information that appears in journals.
Just like a chain fast food restaurant is a sensible place to eat when you are just passing through, a quick review of just what appears in the journals is not a bad strategy if you only have an hour or perhaps half a day to learn something about a topic. But if you are in town two days, such an approach is kind of sad. Claiming to be an expert but only being familiar with the journal literature is kind of like writing a local restaurant guide based on eating at only the restaurants with a drive-through. Finally, believing something to be true just because it is in a journal is kind of like believing that a McDonalds cheeseburger offers perfect nutrition because it contains all four food groups (if you are under 30, you might have to look up that reference and also search "reagan ketchup").
I hereby claim my free happy meal for prompting this post by email (I think).
ReplyDeleteIt's an interesting metaphor, but doesn't really address the (I think more pertinent) question on information architecture:'we know peer review and journal publication has numerous flaws, but what alternative system is better?'
Open post-publication review is interesting (but then when it's been tried the people you'd like to contribute to review turn out not to bother).
Editorial publication discretion and no peer review (à la the journal 'medical hypothesis') to encourage non-mainstream analysis tend to be a breeding ground for the weird. Like this paper (http://bit.ly/AoAu5N) where the authors in all apparent seriousness consider whether 'mongoloid' is an appropriate name for people with Down's syndrome because they enjoy sitting cross-legged, taking part in handicrafts,and share some facial features with oriental populations.
People shouldn't defer to knowledge coming from peer-review journals on authority alone and should be able to be critical consumers of research from all kinds our sources (journals, blogs, books, 'think tank' reports etc). But peer-reviewed sci journals are a decent starting point.
Also, in some politically contentious areas (e.g. climate science), I'd suspect most everyday consumers of information don't get it from the McD's of science journals themselves but second-hand through bloggers, journalists, authors and political commentators (many of whom won't have read the original science themselves and are reporting it second-hand). I think in many areas, it would be *great* if the traditional scientific publication process had the market penetration and consumer appeal of McD's.
Rory,
DeleteI have to admit I am not sure what caused the idea to bubble up -- it had been forming for a while. So I am happy to believe your suggestion that you sparked it.
I will address the question of what works better in a few subsequent posts that I already have in mind or partially written. And, it is worth repeating that the McDonalds metaphor is meant to note both that the system has great value and that it has severe limitations.
Also, like any metaphor, you can find points where it is a bit strained. I agree that the average person does not pay much attention to whether the rumors they here came from a peer reviewed source. However, there is a good chance that they did, because the next level up the information distribution order -- the press and activists -- tend to overvalue the peer review system. So if we consider the portion of the population that does their own research at some level to be the audience, then the metaphor works better.
Indeed, it does not work so well when "regular" people try to access the literature themselves, treating it like fast food. As bad as the health press is, at least the half or so of reporters who have some clue understand that a single journal article is a tiny peephole on the science. People who do not even understand that much, and whose science education consists of the misguided message that science is mostly about a set of indisputable facts (a topic I have written about), have a habit of treating each abstract they read as if it were Philosophiæ Naturalis Principia Mathematica.
Editorial discretion suffers from a set of problems that, like those of peer review, traces back to the limits of the abilities (skill and available effort) of those in power. I am quite confident that the best way to decide what should get some kind of high-honor publication status in a few specific topics is for me to unilaterally decide. But it does not require too much departure from those particular thoughts before someone else would be better than I, but might not be available to help.
I'll be interested to read what your ideas of a better solution are when they come. I struggle to see an alternative that doesn't run into one of the same core problems you identify (lack of sufficient skills/competency amongst the individuals who ultimately arbiter publication and dissemination) in ways just as bad, if not worse, than the current system.
DeleteThere are some interesting hybrid peer review/moderated wiki structures suggested out there though, which seem to have some promise, at least in certain fields.