Interesting bits about peer review and information quality

Randomly crossing my desk this morning:

A 16 minute video (h/t John Droz) with various interesting discussions about how the peer review process (both at the funding level and the publication level) serves as an echo chamber for the views supported by those who are in control, as well as some great observations about how to solve that problem.  The context is climate change, and there is a slight tendency toward piling on some of the dubious arguments that there is nothing to worry about, but there is only a bit of that.  Mostly it is an interesting study of scientific debate.

For those who do not want to watch, I will type out one highlight, from a talk by Michael Crichton, in which he recommends: 

In areas where policy is very important, you don't just give the research grant to one lab.  You give it to three.  And you make sure that two of them are strongly opposed to each other.  You tell them all that each lab will get to review the others' data; you tell them all that they will be published simultaneously; you tell them all that the publication will include the responses of the other labs to their work; and you tell them all that simultaneous to publication in a journal they will publish their data on the net.  If you did that, an enormous amount of baloney that now takes place would be over.

The substantive topic -- where is there legitimate disagreement about climate change, and is the disagreement being downplayed (by the dominant researchers) or overplayed (by news reporters and organizations funded by the likes of the Koch brothers) -- is fascinating for me, because I know enough to be able to make some substantive judgments, but not nearly enough to be an expert and decide for myself.  So basically I am in a position similar to that of most of the people I try to educate about my areas of expertise.  It is a tough position to be in, but educational about the process of learning and knowing.

Of course, one of the simplest rules I come back to and recommend about learning and knowing is when someone says something you know is bullshit, try to avoid believing the rest of what they say.  Case in point are a couple of lines from a new (but apparently not dated) FDA report (pdf) about their "global strategy" (h/t John Connell).  They lead off their discussion of tobacco regulation with:

Unlike food and medical products, tobacco products, including cigarettes and smokeless tobacco, are inherently unsafe.

That is not much different from the usual boilerplate from an tobacco control report and, if you bother to actually think about it more than the authors did, patently absurd.  Medical products are almost all inherently unsafe, posing some risks when used optimally and more when mishandled.  That is why there are so many regulations.  So perhaps what they are trying to say is that tobacco is different because the other products have a net positive expected value, in terms of health.  But, of course, that is not always true either, since there are constantly discoveries that particular medical products are causing more harm than good.  So what they must mean is that they have net positive expected values, to the best of our current knowledge.

But some medical products do not increase health in any "objective" way like increasing longevity, but merely reduce pain or otherwise improve happiness, or compensate for mental health issues -- possibly at some minor cost to physical health.  But smoke-free tobacco products fit exactly that description for many people.  So either some tobacco products have net positive expected value, like the meds, or some medical products do not.  Hmm.  Correcting their statement now requires some further nuances about either the average cost (averaging in smoking with smoke-free products) or a ridiculously gerrymandered definition of "unsafe".

Something is also needed to get the bit about food right.  We know that some food products, like Coke and bacon, have net negative impacts on health.  Though the manufacturers and people who make the silly claim that there is no bad food point out that this is not true if the alternative is nothing at all (that is really what they basically claim -- e.g., Coke is good for you because it provides need hydration, and because sugar is a nutrient).  If the alternative is other food and drink, those products are clearly physically unhealthy.  So the closest true thing to what FDA said seems to be: taken as an entire unsegmented category, food and medical products have net positive effect on health, while tobacco (as it now exists) has net negative physical health effect.

The next line takes their errors in another direction:

Their use takes a deep toll on U.S. public health and economy, resulting in around 443,000 deaths and $193 billion in medical costs and lost productivity each year.

I will set aside the well-known points -- the main criticism of FDA policy, that basically all that cost is attributable to the smoking of menthol and unflavored cigarettes, not "tobacco products" as a whole, but FDA's has focused primarily on every other type of product, and the scientific weakness of the numbers themselves -- and just look at the concept of economic costs.  The FDA document is current; we are in a macroeconomic depression, with huge overhanging unemployment.  So the lost productivity from loss of labor is... zero (more or less).  We have surplus labor in all sectors in our present state, so even if the calculation of lost productivity had been correct when it was done, it would certainly not apply now.

And then there is the little problem of counting lost productivity, but not reduced consumption.  People dying early is clearly bad, but if you are going to try to also count their lost productivity, you need to count their reduced consumption.  Otherwise, taking it back to the medical and food categories, they are using the same logic that says "medical products kill half a million Americans every year" (I made up the number, but it is probably in the right range) or "food products kill 2.5 million Americans every year" (which is to say, every single person who dies could have lived a bit longer had they eaten differently).  In both cases, they obviously extend life more than they cause early death, but if you do not count the savings, the costs are really impressive.

Of course, these technical observations are of little interest to pamphleteers, who are concerned only with impact and not with scientific detail.  These would only matter to someone claiming to be a scientist or scientific organization.

Oh, wait.

This kind of points out the fatal flaw in Crichton's utopian idea:  Most of the time when the science is politicized, those who control the grants are even more politicized than the researchers they are funding.  When is the last time you heard of a government grant going to someone who would disagree with those two claims by FDA, even though the claims are both clearly wrong?