Three weeks ago, in the first of my Sunday "how to figure out who to believe" posts, I started an examination of how Chris Snowdon is much more credible in his criticism of "The Spirit Level" than the authors of that book are. What I emphasized was that Snowdon criticized specific points that the original authors made, challenging their methods and analytic choices, and the original authors in their replies did not address his (and other critics') points. Also the critics offered further rebuttals to the indirect responses by the original authors. Just because someone argues for their criticism does not make it right, of course, but when those who are criticized never respond to credible substantive criticism, it tends to suggest that the criticism is right. This is especially true when the original authors continue to argue their points, as they did with The Spirit Level (if they just become silent, perhaps feeling that their original words speak for themselves, then it becomes a bit trickier to interpret their failure to respond).
Last week I was giving testimony in the matter reported here. (Just for the record, I would not have summarized my testimony the way that reporter did. There is nothing specifically wrong with it, it is just not exactly what I would want to say.) As part of my testimony, I wrote a critique of a government document, and was cross examined about that by a government attorney. The questioning brought to my mind an interesting point of clarification about the principle I talked about three weeks ago. (Indulge me with one more note for the record – sorry about these, but in legal cases it is a necessary hassle: This proceeding was open to the public and I believe there will be a transcript on the public record, so there is nothing secret about this. But just in case it might annoy someone if I did, I am not going to say anything of substance about the testimony, and will make up an example instead.)
One of the points I had made in my written opinion was, to paraphrase, that if you are going to legitimately review a contentious issue and justifiably declare that a particular possible conclusion is right, then you must present the best possible opposing case and rebut it. To make a legitimate argument, you cannot just ignore the information and arguments on the other side, present just one side of the issue, and then declare a conclusion. Instead, it is necessary to acknowledge the opposing claims and explain why they are wrong, are over-matched by your claims, or are otherwise are unconvincing.
To take a very simple example, if you want to claim that you are the tallest person in the room (assume that gets you some kind of prize, so you have some incentive to make the claim), you cannot just say, "I am six-foot-five-inches" (or whatever that is in centimeters – let's say a thousand – I am really lousy with centimeters) and "someone just said 'I'll bet you are the tallest person in the room'". Those tend to support the claim, but we are only hearing one side of the story. What if you are omitting facts like "but that guy over there looks like he is six-foot-seven", or counterarguments like "as soon as someone said I was tallest someone else said 'no he's not – look over there'"? If so, then you have not fully represented the situation. Because of this possible incomplete reporting problem, therefore, if you say nothing about the apparent heights of everyone else and nothing about whether anyone agreed with the claim about you, then a careful listener should become suspicious; you have not even told us who else might have something to say about your claim. Moreover, if that careful listener had already heard someone say the "no he's not – look over there" then there is clearly a problem. Failure to even acknowledge that there is opposing evidence does not prove you are wrong, of course. It might be that you are indeed tallest despite those claims to the contrary. But if you were confident that your claim stood up(!) to scrutiny and all opposing arguments, why did you just ignore the other information?
When I was being cross examined, I discovered that I may have not communicated my point as effectively as I might have done (or maybe it is possible that the attorney was just pretending to misinterpret it – that is one of the hazards of legal matters: unlike in science writing where you trust that most people are trying to understand you and if they do not it is because you communicated badly, in legal matters sometimes people are trying to look for a way to misunderstand you). I was asked what about that government report I found to be good, and when I responded that I was not quite sure how to answer the question, the attorney pointed out that I had said that we have an obligation to present the best points by the opposition and respond to them. I realize that what I had written might be interpreted as some kind of "you should always find something nice to say about someone" type of thing.
I could see why that would create some confusion because what I actually do, and what I am suggesting others do, is present the best arguments that are fielded in opposition to your claims and then explain why they are wrong. (This does not, of course, mean that you need to acknowledge everything that someone says that is right, just address the bits that are in contention.) This is something that people who are not very good scientists (e.g., in my experience, most people who fancy themselves to be epidemiologists) react to as if it were unacceptably combative. Quote someone else's claims and then pointedly argue that they are wrong? How rude! But only by citing the opposing claims can you (a) make it clear that you understand them but still disagree (as opposed to reaching a contrary conclusion simply because you do not get the point) and (b) present an efficient argument that they are wrong.
But wait, someone might say, you want to present the opposing arguments in order to explain why they are wrong? Does that mean you are you assuming that the opposing arguments are wrong? Absolutely not. I am not assuming they are wrong because I believe a particular conclusion; I reached that conclusion, in part, because I have considered the opposing arguments and concluded that they are wrong. If I had concluded without considering those arguments, then my conclusion would not be worth very much. But also, my conclusion would not be worth much if I could not explain why opposing arguments were wrong after having considered them.
In most legal proceedings, it is the duty of an expert witness to inform the primary audience (judge, jury, board, tribunal, etc.) of the truth to the best of your ability and understanding, rather than to be an advocate like an attorney, who often has a duty to do anything within the bounds of the law to advocate for a client's preferred position. This is not a difficult role to step into, because it is basically the same as the ethical duty to society of a scientist, educator, or journalist. So the same principles apply in each case. There is no duty to be nice to those who argue views opposite to what you believe (that is, frankly, hard to do while efficiently presenting the science, though scientists learn that harsh and blunt are part of the scientific process, they are not really non-nice). But there is a duty to give full credit to their points.
And therein lies what must be the best possible clue about who to believe in a matter where you are not a subject matter expert: The path of scientific righteousness passes through a complete and honest appraisal of the best of the other paths. Beware of the guide who tries to tell you that there are no other paths out there.
Update on a related point: Two Sundays ago I wrote about a New York Times article that urged readers to avoid WebMD and instead read the Mayo Clinic website. I pointed out some flaws in that reasoning. In the letters to the editor, a spokesman for WebMD reiterated my point that there is no reason to believe that the Mayo website is less biased. However, he tries to argue that we should be impressed that WebMD separates advertising from content, which is obviously a minimum standard, but not really a response to the points that generated the original criticism. He also points out that Mayo's website is also a commercial entity that accepts advertising, which says nothing about their credibility, unlike the pattern of disinformation that I cited. The spokesman quite reasonably responded to the spurious arguments about who we should trust that appeared in the original article; interestingly, though, he did not respond to the core question about whether we should trust WebMD.
Rather more disappointing, though, was the next letter (same link) by a librarian who suggested that readers should forgo both of those sources and go to Medline Plus instead because it is "without bias". Apparently her basis for claiming this is that it is produced by the U.S. government. I trust I do not have to address either of these absurdities.]
Last week I was giving testimony in the matter reported here. (Just for the record, I would not have summarized my testimony the way that reporter did. There is nothing specifically wrong with it, it is just not exactly what I would want to say.) As part of my testimony, I wrote a critique of a government document, and was cross examined about that by a government attorney. The questioning brought to my mind an interesting point of clarification about the principle I talked about three weeks ago. (Indulge me with one more note for the record – sorry about these, but in legal cases it is a necessary hassle: This proceeding was open to the public and I believe there will be a transcript on the public record, so there is nothing secret about this. But just in case it might annoy someone if I did, I am not going to say anything of substance about the testimony, and will make up an example instead.)
One of the points I had made in my written opinion was, to paraphrase, that if you are going to legitimately review a contentious issue and justifiably declare that a particular possible conclusion is right, then you must present the best possible opposing case and rebut it. To make a legitimate argument, you cannot just ignore the information and arguments on the other side, present just one side of the issue, and then declare a conclusion. Instead, it is necessary to acknowledge the opposing claims and explain why they are wrong, are over-matched by your claims, or are otherwise are unconvincing.
To take a very simple example, if you want to claim that you are the tallest person in the room (assume that gets you some kind of prize, so you have some incentive to make the claim), you cannot just say, "I am six-foot-five-inches" (or whatever that is in centimeters – let's say a thousand – I am really lousy with centimeters) and "someone just said 'I'll bet you are the tallest person in the room'". Those tend to support the claim, but we are only hearing one side of the story. What if you are omitting facts like "but that guy over there looks like he is six-foot-seven", or counterarguments like "as soon as someone said I was tallest someone else said 'no he's not – look over there'"? If so, then you have not fully represented the situation. Because of this possible incomplete reporting problem, therefore, if you say nothing about the apparent heights of everyone else and nothing about whether anyone agreed with the claim about you, then a careful listener should become suspicious; you have not even told us who else might have something to say about your claim. Moreover, if that careful listener had already heard someone say the "no he's not – look over there" then there is clearly a problem. Failure to even acknowledge that there is opposing evidence does not prove you are wrong, of course. It might be that you are indeed tallest despite those claims to the contrary. But if you were confident that your claim stood up(!) to scrutiny and all opposing arguments, why did you just ignore the other information?
When I was being cross examined, I discovered that I may have not communicated my point as effectively as I might have done (or maybe it is possible that the attorney was just pretending to misinterpret it – that is one of the hazards of legal matters: unlike in science writing where you trust that most people are trying to understand you and if they do not it is because you communicated badly, in legal matters sometimes people are trying to look for a way to misunderstand you). I was asked what about that government report I found to be good, and when I responded that I was not quite sure how to answer the question, the attorney pointed out that I had said that we have an obligation to present the best points by the opposition and respond to them. I realize that what I had written might be interpreted as some kind of "you should always find something nice to say about someone" type of thing.
I could see why that would create some confusion because what I actually do, and what I am suggesting others do, is present the best arguments that are fielded in opposition to your claims and then explain why they are wrong. (This does not, of course, mean that you need to acknowledge everything that someone says that is right, just address the bits that are in contention.) This is something that people who are not very good scientists (e.g., in my experience, most people who fancy themselves to be epidemiologists) react to as if it were unacceptably combative. Quote someone else's claims and then pointedly argue that they are wrong? How rude! But only by citing the opposing claims can you (a) make it clear that you understand them but still disagree (as opposed to reaching a contrary conclusion simply because you do not get the point) and (b) present an efficient argument that they are wrong.
But wait, someone might say, you want to present the opposing arguments in order to explain why they are wrong? Does that mean you are you assuming that the opposing arguments are wrong? Absolutely not. I am not assuming they are wrong because I believe a particular conclusion; I reached that conclusion, in part, because I have considered the opposing arguments and concluded that they are wrong. If I had concluded without considering those arguments, then my conclusion would not be worth very much. But also, my conclusion would not be worth much if I could not explain why opposing arguments were wrong after having considered them.
In most legal proceedings, it is the duty of an expert witness to inform the primary audience (judge, jury, board, tribunal, etc.) of the truth to the best of your ability and understanding, rather than to be an advocate like an attorney, who often has a duty to do anything within the bounds of the law to advocate for a client's preferred position. This is not a difficult role to step into, because it is basically the same as the ethical duty to society of a scientist, educator, or journalist. So the same principles apply in each case. There is no duty to be nice to those who argue views opposite to what you believe (that is, frankly, hard to do while efficiently presenting the science, though scientists learn that harsh and blunt are part of the scientific process, they are not really non-nice). But there is a duty to give full credit to their points.
And therein lies what must be the best possible clue about who to believe in a matter where you are not a subject matter expert: The path of scientific righteousness passes through a complete and honest appraisal of the best of the other paths. Beware of the guide who tries to tell you that there are no other paths out there.
Update on a related point: Two Sundays ago I wrote about a New York Times article that urged readers to avoid WebMD and instead read the Mayo Clinic website. I pointed out some flaws in that reasoning. In the letters to the editor, a spokesman for WebMD reiterated my point that there is no reason to believe that the Mayo website is less biased. However, he tries to argue that we should be impressed that WebMD separates advertising from content, which is obviously a minimum standard, but not really a response to the points that generated the original criticism. He also points out that Mayo's website is also a commercial entity that accepts advertising, which says nothing about their credibility, unlike the pattern of disinformation that I cited. The spokesman quite reasonably responded to the spurious arguments about who we should trust that appeared in the original article; interestingly, though, he did not respond to the core question about whether we should trust WebMD.
Rather more disappointing, though, was the next letter (same link) by a librarian who suggested that readers should forgo both of those sources and go to Medline Plus instead because it is "without bias". Apparently her basis for claiming this is that it is produced by the U.S. government. I trust I do not have to address either of these absurdities.]
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.