Unhealthful News 44 - Tree climbing octopus eats reporters' brains

This week my Sunday post (on the topic of how to figure out who to trust) practically wrote itself, or other people wrote it for me.  Some of you might have figured I would address the question I left you with yesterday -- how can you figure out you trust me? -- but I am afraid I cannot just declare an answer to that (it would get rather circular, wouldn't it?).

Over the last week, several news outlets picked up on a random press release about a talk given by Donald Leu, a researcher from the University of Connecticut, who tricked a couple of groups of seventh graders in Connecticut and South Carolina into believing the elaborate internet joke about the tree octopus that lives high in the temperate rain forests of the Pacific Northwest (it is sometimes described as a hoax rather than a joke, but the details make it difficult to imagine that it was intended to fool anyone – e.g., its natural predators include the sasquatch, and we all know that they cannot climb trees; oh, and if you have not seen it, check out that link – it is really cute).  This was reported by some local newspapers, but also the Daily Mail,  and it had minor mentions at Reuters and NBC, and was played up quite a lot by CNN who was still flogging it this morning even after it was apparent that it was not the news story that it was claimed to be.  Basically the research found that almost every child in the study population was convinced when presented with information that the tree octopus was real.  This was spun by the researchers as showing that "digital natives" (those young enough that online sources of information have existed throughout their lives) were not actually very good at sorting out truth from fiction on the internet, and even that the internet was making them gullible. 

The news reports had various simple errors.  Some attributed the tree octopus joke to the researchers, though it is actually one of the classic web jokes, dating back to 1998.  Some said the research was new, though it is actually five years old (forever in internet time, and so not very informative anymore) and the only thing that was new was someone giving a publicized talk about it.  But the real problem, as a few other commentators pointed out, is that the reporters were more credulous than the kids supposedly were.

 Richard Chirgwin writing at the techie chat and networking site, The Register, pointed out weaknesses in the research that were obvious even from the press release  (e.g., small sample size, though the problem was actually worse than what he characterized it as, studying a few dozen kids; it was effectively only a few observations, since there is overwhelming evidence that a group like a classroom full of kids will likely reach almost unanimous opinion due to group dynamics).  He then went on to suggest a few questions that the reporters might have asked before falling for this:

“Was this the only task set for the children?” – this is important because if other tasks were included, they could impact the research. The children may have been put in a trusting frame of mind by other tasks; or they might have been asked to follow up the “Tree Octopus” information at the end of their attention span.

“Did anything about the research protocols, or the interactions between researchers and children, influence the results?” – For example, did the “Tree Octopus” information come after the researcher had been accepted as a trusted source by the children?

“Could any other factors affect the childrens’ research skills?” – For example, have they been taught well? What’s their educational and/or socio-economic background?

“How do you create a control to differentiate between the research skills of different student cohorts?” – Without a control, the research fails to tell us whether the problem is particular to research conducted using the Internet.

Some of those questions were partially answered when Brett Michael Dykes at Yahoo News (yes, Yahoo News wins hands down as the best news source about this story) posed a few questions to the issuer of the press release and got a partial response, that the study subjects were from the best readers groups at schools with "economically challenged" students.  Speaking as someone who fully fit that description in seventh grade I can tell you that getting placed in the skilled group in that environment typically means you are skilled at being able to take in information and provide slightly processed, but absolutely non-critical responses to it (it was years before I was a critical thinker and well into adulthood before I was confident of my critical thinking in the face of someone in authority insisting otherwise).  So when someone from the university comes in and teaches a guest lesson on some wonderful creature from the Pacific Northwest, a place about as foreign to these students as Mars, why would we expect them to doubt it?  Especially if, as the above quoted questions imply, there are many ways that the researcher might have actively tried to convince them of his credibility.  Tricking twelve-year-olds is not exactly difficult.  That is why we do not let them enter into binding contracts.

Professor PZ Myer, in his blog about this, wrote:

I actually use the Pacific Northwest Tree Octopus every semester, in the first lecture of our introductory biology course! After giving an overview of the scientific method and how to ask good scientific questions, I use it as an example: I show them the page, read a few excerpts, and ask them what they think…and always the majority of students are skeptical. The few who will grant it tentative plausibility always follow up with specific questions about the site and about where they can get additional information to confirm it.

It is interesting to observe that they are immediately skeptical, since this points out that the east-coast impoverished seventh-graders would probably have done better if the subject matter was not so foreign to them.  That observation is almost as interesting as the point he was presumably trying to make, that students who have had a few more years to learn scientific reasoning and realize that not everything they are told in class is right do just fine.

But where the reporters failed miserably is not pausing to think:  What would a real live web-literate decent student do if you put them in front of a computer and said "find out what you can about the tree octopus"?  Obviously they would do a search and check out a few websites, or they might go straight to wikipedia.  They would not just look at the one website that the researchers pointed them to (it is not clear if that was the original joke site or a derivative they created to better trick the kids).  "Digital natives" know how to do searches and follow hyperlinks; handing them a single packaged website is like handing them a single printed report.  If that is all they have to work with, what choice do they have but to believe what they were told?

The Wikipedia entry about the tree octopus (which calls it a hoax) went up in May 2006 (figuring out things like that is something that we critically thinking digital non-natives know how to do!).  This actually appears to post-date at least the earliest of the research that was being talked about, though maybe not all of it.  But I am willing to bet that there would have been a simple factual reporting that the octopus was a joke/hoax on the the second or third page that showed up in a search back then (ranked just below the original joke page which is one of the rare pages on an obscure topic that still outranks the Wikipedia entry).  But today Wikipedia would cut right through this hoax.  Like any encyclopedia, it is good at uncontroversial simple factual points.  It tends to be worse than the edited, single-authored, classic paper encyclopedia articles of yore for controversial, subtle, or (ironically) evolving areas of knowledge, but the fact that the tree octopus is a joke is not one of these.

So, if you stop and think about it (as you now have done, at my suggestion – but notice you do not have to take my word for anything – a key point about creating credibility!), there is just no way that this research shows what the authors claimed it showed, and what many reporters so naively believed.  Or as Dykes put it:

we think the overall lesson for the kids here is as follows: Don't believe everything you hear or read, on the Internet or elsewhere -- or, for that matter, in press releases.

If you stop and think about it and realize that the claims you read seem to defy everything you would expect, based on common knowledge and critical thinking, then one of two things are true:  Either the results are genuinely counter-intuitive, in which case the authors should be aware of that fact, should actively acknowledge it, and should explain why the claims are right in spite of what you think you know – at the very least they should give that explanation when prompted.  Or the writers are hoping that you will not notice how strange their claims are, burying the weakness of their analysis in confident statements and sciencey wording, in which case you really should not believe a word they say.

The octopus trick is really not much different from the sketch comedy show bits where someone goes out to the street asking people a question about a non-existent event, public official, or country and recording the answers of the ones who do not realize they are being tricked.  Of course, it might be non-tricked people engaging in the normal linguistic behavior of hearing a variation on what is actually said that makes more sense.  E.g., when someone is asked to comment on "the success of the people's revolution in North Pakistan" they hear "Egypt" and give an answer.  And someone puts it on television and laughs at them, like they did the poor seventh graders.  If someone really wants to research media consumers being credulous they should look at responses to arithmetic-defying economic claims on Fox News or to concerted and believed lies like those documented here by Citizens Against Government Encroachment in Canada.

C.A.G.E. documents on how anti-tobacco extremists pursue a concerted disinformation campaign to make it look like there is more popular support for their cause (creeping prohibition) than really exists, and that there is not great support for the opposition (those who believe in moderation and individual choice).  The reason that this struck me as relevant to the octopus story is because seventh graders are subject to a barrage of anti-drug, anti-tobacco, etc. propaganda to the point that they believe that all decent people are on the same side of the issue and they feel like they cannot have doubts.  It is the same one-sided game of visitors from the university showing them just a single website and implying it is accurate.  Only unlike the octopus case, most grownup inhabitants of the digital world still believe the manufactured impression of one-sidedness (at least about tobacco).  Why?  Well, if there were a concerted campaign to create the illusion that most people know the tree octopus exists then it would require enormous confidence to refuse to believe that too.

Doing research that involves tricking seventh graders who do not even have access to further information is just made-for-tv comedy.  If someone wants to do meaningful research about credulity in the internet age, I would suggest looking widespread campaigns to persuade people of a particular political view, and the factors that can help maintain critical thinking in spite of them.  Tricking someone with one web pages is just that, a trick.  Tricking someone by manipulating the predominant message across the media, including on the entire internet, is a way to undermine democracy.