American Ethnologist To Check Your Facts

Fact checking is all the craze these days. This American Life ran an episode-length retraction of Mike Daisey’s Apple story. Sites like Politifact regularly check politicians on their Truth-o-Meter. Magazines like the New Yorker are proud of their fact checkers, but academic journals rarely bother to check facts. Sure, academics have peer-review, but peer-review is not the same thing as fact checking. An expert on linguistic anthropology who does work in Latin America might be asked to review an article on indexicals in Chinese speech. As peer review goes, there is nothing wrong with this. Said expert will be able to do a good job of evaluating the argument and the relationship of the argument to the data presented in the paper. What they won’t be able to do is to check whether that data is accurate.

With the exception of a few big controversies, such as Margaret Mead or Jared Diamond (who isn’t actually an anthropologist), it is very rare for anyone to go and talk to an informant and ask them if they really said the words attributed to them by the anthropologist. For this reason the American Ethnologist’s recent announcement that they will fact check all articles is truly groundbreaking.

And it raises a number of questions: how will they pay for it? Fact checking anthropology articles is a lot more difficult that fact checking your ordinary piece of journalism. Especially for fieldwork conducted in some of the more remote corners of the earth. Of course, more and more people have internet access these days, and English skills are more widespread so maybe it won’t be as difficult as all that.

Even then, there is still the question of what constitutes a factual claim in anthropology. Will they just be confirming the most obvious statements of fact, or will they ask informants about the interpretation of their words in the text?

And what about privacy? While I trust American Ethnologist not to divulge names, there are serious risks related to divulging name and contact information to anyone, especially those living in countries that might monitor phone calls or email.

Still, I have seen enough questionable research in print that I applaud AE’s efforts to raise the bar beyond mere peer-review. But the details matter and I worry about how the AE fact-checkers will interpret their mandate. It will be interesting to hear reports from the first round of scholars who submit articles under the new regime. If you are one of them, please let us know and we will be happy to publish your account here.

UPDATE: Please note the date of this post.

12 thoughts on “American Ethnologist To Check Your Facts

  1. I think its a great step forward for anthropology. I was asked to blind fact-check an article about language, ideology, and political economy in Taiwan. Will be sure to report back here once I’ve completed my assignment.

    (Yes, I fell for it hook, line, and sinker. Nice one, K.)

  2. For things like fieldnotes, they could easily take an audit approach and have the author submit a copy of the notes. There is no way they can pull off an independent evaluation in which someone confirms sources, but think about it – newspapers don’t do this either.

    And there is often at least a relatively clear line between claims that present themselves as facts and claims that have an explicit interpretive component.

    Maybe the biggest problem is that in your average AE paper there just aren’t too many facts to check ;-)

  3. This is a joke, but it shouldn’t be. It wouldn’t be too hard to fact check journal articles. Since most anthropologists bring recording equipment into the field most quotes could be easily verified, as could witnessing certain events. Costs could be mitigated by using university resources. If you ask me, this is long overdue.

  4. Mary,

    Judging from my email inbox, you are not the only Savage Minds reader who is unaware of certain customs commonly associated with the date of this post. Or with this particular video. I hope these two (wikipedia) links help clear up any confusion you might have.

  5. Scott, it actually is a joke. One may imagine a future in which every anthropologist wears one of those “life camera” things that records everything he or she sees. But a lot of ethnographic insights pop up in times and places where recording is inappropriate, e.g., overhearing political rants or other bits of gossip that will get the speaker in trouble or, less dramatically, when one is privy to conversations where it would simply not be acceptable to record without seeking permission and seeking permission would interfere with the flow of what was happening. I think of sitting in my Daoist master’s storefront temple in Taiwan as clients wandered in to seek his advice. My making every client stop to agree and sign a consent form….Neither polite or likely to win the necessary cooperation. And then, of course, there are the after work drinking sessions in which I participated in Japan, not to mention hanging out in studios where TV commercials were being shot or edited. Once again, “May I record this and would you please sign this….” Totally unrealistic. Not to mention the pain of the fact-checker trying to make sense of what was going on without an omniscient view of the scene.

    The point of peer review is, ideally, that people with similar interests and experience, who are likely to be able to call foul or at least raise questions, are responsible for checking the content of what they read. There are no other fact checkers who will do any better.

  6. Dear John,
    I have a somewhat unique perspective on the issue. I was an anthropologist for many years before making my move over into journalism where now almost all of my work is fact checked. Fact checking is much more aggressive than peer review. The idea is that every factual statement that a writer makes should be challenged and substantiated. Fact checkers understand that not every little piece of data can be substantiated, but there needs to be a good faith effort from everyone involved. This doesn’t mean carrying a recording device at all times, but it does mean letting someone look at your notes to be sure that when you quote someone you have done so accurately. If you describe a major event then that event should have actually happened. It would be asking Geertz if, indeed, he DID go to a Balinese cockfight that perfectly illustrated his ethnographic idea, and not just taking his word for it.

    Every anthropologist that I know takes notes, and most record interviews. Consent forms leave a paper trail for contacts, and although these papers exist, I have never heard of anyone ever looking at them again. When it is possible fact checkers will contact sources to verify what they said. While it is a hassle to find some people (and language barriers and hard to find sources can make this difficult and sometimes impossible) I have found that it greatly increases the reliability of my stories. Often my first drafts are unintentionally riddled with errors–I misheard someone, or even worse, I heard what I wanted to hear, not what was actually said. A fact checker can challenge that and give the source the opportunity to respond so that what actually gets printed is as close to the truth as possible.

    The danger as I see it, is that, at the moment anthropologists have no standards for collecting their material. Ethnographic methods are not generally rigorously taught in graduate school. Methods classes often revolve around grant applications and a half-hearted statistics course. People are expected to learn in the field. Often when they are alone and without support.

    When magazines adopted fact checking standards journalists were not happy. It makes things more time consuming, and potentially opens the door to professional embarrassment. But what we all learn is that everyone makes mistakes while collecting information in the field. And fact checking adds fidelity.

    It will also catch anthropologists who simply invent “ethnographic details” out of thin air. If journalists at the New York Times, This American Life, The American Prospect, Wired and many other major publications have historically simply made up stories, then what basis do we have to think that social scientists wouldn’t do the same thing? The journalists reported to millions of people and had a much better chance of being exposed for their lies, and yet many of them get away with it for years. Is there any reason to think that social scientists are immune from the same human failings?

  7. Scott, you make some good points, but still, I believe, miss a fundamental difference between journalists and anthropologists. The information in news stories is, overwhelmingly, either public record, authorized, or both. The information that ethnographers gather on the fly while participating in the lives they study is often neither. A fact checker can examine a claim like “Government statistics project X” or “Celebrity Y said Z,” and there are independent sources in which the fact checker can confirm the veracity of the claim.

    Consider, however, a conversation I had with my Daoist master in Taiwan. In it, he told me something that was very important to my thinking. He observed that when Daoists claim magical powers, they justify their claims in one of two ways. They claim an original revelation, a vision from a god, or point to texts and their position in a genealogy that justifies their possession of them. The texts are presented as containing the secrets revealed to the founder of the lineage during an original revelation, a vision from a god.

    In neither case is there a way for a fact checker to confirm the veracity of the original revelation, unless, of course, they have a way to speak directly to the god. The same is true of the field note in which I recorded these comments. Even if I would make the field note public, how is the note’s content to be fact checked? It’s my note. I could have just made it up. Or, given the state of my Taiwanese at the time the conversation occurred, I might have mistaken what was said or omitted vital information that would radically alter the interpretation. But how is a fact checker to know? There is no independent source of information. There is only that fading note, my Daoist master is dead, and all we have besides the note itself are the memories of a man in his mid-sixties of a conversation that took place four decades ago.

    Where, then, does the credibility of ethnographic accounts come from? As Andrew Abbott observes in Methods of Discovery, it is often small details that persuade us that the ethnographer was, in fact, an eye-witness to what he or she describes. He notes, for example, Evan-Pritchard’s remark that, while living among the Azande, he used the poison oracle to organize his own schedule and found it about as good a way to do so as any other method he’d tried. Such small details are, however, equally effective in fiction.

    At the end of the day, then, what turns out to be left are the classic tools of humanistic scholarship. My claims about what my Daoist master said can be compared to what others have written about Daoism and Daoist masters to see if they seem wildly different from what others have said. If so, they may be challenged and a scholarly debate begins. The matter may remain moot for a long, long time. Peer review assumes this kind of scholarly debate and that those debating the matter are themselves recognized authorities in the field, better able than the rest of us to detect claims that seem, to those who know a topic, too odd to be believable.

    This is not, however, fact checking in any straightforward journalistic sense, where established facts from independent sources directly confirm or disconfirm an author’s claims. And where one would find such sources when the only available source is the one that needs checking. Yes, how is that supposed to be done?

  8. Responding to the conversation, and not to the actual post (brilliant btw. Had me going WTF for about 5 minutes).

    While I agree that it is important to know that ethnographers are reporting the facts, it is hard for many of us who work in war zones, conflict areas, or other problematic field sites to recover information that anyone can check. I don’t think that journalists go through an IRB but academic researchers do. IRB protocols are strict about who has access to information, especially on dates, places, and events that can resolve in informants being identified or even the danger of potential identification.

    Very often, when working in areas of conflict and violence, we get information after a long time of building trust and assuring the person who is taking a big risk in sharing this information that only the generalized analysis will ever be published and the raw data will be kept confidential. How do we gain trust if we know that someone else will be checking over the data? How do I tell an informant, “look, I know you are putting your life on the line, but someone, I don’t know who, will be going through this interview to make sure that I actually talked to you.” It’s hard getting them to trust the ethnographer, and we are expecting them to trust the faceless bureaucracy behind the ethnographer? Problematic.

    I don’t know a way out of this issue; it is a good question but the solutions are as problematic as the issues.

  9. I find it interesting that so many people here think that the information gathered by social scientists is in some way more rarefied and sensitive than information gathered by journalists. For the record, I wrote a fairly well received book on organ trafficking (http://amzn.to/lARVUD) which was reported mostly from third world countries, depended on the trust of my informants, was researched in several different languages, occasionally used pseudonyms and was also fact checked. I’ve also worked in conflict zones and even occasionally been in areas people were shooting each other. In all cases, when I write up my research, I do so under the assumption that everything I write needs to be defensible to an independent third party researcher. I see not doing so as borderline unethical.

    I have exactly zero faith in the ability of IRB boards to effectively guarantee the quality of social science research in advance. For the most part at universities IRBs are comprised of a variety of professors, not all of whom are in the social sciences, and the generally try to anticipate problems, and usually the takeaway is that you need to have your sources sign consent forms. Call me cynical, but I don’t see how a consent form is going to keep people’s facts straight.

    Also, anthropologists should realize that fact checkers had a variety of tools at their disposal. They can call sources, listen to audio, look at photographs, review field notes, access research databases and consult with legal teams when necessary. They give their best effort, and realize that not every piece of information is essentially verifiable, but the idea is that if someone is making up a lot of data, or routinely misinterpreting information, they will be able to call attention to a pattern. They also respect confidentiality of informants, I have had several fact checkers contact people I work with under privacy arrangements. Indeed, many of my sources appreciate when I tell them that there will be an independent review of my work down the road. It actually adds credibility.

    What is missing from this discussion, however, is how fact checking really adds fidelity to research. If I know that I’m going to be held accountable for what I write, I make double the effort to be sure that I don’t make exaggerations and that my work can really stand up on its own two feet. When that safeguard is not there standards begin to slip. When a fact checker tells me that I have an error in my work, I don’t get angry at the fact checker, I thank them for catching a mistake, or I have a conversation with them about why what I wrote is in fact accurate and then rework the data point so that it won’t raise objections from other readers. It is an important safeguard, and is one of the reasons that I can feel confident when I present my work to the public.

Comments are closed.