An Interview With Reviewers 1, 2, and 3

This is the third post in a sequence called Strange Rumblings in the Meritocracy.

[What follows is an edited and condensed transcript of an interview I conducted with Reviewers 1, 2, and 3. NB: Reviewer 1 and 2 and I had been sitting around for two hours, waiting for Reviewer 3 to show up, when we decided, to hell with it, we’ll just start talking. Reviewer 3 eventually showed up.]

Daniel: I just wanted to thank both of you for taking the time to talk with me. I know graduate students and junior scholars will likely appreciate a peek behind the curtain of anonymous peer-review. For many people it’s their first excursion into the broader discipline beyond the networks of their home institution, professional colleagues, or academic peers. More prosaically, successfully navigating peer-review is the only way any of us will get jobs. I’m guessing, too, that some mid- and senior-level scholars who are not actively involved in journal editing might like hearing what their colleagues say.

I also want to apologize for Reviewer 3. I’ve been getting texts, I think they’re stuck in traffic, or there was a schedule conflict, or there was a sick pet, or a student crisis, or something. I’m not really sure. The message keeps changing. They say, though, that they’ll be here soon. So I guess we should get started, and make due.

I was wondering if we could start off with a basic question. When you review for an academic journal, what do you look for in an article?

Reviewer 1: Well, this sort of gets at a philosophy of reading–I basically try to say “yes” whenever I come across something so that I can evaluate it on its own merits. What does it say it’s going to do? Does it succeed on its own terms? Then I go and evaluate from there. I think there is a danger,  otherwise, in criticizing the author for not writing the piece you would have written.

Reviewer 2: That’s interesting. I actually start from a different point. I try to see where it fits into wherever disciplinary scholarship is headed. It’s very important to me that someone writing an academic article demonstrate a good accounting of where the literature is at the moment, and then, and only then, offer something new to build on.

R1: I see what you’re saying, though, don’t you think there is a problem there? How are you going to innovate or have some kind of breakthrough, if your first step is accountability to what people have written already?

R2: Well, that sort of sounds sensible, but look, if I am asked to review something, it’s because I’m an expert on it. I, my colleagues, my students–we ARE the subject matter, we are the discipline, so you have to acknowledge that work that we’ve done. If you’ve taken our work seriously, you’ll likely end up producing something along the lines that we’ve talked about.

R1: I’m not sure I’m with you, there is a lot of scholarship out there, and I’m leery of presuming that one strand of it has all the answers. I mean to take a small example, economic sociology and economic anthropology deal with very similar sorts of research question, yet often draw on different bodies of theoretical and ethnographic literature. If you widen your aperture even a little bit you’ll take in behavioral economists, who again, on the face of it answer similar sorts of research questions but almost have an entirely different epistemology. Your standard doesn’t seem to allow new work, or at least work on new lines to emerge.

R2: I get where you’re coming from, but this is why it’s called a discipline, and why we make claim to some sort of body of knowledge, we build on what comes before. Taking every single piece as a one-off doesn’t seem to allow us to get beyond one-off speculative studies.

D: This is actually a good place to ask my second question: What type of criticism do you usually offer?

R2: Nine times out of 10, writers are either not adequately current on the literature, or they’ve missed a stream of it that both adds nuance to what they claim and makes their conclusions more robust by comparative analysis. That other one out of 10 has to do with inferences and conclusions drawn from evidence, people either claim way too much, or less frequently not all that much.

R1: There really isn’t any typical criticism I give. I suppose I either buy the argument or I don’t. When I buy the argument I give a pretty idiosyncratic list of suggestions and things I’ve read that I find helpful. If I don’t like the article I often sort of shut down and don’t have much nice to say. I tend to suggest that pieces don’t really fit in the discipline strongly consider other venues.

R2: That sounds like terrible feedback–don’t you worry about consistency?

R1: Well, I don’t think exciting new ideas come from being consistent. I mean you can critique anything, so why not go with my gut? I’ve been doing this long enough that I know what good work looks like.

R2: But what happens if I give you a methodical string of literatures to consider, and you just say you don’t like it? What’s the writer supposed to do?

D: If I may, I confess this happens to me a lot. I tend to get a lot of Jekyll and Hyde reviewing, honestly, right down to tone. Some people like my voice, others want to exorcise it from my writing. It’s tough, too, because a lot of the subjective criticism (I like this, I don’t like this, this should change, this is or isn’t anthropology) is often offered without any sort of argument. I find I have to take a lot of stuff on faith or on authority, and don’t have much time to negotiate or talk things out. It often doesn’t feel like the free exchange of ideas that academia is occasionally billed as.

R1: I get the frustration, but so much of work we do is interpretative. If the interpretation doesn’t work, I just can’t help you.

R2: It’s almost like there should be some kind of empirical structure or process through which you could reliably generate new ideas…

R1: Look I wouldn’t have gotten tenure if I weren’t good at this stuff.

D: Honestly, questions about authority are a good transition to another question: what do you think about the quality of a lot of the work you see?

R2: Quality is a moving target, I often see the work as evolving, so I rarely reject it out of hand. Most stuff I see has the potential to be great.

R1: I don’t really see any pattern. Though, honestly a lot of the bad work is beyond saving.

D: I want to push you on this. You’re often reviewing work of people that have done multiple years of research in their area of expertise. Do you really, rarely see room or space for improvement?

R1: The beautiful thing about anonymous peer review is I don’t have to wear some kind of mask. I find in so much of American academia a reflective politeness replaces any sort of scholastic honesty. I mean, when is the last time you got useful feedback from presenting a conference paper? If nothing else, that’s what I’m offering.

R2: You must be a joy in department meetings…

D: So, and this is for both of you, what kind of tone do you think a reviewer should take?

R1: Like I said, I’m just honest.

D: So telling someone their work is hopeless, or has no place in the discipline, or is shallow and two dimensional, is useful?

R2: Oh of course that isn’t. Just because you’re anonymous doesn’t give you license to be an ass. R1’s comment about saying yes to a text has quite a bit to recommend it. You have to give people a fair amount of charity. Like I said, I, my colleagues, and my students are the experts. I try and bring people up to speed on what we’ve done. It’s not that any of this is inaccessible or impossible, it just takes work to get through. Just about anyone can get there the process is an open one.

R1: You keep talking about this process thing, I don’t know that interpretation of ethnographic data has any kind of “reliable” process. Where does a good idea come from? The master’s tools and all that.

R2: I’m not sure we’re reviewing the same work–everything I see in anthropology has some kind of empirical foundation, some sort of data or information gathered from out there that is used to make a logical argument. You get the evidence, you interpret it according to well known method and theory, you get a result, you either prove something or you don’t. I don’t get what’s magical about this.

D: This point of disagreement is actually a good time to ask my last question, what alternatives if any, do you see to anonymous peer review? Something like open review or what happens at the Open Anthropology Cooperative?

R1: You need the mask of anonymity to say what you really think. Otherwise it’s just banal platitudes and social niceties. Similarly, without the absolute protection of tenure there is not intellectual freedom.

R2: Look, I see this stuff as a more or less empirical process. I think there are probably lots of other ways to get to the bottom of plausible potential directions a bit of scholarship could go. It’d be interesting to see how reviewers’ reports compare in an open seminar of some sort to peer review. It’d be interesting to see if you really do get the same kind of feedback and criticism. I suspect you would.

D: Thanks very much to both of you–it’s such a bummer Reviewer 3 couldn’t make…

Reviewer 3: I am so sorry I’m late! I just have so much stuff to do this time of year! Grading, students, family, oh you just wouldn’t believe. I’m just overwhelmed and could only read the topics you sent out superficially. That said I think peer review is incredibly important, and as I’m sure others of you have noted is an important stepping stone in an academic career. It really helps you see what other people are doing, and is important service for your university and the discipline at large. I agree too that peer review is basically all that is keeping us from pseudo-scientific superstition and a new dark age. And once you start
publishing and getting your name out there, its just great. I mean when you get the best minds together collaborating on review there’s nothing you can’t do.

R1, R2, D: [blinks]

D: Wow. Well, thanks again for talking to me.

4 thoughts on “An Interview With Reviewers 1, 2, and 3

  1. Every article purports to be a contribution to knowledge. Aren’t the critical questions how big and for whom? What about using reviewers from other related disciplines, who serve, in effect, as external examiners, and function to separate the wheat of broader interest from the chaff confined to established disciplinary ruts?

  2. Ya, this is the tricky part of review–there can be fundamental differences in how reviewers approach anthropology, and while this is interesting in the philosophical sense, it can lead to comments that are not only at odds, but completely talking past one another. I get the idea that folks are materialists, idealists, Foucauldians, positivists etc etc. But there has to be a way to set some of this aside to an extent and evaluate papers on their own terms. I’ve had grant reviews where the SAME proposal had comments that ranged from “this is excellent, great methods and theory” to “I don’t even see how this qualifies as science.” The last one is pretty much a literal quote. These kinds of mixed messages (that’s putting it nicely), in my view, really undermine the whole point of the process.

    Thanks for this post, Dan. Sometimes satire is our only hope.

  3. To Ryan’s point, what are we to conclude when supposedly dispassionate reviewers produce utterly contradictory reviews?
    As a case in point, let me mention my own unhappy experience. In early 1998 I submitted a theoretical paper I quite liked to the AA: “Culture, Mind, and Physical Reality: An Anthropological Essay.” Admittedly a rather narrow topic, I nonetheless hoped it might hold some interest for the AA readership. Five months (!) later I received a “revise and resubmit” letter from the editor, together with copies of five “peer reviews.” Two reviews were enthusiastic and recommended publication (one referee dropped the farce of anonymity and identified himself: an eminent anthropologist whose work I greatly admire). His summary comment: “I strongly recommend publication as is” (his emphasis). The other positive reviewer commented: “this lively and lucid paper is worthy of the AA and the AA of it.” A third referee was middle-of-the-road, noting some strong points, some weak points, and recommending the standard scholarly solution of “revise and resubmit.” A fourth referee judged it “not AA material” because it is a “highly philosophical essay characterized . . . by a reliance on anecdote, metaphor, and thought experiment, rather than actual data” (No doubt about his research paradigm! Ah, those Actual Data!). A fifth referee really hated it: “Briefly, the paper is unsophisticated, out-of-date, and very poorly argued.” Ouch.
    The conclusion I drew from those polarized reviews is that it is ludicrous to pretend that anthropology possesses anything like a canon, an established body of knowledge to which individuals can only hope to add minor contributions here and there. Rather, there is no consensus of any importance in the field; anything that matters is a toss up, a free for all.
    With that in mind, I’ve suggested that the most interesting and potentially productive papers are those that receive wildly discrepant reviews. If all reviewers agree that, yes, this paper represents a solid contribution to the field, that only means the paper holds nothing of interest, nothing challenging, nothing worth debating. Reject it out of hand!
    What do we do about this situation, in which peer reviews (obviously built on nothing) hold such importance? As anthropologists I think the answer is to conduct an ethnography of peer review. After all, here is an important feature of the belief system of a tribe we all know, so let’s collect a bunch of actual peer reviews and study them.
    I know this cuts against the grain of well-socialized academics. The peer reviews we receive are meant to be a hidden shame; the last thing we’d want to do is expose them to the light of day. Well, hey, let’s expose them! See how they’re put together. Wouldn’t we expect to conduct just that sort of inquiry into the peculiar beliefs and practices we encounter “in the field”?

  4. What this reflects is the fragmentation of the discipline following the collapse of the grand narratives in the 1980s, in which the process eloquently described by Andrew Abbott in Chaos of Disciplines, accelerated, leaving anthropology not only without clear paradigms but without even the broad consensus that anthropology is a discipline in which humanists can work alongside scientists, and humanists and scientists can learn from each other and together contribute to deeper as well as more rigorous knowledge of humanity. Which to my mind makes attempts to reintegrate the discipline like the one just published by Augustin Fuentes and Polly Weissner [http://www.journals.uchicago.edu/doi/full/10.1086/685694] particularly valuable contributions to knowledge.

Comments are closed.