This is the third post in a sequence called Strange Rumblings in the Meritocracy.
[What follows is an edited and condensed transcript of an interview I conducted with Reviewers 1, 2, and 3. NB: Reviewer 1 and 2 and I had been sitting around for two hours, waiting for Reviewer 3 to show up, when we decided, to hell with it, we’ll just start talking. Reviewer 3 eventually showed up.]
Daniel: I just wanted to thank both of you for taking the time to talk with me. I know graduate students and junior scholars will likely appreciate a peek behind the curtain of anonymous peer-review. For many people it’s their first excursion into the broader discipline beyond the networks of their home institution, professional colleagues, or academic peers. More prosaically, successfully navigating peer-review is the only way any of us will get jobs. I’m guessing, too, that some mid- and senior-level scholars who are not actively involved in journal editing might like hearing what their colleagues say.
I also want to apologize for Reviewer 3. I’ve been getting texts, I think they’re stuck in traffic, or there was a schedule conflict, or there was a sick pet, or a student crisis, or something. I’m not really sure. The message keeps changing. They say, though, that they’ll be here soon. So I guess we should get started, and make due.
I was wondering if we could start off with a basic question. When you review for an academic journal, what do you look for in an article?
Reviewer 1: Well, this sort of gets at a philosophy of reading–I basically try to say “yes” whenever I come across something so that I can evaluate it on its own merits. What does it say it’s going to do? Does it succeed on its own terms? Then I go and evaluate from there. I think there is a danger, otherwise, in criticizing the author for not writing the piece you would have written.
Reviewer 2: That’s interesting. I actually start from a different point. I try to see where it fits into wherever disciplinary scholarship is headed. It’s very important to me that someone writing an academic article demonstrate a good accounting of where the literature is at the moment, and then, and only then, offer something new to build on.
R1: I see what you’re saying, though, don’t you think there is a problem there? How are you going to innovate or have some kind of breakthrough, if your first step is accountability to what people have written already?
R2: Well, that sort of sounds sensible, but look, if I am asked to review something, it’s because I’m an expert on it. I, my colleagues, my students–we ARE the subject matter, we are the discipline, so you have to acknowledge that work that we’ve done. If you’ve taken our work seriously, you’ll likely end up producing something along the lines that we’ve talked about.
R1: I’m not sure I’m with you, there is a lot of scholarship out there, and I’m leery of presuming that one strand of it has all the answers. I mean to take a small example, economic sociology and economic anthropology deal with very similar sorts of research question, yet often draw on different bodies of theoretical and ethnographic literature. If you widen your aperture even a little bit you’ll take in behavioral economists, who again, on the face of it answer similar sorts of research questions but almost have an entirely different epistemology. Your standard doesn’t seem to allow new work, or at least work on new lines to emerge.
R2: I get where you’re coming from, but this is why it’s called a discipline, and why we make claim to some sort of body of knowledge, we build on what comes before. Taking every single piece as a one-off doesn’t seem to allow us to get beyond one-off speculative studies.
D: This is actually a good place to ask my second question: What type of criticism do you usually offer?
R2: Nine times out of 10, writers are either not adequately current on the literature, or they’ve missed a stream of it that both adds nuance to what they claim and makes their conclusions more robust by comparative analysis. That other one out of 10 has to do with inferences and conclusions drawn from evidence, people either claim way too much, or less frequently not all that much.
R1: There really isn’t any typical criticism I give. I suppose I either buy the argument or I don’t. When I buy the argument I give a pretty idiosyncratic list of suggestions and things I’ve read that I find helpful. If I don’t like the article I often sort of shut down and don’t have much nice to say. I tend to suggest that pieces don’t really fit in the discipline strongly consider other venues.
R2: That sounds like terrible feedback–don’t you worry about consistency?
R1: Well, I don’t think exciting new ideas come from being consistent. I mean you can critique anything, so why not go with my gut? I’ve been doing this long enough that I know what good work looks like.
R2: But what happens if I give you a methodical string of literatures to consider, and you just say you don’t like it? What’s the writer supposed to do?
D: If I may, I confess this happens to me a lot. I tend to get a lot of Jekyll and Hyde reviewing, honestly, right down to tone. Some people like my voice, others want to exorcise it from my writing. It’s tough, too, because a lot of the subjective criticism (I like this, I don’t like this, this should change, this is or isn’t anthropology) is often offered without any sort of argument. I find I have to take a lot of stuff on faith or on authority, and don’t have much time to negotiate or talk things out. It often doesn’t feel like the free exchange of ideas that academia is occasionally billed as.
R1: I get the frustration, but so much of work we do is interpretative. If the interpretation doesn’t work, I just can’t help you.
R2: It’s almost like there should be some kind of empirical structure or process through which you could reliably generate new ideas…
R1: Look I wouldn’t have gotten tenure if I weren’t good at this stuff.
D: Honestly, questions about authority are a good transition to another question: what do you think about the quality of a lot of the work you see?
R2: Quality is a moving target, I often see the work as evolving, so I rarely reject it out of hand. Most stuff I see has the potential to be great.
R1: I don’t really see any pattern. Though, honestly a lot of the bad work is beyond saving.
D: I want to push you on this. You’re often reviewing work of people that have done multiple years of research in their area of expertise. Do you really, rarely see room or space for improvement?
R1: The beautiful thing about anonymous peer review is I don’t have to wear some kind of mask. I find in so much of American academia a reflective politeness replaces any sort of scholastic honesty. I mean, when is the last time you got useful feedback from presenting a conference paper? If nothing else, that’s what I’m offering.
R2: You must be a joy in department meetings…
D: So, and this is for both of you, what kind of tone do you think a reviewer should take?
R1: Like I said, I’m just honest.
D: So telling someone their work is hopeless, or has no place in the discipline, or is shallow and two dimensional, is useful?
R2: Oh of course that isn’t. Just because you’re anonymous doesn’t give you license to be an ass. R1’s comment about saying yes to a text has quite a bit to recommend it. You have to give people a fair amount of charity. Like I said, I, my colleagues, and my students are the experts. I try and bring people up to speed on what we’ve done. It’s not that any of this is inaccessible or impossible, it just takes work to get through. Just about anyone can get there the process is an open one.
R1: You keep talking about this process thing, I don’t know that interpretation of ethnographic data has any kind of “reliable” process. Where does a good idea come from? The master’s tools and all that.
R2: I’m not sure we’re reviewing the same work–everything I see in anthropology has some kind of empirical foundation, some sort of data or information gathered from out there that is used to make a logical argument. You get the evidence, you interpret it according to well known method and theory, you get a result, you either prove something or you don’t. I don’t get what’s magical about this.
D: This point of disagreement is actually a good time to ask my last question, what alternatives if any, do you see to anonymous peer review? Something like open review or what happens at the Open Anthropology Cooperative?
R1: You need the mask of anonymity to say what you really think. Otherwise it’s just banal platitudes and social niceties. Similarly, without the absolute protection of tenure there is not intellectual freedom.
R2: Look, I see this stuff as a more or less empirical process. I think there are probably lots of other ways to get to the bottom of plausible potential directions a bit of scholarship could go. It’d be interesting to see how reviewers’ reports compare in an open seminar of some sort to peer review. It’d be interesting to see if you really do get the same kind of feedback and criticism. I suspect you would.
D: Thanks very much to both of you–it’s such a bummer Reviewer 3 couldn’t make…
Reviewer 3: I am so sorry I’m late! I just have so much stuff to do this time of year! Grading, students, family, oh you just wouldn’t believe. I’m just overwhelmed and could only read the topics you sent out superficially. That said I think peer review is incredibly important, and as I’m sure others of you have noted is an important stepping stone in an academic career. It really helps you see what other people are doing, and is important service for your university and the discipline at large. I agree too that peer review is basically all that is keeping us from pseudo-scientific superstition and a new dark age. And once you start
publishing and getting your name out there, its just great. I mean when you get the best minds together collaborating on review there’s nothing you can’t do.
R1, R2, D: [blinks]
D: Wow. Well, thanks again for talking to me.