The five virtues of peer review(ers)

I’ll give ’em to you right up front: mindfulness, honesty, tact, precision, and respect. Got that? Ok, now on to the rest of this post:

These days everyone loves to hate peer review. It is, the story goes, an old fashioned, evil form of collaboration which has been rendered obsolete by shiny new ways of communicating over the Intarweb. I am part of the problem, since I’m not uncritical of peer review. However, peer review has been so pummeled lately that I am starting to feel sorry for it since, no matter how valid many of the objections against it are, it got caught in a perfect storm of pre-existing antipathy and techno-enthusiasm that could sink pretty much anything. So I want to make a point here about peer review which I feel is under-appreciated: the value it has for improving not articles, but reviewers.

I’ve reviewed dozens of papers by this point, and received comments from many reviewers on my own work. I’ve worked in the video game literature which is large and where I can’t identify peer reviewers. I’ve also worked on Melanesian anthropology, a field so small that the pretense of anonymous review is almost comical. I’ve received remarkably helpful feedback and also ridiculous feedback. Being on the receiving end of peer review has instilled in me a deep desire to do good review, and a deep appreciation of how hard that is to do. Bad peer review is like the comments section of this blog in one of its bad moods: vicious, narcissistic, and extremely abstract. Good peer review, on the other hand, requires one to cultivate a wide range of virtues.

First, peer review teaches you to read closely. As academics we can always decide to not read something, or to dislike it because of its tone, or other reasons. To honestly disagree with a paper for peer review, you really need to read it closely and discern what it actually says — not what you think it says or want it to say given your current set of prejudices. The more you disagree with something, the more carefully you have to scrutinize it.

Secondly, peer review requires you to be honest, frank, and candidate about what a work says. A diplomatic person often avoids saying what they think of someone directly to their face. When you can’t avoid speaking directly to someone’s merits, it is just easier to tell people what they want to hear. It’s also very easy to just dump on someone and call their work worthless. But evasion, flattery, and attack are all far different from what is actually needed: rigor, prudence, and honesty.

Good peer reviewers also, third, have to be good writers. There is a veil of secrecy — or worse, the pretense of one — between you and your reader. Any sort of negative feedback always stings (indeed, for some anything but lavish praise stings), and the pain of peer review is amplified by this secrecy. It takes a lot of work to give critical feedback in a way that people will take on board, especially since outright flattery is disallowed. One must strike the right tone.

The fourth good quality of peer review is that you must not only strike the right tone, you must actually have something concrete to say. What specifically is wrong with the paper. How specifically can it be improved? Most academics have good noses and can tell if something stinks. Fewer, however, can figure out what in particular is haywire in a paper. Even fewer can provide suggestions about what sort of remedies might be undertaken. The biggest problem, surprisingly, is that few of us have ever actually sat down and explicitly articulated what actually makes a paper good. We rely on our habitus, which unfortunately (or fortunately?) we can’t just break off a piece of and mail to our collaborators. And less face it, we could stand to be more explicit and precise in our understanding of what makes work valuable — habitus only gets you so far.

Finally, there is one virtue of peer review which seems in particularly short supply: the desire to improve the paper the author has written. Too often, peer reviewers instead complain that the paper is not as they themselves have written it. Sometimes this phenomenon is easily spottable because it is so outrageous: “This paper on X and gender should not be published because the author should have written a paper on X and race”. Often times this lack of respect for the scholarly project of the author hides behind suggested changes which seem to make sense but, if you think about them, are actually unfair. So remember: it’s not their fault that they’re not you. Your job is to understand who they are, and how they and their work could improve.

mindfulness, honesty, tact, precision, and respect: these are not just the virtues of peer review, they are virtues period. Peer review is important because it offers an powerful opportunity to cultivate them. The fact that so many people do not chose to do is, of course, partly caused by the way peer review is architected. But let’s face it: bad peer review happens because good people are hard to find — and that’s something that’s probably true everywhere. I’ve benefitted tremendously from the responsibility to peer review papers because it elicits personal growth. My recommendation? The next time you get lousy feedback from someone, use the incident to strengthen your resolve to become the peer reviewer you wish your paper had gotten.

Rex

Alex Golub is an associate professor of anthropology at the University of Hawai‘i at Mānoa. His book Leviathans at The Gold Mine has been published by Duke University Press. You can contact him at rex@savageminds.org

11 thoughts on “The five virtues of peer review(ers)

  1. Thanks Rex for this post, but I am left wondering, what do you think is wrong ( or not totally right) with the criticism of peer -review?

  2. One minor self-check I’ve noticed being helpful: if you’re thinking of writing a negative comment about a paper you’re reviewing on the basis that it is shockingly out of touch with the literature, has ignored very relevant related work, fails to give due credit, etc., etc., and the papers you have in mind that it failed to engage are all yours or your close collaborators, it might be worth really rethinking that. It’s really easy to see connections in your own work that are very non-obvious if even there at all to a third-party observer, and somewhat unfair to fault another author for not having divined them. Sometimes this boils down to pure careerism (cite me! cite me!), but I think more generously it often is a genuine mis- or over-reading of one’s own work, which is hard to approach from a third-party perspective.

  3. “Secondly, peer review requires you to be honest, frank, and candidate.”

    It also requires careful use of autocorrect!

    But good post otherwise. I would add that I often try to work on improving not just the paper I’m reviewing, but the person’s future papers. If the paper has serious flaws in argumentation, the writer has to learn how to make a good argument. Often a reviewer has to do a job that should have been done by the writer’s graduate supervisors.

  4. Great post. Makes me think also of the role, function, and form of the book review–which is not the same in many ways, but is precisely the same in others. Peer review has the advantage of being (sort of) anonymous, and authors have something of an immediate opportunity for rejoinder. Not so with book reviews, which can be generous and appreciative while still being critical, or just unabashedly nasty, devoid of generosity and any sense that our work, even after it’s published, is still in fact in-progress, in conversation, in use. [Perhaps that’s another matter about which we talk too little: how we think about the book, or any of our published works, as finished products/ culminations that nonetheless can and often do lead to other papers, products, arguments that revise, rethink, extend, answer critiques etc.] To boot, and to my knowledge, book reviews themselves receive little to no editorial input (whereas peer reviews can be ignored by editors or only partially addressed). This, though book reviewers are making arguments, too, sometimes very shaky and evidently blinkered ones, and often either raising or striking at the professional credibility of authors in very public, very material ways—with no real possibility for dialogue. I’d say your final paragraph should be one sent out with every request for a book review, too. Along with a reminder that our professional committments to the discipline require the sort of care and attention to review (peer, book, annual, tenure, etc.) that makes it a critical but helpful, constructive, and meaningful process, too.

  5. While applauding what Rex has written, I would like to take this opportunity to suggest that we also consider the importance of non-peer review. My proposal would be that we encourage journal and book editors to include among their pool of reviewers people who, while serious scholars, are not experts in the geographical or topical area in question. Why? To get an opinion on the interest or importance of the work in question from a source outside the handful of people who will have expertise in both the geographical and topical area. In a world where we wonder about the inability of anthropological ideas to break out of the closed communities in which they are formulated, a comparative perspective could add considerable value and encourage broader debate.

  6. Other–than–SME peer reviewers who are Fred Eggan types (read everything, bridge national traditions) would seem to me like helpful additions to any peer review process. Other–than–SME peer reviewers who are no more than experts in a subject matter that is not under discussion in the item under review would seem to me to be less so.

  7. MT, a valid point. But let’s work with it a bit. How do you measure the difference between qualified and unqualified Other-than-SME reviewers?

    Let us agree at the outset that a medieval historian won’t have much to say about a paper on particle physics. If we look at academia as a space of differences, medieval history and particle physics are simply too far away from each other to expect an expert in one to have anything valuable to say to an expert in the other.

    What, however, of the subspace broadly labeled social sciences and humanities? Here blurring and overlap abound. So, I would argue, do opportunities for fruitful interaction across “disciplinary” or geographical boundaries. Area studies, for example, often benefit from interactions between anthropologists, historians and other social scientists. In China studies, for example, collaboration between anthropologists and historians has been particularly fruitful. One of my current projects is a paper on the keieijinruigaku (Anthropology of Administration) group based at the National Museum of Ethnology in Japan, which has been an enormously productive, twenty-year long collaboration between anthropologists associated with Japan’s equivalent of the Smithsonian Institution and management professors from business schools.

    One can also make a case for people who share similar interests but work in different geographical regions. I am, at the moment, for example, also engaged in a conversation on OAC with Logan Spark, who studies religion and healing in Turkey. He has been writing about attending a workshop on religion and healing in Mexico, where he just encountered susto “fright.” I have mentioned to him that in my work with a Daoist healer in Taiwan, a ritual called “catching frights” was one I saw performed, often several times a day. I am not advocating a return to Frazer-Jung-Joseph Campbell style, “Aren’t they all doing the same thing?” theorizing. I am advocating a closer look at specific differences in how similar rituals motivated by similar concepts are performed in widely separated places.

  8. These are good reflections on peer review.

    “bad peer review happens because good people are hard to find”

    This statement should be extended: Mediocre people are also hard to find for peer reviewing. Editors often cannot find enough reviewers, period (whether or not they are “good” reviewers). This is a real crisis, in anthropology and in scholarly publishing generally.

    Here are some made-up figures: Assume that each manuscript submitted to a journal gets three reviews, and the acceptance rate is 50%. That means six reviews are needed for every published paper (ignore re-submissions and other complications for now). So take the number of papers you publish annually and multiply by six to find the minimum number of reviews you should write each year. If you think you are a good reviewer (as outlined by Rex), then you should contribute more reviews (to help balance out the mediocre and bad reviewers). So now double your annual review figure.

    Since I first did this little exercise a few years ago (after reading about it from a journal editor, in biology I think), I don’t think I have turned down any reviews. But the problem remains of how editors can find reviewers. My own recommendation is that the work be shared with the editorial board. In my experience, only one journal editor has leaned on members of the editorial board to help find reviewers; most either just use the board to review more manuscripts than outsiders, or else they don’t pay any attention to the board at all.

    And to all you readers out there: If you aren’t getting lots of manuscripts to review, contact the most relevant journals and offer your services. There are a lot of smart people reading SM, and many or most probably are (or could be) good reviewers.

Comments are closed.