I recently did a peer review for American Anthropologist, and was surprised (and delighted) to receive a note for them thanking me for my work and telling me that decision the editor made regarding the manuscript and — this is the new part — attaching all the feedback all the other peer reviewers gave the article as well. I’m familiar with this model, which is widely used in the biosciences, and I think it is great . Peer review is central to what we do but we rarely teach it to our graduate students, and the process itself is wrapped in a secrecy which makes learning on the job difficult.
At times peer review is like some sort of kinky Victorian sex act that Foucault would dissect: secret, unmentionable, but totally central to our academic/libidinal economies. People speak of it in hushed tones, afraid of the terrible secrets that will be disclosed if their behavior ever became public. Opening it up like this will help increase the quality of peer review by making review more transparent. I think it will also encourage peer reviewers to not act like total assholes when they review pieces. Which, let’s be honest, is something that needs to be encouraged.
I was curious about how this change was made so I reached out to Michael Chibnik, the editor of AA, and asked him how it came about. Thanks to Mike for answering these questions so thoroughly.
You recently switched to a new format for peer review feedback — can you tell me what that new format is, exactly?
After a decision is made on a manuscript, the peer reviewers are informed of the decision and receive copies of all the reviews of the manuscript. They receive only those comments that reviewers send that are intended for the author(s). They do not receive the “confidential comments to the editor.”
In many cases, manuscripts receive initial decisions of “revise and resubmit.” If a manuscript is resubmitted, we try to have at least one former reviewer (always someone who recommended “revise and resubmit” – not those recommending accept or reject) comment on the revised piece. These re-reviewers have access to the decision and reviews on the second go-round. But people who reviewed the first submission but not the second submission do not have access to the later set of reviews and the second decision. Unfortunately, this cannot be done with the manuscript control system (Scholar One) that we use.
Occasionally, I’ll edit a review before sending it on to an author. This is almost always done when a reviewer makes a recommendation about publication in the comments intended for the author. (I take these recommendations out.) In a very few cases, I have taken out what I regard as unnecessarily harsh remarks or speculation (often wrong) about the identity of the author. But the reviewers (unlike the authors) do not get edited versions of reviews. They get exactly what the reviewers wrote.
Many other disciplines use this format for peer review. Was there one discipline or combination of disciplines that AA was looking to emulate here?
Not really. Brandi Janssen, my editorial assistant at AA, had reviewed an article for a journal in another field and told me that she had seen the other reviews. This struck me as a good idea. I had long been disappointed by the lack of feedback I got from my own reviews of manuscripts. I wondered what the decision on the manuscript was and what the other reviewers had to say about the manuscript.
How was this decision made? Editorial fiat? Was the board consulted? Take us inside the decision-making process of our flagship journal.
I did a lot of consultation. I explained the proposed plan to the 50+ members of the AA editorial board and asked for their comments. I also ran the idea by various people I knew, including editors and former editors of prominent anthropology journals.
The responses were amazingly varied. Some people told me that was done in many other journals and thought it was a good idea. Others were strongly opposed to the idea (some saying that they had never heard of such a practice), thinking that they would lose anonymity even though their identity is blinded. They thought they could be identified by what they had to say. They were worried that their reviews (although anonymized) would leak out in other ways (such as being shared in classes).
I found out in the course of this that many journals – especially in the sciences – do not blind reviews. Reviewers know who the authors are and authors know who the reviewers are. A few peer reviewers for AA have insisted on signing their reviews. One potential reviewer refused to do a review because of the AA double-blind policy in which both authors and reviewers are anonymized.
Overall, about 2/3 of the people who responded to my queries were in favor of the idea, about 1/6 opposed and the rest somewhere in between. I initially decided not to go ahead with the plan because (1) some of the opposition was quite fierce; and (2) some prominent people in the field didn’t like the idea. However, I discussed the idea with couple of friends at the AAA meetings, who came up with a suggestion that alleviated my concerns. They suggested (and we now do this) that potential reviewers agreeing to comment on manuscripts be informed that their reviews would be shared with the other peer reviewers. If people objected, they could let us know and we would seek other reviewers to replace them. (The manuscript control system does not allow us to share some reviews, but not others.) So far no one has objected.
Are other AAA journals making this move?
I don’t know. I’ve heard that one AAA journal already does this, but I am not sure if this is the case.
What sort of feedback have you gotten from peer reviewers?
Not much. Two reviewers emailed me to say that they liked this. One asked about why I made a certain decision on a manuscript, given the reviews. And one commented that the other reviewers were really mean. (This reviewer had made positive comments on a manuscript that was turned down.)
* * *
That’s it. I’d be interested to hear what other people think about peer review processes in anthropology, and thanks again to Mike for making himself available to answer these questions.