I have thought about writing on this subject for a while but it was this Tweet from Britt Koskella http://brittkoskella.wordpress.com/ on the 19th November 2013 that finally stirred me into action.
As an editor (I am for my sins, Editor-in-Chief of Insect Conservation & Diversity http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1752-4598) I love people like Britt. It is such a joy to be able to select their names from the journal data base and assign them a manuscript, knowing that nine times out of ten they will accept my invitation to review a manuscript and that on that tenth occasion they will very kindly suggest an alternative (sometimes two or three) reviewer who will also almost certainly accept my invitation. Britt Koskella, I love you and those like you :-) My reply to Britt was as follows:
You will have noticed that I confessed to doing too many myself; in fact in addition to those manuscripts that I read as an Editor I do on average, forty to fifty reviews for other journals. Like Britt I have a hard time saying no. I am getting better though – I actually turned down two this month ;-)
There is a lot of debate at the moment about the peer review process in general with a number of journals adopting an open mass review process and other journals opting for the as long as the science is sound it is publishable approach. We are, however, mainly, despite its many flaws, still operating on the traditional two referees per paper peer review system.
So how many papers should you referee asks Britt? The general rule of thumb to entitle you to call yourself a good citizen is to agree to referee two papers for every paper that you submit as that is the minimum number of referees that you would expect to look at your own papers. To be on the safe side and to feel that you are making a real contribution to your community, I would suggest that a 3:1 ratio is very acceptable. In my experience as an Editor of two journals and as an Associate Editor on three other journals, there are a number of people who referee many more papers than that and a disturbingly large number of prolific authors whom, as far as I can see, never ever agree to referee papers.
As an Editor, what do I want from a referee? In a nut-shell, someone who reads the paper thoroughly, checks first that the experimental design and statistical analysis are sound; if the experiment is not designed properly then it doesn’t matter how well the paper is written, it is not worth proceeding with; that the appropriate literature is cited (and by this I don’t just mean the referee’s own papers) and that the paper fits the remit of the journal and advances the subject area significantly. I also do not want the referee to say how good the paper is in the comments to authors section and to tell me in the confidential comments that it is crap. If you don’t like it then have the guts to tell the author why, don’t leave it up to the poor Editor to try to explain why he/she is rejecting their paper despite the apparently favourable comments they can see in the referee’s reports. I also expect total impartiality; you might not agree with what you read but unless the methodology is flawed that is not a reason to reject the paper. Be open-minded and fair above all. If you are rejecting a paper, be constructive, authors at the start of their career are not as resistant or as resilient as old timers http://dynamicecology.wordpress.com/2013/11/18/are-you-resistant-or-resilient-in-the-face-of-rejection/. Above all be fair, write your report bearing in mind the sort of review that you as an author would like to receive. Do unto others as you would have others do unto you and that goes double for those of you who don’t referee as many papers as you should! I am very tempted sometimes to do an instant reject on authors who have turned down my invitation to review a paper, especially if I have just accepted one of theirs.
I used to run a course for PhD students about getting published and it always used to amaze them that decisions on whether papers were published or not was dependent on the opinions of two to three people. My response was that if you think that is bad, decisions about grant funding are often made with just as few opinions and those decisions have even greater implications for career prospects.