What to Keep in Mind about Refereeing Work

February 23, 2012

This post is motivated by some of my concerns relating to suggestions about unblinding the referees (as discussed here).  While I agree that there are problems with the current state of the peer review system, I tend to disagree strongly with some proposed resolutions.

During my last year as a graduate student at USC, I was managing editor of the Pacific Philosophical Quarterly.  I’ve submitted my own work to journals, and, in addition, I’ve served as a referee for a few different journals.

I learned a number of things from being “behind the scenes” at PPQ.  For instance, I learned how much of a manuscript’s progress through the review system depends on factors extrinsic to the manuscript.  If there are two equally appropriate people to ask to referee a given paper, who differ on their threshold for “reject” vs. “revise and resubmit”, that could make the difference for what happens to your paper.  It isn’t personal. I didn’t have a scorecard for how “easy” different referees were. It was just the luck of the draw.

I also learned that some of the biggest issues for the peer review system come from the fact that it is treated like a volunteer gig.

Why are so many journals so slow about getting decisions on papers?  There are several possible bottleneck points for journals, but the two biggest, in my experience are: 1) getting potential referees to agree to review a paper, and 2) getting referees who have agreed to review a paper to meet deadlines.

While getting bad comments is frustrating, and can often make a rejection feel particularly unfair, I honestly think that such issues would be vastly less important if all manuscripts were issued decisions promptly (sometimes I joke that it would be fine for most journals to simply provide an efficient distribution of injustice).

So, why do we get such bottlenecks in securing referees and in getting feedback from referees?  Here, I think the answer is blindingly obvious: It is hard to get referees, and hard to get them to prioritize refereeing work, because the profession does not treat such work as valuable.  Combine this with the fact that we all have more work to do than we have time to do it, and that refereeing a paper can often be more tedious than it is exciting, and it should not be a mystery why it is hard to get quality refereeing done in a timely manner.

You know how I can tell the profession doesn’t treat refereeing work as valuable?  There are no real incentives to do such work.  Neither do we entice people to do it with rewards, nor do we punish people for failure to do it.  Since one can only referee as much as one is asked to referee, I’d be inclined to favor positive incentives for people who do referee, rather than penalties for people who don’t, but the basic point is just this: If refereeing work really is an invaluable service to the profession, why aren’t we putting any professional value on its performance?

I’d be curious to know whether any university treats this sort of service to the profession as a serious factor in tenure or promotion. Where by “serious” I mean something more than listing it in the official description of tenure factors.  What I mean is: Does it genuinely count in your favor for T&P to be a good professional citizen? Does it genuinely count against you if you are not?

Maybe we can’t unilaterally change the way university administrations value things for Tenure and Promotion.  I don’t want to throw in the towel on that yet, but it isn’t the only option.

Journals (as an extension of the Publishing Houses) are also dropping the ball on this.  What does someone get for putting in the time and effort to do good review work?  Basically: pride in a job well done.  I know that some publishers compensate people who referee book manuscripts.  This seems like good practice, since refereeing book manuscripts is at least an order of magnitude more of a burden than refereeing a journal article.  But, and this is the important thing: refereeing journal articles is still a burden!

Why on earth are we running things in a way where journal editors are put in the position of sending messages that might as well say this:

Dear Very Busy Academic,

I have a bit of time-consuming work that I need someone to do, and I think you have the right knowledge and skill-set to perform it.  While I know we are all very busy with teaching, researching, mentoring students (and if we have any time or energy left over, perhaps some of it should be reserved for a life outside of work), I am hopeful that you recognize how the peer review journal process is held together with duct-tape and dreams, and will voluntarily pitch in to help out.  In thanks when you finish, I will even send you a form e-mail that says “thanks” (what more gratitude could you hope to receive?)

If you don’t have the time, can you recommend a few other people who are more easily motivated to provide valuable services for no compensation?

Best,

Editor

PS: If you do a good job, you will no doubt become the first person I think of for any similar requests in the future. Yay!