The Evidential Value of Bad Referee Reports

The following assumes that the referee report in question in fact misreads your paper. Often this is not the case, and the advice below will not apply. Other times, this advice will apply in a restricted or qualified form. And of course, your mileage may vary. Also, this advice is descended from advice helpfully given to me as a graduate student by Mark Schroeder.

The Prologue: You have sent your paper off to a journal and the fools, the damnable fools, did not recommend accepting your paper. If you are fortunate, you got referee reports, though it may not feel fortunate, as some of the comments seem like they did not even read your paper, or perhaps like they read another paper entirely. They certainly misunderstood what was going on in your paper. Ugh!

If you are like me, you will have many emotional reactions to this rejection. You may think it makes sense to barrel through those emotions by sending the paper off immediately to another journal, or you may become avoidant and focus on another paper, which does not have this frustration and feeling of insecurity attached. I don’t necessarily have have great advice on how to deal with the emotional element here, except that it is worth recognizing that you probably care a lot about the paper and the reactions to it, and your emotions are real, and important. What I have is advice on a stance that can sometimes be helpful to adopt when reading the referee reports, in order to get more out of them when they are tragically misguided.

The Problem: Referees are extremely non-ideal readers of your papers. They are often hurried, harried, or hostile. They cannot talk to you about their questions, or get clarification. They have been given instructions on what threshold to use when recommending acceptance (which may or may not have been clear, and which they may or may not have attended to, and which often induces them to look for problems which may not exist). They may be less expert on the topic than you would like, because the journal just asked the referees who would be better fits to read other papers quite recently (or for any of thirty other reasons). And they may just be grumpy/distracted while reading it for entirely unrelated reasons.

The False Dilemma: Some people seem to think there are only two options for how to read a bad referee report, and since neither is worth doing, by inference, they seem to think that you should not pay them any mind. Either (a), you take the report at face value, and consider each objection or concern as having prima facie merit or (b), you stew about the referee’s ulterior motives or speculate about why they had such hostility to your paper. Of course, (a) is rejected since the overwhelming evidence of your paper itself clearly makes the case against taking the substance of their objections seriously. And (b) is not worth spending your time on, because you have better things to do with your time: don’t let the haters live rent free in your head (this not only makes you landlord, but a bad landlord, so it is foolish on any accounting). There is, however, a third way.

The Third Way: As stated above, referees are readers, and since they documented their impressions of your paper, that means a referee report is some evidence about how your paper reads. Now, it is not comprehensive evidence, and it is not systematic, and it reflects one member from a potentially weird group of your potential readership. But it is evidence about how your paper reads nonetheless, and, oddly, it does have some virtues. Lots of people who might read your paper will share some of the relevant features of the above mentioned referee. Whether they are quickly skimming your paper, whether they are antecedently hostile to your view, whether they are being sloppy because they are in a hurry, whether they are grumpy because they are just having a terrible horrible, no good, very bad day: very few of us can expect our papers to mostly get ideal readers, most of the time. Also, it is a very good example of how people from the potential referee pool may respond to it. So I think there is some value that can be gained from learning what a hurried/harried/hostile reader got out of your paper. Note, that unlike option (a) in the above dilemma, this does not mean that we take the report at face value, but neither does it mean that we spend any time worrying about the exact motives, mindset, or malevolence of the referee.

Instead, treat this bad referee report as data. Say to yourself: “huh! it was easier than I expected for someone to misread my paper this way.” If it helps refer to them as someone hurried/harried/hostile/horrible. And rather than asking yourself questions like “Should I make the specific changes they recommend?” (you already know the answer is no) or “From hell’s heart I stab at thee; for hate’s sake I spit my last breath at thee.” (this is not technically a question, but rather, a dying curse from Star Trek II and Moby Dick), instead ask: “What can I change about my paper to make it harder for H³-Referees to misread it this way?”

What Can I Change About My Paper to Make it Harder for H³-Referees to Misread It This Way?: This is a good question, and obviously depends a lot on the specific ways the paper is being misread. But I think there are three major categories of mistaken advice worth going over.

1) They want me to talk about X, but X has nothing to do with my paper.

2) They don’t think my point is interesting/major, but my point is interesting/major.

3) They think I am saying/defending Y, but I am not saying/defending Y.

As to item (1), I think this is the most common type of bad advice I’ve seen, and it is also the most dangerous because it is the easiest to accommodate and as a result, people frequently do, and it makes many papers much worse. It’s the result of a bad referee mistaking what your paper is about, or what some part of your paper is about, and so thinking that X is relevant. Now, again, it often comes down to the details. This could always just be because X is that referee’s hobby horse, or they tend to see X as relevant to everything. But some possible things that could be behind it other than this are that you are talking about whatever your paper is about in a way that makes it too easy for someone to get confused, especially if they are reading quickly. Your disadvantage, as an author, is that “the debate”, which pre-exists you, shapes a lot of people’s assumptions and associations, and can lead people to draw conclusions, especially if they aren’t being careful, from one thing you say. You say “counterfactuals” and they fill in all sorts of things. You say “laws of nature” they fill in other things. You put them in sentences near each other, they may assume they know exactly what you are talking about, even if they don’t read the other nearby sentences. Your advantage, as an author, is that you get to control the order you present the words in, what you say about those things, and you can put those words farther apart, or put “notably not related to David Lewis’s account” in bold or what have you. So, instead of adding a section about X, you ask yourself “what would lead someone to think X is relevant to my paper?” Again, maybe there isn’t something. Sometimes referees just project their own nonsense on your paper. But other times that lens is helpful for you to see that, while you knew X was irrelevant, someone skimming your paper might naturally expect section 2 to discuss X, because section 2 seems (when skimmed quickly) like it’s about something other than what it’s really about.

The remaining two points involve applying the same basic observation to slightly different problems. But there is one major important twist for (2): Lots of people never seem to know how to set the expectations correctly early on in their papers. I don’t know why this is, other than that lots of times we don’t really teach people how to write papers; we teach people how to write arguments, and then just assume that if you do that for the length of a paper, it’s a paper. But a paper length argument with an interesting conclusion is not the same thing as a paper whose introduction makes it clear why anyone would or should care about its conclusion. Part of the role of the introduction is expectation setting. Maybe all you are doing in the paper is showing that one way of defending an argument against View R doesn’t work. If you frame this the wrong way, you will get referee reports that tell you all about the fifty other ways of defending View R. You, of course, do not care about the fifty other ways of defending View R, because they were not the point of your paper. Your paper matters because lots of people defended View R in that way, and you want to show that it’s bad for them to not be able to! But somehow, you didn’t write the first page your paper, or the abstract, in such a way that the referee would feel silly listing those as problems with your paper. So, in a case like that, the framing issue is key. Maybe you said why this was crucial later in the paper? Maybe you forgot to explain why. Maybe you explained in detail but the referee missed/ignored it. Again, it’s also possible that there is nothing you can do to the paper to make this improvement. But what you should ask is: “how can I make it harder for a referee to write a report where they both a) paraphrase my description of what my paper is doing, and then b) also say that this is not a worthwhile thing to do?”

Paired with this is the situation where they misdescribe what you are doing, in which case, you need to ask: “how can I make it harder for them to mistake what I am doing?” You might think the relevant standard of clarity is something like “would a reasonable reader be able to understand what I am saying, if they wish to?” This is a great standard for cooperative communication among friends in some glorious hypothetical state of leisurely discourse. In academic journal articles, you want to employ a standard that is something more like, “how can I make it difficult for a recalcitrant reader to misinterpret me, when they are hurried/harried/hostile, having a bad day or distracted, and potentially bringing their own assumptions or baggage to the paper?” These are different standards. The former is like laying your meaning out in the middle of the yard for anyone who wants it to grab. The other is like placing thirty copies of your meaning around the reader, like so many rakes for them to step on, and ensuring that there is no path they can follow that does not end with your meaning hitting them square in the face.

This might sound like a lot of time and energy to put into bad referee reports, but what’s more accurate is that I’ve over-written the blog post a fair bit. The key idea is that you should use their bad report to identify where your manuscript was too easy to misread, and take that as a basis for revision, so that readers have a harder time mistaking what you are doing and why. This way, even the bad feedback can be useful feedback (though, it is not always useful feedback), and the changes you make improve your paper, rather than merely mollifying referees.

Advanced/masochistic authors can even sometimes simulate hostile referees for themselves before sending things off to save time (though this is not recommended for perfectionist/anxious authors).

My other blog has a very narrowly circumscribed purpose (using modern philosophers’ attempts to understand humanity to help me try to understand humanity), and a very broad intended audience (everyone), so this post is not appropriate to write there, because it is advice for professional philosophers on how to deal with bad referee reports. It is advice that I have given out several times, usually verbally or on twitter, and it seemed to make sense to put it all in one place, longform.