Predict Your Reviews

Researchers who submit papers to conferences or journals should do a little exercise whenever they submit: write down a prediction about what your reviews are likely to be. Doing this exercise helps you:

• Understand the strengths and weaknesses of your own work;
• Shift from a mindset of regarding reviewers as an exogenous environmental hazard towards collaborative colleagues;
• Notice communication failures with your reviewers;
• Write author responses more quickly;

The exercise

If you understand your paper well and have just submitted it, this exercise takes about 30 mins as you submit and maybe another 15 extra mins when you get your reviews back.

For each paper you submit:

• Skim it from start to finish.
• Write down the top 3 criticisms you would have as a reviewer.
• Write down your probability estimate that the paper is accepted.

If it takes you longer, that’s ok, it’s probably still worth it. It will be faster the more experience you have reviewing papers and the better you understand your current work.

When you receive reviews back, assess the extent to which the criticism offered by the reviewers are the same as yours.

Interpreting your results

If you agree with reviewers about the main criticisms of your paper, that’s great. It means that you accurately communicated your idea and that you clearly communicated the strengths and weaknesses of the work. Responding to the reviews is now a matter of debating how important your paper’s weaknesses are compared to its strengths. Presumably, if you submitted it, and you clearly understood its weaknesses, you think there’s a case for this.

If you disagree with reviewers about the main criticism that can point to a few possible explanations:

• Possibly the paper does not communicate its idea clearly enough, the reviewers got confused, and they responded in a way that doesn’t really engage with your contribution. These reviews can be the most frustrating to work through, but the only solution really is to redraft and practice explaining the paper to a slightly wider group of people.
• Possibly, the reviewers point to a mistake that you didn’t know about, a body of work you didn’t know about, etc. This is peer review working at its best and giving you a great opportunity to improve your work. Sometimes you can quickly integrate the new perspective or argue that it has only a small effect on your paper’s correctness or importance. Other times you just need to redraft to take this into account.
• Possibly the reviewers were lazy or incompetent. I think this is genuinely quite rare and it’s almost never useful to act as though this is true. Importantly: reviewers who have been confused by your writing will rarely know why they got confused and often won’t even notice that they are confused. As a result they might make claims that seem wrong or incoherent. Your job is to work out where the misunderstanding happened that caused them to get where they are.

Over time, assess your calibration on your acceptance forecasts. If you feel like it, you can make forecasts about colleagues’ papers that you have read carefully to start refining your predictive accuracy, though it’s probably best to only share predictions when requested!

Why I think this helps

A really common failure mode in researchers, especially students but also often for more senior researchers, is to treat journal/conference reviewers as an annoying obstacle that randomly zaps their paper unpredictably. This is a really bad pattern to fall into because:

• It is highly demotivating to be randomly zapped for no obvious reason.
• Blaming the incompetence/stupidity/laziness of your reviewers is a psychologically satisfying response to rejection but it blocks you from learning from the reviews.
• Realistically, most people who actually read your paper will be less attentive and knowledgeable than your reviewers. If your reviewers aren’t getting it, you aren’t writing clearly enough. Predicting criticisms helps with this. You will find that reviews do in fact become more predictable. Not perfectly predictable, and the manifestation of confusion can be a bit random. But more predictable.

This approach also makes resolving reader confusion your responsibility. Perhaps counterintuitively this gives you more of a sense of control over the whole process. I find it very satisfying to know that a confusing and apparently random review is not a case of the reviewer giving me completely arbitrary feedback, it’s a case of the paper not being so clearly written that nobody could possibly get that confused by it.

It also gives you more ability to manage the ups and downs of research outputs. You can begin to reward yourself for having submitted a 50% paper, feeling fairly confident in your assessment, instead of only feeling good about an accepted paper. This lets you target forward indicators instead of needing to rely on lagged stochastic rewards.

Of course, it also helps you write better papers. You can understand the actual strengths and weaknesses before you submit the paper, adopting the mindset of a reviewer, and then address the weaknesses ahead of time.

FAQs

But if I know my paper has shortcomings, shouldn’t I not submit it?

This question occurs to students who are inclined to underconfidence. The answer is no. All papers have shortcomings. It is not concerning if there are problems with your paper. They just need to not be problems that are more important than what makes the paper useful, and need to not undermine the main message/argument/result. The only exception is basic errors of correctness, in which case you should fix it and submit later.

I rewrote my paper to address the reviewers’ confusions but then the next conference the reviewers were also confused about totally different things!

Sometimes this happens. Oftentimes the mistake was in assuming that the reviewers knew what they were confused about. A confused reader will often say slightly random things because as their understanding of the topic departs yours, your sentences become less and less effective at guiding their understanding and you start to speak about different things. Your job is to diagnose the root cause of their confusion, not to answer the surface level question/complaint.

If confusion is my responsibility, what about reviewers who are actually lazy/incompetent or acting in bad faith?

On rare occasions you might come to the conclusion that no reviewer could possibly have gotten confused by your point if they actually read what you wrote. I am confident though that the vast majority of people who think they are in this situation are wrong and should reassess.

If you are quite sure that you are not wrong about this, remember that many of the people reading your paper will also not actually read what you wrote. If you care about those people understanding you too, you might want to think about ways to make your main point even more obvious, possibly using figures/diagrams/abstracts.

At some point it is also ok to accept that although pre-emptively resolving confusion is your responsibility, you don’t need to resolve all the possible future confusions and you do need to prioritise your time on the audiences you care about most. This is also why jargon can be ok: you’re writing for a specific audience, not others, and that’s fine. The crucial thing is to make this a choice rather than an unpredictable random thing that happens to you.

Twitter  Email