Reviewing a potentially interesting, albeit unreadable paper

Question

I am reviewing a paper for a major journal with very high impact factor. Some of the authors are very well known in the field and have several other papers published in this particular journal, such that their objectives and results are likely valid ("likely" is the best you can get in this field, so don't get hung up on this). However, they are not native English speakers and the paper is almost completely unreadable.

Usually, I would immediately reject the paper. Due to their reputation, however, I forced myself to stay with the paper for a couple of weeks. From what I could understand after several attempts, their results might indeed have the potential to be published in this journal, but the editorial board is typically very strict: If I send a report claiming that the paper needs to be pretty much fully rewritten, they will likely reject it. I do need, however, a fully rewritten version even to be sure that I properly understood what they claim, and then to reinforce or change my mind about the paper's relevance.

I am thinking about sending a report where I state what I believe the paper is about, and if this is the case then it might be relevant, but I cannot be sure because it is currently unreadable. If the authors revise it, then I will be happy to read it again. The journal might still reject it, and I'd feel bad because this is essentially blaming people for not knowing English. I then wonder if there is any other way, e.g. contacting the editors directly instead of writing a report. Would this be better? Are there other options?

Answer 1

Reviewer bias in single- versus double-blind peer review

Scientific peer review has been a cornerstone of the scientific method since the 1600s. Debate continues regarding the merits of single-blind review, in which anonymous reviewers know the authors of a paper and their affiliations, compared with double-blind review, in which this information is hidden. We present an experimental study of this question. In computer science, research often appears first or exclusively in peer-reviewed conferences rather than journals. Our study considers full-length submissions to the highly selective 2017 Web Search and Data Mining conference (15.6% acceptance rate). Each submission is simultaneously scored by two single-blind and two double-blind reviewers. Our analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions.

You're describing something that double-blind review is supposed to prevent. If the paper is unreadable, you should tell the editor, regardless of who the authors are.

I then wonder if there is any other way, e.g. contacting the editors directly instead of writing a report. Would this be better? Are there other options?

The result would be the same - the editors receive a "there's potentially useful science here, but the English is incomprehensible" response from you, and then it's up to them to decide what to do (remember editors can choose to accept a manuscript which reviewers recommend reject for). Practically speaking the difference if you write to the editors directly is that they'd have to submit the recommendation for you, which is a minor administrative hassle, but still an administrative hassle.

Answer 2

Stop wasting your time. It takes at most 30 min to find out that a manuscript is "unreadable". Why did you spend weeks with this? That is a disservice to yourself (and your students and your employer and ...). It is also a disservice to the authors, who have to wait for a decision where the editor should have done a desk-reject.

Answer 3

Converted from a comment by another user:

You can of course always contact the editor. However, precisely because it is a high IF journal, it is probably read by many people. Inflicting a "likely" correct and interesting but unreadable paper on them is a disservice. Research is communication, and the accepted language is usually English. You are not blaming people for not knowing English. You are performing a valuable gatekeeping service. The editor might suggest to the authors to have their next paper proofread before submitting. This is unfair to non-native speakers, but sometimes there is no "fair" solution.

I share this opinion. Just let the editors know your honest opinion on the paper; it is up to them to decide what to do. Even if the paper is rejected, this is not the only journal in the world.

I suggest not to contact the authors independently: this would break reviewer anonymity, which is an important safeguard to have (especially in cases like this one, where you are essentially giving a negative opinion on a paper by a well-known author).

Answer 4

It isn't your job as a reviewer to try to decode a badly written paper, nor to second guess editorial responses. Write an objective review saying what you said in the question "potentially interesting but almost unreadable" to the editor. What the editor then does is not your problem.

Answer 5

If you do not understand what the paper is about, do you think one single journal reader will? And certainly none of the readers will try for several weeks to understand a single journal article. If they do not understand it, they will put it aside.

So what is the use of accepting and publishing a paper nobody will be able to understand?

Tell the publishers that the content seems to be interesting and worth for publication, but it has to be rewritten.

Answer 6

Easy. Write some version of what you just did here. Say something along the lines of, this work is technically competent, addresses an important issue, is an important observation for the field, comes from a group who have made many important contributions, etc etc. You can then justify your statements with the scholarly details. You then state that the one major weakness of the paper is that the standard of written English is very poor; so poor that it took you a long time to get through it. (Don't say it took weeks, though). I serve on the editorial boards of several journals and I would not reject a paper on that basis. I make it very clear to the authors that they need to get help from somebody who can actually write in understandable English, and I give them at least one shot at fixing it. On most online review websites there is a place where you can send a message that only the Editor will see. If you're really concerned about this you can write there that you really want to see this paper get published, but that the English is awful.

Answer 7

Ah, the good times of reviewing papers... I miss none of them, so much painful reading. I am not going to provide any particularly smart or new answer, but I feel that sometimes we get so hung up into complex considerations and we don't see that the best approach is simply to go back to the basics. What always worked for me was to begin by asking myself the most basic questions. Do the authors make the problem they are facing clear and the contribution of the paper in that respect?

If this is already hard to detect, it's a rejection. If these points are reasonably clear, then proceed to the second level: do the authors clearly explain their procedure to arrive at those results and provide a (reasonably) fair analysis of their importance? These two questions are harder to answer as they usually involve some very in-depth specific knowledge in areas in which the authors themselves might know more than anyone else.

Even in that case, though, the path for a reviewer is not too difficult. In my opinion, one should always keep in mind that a paper is not the "discovery" itself, but it's an instrument used by the authors to communicate the discovery to others. So, in a sense, evaluating it should not be thought as conferring or not some honors to the authors, but rather as maximizing the benefit for the readers. If the paper does not fulfill that function because is unreadable (I would say for the average reader of that journal, a criteria which might be possibly relaxed by considering a more restricted group, depending on the subject), I think one should definitely ask to resubmit.

Remember, if you do that, you are not dissing the authors, you are just helping many readers learn something potentially important without going through the same pain that you experienced. In a sense, you should see yourself as the hero who takes one for the benefit of group.