Is it unfair for paper circumstances to play a role in the review of the paper? [closed]

Question

Let me say beforehand that this question does not refer to scientific correctness and treats it as undisputed, i.e. the paper is assumed correct in the sense that it clearly states a valid purpose, has an adequate literature review, the theory and methods are scientifically sound.

After a brief introspective, it came to me that I often take into the account under what circumstances a paper was written when assessing the scope / prestige of the venue it is to be published at. This usually includes funding, if any, seniority and number of the authors.

On the one hand, I find it unfair to expect the same scale of experimental devotion from a student-mentor author pair from a developing country with no or relatively little funding as from a research group with multiple grants from a developed country. Given the relevance of the contribution, the first group shouldn't be rejected on the grounds that they don't have a comparable abundance of resources. Also, they shouldn't be discouraged as they managed to make a difference without those resources. And, after all, we do this kind of expectation management in other aspects of life, e.g. one doesn't fail a student because they wrote the assignment in a free text editor, because they can't afford MS Word.

As a concrete example, I once saw another reviewer comment that they would like to see a more significant average deviation on the graphs. This requirement would've required the authors to run tens of multi-hour simulations on a cloud cluster consisting of ~50 high-end instances, each billed by the hour, and it is also rather irrelevant to the proof or the contribution itself. The two authors received a government grant which translated to roughly $1200 for this research at an university that doesn't have subsidies with the cloud provider.

On the other hand, this is clearly a bias. Augmented additionally by the fact that it is not possible to implement during a double-blind review. Also, somewhere in there is an argument how practices to include low-cost research would result in research funding being gradually cut in some way or the other.

So, my question is the dilemma whether it is unfair to let capabilities outside of a research group's direct influence contribute to the review of their paper (again, only once the contribution is established)?

My field is computer science, so, while not strictly pen-and-paper, few resources are needed, relative to other fields, e.g. experimental physics, bio-medicine, etc., to do meaningful research.

Answer 1

No, your review should not change based on knowledge of the authors' circumstances. Your role is to give the editor an objective assessment of the strengths and weaknesses of the author's submitted manuscript. However, you can still help less advantaged authors. Write your review to focus on the strengths and weaknesses of their work, and leave the authors to choose how they wish to address these.

To take your example: suppose that the authors have developed a new algorithm for X, which you consider to be a worthwhile contribution to the field. However, they go on to claim that their algorithm outperforms the current standard, based on only one test-case.

An unhelpful reviewer might write: "This paper has potential, but it cannot be accepted unless the authors run more test-cases". A more helpful review would be: "The algorithm in this paper appears to be a useful contribution, and it is deserving of publication. However, I do not feel the authors currently have sufficient evidence to support their unqualified claims regarding performance, and this should be addressed before acceptance."

By doing this, you leave the authors with a choice: they can either run more experiments, or they can put caveats around their discussion of performance. (Of course, you should only do this if you feel the paper is publishable in such a form.) A wealthy research group will likely choose the former option, a less well-off one might settle for the latter. However, both have been held to the same standard of scientific correctness.

Answer 2

Unfortunately you seem to be specifically asking for opinions which isn't especially valued here. However, I'll give my opinion on this.

A scientific paper (or any scholarly paper) should stand on its own. Either it properly makes a case or it does not. The resources involved in establishing truth should not be a factor, other than to say WOW in some cases. We should neither value nor devalue a work of scholarship based on who did it or how.

The case you mention is instructive. Asking for better significance of results is completely valid. With low significance the likelihood that the results are invalid is increased. Who knows what would happen if more resources were poured into it other than with more data (and a good design) the confidence in the results is increased.

However, in cases in which researchers from places with few resources achieve good results, it is the people themselves who should be especially honored, but not the paper itself. Again, you can say WOW, but the paper stands for itself. We seek truth in scholarship, nothing more or less.

Answer 3

Let me answer this question by other questions:

Would you like to live in a world where pharmaceutic products which the population is administered is based on studies in which a reviewer accepts that given the resources the researchers had at hand it was unrealistic that they could make a clear line between "drug is working or not harmful" to "drug is not working or harmful" can not be drawn, and therefore we accept that we regularly take unproven results as proven?

Would you like to live in a world where simulation software contains simplifications which work "usually" and not "most of the time" when predicting if a structure is stable?

Would you like to live a world where it is ok to accept an algorithm for a self driving car even if it not proven up to the full level of required credibility because the authors could not afford to test it properly?

So, no, sometimes lives, health, well-being and economic progress depend on science being right. We should not just do it sloppy.

Answer 4

So, my question is the dilemma whether it is unfair to let capabilities outside of a research group's direct influence contribute to the review of their paper (again, only once the contribution is established)?

Is it fair that there are people suffering from war, poverty, starvation and all kinds of other serious problems in various parts of the world as I sit in my comfortable chair typing this answer, free to think about academic matters? No, in all honesty it strikes me as deeply unfair. Perhaps I should drop everything else and dedicate my life to correcting all the injustices out there. But, recognizing that both my ability to fix the world’s problems and my willingness to do so are limited, instead I have taken it upon myself to write this answer in this small region of space-time that I currently occupy, and I will try to do the best job of it that I can.

The same principle applies to the situation you ask about. When I review a paper, I take on a responsibility to provide the most honest, competent and professional review I can. Thus, the standards of review I apply must be the same ones that I would want to see applied to my own work, and the review must be as unbiased and uninfluenced by irrelevant factors as I can humanly (and consciously) manage to make it. To do anything less would be to betray the trust of the people who chose to assign me the review and of the journal’s readers who expect papers of a certain quality. I can’t fix all the world’s problems, nor do I have a responsibility to do so, but the one thing I do have a responsibility for is to do the job that I have committed to doing as well and impartially as I can.

The conclusion is obvious. It is unfair that researchers from some parts of the world are constrained by a serious lack of resources that prevents them from doing good research as well as equally or less talented researchers in richer countries. But as a reviewer, it would be wrong for you to “grade on a curve” and set aside your usual standards for researchers who you feel are disadvantaged in some way. Your desire to give those researchers a little “help” is well-intentioned and even admirable, but you must resist that temptation; to make the world a better place, start by doing your job as professionally as you can. Of course, if you still have time and energy left over after that, there are plenty of things you can do try to help researchers from poor countries, so certainly that should be something to consider as well.

Answer 5

Thank you for raising this issue. I have the same sort of questions.

Pondering over your question, there is another bias you don’t mention: overcalibation. When a reviewer asks for another experiment, the reviewer should ask him/herself first, whether this really contributes to the scope and purpose of the article.

Second, the review process is full of bias benefitting established research groups (the power holders). Reviewers from those groups will unconsciously favour their approach/culture/way of thinking. There is a lot of scientific literature and evidence about the bias in publishing. University, reputation, nationality, gender, and as you mention: seniority, funding, venue to be published: it all matters and we did’t even reach the scientific contribution yet.

Personally, I have one simple criterion when reviewing papers: does it propel the academic discussion forward? No paper is perfect, nor it has to be. If a paper is a stepping stone or invites to other relevant research, and it does not contain faults, I accept the work.

I see new publishers emerging in other continents. I my view this is a natural response to the bias of the established system. If we continue this way, science will be segregated, I fear.

The difficulty I have with the statement that we seek truth in scolarship is that ‘ truth’ is not defined and (in my view) has many faces.

Answer 6

There is always some bias, but if you have an established set of criteria that you apply across the board then that seems to be fair, only my opinion though.

If all papers/submissions were double blind reviewed then, as you say, it would not be an issue.

Answer 7

Now you have mentioned computer science, I'd like to tell a loosely-related anecdote.

At earlier days I have seen a chess program for a programmable calculator. The calculator had, like, 12 registers and no larger RAM or swap ability. You'd think you need an array of 64 integers to represent a chess board. What they did, was to go for some endspiel problem (so, less distinct chess pieces) and sort of bit packing. They put each row of the board in a single register.

This way they even had enough spare registers to track other required things in a game.

---

Since this is computer science, hardware matters. But those, who do not have access to a more potent hardware, are not lost, standing in the rain. It is possible to word smarter, not harder and to circumvent the limitations of the hardware by more clever programming and better algorithms.

---

But actually, my personal opinion is stated in the comment above. Affiliation and names should play no role in the review process.

Answer 8

Let's start with the premise that it is impossible to be perfectly unbiased as a peer-reviewer, though it is a worthy goal to work towards. Now, given other types of bias prevalent in publishing, this seems benevolent, because ultimately it is oriented towards equal opportunity. So in principle, I'd argue that this is actually a responsible thing to do.

In practice, there are two issues that come to mind:

(1) The reviewer's ability to assess how well funded a group is. A basic assessment could be based on the authors' country, but there are big variations across universities in any country. Not all groups publicly mention their funding status. It doesn't seem possible to make a correct assessment of spending ability based on the little information available.

(2) This type of bias should be like a valve; it should work one way. That is, you shouldn't hold a well funded group to higher standards just because of their funding. That can open doors to all sorts of discrimination which will rapidly spiral into unsavoury. The goal is to give less economically privileged groups a leg-up, not to use privilege against those who have earned it already.

EDIT: On probing different angles of this issue further, I came across this question which is quite similar on the ethics, though it deals with admissions. Insofar as promoting equal opportunity is concerned, I think there are some very good answers, particularly the one by penelope.