22

Let me say beforehand that this question does not refer to scientific correctness and treats it as undisputed, i.e. the paper is assumed correct in the sense that it clearly states a valid purpose, has an adequate literature review, the theory and methods are scientifically sound.

After a brief introspective, it came to me that I often take into the account under what circumstances a paper was written when assessing the scope / prestige of the venue it is to be published at. This usually includes funding, if any, seniority and number of the authors.

On the one hand, I find it unfair to expect the same scale of experimental devotion from a student-mentor author pair from a developing country with no or relatively little funding as from a research group with multiple grants from a developed country. Given the relevance of the contribution, the first group shouldn't be rejected on the grounds that they don't have a comparable abundance of resources. Also, they shouldn't be discouraged as they managed to make a difference without those resources. And, after all, we do this kind of expectation management in other aspects of life, e.g. one doesn't fail a student because they wrote the assignment in a free text editor, because they can't afford MS Word.

As a concrete example, I once saw another reviewer comment that they would like to see a more significant average deviation on the graphs. This requirement would've required the authors to run tens of multi-hour simulations on a cloud cluster consisting of ~50 high-end instances, each billed by the hour, and it is also rather irrelevant to the proof or the contribution itself. The two authors received a government grant which translated to roughly $1200 for this research at an university that doesn't have subsidies with the cloud provider.

On the other hand, this is clearly a bias. Augmented additionally by the fact that it is not possible to implement during a double-blind review. Also, somewhere in there is an argument how practices to include low-cost research would result in research funding being gradually cut in some way or the other.

So, my question is the dilemma whether it is unfair to let capabilities outside of a research group's direct influence contribute to the review of their paper (again, only once the contribution is established)?

My field is computer science, so, while not strictly pen-and-paper, few resources are needed, relative to other fields, e.g. experimental physics, bio-medicine, etc., to do meaningful research.

closed as primarily opinion-based by Brian Borchers, Enthusiastic Engineer, rturnbull, E.P., Alexandros Jul 30 '18 at 9:38

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    what is fair? of course nothing is fair. i dont even understand why people use fairness as a guiding idea. fair is a state of mind. to your question, the group with more resources has better capacity to push the knowledge base, and that is objectively optimal. i personally consider objectively optimal to be 'fair'. – user96140 Jul 27 '18 at 18:01
  • 1
    Wouldn't it be the best course just to tape over the authors and affiliations and then to review the paper as is? – Oleg Lobachev Jul 28 '18 at 12:34
  • We're assuming this is a "standard" venue, right? There's certainly venues for research with special conditions, e.g. undergrad conference/journals, conferences for research in a specific area of the world, conferences for non-research universities, etc. I would expect less rigor and resources in a undergrad conference than a standard one. – user71659 Jul 28 '18 at 20:17
20

Unfortunately you seem to be specifically asking for opinions which isn't especially valued here. However, I'll give my opinion on this.

A scientific paper (or any scholarly paper) should stand on its own. Either it properly makes a case or it does not. The resources involved in establishing truth should not be a factor, other than to say WOW in some cases. We should neither value nor devalue a work of scholarship based on who did it or how.

The case you mention is instructive. Asking for better significance of results is completely valid. With low significance the likelihood that the results are invalid is increased. Who knows what would happen if more resources were poured into it other than with more data (and a good design) the confidence in the results is increased.

However, in cases in which researchers from places with few resources achieve good results, it is the people themselves who should be especially honored, but not the paper itself. Again, you can say WOW, but the paper stands for itself. We seek truth in scholarship, nothing more or less.

  • 2
    This might be a good meta discussion. I consider all ethics questions to be more or less opinion based. It is also my opinion that this question is not significantly different from other ethics-tagged questions and that it can be constructively answered (as your answer shows, among others). If you however do have some concrete suggestions how to improve the question, feel free to comment or edit. – user3209815 Jul 27 '18 at 13:36
  • 3
    I enjoyed reading this answer, but have two observations: (1) 'Who knows what would happen if more resources were poured into it' could extend indefinitely, as could data collection. Someone with more resources may be familiar with larger-than-necessary datasets and insist on the same. Who gets to draw the line? – user153812 Jul 27 '18 at 17:27
  • 2
    (2) "The resources involved in establishing truth..."is ironically and unintentionally very true. In English, 'truth' is derived from religious terms - from faith, loyalty, steadfastness, and not from 'factual' or 'scientific' terms. Sadly, the availability of resources has very much decided 'faith', historically. Science is not immune from this, since it's the same humans working on it. The 'fact' is that we live in an unequal world and what is routine for one is a luxury for another. – user153812 Jul 27 '18 at 17:39
  • @user153812, the individual researcher has to draw the line. Resources are always finite. Larger studies are generally better, but it may be impossible to achieve. Research, however, need to be repeatable, so that spurious results (which are always possible in statistical studies) can be corrected. At the 95% confidence level, 5% will give the wrong answer. And even that assumes true randomness. Moreover, some questions are so important to answer now that time won't permit a fuller answer as action needs to be taken to, for example, save lives in the near term. A better answer has to wait. – Buffy Jul 27 '18 at 17:47
  • 3
    Precisely. The individual researcher does decide; so unless it is a clearly non-representative datset, reviewers should not object (because they use larger datasets themselves). That is the point OP is trying to make- for some researchers the objection is a minor annoyance, for others it could be near-impossible to resolve. – user153812 Jul 27 '18 at 18:02
20

No, your review should not change based on knowledge of the authors' circumstances. Your role is to give the editor an objective assessment of the strengths and weaknesses of the author's submitted manuscript. However, you can still help less advantaged authors. Write your review to focus on the strengths and weaknesses of their work, and leave the authors to choose how they wish to address these.

To take your example: suppose that the authors have developed a new algorithm for X, which you consider to be a worthwhile contribution to the field. However, they go on to claim that their algorithm outperforms the current standard, based on only one test-case.

An unhelpful reviewer might write: "This paper has potential, but it cannot be accepted unless the authors run more test-cases". A more helpful review would be: "The algorithm in this paper appears to be a useful contribution, and it is deserving of publication. However, I do not feel the authors currently have sufficient evidence to support their unqualified claims regarding performance, and this should be addressed before acceptance."

By doing this, you leave the authors with a choice: they can either run more experiments, or they can put caveats around their discussion of performance. (Of course, you should only do this if you feel the paper is publishable in such a form.) A wealthy research group will likely choose the former option, a less well-off one might settle for the latter. However, both have been held to the same standard of scientific correctness.

  • 1
    This is an excellent answer, I think the proposed solution satisfies everything. – user153812 Jul 28 '18 at 6:29
8

Let me answer this question by other questions:

Would you like to live in a world where pharmaceutic products which the population is administered is based on studies in which a reviewer accepts that given the resources the researchers had at hand it was unrealistic that they could make a clear line between "drug is working or not harmful" to "drug is not working or harmful" can not be drawn, and therefore we accept that we regularly take unproven results as proven?

Would you like to live in a world where simulation software contains simplifications which work "usually" and not "most of the time" when predicting if a structure is stable?

Would you like to live a world where it is ok to accept an algorithm for a self driving car even if it not proven up to the full level of required credibility because the authors could not afford to test it properly?

So, no, sometimes lives, health, well-being and economic progress depend on science being right. We should not just do it sloppy.

  • 1
    The first paragraph of the question: "Let me say beforehand that this question does not refer to scientific correctness and treats it as undisputed, i.e. the paper is assumed correct in the sense that it clearly states a valid purpose, has an adequate literature review, the theory and methods are scientifically sound.". It's extremely aggravating to have to read an answer that has ignored the carefully provided proviso in the question. Nowhere has it been suggested that sloppy science should ever be accepted. – Araucaria Oct 19 '18 at 8:43
4

So, my question is the dilemma whether it is unfair to let capabilities outside of a research group's direct influence contribute to the review of their paper (again, only once the contribution is established)?

Is it fair that there are people suffering from war, poverty, starvation and all kinds of other serious problems in various parts of the world as I sit in my comfortable chair typing this answer, free to think about academic matters? No, in all honesty it strikes me as deeply unfair. Perhaps I should drop everything else and dedicate my life to correcting all the injustices out there. But, recognizing that both my ability to fix the world’s problems and my willingness to do so are limited, instead I have taken it upon myself to write this answer in this small region of space-time that I currently occupy, and I will try to do the best job of it that I can.

The same principle applies to the situation you ask about. When I review a paper, I take on a responsibility to provide the most honest, competent and professional review I can. Thus, the standards of review I apply must be the same ones that I would want to see applied to my own work, and the review must be as unbiased and uninfluenced by irrelevant factors as I can humanly (and consciously) manage to make it. To do anything less would be to betray the trust of the people who chose to assign me the review and of the journal’s readers who expect papers of a certain quality. I can’t fix all the world’s problems, nor do I have a responsibility to do so, but the one thing I do have a responsibility for is to do the job that I have committed to doing as well and impartially as I can.

The conclusion is obvious. It is unfair that researchers from some parts of the world are constrained by a serious lack of resources that prevents them from doing good research as well as equally or less talented researchers in richer countries. But as a reviewer, it would be wrong for you to “grade on a curve” and set aside your usual standards for researchers who you feel are disadvantaged in some way. Your desire to give those researchers a little “help” is well-intentioned and even admirable, but you must resist that temptation; to make the world a better place, start by doing your job as professionally as you can. Of course, if you still have time and energy left over after that, there are plenty of things you can do try to help researchers from poor countries, so certainly that should be something to consider as well.

1

Thank you for raising this issue. I have the same sort of questions.

Pondering over your question, there is another bias you don’t mention: overcalibation. When a reviewer asks for another experiment, the reviewer should ask him/herself first, whether this really contributes to the scope and purpose of the article.

Second, the review process is full of bias benefitting established research groups (the power holders). Reviewers from those groups will unconsciously favour their approach/culture/way of thinking. There is a lot of scientific literature and evidence about the bias in publishing. University, reputation, nationality, gender, and as you mention: seniority, funding, venue to be published: it all matters and we did’t even reach the scientific contribution yet.

Personally, I have one simple criterion when reviewing papers: does it propel the academic discussion forward? No paper is perfect, nor it has to be. If a paper is a stepping stone or invites to other relevant research, and it does not contain faults, I accept the work.

I see new publishers emerging in other continents. I my view this is a natural response to the bias of the established system. If we continue this way, science will be segregated, I fear.

The difficulty I have with the statement that we seek truth in scolarship is that ‘ truth’ is not defined and (in my view) has many faces.

0

There is always some bias, but if you have an established set of criteria that you apply across the board then that seems to be fair, only my opinion though.

If all papers/submissions were double blind reviewed then, as you say, it would not be an issue.

-1

Now you have mentioned computer science, I'd like to tell a loosely-related anecdote.

At earlier days I have seen a chess program for a programmable calculator. The calculator had, like, 12 registers and no larger RAM or swap ability. You'd think you need an array of 64 integers to represent a chess board. What they did, was to go for some endspiel problem (so, less distinct chess pieces) and sort of bit packing. They put each row of the board in a single register.

This way they even had enough spare registers to track other required things in a game.


Since this is computer science, hardware matters. But those, who do not have access to a more potent hardware, are not lost, standing in the rain. It is possible to word smarter, not harder and to circumvent the limitations of the hardware by more clever programming and better algorithms.


But actually, my personal opinion is stated in the comment above. Affiliation and names should play no role in the review process.

  • 1
    This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review – user153812 Jul 28 '18 at 14:37
  • I understand that the first part of the answer is a bit far-fetched (and it probably won't fit into a comment). But the second part actually provides an answer. "Is it unfair for paper circumstances to play a role in the review of the paper?" – "Yes, it is." – Oleg Lobachev Jul 28 '18 at 15:50
-1

Let's start with the premise that it is impossible to be perfectly unbiased as a peer-reviewer, though it is a worthy goal to work towards. Now, given other types of bias prevalent in publishing, this seems benevolent, because ultimately it is oriented towards equal opportunity. So in principle, I'd argue that this is actually a responsible thing to do.

In practice, there are two issues that come to mind:

(1) The reviewer's ability to assess how well funded a group is. A basic assessment could be based on the authors' country, but there are big variations across universities in any country. Not all groups publicly mention their funding status. It doesn't seem possible to make a correct assessment of spending ability based on the little information available.

(2) This type of bias should be like a valve; it should work one way. That is, you shouldn't hold a well funded group to higher standards just because of their funding. That can open doors to all sorts of discrimination which will rapidly spiral into unsavoury. The goal is to give less economically privileged groups a leg-up, not to use privilege against those who have earned it already.

EDIT: On probing different angles of this issue further, I came across this question which is quite similar on the ethics, though it deals with admissions. Insofar as promoting equal opportunity is concerned, I think there are some very good answers, particularly the one by penelope.

  • Regarding this seems benevolent, because ultimately it is oriented towards equal opportunity, imho as Buffy wrote We seek truth in scholarship, nothing more or less. and therefore all other considerations should be secondary. Imho equal opportunity can play a role in the grant of research funds but not when judging the result. This is why I downvoted. – problemofficer Jul 27 '18 at 13:36
  • @problemofficer - First, thanks for taking time to explain your point, really appreciate it. Nevertheless I must disagree on two counts: (1) OP is amply clear that the dilemna in question does not involve any compromise in the veracity of results(first sentence of question). So anything over that is essentially frills; and not being able to afford said frills shouldn't detract from the 'truth'. Just a less pretty way to state the truth. (2) 'All other considerations should be secondary' is fine with me, so long as it is a consideration. I don't think I conveyed anything to the contrary. – user153812 Jul 27 '18 at 17:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.