47

If you disagree with most of the research community about some methodological questions**, how should you act as a reviewer of a paper?

Should you review as most reviewers would? Or should you follow what you think is scientifically correct?

More details on the disagreement. The typical paper in my field is something like this: An algorithm is described (for solving an optimization problem), the algorithm is tested on one of the four famous data sets, the results are reported and discussed (in the sense that: "if there are more than 4 required hubs and the number of sources and sinks is over 100, then my algorithm is 50% faster, otherwise not"). So I have concerns that the algorithms are overtuned to the famous data set, and that they might be less strong if one applied them to data sets of different structure.

6
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – Massimo Ortolano
    Mar 22 at 8:12
  • 2
    In real world applications, over-tuning algorithms to datasets is a growing problem that has seen a lot of criticism in recent years. Whatever your academic community's current stance is on this topic, you should bring it up in this particular case since the kinds of people who may be using this research for practical applications are likely to question such methods, even if the academic community does not.
    – Nosajimiki
    Mar 23 at 16:18
  • Your wording, using "beliefs", rather than reference to things that might be "objective" or at least "falsifiable/verifiable" certainly weakens the rhetorical sense of the question... After all, you have tangible, explainable reasons for your "belief". Reword it? Mar 23 at 17:19
  • 1
    @paulgarrett These are not my words, the word "belief" came in through the edits of other people. Mar 23 at 17:31
  • 1
    Hm. Not good edits. Oh, well, nevermind, ... Mar 23 at 18:27

6 Answers 6

42

I would approach the issue with an open outcome: Don’t demand that the authors do X, but ask them why they don’t do X and request at least a justification for not doing X (in the manuscript). For an issue like in-sample optimisation, you can possibly refer to some suitable paper, that while not from your field is sufficiently general to apply to your field.

You will probably not achieve that the authors actually do X, but you will likely achieve that the authors write a sentence (about why they don’t do X) that is not very convincing or comfortable and reduces the value of the manuscript. This also raises awareness of the issue for the readers and maybe for their next work, the authors will consider improving on this very aspect. And of course there is the possibility that the authors can bring a good reason for not doing X, in which you will learn something (and they should likely bring the argument in the manuscript).

51

As a peer reviewer, you should offer your own assessment of the manuscript at hand. This way, you contribute to and possibly advance the field. If you just followed what others did common mistakes would just propagate, and this would be undesirable, of course.

However, do not expect that others must follow your advice. It is also conceivable that you are wrong, always consider this as a possibility. Editors and other reviewers might come to a different conclusion than you. (Leaving your example aside)

19

If you are willing to put your differences/reasoning in writing to an editor then there is no reason not to judge the paper according to the principles that you think are important and correct.

I question, however, whether it is a dominant view that the training and test data can be the same. Separate random selections from a large enough set would be a different matter.

But some "views" diverge into crankery so you need to be willing to justify your own.

14
  • 1
    To answer your question: In hub location papers (an area in discrete optimization), the results of the optimization algorithms are always reported for two or three data sets from the literature. Algorithms are considered strong if they perform well on those data sets, containing less than a hundred test cases each. I have never seen a paper mentioning training data. Mar 20 at 11:57
  • 23
    "I have never seen anybody separating the data set into training vs. test data or even discussing this aspect at all. " There you go, a low-hanging fruit ready to be picked up by you, write that paper and discuss the issue!
    – EarlGrey
    Mar 20 at 21:16
  • 6
    @JFabianMeier "they are chosen by theoretical arguments or by preliminary tests on the test data set", these are two very different situations. In the former case, why do you think using the whole data for testing is an issue?
    – Luca Citi
    Mar 21 at 3:58
  • 8
    "they are chosen by theoretical arguments" in this specific case, then they're "training" it on a separate data: the theoretical framework, and in this case there is no need for further separation of data, since they don't train anything. On the other one "preliminary tests on the test data", this might warrant more details. But given your comment so far, perhaps try to search for the justification before claiming that there is something fishy on-going in the whole community (it's possible, but you shouldn't rule out that you misunderstood the field as well)
    – justhalf
    Mar 21 at 10:26
  • 2
    "it is discussed (without experiment) that some parameter settings are more plausible than others" This is a big red flag to me - not uncommon for people to do the experimentation and then write up the parameter choice finding some supporting arguments. Of course, that is fundamentally incompatible with the scientific method but there are researchers who do not even see an issue with that...
    – Lodinn
    Mar 22 at 12:16
9

Separating test and training data is necessary to confirm that the algorithm can predict unseen data. If the claim is made that the algorithm, presented with the parameters they used, can predict unseen data, then they need to split the dataset. If that claim is not being made, the case can still be made for it, but it's more nuanced than 'YOU MUST DO THIS FOR EVERY ALGORITHM'.

In my opinion, it's all in what claims are being made of the algorithm. If you feel that the claims being made are not supported by the methods, it's your choice to recommend they either:

  1. dial down their claims so that they reflect the evidence in the paper
  2. perform more analysis to support their claims

It's fine to give the authors the option

2
  • 2
    This would require that the papers state a hypothesis and try to find evidence for and against it. Papers are usually written in a way that they define an algorithm, make numerical tests and then discuss the tables. Mar 21 at 11:22
  • 1
    And it depends on what is discussed as to whether the separation is important. If the discussion says 'the algorithm fits the data' that's fine and is factually accurate (but maybe a note should be added in limitations). If the discussion says 'the algorithm will predict unseen data' then they have not demonstrated that and that claim will need to be proven or removed.
    – E. Rei
    Mar 21 at 15:48
4

Should you review as most reviewers would? Or should you follow what you think is scientifically correct?

I'm not convinced this is a binary choice as you describe it. A good review probably isn't either of these.

Your review is sought, to check the papers adequacy and caliber. As part of that, if they wanted "most" reviewers views, the editors are more than able to ask other more typical reviewers, or a wider range of reviewers, to review it. Therefore you should assume they actually want your view, not a mere echo of what you perceive to be a popular view.

But giving your view need not be idiosyncratic or fringe-y either.

  • If the field's methodology is weak or flawed, there are, presumably, reputable papers that say this. You can allude to those papers, and state that you have concerns that the paper is prone to/may have suffered from the weaknesses X, Y and Z, as described in [list of cites], because [reasons]. You ask the authors to address these concerns as usual.

  • If there are no such papers, or they are not seen as significant in the field (or overlooked), then describe that you see a possible concern that the algorithm may be so tuned to the specific data, that it is unclear if it has general interest, because the authors do not appear to show its performance against appropriate general data. (And if needed: the data they do test against cannot in your view be considered appropriate as a test with general data, because (reasons), notwithstanding that it is a widely used dataset). That too, is a sensible, professional statement.

Really, your review is to state what, in your view, needs to be addressed, in order that the paper become acceptable, adequate, and professionally worthwhile publishing (if not already so). So you are not advocating a view, so much as identifying possible inadequacies (as you personally yourself feel they may exist), that you convey to the authors so they can address them - subject to the editors overriding judgement on the need for this and willingness to publish.

Because you are recounting possible issues to check, rather than advocating a position, you should find that there is a wording that allows you to state you have a concern, without adopting a fringe-y position in doing so.

3
  • Re: "if there are no such papers", you might consider writing one.
    – kaya3
    Mar 23 at 17:01
  • Not relevant for this, wrong timescale. That's a different thing for future.
    – Stilez
    Mar 23 at 22:19
  • Obviously it is a different thing for the future, but it is still something OP should consider doing.
    – kaya3
    Mar 24 at 6:39
-1

So you are saying, what to do with papers which follow best practices of your field, but you think (with good reason) that these best practices are not very good or insufficient. I think trying to change the community by rejecting papers/forcing authors to follow your ideas is not the way to do it; especially since it is quite random which papers reach your desk. So you should:

  • Do the peer review following best practices of your field without pushing your personal agenda; you might mention your concern, but not enforce it; maybe the seed will grow in the researcher

  • If you think you cannot do this, then do not accept peer review requests

  • If you want to change the community: publish papers on that subject, show what has been overlooked, show how crappy some of the celebrated algorithms are on other data, hammer the message home at every conference. Also offer a viable path for research in your field continue, i.e. positive messages, not 'dont do...' but 'do a, b, c, d)

Edit: To clarify: It is not the right way to change a scientific community by imposing standards on publication when you are the only one seeing a problem there. The right approach is to change it through scientific discussion. Peer review is not (well it is a little) scientific discussion, worst it is not public and there is a power imbalance.

Apart from that not being the way, it is also inefficient.

2
  • 3
    "papers which follow best practises of your field" - you mean worst practises? ;-) Mar 22 at 19:47
  • 2
    You can state in your review that you are taking a contrarian position to most in your field and explain why you think the authors should change their manuscript, and let the editor decide. It is always up to the editor anyways. Mar 23 at 4:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .