Should I review papers according to my conviction about correct methodology or according to the usual methodology applied in published papers?

Question

If you disagree with most of the research community about some methodological questions**, how should you act as a reviewer of a paper?

Should you review as most reviewers would? Or should you follow what you think is scientifically correct?

More details on the disagreement. The typical paper in my field is something like this: An algorithm is described (for solving an optimization problem), the algorithm is tested on one of the four famous data sets, the results are reported and discussed (in the sense that: "if there are more than 4 required hubs and the number of sources and sinks is over 100, then my algorithm is 50% faster, otherwise not"). So I have concerns that the algorithms are overtuned to the famous data set, and that they might be less strong if one applied them to data sets of different structure.

Answer 1

I would approach the issue with an open outcome: Don’t demand that the authors do X, but ask them why they don’t do X and request at least a justification for not doing X (in the manuscript). For an issue like in-sample optimisation, you can possibly refer to some suitable paper, that while not from your field is sufficiently general to apply to your field.

You will probably not achieve that the authors actually do X, but you will likely achieve that the authors write a sentence (about why they don’t do X) that is not very convincing or comfortable and reduces the value of the manuscript. This also raises awareness of the issue for the readers and maybe for their next work, the authors will consider improving on this very aspect. And of course there is the possibility that the authors can bring a good reason for not doing X, in which you will learn something (and they should likely bring the argument in the manuscript).

Answer 2

As a peer reviewer, you should offer your own assessment of the manuscript at hand. This way, you contribute to and possibly advance the field. If you just followed what others did common mistakes would just propagate, and this would be undesirable, of course.

However, do not expect that others must follow your advice. It is also conceivable that you are wrong, always consider this as a possibility. Editors and other reviewers might come to a different conclusion than you. (Leaving your example aside)

Answer 3

If you are willing to put your differences/reasoning in writing to an editor then there is no reason not to judge the paper according to the principles that you think are important and correct.

I question, however, whether it is a dominant view that the training and test data can be the same. Separate random selections from a large enough set would be a different matter.

But some "views" diverge into crankery so you need to be willing to justify your own.

Answer 4

Separating test and training data is necessary to confirm that the algorithm can predict unseen data. If the claim is made that the algorithm, presented with the parameters they used, can predict unseen data, then they need to split the dataset. If that claim is not being made, the case can still be made for it, but it's more nuanced than 'YOU MUST DO THIS FOR EVERY ALGORITHM'.

In my opinion, it's all in what claims are being made of the algorithm. If you feel that the claims being made are not supported by the methods, it's your choice to recommend they either:

1. dial down their claims so that they reflect the evidence in the paper
2. perform more analysis to support their claims

It's fine to give the authors the option

Answer 5

Should you review as most reviewers would? Or should you follow what you think is scientifically correct?

I'm not convinced this is a binary choice as you describe it. A good review probably isn't either of these.

Your review is sought, to check the papers adequacy and caliber. As part of that, if they wanted "most" reviewers views, the editors are more than able to ask other more typical reviewers, or a wider range of reviewers, to review it. Therefore you should assume they actually want your view, not a mere echo of what you perceive to be a popular view.

But giving your view need not be idiosyncratic or fringe-y either.

• If the field's methodology is weak or flawed, there are, presumably, reputable papers that say this. You can allude to those papers, and state that you have concerns that the paper is prone to/may have suffered from the weaknesses X, Y and Z, as described in [list of cites], because [reasons]. You ask the authors to address these concerns as usual.

• If there are no such papers, or they are not seen as significant in the field (or overlooked), then describe that you see a possible concern that the algorithm may be so tuned to the specific data, that it is unclear if it has general interest, because the authors do not appear to show its performance against appropriate general data. (And if needed: the data they do test against cannot in your view be considered appropriate as a test with general data, because (reasons), notwithstanding that it is a widely used dataset). That too, is a sensible, professional statement.

Really, your review is to state what, in your view, needs to be addressed, in order that the paper become acceptable, adequate, and professionally worthwhile publishing (if not already so). So you are not advocating a view, so much as identifying possible inadequacies (as you personally yourself feel they may exist), that you convey to the authors so they can address them - subject to the editors overriding judgement on the need for this and willingness to publish.

Because you are recounting possible issues to check, rather than advocating a position, you should find that there is a wording that allows you to state you have a concern, without adopting a fringe-y position in doing so.

Answer 6

So you are saying, what to do with papers which follow best practices of your field, but you think (with good reason) that these best practices are not very good or insufficient. I think trying to change the community by rejecting papers/forcing authors to follow your ideas is not the way to do it; especially since it is quite random which papers reach your desk. So you should:

• Do the peer review following best practices of your field without pushing your personal agenda; you might mention your concern, but not enforce it; maybe the seed will grow in the researcher

• If you think you cannot do this, then do not accept peer review requests

• If you want to change the community: publish papers on that subject, show what has been overlooked, show how crappy some of the celebrated algorithms are on other data, hammer the message home at every conference. Also offer a viable path for research in your field continue, i.e. positive messages, not 'dont do...' but 'do a, b, c, d)

Edit: To clarify: It is not the right way to change a scientific community by imposing standards on publication when you are the only one seeing a problem there. The right approach is to change it through scientific discussion. Peer review is not (well it is a little) scientific discussion, worst it is not public and there is a power imbalance.

Apart from that not being the way, it is also inefficient.