Reject a paper based on relevant but pointless research direction

Question

Disclaimer: this is a hypothetical question, not (immediately) related to any concrete past, present or future reviewed paper.

My context is the no-fly zone between Computer Science and Applied Mathematics. In my opinion, there are some research directions within this field that are pointless. Roughly speaking, papers can be either theoretical or applicative. The former provide insight on formal concepts, model interesting physical phenomena, analyze significant properties of an algorithm, and so on. The latter build algorithms to solve real-world applications, or at least provide preliminary results on contrived problems that promise future performance enhancements. In summary, I am very tolerant as to which is my field of research, and I find interesting results almost everywhere.

However, I observe that there appear more and more papers that suffer from the syndrome of the "parameter change": a model—typically an algorithm—is built that can solve toy problems, but no large-scale application is proved or foreseen; there are no fundamental theoretical problems in either Mathematics or Computer Science (or Physics, Biology, or other fields) involved; then, a second paper changes a minor parameter or adds a light generalization, but still no real applicability is shown; then a third paper. Note that I am not talking about marginal papers, I estimate that half the papers of a top journal (ISI indexed, impact factor, serious publisher...) fall within this description.

Now, my particular problem: I am assigned a paper for review. I am a bit prejudiced by the title and the abstract, but nonetheless I bite the bullet and go through pages and pages of equations, crowned by some computer simulations that allegedly prove the superiority of the algorithm (marginal superiority in an uninteresting toy problem). I am unable to point out any particular error, which is hardly surprising, since what is proved is rather obvious, usually some disguised version of the Stone-Weierstrass theorem or the Lyapunov Stability criterion.

What do you think should be my stance? Some possibilities:

1. Next time, think better and decline the review because you are prejudiced and unable to produce a fair review.

2. Well, the paper is correct, isn't it? Judge exclusively by the content.

3. The editor has asked for your opinion and you think that the paper should not be published. Say so: "It is correct but this line of research is pointless. Yes, I know that there are hundreds of papers within this line, but those are not my business and I am currently reviewing this paper".

What about consequences of the somewhat quixotic answer 3? It looks like my opinion is minority and probably not shared with editors themselves. Could I be berated or blacklisted by the editor?

Answer 1

Generally, as a reviewer, you are being asked for your opinion of the paper's correctness and its significance (importance, level of interest, etc). Evidently in this case you think the results are not significant or interesting enough to publish in this journal, so you should say so in your review. If possible, you should explain why you think so: the authors don't give any compelling applications, the results are only a marginal improvement on what's known, etc.

This isn't reason to decline the review. Thinking the paper is uninteresting isn't "prejudice".

If the editors don't agree with your assessment, they are free to disregard it and publish the paper anyway. But you do have a responsibility to give your honest opinion.

In fact, I would suggest that you evaluate the paper's interest before you start to carefully check its correctness. If you believe it's not significant enough to publish, even if correct, then you can so inform the editors and save yourself the time.

There are some exceptions, where the editors may only be interested in whether you believe the paper is correct. Perhaps they are already convinced it is sufficiently important (from their own opinions or other reviewers), or perhaps "importance" isn't a criterion for publication in this journal (e.g. PLOS One). If that's the case, then the review invitation should make this clear; otherwise, do consider the paper's importance as part of your review.

Answer 2

tl;dr: Let the paper try and convince you it's not pointless; if it hasn't, you may judge it harshly.

Referring to your three options:

Decline the review because you are prejudiced and unable to produce a fair review.

You're not prejudiced against the specific paper or specific author, you have an opinion regarding the significance of such papers. But - do make sure you're not pre-judging a specific paper because you've disliked existing ones which are somewhere between theory and practice.

Well, the paper is correct, isn't it? Judge exclusively by the content.

You're "spoiling" this answer by adding an irrelevant rhetorical question. We're talking about relevance and significance, not correctness. Always judge by the content; but - the content must include an argument for significance and relevance if those are not immediately apparent. For that, context is significant.

The editor has asked for your opinion and you think that the paper should not be published. Say so: "It is correct but this line of research is pointless. Yes, I know that there are hundreds of papers within this line, but those are not my business and I am currently reviewing this paper".

Don't say that until you've read the paper and have not been otherwise convinced. Also, again, the "correctness" is not the issue.

The bottom line is summarized in the tl;dr above.

PS - A paper may have a very interesting method to prove a not-very-useful result, and that may be an independent reason to accept.

Answer 3

I would, and have, chosen option 3. However, I phrase it differently. Make your point using neutral terms, so the focus is on your argument rather than your evaluation. That is useful to the editor, regardless of whether (s)he agrees with your points or not. A professional disagreement is no reason to be blacklisted.

Answer 4

You estimate that around half of the papers in one of the top journals fall into this class of "pointless" papers. Honestly, to me that says that your view of what is "pointless" is so far out of line with what the community as a whole thinks, that you shouldn't be making recommendations about papers on that topic.

Answer 5

As an editor, option 3 is exactly what I'd like you to express. As an editor, and as a member of the community, I agree that we publish a lot of papers that are indeed either (i) pointless, or at least (ii) do a rather poor job motivating why anyone should care.

So to ask the question "Why should anyone care?" is really one of the more important jobs of a reviewer. If you think we should care, then you can check technical correctness, but I entirely agree that a pointless paper, technically correct or not, should not be published. It's the editor's job to ask reviewers about this issue as well.

(I will add that I have trouble seeing what the point is about many papers I find as well. But I've also learned that not all papers do a particularly good job explaining what the point is, and that that is not equivalent to there not being a point. That's particularly true for many pure math papers that contain essentially no introduction that puts the result into context. That doesn't mean that there is no context in which the result is not relevant -- it just means that you have to be an expert in the field to see it. I suspect that many papers in applied maths and other areas fall in the same category: You will understand the point when you know enough about the problem, which most of us don't. Of course, this does not refute the fact that there are many papers that truly are pointless.)

Answer 6

It is correct but this line of research is pointless.

Number theory. I agree that the way you described it sounds like really minor results. But on the other hand, I believe minor results were almost always the stepping stones in mathematics. And if you think the "line of research" itself is pointless, I can't help but think of number theory. Was thought of as pointless with toy problems until cryptography.

So, I'm saying you can't really judge if a line of research will always stay pointless or if the toy problems become interesting problems in the future because we can't know the future.

Answer 7

I have asked a very similar question here:

What good is engineering research with no practical relevance?

However, the majority of the answerers chastised me for failing to see the usefulness of "pointless research" (in your own words). If only these answerers had seen the papers I have seen!! I am too polite to post those research paper on here, but when you are using genetic algorithm to allocate fertilizers (in 2017, when there are extremely robust optimization methods), you might be doing something wrong. When you are rediscovering Markov decision process, you are doing something wrong. When you are modeling the purchasing behavior of actual human beings as dynamical system (with zero stochasticity involved), and proposing a Lyapunov function to "show convergence", you are doing something wrong. It is not about the math, but about the assumptions and practical relevance.

For the sake of producing good research, while being guarded against our own biases, the best we can do is to highlight and question (and question repeatedly) the practical relevance of those "pointless research".

We should not be afraid to ask things like: "How can you extend your results? How can it be implemented? what advantage does it bring as compared to seemingly better and more widely used alternatives? Why is the application of your paper so limited? Is there anyway to use your proposed algorithm for a non-toy problem"

or

"why should anyone care?"

This is the only way we can compete against research in industry and safeguard the prestige of the title of an "academic", if not the quality of research in a particular University or even an entire country.

Some countries are known to output enormous amount of pointless research, and I think it has harmed the reputations of researchers from those countries very deeply.

To your question, if you find that the application is too specific, or that the algorithm is too limited, I would recommend you not hold back when deciding whether or not to push the reject button. But definitely discuss this with several peers first, a consensus should quickly be reached if the paper in question is, indeed, pointless.

This is the only way that academia can innovate. I have seen people doing the same thing that they have done 20 years ago, just with minor tweaks in the setup, so that an algorithm appears slightly different than when it was first proposed some 20+ years ago. It is time for a serious change.

Answer 8

When I read your question and came to the passage "pages and pages of equations, crowned ... or the Lyapunov Stability criterion", my reaction was that, if this is an accurate description of the paper, then you should tell the editor exactly that.

Answer 9

I will admit my bias here: I once had a paper dismissed as "a solution in search of a problem" by a reviewer (essentially the same as 'this is pointless') when:

1. That paper was an integral component of a larger project
2. That paper has been cited 33 times, and is well above my personal h-index.

So I'm a little skeptical of people's ability to evaluate what is useful.

Personally, I tend to review papers based on their technical merits - for me to reject something, it has to be badly done, outright wrong, "This does not say what you think it says", etc. rather than just "I don't like it." I may however leave the authors a comment that the results need firmer linking to some practical outcome (I'm in a field where practical outcomes matter, and 'Is interesting for its own sake' is rarely accepted).

In the comments to the editor section I would note that I don't think the paper will have much of an impact, and is one of a number of examples of the field going down a useless rabbit hole. But I'd leave it to their editorial judgement to determine if they think that's grounds to not accept something.

Answer 10

The referee assignment is offered in the context that you know what is acceptable and appropriate for that journal. If one believes that half of what's published in that journal is not appropriate because of the area the work is in, then such person is not an appropriate referee for that area for that journal. The OP should decline to review such papers.

Answer 11

I would like to offer a slightly contrarian answer, possibly controversial, but do give it a thought.

You are perfectly entitled to your evaluation of the manuscript; after all you would not be contacted if you don't have the appropriate credentials. You are also right to say that this paper is your business and not other similar papers. I urge you to go one step beyond, and consider the impact of your decision.

As a responsible academician, it may be good to think if a 'reject' decision is likely to change the number of "parameter change" papers being published. Unlikely- the editor is more likely to treat it as an isolated case. It certainly raises one voice, but if as you say, 50% of papers published are similar, that voice will be drowned out.

Now consider the effect on the author(s)- they have seen many similar papers getting published in the same journal. It may even be the case that they chose this journal for that very reason. To them, this would appear an unfair and biased decision. This does not help their morale, and for a young researcher, could be demotivating. It is a bit unfair for them to be singled out. (Yes, the review process is probabilistic and inequalities are inevitable, but perhaps we could do our bit to reduce those, when we can.) When the researcher is demotivated, (s)he is less likely to understand nuanced arguments (eg. relevant but pointless) and take them in the right spirit. The paper will probably be subjected to minor modifications and sent to a similar journal, where it may well get published. You, as a reviewer, have little chance of changing the author's research tendencies in isolation. In the end, you will have the satisfaction of having stated your point (very important in science!), but beyond that, whom or what did it serve?

To my mind, an alternative would be to either review this paper (rigorously) based on its relevance and correctness (leaving relevance aside), or to decline the review. Certainly how interesting a paper is should factor in a regular review- but given your strong feelings about this journal/type of paper, this is not really a regular review. I reiterate- I don't believe this is how every review should be done, this is a special case.

If you are serious in your objection to the type of papers being published, you should instead use you academic standing to write to the editorial board, bring up this issue in conferences, or take a stand in public fora. At the very least, the conversation will move beyond you, one editor and one author. You might find that many others agree with you, and that could be a step towards making a really meaningful contribution. Even if that doesn't happen immediately/visibly, you will know you attacked the root cause, not the symptom.

Answer 12

This is a great question. I've been on the "other end" of the situation - I've submitted my paper which "could" solve a real-world problem, but the paper only did a simulation on a toy/contrived problem.

Problem statement: Running all regression tests can be really time-consuming for a team/organization/company. If we could schedule the tests from the lens of game theory by looking at testability as a competition between testers and developers, we could get superior scheduling as opposed to "any" kind of prioritization scheme one could come up with.

This is really really hard to do on a real system with real people with real tests. In fact, we don't even have data on the "success" of tests even if we use open source software (i.e., which tests were the most successful at identifying problems/bugs).

The best we could do in this situation, is abstract out everything about the "real world" and only focus on the algorithm at hand. We tried to model all known prioritization algorithms similarly so that a simulation could give us some idea into its effectiveness.

We got comments on the lines you suggest:

...I (reviewer) am okay with any direction of research, but this is so far-fetched that the authors are imagining this problem in their head when none exists. There is no evidence of any competition between testers and developers in reality and this is not even applicable in the wild.

This caught us off guard - apparently, the reviewer(s) focused more on the "formulation" of the problem than what we were proposing. Our paper's premise was to put forth a better way of prioritizing and scheduling regression tests so as to optimize the chance of finding bugs by randomizing the tests w.r.t. game theoretic models. That is if we can only run 10% of the tests, which to run (randomly) that would maximize our chances of finding bugs at some X% level of confidence.

What I'm trying to say is, it'd have been really nice if some of the reviewers would have just taken a step back and evaluated the paper from the problem's POV vs. everything else. Some reviewers thought it was pointless - we thought it was a great new POV that never existed before and hence we wrote a paper to put the idea out there. Few got quite a few folks who liked the idea but many just didn't get it (or were swayed by their prejudice).

Hence I strongly urge you to see the actual problem being solved - is that an interesting problem? Can it be solved any other way? What's better about "their" way? Can you, the reviewer, even imagine a possible application? Does the paper call it out? What are the merits of that approach?

If the paper holds up to the question above, then it may be a candidate for acceptance vs. being a pointless research direction. Note, the folks writing the paper are more deeply set into the problem and have some idea of its applicability (or not at all, and the reviewer should help with that).