61

Disclaimer: this is a hypothetical question, not (immediately) related to any concrete past, present or future reviewed paper.

My context is the no man's land between Computer Science and Applied Mathematics. In my opinion, there are some research directions within this field that are pointless. Roughly speaking, papers can be either theoretical or applicative. The former provide insight on formal concepts, model interesting physical phenomena, analyze significant properties of an algorithm, and so on. The latter build algorithms to solve real-world applications, or at least provide preliminary results on contrived problems that promise future performance enhancements. In summary, I am very tolerant as to which is my field of research, and I find interesting results almost everywhere.

However, I observe that there appear more and more papers that suffer from the syndrome of the "parameter change": a model—typically an algorithm—is built that can solve toy problems, but no large-scale application is proved or foreseen; there are no fundamental theoretical problems in either Mathematics or Computer Science (or Physics, Biology, or other fields) involved; then, a second paper changes a minor parameter or adds a light generalization, but still no real applicability is shown; then a third paper. Note that I am not talking about marginal papers, I estimate that half the papers of a top journal (ISI indexed, impact factor, serious publisher...) fall within this description.

Now, my particular problem: I am assigned a paper for review. I am a bit prejudiced by the title and the abstract, but nonetheless I bite the bullet and go through pages and pages of equations, crowned by some computer simulations that allegedly prove the superiority of the algorithm (marginal superiority in an uninteresting toy problem). I am unable to point out any particular error, which is hardly surprising, since what is proved is rather obvious, usually some disguised version of the Stone-Weierstrass theorem or the Lyapunov Stability criterion.

What do you think should be my stance? Some possibilities:

  1. Next time, think better and decline the review because you are prejudiced and unable to produce a fair review.

  2. Well, the paper is correct, isn't it? Judge exclusively by the content.

  3. The editor has asked for your opinion and you think that the paper should not be published. Say so: "It is correct but this line of research is pointless. Yes, I know that there are hundreds of papers within this line, but those are not my business and I am currently reviewing this paper".

What about consequences of the somewhat quixotic answer 3? It looks like my opinion is minority and probably not shared with editors themselves. Could I be berated or blacklisted by the editor?

  • 3
    Re: #3 "true but useless" is a phrasing I personally sometimes use. Re: Last line, "blacklisted" may be the word you want. – Daniel R. Collins Mar 3 '18 at 14:23
  • 2
    In chemistry it is pretty common for the reviewers to ask for additional experiments showing an application of the presented method within a major revision. Would this be a possibility in this field too? – DSVA Mar 4 '18 at 2:53
  • 7
    #3 is my preferred option. If they did not want your opinion they should not have asked for it. If they blacklist you for expressing your solicited opinion, then you are better off without them. – emory Mar 4 '18 at 3:32
  • 10
    You might be interested in a similar problem from the metaheuristic community, well documented in this paper. – Gumeo Mar 5 '18 at 7:44
  • 6
    It sounds like you are describing the trend towards researchers producing "least publishable units" in reaction to the publish-or-perish, or a quantity-published as a proxy for quality metric, mentality of some academic circles. – Kelly S. French Mar 5 '18 at 18:20

12 Answers 12

74

Generally, as a reviewer, you are being asked for your opinion of the paper's correctness and its significance (importance, level of interest, etc). Evidently in this case you think the results are not significant or interesting enough to publish in this journal, so you should say so in your review. If possible, you should explain why you think so: the authors don't give any compelling applications, the results are only a marginal improvement on what's known, etc.

This isn't reason to decline the review. Thinking the paper is uninteresting isn't "prejudice".

If the editors don't agree with your assessment, they are free to disregard it and publish the paper anyway. But you do have a responsibility to give your honest opinion.

In fact, I would suggest that you evaluate the paper's interest before you start to carefully check its correctness. If you believe it's not significant enough to publish, even if correct, then you can so inform the editors and save yourself the time.

There are some exceptions, where the editors may only be interested in whether you believe the paper is correct. Perhaps they are already convinced it is sufficiently important (from their own opinions or other reviewers), or perhaps "importance" isn't a criterion for publication in this journal (e.g. PLOS One). If that's the case, then the review invitation should make this clear; otherwise, do consider the paper's importance as part of your review.

  • 6
    +1 for "save yourself the time". The expectation of this advice was a more or less conscious reason for posting. – Miguel Mar 3 '18 at 21:08
  • 14
    "Evidently in this case you think the results are not significant or interesting enough to publish in this journal" When the asker claims to think that about 50% of what the journal publishes, I'd say that they aren't a very good judge of what is important or interesting enough for this particular journal. – David Richerby Mar 4 '18 at 0:49
  • 1
    @DavidRicherby Or maybe the community in general is all too interested in working on problems it finds interesting, not necessary problems that are important or interesting to other people. Case in point: String Theory. – 101010111100 Mar 4 '18 at 8:54
  • 1
    @101010111100 When a community is large enough, they are the “other people”. I don’t try to make my computer science papers interesting to string theorists (or earth scientists, or economists, or historians or...) and it would be unreasonable to demand that they make their papers interesting to me. – David Richerby Mar 4 '18 at 10:45
  • 10
    @DavidRicherby: The reviewer has a professional opinion of what's important, which apparently differs from the mainstream. I wouldn't say that this ipso facto makes them a "bad judge"; reasonable experts can disagree. There's perhaps a separate question here: should one judge the importance of a paper by one's own professional opinion, or by one's impression of the community consensus? I would argue the former, because otherwise how will dissenting views ever be heard? – Nate Eldredge Mar 4 '18 at 16:51
26

tl;dr: Let the paper try and convince you it's not pointless; if it hasn't, you may judge it harshly.

Referring to your three options:

Decline the review because you are prejudiced and unable to produce a fair review.

You're not prejudiced against the specific paper or specific author, you have an opinion regarding the significance of such papers. But - do make sure you're not pre-judging a specific paper because you've disliked existing ones which are somewhere between theory and practice.

Well, the paper is correct, isn't it? Judge exclusively by the content.

You're "spoiling" this answer by adding an irrelevant rhetorical question. We're talking about relevance and significance, not correctness. Always judge by the content; but - the content must include an argument for significance and relevance if those are not immediately apparent. For that, context is significant.

The editor has asked for your opinion and you think that the paper should not be published. Say so: "It is correct but this line of research is pointless. Yes, I know that there are hundreds of papers within this line, but those are not my business and I am currently reviewing this paper".

Don't say that until you've read the paper and have not been otherwise convinced. Also, again, the "correctness" is not the issue.

The bottom line is summarized in the tl;dr above.

PS - A paper may have a very interesting method to prove a not-very-useful result, and that may be an independent reason to accept.

  • 1
    Let the paper try and convince you is a great suggestion. It is incredibly good advice especially in order to avoid other potential biases (lack of seniority, gender, race). Having an arsenal of one-shot rejection reasons makes you imo more likely to bias your reports as you may refrain from using them on your favored groups. – HRSE Mar 7 '18 at 10:03
  • @HRSE: Thanks for the compliment; but - I'm not sure what you mean by "one-shot rejection reasons". – einpoklum Mar 12 '18 at 10:43
  • In my experience, referees are likely to emphasize singular issues in order to reject papers. You may receive reports or see the comments of other referees along the lines of: "Great paper, but I dislike X". The easiest one of those is "not important". The problem is of course that the less mental effort you have to spend on an argument, the more likely you are to use this argument in a discriminatory way. – HRSE Mar 13 '18 at 8:32
  • @HRSE: "Great paper but I dislike X" sounds like "weak accept" to me... More generally, I feel that rejections require long explanations and justification, which considers potential counter-arguments (while acceptance not so much). – einpoklum Mar 13 '18 at 8:39
  • You're in a subject area with great referees then ;-) – HRSE Mar 14 '18 at 0:47
16

I would, and have, chosen option 3. However, I phrase it differently. Make your point using neutral terms, so the focus is on your argument rather than your evaluation. That is useful to the editor, regardless of whether (s)he agrees with your points or not. A professional disagreement is no reason to be blacklisted.

  • Perhaps "unmotivated" can serve as a neutral-sounding substitute for "pointless". – John Coleman Mar 5 '18 at 19:01
  • In addition to being useful to the editor, giving arguments to explain why the article is pointless (and maybe adding ideas on interesting generalizations/concrete possible applications) may help the author. – Taladris Mar 7 '18 at 7:07
15

You estimate that around half of the papers in one of the top journals fall into this class of "pointless" papers. Honestly, to me that says that your view of what is "pointless" is so far out of line with what the community as a whole thinks, that you shouldn't be making recommendations about papers on that topic.

  • 6
    Yeah, I thought about that. But science is not democratic and, after all, it is them who have asked for my opinion. – Miguel Mar 4 '18 at 3:19
  • 4
    Do you think they'd have asked your opinion if they knew you dismiss the whole area as pointless? – David Richerby Mar 4 '18 at 12:11
  • 9
    Maybe. A minority opinion is an opinion. If everyone had the same opinion, there would be no need to review. – Miguel Mar 4 '18 at 13:21
  • 11
    If you want to tell a journal that half the papers they publish are pointless, do that by writing a letter to the editorial board, not by penalizing the unlucky people whose paper was sent to you to review rather than somebody else. – David Richerby Mar 4 '18 at 13:26
  • 13
    @DavidRicherby I strongly disagree. The editor asked OP for his honest professional opinion of the paper. The only proper response is for OP to give his honest professional opinion of the paper. – JeffE Mar 4 '18 at 15:55
13

As an editor, option 3 is exactly what I'd like you to express. As an editor, and as a member of the community, I agree that we publish a lot of papers that are indeed either (i) pointless, or at least (ii) do a rather poor job motivating why anyone should care.

So to ask the question "Why should anyone care?" is really one of the more important jobs of a reviewer. If you think we should care, then you can check technical correctness, but I entirely agree that a pointless paper, technically correct or not, should not be published. It's the editor's job to ask reviewers about this issue as well.

(I will add that I have trouble seeing what the point is about many papers I find as well. But I've also learned that not all papers do a particularly good job explaining what the point is, and that that is not equivalent to there not being a point. That's particularly true for many pure math papers that contain essentially no introduction that puts the result into context. That doesn't mean that there is no context in which the result is not relevant -- it just means that you have to be an expert in the field to see it. I suspect that many papers in applied maths and other areas fall in the same category: You will understand the point when you know enough about the problem, which most of us don't. Of course, this does not refute the fact that there are many papers that truly are pointless.)

12

It is correct but this line of research is pointless.

Number theory. I agree that the way you described it sounds like really minor results. But on the other hand, I believe minor results were almost always the stepping stones in mathematics. And if you think the "line of research" itself is pointless, I can't help but think of number theory. Was thought of as pointless with toy problems until cryptography.

So, I'm saying you can't really judge if a line of research will always stay pointless or if the toy problems become interesting problems in the future because we can't know the future.

  • 2
    No, no, no... no way I am going to be dragged into a pure vs applied math war :) I do not dismiss a whole field, but a particular "assembly line" produced sequence of papers. – Miguel Mar 4 '18 at 15:09
  • @Miguel Be assured, I'm not trying any of the like. Just thinking what other answers may have missed. I understand the "assembly line" problem, though. – SK19 Mar 4 '18 at 16:11
  • 1
    It's also a false equivalence. Just because some number theoretic results have been found very useful in cryptography does not imply that all number theory papers from 30 or 50 years ago are now relevant. It's also not true that for some of these irrelevant papers a good reviewer could not have predicted the lack of relevance. (Note that I'm not trying pick on number theory here: the same observation can be made about many fields -- including my own, numerical analysis.) – Wolfgang Bangerth Mar 6 '18 at 13:23
  • 1
    @WolfgangBangerth I'm not stating an equivalence. I'm simply adding a point of view. Of course not all are relevant today. But that holds for a lot of papers which results have been generalized since. "It's also not true that for some of these irrelevant papers a good reviewer could not have predicted the lack of relevance." I want to emphasize that this may be your belief, but cannot be proven. Any paper you could name as an example could potentially become interesting in the future, unless it has been generalized since and not used extraordinary proof techniques. We simply cannot tell. – SK19 Mar 6 '18 at 15:53
11

When I read your question and came to the passage "pages and pages of equations, crowned ... or the Lyapunov Stability criterion", my reaction was that, if this is an accurate description of the paper, then you should tell the editor exactly that.

11

I have asked a very similar question here:

What good is engineering research with no practical relevance?

However, the majority of the answerers chastised me for failing to see the usefulness of "pointless research" (in your own words). If only these answerers had seen the papers I have seen!! I am too polite to post those research paper on here, but when you are using genetic algorithm to allocate fertilizers (in 2017, when there are extremely robust optimization methods), you might be doing something wrong. When you are rediscovering Markov decision process, you are doing something wrong. When you are modeling the purchasing behavior of actual human beings as dynamical system (with zero stochasticity involved), and proposing a Lyapunov function to "show convergence", you are doing something wrong. It is not about the math, but about the assumptions and practical relevance.

For the sake of producing good research, while being guarded against our own biases, the best we can do is to highlight and question (and question repeatedly) the practical relevance of those "pointless research".

We should not be afraid to ask things like: "How can you extend your results? How can it be implemented? what advantage does it bring as compared to seemingly better and more widely used alternatives? Why is the application of your paper so limited? Is there anyway to use your proposed algorithm for a non-toy problem"

or

"why should anyone care?"

This is the only way we can compete against research in industry and safeguard the prestige of the title of an "academic", if not the quality of research in a particular University or even an entire country.

Some countries are known to output enormous amount of pointless research, and I think it has harmed the reputations of researchers from those countries very deeply.

To your question, if you find that the application is too specific, or that the algorithm is too limited, I would recommend you not hold back when deciding whether or not to push the reject button. But definitely discuss this with several peers first, a consensus should quickly be reached if the paper in question is, indeed, pointless.

This is the only way that academia can innovate. I have seen people doing the same thing that they have done 20 years ago, just with minor tweaks in the setup, so that an algorithm appears slightly different than when it was first proposed some 20+ years ago. It is time for a serious change.

6

I will admit my bias here: I once had a paper dismissed as "a solution in search of a problem" by a reviewer (essentially the same as 'this is pointless') when:

  1. That paper was an integral component of a larger project
  2. That paper has been cited 33 times, and is well above my personal h-index.

So I'm a little skeptical of people's ability to evaluate what is useful.

Personally, I tend to review papers based on their technical merits - for me to reject something, it has to be badly done, outright wrong, "This does not say what you think it says", etc. rather than just "I don't like it." I may however leave the authors a comment that the results need firmer linking to some practical outcome (I'm in a field where practical outcomes matter, and 'Is interesting for its own sake' is rarely accepted).

In the comments to the editor section I would note that I don't think the paper will have much of an impact, and is one of a number of examples of the field going down a useless rabbit hole. But I'd leave it to their editorial judgement to determine if they think that's grounds to not accept something.

  • 7
    "So I'm a little skeptical of people's ability to evaluate what is useful." There's a reason papers tend to have more than one reviewer, after all. – JAB Mar 5 '18 at 21:46
  • @JAB unfortunately, more and more editors use an aggregation rule by which each reviewer has to agree to publication. a dismissive review plus a glowing review does not equal two unenthusiastic but not outright dismissive reviews. – HRSE Mar 7 '18 at 9:51
1

The referee assignment is offered in the context that you know what is acceptable and appropriate for that journal. If one believes that half of what's published in that journal is not appropriate because of the area the work is in, then such person is not an appropriate referee for that area for that journal. The OP should decline to review such papers.

1

I would like to offer a slightly contrarian answer, possibly controversial, but do give it a thought.

You are perfectly entitled to your evaluation of the manuscript; after all you would not be contacted if you don't have the appropriate credentials. You are also right to say that this paper is your business and not other similar papers. I urge you to go one step beyond, and consider the impact of your decision.

As a responsible academician, it may be good to think if a 'reject' decision is likely to change the number of "parameter change" papers being published. Unlikely- the editor is more likely to treat it as an isolated case. It certainly raises one voice, but if as you say, 50% of papers published are similar, that voice will be drowned out.

Now consider the effect on the author(s)- they have seen many similar papers getting published in the same journal. It may even be the case that they chose this journal for that very reason. To them, this would appear an unfair and biased decision. This does not help their morale, and for a young researcher, could be demotivating. It is a bit unfair for them to be singled out. (Yes, the review process is probabilistic and inequalities are inevitable, but perhaps we could do our bit to reduce those, when we can.) When the researcher is demotivated, (s)he is less likely to understand nuanced arguments (eg. relevant but pointless) and take them in the right spirit. The paper will probably be subjected to minor modifications and sent to a similar journal, where it may well get published. You, as a reviewer, have little chance of changing the author's research tendencies in isolation. In the end, you will have the satisfaction of having stated your point (very important in science!), but beyond that, whom or what did it serve?

To my mind, an alternative would be to either review this paper (rigorously) based on its relevance and correctness (leaving relevance aside), or to decline the review. Certainly how interesting a paper is should factor in a regular review- but given your strong feelings about this journal/type of paper, this is not really a regular review. I reiterate- I don't believe this is how every review should be done, this is a special case.

If you are serious in your objection to the type of papers being published, you should instead use you academic standing to write to the editorial board, bring up this issue in conferences, or take a stand in public fora. At the very least, the conversation will move beyond you, one editor and one author. You might find that many others agree with you, and that could be a step towards making a really meaningful contribution. Even if that doesn't happen immediately/visibly, you will know you attacked the root cause, not the symptom.

  • 1
    I understand that I alone am not going to change the trend, but when in doubt I always apply the Categorical imperative: if every reviewer acted the same, then the trend would change towards more meaningful papers. Otherwise I would not be responsible, just taking the easy way. – Miguel Mar 12 '18 at 9:40
  • @Miguel- I appreciate the amount of thought you have given this issue, and thank you for introducing me to the Categorical imperative. Taking the easy way out is not right- hence my last paragraph about raising the issue more publicly. In fact, it may be much more difficult, but probably more 'right'. If I may offer a different Talisman, think of the effect of your action on the last person in the line, irrespective of anyone else. That way you take more ownership of the situation. – AaB Mar 12 '18 at 11:46
-1

This is a great question. I've been on the "other end" of the situation - I've submitted my paper which "could" solve a real-world problem, but the paper only did a simulation on a toy/contrived problem.

Problem statement: Running all regression tests can be really time-consuming for a team/organization/company. If we could schedule the tests from the lens of game theory by looking at testability as a competition between testers and developers, we could get superior scheduling as opposed to "any" kind of prioritization scheme one could come up with.

This is really really hard to do on a real system with real people with real tests. In fact, we don't even have data on the "success" of tests even if we use open source software (i.e., which tests were the most successful at identifying problems/bugs).

The best we could do in this situation, is abstract out everything about the "real world" and only focus on the algorithm at hand. We tried to model all known prioritization algorithms similarly so that a simulation could give us some idea into its effectiveness.

We got comments on the lines you suggest:

...I (reviewer) am okay with any direction of research, but this is so far-fetched that the authors are imagining this problem in their head when none exists. There is no evidence of any competition between testers and developers in reality and this is not even applicable in the wild.

This caught us off guard - apparently, the reviewer(s) focused more on the "formulation" of the problem than what we were proposing. Our paper's premise was to put forth a better way of prioritizing and scheduling regression tests so as to optimize the chance of finding bugs by randomizing the tests w.r.t. game theoretic models. That is if we can only run 10% of the tests, which to run (randomly) that would maximize our chances of finding bugs at some X% level of confidence.

What I'm trying to say is, it'd have been really nice if some of the reviewers would have just taken a step back and evaluated the paper from the problem's POV vs. everything else. Some reviewers thought it was pointless - we thought it was a great new POV that never existed before and hence we wrote a paper to put the idea out there. Few got quite a few folks who liked the idea but many just didn't get it (or were swayed by their prejudice).

Hence I strongly urge you to see the actual problem being solved - is that an interesting problem? Can it be solved any other way? What's better about "their" way? Can you, the reviewer, even imagine a possible application? Does the paper call it out? What are the merits of that approach?

If the paper holds up to the question above, then it may be a candidate for acceptance vs. being a pointless research direction. Note, the folks writing the paper are more deeply set into the problem and have some idea of its applicability (or not at all, and the reviewer should help with that).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.