69

In a modeling paper I am writing, I know that there are some limitations.

Is it a bad idea to point out the limitations? Am I just giving the reviewers more straw for them to reject my paper?

Improve this question
6
  • 24
    definitely clearly point out what domain(s) your model is most valid in and what domain(s) your model is least valid and why. if your model is meant to help someone get a grip on some complex interaction, that only helps when these someones apply the model to where it is meant to be applied. it reduces the efficacy of the model when it gets applied to contexts where it is already known to be flawed.
    – robert bristow-johnson
    Aug 28, 2016 at 3:16
  • 7
    To add to what the others say: If I think the authors haven't considered an obvious issue I'll assume it's because the work is low quality and the authors are incompetent. I'll give my report accordingly. If an obvious issue exists and I don't agree with the authors' discussion of why it's not significant, I'll question it, but I won't assume the authors are incompetent. If I ever thought the authors knew about an issue and are hiding it... every paper of theirs I ever referee from then on would receive a very, very close investigation with no margin for error.
    – Joel
    Aug 29, 2016 at 3:23
  • 5
    Sophisticated readers will always remember that "all models are wrong, but some are useful". If you have no description of the shortcomings then your sophisticated readers may assume you weren't astute enough to recognize them and judge your paper accordingly.
    – eykanal
    Aug 29, 2016 at 13:14
  • 1
    Do not write with the goal of 'getting the paper published' - it doesn't serve the fundamental purpose of writing papers which is to disseminate knowledge and document it for posterity's sake. This is what ails academia today in fact, people hiding facts, even skewing results, or simply churning out tripe that follows the money trail, but teaching us nothing new. It would be refreshing to see something honest and original, submitted out there for a change. Do not fear the rejection.
    – squashed.bugaboo
    Aug 29, 2016 at 20:58
  • 3
    @MichaelGeary, yes. i am saying that, assuming the author knows where his/her model is valid and where the model is less valid, that the value of the published work is increased by clearly describing the "domains of validity" (or whatever term you wanna give it). leaving it out decreases the value of the published paper.
    – robert bristow-johnson
    Aug 30, 2016 at 20:50

9 Answers 9

Reset to default
151

Science is not about getting papers published; it's about writing papers that move a field forward. So don't let worries about reviewers prevent you from adding content that you feel should be there. Indeed, modeling papers that hide their limitations are much less effective at advancing a field than those that clearly explain them.

Good reviewers recognize this and often look in the Discussion for a clear exposition of the model's limitations as well as its strengths. That said, to be worth anything a model must make useful contributions despite these limitations, and it's your job to explain how your model does that.

Improve this answer
5
  • 32
    This, so much this. Address the obvious questions in the paper. A well reasoned discussions of the strengths and weaknesses of a model or an experiment is a wonderful thing.
    – Jon Custer
    Aug 28, 2016 at 0:36
  • 41
    speaking as a software dev often trying to understand an algorithm from a paper, the paper is almost completely useless without a discussion of the limitations. It makes it much harder to actually determine if the work is in any way useful.
    – Leliel
    Aug 28, 2016 at 5:38
  • 7
    @Mindwin Science is inherently social. As long as papers remain the primary venue for sharing results, models, etc., science is inextricably linked to the act of writing papers. It's just not about writing papers to get famous.
    – Corvus
    Aug 29, 2016 at 20:38
  • 2
    Agreed. However, it is an unfortunate reality that the goals of advancing science and advancing one's own career are not always perfectly aligned. Admitting the limitations of your own work is a perfect example of this. In some cases it may indeed be better for a researcher's career to not mention a significant flaw and hope no one notices when publishing. I still think your advice is the right thing to do. But at times there is a cost for it.
    – user24098
    Aug 31, 2016 at 8:44
  • 2
    To elaborate on this, in general, a discussion of limitations is expected and strengthens your paper. But in certain cases, there might be a specific limitation that greatly reduces the value of the work (or even renders it useless). There may be a personal cost to admitting such a flaw, as opposed to just trying to "get away with it".
    – user24098
    Aug 31, 2016 at 8:47
41

Personally, as both and author and a reviewer, I strongly prefer papers that are clear and honest about their limitations. I typically see good declarations of this sort take one of several forms:

  • The results are limited to a specific class of cases, but those cases are quite interesting and important.
  • The results are limited, but they are still a major improvement over the prior state of the art.
  • The results are limited, but the work needed for necessary extensions to broader coverage is clear and unlikely to pose major scientific obstacles.

Clearly, if you are submitting a manuscript, you believe your work is interesting and valuable despite its known limitations. Explicitly stating those limitations gives you a chance to make that case for value clearly and directly.

If, on the other hand, you attempt to hide your limitations, many reviewers may notice them in any case, and may hold them more harshly against you.

Improve this answer
14

From my experience reading chemical literature, there is nothing more suspicious to my eyes than a paper which contains nothing but a recipe and a few successful results. That works if you are going to bake cookies, but chemical syntheses of that caliber are always a gamble.

Any really good scientific paper will discuss not only what succeeded, but also what failed, and why. There is absolutely no requirement that papers be written as if you were trying to sell something and therefore only contain positive points. The really good chemical journals (e.g. Organic Syntheses) will contain discussion of any pitfalls or possible failure points as well as discussion of potential hazards, etc. - and that's what sets them apart from other less reputable sources!

I would rather have a 15-page review that exhaustively covers the mechanism of some exotic chemical transformation - rather than a two-paragraph paper which frankly could have been written on the back of a napkin at lunchtime.

Improve this answer
0
9

I would like to add to the other answers by looking at this issue from another perspective. Sure,if the reviewer realizes you left out certain parts on purpose, he might have a hard time trusting your honesty on other parts of the paper (making the review process more difficult).

However, even with the premise that you did not leave out these parts on purpose, failing to see and mention relatively obvious limitations of your work can give the impression of inaccuracy and oversight in your research. Naturally, this can create doubts about the accuracy of other sections of the paper.

Improve this answer
5

Write about them

It is not about selling a product but adding knowledge to the community. Especially in science, knowing its errors and limitations is a crucial part of every result you provide.

Every (good) reviewer knows that.

Improve this answer
0
4

Pointing out the weaknesses will definitely make the paper higher quality and more useful.

It may make it either more or less likely to be published. That's definitely something to consider even if you are altruistic - the paper isn't going to help many people if it's not published. But all things considered I'd say including the caveats will make it more likely to be published.

If the reviewer realizes the limitations himself, and they are significant and worthy of mention, their omission would make the writing appear either clueless, negligent or dishonest.

If the reviewer doesn't realize them himself, their inclusion would still give the paper more credibility, make it clearer and better framed, and give the sense that you know what you're talking about; and the paper would still be worthwhile for the results it does have. Omitting the limitations could make the paper seem "too good to be true" and make the reviewer wonder what he's missing.

And there's the remote chance that the reviewer would buy into a paper that appears to have fantastic results, and reject it if he is hinted the results are a bit more mundane. Such a reviewer is probably gullible and uninformed about what makes a good paper, and it's a bad habit to assume your reviewers will be such.

So, however you look at it, mentioning the weaknesses is definitely the way to go. You shouldn't overplay it though, it is expected that you will be mostly positive about the paper and not go out of your way to discredit it. No need to write "Ultimately, the analysis in this paper is totally useless, because the model makes assumptions X, Y and Z which are completely unrealistic."

Improve this answer
3

You must include all of the parameters in which your model operates including any that you don't understand or have a possible confounding effect.

This is science. To not include them is evangelism.

Including the limits and known issues of your work shows that you have thought the subject through and still consider your work to add to the topic. To NOT include limits will force reviewers to do this work for you and they will not be appreciative.

There is nothing wrong in noting the edges of your work. Science is the exploration of things we don't know. If we knew them there would not be limitations and you wouldn't be writing about them.

Improve this answer
2

Finding a useful model is craft, art, and skill. Very important, quite valuable, not science. How and what this model fits and when it stops fitting for what reason: that's science.

You are contemplating omitting the one thing that makes your paper worthwhile as a paper. "I thought I'd do it this and that way, and it worked." is not an academic paper but a lab report. It provides no insight.

If you are handing in a lab report and skip the science, you may or may not get your paper accepted. But then others will get the credit for actually working out the science, and it will be to your detriment.

Improve this answer
1

High moral standards & selfless service to the greater good of humanity aside, there is still a very good selfish reason for pointing out the limitations of your work: if you do it yourself, you can then discuss it on your terms right there, whereas if reviewers find it, you couldn't. What I mean is something like "My method has such and such limitations, BUT it's still good, because

  • There are still many important cases where it DOES work...
  • ... or it covers a few cases not covered by any existing methods
  • It may be possible to overcome some of the limitations in our future works
  • It looks promising because of blah-blah-blah
  • Etc."

In every research paper I've read (or written), there was a "Discussion/Limitations" section, composed more or less like that. If you don't write it yourself, a reviewer would kindly do that for you, although it would probably look more like "the proposed method seems to have such and such limitations, whether it's still good I have no time to research - reject."

Furthermore, unless your field is completely new and revolutionary, a typical reviewer has already read tons of papers on the subject, has probably done some research of his/her own, and knows quite well what are the typical problems and difficult cases - they expect you to know and discuss them, including inevitable failures of your approach with some of them. If they don't see such discussion in your paper, it gives them impression of you being incompetent, or dishonest, or both.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .