14

My team has been trying to replicate a study published in a journal. After a month of effort my team is convinced that the details provided in the paper are incorrect and the study cannot be replicated with the information provided in the paper.

We now want to publish the discrepancies in the paper along with proposed modifications.

I want to know:

  1. What article type (Commentary, Letter, Note, etc ) would fit for the proposed submission?
  2. Should it be submitted to the same journal or could it be submitted to any similar interest journal?
9
  • 27
    You spent a few days on a study that potentially took the researchers years, but you're sure it's wrong? Have you considered contacting the authors for assistance?
    – Ian
    Commented Jan 31, 2023 at 12:12
  • 9
    Few days is one month. There are clear discrepancies in the paper. The mathematics of the paper is just not right. We have tried to contact the authors, haven't received any reply.
    – pkj
    Commented Jan 31, 2023 at 12:35
  • 5
    Yeah, a month seems a little early. I spent 2 years reproducing a result form a paper when I was a PhD student. It did eventually work, but i was sure hard work to do. Commented Jan 31, 2023 at 17:08
  • 11
    @Ian reproducing a result should take much less time than the first work, since the method is already outlined, so I disagree with your sentiment there. But agree that OP should've mentioned their attempts at contacting the authors in the question, since it's part of due diligence in reproducing a result.
    – justhalf
    Commented Feb 1, 2023 at 7:44
  • 20
    to the commenters: how long it takes to reproduce a result is clearly very strongly dependent on what kind of thing the result is - I don't see how you could give an estimate either way without more information.
    – N. Virgo
    Commented Feb 1, 2023 at 8:28

4 Answers 4

13

Journals publish less-than-perfect papers so often that it's seldom even noteworthy. I can think of several often cited papers from the 20th century with glaring typos in the math formulas, and/or with gaps in proofs, which were fixed in later papers.

As your first step, you should contact the article's "authors", using the contact information in the paper, or on their departmental web site. You should include all the authors, not just the designated "corresponding author", unless there are too many of them. You should include all your detailed and specific criticism of their paper in your initial e-mail, rather than a "teaser".

Example of a bad e-mail: "I found problems in your paper, please contact me so we can discuss." - many people would assume this to be spam/crank e-mail, and not even bother responding.

Example of a better e-mail: "You state in your paper that because water is wet, therefore the sky is blue. But I see a gap in your argument because you assume that the quadratic variance term is zero." Unless you have reasons to suspect intentional misconduct, you should assume an honest mistake, and treat the authors like you'd want to be treated if you were in their shoes.

Optional: "We further proved a weaker result, that the sky is blue-green, and found some counterexamples of the sky being green." If you've made some non-trivial advances, then you may choose to share them with the authors or not. Could you use them in your own publication? Are you worried that the authors would steal it?

The authors should immediately acknowledge the receipt of your e-mail, and within a few weeks either explain to you why this is not a problem, or send a correction / retraction / clarification to the original journal, graciously thanking you for finding the problems. It would be unusual for the journal to decline to publish a correction submitted by the authors. Or they may even argue that the problems are unimportant, but you may disagree.

If the authors blow you off or you're otherwise still not satisfied, then the next step may be to post your criticism on pubpeer. Other people may comment on your criticism there, and help you improve it. Or the authors may respond to your satisfaction this time. You may decide to skip this step and go straight to the letter to the editor.

The editor should acknowledge its receipt right away, solicit feedback from the authors and the reviewers, and hopefully publish your letter, together with any rebuttal from the authors. But they may totally blow you off, especially if it's not a reputable journal.

The next step may be to write a little paper on your own. Frankly, a paper focusing mostly on the problems in another paper is unlikely to be of interest to many people. You should try to include some new results if possible, so people reading it won't feel that it's a total waste of their time. Posting a preprint on arxiv or the like may be more realistic than a journal.

3
  • 6
    Another template email might be: "Hi Prof X, We read your paper about fooing bars with interest. Our research focuses on fooing buzzes, so we thought your result was interesting and though to try your fooing method. Unfortunately we couldn't get it to work, even on bars. You don't mention the composition of Buffer X. We'd assumed it concated super-foo-ium but perhaps its based on a different foo-donor? We'd really appreciate your help getting the fooing protocol up and running. " Commented Feb 1, 2023 at 7:56
  • Often papers name a "corresponding author", are you suggesting to contact all the authors even though a corresponding author has been identified? Commented Feb 1, 2023 at 12:25
  • 2
    Ian -- thanks! @FerventHippo yes, absolutely, I'm suggesting exactly that. I've heard multiple stories where the designated "corresponding author" died / retired / switched careers / lost interest / went missing / was too lazy to forward... and the other authors never heard about the failed contact attempts until much later. Unless it's one of those dozen+ author papers, please don't rely on the known weak link in communications, but rather do let every author know initially about your concerns. The subsequent conversations can involve fewer people. Commented Feb 1, 2023 at 12:50
7

I don't think we can answer your first 'article type' question, as different journals call different types of submission by different names. Look at the journal's instructions to authors and see what they say should be submitted to comment on previously-published work, and follow that. Or ask the journal directly if it's unclear.

In terms of the 'where' question, you can of course submit it to any relevant journal. But if it's a narrow criticism of a single paper's methods, it's unlikely that any journal other than the original publisher of the article would think it's sufficiently noteworthy. If the result you're questioning is particularly high-profile, or the method is widely- and incorrectly-used, there may be more interest from other journals, but it doesn't sound like that's the case here.

2

First of all, as the comments have already indicated, I, and most reviewers, would be skeptical that the paper in question has various errors/discrepancies. Maybe there are some notation issues, different understandings, or other typesetting glitches, but these errors are different than what you seem to be implying, which is that the paper's claims are "wrong". You should make it clear in your question whether you think this original paper is simply flawed, or whether you think this original paper has serious issues and should be retracted or corrected (I get the impression that you're implying the latter).

If there are errors, you need to make the distinction between a technical error which is mostly harmless and due to imperfect human understanding, as opposed to an intentional or negligent error on the part of the study's authors. Just correcting some notation/equation issues is not significant enough to warrant publication by itself, at least in the fields I am familiar with (e.g. various areas of engineering). On the other hand if you think there was intentional or negligent errors on the part of the original study's authors, and you have some evidence to back up this claim, then it sounds like you should contact the journal's editors with your concern and evidence.

Ultimately, to warrant publication, it is expected to make a significant new contribution. Correcting some typos or pointing out problems in previous papers shows that you understand the material, but that by itself is not a significant new contribution. If the paper you're correcting is especially noteworthy, I suppose the correction might warrant publication by itself as a letter length paper. But you really need to think in terms of making a significant new contribution, as opposed to your current mindset which comes across as more "this paper in the literature had various problems in it" (which may very well be true, but saying that by itself is not a contribution).

Lastly, you raise the question of reproducibility: well in a lot of engineering, it's basically impossible to reproduce a study unless you also have the code. The code might not be available for a number of reasons (e.g. funding might impose that the funding agency has sole access to the code for some time; or maybe the authors have a follow-up in mind and have kept the code private intentionally until the follow-up is completed). Also, the authors might not reply to you for various reasons (e.g. they're on vacation, or have too many emails and missed yours). This is all to say, a lot of the literature is, unfortunately, NOT reproducible (at least in the strict sense of what 'reproducible' means).

You should focus on your paper's contribution (not the contribution of previous papers), and if some paper in the literature gets some parts wrong, then of course you should present the correct way in your paper, but it's not needed to call out specific authors/papers as being 'wrong' or 'bad'. If your way is truly better, rest assured that other people will (eventually) notice this as well, and tend to cite your paper more.

1

I do not think these kind of omissions are publishable unless the paper is highly cited and the results are misleading.

If this is in computer science, a significant portion of the papers are in the same state. Replication therefore requires guesswork. Although this makes the paper a low quality one, there are many of these even in top journals. General response to these are simply contact the author to obtain additional information or source code of the project. I suspect most of these authors think that explanation that they provide is enough, but since only they have complete knowledge, explanation given is often too little. Even when there is an algorithm, it generally does not cover entire method.

To summarize, while this trend of low quality, very hard or impossible to replicate papers do slow down the speed of research conducted, it is considered normal and you will not be able to get back anything for the time you have lost.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .