During the reading of a paper, published in the proceeding of a prestigious computer science conference, I noticed a logical error in the results presented in the paper.

I contacted two of the authors who are my colleagues (one is a PhD student in the same lab where I am a student) to ask for clarification. After hearing what they had to say, I was under the impression that they mainly tried to convince me that the results were OK. One argument they used was that because the paper was already published in a prestigious conference and passed the peer review, the results couldn't have been wrong. That didn't satisfy me; rather, their answers made me suspicious that at least parts of the results, presented in the paper, are fake (I mean, are not real). Moreover, that the authors tried to hide the problem with the results. Finally, I believe I have strong evidence to support my belief, which, at the moment, is that the results are fake.

What is the normal course of action in a case like this? If someone has some evidence indicating that a paper published in proceedings of a (computer science) conference includes fake results, what should that person do?

Additional information:

It's been more than two years since the paper was published.

The evidence that the paper is "fake" rather than merely wrong is that the method claimed would not be computationally tractable for the claimed data size without some special innovation that was not described by the authors.

  • 44
    " One argument they used was that because the paper was already published in a prestigious conference and passed the peer review, the results couldn't have been wrong." That is a terrible argument indeed.
    – Clément
    Oct 17, 2022 at 20:17
  • 61
    Fake, or possibly incorrect? Quite a difference.
    – Jon Custer
    Oct 17, 2022 at 21:02
  • 16
    Echoing @JonCuster's comment, it is surely important to distinguish "fake" from "incorrect" (not to mention intent...) Oct 17, 2022 at 22:40
  • 19
    It is always safer to ask a question -- "I'm not understanding this point, can you clarify" -- than to challenge. That avoids either accusing the other person of being wrong or making a fool of yourself if the error is yours. And clearly, if you're saying "some evidence", you aren't in a position to challenge directly and get a publication credit out of it...
    – keshlam
    Oct 18, 2022 at 6:49
  • 14
    For a concrete example, I wrote a branch-and-bound algorithm for a combinatorial problem where the algorithm's complexity was like 2^O(n^2), and the algorithm would take days or weeks when the problem size was 15, 16, 18 or 19, but for 17 my algorithm "got lucky" and found the solution in under a second. As far as I could tell, there was nothing special about the case n=17, it's just that the algorithm incidentally found a good solution early, which allowed it to reject the vast proportion of the search space which could not contain any solution better than that one.
    – kaya3
    Oct 19, 2022 at 4:25

8 Answers 8


There are two questions in the scenario that you posit.

  1. Is there a real error in that conference paper, as you seem to think ?

(Let's put to one side whether the error is an honest mistake or a deliberate attempt to deceive since this would be almost impossible to prove and any effort to do so may well signal a sort of false vindictiveness on your part.)

  1. If an error really is there, what should be done about it ?

This erroneous conference paper should only be a real concern to you if its assertions are likely to be a problem to you in your own anticipated published work for your doctoral thesis.

If it's not a particular issue to you in your own work,i.e. if it's just something wrong that you noticed in another person's work, then you just note it and proceed with your own studies. You should do nothing about it until after your next position is secured.

If it really is going to become an issue with the reasoning that you are adopting for your research and the eventual assertions coming from your own work, then you will have to address it with your supervisor. It then becomes a matter for your supervisor to consider.

There are sensitivities involved here since the authors of the erroneous work were (or still are ?) in your own department. But it is your supervisor's job to negotiate a way through these sensitivities that is at least fair to you.

Given what you have told us so far, that is as much as I can advise for now.

And please change your SE name to something that will not cause you or your organization to be identified by readers here.

You can ask SE Academia moderators to remove @ references to your name.

  • 3
    It seems unethical to me to let other people use incorrect results that you know are wrong (which will obviously cause them to either mislead others or waste lots of their time) just because it isn't an issue for you. Obviously that only applies if you're pretty certain that they are wrong, but still.
    – Voo
    Oct 20, 2022 at 13:54
  • 1
    @Voo That is a single perspective. Here is another: it seems unethical to me that a Y2 PhD student be the one to first publicly identify errors in a colleagues' papers and to press for its correction lest others expend time and resources on it; it is not strictly their responsibility, their omission in not doing so will not be held against them as this publication 'passed' more senior research faculty and a PhD student has far more pressing things to do with their work and free time. Academia offers many booby traps for earnest people. Let's not get OP snared in one. If it really is an error.
    – Trunk
    Oct 20, 2022 at 14:19
  • 4
    "other people aren't doing anything either" is not an excuse. But yes absolutely - not doing anything is the most convenient and easy solution - unethical solutions usually are, that's why people pick them.
    – Voo
    Oct 20, 2022 at 15:23
  • 1
    Certainly OP should only take action if they have a high degree of certainty and after excluding other possible explanations. Which might very well not be possible and that's fine. The unethical part is the "I know it's wrong, but I'm not doing anything because it might impact me". Inaction that causes harm to others because it's more convenient for you seems pretty clear cut.
    – Voo
    Oct 20, 2022 at 16:57
  • 1
    The unethical part is the "I know it's wrong, but I'm not doing anything because it might impact me". Inaction that causes harm to others because it's more convenient for you seems pretty clear cut. That's where you and I part company then. I know several people who did the inconvenient thing in academia (and elsewhere) for your sort of reasons. It cost all concerned a lot. Some lost their future in academia. Others had to change profession owing to CV impact. OP comes here looking for advice on the right thing to do. But he expects to preserve his doctorate and his academic career.
    – Trunk
    Oct 20, 2022 at 17:06

Let’s hold our horses here for a moment. Unlike the other answers so far, I see three options:

  1. The results are “fake”, e.g. maliciously, knowingly fabricated
  2. The results contain a mistake, made in good faith
  3. The results are absolutely fine and you’re misunderstanding something

Option 1 is an extremely serious allegation and you should make sure you’re certain of misconduct (as opposed to honest mistake). This is not only to protect these colleagues: if you accuse someone (especially in your own group) of faking data, and are later found to be in the wrong, you will acquire a reputation that’s going to weigh you down a lot. Your use of the word “fake” is also confusing: do you mean “fabricated” (getting one number but writing down a different one) or “incorrect” (overlooking a flaw in the calculation and getting the wrong number, but writing it down faithfully)?

Option 2 is very common and you have already done the right thing: spoke to the authors about it. Since it’s your current group, it’s much better at this point to discuss your concerns with your supervisor than to escalate or publicise this. If it’s a mistake, it’s still potentially embarrassing for the group and your supervisor will probably be grateful to be alerted. If on the other hand there is indeed misconduct, and assuming your supervisor is not aware or complicit, it’s in their interest to resolve the situation delicately, and they have many more tools at their disposal than you (e.g. issuing a retraction, disciplining the relevant researchers).

Option 3 is also possible. Since we only have your side of the story, it’s very hard to tell the difference between “they mainly tried to convince me that the results were OK” and (from their perspective) “we tried to explain our results to OP but they’re still stuck on that step”. Can you work through the paper with a peer and see if they also agree with your interpretation? You should try to do this neutrally: “I’m not sure I agree with this, what do you think” and not “This bit here is clearly fake”

Finally: you may want to avoid using your real name when discussing serious scientific misconduct on a public forum.

  • 19
    Good answer! I'd also add that OP not being able to distinguish clearly between 1 and 2 in their question makes me think that 3 is by far the most likely scenario. Oct 18, 2022 at 14:50
  • 1
    OP says it's a "logical error" - this could be important in certain areas of computer science as it could lead to false conclusions. I hope no funding allocations were provided on the basis of any false conclusions as this would make things embarrassing for everyone concerned - although a fessing-up would have to be done if a real error was made. On the other hand the logical error may be harmless and offer no advantage to the research group. If error it was - and OP hasn't passed it to anyone for review yet, it seems.
    – Trunk
    Oct 18, 2022 at 15:20
  • 28
    One of the more important lessons to learn as a PhD student or young researcher is to always assume option 3 by default (then work from there). Many posts on this site indicate that few people ever consider it, though. Because it is almost always option 3. Many times I've been in the exact same situation as the poster, then realized my mistake after weeks of swearing to high heaven that I couldn't possibly be wrong. Do not start fights by hurling megaton allegations without serious, serious proof.
    – Jerome
    Oct 19, 2022 at 2:06
  • 10
    @Jerome that’s my experience too. Option 3 is really common, we all get things wrong, and a bit of humility at this stage can prevent a lot of embarrassment later on. There’s a scathing paper review I sent early in my career I can’t ever think of without cringing.
    – Ottie
    Oct 19, 2022 at 6:50
  • 4
    Is there a 4th option that is kind of the child of both 1 and 2? i.e. The paper was originally published in good faith under the umbrella of 2. The author has since realized the error but refuse to acknowledge it and continue to defend the paper as accurate. "It was accepted, it can't be wrong."
    – Mr.Mindor
    Oct 19, 2022 at 17:11

As mentioned in the comments, there's a major difference between "fake" and "incorrect".

If the results are fake, then you are alleging scientific misconduct. If true, this will have serious consequences and you will need serious proof. The process varies by institution, but here's an example from the University of Manchester. Here misconduct is defined as fabricated results, falsified results, plagiarism, failure to meet legal/ethical/professional obligations like ethical guidelines when conducting human research, etc. If you observe these things then you are required to report the fraud (clause 5.2). There is a person at the institution which you can report the alleged fraud to, to whom you file a report. You are given legal protections for raising concerns about fraud, but you need to bring evidence. Once fraud is reported the university will investigate, and the affected people will get the chance to defend themselves. Needless to say, this will have serious consequences at personal level, regardless of whether the allegations are true or false.

If the results are incorrect, then it's a different matter. The "standard" way of dealing with this is by writing a new paper highlighting the error. Published papers can be wrong, so "the paper was already published in a prestigious conference and passed the peer review, the results couldn't have been wrong" is not true. Here's an example.

In your case, given that the author is a member of your research group, I would expect to start internally. If the results are fake, then I'd talk to the supervisor first, before escalating if it looks like the supervisor is also involved in the fraud. If the results are incorrect, then I'd convince the other author first, or at least the supervisor. If you can convince the supervisor there's a problem, chances are they'll tell you what to do next (e.g by getting more data).

  • 1
    I'm still a bit surprised that a paper was published in JHEP essentially stating that certain particles could exceed the speed of light in a vacuum, is that what the article is implying?
    – Tom
    Oct 18, 2022 at 7:41
  • 2
    @Tom it was, yeah (past tense since the result has been discredited). physics.stackexchange.com/questions/14968/…
    – Allure
    Oct 18, 2022 at 8:03
  • 17
    @Allure I remember that one! It was a faulty optical fibre connection right? Even the authors didn't "believe" it, but there comes a point when you need to publish what you observe or everyone is going to hear rumours anyway. Particularly with a collaboration that big.
    – Clumsy cat
    Oct 18, 2022 at 9:05
  • @Clumsycat That makes sense I suppose.
    – Tom
    Oct 18, 2022 at 10:18
  • 3
    @Tom The manuscript was never published. The last paragraph of this answer physics.stackexchange.com/a/111137/36194 sums it up best. Oct 18, 2022 at 13:11

Confronting them or fighting with them is not going to get you anywhere. Not anywhere positive, anyway.

But you can always write a follow-on paper that improves/corrects the error and submit it, perhaps to the same journal. It will force you, however to completely verify your claims. That would be one of the valid ways to proceed if you were at a different institution.

The other alternative is to create a note pointing out the error, and the effect on the conclusions of the paper and send it to the editor. This is a little bit less work, but you still need to be very sure of your arguments.

And if you are alleging "fake" rather than mistaken conclusions you need to be especially certain that you've got it right. But if you can find the correct result, then you don't need to charge them with misconduct. The community will do that if they see the alternative.

Having been published in a prestigious journal isn't proof of correctness, of course. There are plenty of counterexamples, both in technical and non-technical publications.

  • 4
    OP wrote of a conference, which is more important in CS than journals (unfortunately). So there is no next issue. This happens far too often in CS with no good forum for discussing problematic results. Oct 18, 2022 at 4:41
  • 1
    Conference papers are almost the standard form of publication in computer science.
    – Tom
    Oct 18, 2022 at 7:42
  • 1
    The follow-on paper (or correction "letter" to that journal if they publish such things) might be a way to absolve the department for this thing - if error it be. Please consult with an honest academic in the university you attend before making any decision on action. If finding a member of faculty that you feel you can trust is not possible, it's best to let the matter be sorted out after your own doctorate is finished and you get a position elsewhere.
    – Trunk
    Oct 18, 2022 at 15:03

To add to the other answers, let me expand on Option 3 of the great answer by Ottie: Consider the possibility that you are wrong. You spare us the details, but of course details matter to make a decision on the fact. If I understand you correctly, you base your belief that the results are wrong because they seem to have solved an NP problem in reasonable time. Strictly speaking, an instance of an NP problem, even a very large one, can still be solved in reasonable time. Also an instance can have a property that can be exploited or can be solved by a heuristic and the solution can be verified as optimal in polynomial time (that is actually the definition of NP, a problem can be verified in polynomial time). What I am trying to say is that it is very easy to draw a false conclusion based on the fact that something is NP. The story of encryption based on the insolvability of the knapsack problem comes to mind https://en.wikipedia.org/wiki/Merkle–Hellman_knapsack_cryptosystem. As you can see, you would not be the first one to believe falsely that something is computationally infeasible.

If you want to move forward from your obvious frustration with the authors and the paper, you should go through their explanations again, assuming that it is correct. If you want to involve someone else (after announcing results to be fake the authors are probably out), you should follow the advice given to you and asking in help understanding each step. It might be that there is a relatively minor error, such as claiming a sub-optimal but close solution to be optimal. Only after making doubly and triply sure should you escalate to your supervisor. If the supervisor does not react, then you could go to the university level or use one of the publication checking outlets that you have been told about. This is to make sure that you (1) do not make a false allegation that (2) could have serious impact on your career at your institution. If the supervisor does not find a problem, you have the choice to believe the supervisor's judgment or not. If not, you are making a strong statement and strong statements need strong proof.

  • 4
    Based on the OP's addendum in the question's edit history, I think your answer is the correct one. OP has made a logical mistake by supposing that "NP hard" means each individual instance of the problem cannot be solved efficiently.
    – kaya3
    Oct 19, 2022 at 4:39
  • 1
    I suspect that this is true, but we do not know. OP could be onto something. Oct 19, 2022 at 9:23

I'm a fresh graduate and am not very experienced, but your question doesn't seem to be difficult to answer. You have two options: (1) you can do nothing, and (2) you can start a fight. To start a fight, you have two options: (2a) you can fight in the battlefield of research journals by publishing a critical article or a comment, and (2b) you can fight by lodging a formal complaint with the universities the authors work at.

The article was published in the proceedings of a conference, which means that publishing a comment there isn't an option. And I don't think that a journal will accept an article whose purpose is to criticize a paper published in conference proceedings. This means that Option (2a) is unavailable unless the authors publish the same results in a journal.

Option (2b) should be used only if you can prove that the results were fabricated.

At any rate, you should think twice before starting a fight. Ask yourself the following two questions: (i) What's in it for me? In other words, is this worth spending time and making effort, or is it better to focus on my own research instead? What will I gain from this fight? (ii) Do I have a proof? To use Option (2a), you have to have a proof that the results are wrong. To use Option (2b), you have to have a proof that the results were fabricated.

I personally would start a fight only if someone published results contradicting mine.

  • You can also fight by complaining (not via a paper, but simply via e-mail with the relevant data attached; if you consider this to be your 2a, then sure) to the journal editor, right?
    – justhalf
    Oct 18, 2022 at 4:26
  • @justhalf yes, sure, you can, I'm just unsure what the editor could do then. Btw, the paper was published in conference proceedings, not a journal
    – Mitsuko
    Oct 18, 2022 at 7:57

There are many fake results out there, most of them are never denounced or corrected. People who blow the whistle through the official channels (journals, institutions) often suffer more than the fraudsters. Official procedures tend to take years, and to waste an unreasonable amount of your time even if you ultimately prevail. See Leonid Schneider's blog for many examples in biology and medicine.

Fortunately, there are ways to raise concerns without jeopardizing your career. In particular, PubPeer allows you to publicize your remarks anonymously.

You do not need to be sure that there is fraud for raising concerns, provided you do it in a professional manner. After all, it is the author's job to explain their results clearly and convincingly. If their work is correct they should be able to easily answer your concerns.

Asking people who suspect scientific fraud to be absolutely certain and to go through official channels, is one way to make sure that most fake results will never be corrected.


Whether your suspect simple error, outright fraud, mind control by nefarious evil forces, or whatever, the best way to challenge results is to attempt to replicate them.

If you faithfully follow their methodology, use the same input, and get the same output, then you'll have learned something.

If you get different results, then you can publish those results, and third parties will take note and get involved.

You must log in to answer this question.