What should I do if I have some evidence pointing out that the results presented in a conference paper might be fake?

Question

During the reading of a paper, published in the proceeding of a prestigious computer science conference, I noticed a logical error in the results presented in the paper.

I contacted two of the authors who are my colleagues (one is a PhD student in the same lab where I am a student) to ask for clarification. After hearing what they had to say, I was under the impression that they mainly tried to convince me that the results were OK. One argument they used was that because the paper was already published in a prestigious conference and passed the peer review, the results couldn't have been wrong. That didn't satisfy me; rather, their answers made me suspicious that at least parts of the results, presented in the paper, are fake (I mean, are not real). Moreover, that the authors tried to hide the problem with the results. Finally, I believe I have strong evidence to support my belief, which, at the moment, is that the results are fake.

What is the normal course of action in a case like this? If someone has some evidence indicating that a paper published in proceedings of a (computer science) conference includes fake results, what should that person do?

Additional information:

It's been more than two years since the paper was published.

The evidence that the paper is "fake" rather than merely wrong is that the method claimed would not be computationally tractable for the claimed data size without some special innovation that was not described by the authors.

Answer 1

There are two questions in the scenario that you posit.

1. Is there a real error in that conference paper, as you seem to think ?

(Let's put to one side whether the error is an honest mistake or a deliberate attempt to deceive since this would be almost impossible to prove and any effort to do so may well signal a sort of false vindictiveness on your part.)

2. If an error really is there, what should be done about it ?

This erroneous conference paper should only be a real concern to you if its assertions are likely to be a problem to you in your own anticipated published work for your doctoral thesis.

If it's not a particular issue to you in your own work,i.e. if it's just something wrong that you noticed in another person's work, then you just note it and proceed with your own studies. You should do nothing about it until after your next position is secured.

If it really is going to become an issue with the reasoning that you are adopting for your research and the eventual assertions coming from your own work, then you will have to address it with your supervisor. It then becomes a matter for your supervisor to consider.

There are sensitivities involved here since the authors of the erroneous work were (or still are ?) in your own department. But it is your supervisor's job to negotiate a way through these sensitivities that is at least fair to you.

Given what you have told us so far, that is as much as I can advise for now.

And please change your SE name to something that will not cause you or your organization to be identified by readers here.

You can ask SE Academia moderators to remove @ references to your name.

Answer 2

Let’s hold our horses here for a moment. Unlike the other answers so far, I see three options:

1. The results are “fake”, e.g. maliciously, knowingly fabricated
2. The results contain a mistake, made in good faith
3. The results are absolutely fine and you’re misunderstanding something

Option 1 is an extremely serious allegation and you should make sure you’re certain of misconduct (as opposed to honest mistake). This is not only to protect these colleagues: if you accuse someone (especially in your own group) of faking data, and are later found to be in the wrong, you will acquire a reputation that’s going to weigh you down a lot. Your use of the word “fake” is also confusing: do you mean “fabricated” (getting one number but writing down a different one) or “incorrect” (overlooking a flaw in the calculation and getting the wrong number, but writing it down faithfully)?

Option 2 is very common and you have already done the right thing: spoke to the authors about it. Since it’s your current group, it’s much better at this point to discuss your concerns with your supervisor than to escalate or publicise this. If it’s a mistake, it’s still potentially embarrassing for the group and your supervisor will probably be grateful to be alerted. If on the other hand there is indeed misconduct, and assuming your supervisor is not aware or complicit, it’s in their interest to resolve the situation delicately, and they have many more tools at their disposal than you (e.g. issuing a retraction, disciplining the relevant researchers).

Option 3 is also possible. Since we only have your side of the story, it’s very hard to tell the difference between “they mainly tried to convince me that the results were OK” and (from their perspective) “we tried to explain our results to OP but they’re still stuck on that step”. Can you work through the paper with a peer and see if they also agree with your interpretation? You should try to do this neutrally: “I’m not sure I agree with this, what do you think” and not “This bit here is clearly fake”

Finally: you may want to avoid using your real name when discussing serious scientific misconduct on a public forum.

Answer 3

As mentioned in the comments, there's a major difference between "fake" and "incorrect".

If the results are fake, then you are alleging scientific misconduct. If true, this will have serious consequences and you will need serious proof. The process varies by institution, but here's an example from the University of Manchester. Here misconduct is defined as fabricated results, falsified results, plagiarism, failure to meet legal/ethical/professional obligations like ethical guidelines when conducting human research, etc. If you observe these things then you are required to report the fraud (clause 5.2). There is a person at the institution which you can report the alleged fraud to, to whom you file a report. You are given legal protections for raising concerns about fraud, but you need to bring evidence. Once fraud is reported the university will investigate, and the affected people will get the chance to defend themselves. Needless to say, this will have serious consequences at personal level, regardless of whether the allegations are true or false.

If the results are incorrect, then it's a different matter. The "standard" way of dealing with this is by writing a new paper highlighting the error. Published papers can be wrong, so "the paper was already published in a prestigious conference and passed the peer review, the results couldn't have been wrong" is not true. Here's an example.

In your case, given that the author is a member of your research group, I would expect to start internally. If the results are fake, then I'd talk to the supervisor first, before escalating if it looks like the supervisor is also involved in the fraud. If the results are incorrect, then I'd convince the other author first, or at least the supervisor. If you can convince the supervisor there's a problem, chances are they'll tell you what to do next (e.g by getting more data).

Answer 4

Confronting them or fighting with them is not going to get you anywhere. Not anywhere positive, anyway.

But you can always write a follow-on paper that improves/corrects the error and submit it, perhaps to the same journal. It will force you, however to completely verify your claims. That would be one of the valid ways to proceed if you were at a different institution.

The other alternative is to create a note pointing out the error, and the effect on the conclusions of the paper and send it to the editor. This is a little bit less work, but you still need to be very sure of your arguments.

And if you are alleging "fake" rather than mistaken conclusions you need to be especially certain that you've got it right. But if you can find the correct result, then you don't need to charge them with misconduct. The community will do that if they see the alternative.

Having been published in a prestigious journal isn't proof of correctness, of course. There are plenty of counterexamples, both in technical and non-technical publications.

Answer 5

To add to the other answers, let me expand on Option 3 of the great answer by Ottie: Consider the possibility that you are wrong. You spare us the details, but of course details matter to make a decision on the fact. If I understand you correctly, you base your belief that the results are wrong because they seem to have solved an NP problem in reasonable time. Strictly speaking, an instance of an NP problem, even a very large one, can still be solved in reasonable time. Also an instance can have a property that can be exploited or can be solved by a heuristic and the solution can be verified as optimal in polynomial time (that is actually the definition of NP, a problem can be verified in polynomial time). What I am trying to say is that it is very easy to draw a false conclusion based on the fact that something is NP. The story of encryption based on the insolvability of the knapsack problem comes to mind https://en.wikipedia.org/wiki/Merkle–Hellman_knapsack_cryptosystem. As you can see, you would not be the first one to believe falsely that something is computationally infeasible.

If you want to move forward from your obvious frustration with the authors and the paper, you should go through their explanations again, assuming that it is correct. If you want to involve someone else (after announcing results to be fake the authors are probably out), you should follow the advice given to you and asking in help understanding each step. It might be that there is a relatively minor error, such as claiming a sub-optimal but close solution to be optimal. Only after making doubly and triply sure should you escalate to your supervisor. If the supervisor does not react, then you could go to the university level or use one of the publication checking outlets that you have been told about. This is to make sure that you (1) do not make a false allegation that (2) could have serious impact on your career at your institution. If the supervisor does not find a problem, you have the choice to believe the supervisor's judgment or not. If not, you are making a strong statement and strong statements need strong proof.

Answer 6

I'm a fresh graduate and am not very experienced, but your question doesn't seem to be difficult to answer. You have two options: (1) you can do nothing, and (2) you can start a fight. To start a fight, you have two options: (2a) you can fight in the battlefield of research journals by publishing a critical article or a comment, and (2b) you can fight by lodging a formal complaint with the universities the authors work at.

The article was published in the proceedings of a conference, which means that publishing a comment there isn't an option. And I don't think that a journal will accept an article whose purpose is to criticize a paper published in conference proceedings. This means that Option (2a) is unavailable unless the authors publish the same results in a journal.

Option (2b) should be used only if you can prove that the results were fabricated.

At any rate, you should think twice before starting a fight. Ask yourself the following two questions: (i) What's in it for me? In other words, is this worth spending time and making effort, or is it better to focus on my own research instead? What will I gain from this fight? (ii) Do I have a proof? To use Option (2a), you have to have a proof that the results are wrong. To use Option (2b), you have to have a proof that the results were fabricated.

I personally would start a fight only if someone published results contradicting mine.

Answer 7

There are many fake results out there, most of them are never denounced or corrected. People who blow the whistle through the official channels (journals, institutions) often suffer more than the fraudsters. Official procedures tend to take years, and to waste an unreasonable amount of your time even if you ultimately prevail. See Leonid Schneider's blog for many examples in biology and medicine.

Fortunately, there are ways to raise concerns without jeopardizing your career. In particular, PubPeer allows you to publicize your remarks anonymously.

You do not need to be sure that there is fraud for raising concerns, provided you do it in a professional manner. After all, it is the author's job to explain their results clearly and convincingly. If their work is correct they should be able to easily answer your concerns.

Asking people who suspect scientific fraud to be absolutely certain and to go through official channels, is one way to make sure that most fake results will never be corrected.

Answer 8

Whether your suspect simple error, outright fraud, mind control by nefarious evil forces, or whatever, the best way to challenge results is to attempt to replicate them.

If you faithfully follow their methodology, use the same input, and get the same output, then you'll have learned something.

If you get different results, then you can publish those results, and third parties will take note and get involved.