0

To be honest, I'm not sure how exactly to word my question in order not to cause any miss-understanding.

During my literature review sometimes I encounter with researches which propose an approach with promising results. Then when I try to use it or to investigate its theoretical principles I notice it is not as straightforward as it was mentioned in the paper to expect such great results. Then I contact the guy asking for more details about the method or even for the code to implement the approach and reproduce the results, but I hear no answer from them or some thing like "sorry we do not know where the code is now!" or "we have no right to give the code" or some vague answer which is not helpful.

Well, these papers always have tables and numbers related to real data to support their hypothesis, but sometimes I wonder if it is even rarely possible than the authors were faking something? or I'm totally wrong about it?

Even some of these papers are published in high ranked conferences which makes it harder to be skeptical about them!

So my question is how much should we trust works of others in academia even if they published their work in well-known conferences?

p.s: my field is computer science and mostly about algorithms and methods and the implementation results which can support the claimed performance for the algorithm.

  • 1
    Welcome to the reproduciblilty problem in academia. It's a well-known problem. – tonysdg Nov 3 '16 at 18:08
  • I'm not sure what your question is. Based on your 2nd-to-last paragraph: Are you asking whether you should suspect data fabrication if you find that some idea is not as straightforward as presented in the paper, or if the researcher says they cannot share their code with you? Or something else? Please edit your post to clarify. – ff524 Nov 3 '16 at 18:10
  • My question is how much should we trust works of others in academia even if they published their work in well-known conferences? – Bob Nov 3 '16 at 18:18
  • 1
    @Bob: Are you also assuming that you're replicating their entire experiment? Down to the hardware they used, the system and non-system libraries they used, etc.? – tonysdg Nov 3 '16 at 18:28
  • 1
    @DilipSarwate I think you are asking something completely unrelated to what Bob is asking. If you want to ask about quality control of conferences vs journals, you should ask a new question. – ff524 Nov 3 '16 at 19:14
1

I think there is more than one type of "trust" here: "trust" as in whether to believe the authors are being honest, "trust" in terms of the state and certainty of knowledge in a field, and "trust" in the value of the peer review system.

Although misconduct is possible, and it is worthwhile to consider, another possibility is that the approach used in a paper relies on assumptions that were either taken for granted or are stated in the paper but you have overlooked them. For a signal-processing algorithm, for example, there may be an assumption that external noise has a particular color (white, pink, etc) - with a different color noise, or with real world signals, the algorithm may not work as well; for code optimized for a particular architecture you might get different results with different hardware or a different software environment. You shouldn't "trust" that a result in one context will apply to all other contexts that have not yet been tested.

Peer review is meant to improve the quality of and confidence in published material - it is not a flawless fact-checking system. Even when everything in peer review goes perfectly, the best you can assume from a peer reviewed paper or presentation is that the methodology is sound and the conclusions are reasonable based on the data. You should never "trust" a published work as the last word on a topic, and be wary of far-reaching claims - these are often stated for the potential of the technique, not necessarily the implementation as presented - these statements are intended to gain support for further work on the topic, so feel free to treat them with some skepticism. I wouldn't classify any of these as misconduct, just a potential for miscommunication between what is written and how you are expected to interpret what is written.

Lastly, the issue of sharing code differs by field, but at least in the biological sciences it has become standard for journals to ask authors to make analysis code and raw data available on request. I am not sure about the norms in computer science, and it certainly might depend on the potential for commercial application. It seems unlikely that someone would truly have "lost the code" to a recent paper, but if you are going back more than a couple years it is certainly possible that was indeed the case.

It is also possible, if you are a student, for example, that the authors felt your request was bothersome. How did you ask? "Can I see your code?" is a lot different from "I implemented the algorithm you published in 2014, and I got a different result from what you published. Here is the code I used - would you be able to share your original code, or have a look at mine to help find the discrepancy?" If you are truly skeptical, this is the approach I would suggest - science would benefit greatly from these types of replication and confirmation attempts.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.