214

There is absolutely no reason that you can't publish a paper that says "We compared our method to methods X and Y. Since code the original code was not available for X and Y, we reimplemented the methods to the best of our ability. The code for these reimplementations is available in supplementary files A and B. Our new method out performed the ...


68

I do not think that this is an ethical question in the first place. Reproducibility is not harmed by the requirement that money needs to be spent on buying software or setting up an experiment (or are there open-source particle accelerators?). While open source solutions are preferable for many reasons, there are clearly cases where using some proprietary ...


56

A chapter that does that is fine; I had one in my thesis. It can for example serve to set the stage for your original work. But someone with a PhD has to be able to do more than just repeat the work done by others, and the thesis has to show that capability. Replications are valuable, but a thesis consisting solely of replications won't show what it needs to ...


54

I think there are two kinds of reproducibility: The ability of someone else to run your code and obtain the same output. The ability of someone else to write their own code that does the same thing as yours based on your description and on examination of your code (reproduction from scratch). The second kind of reproducibility is much more convincing, ...


51

This question has become an important one as the push for greater reproducibility in computational research grows. Use of closed-source software is an acceptable part of research in most fields. Nevertheless, the following viewpoint enunciated by John Claerbout is becoming more widespread: An article about a computational result is advertising, not ...


50

Just publish. Publish your attempts to replicate the findings, documenting the discrepancies, together with the nice results you've obtained by extending their work. Consider sending a draft to the original authors for their comments.


34

Yes, "redoing the work of masters has been seen as a very effective tool for training." but the Ph.D. dissertation is not training! It evidence of what you have learned from the training.


33

Write a paper explaining what the errors are and how they invalidate the results of the papers in question, then submit it to a journal with good visibility and get it published. Writing directly to the journal editors is appropriate only if you have good evidence that the errors in question are deliberate (e.g., the authors have fabricated data in order to ...


29

I have mathematically proven that the central assumptions and claims in that series of papers were wrong and/or incomplete. This is a very complicated statement and it is important to understand it in order to know what to do. The two issues are assumptions and claim. People often make assumptions to solve difficult research problems under the assumed ...


29

People can be dishonest. They can also make honest mistakes and publish bad science. Don't assume that it is you who has an inferior result. And don't assume that a doctoral committee won't believe you. If they are competent to judge you without the earlier results they should be competent to understand what you have done. However, I have two suggestions. ...


28

to finish my degree I need to build methods outperform what is already there No, that is not true. You need to deliver a piece of proper scientific work and advance knowledge and that does not depend on what direction your findings point. Of course, things are easier and more pleasant if your implementation is better. But the actual scientific part of your ...


27

Computer science is a particularly friendly environment for releasing material in advance of publication. In fact, there are a number of methods for doing so that provide a clear archival time-stamp on your work, including university technical reports, arXiv, and big repository sites like github and bitbucket. Moreover, unlike many other disciplines, such ...


26

This will, as it seems nearly all questions on this site, vary based on field. My answer applies to Epidemiology and medical research only. Your mileage may vary. It is very common for this to happen in my field. There has been an increasing emphasis on using meta-analysis and systematic reviews to summarize bodies of work, and with those there is almost ...


24

I think contacting the editor of the journal is your best bet. Contacting grant agencies will most likely not warrant a reply, and I don't imagine many of them have stipulations for sharing code (yet). That said, I have been in a similar position numerous times, and I have had very little luck every obtaining the code. The editor will most likely not be ...


24

As a researcher, I'm generally very keen on including as many implementation details as possible. However: There is rarely the room available in a paper to describe every detail of an implementation. Describing every detail of an implementation in prose often takes substantially more work than writing the program in the first place. The meaning of code (...


23

If there are other reasons to reject the paper, then it's certainly unnecessary to request code/data. If the paper looks like something that might be accepted, then you should ask yourself: Can I certify the correctness and significance of this work, to the necessary degree, with the information that is available? Of course, the key is the phrase to the ...


23

The gold standard is blind, independent replication. However, this almost never happens because it costs a similar amount (of materials and time) and there is less benefit to the researcher who repeats the experiments, so there is little incentive. Next down would be independent replication by another group, then replication within the same group. The bare ...


22

Ask for the code. Please do, however, explain why you want it and what you intend to use it for. Personally, I would be quite happy if someone contacted me about my research, and would try to give all the necessary tools to recreate my data. Especially so if I am not currently working on a follow up piece. That said, I've asked for parameters, codes, ...


20

@posdef's comment is already very good. I would definitely suggest contacting the authors and trying to get exact agreement also on the details that they didn't describe in the original paper. The original authors should be quite interested in an independent replication, so they should have an incentive to collaborate. Depending on how much they contribute, ...


20

Law is concerned with what you do; ethics is more about what is a better choice and why it is better. The answer to your question depends on why you use the proprietary software. If this is the only way of achieving the result, then of course you can and should use it. It does not matter if the code is proprietary — if this is the only way to solve the ...


18

To supplement @David Ketcheson's answer with a "Yes, but..." I agree that there are two types of reproducibility - CrossValidated discusses them with some degree of frequency. There is, as has been mentioned, "Can I click 'Run' and get the same answer you did" reproducibility, which I generally don't find very compelling. There's also "Could I repeat your ...


17

Publishing results that contradict previous publications can be awkward, but if you can show that your method is correct beyond reasonable doubt, then it shouldn't be a problem. No code is guaranteed to be completely free of errors and no result is guaranteed to be correct just because it is published. You don't say much about the nature of your ...


17

There are a few things that I would consider when choosing a data repository: Does it let you release your data under a license you're happy with? Applying too restrictive a license can prevent anyone from doing anything useful with the data, so think about what you're prepared to allow. In particular, remember that most of the research done in academia ...


17

There are multiple answers to your question. Academic publishing assumes good faith. At least in the circles where the majority of people are genuinely interested in the advancement of science. This is the only viable mode of sharing scientific information between adults, although it sometimes fail as we all know it. Publishing should firstly serve the ...


15

Given that you don't have space enough to present all three sets of results, I would suggest that you choose the parameters that best illustrate the scientific point that you are aiming to make. When you state the parameters, however, you should add a parenthetical note or footnote that says something along the lines of: Note that we have chosen these ...


15

I would say it depends: a) If in medicine, you are the first one to replicate an effect in an independent study, then you actually did something original, namely confirming the effect b) If in physics, you refine an experimental design to verify an uncertain outcome, or clean up a measurement to exclude artifacts, and make the effect more clear, it is also ...


13

I'm not sure anyone is supposed to repeat experiments, in any formalized way. In engineering, and I suppose physics, you end up repeating experiments because you are looking to use a method to solve a problem. So if someone has published a method to solve a particular problem, researchers will often try it in order to see how well it works for their system. ...


12

Let me start with a disclaimer. I generally subscribe to the free software community perspective that proprietary software is questionable ethically, and best avoided if possible. I realise this perspective is not commonly held in scientific circles. Having said that, sometimes proprietary software is a necessary, or at least not easily avoided evil, and I'm ...


12

If you consider the data of permanent value and essential to your paper, then you should try to preserve it permanently. The question in that case is not how long to register a domain name for, but rather how to find an organization that's willing to archive the data in the long run. If you don't have that, then it's going to disappear eventually anyway. ...


12

(Disclaimer: I have no personal experience with such a situation, so I'm just going off plain common sense. That said...) It sounds like you've already taken every reasonable step to discover the source of the discrepancy, and you're now left with just an "unexplained deviation" between your results and theirs. You also say that the discrepancy doesn't ...


Only top voted, non community-wiki answers of a minimum length are eligible