217 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

There is absolutely no reason that you can't publish a paper that says "We compared our method to methods X and Y. Since code the original code was not available for X and Y, we reimplemented the ...
user avatar
  • 30.4k
68 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

I do not think that this is an ethical question in the first place. Reproducibility is not harmed by the requirement that money needs to be spent on buying software or setting up an experiment (or are ...
user avatar
60 votes
Accepted

Why isn't it the norm to have research repeated immediately by other academics?

Researchers make or break their reputation on ground-breaking, innovative, and high-impact work. Reproducing the results of another group provides essentially no benefit to any individual group (even ...
user avatar
  • 7,191
56 votes

Why not make students reproduce work?

A chapter that does that is fine; I had one in my thesis. It can for example serve to set the stage for your original work. But someone with a PhD has to be able to do more than just repeat the work ...
user avatar
  • 34.2k
51 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

This question has become an important one as the push for greater reproducibility in computational research grows. Use of closed-source software is an acceptable part of research in most fields. ...
user avatar
41 votes

Author has published a graph but won't share their results table

There is no results table to go with the graph. Use Datathief: https://datathief.org/ Despite the name, in most countries using Datathief is explicity allowed by copyright law. Data cannot be ...
user avatar
39 votes
Accepted

Is it appropriate to re-use data from previous publication for a new study?

Yes, that's not only appropriate, but -- next to replication -- a main purpose of publishing data. Not being able to build on published data would greatly limit the accumulation of knowledge and lead ...
user avatar
  • 34k
34 votes
Accepted

What to do when the claims in a paper are proven to be wrong?

Write a paper explaining what the errors are and how they invalidate the results of the papers in question, then submit it to a journal with good visibility and get it published. Writing directly to ...
user avatar
  • 8,898
34 votes

Why not make students reproduce work?

Yes, "redoing the work of masters has been seen as a very effective tool for training." but the Ph.D. dissertation is not training! It evidence of what you have learned from the training.
user avatar
  • 357
31 votes

Why isn't it the norm to have research repeated immediately by other academics?

experiments re-run by other academics for cases where it is relatively low cost and fast to do so The proportion of novel experiments which can be quickly and cheaply replicated is extremely small ...
user avatar
30 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

People can be dishonest. They can also make honest mistakes and publish bad science. Don't assume that it is you who has an inferior result. And don't assume that a doctoral committee won't believe ...
user avatar
  • 265k
29 votes

What to do when the claims in a paper are proven to be wrong?

I have mathematically proven that the central assumptions and claims in that series of papers were wrong and/or incomplete. This is a very complicated statement and it is important to understand it ...
user avatar
  • 103k
28 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

to finish my degree I need to build methods outperform what is already there No, that is not true. You need to deliver a piece of proper scientific work and advance knowledge and that does not depend ...
user avatar
27 votes
Accepted

Open Source Software and Reproducibility in Computer Science

Computer science is a particularly friendly environment for releasing material in advance of publication. In fact, there are a number of methods for doing so that provide a clear archival time-stamp ...
user avatar
  • 181k
27 votes

Why isn't it the norm to have research repeated immediately by other academics?

Because not all research is interesting or important enough to other people for them to spend their own time and resources on it. Because replication is frequently made difficult when original ...
user avatar
26 votes
Accepted

Can I request the code behind a research paper from the author?

Ask for the code. Please do, however, explain why you want it and what you intend to use it for. Personally, I would be quite happy if someone contacted me about my research, and would try to give ...
user avatar
  • 787
26 votes
Accepted

Can I trust a paper that hides implementation details?

As a researcher, I'm generally very keen on including as many implementation details as possible. However: There is rarely the room available in a paper to describe every detail of an implementation. ...
user avatar
24 votes
Accepted

Ignored Requests Open-Source Software from a Research Group

I think contacting the editor of the journal is your best bet. Contacting grant agencies will most likely not warrant a reply, and I don't imagine many of them have stipulations for sharing code (yet)....
user avatar
  • 3,229
23 votes
Accepted

Who is supposed to repeat experiments?

The gold standard is blind, independent replication. However, this almost never happens because it costs a similar amount (of materials and time) and there is less benefit to the researcher who ...
user avatar
  • 2,749
20 votes
Accepted

What should I do if I cannot reproduce experimental published results

@posdef's comment is already very good. I would definitely suggest contacting the authors and trying to get exact agreement also on the details that they didn't describe in the original paper. The ...
user avatar
20 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

Law is concerned with what you do; ethics is more about what is a better choice and why it is better. The answer to your question depends on why you use the proprietary software. If this is the only ...
user avatar
19 votes

Author has published a graph but won't share their results table

Maybe not very nice, but hardly "misconduct". You might need to reproduce more of their "experiment" to get new data. Their data is their own, I think. It may be that the reviewers ...
user avatar
  • 265k
17 votes

When replicating results, what should I do when the author didn't use the method the paper said they did?

Given that you don't have space enough to present all three sets of results, I would suggest that you choose the parameters that best illustrate the scientific point that you are aiming to make. When ...
user avatar
  • 181k
16 votes

Why are papers accepted even if they don't release code or data to allow reproducibility?

There are multiple answers to your question. Academic publishing assumes good faith. At least in the circles where the majority of people are genuinely interested in the advancement of science. This ...
user avatar
  • 26.7k
16 votes

How to answer a reviewer asking for the methodology code of the paper?

You cannot answer “convincingly”, because your assumption that it is okay not to provide the code is incorrect, and the reviewer is correct to ask for it. And the fact that it’s time-consuming to ...
user avatar
  • 170k
15 votes
Accepted

Can you do a PhD solely on repeating experiments of others?

I would say it depends: a) If in medicine, you are the first one to replicate an effect in an independent study, then you actually did something original, namely confirming the effect b) If in ...
user avatar
  • 3,017
15 votes

What would you do if the author of a published CS article does not respond to code requests?

Simply nothing, but trying to implement it yourself! According to my experience, there would be four potential reasons because of which authors may refuse to share their code: 1- Their code needs to ...
user avatar
  • 6,080
14 votes

Who is supposed to repeat experiments?

I'm not sure anyone is supposed to repeat experiments, in any formalized way. In engineering, and I suppose physics, you end up repeating experiments because you are looking to use a method to solve a ...
user avatar
  • 1,188
12 votes

Author has published a graph but won't share their results table

Authors are not in any way obligated to comply with random requests for additional information about their paper, nor do they even need to respond at all. If they provide what you ask for, it would be ...
user avatar
11 votes

What should I do if I cannot reproduce experimental published results

There is nothing wrong or rude about contacting the author and asking for some help in this matter. You should identify not only who you are, but what your institutional affiliation is (and as I ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible