219 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

There is absolutely no reason that you can't publish a paper that says "We compared our method to methods X and Y. Since code the original code was not available for X and Y, we reimplemented the ...
Ian Sudbery's user avatar
  • 38.3k
69 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

I do not think that this is an ethical question in the first place. Reproducibility is not harmed by the requirement that money needs to be spent on buying software or setting up an experiment (or are ...
CrepusculeWithNellie's user avatar
64 votes
Accepted

Why isn't it the norm to have research repeated immediately by other academics?

Researchers make or break their reputation on ground-breaking, innovative, and high-impact work. Reproducing the results of another group provides essentially no benefit to any individual group (even ...
Andrew's user avatar
  • 8,260
56 votes

Why not make students reproduce work?

A chapter that does that is fine; I had one in my thesis. It can for example serve to set the stage for your original work. But someone with a PhD has to be able to do more than just repeat the work ...
Maarten Buis's user avatar
  • 43.7k
52 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

This question has become an important one as the push for greater reproducibility in computational research grows. Use of closed-source software is an acceptable part of research in most fields. ...
David Ketcheson's user avatar
44 votes
Accepted

Is it okay to upload code I wrote for replicating someone else’s simulation study?

You ask about uploading three different things, so it's worth answering with regard to the three parts. Your own code: Definitely acceptable to upload it. You wrote the code so the code is your ...
CrimsonDark's user avatar
  • 11.2k
41 votes

Author has published a graph but won't share their results table

There is no results table to go with the graph. Use Datathief: https://datathief.org/ Despite the name, in most countries using Datathief is explicity allowed by copyright law. Data cannot be ...
Anonymous Physicist's user avatar
39 votes
Accepted

Is it appropriate to re-use data from previous publication for a new study?

Yes, that's not only appropriate, but -- next to replication -- a main purpose of publishing data. Not being able to build on published data would greatly limit the accumulation of knowledge and lead ...
henning's user avatar
  • 35.1k
34 votes

Why not make students reproduce work?

Yes, "redoing the work of masters has been seen as a very effective tool for training." but the Ph.D. dissertation is not training! It evidence of what you have learned from the training.
user61975's user avatar
  • 357
32 votes

Why isn't it the norm to have research repeated immediately by other academics?

experiments re-run by other academics for cases where it is relatively low cost and fast to do so The proportion of novel experiments which can be quickly and cheaply replicated is extremely small ...
Anonymous Physicist's user avatar
30 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

People can be dishonest. They can also make honest mistakes and publish bad science. Don't assume that it is you who has an inferior result. And don't assume that a doctoral committee won't believe ...
Buffy's user avatar
  • 365k
28 votes

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

to finish my degree I need to build methods outperform what is already there No, that is not true. You need to deliver a piece of proper scientific work and advance knowledge and that does not depend ...
cbeleites unhappy with SX's user avatar
28 votes

Why isn't it the norm to have research repeated immediately by other academics?

Because not all research is interesting or important enough to other people for them to spend their own time and resources on it. Because replication is frequently made difficult when original ...
Nullius in Verba's user avatar
27 votes

Are experiments described in scientific papers *actually* peer-validated today?

There are two related but separate processes in play here. On the one hand, papers are peer reviewed, i.e., when you submit a paper describing your experiment, results and conclusions to a conference ...
Stephan Kolassa's user avatar
24 votes
Accepted

Who is supposed to repeat experiments?

The gold standard is blind, independent replication. However, this almost never happens because it costs a similar amount (of materials and time) and there is less benefit to the researcher who ...
arboviral's user avatar
  • 2,799
21 votes

Are experiments described in scientific papers *actually* peer-validated today?

When laypeople are introduced to the idea of replication, they imagine scientists constantly redoing every paper that comes out, to "check it". At least I did. This is not the case in most ...
gomennathan's user avatar
20 votes

Is it ethical to use proprietary (closed-source) software for scientific computation?

Law is concerned with what you do; ethics is more about what is a better choice and why it is better. The answer to your question depends on why you use the proprietary software. If this is the only ...
Dmitry Savostyanov's user avatar
20 votes
Accepted

Are experiments described in scientific papers *actually* peer-validated today?

Others have discussed the official process. Generically, the thing is, the process is messy. There is no megalithic authority presiding over science, nor academia in general. We don't have a "...
Boba Fit's user avatar
  • 1,859
19 votes

Author has published a graph but won't share their results table

Maybe not very nice, but hardly "misconduct". You might need to reproduce more of their "experiment" to get new data. Their data is their own, I think. It may be that the reviewers ...
Buffy's user avatar
  • 365k
17 votes

Is it okay to upload code I wrote for replicating someone else’s simulation study?

Other answers here seem to focus on the issue of copyright. I will add that it is also a good idea to publish your code and results in the interest of documenting replication of the original study. ...
Thomas Arildsen's user avatar
16 votes

Why are papers accepted even if they don't release code or data to allow reproducibility?

There are multiple answers to your question. Academic publishing assumes good faith. At least in the circles where the majority of people are genuinely interested in the advancement of science. This ...
Cape Code's user avatar
  • 27.1k
16 votes

How to answer a reviewer asking for the methodology code of the paper?

You cannot answer “convincingly”, because your assumption that it is okay not to provide the code is incorrect, and the reviewer is correct to ask for it. And the fact that it’s time-consuming to ...
Dan Romik's user avatar
  • 190k
16 votes

How does “reproducing other labs' results” work?

With limited time & resources, it seems researchers would stick to their own project, however adjacent new results may be. Well, yep, there's the problem. Most labs have funding to perform a new ...
Azor Ahai -him-'s user avatar
16 votes
Accepted

How much should I tidy up code before I share it?

I want you to imagine yourself in a few years from now. You're writing up your PhD thesis and need to check that result you presented in that undergraduate conference back in 2023. You can't really ...
astronat supports the strike's user avatar
15 votes
Accepted

Can you do a PhD solely on repeating experiments of others?

I would say it depends: a) If in medicine, you are the first one to replicate an effect in an independent study, then you actually did something original, namely confirming the effect b) If in ...
Sascha's user avatar
  • 3,700
15 votes

Who is supposed to repeat experiments?

I'm not sure anyone is supposed to repeat experiments, in any formalized way. In engineering, and I suppose physics, you end up repeating experiments because you are looking to use a method to solve a ...
Arnfinn's user avatar
  • 1,198
14 votes

How does “reproducing other labs' results” work?

The other answers are so far covering situations where some result is reproduced because of doubts. But there are also situations where the published result allows (is the only or presumably best ...
cbeleites unhappy with SX's user avatar
12 votes

Who is supposed to repeat experiments?

Those whose research benefits from it While it would be nice to repeat experiments, in practice we as a society don't really want1 scientists to repeat them just because, we want them to create new ...
Peteris's user avatar
  • 8,251
12 votes

Author has published a graph but won't share their results table

Authors are not in any way obligated to comply with random requests for additional information about their paper, nor do they even need to respond at all. If they provide what you ask for, it would be ...
NotThatGuy's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible