Questions tagged [reproducible-research]

Questions related to research in which data and/or code is made publicly available so that others may verify the findings and build upon them.

Filter by
Sorted by
Tagged with
84
votes
12answers
12k views

Is it ethical to use proprietary (closed-source) software for scientific computation?

I have data from an ordinary scientific experiment. Now, in order to analyse the data, the most convenient way for me is to use some proprietary, closed-source software. Is it ethical to use such ...
63
votes
5answers
8k views

Why not make students reproduce work?

Typically, PhD theses are expected to have novel results. As science advances, it is becoming increasingly difficult to accumulate enough knowledge to begin a project that pushes the boundaries. ...
61
votes
8answers
5k views

What should you do if you cannot reproduce published results?

I am trying to reproduce published results in a paper. Those results come from numerical simulations. The original authors and I do not use the same software, and theirs is proprietary (I don't have ...
60
votes
6answers
12k views

How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

I am graduate student, to finish my degree I need to build methods outperform what is already there. An issue that I came across with, is that two papers reported way (I mean more than 20%) more than ...
57
votes
7answers
3k views

Popular proprietary program or obscure open source substitute for reproducible research?

For a long time, I had been using open source software for my work to boost "reproducible research". I believed that if I made my codes open source and the softwares to run those codes in were open ...
39
votes
4answers
10k views

Requesting raw data from previously published research

I am working on a mathematical model to help explain some experimental results and generate new hypotheses. Unfortunately, I have neither the resources nor interest to gather experimental data, but in ...
36
votes
3answers
3k views

How to get the data to reproduce a published result?

I'm studying published journal articles on my topic, and I'd like to reproduce their analysis. How can I get a copy of their dataset and enough specifications to reproduce their numerical results? ...
33
votes
4answers
20k views

Can I request the code behind a research paper from the author?

I am in the field of computer science. It is often the case when I am reading a paper I start to wonder, "Wow stunning results, however, I would like to prove that." or "How exactly did he get these ...
30
votes
5answers
1k views

Data publication basics - where, why, how, and when should I publish my unpublished data?

Many researchers have unpublished data. Some of this data may never be published as a manuscript. But I would like to make scholarly contributions of data that I have no intent on publishing, e.g. by ...
28
votes
8answers
4k views

Who is supposed to repeat experiments?

A follow up to this question. There is an issue in science with experiments not being repeated. I asked if PhDs can be done on repeating experiments and the answer is kind of mixed. Most PhDs aren't ...
27
votes
5answers
4k views

What to do when the claims in a paper are proven to be wrong?

What can be done to correct an error that might be hiding a certain author's sole purpose of publishing a paper, no matter the soundness of their method? In a nutshell, the particular case that ...
27
votes
4answers
3k views

Why are papers accepted even if they don't release code or data to allow reproducibility?

Recently, we had a lecture about Reproducible research on of the slides was: Reproducibility: start from the same samples/data, use the same methods, get the same results. Replicability: ...
24
votes
2answers
2k views

As a reviewer, how much raw data or code should you request?

The topic of reproducible research is attracting quite some press these days, yet much remains to be done. In this spirit, I am asking what can be a reviewer's role in this — I'll explain fully below. ...
22
votes
5answers
2k views

Ignored Requests Open-Source Software from a Research Group

For more than 6 weeks now, I have been attempting to contact a post-doc or their (former?) PI to request access to either the source code or software of a tool that was published in BMC Bioinformatics....
19
votes
3answers
2k views

What should I do if I cannot reproduce experimental published results

I found a useful advice if I cannot reproduce computational published results in (What should you do if you cannot reproduce published results?). However I would like to ask the same situation with ...
18
votes
5answers
4k views

Can I trust a paper that hides implementation details?

In Computer Science, sometimes we're assigned to reproduce the results presented by a paper that already have been published. In many cases (most of them) the code is not available, making the task ...
18
votes
3answers
284 views

How to replicate others' results quickly?

Lately, I've been getting the impression that before I can conduct cutting edge research, I have to first master state of the art methods/techniques. Often, this requires spending (i.e. wasting) lots ...
18
votes
4answers
745 views

Is there evidence that open source research software produces more uptake of methods?

In recent years there has been a drive to make science more open. This includes making the software used to perform research open source. The main argument in favour of this idea is that research ...
18
votes
3answers
651 views

Do we know what proportion of scientific reproducibility issues are fraud-related?

A few days ago there was an article on the LSE blog about scientific reproducibility which made little sense to me until I realised they were equating 'low reproducibility' with 'scientific fraud': ...
16
votes
3answers
2k views

Open Source Software and Reproducibility in Computer Science

As a PhD researcher in computer science, my scientific results often take the form of algorithms, which in turn come with implementations. I am convinced that releasing these implementations under an ...
16
votes
6answers
966 views

Licence that converts to GPL after publication but is copyrighted prior to publication

I would like to post an repository on github that includes data and analysis code in R. The analysis and data forms part of a journal article submission. I'm happy for people to view the code and data ...
16
votes
2answers
493 views

I've reproduced a paper: where shall I share the code and findings?

The majority of papers don't share their code. However, I sometimes want to reproduce the results, and therefore re-code the experiments to try to reproduce the papers. Where shall I share the code, ...
15
votes
3answers
1k views

Can you do a PhD solely on repeating experiments of others?

In here, they say there is a problem with results not being replicated. I know a PhD is supposed to be "original" research. But could you do a PhD solely on replicating the results of others? ...
15
votes
4answers
2k views

How does a PhD student go about doing a meta analysis of a research question?

I just thought that I would put this question out here because I am currently working on a meta analysis of a particular research question in my field (HCI/information science). Moreover, this ...
14
votes
5answers
657 views

Has it ever happened that one paper's findings were contradicted by another?

Since reviewers don't check the experimental results by trying to reproduce the experiment, is it possible for someone to submit a paper which basically says "Method X was proposed in paper Y and ...
13
votes
3answers
312 views

Where do you place the link to open data in a manuscript?

Assume you are writing a manuscript and you have uploaded your raw data to a relevant hosting site (e.g., The Dataverse Network, FigShare, etc.). Where in the manuscript should you indicate a link to ...
12
votes
2answers
578 views

How do you validate a scientific paper that uses a non-public cell/animal line?

Recently my lab published a paper that uses a mouse line from another lab. Without going into too much detail since not everyone here is a Biologist, this other lab has a mouse with a reporter gene ...
11
votes
5answers
281 views

Are virtual machines good for reproducibility and open science?

Virtual machines (VMs) provide a way for scientists to package not only scientific software, but also data, external dependences, and even entire operating system configurations, facilitating a ...
10
votes
1answer
414 views

How long should I register a domain name that will be referenced in scientific publications?

I have a website (with data and modeling interface) that will be referenced in a series of scientific publications that are beginning to be published. Currently the website is along the lines of ...
10
votes
2answers
4k views

What is the difference between repeatability, replicability and reproducibility?

I have seen many instances where authors used the term "reproducibility" and "replicability" interchangeably in the social and behavioural sciences. Sometimes, they distinguish between the "...
9
votes
5answers
276 views

Are there any proposals to make sharing code and data with publications a requirement?

Depending on the project, many researchers have to rewrite code and reproduce datasets from previous works in order to perform experimental evaluations. Although some dataset are understood to contain ...
9
votes
3answers
559 views

What exactly would count as copyright infringement in a replication paper?

I have been following the review process of a replication journal In particular, it has an open review which can be seen. On a particular submission they are having a debate about whether copying ...
9
votes
2answers
601 views

Pedagogical pros and cons of requiring undergraduate students to follow open/reproducible research practices?

I regularly teach an undergraduate course in which students are expected to complete projects that involve writing code. I also supervise undergraduates in other project and thesis work that involve ...
8
votes
1answer
178 views

How should I split my code and data between an open repository and the article's supplementary files?

I am working on a paper that involves some data and a piece of code to analyse the data. I want the results of the analysis to be reproducible and the code to be open. This concept of "reproducible ...
8
votes
2answers
191 views

Reproducing work of others - name them as conflicts of interest?

We are currently reproducing several papers in a specific sub-field of computer science under different experimental conditions. When we submit the paper, we will be required to declare people who may ...
7
votes
1answer
2k views

Forum for discussing research paper

I am trying to implement an algorithm given in a paper. I have some questions which I would like to clear out since implementation requires complete clarity on all the concepts/algorithms/arguments ...
6
votes
2answers
247 views

Is it beneficial for a student to publish their research code and data?

There are at least four benefits of releasing the code and data produced during a thesis or research project (citing textually this laboratory): Allows to reproduce figures in the revisions of a ...
6
votes
2answers
291 views

Am I responsible for not reproducibility of collaborators’ result?

Recently, I have ran into some issues about reproducibility of some of my research results which are obtained by our collaborators from different university. In fact, in our current repository there ...
6
votes
2answers
249 views

How to address an author showing replicated results from his paper?

I have replicated a method of interpolation recently proposed by certain author, and I want to tell him that I have done so. I suppose the author is interested in knowing that students are engaged ...
6
votes
3answers
153 views

Are there any experiment reproducibility databases?

Quite often when I read a medical research article I wonder whether other researchers tried to reproduce the experiment. I could go through the list of papers that cite the article but I find this ...
5
votes
2answers
1k views

When replicating results, what should I do when the author didn't use the method the paper said they did?

I am replicating a model to report on how well it does on a new evaluation metric I am proposing. The method/hyperparameters of the model are given in a paper about it, where it breaks all benchmarks ...
5
votes
3answers
223 views

Is it acceptable practice to make the source code of a re-implemented paper available online

It is the usual practice that researchers re-implement others' work for the purpose of comparison or result verification. If for instance researcher A re-implements researcher B's paper and improves ...
5
votes
4answers
522 views

Ethical considerations in publishing raw data

Apart from taking measures to maintain confidentiality, do I need my ethical approval to specify permission to publish raw data along with my study? I'm trying to facilitate reproducibility in my ...
5
votes
1answer
151 views

Ask the author of a paper for suggestions of where to publish replication results

I based my thesis on a published paper and ask its original author if he was interested in seeing the results. After approaching him (with your help provided: How to address an author showing ...
4
votes
1answer
307 views

What is the best philosophy and practice to use smartphones for academic life? [closed]

I am doing research in theoretical physics, and have learned that smartphones can bring in a lot of distractions to your life if you want to focus on your research and study. Like facebook, twitter ...
4
votes
2answers
185 views

How to deal with non-reproducible research?

For context, I am a PhD student in bioinformatics, working partly in development of new statistical models and partly in applying models to data. I recently came across a paper which describes an ...
4
votes
1answer
341 views

Where can I find examples of research misconduct reports?

I am writing a report documenting evidence for research misconduct. The report is to be submitted to the ethics board of a university. Are there publicly available examples of such reports? Not the ...
4
votes
1answer
121 views

Reproducible science: do entities exist requiring Level 3 of TOP standards?

Quite many (if not most of) scientific articles, related to computational sciences, which I have seen, do not share both data and code leading to the figures. This may have of course good reasons ...
4
votes
2answers
258 views

Reproducible and exact calculations vs. approximate and slightly random results in experiment

When experiments are performed and reported in an academic article, I learned that reproducibility is important. I am implementing a system where I stand between the choices to use GPU acceleration to ...
3
votes
2answers
187 views

How can we perpetuate reproducibility?

Two years ago, Roger Peng discussed in his blog the ramifications of requiring reproducibility for the first producers of research objects, but not for reusers. He then proposed What I think a good ...