3

For the sake of high-quality science diffusion, there is no doubt that reproducible research is a good practice as it helps to demonstrate the robustness of the proposed methods and boosts the possibilities of getting more citations for a published paper. Needless is to say that it also allows new researchers to continue the work of a previous mentor. For students wondering if they should or should not release their code in a public repository, this was a topic already discussed here: Is it beneficial for a student to publish their research code and data?.

However, my understanding, from this material, is that reproducible research is a practice followed by institutions in which the government takes charge of the salary earned by researchers, their equipment, and travel expenses. Given that public funds are used in this context, there is no point in publishing results in places that would keep the knowledge inaccessible to the people who need it (and pay for it with their taxes). In fact, many journals offer now the possibility of publishing papers as Open Access. Obviously, this exempts the case of national security affairs that need to be kept in secret.

For independent researchers or non-granted students, the things change a little bit, as they don't have to follow the reproducible research practices because they pay for most (if not all) of the expenses during their studies. My question is, what are the incentives (if there is any) for this group of people to release their code on the internet if they are not looking to exploit it commercially? Is reproducible research something that they can use to improve their chances to get a job in academia or an R&D department in the industry?

Note: by reproducible research, I am taking the definition stated here: all the results from a paper can be reproduced with the code and data available online.

  • I'm not sure what you mean by reproducible research. It reads as if you mean being able to independently run a given program on a given dataset to verify the results. I think a better definition requires much more. Can the hypotheses of a study be accepted via completely independent methods, data, and models? It isn't about the honesty of the first reporter, it is about the likely truth of the statements. Running the same code on the same data isn't independent in that sense. Please explain a bit more what your understanding is. – Buffy Nov 22 '18 at 16:34
  • Unfortunately, though you ask a valid question, I think the definition is too narrow and flawed. There is a problem in the scientific world that many reported results can't be validated based on further study. While making the process transparent (which is what your definition enables) it isn't quite the same thing. Even scientific studies done at 95% confidence, if done well, will report the wrong result 5% of the time, by definition. One learns more by, say, running a given code on a different data set, or different code on a similar-not-identical set. IMO, the term used is misleading. – Buffy Nov 22 '18 at 16:48
  • Oh, I see @Buffy. The definition of reproducibility that the LCAV provided does not consider the confidence degree that you mentioned. Perhaps, can we talk about soft reproducible research? Anyway, does it represent an incentive for independent researchers? – JFonseca Nov 22 '18 at 18:09
  • 1
    @JFonseca: Sorry, my comments appear to have been quite unclear. "how bizarre is to ask for incentives when doing reproducible research" - that's not bizarre at all. As I said, the incentives are just probably neither the institution's practice nor the expectation to keep a marketable concept secret. "he should just release his code for the sake of the humanity" - no, as I wrote, it may happen for an increase in reputation or becoming more well-known in one's field. As for your last sentence, I'm afraid I do not understand what you are trying to say. – O. R. Mapper Nov 22 '18 at 22:25
  • 2
    You shouldn’t need incentives to do “reproducible research”, you should do it because it’s good science and doing good science (and showing that you can) is good for your career. Open Access for tax-funded research is important too but really a separate discussion. You should be ensuring your results can be reproduced even if you’re in a private company or publishing high impact papers. – Tom Kelly Nov 23 '18 at 10:56
1

I hope I understand the intent here well enough to offer guidance. I don't think there is any inherent incentive for most researchers to publish more than they do. The current system works well enough for them, so the added effort makes little sense.

The reason that many US government agencies publish more than the minimum is that much (most?) government funded research here has, traditionally, been considered as owned by the American public. Public Domain, if you like. On an orthogonal dimension, the fact that a lot of science has become politicized, implies that many want to be as transparent as possible so that the inevitable conflicts and charges can be openly refuted. But an additional reason is that "public servant" researchers are also generally interested in others being able to leverage and extend the work they do - for the greater good. They don't work for a profit motive and so have little incentive to hold things close.

If individuals want to publish code and data, it is, for the most part, permitted, as long as journals don't impose restrictions for reasons of copyright, etc.

However, a beginning researcher should carefully consider what openness implies. In particular, while you retain copyright (which you can license), you complicate the possibility of obtaining patent on your ideas. It may even be that others, with whose policies you disagree, can use and monetize your ideas against your wishes. Openness in general is good, and being willing to "reveal all" is commendable, but be aware of the implications when you do.

  • "I don't think there is any inherent incentive for most researchers to publish more than they do." .. or would like to. I think many researchers would like to publish quite a bit more, if just they had the time and other resources available. – O. R. Mapper Nov 22 '18 at 20:58
  • Thanks, @Buffy and O.R.Mapper, is there any standard way to determine by beforehand if a journal is fine with you if you decide to release your code on the Internet? I am asking because it may exist some conflict of interest if your code is also used to generate the figures included in the paper. – JFonseca Nov 23 '18 at 14:32
1

My question is, what are the incentives (if there is any) for this group of people to release their code on the internet if they are not looking to exploit it commercially? Is reproducible research something that they can use to improve their chances to get a job in academia or an R&D department in the industry?

Whether independent or not, the incentives for publishing your software code and/or data resources are:

  • Some conferences/workshops encourage the submission of papers devoted to new code/resources (i.e. a quite easy publication).
  • Assuming your code/resources can be found, gain popularity and are useful to the community, you can get a lot of citations: every time somebody uses them, they will refer any paper you ask them to cite (this must be clearly stated in the license info).
  • Mentions of your open-source software / open-access datasets in your CV are seen positively in the academic community (for the reasons that you mention), so it increases your chances to get an academic job. It can also be seen as a showcase of your skills when applying in the industry.

So overall yes, it does strengthen your research profile.

However it can take a lot of time: polishing the code, writing some clear documentation, maintaining the software, etc. So there is a trade-off between devoting time to this or to other research activities. I assume that this question is especially important if you are an independent researcher.

Edit following OP's comment: for independent researchers it can be a good way to keep in touch with the research community, especially if you don't have the time or financial resources to go to conferences.

Disclaimer: my field is Natural Language Processing, which involves Machine Learning with unstructured data. Results often vary depending on the dataset and the exact experimental setup, and reproducing an experiment can take a lot of time. Thus reproducibility is a major issue in the field and open access resources are strongly encouraged.

  • Thanks @Erwan! It is also a way to keep in touch with the researching community (when your job and career do not match) and receive valuable feedback to improve your contributions. – JFonseca Nov 23 '18 at 3:25
  • Good point, I'll add it to my answer. – Erwan Nov 23 '18 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.