8

In academia, researchers live by the mantra of "publish or perish". This is because the contribution to society that academia gives is the expansion of human knowledge.

However, given the importance of software to conduct research, I don't understand why implementing many algorithms or building API's for research purposes is not equally relevant when trying to get funding.

See, for example, how few papers in math, computer science and software engineering provide source code, despite the entire focus of the paper being on the implementation of an algorithm and its performance in real hardware.

Moreover, see how much research relies on API's. We have linear algebra API's, calculus, computer visions, physics simulations, etc. Many of which are paid software (e.g., Matlab).

If someone is willing to spend time developing an API for a given subject (a computer vision library, a rendering algorithm, quantum physics / computing simulations) and that software provides new and useful tools in open software and for free so that researchers can conduct their research more efficiently, I don't see how this is not equivalent in usefulness as making a paper.

  • 9
    In fields like statistics many of the best-cited works are software implementations of algorithms, vignettes on those implementations are published like any other paper, and the authors of the most popular ones are probably known best by those contributions. A professor in neuroscience I knew as an undergraduate advised that the best career move he made was sharing a piece of data analysis software which quickly spread in his field and drew a lot more attention to his work than any other publication. In summary, I disagree a bit with the premise here. – Bryan Krause Oct 4 '18 at 20:11
  • 1
    Why do you want to be an academic? If you’re good and passionate about implementation you should look for a programming job that will reward exactly that. – Cape Code Oct 5 '18 at 3:03
  • I love both things. But mostly, what I enjoy the absolute most is learning an algorithm and implementing it from scratch to fully understand it and analyze it. I also love doing research. But most importantly, I feel there is a huge lack of open software resources for science. And I believe this is a problem that needs to be fixed. – Makogan Oct 5 '18 at 3:06
  • 1
    I've written a longer answer below, but the premise that scientists can't get credit for their software is wrong today. – Wolfgang Bangerth Oct 5 '18 at 3:55
  • 1
    1) many software has publication attached, which can be among the highest cited papers ; 2) publications are about noteworthy results of research, while coding already existing algorithms is something totally different. Apples with oranges. – Greg Oct 6 '18 at 17:40
3

Software is a research product has value and is seen as valuable, but you as a researcher need to ensure that you extract the value out of it. Extracting value out of any research product requires doing some kind of formal write up and distributing the ideas among the relevant audience. For software, this is many times in the form of a software publication. Many traditional scientific computing journals take a software paper and these papers can do quite well in terms of citations. For example, Shampine's "The MATLAB ODE Suite" is one of the most read and cited articles in SIAM Journal of Scientific Computing.

Taking that paper as a reference, notice that it identifies the problem it's solving, explains how it had to modify the algorithms to achieve its goals, and the kind of results it can achieve. This then gives a citable resource and many of the people who use the paper will then cite this source. Software tends to be under cited, but since it can get so widely used you will find that successful software has some of the most cited publications. Another classic journal like this is ACM Transactions on Mathematical Software (ACM TOMS). More recent software journals are the Journal of Open Research Software and SoftwareX.

In a slightly different direction, the Journal for Open Source Software reduces the importance of the paper by making the software and its documentation part of the peer review. As an example, Optim.jl has a JOSS paper which you can see is much shorter than traditional academic papers, but at least summarizes the value of the software in a citable form.

When applying for jobs, you'll notice that people naturally assign a different value to software papers. While it is definitely changing, some professors I know take hardline stance that software is just implementation and doesn't extend knowledge and will value it less than other papers (which is why these exact points should be refuted in a publication about said software). In contrast, some individuals care more about impact or citations, and so a widely used software will have a very large effect on them. Generally in my experience, the value of software seems to be increasing over time, with senior professors more likely to take the former stance and newer professors more likely to take the latter.

But my advice would be to just do whatever you think has a high impact to the research community. If it does have a big impact, people will notice, and sometimes just being noticed is what you need.

  • Thanks for the advice, this answer seems well balanced and rounded – Makogan Oct 6 '18 at 16:32
4

I am not sure what field you are in, but to me, this question almost doesn't make sense.

Nothing is equivalent to a paper ... except a paper. You may as well ask why doing research without publishing it isn't equivalent to a paper. That's just not the model we have.

In my fields, if you publish a library/toolbox, you write a paper on it. If you put up an open dataset, you write a paper on it. If you do research, you write a paper on it. There are many people who focus on developing toolboxes, and many people who might never touch development at all and will only ever publish "actual" science.

In other words, it's not validated and "counts" until it's been at least somewhat peer-reviewed.

  • 1
    'In other words, it's not validated and "counts" until it's been at least somewhat peer-reviewed.' - I think this statement just leads to the obvious follow-up question that is not answered in your answer: Why don't we have a publication mode that includes peer-reviewing libraries/toolboxes? – O. R. Mapper Oct 5 '18 at 7:28
  • 1
    @orm We do? At least my field does. They're called journals – Azor Ahai Oct 5 '18 at 14:52
  • 1
    @O.R.Mapper Oh, I see what you're asking. No, I don't think we're there yet. I don't know how big libraries/toolboxes are in your field, but in mine, they'd never get fully peer-reviewed because they are too big. They're usually up on GitHub, though. – Azor Ahai Oct 5 '18 at 15:16
  • 1
    Of course, any such work (including convential papers) is only useful on a detail level to its target audience. Likewise, someone who doesn't even remotely use libraries probably wouldn't be reviewing them, at least not for the aforementioned aspects. – O. R. Mapper Oct 5 '18 at 15:53
  • 2
    @O.R.Mapper there are journals like that. The Journal of Open Source Software (JOSS) is one leading example where the paper is short but the code and its documentation are considered part of the paper. So this is a journal which performs peer review of software and toolboxes, and has seen a lot of adoption among scientist open source developers. – Chris Rackauckas Oct 6 '18 at 15:45
4

You are mistaken in your assumptions. Scientific software is valued and there are now in fact a number of journals dedicated to exactly this -- I know this among other reasons because I happen to be the Editor-in-Chief of the ACM Transactions on Mathematical Software (ACM TOMS). There is (actually: was) also the Computational Science and Discovery journal, and then there is Computing in Science and Engineering (CiSE). All of these publish articles that describe computational software.

More generally, there are substantial efforts at making sure that the authors of scientific software get credit that is comparable to other kinds of publications such as journal papers. I would encourage you to take a look at the WSSSPE series of workshops, the Force11 task force, and a number of other efforts to ensure that software can be cited in much the same way as papers can (e.g., via the Zenodo service).

3

There are different kinds of value. And different people measure it differently.

Scientific research resulting in traditional publications is about extending knowledge. The value is in the knowledge.

Implementation is normally about building a product of some sort, whether commercial or not. But to implement something for which the knowledge is already present, doesn't extend knowledge, so isn't the same sort of value as research resulting is new ideas. It isn't less valuable, just differently valuable. An implementation may enable something else to be done, without extending knowledge. Building out scientific libraries feels like this sort of thing.

But you are wrong in some ways in your assumptions. Some people get at least a local reputation boost by building stuff. That is certainly true in IP based businesses such as Google, but also in some universities. It isn't either-or. I've been given props for some of the stuff I've built whether it was, strictly speaking, research or not. My stuff mostly enables students to learn something, rather than knowledge extension per se.

But also, a lot of doctoral research in CS is, in fact, based on implementation. One famous doctoral thesis explains how to build a compiler based on the semantics of a language rather than (the usual thing) it syntax. It was a big deal. But Peter Lee, had to build it to demonstrate that the idea was sound. The implementation wasn't the dissertation, but the research and the extension of knowledge that the implementation proved sound was. Also in the compiler field, if you have an idea to make a garbage collector for a new language run 3% faster and use 3% less memory than the best known solution you can get a doctorate. But you have to build it and measure it (extensively) to prove the idea.

So, the implementation may not be, in itself, the valuable thing, but it enables the verification of a new idea - knowledge extension.

  • However things like the encyclopedia, survey papers, books... Are all built around existing knowledge without extending knowledge. And all of those come from academia as well. Moreover, outside of the sciences, for example in literature. You are not really extending more knowledge, rather adding a new analysis of an author or genre, which adds knowledge without really extending it. It seems to me there is a lot of benefit in encouraging academics into developing better research tools, not only extending knowledge. – Makogan Oct 4 '18 at 19:52
  • 2
    @Makogan And nobody becomes professor by writing an encyclopedia. – user9646 Oct 4 '18 at 19:59
  • 5
    @Makogan And (almost I guess) every professor uses a chair to sit. – user9646 Oct 4 '18 at 20:13
  • 2
    @Makogan I think my chair is the one object that I use the most (together with my desk). And I disagree that professors are the best people to make encyclopedia. Some professors yes, others, not so much; and non-professors can be just as qualified. There is a non-trivial aspect of vulgarizing in writing encyclopedias. – user9646 Oct 4 '18 at 20:42
  • 1
    This is the traditional standpoint in mathematics departments, and I find it's very wrong/incomplete. The implementation is usually a source of its own research and results. BUT, the researcher needs to take the steps to market said software for this value. Many times they might write the first real optimized implementation of some method: having benchmarks of production-quality implementations can be ground breaking for the field and provide new directions. However, the researcher needs to take steps to share this kind of knowledge. Software-based journals are good outlets for this research. – Chris Rackauckas Oct 6 '18 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.