19

Developing useful software products is becoming an increasingly important research activity in many fields. However, it seems the proper credit/incentive system has not been fully worked out.

The situation becomes particularly problematic when large and complicated software products depends on many individual components developed by different research teams.

Consider a situation in which a researcher spend years of effort to develop a software component C that solves a very specific but very difficult fundamental problem. This researcher then publishes a paper on C.

Another group then develops a more general software B in a few weeks, which is a thin wrapper for C. It basically reduces a more general class of problems to specialized problems that C can solve and then feed to C. With everything being open source, B can directly incorporate C into itself in various ways. Let's assume this is done legally, and the README file in B contains flattering acknowledgement to C. This group then publishes a paper on B, which properly cites the original paper on C. (Best possible situation)

Fast forward a few years. Since C solves only very specific problems, no one uses it directly (other than B). The more general class of problems that B can solve (thanks to C) happened to become a hot topic. And the citation tally is now:

  • C: 1 citation (just the paper on B);
  • B: 1200 citations.

This seems like a terrible situation for the author of C. And this is not purely hypothetical. I have seen this happen to several people already.

More broadly, this credit/incentive structure may be bad for a field. In particular, any one in tenure track position probably shouldn't be developing software for solving specific but fundamental problem (even though they are likely in a career stage in which they are most capable of doing exactly that).

My question is, taking what I described as given (I know it happens, so there's no point explaining why such situation does not exist), how original developer of fundamental software components could position themselves (other than keeping things closed) to get proper credit?

More broadly, how can a field setup proper structure to incentivize the development of such software?

10
  • 4
    Why is recognition in the form of citations important to you? Why don't you write B1, B2, ... BN yourself? Lots of papers only have one citation. Write more software. People who only write one paper don't normally get a lot of citations.
    – Buffy
    Dec 9, 2021 at 20:59
  • 1
    Maybe the insight that this software can be used generally is actually worth more than the software itself. But if that is not the case here: fortunately there are more and more scientists who can judge these situations correctly instead of only counting citations.
    – Louic
    Dec 10, 2021 at 6:10
  • 22
    "The whole thing that makes a mathematician’s life worthwhile is that he gets the grudging admiration of three or four colleagues." -- Donald Knuth, Computer Scientist
    – usr1234567
    Dec 10, 2021 at 7:51
  • 4
    One idea is to develop a measure, similar to Google's PageRank algorithm, that measures papers not only by their direct citation count, but also by indirect count. So if C is cited by a highly-cited paper, the rank of C will increase too. Dec 10, 2021 at 12:13
  • 2
    If B was so important, and follows immediately from C, then the researcher should have made B themselves. I'm sure you're familiar with the concept of low hanging fruit. Do you expect some kind of reward for neglecting to pick it? On the other hand, if perhaps B was not obvious, or B was more work than you imply, why should C get the credit for that work?
    – Taw
    Dec 10, 2021 at 20:55

6 Answers 6

26

Create a culture where evaluation of researchers is done by expert judgement rather than crude benchmarks.

9
  • 7
    Sorry, I don't follow. Say more?
    – Buffy
    Dec 9, 2021 at 21:01
  • 3
    The experts in the field know that C is more important than B - so if evaluation is done by expert judgement than C gets more credit than B. Dec 9, 2021 at 21:32
  • 19
    Yes, if everyone did the right thing then the world would be a better place. However, ...
    – Buffy
    Dec 9, 2021 at 21:34
  • 1
    @Buffy I agree, that’s why I upvoted this answer for proposing that people (and universities) should do the right thing, and for explaining what “the right thing” is.
    – Dan Romik
    Dec 10, 2021 at 0:21
  • 4
    Well, as much as I think that this is how it should be, this does not address the question, namely how the "original developer of fundamental software components could position themselves (other than keeping things closed) to get proper credit". This is something that the community should be doing. The developer with the 1 citation probably doesn't enough influence to make a noticeable difference. Also, it is reasonable to assume that these pea-counting habits will remain while hiring committees do not have the time to evaluate candidates extremely carefully...but will this ever change?
    – DCTLib
    Dec 10, 2021 at 9:52
6

One thing is getting journals (and referees) better at referencing software dependencies. I've seen similar situations where large code B depends on some code C, everyone cites B but forgets to cite C. The only way is start yourself, making sure sure to cite everything, then when you referee papers, if they only cite B then request they cite C as well.

At the end of the day people should be citing the relevant literature, when they don't then your job as a referee is to request they do. Also when giving talks make sure to be more explicit that while you used B, that it is based on C to remind people about the link.

1
  • Isn't this a chicken-and-egg problem? Most people will know the existence of C. That's the problem. If C is already well known, then of course there is no reason to count citations.
    – Bilbo
    Dec 10, 2021 at 20:17
2

how original developer of fundamental software components could position themselves (other than keeping things closed) to get proper credit?

This is a very subjective question, since "proper" will be defined differently by different people. In the original Q OP mentions citations.

But citations are not even a measure of how useful something is. It is a combo of visibility of the original paper, PR campaign, having friends in the field, and how useful is the final product. That is not bad, because awesome but invisible or unusable software should not be rewarded.

So, the original developer should realize that current scientific system rewards very specific behaviors, where generating valuable products is only part of the equation.

If the want to develop fundamental components, they should:

  • integrate their work into larger eco-system through collaborations
  • spend time not only coding, but building community around their solutions or ideas
  • they should not ship just the bare-bones solution, but a valuable toolkit that can be applied, or at least a protocol that will help others develop applications.
0

Perhaps you should look at your software license. Why shouldn't it have a clause requiring that software that is using it require that papers that cite that (using) software also cite it?

If there was sufficient interest in a project group, or the community at whole, an open source software promoting organization, or a university, could task lawyers with creating such a license and the whole community would benefit.

(It would be not much different than, say, current BSD, which requires all using software, when distributed, to properly cite the copyrights of all BSD-licensed software it depends on, directly or indirectly.)

8
  • 1
    Is there any example of license? I cannot think of any. Also, if people don't even know about C, they certainly won't know what its license says.
    – Bilbo
    Dec 10, 2021 at 20:52
  • 1
    If people are using C in their work sufficiently to cite it they will also know what its license says. If not, that implies a great deal of disrespect of software and licenses in the academic research world which does not occur in the commercial world, where this is taken seriously. And would be surprising, to me, because the academic world is itself very concerned with plagiarism.
    – davidbak
    Dec 10, 2021 at 20:54
  • I don’t think you understand how academic citations work. Authors cannot be legally mandated through a software license to cite someone’s work that they do not wish to cite. Such a requirement would be seen as blatantly unethical and would be completely unacceptable to both paper authors and journal editors. Personally if I ever ran across any software trying to impose such a requirement it’ll go straight into my trash bin, never to be looked at again.
    – Dan Romik
    Dec 11, 2021 at 8:33
  • @DanRomik - I do understand how academic citations work. And your choice to not use software that carries such a license is indeed your choice. Just as it is for the many commercial companies who also choose to use or not use software depending on (among other things) its license. In this particular case, it would mean you couldn't use derived software B. And since B's own creators accepted foundation layer C's license before building on it - that would be fine with them too. And since it is fine with you: no problem. If you choose to use it and NOT cite it: whose ethics is implicated?
    – davidbak
    Dec 11, 2021 at 17:30
  • @davidbak a citation is a specific kind of signal that an author sends to her readers (with the implicit approval of the journal). The convention is that if author X cites work Y, then X believes work Y deserves to be cited and that the citation serves the needs of the readers and is not added gratuitously to serve someone’s ulterior motive. But now you are suggesting to create a legal framework whereby if author X cites work Z, they will also be legally required to cite work Y regardless of whether they believe adding such a citation is warranted and respects that convention. …
    – Dan Romik
    Dec 11, 2021 at 17:52
0

I support the answer of Alexander Woo, but would like to elaborate on "expert judgement rather than crude benchmarks". In my opinion the benchmarks will always be there, whether we want it or not. This is simply not practical to pull experts from their main jobs to do evaluation of other scientists.

Benchmarks deserve their existence. However, it is up to experts to devise clear criteria for the evaluations. One such criterion could be to exclude engineering and software engineering from the academic evaluations. We should evaluate academicians according to academic criteria, scientific programers according to 0.5 x software engineering + 0.5 x academic criteria, and programers according to industrial standards.

The problem is in the self-perception of scientific programmers. Very often they position themselves both in academic field and industry to maximize their chances of finding a job. Pure mathematicians, string theorists, philosophers, or social scientists do not have such an escape.

0

The NIH asks applicants to fill out a "Biosketch" that is used to assess the qualifications of the PI (and other key members) of the proposal.

It asks for up to five "Contributions to Science", each of consists of a half-page of text and up to four references to "research products", which need not be publications or even your own work. This would be an ideal place for the authors of C to explain how their work underpins the more popular package B. Indeed, I've heard that people have had success doing something similar with data resources: "I collected/curated data set X, which others have used to do impactful things Y and Z." The one important caveat is that NIH does not allow hyperlinks in grants, so you can't directly link to the README, but I imagine you can allude to it.

Here's the Biosketch Instructions, which include a sample.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .