10

It is encouraged to cite relevant software in research papers (e.g. MIT guide). The question is,

Are these citations indexed at all by any engine?

I can hardly find any software on e.g. google scholar to navigate the citations. For example, SageMath devs request to notify them manually about each citation to collect them on their page.

I am of course talking about software without an accompanying research paper.

1
  • 1
    Note that the first step in tracking software citations is making the software findable (check out the FAIR concept). This means that really the software needs a DOI. Unfortunately this is still rare but changing. Besides JOSS, software can also be checked in to Zenodo to get a DOI. Once it has a DOI and is identified by metadata as software, indexing should be possible. Also, referencing a research paper is not the same as referencing the software - this is a holdover. Answer will follow later... Mar 21, 2021 at 8:11

3 Answers 3

6

If by indexing you mean, "are citations for a particular piece of software trackable", the answer is "yes, they can be".

Indexing requires identifiers

The key to indexing is to give your piece of data - software, papers, whatever - a persistent unique identifier. This identifier is traceable back to a particular version of something, e.g., a specific data file or a particular release of the code, or a corrected paper. One example of this unique identifier is a Digital Object Identifier or DOI.

Until recently, typically only journal papers published by larger journals were given a DOI. In the last few years organisations such as Zenodo.org have started to set up the infrastructure to assign DOIs to other digital works, including software. This means that any piece of software can have a DOI and is potentially indexed. See e.g., this search on Zenodo for a list of software.

From that list of software I clicked on a totally arbitrary result that I'll use for the rest of this answer.

The DOI of the software is 10.5281/zenodo.2585902. This DOI can be resolved using the https://doi.org/ service. Or, I can link to https://doi.org/10.5281/zenodo.2585902 and I will be redirected to the source code immediately.

Identifiers can be cited

Once a digital object has a DOI, it can be cited.

On the Zenodo page for my example software I can download a citation in a Bibtex format:

@software{tanlabcode_2019_2585902,
  author       = {tanlabcode and
                  software-github},
  title        = {software-github/SCRABBLE: make it suitable on CRAN},
  month        = mar,
  year         = 2019,
  publisher    = {Zenodo},
  version      = {0.0.1.1},
  doi          = {10.5281/zenodo.2585902},
  url          = {https://doi.org/10.5281/zenodo.2585902}
}

Zenodo also generates citations in other formats like Mendeley, CiteULike, and others.

I can now include that citation in a paper. It is crucial that the reference list includes the DOI so that the citations can be tracked.

We should also note that citing the first research paper where a piece of software is mentioned is not the same as citing the code, especially as the code will change over time. The great thing about a DOI from Zenodo or JOSS is that you can cite a specific version of the code directly.

Citations can be tracked

At the bottom of the Zenodo page for the software you’ll see a box listing citations for this publication. This is based on exchanging citation data (references to the DOI) with other services. Together these services cover much of the international publishing community and data repositories. It can take a week or two, but usually citations show up.

Another important tool is crossref.org which keeps track of all uses of a DOI. I can search the metadata index for that DOI. Unfortunately the random piece of software I chose has no citations yet :(

As mentioned by @moinmoin the process has been streamlined a bit by the Journal of Open Source Software (JOSS).

DOIs in indices

Although software and other digital objects can be indexed, this does not mean that they are. As pointed out before, Google Scholar is one of the few that seems to do this; Web of Science and others don't appear to include non-traditional DOIs at this time.

You should expect increasing indexing (and awareness) of non-article DOIs in the next few years as employers and authors see the benefit of being able to track non-journal publications, such as data sets, software, and other digital products. It gives everyone involved much more recognition than would be possible with traditional papers.

What SageMath should do

Let's look at your example, SageMath.

  • They should check their software into Zenodo or submit it to JOSS. They would then get a DOI.
  • They include the DOI prominently on their web pages.
  • they update the DOI with each new released version of their code
  • Users of SageMath should then be encouraged to cite that DOI.
  • Usage of SageMath can then be tracked through CrossRef.
5
  • Accepting this answer as the most complete one! Thank you! Mar 22, 2021 at 22:08
  • by "update the DOI with each new released version" do you mean get new DOI or update the info into the old DOI (if it's somehow possible)? Mar 22, 2021 at 22:08
  • 1
    Zenodo has a system whereby you can give each new release (e.g., 1.0, 1.1, 2.0) a new DOI. If you use a code repository like GitHub you can tag a commit as a release. This means that you get a DOI that is specific to a single commit. In Zenodo you do this by just adding a new version. Try it out at sandbox.Zenodo.org! Mar 22, 2021 at 22:13
  • 1
    For balance, I would note that there alternatives to Zenodo: Figshare also allows you to get a DOI assigned to software, and that DOI can be revised with new versions. By which I mean that there's a generic DOI of figshare-blah, and a series of DOI versions figshare-blah.v1, figshare-blah.v2, etc., with the generic resolving to the most recent. (I don't know, and it's not clear from above, how Zenodo's DOI versioning works.)
    – Lou Knee
    Mar 23, 2021 at 22:59
  • 1
    Once citations are indexed then the next step will be to apply metrics, such as counting them... so should the different version DOIs be counted separately, or all added together? I don't think either is correct in general.
    – Lou Knee
    Mar 24, 2021 at 0:30
7

Google Scholar does sometimes parse citations to codes; here is an example of the R library rvest:

30 citations to a R library

In addition, as MoinMoin implied, many researchers write so-called data papers or software reviews that accompany newly published datasets or softwares. They are submitted to journals and undergo peer-reviews like other kinds of scholarly articles.

Journals increasingly support such data papers or software reviews. I am not only talking about journals that are specifically dedicated to such document types (e.g. Springer Nature's Scientific Data), but also to specified submission guidelines for this kind of documents at other journals (e.g. at PLOS ONE)

Some bibliographic databases capture such documents with a special category. At Web of Science, for instance, you can refine the results to 'data papers', 'database reviews' or 'software reviews':

Web of Science menu

2
  • I've never heard of using a "data" paper to introduce a software Mar 20, 2021 at 20:26
  • @AzorAhai-him-, you are right about the nuance. I edited the post a bit. Thanks for the hint.
    – anpami
    Mar 21, 2021 at 8:19
6

an attempt to answer your question:

To an extent this is a known problem -"how do I get credit for writing scientific software?" when our community largely tracks citations via Scopus, ORCID, Google Scholar etc., and also when software is versioned and in continual development, not usually one-and-done like a paper.

I do not believe (read: have never seen) any kind if citation counting for the "miscellaneous" references (to borrow the common entry type used in BibTeX for software) without a proper identifier, like a DOI. I also anticipate that there's confusion in citation counts when trying to sum over versions of a software package.

some additional thoughts on addressing the problem:

Fortunately, being a known problem in scientific software development, a current attempt at a compromise solution that avoids writing an publishing a traditional, formal paper is The Journal of Open Source Software (JOSS). The JOSS papers are essentially taken from the documentation, so if you write up the code properly, it is supposed to be minimal effort to generate the paper. They then send it out for peer-review, and assign a DOI on acceptance.

JOSS also has the advantage of putting all the relevant links in one place at one place (DOI) - paper, source code, releases, etc. JOSS has the backing of NumFocus, who also have a few other open journals and are a major funder (donation bundler) for a large number of scientific, open-source software projects (Julia, Pandas, NumPy), which is a long way of saying that JOSS is a legitimate and accepted part of the Scientific FOSS ecosystem and not some scam journal.

5
  • @libor thanks for the link - I use NumPy enough that I should be citing it now that there's a paper/DOI. I think it's it's important for the people who write and distribute the software I use in my research to also get the credit that is often missing when software can't be or isn't cited.
    – DoctorSoup
    Mar 20, 2021 at 15:13
  • 1
    @libor I guess I also have a much less cynical worldview than you: I don't care one way or another about whether Nature's IF will substantially increase, it's Nature and whether the people working there get a bonus at most maybe tangentially affects access when we start talking about what journal subscriptions cost. I personally find that the smarmy/snarky comments that are not directly related to the question detract significantly from the usability of comments (separating the wheat from the chaff to get the useful information) and negatively affect the (certainly my) user experience.
    – DoctorSoup
    Mar 20, 2021 at 15:17
  • 1
    In the interest of being polite, I will explain that I find throwing around "middle managers' bonuses" as opposed to just saying that it's likely profitable (since you didn't link to those managerial contracts with bonuses for increasing the IF, so I strongly suspect that that is not "a factual statement") is (to me) a snarky, obnoxious comment. And it's ok that you don't care about my user experience, but the added sarcasm ("ask for a refund") is certainly rude and derisive and, once again, contributes nothing of value to the discussion.
    – DoctorSoup
    Mar 20, 2021 at 17:24
  • Forget it. Feel free to edit the information I gave you into you answer. Or not. In the interests of being polite I'm not interested in engaging with you.
    – user133933
    Mar 20, 2021 at 17:38
  • JOSS is a nice reference, thank you! Mar 22, 2021 at 22:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .