Contributions of indirect research toward getting a post-doc

Question

I am an engineering student, however, my graduate research is more inter-disciplinary (computational science). Which is good for me in some ways, because one of my better skills is programming, since I've been doing it avidly as a hobby for a very long time. Because of this, it would be much easier for me to create a widely used computational science software tool rather than a highly-cited journal article.

With the specific aim of getting a post-doc at a top-ranked university, how does creating a useful piece of software (like an original, optimized quantum monte carlo implementation for example) compare to a widely-cited journal article? In other words, if done well, both would be highly-cited, but do frequent citations of software carry the same weight as top-tier research articles?

Answer 1

The following applies to the neuroscience and the biological sciences.

In almost all cases, the article will win. This is typically due to the attitude that writing software isn't research; it's implementation. In some cases, you will have to overcome technical challenges when implementing the software, but for the most part you'll simply be coding up techniques that were developed and iterated upon by a different research group.

Note that there do exist labs that both developing analytical techniques and then have a team as part of the group who translates that into software development. The Wellcome Trust Center in UCL comes to mind here; they have developed a number of techniques used for detecting brain activity, as well as brain mapping, and they maintain a software suite that allows researchers to use their techniques. There are a number of similar groups in different fields (e.g., medical image analysis (project)(group), MEG analysis (project)(group)). These labs have typically demonstrated over time their strength in developing research software, and are able to obtain grants to specifically support development of their software.

Answer 2

There's a thresholding effect for software. If you create a piece of software that has a huge number of downloads and has visible impact, that will count for something. However, with papers, you wouldn't need to have the same level of impact to have the same effect on interviewers.

The problem really is that academics don't know how to evaluate the impact of software development, and would need help doing so. I imagine that if your potential post-doc advisor has used your software, you won't need to do much explaining. But if not, and your research credentials are otherwise weaker, it might be difficult to justify.

Answer 3

Attitudes are slowly but surely changing--here is an indication. As of January, the NSF required that the Publications section of NSF biosketches be changed to Products. It is remarkable how important names are to bureaucrats; in this case, the change reflects the awareness that the publications category excluded much of the work of scientific programmers, inventors and the producers of data sets (data sets are routinely referred to as products in the geo and social sciences). The previous biosketch format tended to make the inclusion of software and data sets (not to mention other contributions) under Publications often seem forced. The practice of citing technical reports and papers on software instead of directly citing it ought to be questioned. Given the ever increasing importance of software and data set production for many kinds of scientific research, it is a welcome development not to have to diminish significant and specialized contributions by citing published reports on products that could be cited directly (in contrast to certain inventions) or by hiding them under Synergistic Activities. In time, administrators will get the message that software production ought to count more than it does.