Should I write a paper on open-source software I've built?

Question

I have been fairly active in the open-source software community with a few projects that have gotten some attention. However, my reputation in the academic/publications world is a little bit low and I'd like to improve that.

I've seen papers written for all kinds of things ranging from simple summaries of a given project, up to discovering new ideas.

I'd like to write a paper about a specific open-source software project that I've built but I'm puzzled whether it would be appropriate to write a paper about it. After all, not many important software projects were published in the form of a publication.

Edit: as a concrete example, this is one project I've built: http://lmatteis.github.io/void-graph/ - it can visualize RDF structures as a dynamic graph which can be saved in SVG and used in presentations or slides. Would you find this appropriate as the subject of a publication?

Answer 1

I'd like to write a paper about a specific open-source software project that I've built but I'm puzzled whether it would be appropriate to write a paper about it. After all, not many important software projects were published in the form of a publication.

Edit: as a concrete example, this is one project I've built: http://lmatteis.github.io/void-graph/ - it can visualize RDF structures as a dynamic graph which can be saved in SVG and used in presentations or slides. Would you find this appropriate as the subject of a publication?

I can help with specific suggestions for suitable venues in your area (that accept system papers) in order of decreasing impact for improving your research rep:

• Semantic Web Journal accepts Tools and Systems papers. You need to be able to demonstrate real-world impact of the tool/system. Effectively, this track is for paving the cow-paths: publishing about tools/systems that are already well-known in the community but don't have sufficient scientific contribution for a research track (and incentivising the developement of tools/systems that are useful for the community). First step is to get your tool/system to be well-known.

• WWW Demo Track: Write a short 2-page paper on your idea, pack it with as much academically-restrained ehthusiasm and technical detail as you can and hopefully you'll get to present it at the WWW conference. These 2-page papers will be published in the supplementary proceedings and will be indexed in DBLP. The criteria for demo papers as WWW is (in my opinion) often fickle ... a lot of demo papers are borderline/rejected full papers. Otherwise reviewers follow their own whims.

• ISWC|ESWC Demo/Poster/Challenge tracks: Probably you want to aim for a demo track. Submit a four or five page (LNCS) paper to ISWC or ESWC describing your demo. Main emphasis for reviewing is on the novelty of the system itself, technical soundness, and how nice a conversation-piece it will make at the poster/demo session. Demo papers are sometimes (not always) published as a CEUR proceedings, which will sometimes end up in DBLP.

The first option is essentially free (money wise) for you.

The latter two options will incur the cost of attending the conference to present a demo. If you are an independent researcher, that might not be an option: it might be a high cost for little reputation gain. But it depends on your long term goals.

---

Another option is to find your inner scientific contribution and go for a research track submission.

The most important aspect is that an expert in the area will learn something about the area that they didn't already know and couldn't find out about elsewhere (without doing the research themselves). As a reviewer in a research track, after reading a paper I will always ask myself: did I learn something? What did I learn? What is its nature (theoretical, experimental, analytical, synthesis, etc.)? Where else could I have learned that?

As an author, I apply the same principle in reverse: what is the reader going to learn from this paper and how can I highlight it and frame it in the proper "research-speak"? (This may appear cynical, and perhaps it is a little, but being able to identify, highlight and sell your core contributions is a delicate art that does lead to better papers ... as well as higher success in peer review.)

Ultimately, with experience on your side, it sometimes doesn't require much effort to find an angle from which something can be turned into a scientific contribution.

---

Also, take encouragement from the fact that many of the most highly cited papers/references in the Semantic Area refer to software projects or systems of various types (Google Scholar citations): Jena (856), Sesame (1346), Protege (1060), DBpedia (1344), OWL API (265), and so forth.

Being system papers, all of these papers were (arguably) arguable in terms of scientific contribution.

Likewise, many authors in the area have made their names through works that are inherently practical while being based in industry (e.g., HP Labs, Talis, Bell Labs). Looking through the author list of some of the papers above will throw up some such names.

Answer 2

I had faced this exact situations a few years ago. I had developed an open source project, which went pretty popular and started being used by many corporations across the globe. Since, I was looking for strengthening my presence in academia, I planned to write a paper on it and sent to a decent journal. The result was rejection with a plethora of useful comments and suggestions. It took me almost an year to publish that paper in a reputable conference.

So, the lessons learnt were (and possibly applicable to you as well)

• The expectations of research tracks is not only that your thing just works, but you need to justify how and why it works
• There should be results (mostly analytical), to demonstrate that your technique works and hence can be used by other people referring/reading it.
• You need to answer why your work is important and what it contributes to the field
• Moreover, most conferences have or accept papers from industrial or application tracks, where you can describe your work in an implementation centric way (possibly giving you some leverage from nuances of research papers as mentioned above)

Answer 3

Yes, you should. But on the other hand, do not necessarily assume it will be accepted easily. Some advices:

• Your best bet is a demo track in a CS conference (as Peter suggested). For CS demo apps maximum number of pages is usually four, so use them wisely.
• Check all major CS conferences (VLDB, SIGMOD, EDBT and those focusing on Linked Data - RDF). Check their demo track. Read the papers from years past. How is your tool compared to these efforts?
• You must understand that academic criteria is different than industry criteria for what is good or not. The easier way to reject a paper (even a demo paper) is the "infamous" question: "What is the scientific contribution of the paper?" To be published and accepted, you must prove that your work has multiple scientific test-cases and usages. It is not about what is the most excellent work of CS engineering. Otherwise, excellent apps like Photoshop, Worpdpress would get best paper awards.
• Another main question you need to answer on your paper is: Why is your tool better than every one's else (you must provide such a comparison => you must be familiar with what similar tools are available). There are multiple RDF tools or graph tools that may handle thousands / million of nodes. Can your tool support such sizes? (I have worked with SVG many - many years ago and at that time, SVG was not scalable for those sizes. So, also keep that in mind).
• You must describe your tool completely: Tool Architecture (frameworks used, server architecture), what it can do (screenshots, mini-tutorial), where the tool may be downloaded / demoed, what calculations may be done with your tool etc...

Of course this is not an exhaustive list. They are just hints to show you what you should aim for. Note, there are many CS researchers who would like to work with proficient open-source engineers (like yourself) on various ideas, so if you want to expand yourself into academic publishing, you will find your way. Good luck!!

Answer 4

Yes - for example, in my field, there is one fairly good journal that allows a type of short journal article on new "tools", and I've read more than one PLOS paper on an open-source application. As others have mentioned, there still needs to be some research content - likely a worked motivating example comparing your package to other means of analyzing the same information, preferably an example whose scientific importance is pretty clear to begin with.

But you should absolutely consider it - it's easier to give you credit for using your software if it's linked to a paper.

Answer 5

Depending on the area, some journals and conferences accept "application notes", a specific kind of paper that puts much more weight on that significant open source software has been written. Researchers that use such software must normally cite this publication so may also be high impact journals and articles. Search for the journal with this profile. You still must show in the introduction why your work is important and better than some known, older alternatives.

The alternative "classic" approach is to look if some scientifically new results and conclusions have been obtained, or maybe some new algorithm have been proposed and evaluated, with less care if the newly written code is popular or even usable in practice.

Answer 6

As a user of open source software, I would appreciate the authors of said software producing a paper about it.

Then when I use the software I can cite the paper. Sure I can cite the software itself, or its manual, or a book about it.

But citing a academic paper looks better, and assuming there is free/easy access to the paper, anyone who doesn't know what the software is, or doesn't feel it is suitable for academical use.

I'm writing a project proposal presently, I cited papers for SciPy, IPython, and several others. However I had nothing good to cite for Subversion. I may end up citing the software itself, as I did for Python.