I have evaluated a software project which has been created by a consortium of top scientists in the field. However, the project itself it doesn't really work, and has only been developed as a proof-of-concept rather than an end product (i.e., it works only with 2-3 scenarios).

This software application should perform 4 steps in order to successfully execute. Each step takes an input file, and produces an output file. The output file from the previous step is used as input in the current step. Initially, you start with 1 file. This file is used as input for step 1. After step 1, another file is produced. Let’s call the input file general_input_file, and the output file general_output_file. When general_input_file is loaded into the application, general_output_file should be produced. Now, I have an input file which I will call my_input_file. I expect the application to produce my_output_file. However, the application only accepts specific_input_file and will produce a specific_output_file. This means that it only works with 2 files which has been previously generated. Both files exist in the project. When I look at the part of the project which should process the general_input_file, there is a statement which looks like this: if the name of the given input file equals specific_input_file, then return specific_output_file. This is a file inside the project. Otherwise, try to process the generate_input_file and generate general_output_file. At this point the software breaks. A number of exceptions are thrown, and debugging and fixing this is beyond of the work that I’m doing.

The question is: in the paper, how do you approach this issue? And how do you argue, in the paper, that the reason you're unable to evaluate the software in a different scenario is due to the limitations of the software? What is the best wording to be used, without being offensive to the authors?

  • 5
    "only been developed as a proof-of-concept rather than an end product" - this is normally the highest aimed for goal of any software development that happens in research. That in itself is not a sign of "low quality", it is by design, as research is not conducted to produce full-fledged, marketable products in most cases. And this can well extend to your observations about the coding quality, as maintainability or top performance are not necessarily a priority. – O. R. Mapper Oct 31 '15 at 14:43
  • 1
    I'm not sure we can answer this question without more context, since the appropriate level of development for software depends entirely on the goals associated with the project. – jakebeal Oct 31 '15 at 15:02
  • 4
    It seems that this depends on at least three questions that you don't talk about here: (1) what is the goal/audience of your paper?, (2) what does the consortium claim their implementation can do, and how does it stack up?, and (3) why is it obvious, and why does it matter, that coding was done by someone inexperienced? For (2), if the project claims the code can transform any input_file to a correct output_file and it can't, this is part of (2); and for (3), I hope it is more than style; but only you know. – gnometorule Oct 31 '15 at 15:36
  • I am still confused as to why this actually matters for your paper or why you are writing a paper on evaluating a software project. Could it be that you are peer-reviewing the project in some way? If a paper on the respective project is already published, I suggest to write a comment/reply to that paper, if the journal allows for that. – Wrzlprmft Nov 7 '15 at 10:54

When I look at the part of the project which should process the general_input_file, there is a statement which looks like this: if the name of the given input file equals specific_input_file, then return specific_output_file. [...] Otherwise, try to process the generate_input_file and generate general_output_file. At this point the software breaks.

By your description, it sounds like the code you are looking at is cheating by returning a precomputed output known to be correct if the input matches a single sample input the authors included with their code (presumably to demonstrate its correctness and because including a sample input was required to get their paper accepted). For all input instances other than this one sample input, the code does something entirely different, which doesn't produce correct results.

This sort of behavior would without question merit a failing grade in a programming course homework assignment or exam. If done in the context of a commercial product, similar behavior would justify the resignation of the CEO and a multibillion dollar scandal. So, saying that the code is "of low quality" seems to me so euphemistic as to be itself a dishonest statement. The way I would describe it is: if the authors are claiming that their code is correct, then they are lying.

Now, I could discuss your actual question of how to discuss this situation in your follow-up paper "without being offensive to the authors," but honestly, I don't see the point. I would ask instead, why would you want to not be offensive to the authors? It's not just that the authors' algorithm may be wrong and that their code is of poor quality; they are seemingly committing academic fraud by submitting an incorrect algorithm with code dishonestly tweaked to make it seem like the algorithm is working. I'm willing to leave a 3% chance that some other more innocent explanation can be proposed, but given the description you've provided I really can't think of one. I'll be happy to reconsider if you provide a bit more detail on how the authors' claims are presented in the paper.

| improve this answer | |
  • 2
    It is hard to understand what is their actual claim about the code. The project page is as follows: they have a bunch of deliverables where they discuss the approach they take into this project, and present a few examples (based on their scenarios?). Another part of the project presents the code of this project (or at least some components of the project). There is a possibility that the code might be out of date (i.e., the last version might not be posted). This project ended a couple of years ago, and a follow up project started this year (I expect they will build on/extend this project) – Andrei Oct 31 '15 at 16:37
  • 1
    Well, the if clause that you described that matches a specific output to a specific sample input file sounds by its nature dishonest. But we're obviously missing a lot of details so it's hard to say anything more substantial. – Dan Romik Oct 31 '15 at 16:45
  • 1
    Here is some information on cheating in chip benchmarking tests: bgr.com/2013/11/27/samsung-benchmark-cheating-banned-htc – Dan Romik Oct 31 '15 at 18:16
  • 3
    @DanRomik: I agree it's a red flag, but I can easily think of an innocuous explanation. The subject of the research might be, for example, a specific algorithm to process certain data or extract some information. The input for this algorithm could be required to (if only for reasons of simple implementation) be in a specific custom format X. To CS people, it might be obvious that Excel documents can be converted to that format, if someone writes a converter. To non-technical people, however, connections of this kind are often not evident. They are not satisfied if they ... – O. R. Mapper Oct 31 '15 at 18:38
  • 1
    ... see the prototype open and process a file in format X and are told that a converter can be written to process Excel documents. They typically have no concept of the complexity (or lack thereof) of a transformation between structure-equivalent data. Silly as it may seem, they want to see how an Excel document is selected and processed. Consequently, unless one wants to spend time for discussions about irrelevant topics such as concrete file formats, the prototype will be adapted appropriately for a demo whose actual point is to show the workflow and the core algorithm. – O. R. Mapper Oct 31 '15 at 18:39

The least offensive way to indicate that you tried to use system-X (and it didn't work) is to say something like

We also attempted to process our data using system-X [1023], but were unable to do so successfully due to run-time errors.

That avoids assigning any blame, whilst acknowledging the existence of the project. I am guessing that there is a very high chance that at least one of the scientists associated with system-X will be a reviewer for your paper. If the system doesn't work, they have still gotten a cite. If there is a newer version of the software that does work, you may elicit a useful comment.

You might also try sending an email to the authors to see if there is a way around the error, or if there is a newer software release. Including the exception log as an attachment, and avoiding mentioning having looked behind the curtain may be diplomatic.

| improve this answer | |

Does it need to work?

I know the question is weird, but researchers are not software engineers. If the objective of the project is to create a fully functional piece of software, then the low quality of the software is worrying. In the other hand, if the objective of the project is to devise new ways of doing stuff, then this software is a prototype, a proof of concept, as you mentioned, then working in a few cases is a remarkable feat.

Once the proof of concept is done, the software engineers can come and turn this into a product, which is a fully functional software. Don't make researchers do engineer's work, they are bad at it :). To each its own.

Understand the context to understand the results..

| improve this answer | |
  • 1
    I don't believe this addresses OP's question. He is not asking whether he needs to be critical of the software he is evaluating, but rather how to be critical without offending anyone. – 01010110011001 Oct 31 '15 at 14:51
  • 2
    "Don't make researchers do engineer's work, they are bad at it" - I find the second part of this statement untrue as a general rule, and even slightly offensive (given that at least in applied CS, a significant part of people switch to the industry as professional developers right after receiving their doctoral degree). Lots of research code is not of low quality (when measured to code of marketable products) because its developers cannot do any better, but because priorities in development are simply very different. – O. R. Mapper Oct 31 '15 at 14:54
  • @O.R.Mapper Writing code is very different from developing a fully functional software. And that takes experience and a skill set broader than just good programming skill. You can go from academia to industry and vice versa, of course, but you won't be a senior anything. Both problems are more complicated than that and need experience. More than that, a proper software needs documentation, use cases, etc. For a researcher, that would be a colossal waste of time, better spent writing articles... – Fábio Dias Oct 31 '15 at 17:50
  • @FábioDias: All right, so you meant something like "Don't make researchers do engineers' work, they are sometimes inexperienced at it (similar to engineers who have started working only one or two years ago)". As it stands, I first read your statement more like "Don't make researchers do engineers' work, they are hopelessly incompetent at it (similar to cooks or gardeners who are asked to develop a software)", hence my slight disagreement. – O. R. Mapper Oct 31 '15 at 18:48
  • @O.R.Mapper yeah... Let's just call it an unfortunate and incompetent attempt to be funny :) – Fábio Dias Oct 31 '15 at 21:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.