101

I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.

On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.

My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?

  • 38
    I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it. – Henry Aug 29 '18 at 0:27
  • 7
    If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic). – Peter Cordes Aug 29 '18 at 4:54
  • 7
    Relevant question on this site: Should I share my horrible software? – Dirk Aug 29 '18 at 6:18
  • 7
    @DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher. – Jim Clay Aug 29 '18 at 13:56
  • 5
    Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code. – allo Aug 29 '18 at 14:15

10 Answers 10

101

If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.

However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”

  • 2
    Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system. – Hakaishin Aug 29 '18 at 7:55
  • 1
    @Hakaishin Rejecting the paper is still an option. – Mast Aug 29 '18 at 10:01
  • 27
    @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers. – xLeitix Aug 29 '18 at 10:35
  • 39
    @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it. – aeismail Aug 29 '18 at 12:13
  • 2
    @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to. – Nic Hartley Aug 30 '18 at 22:59
44

The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.

The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.

  • 3
    The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation. – aeismail Aug 29 '18 at 2:46
  • 22
    If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case? – Ben Aug 29 '18 at 3:44
  • 6
    In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor. – aeismail Aug 29 '18 at 12:21
17

Yes, you should comment and possibly more.

You've said it yourself:

A lot of results in the paper depend heavily on computer computations.

Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).

Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?

So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)

If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.

Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.

12

Messy code affects reproducibility

You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.

In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, a fact that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.

If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, by forcing them to learn a bit about writing nice code, you'd be doing them a favor.

  • 4
    "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time" – ivanivan Aug 30 '18 at 3:19
10

Let me touch briefly on an aspect that hasn't appeared on the existing answers.

My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?

Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.

Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.

Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.

This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!

4

I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.

If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.

Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....

  • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader. – Nisarg Aug 30 '18 at 13:44
  • 4
    @NisargShah, an IDE can't give your variables meaningful names. – Wildcard Aug 30 '18 at 21:26
3

The authors provide a link in the article, so the code is either consider considered a reference or part of the research. Whatever the situation, this raises questions:

  1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.
  2. Is there a license to the code? If not, its status is not at all evident.

As a reviewer, it is up to you to decide what to do. Possible actions include:

  1. Do not comment on the code.
  2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.
  3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.

  • 1
    The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.) – DetlevCM Aug 29 '18 at 11:27
  • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them. – Pierre de Buyl Aug 29 '18 at 11:42
  • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.) – DetlevCM Aug 29 '18 at 12:51
2

I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is. I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.

To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.

Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.

0

My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.

So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.

  • 1
    If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them. – RDFozz Aug 31 '18 at 15:13
  • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer. – akhmeteli Aug 31 '18 at 23:29
  • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve). – RDFozz Sep 1 '18 at 0:42
  • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code. – akhmeteli Sep 1 '18 at 1:27
0

I am not aware of any scientific journals that base their decisions on the code. Whether it is messy or not is not a concern relating the journal. That it should be reproducible should be a concern but I am not aware of any journals that make this a criteria, except for some initiative for some communities, e.g. NIPS, CVPR. In my opinion, it is already a bonus that they chose to release it (and in some cases people are not even allowed to release them due to their organization's rules). If you have any claims that the code is not correct or does not do what it is said to do, it might be a different story but you need to justify this. If you only want to make a comment on the sloppiness of the code, I think you should clarify your commentary and emphasize that it doesn't affect the overall review, I agree with a comment made earlier " convince the authors that it is in their self-interest to fix those issues." Just make it known that your commentary about the code is not related to the review of the paper. Also, please don't discourage people who release their code even if it is "messy".

protected by Alexandros Aug 31 '18 at 15:39

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.