During peer-review, should I comment on the authors' messy code?

Question

I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.

On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.

My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?

Answer 1

If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.

However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”

Answer 2

The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.

The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.

Answer 3

Yes, you should comment and possibly more.

You've said it yourself:

A lot of results in the paper depend heavily on computer computations.

Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).

Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?

So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)

If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.

_{Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.}

Answer 4

Messy code affects reproducibility

You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.

In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, a fact that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.

If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, by forcing them to learn a bit about writing nice code, you'd be doing them a favor.

Answer 5

Let me touch briefly on an aspect that hasn't appeared on the existing answers.

My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?

Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.

Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.

Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.

This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!

Answer 6

The authors provide a link in the article, so the code is either consider considered a reference or part of the research. Whatever the situation, this raises questions:

1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.
2. Is there a license to the code? If not, its status is not at all evident.

As a reviewer, it is up to you to decide what to do. Possible actions include:

1. Do not comment on the code.
2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.
3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.

Answer 7

I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.

If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.

Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....

Answer 8

I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is. I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.

To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.

Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.

Answer 9

My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.

So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.

Answer 10

I am not aware of any scientific journals that base their decisions on the code. Whether it is messy or not is not a concern relating the journal. That it should be reproducible should be a concern but I am not aware of any journals that make this a criteria, except for some initiative for some communities, e.g. NIPS, CVPR. In my opinion, it is already a bonus that they chose to release it (and in some cases people are not even allowed to release them due to their organization's rules). If you have any claims that the code is not correct or does not do what it is said to do, it might be a different story but you need to justify this. If you only want to make a comment on the sloppiness of the code, I think you should clarify your commentary and emphasize that it doesn't affect the overall review, I agree with a comment made earlier " convince the authors that it is in their self-interest to fix those issues." Just make it known that your commentary about the code is not related to the review of the paper. Also, please don't discourage people who release their code even if it is "messy".