How much should reviewers care about other things than an article's content?

Question

This is a follow-up question to a previous one asking for the potential impact of the typesetting tool (at the review level) on the chances of acceptance for a given paper.

I'm interested in the general case where reviewers will get a "feeling" of the way the article is presented, be it the writing style, British vs. American English, English from non-native speakers, choice of font, or like in the linked question, typesetting and any other criterion that is not related to the content of the work presented.

To be clear, I'm not asking about the more obvious or well-documented sources of bias, like the sex, name or geographic location of the authors. This can possibly be addressed by double-blind review.

As an example, I'm used to journal submissions in the form of a PDF with the least possible amount of formatting: plain text, typically not justified on both sides, lines numbered, usually double-spaced, figure captions at the end, figures concatenated at the end by the electronic submission system. Where I publish, typesetting is a journal's thing and as an author I'm happy to delegate that hurdle.

I once reviewed conference proceedings where the submission was very informal and couldn't help but having a "feeling" from the looks of the submissions. Authors who used LaTeX, which is unusual in my field, didn't look like they had a lot of experience publishing in this field.

I think that ideally this feeling should be silenced and I tried to do so as much as I could. However, reading comments and answers on this other thread made me wonder if that was a sentiment that others shared. My question then boils down to:

Do reviewers have a duty to try to ignore their biases or is the "gut feeling" an important part of the review?

As a final note, I'm not looking for arguments in favor or against the use of LaTeX in general, or in the specific case I mentioned. I find LaTeX to be very well suited for the applications it's intended for and I simply used that example to illustrate something that is not content but could (and apparently does) induce a bias.

Answer 1

I think the gut feeling is helpful during the evaluation process - one can probably not fight it anyway, so I think one should just accept that it is there. However, the gut feeling must not hinder a review based on the content of the paper.

So, what should not happen is, e.g.:

• Stop reading after a few pages because you don't like the look, typesetting, language, variable names… (you may, however, stop reading if the style hinders your understanding, which is something completely different). Even if you hate Times New Roman or Arial, be professional and keep reading.

• Keep worrying about issues as above when it comes to write the comments (with the exception of the language, but I am always very careful about comments on language). Comments should mostly address the content of the paper, especially your recommendation should not be influenced by gut feeling. So once you read everything carefully, form an opinion not based on the look of the paper, but on the content (which you have digested at that point and hence, you should be able to evaluate it without taking the look into account).

One exception is figures: If figures are poorly formatted and don't allow to extract the information, this is a serious issue (too small font, pixel graphics with poor resolution, colors that are indistinguishable for common color blindness…). But also here it is more about "can I extract the information" and not about "do I like the look".

Be extra careful with gut feeling if you review something which is borderline for you field. As can be seen in the above comments, the styles of submission vary greatly between fields (extreme example: Some field have all floats, i.e. tables and figures and such, on extra pages at the end of the documents, other field have submission that basically look like a final article, some fields use double spacing, some don't…).

Answer 2

We're supposed to review papers, not authors. But, even in fields with blind review, things like formatting can reveal (or just suggest) a lot of information about the author, which we should try to ignore.

If it's a feature of the formatting which is actually going to disappear in publication (like font choice or number of columns) it's genuinely irrelevant: it tells you nothing about what the published article would be like, so any reaction is personal judgement about the author rather than the work.

Formatting decisions that will remain, like some features of figures and tables, are harder. Those, I think, should be treated like any other expository feature: if you don't like it, you make yourself focus on the question of whether it actually makes the paper less clear or harder to read, or is merely unpleasant. In the former case, you hold it against the paper, in the latter you don't.

One thing to keep in mind is that we don't want to reject people from nearby fields with different writing conventions who have something good to say. Many of the signals your seeing are mostly indications that the person is an outsider to the area, which really shouldn't be held against them. (Note that an outsider is already at a disadvantage, because conventions that matter to the refereeing, like how to organize a paper; how to handle that is, I think, a more difficult question.)

Answer 3

Reviewers should care about matter of text, not aesthetics. Most things you describe belong to the editorial board and editorial staff competences.

That said, I believe that, most of the times, there are some guidelines that a submission must fulfill in order to be passed for review. I find it a basic decency towards reviewers to hand out a properly formatted document. And usually it's not the author who does so.

I would say it's not a matter of pedantry but rather hygiene. It's simply easier and more pleasant to read a document with basic consistent type setting. You can focus better.

"Authors who used LaTeX to write a 5-pages article with 6-7 equations and 10 citations felt like pedantic undergrads with too much time on their hands. Or at least it didn't look like they had a lot of experience publishing in this field."

I don't know, but my personal experience is that it's easier to write in latex anything more than 3 pages, or with at least 1 equation, especially if you have some experience. You do not care about anything else than the matter. You get free bibliography and cross-references. Very easy. And it looks half decent, you get basic type setting for free too. I guess it's just a matter of workflow.

Your sentiment can be turned around very easily. Would you be happy if students handed in assignments written on scrap paper? Probably not. Would you say, with out reading, the content is flawed. Also probably not.

Answer 4

Much of this is down to the journal not the authors. The authors' usual favourite journals will affect what they submit elsewhere.

Taking a journal in my field as an example: the combination of a tight page limit and the submission system means that all authors use the journal's template (LaTeX or word). The LaTeX template makes both a journal-article-style version and a double-spaced, single column, figures at the end (but still fully justified) version available to reviewers (authors can also download these). My understanding is that starting from word you get something close to the latter. In this case, while the journal does some formatting, the authors are expected to provide something quite formatted. In particular the figures should be publication-ready at the first submission.

An author coming from that background to a conference with a much less formal style may well take the same approach -- I would. And yes, I prefer LaTeX even for writing letters -- for me it's quicker and easier to use, even more so if there's an equation or two, or even a single citation (as I use a master bibtex file). Other people prefer other tools, and much more of the variation in the aesthetic quality of the work is down to the author than the tools they use.

Many people struggle to hit "informal" without feeling like their submission is sloppy, so will try harder than is necessary, or they go for quite a formal style by default. Even quite closely related disciplines vary significantly in their levels of formality in style (at an engineering conference I saw suits and ties, but equations made by abusing normal text formatting in the abstract book; an applied physics conference was quite a contrast on both fronts). The most intersting talks at a conference are often those from the edges of the discipline -- just the people who might be expected to miss the house style a little.

Now as a reviewer, obviously failure to follow simple instructions is hard to ignore, as is general sloppiness (e.g. if figures are expected to be publication-ready and they look like a school project drawn in paint). It can also be hard to draw conclusions as to the quality of the content if it's presented really badly. But the specific tool used are irrelevant.

Answer 5

What I usually tell to my students is the following.

Let's suppose that you have written a weak paper. And you will write a weak paper one day or another. If, in addition to this, your paper is also full of grammatical errors, badly formatted, with an incoherent structure and poorly drawn figures, your chances of being published will be next to zero. Because not only will you likely annoy the reviewers, but also because the few positive parts of your work will remain hidden.

If, instead, you do your homework and do your best to prepare a well-written and formatted manuscript, the reviewers will be probably more willing to give your manuscript a second chance. Even if your work is weak, its positive sides will be more evident. Of course you will likely have to do a couple of rounds of major revisions, but that's far better than being rejected right from the start.

With a bit of practice, writing a well-formatted paper takes roughly the same time than writing a badly-formatted one: why taking the risk of spoiling months, possibly years, of work?

Answer 6

Most people I know use a "fail-fast" strategy when reviewing papers- depending on the size of the conference/journal and the field you may be asked to review many papers, and you might be asked to review multiple times a year. It takes hours (for argument, let's say four hours) to really deeply read and understand a paper. Let's do some math: if I'm asked to do 10 reviews and I spend four hours on each paper that's 40 hours of work, or an entire work week on top of other obligations. That's never going to happen. If I find a good reason to exclude a paper early, I'm going to stop and reject it. If nothing is obviously wrong then I'm going to settle in and read the paper more closely.

Common reasons to exclude a paper are:

1. Paper's language is so poor that I have to work really hard to just understand what is being said
2. Paper is not in scope of journal/conference
3. Paper is incremental
4. Major technical flaws or incorrect statements

To be clear- nobody has a checklist of reasons that they use to immediately reject papers they don't want to read. These are basic problems that just jump out at reviewers and cause them to not want to read the rest of the paper.

To answer your question- bias can definitely contribute to a feeling of "I'm busy and I just don't want to review this paper." This sounds harsh, but I guarantee you that the odds are very low that someone manages to produce quality technical content but cannot master basic things like typesetting. If you combine that bias with a good fail-fast reason it can absolutely kill your paper before it's really considered. Tracking down all the typos and typesetting errors is tedious, but it's the easiest and simplest part of writing a paper. If they can't get that right, I don't have high hopes for the rest of the content.

Answer 7

My perspective on many of these questions: The goal should be to evaluate the science behind the paper, not the authors of the paper so long as these things do not interfere with the clarity of the paper overall.

British vs. American English

Don't care. The biggest issue for me is that British vs. American English starts me guessing where the paper came from in double-blind review, which I have to avoid.

English from non-native speakers

I only comment on this when it's sufficiently off from native English that it's actively hampering the comprehension of the paper.

Your LaTeX example

I'm actually in the same boat you are - for me, LaTeX or other typesetting tools are very much not the standard in the field. The appearance of that, or some other things that aren't "normal" for the field (for example, reporting regression coefficients rather than effect measures) are indicators that someone is coming in to the field from the outside.

That doesn't necessarily mean I'll judge their paper more harshly, but I may end up suggesting it come back to field standards for reporting results, make sure the results are applicable to readers of the journals, etc. To use a common example, a proof, no matter how correct, is almost useless to a clinical audience.

Do reviewers have a duty to try to ignore their biases or is the "gut feeling" an important part of the review?

I think this is a little bit of a false dichotomy. Some of these biases can serve as useful heuristics in evaluating a paper, but yes, you need to be vigilant that "Clearly not native speaker, therefore bad paper" doesn't start creeping into your reviews.

Answer 8

Research is a human activity, and humans always have biases. While we should be aware of our biases and try to minimize their impact, we can never avoid them entirely. Trying to consicously compensate against biases ofter leads to overcompensation or undercompensation.

If you fail to adhere to the social norms in your field, some people will be biased against you. Writing papers that look different is one way to break the norms. If the paper is a clear accept, the biases will probably not matter. The effect will be more significant for borderline papers, which are accepted and rejected based on the gut feeling anyway.

Answer 9

This answer deviates from the typesetting aspects and deals with the more general question in the title.

There were two cases, where I accepted an article for other reasons than its content. The first was a long article, where my impression was that it is correct and difficult, but the question was absurdly artificial, in the sense that it appeared to be invented for the sole reason to get another publication. As the journal was pretty good, for me this was a reason to reject the article. Then I noticed that the article was written by some demi-god, which had for some reason escaped me up to then. A quick literature search showed that he had not that many publications, and all of them had a great impact (not in the bibliometric sense). So my impression changed from "artificial" to "too deep for me", and I accepted.

The second case was an article which was full of errors, but this article was an invited article for a special issue celebrating a person A, and the author B was a good friend and collaborator of A. Usually I would have rejected the article, but out of respect for A I sat down and pointed out the flaws, received a revision with lots of new errors, wrote another report, and finally the article got published.

In both cases I think the deviations from the "pure" procedure were justified.