Reviewing papers for MDPI: general feeling as a reviewer

Question

So simple question for those that already reviewed for MDPI: how did you feel about the quality of the other reviewers (in general)? I have reviewed a handful of papers now as I feel it is necessary (in the end others review my papers too), but I am starting to have serious doubts about the "level" at MDPI. Maybe it depends on the journal/paper etc, but for most of the papers I reviewed it ended in me feeling rather idiotic to even bother. (on a side note: I avoid MDPI to publish in, it is really my last resort to publish something in, if I have no other option, but this does not really happen).

To summarize: One paper was a copy paste of another paper, they just changed the molecule they used. Now I can get this, but there is a difference between bluntly copy pasting (even certain mistakes from this previously published paper in MDPI) were kept(!!). I was the only reviewer to actually review this paper. The other 3 pretty much just gave it a go to be published. Which made no sense, the language was awful, there were critical errors in the paper. On top of that, after review round 1, all of a sudden there were extra authors on the paper (and the editor never said anything about this even though I mentioned this in my review report).

Another paper: 2 reviewers, I had some minor remarks, but still some issues and the paper also had several issues language wise. The second reviewer just accepted the paper as it was and congratulated the authors on the splendid work. The paper was mediocre. Nothing special and it contained some errors.

A third paper: I wrote a long list of errors, small things, some bigger ones. In general the paper was again mediocre but ok. It could be published if they removed these issues. The 2 other reviewers barely mentioned anything. They just had a few points (like 3-5) that were really minor things. It seemed they never really bothered to read the paper.

Now I know this happens in other journals as well. I even reveiced "reviews" from reviewers while it was obvious they hardly bothered to read my paper, but MDPI seems to top the cake to be honest.

The duality I am facing now is: should I bother to review for MDPI again (and at least try to improve the manuscripts) or not bother anymore and only review for other, more reputable, journals? I might be a bit naive as I know other (reputable) journals might also have shitty reviewers.

I specifically ask about the experience of reviewing for MDPI, not about their reputation MPDI in general.

Answer 1

Just a personal remark (that to some extent answers the question, for me personally). I usually have many things to do and am asked to review regularly, by several journals. I will strictly do reviews on a first come first serve basis. I will not be hassled by tight deadlines, and for sure a tighter deadline will not make me prioritise a review. Furthermore I decline as a matter of principle all requests that give me three weeks or less time, as this will not normally work according to the first come first serve rule.

More radically, as people know how busy everyone is, I think that if somebody sets a review deadline at less than three weeks, they implicitly communicate that they don't want a thorough high quality review, and consequently I won't review for them. This rules out reviewing for MDPI as far as I know them, but note that this is not exclusive to MDPI, and their reputation is not the issue here. However I'm not surprised in the least that they get crappy reviews given the time frame in which they're working.

I do know that high publication and reviewing speed is seen as desirable by many, and if somebody else is able to produce a good quality review in 10 days, fine by me. Still I value quality over speed, and quality takes time in my world.

Later addition: Once more not exclusive to MDPI, but... the problem with commercial publishers who use the model that the author and not the reader pays is that they have an economical incentive to accept papers. This will often play out as accepting papers more easily, and being generally more positive than critical reviewers, including potentially ignoring reject recommendations, although this will depend on the individual editor and journal. Some may be concerned enough about the scientific reputation of their journal to not accept all too bad stuff, others may accept far more than a competent and critical reviewer thinks they should.

Answer 2

I both reviewed for and published with MDPI multiple times. My experience is in line with what is often written here: MDPI will simply look for other reviewers if they dislike your peer review report. A few months ago, I reviewed a seriously flawed paper in my field (nutrition). The authors completely misinterpreted their data. Reviewer #1 asked for minor revisions while I could not recommend the paper for publication (based on the flawed methodology). They asked for a third reviewer who finally approved the paper. It is now published - review reports did no not appear because the authors did not select open peer review.

In 2022, I submitted 3 papers to MDPI. One to Nutrients, which is one of their flagship journals. Two (well-known reviewers) asked for minor revisions which made me skeptical... In this case, peer review was open. In another instance, the same happened. A paper that has been rejected by other Q1 journals underwent minor revisions and was published in 21 days with MDPI. Finally, I had a paper with 3 reviewers that asked for major revisions. After 3 rounds, all reviewers were satisfied and indicated that the paper is suitable for publication. MDPI rejected it and invited me to re-submit it to another MDPI journal.

For me, MDPI is gambling. It is not transparent and I would only submit papers to MDPI that you may not publish elsewhere. I am honest about that. As for peer reviewing, I only review articles for MDPI from people I know. The constant pressuring for deadlines makes me uncomfortable. 5 days for a major revision is simply not enough in my opinion.

Answer 3

Disclaimer, I worked at MDPI at the time most of this answer was written.* I also have had extensive experience in academic publishing, albeit for other publishers.

The question of whether reviews at MDPI are worse than at other publishers is difficult to answer, because to write a full answer one needs access to every journal's & every article's reviews. I don't have that kind of birds-eye view of the operations. Individual editors see only a subset of their journal's articles. More senior editors can see all the journal's articles, but there are a lot of them, so they are not likely to check every article's reviews. I'll still write an answer based on the reviews I've seen firsthand. The following numbers are for the journals I work with. Because different editors do things differently, what I see might not apply to the journals you see.

For the reviews I have seen firsthand (most of these papers are still under review, hence the relatively low number of reviews per paper):

1. Number of papers: 11
2. Number of reviews: 15 (all written by different reviewers)
3. Number of reviews that include detailed comments: 10
4. Number of reviews that do not include detailed comments: 5

Definition of "detailed" I'm using is if the review is >2 paragraphs in length, after excluding requests to cite X paper or simple English fixes. This definition is not ideal, but it's simple, and does not require subjective judgment.

Several of the reviews with detailed comments are extremely detailed reviews, spanning several pages, with two reviewers running their own calculations and including those calculations in the review (!). Among the reviews that do not include detailed comments, there was one review that basically asked for some citations without saying much else. The other reviewers still clearly read the paper, but didn't have much to say.

Compared to the reviews I saw at other publishers, broadly speaking, these reviews are excellent.

However, I have several comments about these numbers:

• Peer review is a rather stochastic process, and some reviewers will just write poor reviews. We can see this in the Who's Afraid of Peer Review? sting from 2013. Briefly, a Science correspondent submitted a fake paper to several open access journals, and:

Of the 255 papers that underwent the entire peer review process to acceptance or rejection, about 60% of the final decisions occurred with no sign of actual peer review. ... Only 36 submissions generated review comments recognizing any of the paper's scientific problems.

So about 40% of 255 papers (=102) were peer reviewed, of those 36 of 102 (~35%) noticed the paper's scientific problems. In other words, about 65% of peer reviews did not notice the issues. Even if some of these reviews are themselves faked, the numbers are large enough that it seems likely that some reviewers simply didn't notice or care enough about the problems. This would mesh closely with my own experience. I clearly remember one Distinguished Professor at a major R1 university in the US writing a review that said nothing except "I recommend revision, and you should cite the following papers all written by me", so seniority & publishing record doesn't preclude a reviewer from writing poor reviews.

• It should be obvious that if the editor reaches a reviewer who is very familiar with the topic of the paper, then it's more probable that the review will be good. Conversely, if the reviewer is not really familiar with the topic, they more likely to either write a poor review, unless they "get familiar" with the topic (which is a time-consuming process).

Since the MDPI standard operating procedure is to have MDPI staff identifying and inviting reviewers, and since the processes at MDPI are so accelerated (we request reviews in <2 weeks), I think it is relatively likely that MDPI reaches reviewers who aren't the best-qualified to review. In theory, these reviewers should decline to review the article, but some number surely accept anyway and then write a poor review. That could explain what you're seeing. If this is indeed the explanation, it'd indicate the editor(s) you interacted with aren't very good (whose name is on the emails inviting you to review?).

• As I wrote above, compared to the reviews I saw at other publishers, these reviews are excellent. However, this could also be explained by me becoming a better editor (which definitely happened - my current journals are closer to my expertise, and I have been putting in a lot more effort than I used to when selecting reviewers).

For the record of the 15 reviews above, 3 of them were by reviewers not invited by me. Of those three reviews, one is very detailed (this review is one of the two where the reviewers did their own calculations); the other two are not.

• MDPI allows reviewers to volunteer. One of these papers had three volunteer reviewers, but all three were plainly unsuited to reviewing it (no relevant expertise), so they would likely have written poor reviews too. I don't know why they volunteered. I declined all of them, but another editor might not have, which would probably lead to what you saw.

Finally: for the question "should you bother?" I think this linked question is highly relevant. Ultimately it's up to you, and you should not feel pressured to review. There is one thing that I do wonder about though: if you are the only reviewer for a manuscript, would you feel troubled? If yes, then that seems problematic for a lot of journals (I've written some papers that received had only one reviewer myself). If not, then what the other reviewers do doesn't seem like a major concern, and certainly a less important one than whether the authors take your comments seriously.

Update: I've since had the chance to look at more papers handled by other editors. Roughly 50% of the reviewers for the papers I saw had no relevant expertise in the topic (here I count, e.g., Stephen Hawking as having no relevant expertise for a surface science paper, even though surface science is a branch of physics and Hawking was definitely a physicist). Interpret as you will.

*I am no longer affiliated with MDPI.

Answer 4

I am a young researcher in the field of Solid Earth (geology and seismology). I have reviewed 6 papers and so far, I have a good impression of the journals I review for, namely Applied Sciences, Earth, Sustainability, and Remote Sensing.

Among them, I rejected 2, suggested major revisions 4. All of the manuscripts followed my suggestions with 2 rejected and the rest improved a lot after major revisions. None of them has been published and is still in revision (maybe from another review) with the time from submission ranging from 1 months to 6 months.

So I think in my field, MDPI journals are pretty fine. It seems the quality heavily depends on each journal, and I hope MDPI is improving its reputation.