I really can't understand why an inexperienced PhD student should be allowed to review a research article written by researchers of considerably more experience and expertise.
How do journals/editors select referees for submissions?
41
-
49Are you seriously suggesting only the most experienced researchers should be allowed to review? I suspect Stephen Hawking might feel that he has other things to do than review your papers. Also, a fundamental premise of publishing is that you publish so that everyone can review it. Research with rules about who is allowed to review it is highly suspect. – MBentley Jun 14 '16 at 14:32
-
19Note that the majority of users of this site are PhD students. Answers up-votes and rank will likely reflect that. – Cape Code Jun 14 '16 at 14:47
-
4@John: I should remark that one common way for an editor to find a referee is for the editor to look through the database of authors who have published in the same journal. – Willie Wong Jun 14 '16 at 14:48
-
63I feel like what's really being asked is "Some reviewer gave me a crappy review because they clearly didn't understand my work and I'm upset because I'm sure it was some snot-nosed 2nd-year PhD student and my work is good, dammit." Which I completely understand. But that means the question should really be "Why isn't there a standard method at Journals for reviewing reviewers? Why isn't there more oversight of reviews by managing editors? Isn't allowing any random PhD student to reject a paper out of hand bad for the state of the literature?" – sintax Jun 14 '16 at 16:34
-
3If you got a bad review the paper wasn't as good as you thought it was. I – Stevetech Jun 15 '16 at 18:52
150
Yes, it is fair.
- The quality of reviews written by highly experienced researchers is low.
- Many manuscripts have obvious flaws that should be spotted by the inexperienced.
- Reviewers are (in theory) supervised by the editor.
- Novice reviewers are more enthusiastic than experienced reviewers.
- If the paper cannot be understood by a PhD student, it will not be useful to very many people.
- PhD students may have more specialized expertise because they work on fewer projects, as compared to senior researchers (per @nayrb)
Of course, it depends on the individual PhD student. Some are better prepared to review than others.
answered Jun 14 '16 at 12:31
-
22I agree with the overall sentiment of your answer, but pretty much all of your bullets are at best heuristics. Obviously not all experienced researchers write bad reviews, and there certainly are valuable research contributions that will fly over the head of an average PhD student. – xLeitix Jun 14 '16 at 12:52
-
3"The quality of reviews written by highly experienced researchers is low" - from my experience this is sometimes true, but I would not take that for granted. Of course this depends on the PhD, researcher and definition of quality. – Bitwise Jun 14 '16 at 12:58
-
22@xLeitix Is there a non-heuristic way to select reviewers? – Anonymous Physicist Jun 14 '16 at 12:58
-
3@AnderBiguri It's a little worrying that you seem to know who the other reviewer was. – David Richerby Jun 14 '16 at 15:10
-
10You should also mentioned that many PhD students have greater expertise in a subfield due to specialization than the majority of postdocs or even professors. – abnry Jun 14 '16 at 15:53
65
In many disciplines of science, asking Ph.D. students to review papers is considered to be fair by the community. I can only speak for computer science here, but the following arguments may apply in other fields as well.
When a Ph.D. student reviews her/his first papers, this is typically done for his/her advisor. Advisors often check the reviews before turning them in, and thus avoid highly unfair or imprecise evaluation of the work of others.
Reviewing work is one of the best ways to get started with writing your own work. You get to know first hand what can make a paper weak and what you want to read as a reviewer. This is very valuable to the student.
A single review on its own is seldom the reason for acceptance/rejection of a paper. If there is a strong disagreement between the reviewers, then this is normally discussed in the case of conferences (which are prime publication venues in CS), or the editor has a look and will gather further information.
Ph.D. students often simply take more time to review a paper. Thus, they can find flaws in papers that senior researchers overlook in quicker reviews.
-
14Indeed. To your list, I would add that some PhD students have already published, so they do have experience concerning scientific publication. Also, if unsupervised by advisor (point 1), the editor might know them and consider their student status while looking at their review. – Emilie Jun 14 '16 at 12:28
-
Agree: the time I most spent reviewing a single paper was during my own PhD – mgalle Jun 14 '16 at 12:59
-
Thanks for your comments. It is true that students do take more time in reviewing the articles. – John_dydx Jun 14 '16 at 13:03
-
@Emilie +1, in my field, half of the PhD students already published an article during their Master. – gaborous Jun 16 '16 at 12:25
32
The assignment of referees to an article is not a random process where a computer draws a random number and matches it to "active researcher X" in the community. At least in my field the editors play a huge role in asking the "right people" to referee the article.
By making the focus on "inexperienced graduate students" you are focusing on the wrong things. An inexperienced graduate student who, nevertheless, has done research in similar fields and therefore has the expert knowledge to say something meaningful on the manuscript is certainly a much better referee than Prof. Dr. Messenberg whose recent research only overlaps with that of the manuscript in so much as that they use the same species of animal subjects.
The question you should ask is: "Should utterly unqualified persons be allowed to referee journal article submissions?" And the easy answer is, "No, since it is called peer-review not random-Joe-Schmoe-review."
You ask then: "How do we know whether someone is qualified?"
Answer: "You trust the editors; if you don't, don't send the manuscript to him/her/that journal."
As an aside: there are two ways that I've seen where Graduate Students come to review a paper.
The graduate student is passed the paper by his or her advisor to referee. Good advisors will only do so for appropriate papers, and may even provide guidance on how to referee a paper. Bad advisors are, well, bad (if an advisor gave an unsuitable paper to a student to referee, do you really think he himself will write a good referee report?)
The graduate student has independently came to the notice of the journal editor due to a previous interaction (conference presentation, paper submission, research discussion), and the journal editor feels that the student can fill the duties of the referee.
answered Jun 14 '16 at 14:19
-
2Exactly! If you submit a paper to a journal where you don't trust the editors to choose good referees, then you are doing it wrong. I can't imagine any plausible scenario in which you would trust the editors' judgment with the one exception that you fear they might choose a graduate student. – Anonymous Mathematician Jun 14 '16 at 14:43
-
I agree with the points you make, but, out of curiosity, how would you address the same question for conferences? E.g., in CS, which is mostly conference-driven. Since the PC changes every year, the "trusting the editors" parts is a much weaker point (or so it seems to me). – Clement C. Jun 14 '16 at 16:57
-
@ClementC. : I would shake my head in your general direction and give you my condolences, while secretly having a bit of schadenfreude at your expense. Or more seriously: I have zero idea how conference-driven fields like CS are run. But if a field has systematic issues where one cannot necessarily trust the conference chair (is that the right term?), that issue should probably be addressed as a higher priority than the symptom of whether graduate students get to review submissions. But that's just my knee-jerk reaction, since I have no experience nor expertise to answer your question. – Willie Wong Jun 14 '16 at 18:17
-
1@ClementC.: I'm so happy to be able to introduce you to Avenue Q. Yes, you're welcome. – Willie Wong Jun 14 '16 at 18:26
-
2@ClementC.: To mirror the words of the answer: In CS, the conference chair is not a "random Joe Schmoe", either, nor a lonely master who makes all decisions single-handedly. Rather than that, there is a program committee, which is typically populated by more or less experienced researchers with strong connections in the community; the "higher" the conference, the more well-known will various of the program committee members be. Hence, there is not really a trust issue there, either. – O. R. Mapper Jun 14 '16 at 18:49
4
Of course it is great practice for the student (critical reading improves writing as other pointed out).
But it is also fair for the authors of the paper. In general PhD students, specially towards the end, are (or should be) highly specialized in their specific (sub-) subfield and should know the literature better than anybody else. While this does not make them apt to review any paper, they are often the best person for reviewing something very related to their field.
4
Why isn't there a standard method at Journals for reviewing reviewers?
There are some standard methods at some journals (e. g. keeping track of response time and such) and also there are the editors who keep track of the quality of reviews. Any standardized method on reviewing quality of reviews will most likely be flawed, however.
Why isn't there more oversight of reviews by managing editors?
I don't think that this question is based on a right premise.
Isn't allowing any random PhD student to reject a paper out of hand bad for the state of the literature?
Reviewers do not reject papers. Reviewer suggest rejection. Editors reject papers. Editors don't select random PhD students.
Why isn't there any formal training required (e.g. certification) before a less experienced researcher is allowed to review journals?
Who would you like to grant a reviewer certificate? Experienced and reputed researchers probably. These researchers should be able to judge the quality of reviewers and then decide which reviewer could be qualified to review a paper and should choose stronger reviewers for more critical/potentially ground breaking papers and less experienced reviewers for more "standard" papers. Oh wait, that's exactly what editors actually do these days.
3
The title and body of your question are asking different things. The title asks whether it's fair for PhD students to review research papers, which I think is certainly the case.
The alternative would be for full participation in the research community to require formal certification. For example, should a figure well-known in a field, with decades of experience (say, as an engineer), be refused permission to fully participate in the research process simply for not having a PhD? Otherwise, would they cross some arbitrary threshold of "experienced enough"? What if that figure used their savings to self-fund a PhD? Would any existing permission be revoked, since they're now a student? What if they abandoned it half way through, would they go back to their prior status? These kind of issues seem like an unnecessary administrative burden, for a restriction which needn't exist, and would seem unfair to me, so in general I would say that it's fair for PhD students to review research papers.
The body of your question asks something different: whether inexperienced PhD students should be allowed to review research papers. I think the other answers tackle this well, but I would also say that performing reviews is one way of gaining experience.
For the record, I am currently a PhD student, I was following work in my field for years before I started, and I have performed reviews.
-
When I got my first review to do (as a fresh PhD student a few months after graduating with my Diplom [Master]) my professor said: "Well, your Diplom means that you are a qualified professional with all professional rights and duties as a chemist." I'd say the Diplom or Master is a quite formal certification... – cbeleites supports Monica Mar 20 at 22:04
3
You are asking:
Is it fair that a PhD student should be allowed to review a research paper?
Yes:
It should be checked that a paper is written in a clear enough way that anyone with a reasonable background in the research area should be able to understand the paper. A PhD student is more likely to pick up “assumed information” that should be included in the paper, but is not, then someone that is a leading researcher in the given subfield.
A Phd student can also check that the “logic” in the paper is correct etc, and is more likely to rework all the proofs then someone with less time.
But if you had asked:
Is it fair that only PhD students should be reviewing a given research paper?
No:
- It needs an expert in the subfield to know all the past papers on the subject.
- It needs a long standing expert to know how a paper relates to what is being done in other fields.
Therefore a mix of reviewers is needed.
-
Would the down voters like to comment, as my answer seems to be getting as many down votes then upvotes. – Ian Jun 15 '16 at 13:45
0
Your question is simply ambiguous. You ask:
why an inexperienced PhD student should be allowed to review a research article?
This can be read as one of the following two different questions:
"Why an inexperienced researcher (that is by chance also a PhD student) be allow to review?"; or
"Why a PhD student, which I assume in advance to be inexperienced, because all PhD students are inexperienced, be allowed to review?".
For the first interpretation, the editor or program committee should take the inexperience of a reviewer into consideration. For the second interpretation, it lies on a false premise, and so is wrong.
