26

Recently I have been appointed as a reviewer for an IEEE journal in CS. This journal is indexed on Scimago and its ranking is a Q3 journal.

I know that one cannot know all the topics covered in the diverse areas of CS, but as a reviewer, I have been assigned some papers that do not specifically belong to my field of expertise. Even though the journal appears to check the familiarity of the reviewer with the topic before reviewing it, I just feel it is not close enough.

Apart of that sometimes I get some papers in which the practical part or the methodological part is jumped quickly, and for me, it lacks consistency and is rather descriptive; I know the limitations of space, but I believe some authors should put more effort on that section.

Also, how can I know that the results that they present are for real? I have read many cases in which some authors fake their own results or push them to show nice results when in reality is not like that.

What to do in the aforementioned situations?

  • What is the name of the Journal? What are your fields? – Coder Jun 8 '17 at 11:03
  • 22
    I get some papers in which the practical part or the methodological part is jumped quickly, and for me, it lacks more consistency and to be more descriptive -- Then say exactly that in your review! – JeffE Jun 8 '17 at 13:09
  • 4
    Your title says "dubious results" which is a strong statement. Your question doesn't read that strong -- it sounded like you were doubtful only because you've heard of cases of authors faking their results. Can you clarify? – aparente001 Jun 9 '17 at 5:29
  • This question appears to contain at least two very distinct questions. It may be better to ask them separately. As for "Also, how can I know that the results that they present are for real?": You cannot (in general). Apply some plausibility tests, check the calculations and then, if still in doubt, tell the editor about your gut feelings. He has to decide. – Trilarion Jun 9 '17 at 13:04
35

In any field, once you start reviewing, you're going to see a lot of problematic submissions, because there are a lot of people out there who submit really bad manuscripts. In some cases, the authors are unaware of their problems, while in others they're trying to slip a minimal paper past peer review because they aren't really interested in the science---they just need the publication to graduate or get promoted.

Your job as a reviewer is to make sure that what gets published is credible. That means that if you don't understand something or don't feel that the evidence presented supports their conclusions, you have a duty to report that as a problem with the manuscript. Depending on the severity of the problem, you might recommend either revision or rejection---and the handling editor will make the actual decision.

You shouldn't, however, demand that authors to do new work just because you think it would be interesting. You also shouldn't expect to know every tiny detail of their methodology. So, where to draw the line? Here is how I think about it:

  • The authors need to provide enough methodological detail that no significant implementation decisions are missing.
  • The conclusions need to be well-supported by the evidence presented.
  • The overall thrust of the paper should be interesting to a typical reader of the journal (this one varies highly by journal).

None of this will allow you to detect clever fraud, but not-so-clever faking often leaves a clear trace (data that looks too nice), and cherrypicking of data is usually detectable because the authors present data from very few samples or trials.

  • 19
    Also, keep in mind. It is not your job as a reviewer to know for sure that their results are correct. If they print fraudulent/cherrypicked data, it will be dealt with when the next researcher tries to duplicate their results, or tries to build on their incredible new method and finds that it does not work as claimed. There will be some author who will write a counter-paper (if it is worth the time to debunk). All of this will reflect on that author lying, none will reflect on you for letting in a paper where you would not have been able to disprove the results without thorough testing. – EvSunWoodard Jun 8 '17 at 15:46
  • 5
    @EvSunWoodard yes, but often "when the next researcher tries to duplicate their results, or tries to build on their incredible new method" is never. It's not the reviewer's responsibility to ensure correctness, but if there is reason to suspect fraud, I think the reviewer ought to use due diligence and perform some checks if possible. – user24098 Jun 9 '17 at 11:59
  • @dan1111 - If no one ever builds on the method, it's probably not doing any harm either. In an ideal world, everyone would just be open and honest with their findings. Obviously, if you have some reasonable way to check some figure or method, you should give it a shot. But you shouldn't feel you are personally responsible for the contents of a paper just because you reviewed it. – EvSunWoodard Jun 9 '17 at 15:27
  • @EvSunWoodard not necessarily. Plenty of results are relied on or built on by further work without being checked directly. I agree that you aren't personally responsible for the contents of the paper by reviewing it, though. Your responsibility is simply to do a diligent, thorough review. This includes checking what you can when you have doubts, but often you can't realistically verify the findings. – user24098 Jun 10 '17 at 8:56
21

If you feel unable to referee a paper (due to any circumstance), refer it back to the editor, possibly with a list of colleagues which you suggest.

  • 2
    This is good advice. In fact, most journals that ask me to review papers include this in their letter to me. – GEdgar Jun 8 '17 at 11:51
2

Also, how can I know that the results that they present are for real, I have read many cases in which some authors fake their own results or push them to show nice results when in reality is not like that.

Unfortunately, that's not uncommon. However, we can't assume fraud. Doubt is not enough, if you can know for sure that the results are fabricated, then put that in the review. Otherwise, it is up to those who replicate later to find out if the results are not reliable.

I get some papers in which the practical part or the methodological part is jumped quickly, and for me, it lacks more consistency and to be more descriptive;

This is more of a problem. Chances are that others reading the paper will also find the lack of consistency and coherence troublesome. Depending on how bad it is, you can either suggest a revision or reject on this basis. If it is incoherent or obfuscated then you owe it to future readers to give feedback on this.

but as a reviewer, I have been assigned some papers that do not specifically belong to my field of expertise, and even though it appears to check the familiarity of the reviewer with one topic before reviewing it, I just feel it is not enough.

If you are not comfortable with the subject, basic decency is to return it to the editor. I would never suggest rejection a paper that I do not fully understand, I would just return it to the editor. Do to others as you would have them do to you.

  • 4
    As to the last part, it might be possible to distinguish between a paper that you don't fully understand and one that is not understandable. The former should be returned to the editor, the latter should be rejected (or major revisions demanded). – Nate Eldredge Jun 8 '17 at 18:12
  • I agree, if you know the subject well and the paper is incoherent or deliberately obfuscated then it is a different thing. – dan Jun 8 '17 at 18:14

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.