5

I know it's a free process and it depends on reviewer if they want the data and for author if they want to submit it.

Questions:

  1. What are the chances of a reviewer asking for data?

  2. Why might reviewers ask for data?

Improve this question
7

7 Answers 7

Reset to default
10

I have asked, or been asked, for data a vanishingly small but non-zero number of times. But I'm pretty secure in asserting that it's rare.

As for why - I would hope they would specify this when they requested it. Among the reasons would be to see if your results are genuinely obtainable from your methods. This is not necessarily just looking for fraud, etc. but checking to see if your writing is clear enough that by "following the recipe" they get the same answer.

Another possibility is they want to try an alternative analysis. For example, in my field, there is a variable selection method that is commonly used and also wrong. Generally, I ask authors to re-do their analysis with a more appropriate method as a reviewer, but it's entirely possible this won't change the answer. Having the data would let me check for myself.

Improve this answer
5
  • Wait, so you are publishing something which cannot be reproduced since you don't include the data needed to reproduce it?
    – user64845
    Commented Feb 22, 2018 at 18:37
  • 2
    @DSVA Clearly you'd be surprised at how these things work in some fields. Sometimes the data is a controlled, owned resource, which may be monetizable. Sometimes there are legal barricades (you can't just arbitrarily disseminate people's medical information, for example). In exceptional cases, the size of the data may just be so humongous the only reasonable way to transfer it is to physically ship it.
    – zibadawa timmy
    Commented Feb 22, 2018 at 18:47
  • 2
    @DSVA No, I’m not placing medical records on the public domain
    – Fomite
    Commented Feb 22, 2018 at 19:01
  • @Fomite well, in that case the study could be reproduced with different people, right? Your study is not about how you got the results from the available data but about the data itself. In this case the data itself is not needed to reproduce the study since one could collect the data him/herself. That's different for example in CS where you develop an algorithm and demonstrate the davantages on a data set. In this case the code and data set is needed to reproduce the results.
    – user64845
    Commented Feb 22, 2018 at 19:29
  • 2
    @DSVA Indeed - and my field tends to emphasize replication over "I ran your code on your data and got the same answer"-style reproducibility. It's just good to bear in mind that different fields have different standards, needs and restrictions.
    – Fomite
    Commented Feb 23, 2018 at 1:31
5

What are the chances of a reviewer asking for data?

It would be different from one field to another. For example, there is a small chance to be asked for more data (and very tiny chance of being asked for raw data in the fields such data analysis/science and machine learning, and optimization. The odds will increase when the claims are very surprising (too good to be true!) or not clearly backed by theoretical proofs and discussions.

Why might reviewers ask for data? To investigate the correctness of the presented conclusions and claimed achievements based on the obtained data. The reviewer/editor usually provide enough details on why such deep investigation is necessary and how it can affect their final decision.

How to decrease the chance? To decrease the likelihood of such requests, that may make the review process even longer, one can:

  • provide as much supplementary data as possible right at the time of submission,
  • share codes (if applicable) via Git repositories to give the reviewers the confidence that the results are genuine and correct, and
  • provide sound and clear theoretical proofs (when applicable) that support the claims regardless of the quality of the data.

In general, it is really appreciated to make the researchers and their outcomes reproducible. That's why many researchers publically share their valuable datasets/outcomes.

Further Readings:

  1. How to share a scientific dataset with the research community?
  2. How to share to mention/publish large datasets?
Improve this answer
3

In my field (structural biology), we have repositories where we are required to deposit data and models prior to publication. The deposition date establishes precedence, and the data and models are only made public once the associated publication comes out. The repositories will grant the editor and reviewers of the article access to not yet released data and models if they request it during the review process. I don't know how often they actually ask, but I like the fact that the repositories make this possible in a formal and non-awkward way.

Improve this answer
3

What are the chances of a reviewer asking for data?

If it is important for reproducing your results, they should be high.

Why might reviewers ask for data?

Because scientific research is built upon reproducibility of results.

Improve this answer
2
  • As other answers and comments have mentioned, there are many reasons to not distribute data, and "reproducibility" in many fields is more about "getting the same result with new data" rather than "getting the same result with existing data." Reproducing results with the same data seems mostly limited to algorithmic work, i.e. in CS, where you might be talking about reproducibility across computational environments.
    – Bryan Krause ♦
    Commented Feb 22, 2018 at 20:36
  • To provide one example in computational chemistry, raw structural data is often needed (but not often supplied) if you want to do a comparable calculation on the same system but with a different method or computational program/package.
    – user44476
    Commented Feb 22, 2018 at 22:14
3

You are asking the wrong question. Do not care about whether the reviewers might ask for your data. Ask yourself if there are any real reasons why you should not just provide the data!

Personally, I never had the case that a reviewer asked for my data if I did not include it. But when I included it, I had several reviewers who pointed out that my paper comes with data. This was a huge bonus in the review and, as far as I can estimate this, was one of the reasons my paper got accepted.

Despite from getting your paper accepted, publishing data is a great contribution to science. Other researchers are able to reproduce your results, discuss it in more detail, and may advance faster.

Improve this answer
2

I've seen hundreds of reviews as an editor, and don't remember ever seeing a request for raw data. So the answer to your first question is "almost zero". If it does happen, I imagine the reviewer will have to explain to the editor why (s)he needs the raw data, and if the editor is convinced (s)he'll pass the request on to the authors.

EDIT: The reviews I saw were in physics. Things might be different in other fields.

Improve this answer
0

It depends on the scope of the article, as well as the scope of the journal/conference.

A paper which can be very acceptable at a conference, without any experimental results or data sets, might get "strong rejection" from reviewers of another conference.

One of my manuscripts was rejected because

The paper just provides some theoretical results, without demonstrating the practical effects on real-world scenarios.

The previous version of the same manuscript was rejected by another set of reviewers of another conference because

Although the work has good empirical data and clear experimental results, the lack of theoretical work leaves me no choice but recommending a rejection.

The best way to understand this is to check other accepted papers in that conference/journal. Do those papers mention the experimental results as a crucial part, or do they rely on their theoretical results and provide some basic experiments? The answer to this question might help you to decide whether the editors will ask for your data set.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .