2

I'm working in an empirical field and have recently had a paper about a data collection effort and subsequent analysis thereof which was provisionally accepted, conditional on revisions. The negative remark most difficult to address is that the dataset I acquired was considered relatively small; How can I justify the small size of the dataset in a sound manner due to practical difficulties in acquiring data?

It is not possible for me to collect any more data in terms of money, manpower or even time... which all add up to more or less to "research capital". I have seen other datasets used which are not much bigger, but there are also corpora which are bigger by a significant margin. Likewise, some datasets are quite similar while others are completely unrelated to mine. Finally, while some corpora are freely available, for the majority of the papers in my field, it is quite rare for publications to explicitly state how one can acquire the data for their own purposes, and so the actual ability to even aquire data second-hand seems to be spotty.

Basically, I would need to reformulate the paragraph above into some form which I can put into the paper which needs revision; What can I do?

Improve this question
6
  • 2
    This boils down to statistics, in particular a power analysis. Either your dataset is large enough to support your conclusions with reasonable certainty or it is not. Difficulties in obtaining data are not relevant for that question. However, the reviewer didn't reject the manuscript, did they?
    – user9482
    Commented Dec 15, 2017 at 8:38
  • @Roland no, none of the reviewers rejected the paper, but there were many remarks about things I have to address. Most are not a problem, but this is the one which I really "can't do much about" except, as you recommended, to prove that at least the results for this specific study are significant. However, the paper was primarily about the dataset itself, so if it's not resuable then it's a huge black mark against it.
    – errantlinguist
    Commented Dec 15, 2017 at 8:52
  • 2
    "However, the paper was primarily about the dataset itself" There is always the "it's useful for meta-analyses" argument if the data is sufficiently rare.
    – user9482
    Commented Dec 15, 2017 at 9:05
  • 2
    Could it be that the reviewer want you to acknowledge and discuss the problems of small sample size, i.e. that an underpowered study could result in overestimated effect sizes, and thus, a false positive?
    – Mark
    Commented Dec 15, 2017 at 9:07
  • @Mark yes, possibly-- I hadn't thought about that. The comments were not directives like "add exact p-values and describe how you defined confidence intervals" but rather actual "comments", e.g. "the dataset is quite small..."
    – errantlinguist
    Commented Dec 15, 2017 at 9:14

2 Answers 2

Reset to default
2

This answer will expand on some of the notes in the comments and OldDoc's answer. I would suggest three things:

  1. In the data section, acknowledge that the sample size is relatively small. Note that this is in line with other published work addressing the question, and that data acquisition related to this question faces challenges A, B, C... Explain that the analysis and discussion will address the relatively small sample size.
  2. In the analysis/results section, make sure to report the p-values and confidence intervals. The goal is to account for the small sample in the statistical analyses.
  3. In the discussion, explain how having a small sample size affects the power of/your confidence in your results and the generalization of the conclusions. Help the reader understand in what ways a small sample is a limitation for your work without selling it short.
Improve this answer
6

As has been said, a satistical analysis of your data set will clarify with how much confidence and accuracy you can conclude your findings. This an essential part of the scentific process

In the natural sciences particularly, data collection can be problematic and limited by time and budget, but the community are aware of this, as should be your reviewer.

All that is required is that you report your findings appropriately even if you don't necessarily like the answer! In extreme circumstances you may have data sets that are so scant or varied you simply do need more work. It happens.

Science progresses with knowledge which includes less than optimal data, as does helping those that come after you do better by explaining in your paper how you could improve your own approach.

I agree the reviewer is probably asking for this kind of analysis and not why you were unable to do as much work as you would have liked

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .