In experiments involving study participants, why are some studies allowed to only have a small number of test subjects?

Question

I noticed that in my field, studies involving testing with human subjects often only show a small number (N>=10) of them. While I understand that researchers are often limited on funding and time, I always thought studies like that would require a certain minimum of subjects far greater than that.

Different sources I found (e.g. this one p.3, this sample size calculator, or Wikipedia) seem to support that impression.

My field is audio signal processing, and the studies that caught my attention were about speech intelligibility, hearing comfort and other measures related to human hearing.

My question is:

Why do some studies only involve a small count of subjects when statistical theory suggests a high number of participants is required to make meaningful assertions?

(I assume that there is something I missed or something that got mixed up in my head, and I'd really like to find my error)

Answer 1

Although large samples produce more robust results, small samples are not completely devoid of usefulness - they're just more likely to be wrong. As long as the reader keeps this in mind, there's nothing fundamentally wrong with publishing these small-sample studies.

Further, sometimes you just cannot get enough samples to make an assertion with confidence. Examples:

GW170817 was a one-of-a-kind neutron star merger event. Since the rate of neutron star mergers isn't well-known, one could easily wait another five years and not get another sample (plus you'd need hundreds of samples to make the kind of statements you're looking for).

The Berlin Patient - there's no cure for AIDS. Even if this single case were a fluke, it's still worth reporting since it's a signpost telling researchers where to look.

Habitability research - The question of whether aliens exist captures the public imagination. Like it or not, we have to approach the problem from what we know, and we know only one planet with life (i.e. sample size of one). You could of course just decline to work in the field until we've discovered several hundred alien civilizations, but then you could be waiting forever. Plus with no idea what to look for, observational astronomers would have a much harder time finding these civilizations.

Answer 2

In usability research, just 5 participants will reveal about 75% of the problems with a system. This seems to fit very well for qualitative research (so perhaps the hearing comfort studies you mentioned?). From my own experience of running studies with 8-32 participants, it only took running the first few participants before every subsequent participant felt like a repeat.

Sure, you might not be able to get statistical significance with 5 participants, but you can learn a lot.

_{Figure taken from Nielsen's article Why You Only Need to Test with 5 Users.}

Answer 3

Of course this depends a lot on what you're after. If you are looking for differences or some other effect within an individual, then your sample size are measurements within the individual rather than number of individuals. In such scenarios it is legitimate to study only a single (or a few) individuals.

Also, if your individuals are not studied per se but act as "instrument" (testing answer by @AustinHenley, or setting up a sensory panel, i.e. you use N individuals to smell/taste the effect of something you do/produce) you need a number to cover most of the important general variation we have in the population. But that number actually isn't that large because typically only people who are good at the task at hand will be used. I.e., smell-blind people do not apply for a sensory panel in general, you only need to cover the variation in receptors you encounter in people who are good at smelling. And that are far fewer than, say, the number of people you'd need to look at to find out which proportion of people are good at smelling, or the number you'd look at in order to be able to predict, say, the probability that a randomly chosen unknown person will like the smell/taste of what you produced:

If you need to make conclusions about a population or about applicability of your findings to unknown individuals of a population (if we train our method on an individual, this is how well it will do), you need to representatively cover the variability in that population, and this requires large sample size.

---

For the population questions, I fully agree with @Buffy's analysis, but I'll go a bit further about possible reasons.

• The maybe most scary (to me) reason is that it is IMHO perfectly possible/far too easy to publish studies with low quality due to far-too-small sample size.
  Note that I've been working in a life-science field, i.e. close to where we have lots of reports about studies not being reproducible since years, and at least in my sub-field not much of an effect on the sample size.

• One particular problem I see with small sample size studies is not scientific (as in limitations of conclusions that can be drawn) but political. The "politics" of academia focuses very much on novelty. This basically means that once a small sample size study is published, any larger follow-up study has to present extremely good arguments to overcome the "this is known already" bias in funding.
  This means, in addition to being uncertain, small sample size studies may prevent getting certainty.

• In (industrial) experimental design it is often recommended to start with a preliminary experiment using, say, 1/3 or 40% of the available resources. Then do a preliminary analysis and if necessary re-adjust the allocation of the remaining resources accordingly.
  This takes time and means effort. However, if the preliminary results are good, they can be published. The follow-up study will then potentially face a lack-of-novelty hurdle to publication.

• Master's and PhD theses by definition are the work of one student. This limits how much work can happen in one study. And it pushes towards experiments that are inconsistent in the long term: students cannot be abused as lab robots (which would help getting together good sample sizes - but good lab technicians are more expensive...) as they need to contribute scientifically. One of the easiest ways to do that is to improve the experiments over what has been done before - leading to large numbers of small sample size studies/fragmented series of experiments.

• The requirement of "own work" sometimes causes students to not speak openly about their project and not seek advise. Every so often that leads to flawed experimental design and/or too small sample size and this is realized only during data analysis (or not at all). But then it is often too late to do anything but try to rescue the existing data. And rescued into a paper it must be, because otherwise the student won't have the paper they need.

• I see another potential conflict in mixing the evaluation of scientific work (as in grading the PhD student's work) and arriving at scientific findings:

  • if a student in their well-planned and well-conducted study finds the "desired" effect, that implies both that the student did their work well, and we have a scientific advance. All is fine.
  • However, if things don't work out nicely, things are much more difficult: Was the failure due to the student not working well (I come from a wet lab field)? And/or is there no effect? In other words, the student not finding an effect has to put in much more effort in demonstrating it is not their fault.

  Now consider putting a set amount of effort either in n underpowered studies or in 1 with good sample size. If the one with large sample size fails, you don't have a single paper*. If a small sample size study doesn't find an effect it is simply not published, and you move on to the next, because it is usually too much hassle to make a paper publishable with negative finding. But keep in mind that small sample size papers are not only underpowered, but they also "provide" a high "chance" to produce false positive findings (i.e. overestimate effect) - and that means a paper.

• Our brain tends to underestimate the effect of chance. This means without doing the statistics, we're likely to intuitively underestimate chance and be overconfident in our findings.

• There are studies where it is practically speaking impossible to obtain anything close to the required sample size**. And there's nothing wrong scientifically with case reports and small studies as long as they are clearly indicated as such, and the conclusions take the required caution and limitations are clearly stated.
  Often, it is also practically impossible to obtain representative samples. I'm still OK as long as a) limitations are openly stated, and/or b) even applicability is checked by plugging in reasonable guesstimates for prevalence/incidence/class frequencies to at least give a critical thought to suitability for the application in question.
  However, I see far too many studies that needlessly "rescue the world on the basis of 3 mice" or that work on, say, 20 samples of tumor tissues that are anyways cut out of patients with a disease that is neither rare, nor is the tumor volume small, nor is all the tissue needed for correct diagnosis in order to properly treat the patient.

• Finally, I consider it the worst possible waste of experimental effort (and that's even worse if test subjects/animals are involved) if a study is so small that a back-of-the-envelope calculation would have discovered that even in the best case no practical conclusion can possibly be drawn.
  (e.g. reserving 4 independent test patients in development of a medical diagnostic [pos/neg for some disease] out of your 20 patients. Assuming all 4 are correctly classified, this gives you an observed accuracy of 100 %. But the 95 % confidence interval for accuracy ranges from around guessing to perfect.)

---

* real life example:

1. I've met a PhD student who was working with a ≈ 500 patient sample size in my field. Measured them all, found there was a flaw in the design of the measurements (not properly randomized order of measurements, thus strictly speaking drift in time of the measurements could not be excluded as cause of the observation - which however looked like real effects from a physical/spectroscopic point of view). Realized this, and re-did all the measurements. Did not find the effect which was expected from long experience of the supervisor with similar situations and preliminary experiments. It is not clear whether there is truly no effect after all, or whether something else went wrong with the experiments (non-negligible risk for what they do), or both.
2. Same field, other university, other students get their PhD on the basis of, say, 20 patients. Not randomized, never thought about experimental design, not to speak of limitations of their study.

I'd judge student 1 much better than students 2 in terms of the scientific work they did. But they'll have to face a struggle to convince their committee of that: the committees of 2 clearly were not aware of the limitations, nor were the referees of the papers. So students 2 happily publish multiple papers in the time where student 1 struggles whether their study can be published. And student 1 did admit to a mistake, so there's evidence of student 1 not working well (they did make a mistake after all) - whereas there is no such evidence in the studies of 2....

** I'm also fine with the practical limitation that in many interdisciplinary fields if you need significant work of others (in our case, e.g. reference diagnoses by pathologists reviewing every single of our samples) those others may want to see preliminary data published in order to make sure they don't waste their effort with people who don't know what they are doing and/or are not serious about the application.

Answer 4

I will go out on a limb here and suggest that you are correct. Too many studies are too small to have meaningful results. I fact a google search for [scientific studies not reproducible] turns up some scary results. One is that only 10-30% of studies published in journals can be replicated.

There are many reasons for this, but poor research design is a main culprit, including too-small sets of subjects.

As to the reason for the small sample sizes you can point to both money and time. It takes more money and time to sample a larger set in all but trivial situations. But it can be hard or impossible to get a larger body of subjects for many studies. The potential universe of subjects may be widely distributed in space but sparse within the larger population. Sometimes many factors may be required for acceptance, leading to rejection of many. Some studies may just run up against opposition from the population due to many factors, including simply that it takes effort on the part of a subject.

In addition to the size factor is that many studies rely on using students at the same university as subjects. These folks are not representative of the population as a whole unless the population itself is defined to be university students.

Another problem with samples is that the subjects are always assumed to be "randomly" chosen from the overall population, but it is very difficult to assure that in many cases. This is the problem with using university students, in fact.

Small sample sets are perfectly natural for preliminary studies that can give direction and provide refinement to the design of a larger study. But, as you suggest, it is hard to extrapolate results from a small sample to a large population if that population has much variability.

Moreover, small studies cannot, by the nature of statistics, capture subtle effects. If you are trying to determine if more "mumbles" have characteristic "A" or characteristic "B", then a small sample size of "mumbles" will work if, indeed, the actual population breakdown is 70-30, but statistically unlikely if it is 57-43.

For reasons like the above, many researchers are moving from a standard 95% confidence interval in studies to, perhaps, 99% (or more). This helps weed out many of the design issues that cause the problem of non-reproducible studies. But that is risky for students who want and need some results to put into a dissertation and are limited in both time and money. But while small sample size is fine for initial, exploratory, studies, it is not for determining the state of reality.

Answer 5

The main reason that psychoacoustic experiments only utilize a small number of subjects is that the goal is generally not to determine the average performance of the population, but rather to characterize the limits of perception. With an N of 1, you obviously do not know if you have measured that individuals limit of perception, or got lucky and measured the limit of perception of the best individual. Based on the history of the field, from a bayesian vantage, there is a ridiculously strong prior that the first author of a study will be the best trained and most highly motivated subject and have the best performance. There is also a strong prior that with sufficient training and motivation that a substantial percentage of young healthy subjects can achieve similar performance. Studies typically conclude that humans can do task X at a minimum of level Y. There is so much data for each of the few subjects, that one can conclude that each subject's performance is highly unlikely to be from chance and that it actually represents performance. That said, it tells you little about the performance of a random individual.