12

The research (Delphi study) started with NN study participants in Round 1 (gathering demographics). Ten percent of the participants dropped out and did not complete Round 2 or (of course) Round 3.

I'm unsure how to report this.

Do I report the demographic results and then add a note to the Round 2 and Round 3 results ("Note: Of the NN people who completed Round 1, only Nn completed Rounds 2 and 3.")

Or do I remove from the Round 1 results the data gathered from the drop-outs?

Or ... ?

  • 5
    Read a few longitudinal studies, you will know how to report this. – ceoec Apr 20 '15 at 18:12
  • 10
    From the, title I thought you meant some of your coauthors dropped out, not respondents. I think 'respondents' might be a better phrase than 'study participants'. – smci Apr 20 '15 at 22:22
  • 2
    Or, actually, an explicit 'study participants' would also help. As it is the title is quite ambiguous - particularly for people whose fields of study don't include studies with participants. – E.P. Apr 21 '15 at 1:15
  • 2
    "Participants" is an APA style guide approved and extremely common term in psychology. No psychologist would search for "respondent". Unless "respondent" is really common in some other field, I'd recommend that we stick with "participant" here so later generations will more easily find this question. – S. Kolassa - Reinstate Monica Apr 23 '15 at 15:02
  • 2
    Would it be helpful for future searches if I rewrote the question as "... study participants (respondents) ... "? – RJo May 18 '15 at 22:07
19

You should present demographics for every round available for accuracy, completeness, and your own personal sanity. Drop-outs and people lost to follow-ups, are still data points, especially in medical/psychological/sociological studies. They may not have any associated data, but they were recruited and participated at least during the initial phase of data collection (demographics), and not counting them can imply other things.

Anyways, I like using an example to show why it helps present a clearer image.

Let's say 100 animals sign up for a study at Round 1. The demographics are as follows: 50 dogs, 50 cats.

However, when Round 2 rolls around, 20 cats are nowhere to be found. The results are collected from the remaining subjects; 25 dogs are peanut butter lovers, and 15 cats are peanut butter lovers.

If you only say that 20 animals dropped out, the information presented here doesn't mean very much, since you don't know what animals dropped out. In actuality, both dogs and cats had a 50% split based on the population of data collected, but presenting information only partway can be misconstrued as perhaps it was 25/40 dogs and 15/40 cats, because you haven't provided any. In addition, neglecting to mention that you originally had 50 dogs and 50 cats and only presenting that you had 50 dogs and 30 cats in the final results could indicate selection bias or a lack of interest, as opposed to losing cats to follow-up exams.

So you would present in a nice table or summary:

During Round 1, 50 dogs and 50 cats were recruited for the peanut butter study. However, 20 of the original 50 cats (40.0%) dropped out before Round 2 testing and could not be replaced. During Round 2 testing, it was found that 25 of 50 dogs (50.0%) and 15 of 30 cats (50.0%) preferred peanut butter.

  • 1
    Your answer and example were helpful because it illuminated for me the importance of traceability of results from one round to the next. – RJo May 18 '15 at 22:24
0

Some time ago, we developed a work titled as Adaptive Q-Sort Matrix Generation: A Simplified Approach [1] to support our research. This work aims to implement a system related to the DELPHI method. In particular, the goal was to develop the Q-Sort method for information retrieval of an experts' panel. The reason why we did it was to provide a new and simple algorithm to generate the Q-Sort matrices that adjust to the size of a given survey. Therefore, we can have more questions whose weight is null for the outcome of the round. On the same hand, giving experts the need to prioritize some questions, above others, in order to reach a consensus in a more direct way.

[1] Oliveira, B., Calisto, F.M., Gomes, L. and Borbinha, J., Adaptive Q-Sort Matrix Generation: A Simplified Approach.

7

I fully agree with the other answers that you should do statistical analyses on your dropouts, and report and think about the results. Did people who dropped out differ significantly from participants that stayed on? For instance, more women may have dropped out, or more men, or the less successful in initial rounds. If so, you may have confounding effects like selection bias, which you should discuss. (Or you may already have your next research idea right there ;-)

As others write, don't just drop data. Data is precious. Use all you have.

The CONSORT group (which stands for "Consolidated Standards of Reporting Trials") has some materials. It also publishes a flowchart template (MS Word doc) that seems to be becoming the norm for reporting dropouts in the course of trials. I know of a few journals that require exactly this kind of flowchart for submission, which will usually end up in the online supplement of the article. I find such a structure enormously helpful, certainly more so than a free text description that one needs to wade through. I'd strongly recommend you include this kind of flowchart.

  • 3
    Although the research did not involve trials, the pointer to CONSORT was helpful for the flowchart and for two other items I found there: a checklist and an explanation of research methods for randomized trials. Collectively, the content has increased my understanding about research in general and solutions for presenting data. – RJo May 18 '15 at 22:19
25

It is a very bad thing to remove people who dropped out from your data set. The problem is that you do not know whether dropping out is correlated with the effect that you are studying.

For an extreme example, consider a study on the effect of being shot at on soccer ability. In round 1, people play soccer, then they get shot at randomly with a gun that might or might not hit them, then they play round 2 of soccer, then they get shot at again, and then they play round 3 of soccer. Of course, anybody who actually gets hit when they are shot will probably drop out. If you eliminate those people, you will vastly underestimate how badly soccer players are affected by people shooting at them.

This may seem like a rather extreme example, but things very much like it happen frequently in biomedical or psychological studies, just with less obvious causal connections.

Report exactly what happened, and take the missing people into account when you are computing your effect size. If you need help on the technical aspects of that, you should ask on Cross-Validated.SE.

  • 7
    could the example be a bit less graphic, like Chefs using sharp blades and cutting a finger, instead of shooting people – user-2147482637 Apr 21 '15 at 4:36
  • 14
    @user1938107 Why? The current example makes the point well, and small finger cuts probably do not have a large effect on cooking ability. – Potato Apr 21 '15 at 4:44
  • 1
    @Potato Because i find it unnecessary to convey the point. If the goal is an extreme example of user studies, it could easily be phrased as: A medical research project testing the efficacy of a pill formula to fix migraines on human subjects against the current leading brand. There are 100 subjects, 50 have the new pill. Out of the 100, 15 people are put into a coma from the new medication (or die, if you enjoy it so much). Those 15 can not answer round 2 questions if it makes them sleepy. You would not remove the 15 people. – user-2147482637 Apr 21 '15 at 9:54
  • 9
    The example is unnecessarily extreme - as most soccer players would dive on the floor in agony and be unable to continue playing if they were hit with a child's foam toy... ;) – Michael Apr 21 '15 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.