14

I just thought that I would put this question out here because I am currently working on a meta analysis of a particular research question in my field (HCI/information science). Moreover, this question is new and has not been answered in very detail so far.

I have done most of the usual ways described in other resources as follows:

  1. I have done a comprehensive review of the relevant literature pertaining to this particular research question. This is a new area and the number of directly relevant papers are ~<50. I have also compiled an exhaustive bibliography of the indirectly relevant papers in this area. That number is ~<150.

  2. Out of this set of directly relevant literature, I have identified 23 quantitative empirical studies, 14 qualitative empirical studies and the rest are theoretical pieces/position pieces/framework papers.

The challenge I am currently facing is how to meta-analyze the data. In HCI, it is not common to freely distribute empirical datasets (although signs of change are imminent in the atmosphere. :)) and I have sent out polite emails to the relevant researchers inquiring if I could have some manner of access to these datasets. The rate of response is ~50% so far in responding to my email but only about ~5% want to actively share their datasets.

Therefore, the only other option (that I can see right now) is to compile, summarize and make sense of data and results already reported in the qualitative and quantitative papers.

I was wondering if any of you experienced (and also not-so-experienced :)) academicians/researchers had any insights into how to actually go about doing a meta analysis from the bottom up.

The main online resources which I have been using so far to tackle this problem is given

Please do note that I am not expecting any detailed step-by-step "spoonfeeding" response. Pointing me to some helpful resources is fine. In addition, personal anecdotes or valuable experiences will be really appreciated.

Thank you for taking the time to read this question.

  • 2
    For meta-analysis you don't typically need the original dataset, only summary statistics related to the study and intervention (which hopefully are available in the publications). You might also want to add the Cochrane Collaboration to your list of websites. – Andy W Jul 22 '13 at 12:05
  • Thanks! The Cochrane Collaboration website is really cool and looks very useful. It looks specific to the biological sciences or medicine though but I am sure that many of the main concepts carry over to HCI. – Shion Jul 22 '13 at 18:25
  • 2
    Also note that Cross Validated is always able to lend a helping hand with any statistics questions. – ThomasH Jul 23 '13 at 19:51
  • @ThomasH thanks for the tip! I do visit there occasionally as well. – Shion Jul 23 '13 at 20:19
  • 1
    The Handbook of Research Synthesis is very helpful on these details. – Thomas Jul 26 '13 at 9:05
6
+100

Good luck to you. I'm trying to do something similar and found that few HCI papers publish enough summary statistics to conduct a proper meta-analysis. Indeed, a lot of the time, their stats seem quite sloppy.

I've styled my analysis similar to two review papers I found. One from the HCI area as well (Dehn & Van Mulken, 2000) and one from a bit more outfield (Jones & Gosling, 2005).

Neither is a true meta-analysis, but they get as close to formal as I think it's reasonable to get when an actual meta-analysis is simply not an option.

  • added the papers to my current reading list and upvoted your answer by one. if I don't get anymore in a reasonable time frame, I will accept yours. – Shion Jul 23 '13 at 20:20
  • accepted and also awarded bounty to you. – Shion Aug 2 '13 at 17:54
  • Isn't sloppy stats almost the rule in computer science? Performance measurements often don't even report standard deviations. – Blaisorblade Oct 12 '13 at 0:20
  • @Blaisorblade Unfortunately yes, but HCI straddles more than one field. Most conferences I go to, the split between computer scientists and sociologists/psychologists is nearly 50/50. The latter should know a lot better and the former should stop making excuses or take the science out of their name (NB: I'm the former). – ThomasH Oct 12 '13 at 10:19
  • 1
    @Blaisorblade There are some papers in the HCI community specifically as well, but it's a manifold problem. For the most part, neither authors, nor reviewers nor editors have the necessary statistical background to distinguish good from bad statistical practices. So no one along the chain of publication stops the papers being published, so the practice continues. Reviews are one way of drawing attention to the problem, so I plan on having a section "giving out" about other people's stats. – ThomasH Oct 12 '13 at 14:26
2

One take on this, regardless of field, is to create a framework to place the existing research in. Perhaps you have two dimensions, colour (red, green, yellow) and smell (sweet, sour) - and you review all the prior literatures and place it into your framework 'buckets'.

What this is really doing, and why you want this in your dissertation, is setting up your contribution. By classifying all the prior research, you will have (hopefully!) identified a hole, which your work is going to fill. So... choose your buckets carefully!

ps, I realize this is not a statistical answer - but I hope this is of use, or perhaps sheds some light that will help you see a useful way forward.

2

I really second the use of the Cochrane Collaboration website for meta-analysis and systematic review.

Another good resource is the PRISMA checklist which is often used for journals and reviewers in health fields when evaluating papers. PRISMA also has good guidelines for how to format/present your included papers and create a flow diagram of your review process (again, often required in health-related journals). Good luck!

  • +1; came here to mention PRISMA, which breaks down the whole process into very clear straightforward steps and is a gold standard for systematic reviews and meta-analyses. It sounds like the OP is specifically looking for advice on what to do in a field where researchers don't like sharing data though - you might want to add something addressing that specifically. – arboviral Aug 5 '16 at 10:58
1

One note, besides the very helpful ones you have already gotten. While you have embarked on a "meta-analysis", which often has the very specific goal of producing a single (or small number) of summary estimates for an entire field, you should not view having to fall back on writing "just" a systematic review as a failure.

Instead, "the literature in this field is incapable of being statistically summarized in its present state" should be viewed as a finding in and of itself.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.