How to identify studies that should be replicated?

Question

In psychology voting on which studies should be replicated is established on a website. The ReplicationWiki (that I founded) offers a voting option for studies in economics and related fields, but it is not yet used much. I already saw a couple of questions on specific papers here in the academia stack, in the cross validated stack (c1, c2, c3), and in the quantitative finance stack, and added the studies to the wiki and a link to questions here (1, 2, 3, c1, c2, c3, quant).

There are many other places where such questions on replication of specific studies can be found, e.g., in the forums for specialized software (d). To get further suggestions I would like to know if you have already replicated studies yourself or with your students. What would you think would be a mechanism that motivates to share such experience and help to identify the studies that should be replicated? And the replications that are already available somewhere but not yet listed in the wiki? Where would be the right place to ask?

Answer 1

If I was going to look into replicating a study, I would look for one that satisfies some mixture of these two ideas:

• Is there something to gain from the replication? Do I find the original study/results important? Would a replication add to the field? Have I ever wished someone would replicate this?
• Is there reason to believe something would be different? Do I think their conclusions are wrong? Or could I generalize the original result by changing something?

For example, there is a paper that I heavily cite in my work but it is getting relatively old (software engineering does change with time, I think!) so I have wished someone would replicate it. It isn't possible for me to replicate it completely but regardless of the result, I think it would help the research community.

Note: This is coming my from perspective in a field that doesn't particularly value replications (unlike medicine and psychology).