Can we first claim and then find the evidence when we write the literature review (i.e., using confirmatory technique)?

Question

I have a question regarding a researcher’s practice in writing.

Let me give you some background about this issue. Here’s what this researcher does when writing introductions and literature reviews. He makes certain speculations first and then proceeds to making claims based on those initial speculations. It is worth noting that the claims he makes are sometimes too specific. Following that, he starts looking for evidence to support his claims.

I have tried many times to convince him and have told him that this is not the right way to write a research paper. It’s more like writing a composition rather than an academic piece of writing which aims at making contributions to the world of education.

Here’s part of what he recently wrote in an email in response to me:

“I disagree that making claims and looking for evidence supporting them is "not academic". Maybe I am wrong but this approach is less exploratory than the method you have in mind. For example, [Michael] Kane's framework is based on this method: Claim --> evidence. Kane specifically stresses the importance of opinions/judgments etc. I personally do not prefer to tie my mind merely to the literature --I do not find it much academic in the sense it does not engage me in critical thinking. My method is: "The theory I wish to develop does not exist out there; I develop and support it"”

I would like to have your opinion regarding this issue. Is what the researcher claims really right? If not, can you introduce some published sources which have attacked this practice?

Answer 1

Claim -> evidence is a perfectly valid approach. It is just that evidence is not the same as only looking for supporting information, it means critically scrutinizing all aspects that speaks for or against the claim and then drawing an educated conclusion on whether the claim is good or not. In the end the "evidence" is the net outcome of positive and negative support to the claim. So to me the claim that one should come up for an idea an then look for support is either poorly phrased, leading to misunderstanding, or plainly bad.

Answer 2

You'll find some evidence for pretty much anything

It is okay to make assumptions and verify them afterwards. However, there is a problem with "looking for evidence to support his claims" - verifying claims requires looking both for evidence that supports the claim, and also for evidence that would disprove the claim.

Doing only one half of that will be a failure since if the claim is contested, then you'll find only what you're looking for even if the 'other side' has overwhelming support. There are many proven false things that have also some published supporting evidence.

You don't know if the claim is contradictory

There are many cases when seemingly obvious things aren't actually true. If a seemingly obvious issue is a core of your argument, you should make sure that it's true - and that includes looking also for contradictory evidence; and you know that it's not contradictory only after checking it. If it turns out false, you'd want to say "There wasn't any published data about the falseness", not "Well, it was on the first page of google but I didn't bother to look for it".

Answer 3

Another way to frame this matter is in the context of the "confirmation bias." This is the tendency to favor evidence to support one's stance, while ignoring contrary evidence. The social psychology literature has produced an abundance of research to show this is a very common (and unintentional) bias. I think it is perfectly acceptable to make a claim (or hypothesis) that is informed by good theory, and then seek out the relevant evidence that can help refine the claim (or hypothesis).

The problem is not in the process of making the claim and then reviewing evidence -- rather, the problem occurs during the actual review of the evidence. If your colleague isn't aware of the possibility of a confirmation bias, it is likely that she or he will look only at the evidence that aligns with the claim. One way of minimizing the confirmation bias is to begin the search process with more of an open question that allows one to capture the full range of evidence, as opposed to just one of many possible sides.

I highly recommend the book "Thinking, Fast and Slow" by Daniel Kahneman, which does an excellent job of describing many decision making processes and where people can go wrong.

Answer 4

Honestly, I suspect that a lot of research is done this way. Read a couple of papers, get an idea, collect data and then write the paper. During the paper writing one has to do a reasonable literature review (at least in my field) to give context to your work and why it is better. This isn't the best way of doing it.

I believe this really separates researchers from great researchers. The great researchers have a strong knowledge of all the background papers and understand how they relate to their topic.

As someone who has done research for 20+ years, I would really encourage any newer researcher to get a good grasp of the background papers and create an understanding of how they fit together and how your work fits into theirs.

We all build on the shoulders of the past, it is rare to create a new research topic that is not based on past research.

The person you mention, I think, is not creating objective work, it seems a little more subjective (though it is hard to distance yourself from your own ideas :-) ). My feeling is that you need to do the right thing and realize that what they are doing is not a great model (which I think you have realized already).

I think there is a difference between assumptions and hypothesis. Assumptions, to me, seem to be unsubstantiated ideas. Hypotheses, to me, are substantiated ideas. A hypothesis typically comes from a strong knowledge of the background literature and a few good ideas you have come up with.

Just my 2 cents.