How to think while comparing approaches

Question

I'm a CS master student. Now I'm doing my masters thesis and the contribution of my thesis is to compare different approaches/methods of one topic (e.g. clustering of text documents). What I have done so far is look at the state of the art on the topic and read the papers. However, now I need to think of what to compare. Of course there is the obvious comparison question: which method gives the best results? But that is so obvious. My supervisor once suggested to see how those methods compare their results and see if maybe there is a better way to compare the results of the methods. It was helpful to me to think about questions like this. But still I'm so stuck and I can only think of obvious stuff. This is the first time I am doing something like that.

I'm sure some of you went through something like this, so I was wondering if you can even tell me some /basic/ stuff and questions that people address when they compare methods in computer science. My main questions to you are:

1- When I read the papers of the methods to compare, in which way I should read it? Critically? Questionable? What exactly to look for in them?

2- Is there a general scheme for comparing methods in academia?

3- As for a masters thesis, what stuff is a must-do for comparisons?

4- Any great references/papers related to comparisons that could help me?

Answer 1

Not knowing the details of your project, I'll offer some general suggestions. In a sense, it seems to me you're asking two problems:one question is how to read papers, the other question is how to compare methodologies.

As far as comparing methodologies, the key concept in my mind is that of a "metric" or some means of judging/ranking methods. In CS, for example there are many such metrics: computational complexity, run time, stability, theoretical results regarding convergence, etc. Metrics often are often context dependent, meaning that different classes of users may have different ways to rate/rank a method. An algorithm may give an exact answer, which to a theoretician may be great, but if the algorithm can't function in real-time, another use may prefer another method that gives an approximate answer but quickly. So know who is going to use the methodology often helps you understand what metrics are really significant. So two thoughts to keep in mind are the methods you learned in class for judging how good an algorithm was and also who is going to use the algorithm (what are their concerns/needs and which methods best addresses them).
A suggestion in this regard that will also help you with your paper reading is to look at what the authors of each paper say are the advantages of their method. In the introduction every author will give a reason why their method is "better" than the other (existing) methods.

Answer 2

One way to approach the problem is to ask: what can method A do better than method B? What can B do better than A? Can we devise a test to measure it quantitatively?

Then you may find that A beats B at everything all the time, or that A only beats B in cases that seem relatively unimportant. In any case, you'll then have something interesting to say regarding A vs. B.

Now consider C. Does it do anything better than A and B? Maybe you can use an existing test, or maybe you need a new one. And so on for C and D.

Now, that said, very often you have standard metrics that you should use instead (e.g. look at a ROC curve or % errors or execution time or whatever). But when you're unsure how to form a comparison, inventing one that demonstrates a difference in some aspect of behavior (correctness, speed, memory usage, etc.) is often a good place to start.