How to do evaluation when the methodology is unique?

Question

When it comes to the evaluation of a methodology (or an algorithm (CS)) we usually see how well the methodology actually is by comparing it with other similar methods that achieve the same thing.

However, in my case, the method we are proposing is unique and novel and, to best of my knowledge, isn't used anywhere else. Is it valid to evaluate it on its own, for example by simply measuring its time/complexity against n datasets? How would we perform evaluation for methods that can't be compared against others?

Answer 1

TL;DR: if you don't know how to evaluate your work, you probably don't know exactly what research question you are working on.

How would we perform evaluation for methods that can't be compared against others?

I work in applied computer science, and this qualifies as the Number One statement my master students usually have when I ask them how they plan to evaluate their work - "I can't compare my tool / approach against anything - no existing tool has exactly the same scope, so the comparison won't be fair!".

This is of course true, but it also shows a misunderstanding about the research question that the work is addressing. Essentially, the "compare against a standard tool" approach is a valuable evaluation method if the research question is "Can we improve the performance/quality/whatever of an approach that does X?". If the research question is rather, as presumably in your case, "Can we find an approach that does X?", the evaluation of course needs to be different.

In that case, you need to find an evaluation that actually shows what you claim your approach does. Some recent examples from my students:

Is your claim that it makes sense to show software developers certain information, which isn't visualized by any other development tool? In that case you need to set up a study where you show that developers that see this piece of information do something better than developers that don't. Is your claim that taking costs of cloud resources into account when scheduling tasks allows for cheaper execution with the same quality? Set up an experiment where you compare costs with and without taking costs into account for a number of representative workloads.

Orthogonally, it often makes sense to do a partial comparison to existing approaches and tools, to also check how your approach fares in comparison to standard tools on standard problems, essentially to make sure that the improvement your approach presumably has in a "new" dimension does not mean that something else does not work anymore. For instance, in the development tool example above, you may also want to check if the new visualization distracts people so much that they now fare less well in usual development tasks than with standard tools.

Answer 2

There are two things that you need to evaluate with your method:

1. Does the method actually do what you want it to / claim it does?
2. What improvement does the method offer over prior work?

No matter what you are doing and how unique it is, you need to be able to validate #1 (preferably experimentally, but simulation and analytic methods can be fine as well). If you can't do that, you aren't doing science.

The improvement, however, may be either qualitative or quantitative. If it is a quantitative improvement, you must be able to validate by direct comparison (e.g., 30% faster, 5 db more gain, quadratic rather than exponential growth). For a qualitative improvement, however, you may not need to compare if you can demonstrate that prior methods simply do not address certain aspects of your problem (e.g., you can predict upcoming frozbozz failures, but nobody else has even tried). In this case, the comparison with other methods is most appropriately done in the literature review.

Answer 3

When proposing a new technique, where there isn't an equivalent to compare performance with, I tend to do as you suggested, and measure the time/complexity, and give some real world data points (it would take X time to operate on realistic situation Y).

Depending on the specific work, you could present an analysis of it based on rationale (for example, evaluating the security of a proposal by explaining the assumptions and demonstrating it is secure against certain threat models).

If there really is nothing which can do the same task, then you can't do a comparison. But perhaps there is another technique which, while technically different, can be used to solve a similar problem to your technique? You could compare performance within the context of that problem in that case.