I'm comparing my method against existing methods by doing some experiments on one dataset. The existing methods do not provide results on the considered dataset. My question is: Can I use default parameters from the existing methods? The fact is that there are plenty of existing methods and it will take a lot of time to fine-tune for each existing method. Thank you.


Can I use default parameters from the existing methods?

Yes, but should you? Default parameters are unlikely to provide optimal performance, so you may like to try various parameters.

| improve this answer | |
  • Do you mean we should fine-tune for all existing methods that we want to compare? – lenhhoxung Jul 1 '19 at 9:31
  • 8
    @lenhhoxung It depends what you want to achieve. You'll get misleading results by comparing your method with optimised parameters on your dataset to other methods with unoptimised parameters on a dataset they weren't necessarily design for. Perhaps you can find some other basis for comparison. – user2768 Jul 1 '19 at 9:41

In my view this depends on a number of things:

a) Are the default parameters of the competing methods advertised as being widely applicable?

b) Have authors of competing methods provided appropriate guidance for how to tune?

c) Does what you are proposing require a lot of tuning, and did you tune yours in order to beat the others?

Answers a) yes b) no c) no point in the direction that you can use other methods with default tuning. If the resulting score is 3:0 or 0:3, my advice is clear. If it's 1:2 or 2:1, well, one needs to look at details, for example, is provided guidance by others so weak that chances are 95% of the users will use the default tuning anyway etc.?

| improve this answer | |

Assuming that your goal is to publish a paper about your method, a way to answer your question is to ask yourself: given clear indications about how your method is evaluated and how the other methods are evaluated, would a reviewer consider this comparison inadequate, sufficient or convincing?

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.