4

Unfortunately I do not have any prior research experience, therefore I had a few questions while I was studying papers on cluster analysis.

Can a researcher legally and with permission use the benchmark values of the pre-existing algorithm to compare with theirs? This question came to me after I noticed identical results reported in multiple papers for some population based cluster analysis algorithms. The different papers propose a new algorithm and then compare the performance of it with some other algorithms over some common dataset. I have noticed that, for some algorithms the values reported are an exact match. The nature of these algorithm is stochastic, and also the different authors claim to have run the algorithms different number of iterations and then use the mean. Therefore no two sets of run will result with the same mean, min, max and standard deviation.

My suspicion is either it is permissible to reuse the benchmark results with permission. Or a common benchmarking framework exists (still, how can two papers reporting using different number of iterations land into the same identical result?)

I have reproduced the work of the papers using Octave and benchmark is similar to what is reported in the papers, but is definitely not identical.

Also, how generally these are done? If a researcher has to implement 5 algorithms to compare, then do they implement their own and then verify and compare or request the version the author has used and then benchmark with respect to that specific piece of code to maintain unbiased experimentation?

[EDIT]

Quoting the results from another paper (with citation) is all right, but using it in the benchmark comparison table is something which is practiced in research?

  • It depends on the context. Maybe you could point to an article that does this and the table that is bugging you so that we could help you out? – Bill Barth May 15 '14 at 11:49
4

Quoting the results from another paper (with citation) is all right, but using it in the benchmark comparison table is something which is practiced in research?

I would wager it is not done enough. It is certainly ok, even required, to compare your results to earlier work, and the more fair and unbiased you can do this, the better. It is perfectly acceptable to say "on the right-hand side of Table XY, we report the results for this problem previously achieved in [1]" (and then list them exactly as reported).

The one thing to make sure, though, is that when you want to say that you are better than a previous work, the comparison is actually fair. For instance, one sometimes sees papers where the authors use a long-running deterministic algorithm (which you need a map/reduce cluster to execute on) and claim that their work is better than an earlier heuristic approach, which was built for speed and on-line execution. Don't do that.

|improve this answer|||||
1

You are definitely allowed to quote the results from another paper, with citation. You can also email the author and ask for the code (or grab it from their webpage). In this case, citation is also the right thing.

Depending on the benchmarks in question, the results may yield the same results to the accuracy presented on the paper (if I get a value of 0.98547856842, I will probably round it to 0.985). If you repeat the same benchmark on the same dataset enough times, you can converge to the same numbers to the accuracy printed. It is even easier if the dataset is small.

Another way to enforce replicability is to use the same random seed.

This said, I would be suspicious if they both report the exact same results. If you have the mean, standard deviation, and population size, you can do a statistical test to check the probability of them being reported equal.

If they just copied the results from the other paper without citing it, it would be a bad practice, but it does not invalidate the research.

|improve this answer|||||
  • If the algorithm starts from the same place it may converge into the same,or if the dataset is small,it will converge into a possibly the global optima result in identical values. But in my implementation (random seed) different sets of runs with,say,20 iterations,get similar results,but never identical. Not only two, but I see such similarity in atleast 4 papers, published in prestigious journals. Quoting the results from another paper (with citation) is all right, but using it in the benchmark comparison table is something which is practiced in research? – phoxis May 15 '14 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.