In my area (computer science), many researchers have employed some algorithms to solve the problem where most of them used their own private datasets.

I collected several public datasets and have benchmarked the performance (the accuracy) of the state-of-the-art algorithms on several public datasets. I have not seen anyone have done this so far.

However, 1. The benchmarked algorithms are existing algorithms. None of them is my own proposed algorithm. Most algorithms have been used for the researchers' private dataset. 2. The datasets I used are public dataset which has also been used in other studies but there are some algorithms that have not been employed for these datasets. Those algorithms have not been investigated on these public datasets.

The purpose of I'm doing so is: 1. To investigate the accuracy of the existing algorithms employed in my area across various datasets. 2. To show that the accuracy of a particular algorithm is not consistent across various datasets and there is a need for a combination of algorithms.

Is what I have done can be considered as one of the contributions of my thesis before I proceed to the next contribution which is to propose a combination of algorithms?

  • 1
    It's opinion. Ask the people who evaluate your work. Their opinions are the only opinions that determine if you get your PhD. – Anonymous Physicist May 17 '20 at 4:36
  • I have done something similar, and it has been well received. My contribution was to show how existing algorithms perform on a common platform because everyone does their own thing. While doing the research, I also have new findings or insights. – Prof. Santa Claus May 17 '20 at 9:36
  • Are you confusing "algorithm" here with "program"? Algorithmic analysis is abstract and doesn't require running to reach conclusion. Programs on the other hand are messy and their concrete performance can be affected by many things, including the specific architecture on which they are run and the quality of the compilers used to translate them. – Buffy May 17 '20 at 9:48
  • @Buffy I believe I can say "algorithm" here with "program" – Aqee May 18 '20 at 0:54
  • @Aqee, you can optimise an algorithm on paper by changing the way the calculations are done. A "program" is one implementation of an algorithm (e.g. a Python implementation of, say, grid search vs a C++ implementation of grid search). Optimising programs might involve a profiler and tweaking lines of code to make it run faster. Optimising an algorithm might involve tweaking variables to make it more accurate. I think you mean "performance" in terms of how accurate something is on a given dataset, though (not how fast it runs on your fancy GPU). – Pam May 18 '20 at 9:23

What you propose sounds like an in depth experimental review. You would certainly be able to get a paper out of it and, if you made the datasets easy to access and easy to use, you could gather a respectable number of citations. There’s a good example (and another) of this in the Visual Object Tracking community. Both of these are novel because of their objective evaluation or novel dataset. Done well, there is no reason why this can’t be considered a novel contribution to research. By that argument, they would count as a PhD contribution. But you should ask your PhD supervisor to confirm. They will know your field and your school and the normal rules that apply.

On the other hand, if you plan to create and evaluate your own algorithm, then putting together an evaluation framework is necessary anyway. Having comparisons with existing methods will only strengthen the contribution of the new algorithm, so it is simply a matter of whether you write your work up before or after you have your own algorithm.

  • Yes, the papers you shared are similar to what I meant. BTW, what is "evaluation framework" that you are trying to say? – Aqee May 18 '20 at 0:51
  • @Aqee by "evaluation framework" I mean all the code you'll need to write to loop through all your gathered datasets and all your gathered (or implemented) algorithms. When you write your algorithm, you'll just add it in with all the others, that way they'll all be evaluated in the same way. – Pam May 18 '20 at 9:17

Disclaimer: I do not have a CS or related professional or academic background. I decided to write an answer based on my more general point of view and my experience on algorithm/ programming practices in areas they are applied. Please consider carefully.

As already said, you should discuss this with your supervisor in very certain terms.

What I am writing must be taken with care, as I come from a different area with different conventions A replication of existing algorithms on existing datasets (something mechanical, a repetition of prior knowledge) can be in a PhD but is not considered original work. Applying an algorithm is a new context ("method A in context B" type of research) can be considered novel but under certain circumstances, e.g. if there is no prior application in that type of data, if the algorithms are not widely used in the field, if the earlier applications are old/ the dataset has changed (there is a factual reason to rerun the algoritm) or if there is an alteration with something you defined or borrowed from elsewhere. The most well-founded cases are the first and the last - the others can be debated - and even then it depends upon the difficulty and how new and widespread the algorithm is. In terms of a technical contribution, you definitely need something new to be introduced, most preferably something you came up with.

It think that the exercise is very useful in a PhD, for benchmarking or evaluation framework, but would hesitate to consider it a contribution, especially technical.

  • What is your area? – Aqee May 19 '20 at 1:40

I would urge just one (more) point of caution. When you are benchmarking other people's work "objectively" be careful to be sensitive. There's a danger that you can offend someone in your field by clumsily suggesting that their claims of performance are not all that great. It may be that you could work with that person in the future or that as a PhD student your study may not be as objective as you might think.

All I'm saying is that it is important to discuss this approach with your supervisor or someone experienced whom you trust.

I'm not suggesting that you shouldn't challenge results or be scared to say something that might take the shine off someone else's contribution, just that you should tread carefully.

I myself am hoping to do something similar, i.e. pick up algorithms that people have "benchmarked" and run them against more benchmarks on the way to hopefully combining them in novel ways. Good luck.

  • 1
    Well, it is very normal I think if we can challenge someone else's approach in an academic way. In the case, we may work with that person, as as a researcher, that person should be excited how the problem can be solved better – Aqee May 19 '20 at 1:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.