I am developing an algorithm for a type of classification problem that is a small-but-growing area of AI research. Until two years ago, there was no standard benchmark for this type of problem. In 2016, a benchmark dataset was made publically available to solve this issue and most subsequent papers in this area have used it to evaluate their algorithms.

However, the dataset is of poor quality. It is very small with <100 examples, many of which are slight variations on the same data. Also, the ground truth is dubious for several examples. This makes it hard to compare the performance of different published algorithms since they are nearly all within the margin of error.

I can think of four options:

1. Publish my algorithm using the flawed benchmark. Its apparent performance is comparable to other methods, and it has enough other advantages (fewer parameters, etc.) to be published this way, but probably not in a high-impact venue.

2. Publish my algorithm using a more robust benchmark that I create on my own. I could add legitimacy by implementing previously-published algorithms to compare on the new benchmark, but I feel like this would still give the impression of moving the goalposts.

3. Publish my algorithm using a more robust benchmark that I create on my own, in addition to the flawed benchmark. Many papers test their algorithms on additional custom datasets (though usually simulations) so this is not unusual. However, this would imply I consider the flawed benchmark valid and that would still be the results most people look at.

4. Separately publish a more robust benchmark dataset and validated criticism of the flawed benchmark. I could then cite this when I later publish my algorithm. However, this will delay publication of my algorithm and still depends on my new benchmark being well-received.

Because I am still a student and this would be my first work in this field, I am wary of bucking current practices too much. What should I do? Are there other factors I should consider?

I am developing an algorithm for a type of classification problem that is a small-but-growing area of AI research. Until two years ago, there was no standard benchmark for this type of problem. In 2016, a benchmark dataset was made publically available to solve this issue and most subsequent papers in this area have used it to evaluate their algorithms.

However, the dataset is of poor quality. It is very small with <100 examples, many of which are slight variations on the same data. Also, the ground truth is dubious for several examples. This makes it hard to compare the performance of different published algorithms since they are nearly all within the margin of error.

I can think of four options:

1. Publish my algorithm using the flawed benchmark. Its apparent performance is comparable to other methods, and it has enough other advantages (fewer parameters, etc.) to be published this way, but probably not in a high-impact venue.

2. Publish my algorithm using a more robust benchmark that I create on my own. I could add legitimacy by implementing previously-published algorithms to compare on the new benchmark, but I feel like this would still give the impression of moving the goalposts.

3. Publish my algorithm using a more robust benchmark that I create on my own, in addition to the flawed benchmark. Many papers test their algorithms on additional custom datasets (though usually simulations) so this is not unusual. However, this would imply I consider the flawed benchmark valid and that would still be the results most people look at.

4. Separately publish a more robust benchmark dataset and validated criticism of the flawed benchmark. I could then cite this when I later publish my algorithm. However, this will delay publication of my algorithm and still depends on my new benchmark being well-received.

Because I am still a Ph.D. student and this would be my first work in this field, I am wary of bucking current practices too much. What should I do? Are there other factors I should consider?