I'm doing research on object detection for aerial images (drone capture from above).

And many datasets I think their ground truth is not perfect (but also buggy), some object obviously but be ignored by annotator while some objects took long time to recognized with guesses is labelled ? But for some reasons (they love publishing), the datasets got a lot of citations and results on boarding blah blah from variety of methods.

In that case, should I do something ? (I'm newbie to academia world and I know nothing much)

Get higher metrics and claim SOTA on something not good is quite confusing in my opinion. And from that, lead to a question that those question is really workable or just a trick only work for those bend ruler ?

  • 3
    I suppose it depends on what degree of imperfection the datasets have. Is one in ten images mis-annotated? One in one-hundred? Half of them? Is the analysis done on these datasets likely to be improved if the datasets are cleaned up? I imagine that approaching this from the angle of "I improved the annotations in this widely used dataset, lets see how that affects these results" would be recieved differently than "people are intentionally using bad data to get results they want." Sep 4 '20 at 14:19
  • What resolution is expected? Sime cameras can pick up the detail of a black cigarette box on a white card from 100 miles or more? Or isn’t the resolution that good?
    – Solar Mike
    Sep 4 '20 at 18:11
  • The benchmark to which we compare the quality of datasets is not perfection, but rather other alternative datasets. Are there better datasets available? How large is the impact of the flaws in this dataset, do they justify throwing it out and making a different dataset? Can you quantify the accuracy of the dataset? E.g. in natural language processing it's common to evaluate inter-annotator agreement on a subset of the data and use it as a metric of both the rate of mistakes (it's expected that it won't be 0, even in double-reviewed data) and the inherent subjectivity of the annotation schema.
    – Peteris
    Sep 4 '20 at 20:41