How should one react and respond when reading questionable publications in highly-regarded conference proceedings?

Question

I am writing to ask for the community's advice in both reacting (regulating my own thoughts) and responding (whether worth the time to respond at all) to situations involving questionable publications.

I mentor high school students to work on computer science projects, and one of my students wanted to explore an image classification project. To evaluate their performance, I went to Google Scholar and found the first few results papers for their data benchmark. I was surprised to see the paper reporting 99% accuracy in their conclusions - this is remarkable for the task. Upon further examination, they were reporting only training accuracy -- those near ML know this to be a cardinal sin, as any model can be overfit to perfectly learn data. The publication had further oddities, such as a far-too-simple CNN structure to be published in 2020, given that any commonly-adopted CNN (ResNet, AlexNet, etc.) would readily exceed performance of their simple, tutorial-level architecture.

"Ok", I tell myself, "Someone took their 1-week class project, and turned it into a predatory conference to be 'published'. No big deal, this happens all the time." (This was SAI Intelligent Systems and Applications 2021, just to alert others.)

But here's where I became perplexed; the article was cited by another article published in a highly respected conference proceedings for our field. This conference is on the top-10 list from Google Scholar for its category. I went to the citing paper, and read the line - it was a clearly generic statement to which the cited paper had no relevance or strength.

Clearly, reviewers for the conference made a (reasonable, trusting) oversight (no one has time to check every reference), and perhaps even the cited paper was not refereed.

I find myself feeling frustrated to see how quickly articles like this become part of our academic digital clutter, especially when these artifacts are used as metrics for our "worth" as scholars on the job market, up for tenure, etc.

Is there something productive that can be done with this energy? Is it best to let things like this go? Does this happen often?

Thank you in advance for advice from others in the academic community.

Answer 1

My first reaction would be "typical, not another one". My second reaction might be: This would make a great teaching example for how academic publishing works.

Firstly as a demonstration that peer review doesn't guarantee correctness. Almost all papers contain flaws. This isn't to say that peer review is hopeless or pointless. I'm sure without peer review papers would contain more flaws, and I'm sure peer review at good venues weeds out the worst offences (like, I suspect the paper you discuss only reporting accuracy on the training set wouldn't have gotten accepted at a reputable venue). Even pretty bad mistakes get through sometimes, but it reduces the probability. I might discuss with students the difference between "reputable", as in has a proper and trusted peer review reputation, with "glamorous", as in famous for accepting the most "exciting" or "outlandish" (depending on your preference) results.

Secondly its a good example of how:

1. Something being cited has little to do with its quality
2. Citations shouldn't be blindly trusted.

I bet only 1% of citations are ever checked by anyone. As a reviewer, I certainly don't check citations, unless its something that a) feels wrong to me and b) is an important point for the central conclusions of the paper.

That these things are common is why the most important thing we can teach our students is critical thinking. Can they judge the quality of the claims for themselves, if so, then they should. If they can't (e.g. I wouldn't have been able to judge the structure of a CNN, although even I know that performance on training data is the wrong metric), then what do people can make of it? Are the results broadly accepted in the field? Have similar results from other groups supported the main conclusions? Has the work been built on and expanded? Can they ask an expert for their opinion?

In terms of correcting the record: if this rogue citation really does mean the central conclusions of the main paper don't stack up, there are things you can do. Although I'd make sure that it really is central to the main conclusions of the paper and that the paper itself is important. You could tweet about it. You could leave comments under the paper if the hosting site has a comments section, or on PubPeer.

Finally, it terms of screwing up the metrics, the best thing to do is to join campaigns pushing for people not to be judged on numerical metrics like impact factor, H-index or citation count. You could push DORA at your institution.

Answer 2

There are essentially three approaches to this problem.

The first one is to start a guerilla war and challenge the paper directly. A lot of noise, not likely to be very efficient overall, you will end up fighting windmills.

The second one is to work with your local community: Ian has covered these options nicely. If it is an one-off, this option is preferable - you could show it as an example to your students to teach them about what could go wrong about real research.

Finally, if you think a problem is wide-spread enough, you might try address it via the regular means of scientific communication. Raise awareness by giving talks at conferences and writing op-eds; this is what is traditionally being done when one thinks there is a big problem in their field. One could turn the unsightly trend into a research item of its own by provide insights about what kinds of methodological flaws are the most prevalent and what should the scientific community pay extra attention to do their research and reviewing better. I would argue this is the most impactful (but also demanding) option available in this situation.

Answer 3

In my discipline, engineering, many articles in top journals are poorly written and have numerous technical flaws. There are many reasons that led to this 'pollution'; one of which is that experienced reviewers no longer review papers given the volume of submitted papers. Another reason is that there are many cases whereby reviewers simply accept their mate's papers in return for favorable reviews in the future; yes, this is un-ethical but they are playing the system.

How should one react? pissed. Can one do anything? nothing really. The system is corrupted. At the end of the day, we judge what is good quality ourselves, and not rely on the 'brand' of a journal or conference, or even an author's name.