Which set of results to cite when there are contradicting numerical results between articles

Question

I am working on a literature review of various machine learning models doing the same general tasks, both to learn the theory and challenges involved and also to find the current state-of-the-art and average expected performance to compare the experimental result to. In my search, I found that some articles measuring the same model, on the same dataset, with very different final results.

Seeing as this is about machine learning, which inherently has some amount of result instability, not to mention that each lab will have its own model build, pipeline, and practice, some amount of it is expected. Furthermore, all the articles are from IEEE, and the authors are from famous universities, so they are about as trustworthy as it is going to get. It still leaves the question of which set of results to cite, however.

The way I see it, there are a few options:

1. Cite the model's results from the article it came from: The authors obviously know how to get the best performance out of their own work
2. Cite a model's results from the article it did not come from: Prevent any bias, however unlikely
3. Cite all results from unaffiliated review articles: Prevent bias, and also give a "neutral playfield" for all competing models. The downside is that such reports are almost always a few years behind current developments
4. Try to test the models myself: Unless I decided to write my own review article comparing the models, I highly doubt that will be approved. In addition, the limited resources and time I have for this may not allow for a good or fair comparison, nor is it going to be much similar to any current results available

Edit: To give a simple example of what I mean by "contradicting numerical results" in machine learning (ML), assuming that the current standard performance is against a (sometimes not even an ML model, just a mathematical algorithm) with average error Ep = 0.6 (cited as [3]). A and B are 2 models from 2 different labs and follow different model families:

Model A paper [1]:

We have found that the newest Model A has an average error Ea = 0.5, ahead of SOTA B [2] with Eb = 0.8 and production model [3] Ep = 0.6

Model B paper [2]:

We have found that the newest Model B has Eb = 0.57, above industry standard [3] Ep = 0.6, and better than Model A [1] with Ea = 0.75

Assuming a lab C, which is not connected to either of the above, deigns to write a review paper about them, they might write [4]:

The tests conducted have shown that model A [1] has an average error of Ea = 0.7, similar to model B [2]

Answer 1

If you are reviewing a field, then you need to be complete. Picking and choosing is a form of bias, though you can explore whether newer results are an advance in validity over older.

So, you need to cover it all, or at least all of it post some reasonable date.

If you leave out a study you may be doing a disservice. Some models, approaches, etc, can be valuable in the extension if not done that way originally.

And yes, you can say that there is potential variability in results based on other factors than the models themselves. Those differences may actually be important - too important to leave out of a review.

Your last suggestion seems to take you beyond the realm of review, however.

Answer 2

Of the options you suggested, only the first seems reasonable:

Just cite the model's results from the article it came from: The authors obviously know how to get the best performance out of their own work

A review is supposed to report what the literature says, with some intelligent commentary. So, for you, that would mean:

• Report what the literature says. If there are contradictions in the literature, then report it as such. That is useful information to readers, rather than hiding that fact by selectively picking what you think are the "best" results.
• Give some intelligent commentary. You should admit that you are not sure why there is so much variation in the literature, but try to intelligently suggest some plausible reaons. If this is quite an important issue, then it would be worthwhile for you to run the model yourself and see what you get. Although this should not be a formal part of your review, your experience could give you some insights as to why some results in the literature vary from each other. This would add value to your readers.

I do not think any of your other suggestions would be very helpful:

Cite a model's results from the article it did not come from: Prevent any bias, however unlikely

Maybe I don't understand what you are saying here, but I do not see any logic (or integrity) to citing results from articles from which they did not come.

Cite all results from neutral review articles: Prevent bias, and also give a "neutral playfield" for all competing models. The downside is that such articles are almost always a few years behind current developments

From the scenario you describe, how would you determine these so-called "neutral" review articles? You would be introducing a value judgment of "neutrality" that might be difficult to justify.

Try to test the models myself: Unless I decided to write my own review article comparing the models, I highly doubt that will be approved. In addition, the limited resources and time I have for this may not allow for a good or fair comparison, nor is it going to be much similar to any current results available

I did suggest that you run a quick model for your own understanding, but not for publication of the results. Any such models would not normally be publishable because they would have to be subject to peer review in order to merit comparison with the literature that you are reviewing.