I am working on a literature review of various machine learning models doing the same general tasks, both to learn the theory and challenges involved and also to find the current state-of-the-art and average expected performance to compare the experimental result to. In my search, I found that some articles measuring the same model, on the same dataset, with very different final results.
Seeing as this is about machine learning, which inherently has some amount of result instability, not to mention that each lab will have its own model build, pipeline, and practice, some amount of it is expected. Furthermore, all the articles are from IEEE, and the authors are from famous universities, so they are about as trustworthy as it is going to get. It still leaves the question of which set of results to cite, however.
The way I see it, there are a few options:
- Cite the model's results from the article it came from: The authors obviously know how to get the best performance out of their own work
- Cite a model's results from the article it did not come from: Prevent any bias, however unlikely
- Cite all results from unaffiliated review articles: Prevent bias, and also give a "neutral playfield" for all competing models. The downside is that such reports are almost always a few years behind current developments
- Try to test the models myself: Unless I decided to write my own review article comparing the models, I highly doubt that will be approved. In addition, the limited resources and time I have for this may not allow for a good or fair comparison, nor is it going to be much similar to any current results available
Edit: To give a simple example of what I mean by "contradicting numerical results" in machine learning (ML), assuming that the current standard performance is against a (sometimes not even an ML model, just a mathematical algorithm) with average error Ep = 0.6 (cited as ). A and B are 2 models from 2 different labs and follow different model families:
Model A paper :
We have found that the newest Model A has an average error Ea = 0.5, ahead of SOTA B  with Eb = 0.8 and production model  Ep = 0.6
Model B paper :
We have found that the newest Model B has Eb = 0.57, above industry standard  Ep = 0.6, and better than Model A  with Ea = 0.75
Assuming a lab C, which is not connected to either of the above, deigns to write a review paper about them, they might write :
The tests conducted have shown that model A  has an average error of Ea = 0.7, similar to model B