5

I am working on a literature review of various machine learning models doing the same general tasks, both to learn the theory and challenges involved and also to find the current state-of-the-art and average expected performance to compare the experimental result to. In my search, I found that some articles measuring the same model, on the same dataset, with very different final results.

Seeing as this is about machine learning, which inherently has some amount of result instability, not to mention that each lab will have its own model build, pipeline, and practice, some amount of it is expected. Furthermore, all the articles are from IEEE, and the authors are from famous universities, so they are about as trustworthy as it is going to get. It still leaves the question of which set of results to cite, however.

The way I see it, there are a few options:

  1. Cite the model's results from the article it came from: The authors obviously know how to get the best performance out of their own work
  2. Cite a model's results from the article it did not come from: Prevent any bias, however unlikely
  3. Cite all results from unaffiliated review articles: Prevent bias, and also give a "neutral playfield" for all competing models. The downside is that such reports are almost always a few years behind current developments
  4. Try to test the models myself: Unless I decided to write my own review article comparing the models, I highly doubt that will be approved. In addition, the limited resources and time I have for this may not allow for a good or fair comparison, nor is it going to be much similar to any current results available

Edit: To give a simple example of what I mean by "contradicting numerical results" in machine learning (ML), assuming that the current standard performance is against a (sometimes not even an ML model, just a mathematical algorithm) with average error Ep = 0.6 (cited as [3]). A and B are 2 models from 2 different labs and follow different model families:

Model A paper [1]:

We have found that the newest Model A has an average error Ea = 0.5, ahead of SOTA B [2] with Eb = 0.8 and production model [3] Ep = 0.6

Model B paper [2]:

We have found that the newest Model B has Eb = 0.57, above industry standard [3] Ep = 0.6, and better than Model A [1] with Ea = 0.75

Assuming a lab C, which is not connected to either of the above, deigns to write a review paper about them, they might write [4]:

The tests conducted have shown that model A [1] has an average error of Ea = 0.7, similar to model B [2]

8
  • 7
    "To the best of the author's knowledge, the following results were obtained: 0.5 in [1], 0.57 in [2], 0.8 in [3]. While it is, to the best of the author's knowledge, not clear how the differences can be explained, all of them imply.."
    – user111388
    Mar 30, 2023 at 17:27
  • Fair enough, but the problem with the machine learning model is that sometimes the difference in results is only ~1-2 %, and citing everything can lead to ridiculous moments of modern research less accurate than algorithms from 1993, or worse, not even as good as its own predecessor, which means its article should not have been published in the first place (?) Mar 30, 2023 at 18:56
  • If there are really large amounts of different results, I would make a complete table listing the authors, paper, year, result and then, in the normal text, especially emphasize the "non-ridiculous" research and additional important things (eg that while some result doesn't improve the result, it gives some other perspective etc.)
    – user111388
    Mar 31, 2023 at 8:49
  • 1
    Of course, it depends on the result but there are many reasons why a worse result should be published.
    – user111388
    Mar 31, 2023 at 8:50
  • True, true, especially when the new model is the first of a new family. Due to the nature of many of the articles, though, there is rarely a non-Review Paper set of data that does not contain some level of "Huh?" to it. For example: Model A paper [1]: "We have found that the newest Model A has average error Ea = 0.5, ahead of SOTA B [2] with Eb = 0.8 and 90s production model [3] Ep = 0.6" Model B paper [2]: "We have found that the newest Model B has Eb = 0.57, above industry standard [3], and better than Model A [1] with Ea = 0.75" Mar 31, 2023 at 9:43

2 Answers 2

9

If you are reviewing a field, then you need to be complete. Picking and choosing is a form of bias, though you can explore whether newer results are an advance in validity over older.

So, you need to cover it all, or at least all of it post some reasonable date.

If you leave out a study you may be doing a disservice. Some models, approaches, etc, can be valuable in the extension if not done that way originally.

And yes, you can say that there is potential variability in results based on other factors than the models themselves. Those differences may actually be important - too important to leave out of a review.

Your last suggestion seems to take you beyond the realm of review, however.

3
  • 1
    Yes, include all (reputable). Mar 30, 2023 at 17:43
  • 1
    The last suggestion is mostly so that people do not assume infinite time/resources for this problem by prefacing that "Unless it is the point of the project itself, that is NOT going to happen" Mar 30, 2023 at 19:00
  • For covering all sources, feel free to check out my comment on the question, but the short version is: Just because everything is legitimate, does not mean everything makes sense Mar 30, 2023 at 19:09
1

Of the options you suggested, only the first seems reasonable:

Just cite the model's results from the article it came from: The authors obviously know how to get the best performance out of their own work

A review is supposed to report what the literature says, with some intelligent commentary. So, for you, that would mean:

  • Report what the literature says. If there are contradictions in the literature, then report it as such. That is useful information to readers, rather than hiding that fact by selectively picking what you think are the "best" results.
  • Give some intelligent commentary. You should admit that you are not sure why there is so much variation in the literature, but try to intelligently suggest some plausible reaons. If this is quite an important issue, then it would be worthwhile for you to run the model yourself and see what you get. Although this should not be a formal part of your review, your experience could give you some insights as to why some results in the literature vary from each other. This would add value to your readers.

I do not think any of your other suggestions would be very helpful:

Cite a model's results from the article it did not come from: Prevent any bias, however unlikely

Maybe I don't understand what you are saying here, but I do not see any logic (or integrity) to citing results from articles from which they did not come.

Cite all results from neutral review articles: Prevent bias, and also give a "neutral playfield" for all competing models. The downside is that such articles are almost always a few years behind current developments

From the scenario you describe, how would you determine these so-called "neutral" review articles? You would be introducing a value judgment of "neutrality" that might be difficult to justify.

Try to test the models myself: Unless I decided to write my own review article comparing the models, I highly doubt that will be approved. In addition, the limited resources and time I have for this may not allow for a good or fair comparison, nor is it going to be much similar to any current results available

I did suggest that you run a quick model for your own understanding, but not for publication of the results. Any such models would not normally be publishable because they would have to be subject to peer review in order to merit comparison with the literature that you are reviewing.

4
  • By "neutral" review articles I mean Review Papers written by a different lab, without sharing any author with any of the previous research model articles, and not written to prove their own model. This normally means their result only includes a comparison between models, or showcasing their dataset(s) Apr 3, 2023 at 12:06
  • @NamNguyenHoang, OK, I see what you mean. I still don't like the term "neutral", though. "Neutral" sounds to me along the lines of "scholarly objectivity", which I consider to mostly be a convenient fiction. (I think that scholars should strive to be self-critical, without presuming that we can be truly objective.) I would say something like, "research whose authors were not involved in the models that they review". But that's just my opinion.
    – Tripartio
    Apr 3, 2023 at 13:10
  • How about "uninvolved" or "unconnected" then? Would rewording it that way make things more clear for you? Apr 3, 2023 at 13:13
  • 1
    @NamNguyenHoang "Unconnected" sounds good to me.
    – Tripartio
    Apr 3, 2023 at 17:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .