Timeline for How to handle reviewing a paper which I suspect has been generated by AI?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 24, 2023 at 21:59 | vote | accept | m123 | ||
| Jul 24, 2023 at 8:57 | comment | added | Thomas Padron-McCarthy | I read some of the comments as the OP shouldn't tell the editor anything at all about the paper being possibly AI-generated? My advice would instead be to let the editor know all of what you have found, and strongly suspect. This paper could be all sorts of things (an experiment, some sort of joke, or even something malicious by someone else than the stated authors), and I don't think one should hide this possibly very important information from the editor. | |
| Jul 22, 2023 at 18:32 | comment | added | Nordine Lotfi | I know this might be obvious, but you mentioned "LLM can indeed scrape", and I believe that it would be more correct to say, even if it's less short: "people that train or create dataset for training LLM can indeed scrape your post". A slightly related topic to that would be "automated agent", but most of the existing ones that use LLM are just using recursive prompting to create the illusion of "autonomous agent", but it's not gonna work as expected for a lot of reasons. Just Fyi :) | |
| Jul 22, 2023 at 4:09 | comment | added | justhalf | @Kimball we already know about poor paper writing (which may or may not be generated by bot) for cheap publications for quite some time (even before chatGPT) :) | |
| Jul 22, 2023 at 3:09 | comment | added | rydwolf | "Whether they could use that information to learn how to do better is unclear." The answer to this is that they'll improve only by learning grammar and common word combinations from SE, same as any other source of text. The LLM won't actually comprehend any of what the answer is saying; if I write "GPT 4.0 really sucks at citation accuracy" it won't take away from that that it should improve on citations, it'll take away from that sentences with that structure sound natural. Training an LLM is just teaching it patterns, it doesn't do any "thinking" on what it's read (nor is it self-aware). | |
| Jul 21, 2023 at 19:54 | comment | added | eps | @m123 Maybe in this case the whole paper was just ai. But you shouldn't assume this in general because it's not a binary - the best stuff coming out of AI right now is a combination of human/ai intelligences. Even if all the words are ai written, there could be a large amount of behind the scenes prompt engineering. I would be shocked if in 10 years using ai to summarize human work into academic papers isn't commonplace, particularly because a lot of academic papers are poorly written. All tools can be used poorly or improperly, what matters is the results | |
| Jul 21, 2023 at 16:24 | comment | added | Nate Eldredge | I would leave this up to the editor. In this day and age, when they get a report that a paper is nonsense, they should always consider the possibility of it being machine generated. I don't think they need you to point it out. In any case, the editor would have to judge it for themselves before taking any action. | |
| Jul 21, 2023 at 14:14 | comment | added | Kimball | @NateEldredge On one hand I agree that you shouldn't spend much time on it. On the other hand, if somebody in academia is just generating papers to get cheap publications, I think the community should know about it. | |
| Jul 21, 2023 at 9:29 | comment | added | m123 | @Nate Eldredge: Or you advise me to reject the paper because of its poor content only? Do you advise me to mention my guess confidentially to the editor with some reasons? | |
| Jul 21, 2023 at 9:29 | comment | added | m123 | @Nate Eldredge: Yes, I am overthinking this. You know, I am thinking to the unethical part of the action of such authors. This time we can reject it because of its poor context. But what if it wasn't like that? should we ignore the immoral part and just judge the correctness of the results? without considering its authenticity? It seems that we can even publish the correct outputs of AI as long as useful to read, but even in this case I think it must contain the name of the used "AI" as the author, not others. If I share my suspicion, can it be useful to prevent such a thing in the future? | |
| Jul 21, 2023 at 2:36 | comment | added | Nate Eldredge | @m123: I feel like you are way overthinking this. Obviously it's a bad paper. Briefly explain why it's a bad paper, click the "recommend reject" button, and move on with your life. If you're going to devote this much time and attention to reviewing a paper, spend it on one that actually has a path to being published and benefiting the community in some way. This one does not. | |
| Jul 21, 2023 at 0:18 | history | edited | Ethan Bolker | CC BY-SA 4.0 |
added 278 characters in body
|
| Jul 21, 2023 at 0:14 | comment | added | m123 | Can I tell my guess to editor based on my written evidence here? Does the format of citations support my point of view? If the editor disagrees with my guess, is it considered a negative point for me? (For example, I get considered to be unable to precisely decide about these plagiarisms) | |
| Jul 21, 2023 at 0:10 | comment | added | m123 | Thanks for your guide. Since the citations are dense at one section, they can be added manually. Although, some of them that I checked does not contain the cited quotes, their titles are apparently related. Additionally, some of the cited papers does not contain the title of the paper. They only contain the name of the author, the journal name, pp, and year. But, there are also some cited papers that contain the cited paragraph. (The passage is not coherent at all and the paragraphs do not follow any specific order, goal, or coherency. They are some independent paragraphs). | |
| Jul 20, 2023 at 23:17 | history | answered | Ethan Bolker | CC BY-SA 4.0 |