73

Large language models (LLMs) are complex neural networks that can capture and represent world knowledge present in the human-generated text as their weights. LLMs are trained on a large collection of textual documents; for example, billions of webpages that are crawled from the web by the Common Crawl project's web crawler. Moreover, an application of LLMs is to generate text in response to a query which can be a set of keywords, a question, or even a simple sentence.

For example, the conversation in the image below is generated by an LLM called ChatGPT, created by OpenAI (a startup company working on LLMs).

ChatGPT example

The text in the box with a slightly darker background is generated by ChatGPT in response to the question on the first line.

As you can see, the generated response is similar (or even indistinguishable) to a reply that a human might have given to the question on the first line.

Now, my question is: is it OK to generate parts of a research paper, for instance, background or parts of the introduction section, using an LLM such as ChatGPT? Naturally, I would give appropriate credit to the creators of the model.

3
  • 16
    What is your goal that you hope to accomplish by using AI output in a paper? Are you hoping to write a paper in less time without loss of quality? Are there some things you're not sure how to write about? I personally can't figure out what you would gain from this. Dec 3, 2022 at 6:55
  • Comments are not for extended discussion; this conversation has been moved to chat. Please see this FAQ before posting a comment below this one; we can only move comments to chat once.
    – cag51
    Dec 6, 2022 at 0:51
  • OpenAi is a startup? Jun 9, 2023 at 7:53

12 Answers 12

96

There are some aspects of academic writing that you need to consider if you want to use AI-generated text passages. I'll address four of them in my answer: Licensing, Plagiarism, Argumentation, Referencing.

Licensing issues

Language models are trained on source texts, and these source texts are typically harvested in great volumes from publications (online or print) without much concern for the license under which the text was published. Thus, the input for the language model may containmaterial that is not licensed for this type of processing. Consequently, the status of the model and of the work generated by the model is a legal gray area. Here's an article that discusses this "legal minefield" for AI-generated images, some of which are based on the OpenAI software. Much of the discussion applies directly to AI-generated texts.

Accusations of plagiarism

To my understanding, AI-generated content depends on associations between training data and classification data. Input data might be for instance a collection of photographs together with a classification matrix. This matrix expresses whether each image contains e.g. a car, a bicycle, a staircase, a zebra crossing, a traffic sign etc. Training an AI basically means to develop a mathematical model that expresses the learned associations between the images and the classifications. The resulting model can make a prediction whether an image that was not part of the training data also shows a bicycle. In this way, the model functions as a classifier. But you could also reverse the model logic by asking the model to generate an image that contains a bicycle. The AI model will use the learned associations between image information and classification to produce an AI-generated image of a bicycle.

A potential problem in this process may arise from data scarcity. If the training data contains only very few images of a bicycle, the generated image will be very similar to the input. This means that if e.g. the prompt for your text generator is highly specific, the output text may be very similar to the few input texts that the AI can draw from to generate the text. This is called AI plagiarism.

So, the AI-generated text may be a direct plagiarism of another published text. But to you as the end user, there is no way of knowing how similar it is, and there's no way of checking the actual texts that the model used to generate the text based on your prompt. This means that if you publish an article that contains AI plagiarism, and if somebody realizes the similarity between your article and other published material, they can rightfully charge you with plagiarism. Your only excuse would be that you didn't write that section, the AI did. But in an academic context, this will still ruin your academic reputation.

Argumentation

Academic writing depends on coherent and cohesive argumentative elaboration. While AIs have made huge progress in that regard, this is still something that needs to be checked very carefully. AI models don't understand the argumentation of an input text in any sense that relates to human understanding. At best, an AI model generalizes over argumentative patterns recurring in several input texts, and it uses these generalizations to create a text that resembles these argumentative patterns. It is your task as an author to ensure that the generated argument is indeed valid. You can only do this if you know the subject matter well enough – but you can ask an AI model to generate a text for a topic that you're completely ignorant of. Much responsibility is required at this point.

Your example text illustrates this. To me as someone who isn't an expert in reinforcement learning, the description sounds reasonable. But I can't judge whether it's actually correct. I would never dare to use this passage without reading up on the difference in detail so that I can verify the passage. But then, I could just as well write the section myself.

Referencing

One rule that every first-year student should learn is that academic writing needs proper referencing. If you make claims, you should back up the claims by references. If you report previous work, you need to provide sources.

However, an AI model doesn't know about referencing. It may learn that after particular text patterns other typical text patters occur. So an AI may learn that sentences from the academic genre very often end in sequences like (ABCDE 1234). because very often, the input texts will provide references for a statement in a sentence at the end of the sentence, and the format of this reference is often (Author Year). However, the AI doesn't know what this pattern means. It doesn't know that you can't use the most probable letter-number sequence here, but that it has to refer to an existing source. Obviously, this is less than useless – it's actively invalidating the academic quality of your text.

Alternatively, the AI may not produce any reference, which is also unacceptable and requires your intervention as the author. The generated text in your example doesn't contain references – but it contains many claims for which I as a reviewer would ask for sources to back up the claims. Again, it would be your task as the author to find sources for each of these claims. And since the AI didn't use sources in any sense of term "source" as used in academic contexts, it may even be impossible to provide appropriate references for an AI-generated text that satisfies the very strict standards for referencing.

Summary

AI-generated texts are an interesting development, but it's a topic that is so novel that there are many unresolved issues (both legal and ethical), and also many aspects that ask for a high degree of responsibility and knowledge from the author.

My assessment may change in the future, but as of today I'd advise against using AI-generated texts in academic writing.

12
  • 7
    An anecdote about the plagiarism aspect: I asked ChatGPT a fairly specific question copied directly from stack overflow. The question was originally posted a while ago, so I assume it is likely in the training set for ChatGPT. The response I got was a correct answer that was the same conceptual solution as the human answer posted on stack overflow, but was clearly "written" from scratch; the structure and wording of the response was not close to the human answer. Dec 3, 2022 at 0:51
  • 32
    An alternate potentially relevant example of the plagiarism aspect would be the currently ongoing class action lawsuit about GitHub Copilot. The exact context is a bit different (programming as opposed to scientific writing), but the core premise of the suit is arguably broadly applicable to most types of AI trained on data harvested without being licensed for that purpose. Dec 3, 2022 at 3:03
  • 7
    "However, an AI model doesn't know about referencing" On last night's The WAN Show (Youtube), one of the hosts actually was able to get the ChatGPT to generate documents with extremely convincing MLA style references to websites. It linked to real and relevant websites, but not necessarily a real page; someone grading/reviewing would have to manually check each reference to know it wasn't real. And that's with OpenAI currently having a bunch of restrictions in place that might be hamstringing its ability to generate more authentic references.
    – anjama
    Dec 3, 2022 at 14:41
  • 11
    @anjama: I'm pretty sure the answer's point is that things like chatGPT aren't actually understanding that referencing the real sources of the information used in the text they generate. They're just learning that sentences/claims in such texts are often in the style of This is a claim (Source 1996). So it can certainly generate "convincing" references, but that doesn't make them accurate ones – even if the linked page actually exists, that doesn't mean the source supports the claim it's being cited for.
    – V2Blast
    Dec 4, 2022 at 20:12
  • 7
    I tried a version of this, by feeding part of an academic chapter I'm writing in the humanities to GPT-3 to complete. It invented an extremely plausible academic at a real university to cite. It took me a comparatively long time to convince myself that the person didn't exist, and neither did their book. Dec 5, 2022 at 3:13
75

No, it is not acceptable to use a language model such as ChatGPT to generate parts of a research paper. Research papers are expected to be the original work of the author(s), and using a language model to generate part of the paper would be considered a form of plagiarism. Additionally, research papers require a high level of accuracy and attention to detail, and a language model may not be able to provide the level of accuracy and precision that is needed in a research paper. It is always best to do your own research and write your own paper.

This is the answer given by ChatGPT.

It doesn't recommend using itself for generating a research paper.

7
  • 25
    This answer made me laugh
    – Neuchâtel
    Dec 4, 2022 at 4:16
  • 2
    @edelweiss: I'm reminded of the tape recorder classroom scene in the 1985 movie "Real Genius". Dec 4, 2022 at 19:42
  • 9
    This answer is also useful in that it will enter the data future text generators will compose their answers of :-). As an aside, this shows that there can be self-reinforcing drift in generated content. It's easy to imagine a dystopia where most people just listen to, reinforce and replicate machine generated prejudice and nonsense. Dec 5, 2022 at 8:04
  • 1
    +1 for ... everything. Thanks. I personally bow to our new overlords.
    – AnoE
    Dec 5, 2022 at 9:00
  • 4
    +1 I know this answer goes against so many rules of this site but whatever.
    – JiK
    Dec 5, 2022 at 11:04
39

I think, right now, it remains fairly dangerous to do so, at least without extensive oversight. I'd apply similar logic to other software tools that have been around much longer, like grammar or spellchecking software - it's okay to use the software to suggest or assist the author writing, but at the end of the day the author needs to be able to take responsibility for what is written and make the decisions.

LLM AI-generated text has gotten quite good at mimicking real human writing, but it isn't capable of doing the sorts of logical checks that humans need to do when they write. I'm especially worried about writers who use these tools because they do not feel confident in their own writing ability, particularly in a language they're less comfortable in. They may know their topic, but not understand the generated writing well enough to verify it.

Remember that research is meant to be somewhat novel; an AI can only be trained on what already exists. It might generate good summaries of topics that have been summarized many times before, but that doesn't mean it's equipped to actually synthesize knowledge in a forward-looking way. In my experience reading examples so far, I've certainly seen some impressive language constructions, but they feel to me a lot more like a novice university student trying to smash together an essay at the last minute, especially one trying to inflate their word count to get as close as possible to some instructor-mandated length. They have the essay feel down, but not the content. The example you posted seems representative of that - the first sentence, even:

The main difference between (adjective) (noun) and (noun) is (adjective)

Whoa, that's like, profound. Note that none of what is written elaborates on what it means to be "deep" in the context of reinforcement learning. It would be absolutely worthless to have any of this in a paper introduction. Even if the text were better, without citations to where the work came from, you have to admit that you're just trusting an AI's compilation of some library - that's not the same as you personally reading and trusting and vouching for the information. It's not merely the creators of the tool that need to be referenced, but the creators of the knowledge.

As others have mentioned, I do think disclosure and credit is important, but right now I think there are more serious issues to be concerned with.

8
  • 2
    A potentially useful answer (+1) but I'm not sure it's possible to engage in "extensive oversight" as a user of these AI models. The main issue is that, as a mere user, you don't have access to the training data to "oversee" what was done in the algorithm. Consequently, I would think that any oversight that is attempted would still be unable to solve the problems raised in the other answers.
    – Ben
    Dec 2, 2022 at 20:23
  • 2
    @Ben I agree with you that I'm not sure it's possible; I feel like most people who are interested in using this technology aren't really imagining any oversight at all, so I figured that would be enough to warn them off. An example of "extensive oversight" that could actually work might be using an AI-generated paragraph to remind yourself "oh yeah, I should talk about (this thing that I normally take for granted but my audience might not)" - I don't think that's what most people think of when thinking of this technology, but it might be a more valid use.
    – Bryan Krause
    Dec 2, 2022 at 20:52
  • 6
    To clarify, I don't mean to refer to "oversight" as in the generation process, but in the product, the same way one would oversee spelling or grammar check recommendations from software.
    – Bryan Krause
    Dec 2, 2022 at 20:53
  • 4
    Fair enough. I think the danger pointed out in other answers is that you could still end up essentially plagiarising other material without realising it (if that material went into the AI training data) and your proposed oversight of the product would not alert you to that.
    – Ben
    Dec 3, 2022 at 0:41
  • 1
    @leftaroundabout It fails to describe any of that, though. The text is only useful if you already know what deep learning is, in which case you likely don't need to read any of this background.
    – Bryan Krause
    Dec 5, 2022 at 13:26
22

#GivingCreditstoChatGPT is an act of supporting plagiarism!

I am not familiar with legal/copyright issues, so I will not address them here.

Inspite of its capability to generate well-structured "informative" text, ChatGPT is far from being intelligent (and consistent). It is very dangerous to blindly rely on this tool.

For example:

What is p-value?

A p-value is a statistical measure that is used to determine the likelihood that a given result occurred by chance if the null hypothesis were true. If the p-value is low, it suggests that the null hypothesis is unlikely to be true. On the other hand, if the p-value is high, it suggests that the null hypothesis is likely to be true.

Another answer for "What is p-value?"

‪A p-value is a statistical measure which is used to indicate the probability of a given hypothesis being true. It is calulated by dividing the observed results of a test by the expected value of the test‬. The lower the p-value, the more likely the hypothesis to be true.

The best answer (the 4-th try)

A p-value is a numerical value used in statistical hypothesis testing that represents the probability of obtaining a test statistic at least as extreme as the one observed, assuming that the null hypothesis is true.

If I have a student, and he/she keeps giving me a different answer (ranging from far from being correct to very close) everytime ask him/her the question "What is p-value?", I would say that this student just scrapes the entire internet, and gives me whatever the first result that he/she finds "useful" without knowing what it is. That is the problem of ChatGPT. It does not know. It does not understand. AI is not intelligent.

Also, that also implies that it is hard for others to locate the source that you get information from. The answer you get for asking the question above on ChatGPT is not fixed. Eventually, it will be impossible to do that.

Last but not least, in my opinion, ChatGPT could be an useful tool for productivity, and there is no reason why you should not use it if you found it helpful for your work as long as you acknowledge its shortcomings. Don't trust it blindly.

6
  • 4
    The danger of a "great tool for productivity" is producing what? Definitely a risk of quantity over quality here.
    – J W
    Dec 3, 2022 at 12:13
  • 5
    You don't know how tremendously happy I am to know someone else thinks the same thing. AI is in general over-glorified instead of being seen for what it actually is. It is not even remotely intelligent and does not, in any case, take into account or understand context. Without taking into account context, it cannot possibly be called intelligent. Dec 3, 2022 at 12:20
  • 3
    @JW I edited it. Not what I intended to say, but I still believe many people will find such a tool useful. Quality-control is needed, just as you need to do quality-control for things you find on Google. It is important to acknowledge what it can do and cannot do. However, what makes me feel uncomfortable is that we don't know the source which this tool gets the information from.
    – Neuchâtel
    Dec 3, 2022 at 13:21
  • 4
    @NordineLotfi I often use the term "statistical learning" instead of "machine learning". It is obvious. Many "machine learning" techniques were invented by statisticians before computers even existed. Also, is there any machine learning model which does not rely on an implicit assumption about probability distribution? I doubt that. Machine learning also does not "learn" anything from the data. It is inherently based on statistical models which are by no means intelligent.
    – Neuchâtel
    Dec 3, 2022 at 13:31
  • 2
    The more you talk, the more I relate to and agree with what you say. I said the exact same thing once in an argument/discussion, but mine was probably more direct because I said "AI is just statistics", but now, I feel like it's closer to "brute-forcing with statistics" instead. Dec 3, 2022 at 13:45
8

It's worth mentioning that OpenAI expressly forbids using their API to generate content without clear attribution including how the AI was used. (Note: this is applicable at least to the OpenAI API Platform, and may be applicable to other language models, though it is unclear if it applies to chatGPT.) This is in addition to the ethical and legal issues on the publication side that the other answers discuss.

From OpenAI's website (emphasis mine):

Content co-authored with the OpenAI API policy

Creators who wish to publish their first-party written content (e.g., a book, compendium of short stories) created in part with the OpenAI API are permitted to do so under the following conditions:

  • The published content is attributed to your name or company.

  • The role of AI in formulating the content is clearly disclosed in a way that no reader could possibly miss, and that a typical reader would find sufficiently easy to understand.

    ....

For instance, one must detail in a Foreword or Introduction (or some place similar) the relative roles of drafting, editing, etc. People should not represent API-generated content as being wholly generated by a human or wholly generated by an AI, and it is a human who must take ultimate responsibility for the content being published.

The website also has a suggested blurb that can be used:

The author generated this text in part with GPT-3, OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed, edited, and revised the language to their own liking and takes ultimate responsibility for the content of this publication.

I suspect that other large language models will have similar use policies.

In short: it's not just up to the venue and the authors. By using a large language model published by some company, you are obligated to read and follow the terms of use for that model, which may include proper attribution.

3
  • You are citing from the content policy of OpenAI's API, but the OP's question was about ChatGPT, which is a different product. Sep 30, 2023 at 12:37
  • 1
    @ChristophThiede That's interesting, I assumed this would also apply to chatGPT, but I don't know. The general terms of use doesn't mention attribution. Sep 30, 2023 at 15:44
  • 1
    Also, in the brand guidelines, they say "If publishing text or images generated by an OpenAI model, you are not required to attribute." Sep 30, 2023 at 21:33
6

It's difficult to give a conclusive answer. AI-assisted writing is fairly new, and the legal and ethical understanding of it is in flux. I would caution against using AI-assisted writing in published work for a few reasons related to plagiarism and copyright.

AI tools are trained on large datasets, often including published, copyrighted material. You can cite the tool itself, but what about the publications it learned from? Should those be cited as well? Are you certain that it won't produce a paragraph similar enough to a former work to constitute plagiarism? Do authors have a responsibility to ensure that the tools they use aren't plagiarizing or infringing on copyright, or does that responsibility lie solely with the developer of the tools?

At the moment, there are no clear legal answers. It would be prudent to wait until AI technology—and legal and ethical opinions of it—mature.

1
  • The legality is really the least important issue. The actual problem is that so-called AI can't do what its proponents advertise: Reason intelligently. It blindly puts together text fragments based on statistical patterns in the (huge) training data. Jun 9, 2023 at 8:02
6

The example in the question body generates the impression that large chunks of text are to be generated automatically and submitted without editing. However, I'd like to approach the question as asked in the title and more from the perspective of what I consider acceptable usage of AI models as tools.

Summary

  • I think such AIs can be valuable tools to help generating initial draft texts, once the current hype is over and they get to be used for what they are good for. Though under much closer human supervision than the question envisions: These models are giant autocompletion engines, and could/should be used as such; they are not trained to predict reality or truth (other than the reality of what may occur in a text).
  • We are already using deep models - possibly without realizing - for several tasks during research and paper writing without particular concerns.
  • Whether we find it acceptable or not to use such tools to help with proper scientific writing, we'll anyways have to learn how to deal with auto-generated spam papers.

TLDR

First of all, I'm in STEM (for this question: as opposed to sciences/disciplines that are more working directly on language such as literature or also law). The science I generate is partly experimental and partly computational (as in crunching the numbers of those experiments). Fields whose core is in the language may need to develop their own standards.
(Also, I'm into machine learning, but not into natural language processing.)

However, keep in mind that we are already using automated tools:

  • Automated theorem proving in maths

  • Google uses a natural language processing model in their search engine.

  • Translators such as deepl come to mind as well. Particularly in case you're not a native English speaker and look for translations of sentences explaining your thoughts in your native language.

  • Spell checkers are state of the art. Thesauruses or synonym and antonym dictionaries have been around since ever, nowadays there are also rephrasing tools. Newer tools like grammarly were explicitly recommended in a scientific writing course I lately attended. Note that grammarly can rewrite sentences to make them shorter and more legigible.

  • I happily use tab completion, including the variety that predicts what word I want to write next when writing a text.

  • Similarly, speech (voice-to-text) or character recognition (OCR) software uses deep learning.

  • At my secondary affiliation JKI, colleagues develop cadima for helping with systematic reviews. Behind the scenes, they work with (and on) natural language processing AI models for that tool as well.

AFAIK, using such tools is not considered questionable.

As a side note: when I worked on my very first paper, I was at an institute that still had a secretary/assistant. She drew the illustrations about our methodology according to our instructions and got properly acknowledged similar to how one would acknowledge who helped with field or lab work, but did not contribute intellectually to a paper. There is nothing ethically questionable in this practice, neither.

I'd tend to "acknowledge" models like other software relevant for the paper: either in the text or as reference if there's a scientific publication about it. Since that is done not only for, say, the package that computes a statistical model but also for packages like ggplot2 that produce diagrams and figures, why not for packages that generate (parts/early drafts) of the text.


The aforementioned scientific writing course discussed a work flow that has distinct steps for deciding which content, text generation (and recommended freewriting, possibly using voice-to-text software) followed by editing, revising (and reorganizing) the produced text chunks.

GPT-3 and similar models predict how a given text continues. This is IMHO important to understand: It's not answering questions in the sense we'd discuss question and answer, for those models, answers are text that is predicted to come after a question.

It also means that some behaviours that may be considered less desirable are actually what follows from the precise prediction task of those models.

In particular, since the "answers" are in a sense extrapolated from the initial text/keywords, we should expect that this process is unstable (see the different "explanations" of p-values in @edelweiss' answer). (I also wouldn't be astonished if this extrapolation behaviour turns out to be related to the possibility of the model generating abusive or heavily biased language: such extrapolations have a tendency to amplify things)

I think we also need to keep in mind that when talking about writing a scientific paper, we're talking about novel (as in new), but real (not fictional) content. This means we're asking for even more extrapolation: what we want to write about must be outside the body of text known to the model.
And, GPT-3 is not a domain-specific model for predicting how a scientific text goes on, its training base comprises all kinds of text, including invented ones such as fictional or even fantasy texts. Note that the AI generated content has often been described as dream-like - which may be how we humans perceive such instability in the prediction. (It would be interesting whether such a model trained purely on scientific texts and textbooks would produce less fantasy)
(We also should not wonder about "fake news" being produced; and it should by its set-up be able to predict also how fake news stories go on.)

This suggests to me that these models may best be used as very sophisticated auto-completion tools (with tight control by us of the content - and that possibly tighter control is needed for generating/auto-completing parts of a scientific text than for, say, a web page or blog post on some well-known topic), and maybe as a tool for the free writing stage, generating (even a variety of) sentences or paragraphs from maybe a list of bullet points.

I'd gladly use an AI tool with a voice-to-text of my talk as starting point, applying a "scientific paper" language style to generate an initial pre-draft version of a manuscript. ...*dream*...

When writing a scientific paper, we provide (and are responsible for) the knowledge content. I care less about "who" puts the thoughts into nice-to-read English, and personally find the use of also very sophisticated tools entirely acceptable.


relevant blog posts and papers

(One of these posts mentions that also plagiarism detection software will become more sophisticated, establishing "writing fingerprints" of people. Unless these fingerprints are transparently computed, and based on proper science, I'd have the same type of concerns with plagiarism accusations on such a basis that I have about predictive policing, in particular based on intransparent (and likely not well validated) models.)


I don't want to speak (much) on the licensing issues:

  • If ChatGPT is found to have been trained on material that was not properly licensed (see @AustinHemmelgarn's comment on the respective lawsuit) it will likely be taken down. A different AI model trained on a legally available body of text will likely be developed (possibly by someone else) and provide similar functionality - so the rest of these answers is not affected.

  • As Schmuddi discusses, copyright of content generated by AI is not clear.

  • However, I'd expect for an AI model (software) that's used basically as autocompletion tool, i.e. under close (intellectual) supervision of humans, that the copyright lies with those humans. This is IMHO the more relevant and realistic scenario for a proper scientific text being written using an AI.
    (Consider: if Microsoft were to loose a lawsuit for copyright infringement for something incorporated in the Word software. Whatever happens to the future of Word, your text produced with the help of Word stays yours.)

4
  • 1
    An interesting and highly related question over on Law SE today: law.stackexchange.com/questions/86936/…
    – Jon Custer
    Dec 5, 2022 at 23:33
  • @JonCuster: yes, though that refers (also) to a different "level" of plagiarism, i.e. ChatGPT was likely trained on data that was obtained illegally. (see AustinHemmelgarn's comment above about the law suit against OpenAI/Github) Whether identical code produced by an AI that was provably not trained on that material (which is extremely unlikely here) is plagiarism is an interesting question. AFAIK, in IP right, the possibility of two humans accidentally producing the same IP content is acknowledged, and not considered an infringement. Academic rules are yet different, since more literature Dec 6, 2022 at 0:10
  • ... search is required. Whereas e.g. in patent right, a inventor may keep their invention secret (but prove the date, e.g. with a notary) and when later someone else files a patent with that claim, the first inventor can go on using their previous knowledge. AFAIK, they are not required to screen patent applications whether their secret invention shows up anywhere. Whereas the authors of a paper are expected to make sure their state of the art is up to date at submission. Dec 6, 2022 at 0:13
  • And this: meta.stackexchange.com/questions/384410/…
    – Jon Custer
    Dec 6, 2022 at 0:29
6

As long as your text is clear and your research is sound, you may use any tool you prefer to aid you in the writing. Indeed you'll have to proofread the text and make sure that everything is correct. You'd have to check if the license of the particular model you wish to use permits you to use it in such a way or if there are some limitations/attributions required.

If you believe that the text generated by such models is good enough, I don't see why you should not use it.

EDIT: as pointed out by other people these kinds of models may have licensing issues with the data used to train them, this is indeed a big problem. Also as pointed out by others, these models may plagiarize other authors if they have little information about a topic. This said, I stand by the idea that if the final text is a publishable text, with a sound research and that satisfies all the requirements to be publishable, you may use any tool you prefer to write it.

8
  • 8
    Whoever downvoted this might explain the vote. It would be useful information for the OP.
    – Buffy
    Dec 2, 2022 at 14:36
  • 3
    (+1) for answer. @Buffy: When reading the question I was thinking along the lines of this answer, but I did feel a bit uneasy about my thinking. Nonetheless, and this was still before I read stanton63's answer, I couldn't come up with any counterargument that wouldn't also (at least in times past) argue against things like spell check, use of calculators (to say nothing of computer algebra systems), online translators (naturally subject to additional human editing, at least at the present time), etc. Or going further back in time, dictionaries, math tables, mechanical drawing devices, etc. Dec 2, 2022 at 15:30
  • 14
    What if my preferred tool is Cut and Paste from an unreferenced source? Spell checkers and grammar checkers are pointing out possible issues, and I override them all the time since I know better. Wholesale production of bulk text is a different beast.
    – Jon Custer
    Dec 2, 2022 at 16:09
  • 9
    I was not the first downvoter, but I have just added a downvote (-1). The assertion that you may use any tool you prefer as long as you are clear and your research is sound seems obviously incorrect to me (and very dangerous advice). It ignores all the serious issues of licensing/plagiarism, etc. that are set out in the main answer. To go further, what if you instead use the "tool" of copying another researcher's paper without credit/citation? May you do that so long as the text is clear?
    – Ben
    Dec 2, 2022 at 20:26
  • 4
    @Jon Custer: It seems I read between the lines a bit when reading the OP's question "is it OK to generate parts of a research paper", because I didn't think this was LITERALLY what the OP was asking, but rather the question was intended to be about the use of this as a tool for helping smooth out English written by a non-native speaker, and whether this is cheating in some way. I assumed the use would be along the lines of the user, for example, looking at how certain phrases appear and maybe making use of these phrases in their own writing, (continued) Dec 2, 2022 at 21:58
5

An answer to the literal question in the title, "Is it OK to generate parts of a research paper using a large language model such as ChatGPT?", is probably yes. At least if your research paper happens to be about LLMs, trying to understand how and why they work and how to improve them.

A fairly recent and somewhat extreme example is this article, mostly written by GPT-3. The (co)authors asked the LLM to write a paper about itself (using multiple queries). As a consequence, that paper contains a lot of sections written entirely by the LLM.

Whether or not something like that really is acceptable (or should be) is, just as mentioned in the other answers, still up for debate. This particular paper is still in peer-review (or it was at least in June 2022, when the (co)author wrote an opinion piece on their own paper and the ethical and moral questions it poses in Scientific American).

5

I'll add to the other answers that some conferences explicitly forbid some use of language models, so if you plan to use some language models you should make sure it doesn't violate some policy of your publication target.

E.g., https://icml.cc/Conferences/2023/llm-policy:

The Large Language Model (LLM) policy for ICML 2023 prohibits text produced entirely by LLMs (i.e., “generated”). This does not prohibit authors from using LLMs for editing or polishing author-written text.

2

Times have changed. These days the answer is broadly "yes", but 1) you can't list the AI as an author, 2) you must disclose where you used AI, and 3) you are still responsible for the content of your paper, even if it was written by AI.

The use of artificial intelligence (AI) tools such as ChatGPT or Large Language Models in research publications is expanding rapidly. COPE joins organisations, such as WAME and the JAMA Network among others, to state that AI tools cannot be listed as an author of a paper.

AI tools cannot meet the requirements for authorship as they cannot take responsibility for the submitted work. As non-legal entities, they cannot assert the presence or absence of conflicts of interest nor manage copyright and license agreements.

Authors who use AI tools in the writing of a manuscript, production of images or graphical elements of the paper, or in the collection and analysis of data, must be transparent in disclosing in the Materials and Methods (or similar section) of the paper how the AI tool was used and which tool was used. Authors are fully responsible for the content of their manuscript, even those parts produced by an AI tool, and are thus liable for any breach of publication ethics.

0

No, using language models like ChatGPT to generate parts of your research paper is not advisable. Several issues arise from the direct usage of artificial intelligence services such as OpenAI ChatGPT, BingAI, BardAI, and so on in the education sector. Some of the common issues are:

1. Plagiarism Accusations, 2. Licensing Issues, 3. Referencing Issues, 4. Originality of the Sources Issues

If you are confused about any concepts, you can ask these services to explain the meaning. However, after grasping the concepts, you should do your own research, learn from it, and use it in your research papers.

One more thing. Someone once said, "In your life, all kinds of troubles will come looking for you all the time. You don't need to go out and look for them."

A legal team in New York City recently got reprimanded by a Federal Judge for writing a legal brief with the help of ChatGPT. Turns out, none of these cases that ChatGPT had cited to the lawyers actually existed.

NYC judge scolds lawyer for using ChatGPT to write brief full of ‘legal gibberish’; attorney swears robot ‘duped’ him | Yahoo! News | Friday, June 9, 2023

To conclude, I think that it is okay to use ChatGPT within ethical and code of honesty constraints, but beyond that, you are going to attract all kinds of negative attention and consequences you never expected to receive.

Good Luck to you.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .