The example in the question body generates the impression that large chunks of text are to be generated automatically and submitted without editing. However, I'd like to approach the question as asked in the title and more from the perspective of what I consider acceptable usage of AI models as tools.
Summary
- I think such AIs can be valuable tools to help generating initial draft texts, once the current hype is over and they get to be used for what they are good for. Though under much closer human supervision than the question envisions: These models are giant autocompletion engines, and could/should be used as such; they are not trained to predict reality or truth (other than the reality of what may occur in a text).
- We are already using deep models - possibly without realizing - for several tasks during research and paper writing without particular concerns.
- Whether we find it acceptable or not to use such tools to help with proper scientific writing, we'll anyways have to learn how to deal with auto-generated spam papers.
TLDR
First of all, I'm in STEM (for this question: as opposed to sciences/disciplines that are more working directly on language such as literature or also law). The science I generate is partly experimental and partly computational (as in crunching the numbers of those experiments). Fields whose core is in the language may need to develop their own standards.
(Also, I'm into machine learning, but not into natural language processing.)
However, keep in mind that we are already using automated tools:
Automated theorem proving in maths
Google uses a natural language processing model in their search engine.
Translators such as deepl come to mind as well. Particularly in case you're not a native English speaker and look for translations of sentences explaining your thoughts in your native language.
Spell checkers are state of the art. Thesauruses or synonym and antonym dictionaries have been around since ever, nowadays there are also rephrasing tools. Newer tools like grammarly were explicitly recommended in a scientific writing course I lately attended. Note that grammarly can rewrite sentences to make them shorter and more legigible.
I happily use tab completion, including the variety that predicts what word I want to write next when writing a text.
Similarly, speech (voice-to-text) or character recognition (OCR) software uses deep learning.
At my secondary affiliation JKI, colleagues develop cadima for helping with systematic reviews. Behind the scenes, they work with (and on) natural language processing AI models for that tool as well.
AFAIK, using such tools is not considered questionable.
As a side note: when I worked on my very first paper, I was at an institute that still had a secretary/assistant. She drew the illustrations about our methodology according to our instructions and got properly acknowledged similar to how one would acknowledge who helped with field or lab work, but did not contribute intellectually to a paper. There is nothing ethically questionable in this practice, neither.
I'd tend to "acknowledge" models like other software relevant for the paper: either in the text or as reference if there's a scientific publication about it. Since that is done not only for, say, the package that computes a statistical model but also for packages like ggplot2 that produce diagrams and figures, why not for packages that generate (parts/early drafts) of the text.
The aforementioned scientific writing course discussed a work flow that has distinct steps for deciding which content, text generation (and recommended freewriting, possibly using voice-to-text software) followed by editing, revising (and reorganizing) the produced text chunks.
GPT-3 and similar models predict how a given text continues. This is IMHO important to understand:
It's not answering questions in the sense we'd discuss question and answer, for those models, answers are text that is predicted to come after a question.
It also means that some behaviours that may be considered less desirable are actually what follows from the precise prediction task of those models.
In particular, since the "answers" are in a sense extrapolated from the initial text/keywords, we should expect that this process is unstable (see the different "explanations" of p-values in @edelweiss' answer). (I also wouldn't be astonished if this extrapolation behaviour turns out to be related to the possibility of the model generating abusive or heavily biased language: such extrapolations have a tendency to amplify things)
I think we also need to keep in mind that when talking about writing a scientific paper, we're talking about novel (as in new), but real (not fictional) content. This means we're asking for even more extrapolation: what we want to write about must be outside the body of text known to the model.
And, GPT-3 is not a domain-specific model for predicting how a scientific text goes on, its training base comprises all kinds of text, including invented ones such as fictional or even fantasy texts. Note that the AI generated content has often been described as dream-like - which may be how we humans perceive such instability in the prediction. (It would be interesting whether such a model trained purely on scientific texts and textbooks would produce less fantasy)
(We also should not wonder about "fake news" being produced; and it should by its set-up be able to predict also how fake news stories go on.)
This suggests to me that these models may best be used as very sophisticated auto-completion tools (with tight control by us of the content - and that possibly tighter control is needed for generating/auto-completing parts of a scientific text than for, say, a web page or blog post on some well-known topic), and maybe as a tool for the free writing stage, generating (even a variety of) sentences or paragraphs from maybe a list of bullet points.
I'd gladly use an AI tool with a voice-to-text of my talk as starting point, applying a "scientific paper" language style to generate an initial pre-draft version of a manuscript. ...*dream*...
When writing a scientific paper, we provide (and are responsible for) the knowledge content. I care less about "who" puts the thoughts into nice-to-read English, and personally find the use of also very sophisticated tools entirely acceptable.
relevant blog posts and papers
(One of these posts mentions that also plagiarism detection software will become more sophisticated, establishing "writing fingerprints" of people. Unless these fingerprints are transparently computed, and based on proper science, I'd have the same type of concerns with plagiarism accusations on such a basis that I have about predictive policing, in particular based on intransparent (and likely not well validated) models.)
