In the long run, presumably the object should be to teach students the skills they'll need in a world where GPT exists - which would include include the ability to use GPT for the things it's good at, while still maintaining enough skepticism about the technology and enough understanding of the code to avoid embracing its mistakes uncritically.
One interesting strategy I've seen, via Reddit: https://old.reddit.com/r/ChatGPT/comments/117gtom/my_friend_is_in_university_and_taking_a_history/
My friend is in university and taking a history class. The professor
is using ChatGPT to write essays on the history topics and as the
assignments, the students have to mark up its essays and point out
where ChatGPT is wrong and correct it.
As a programming-oriented example, I just asked ChatGPT the question that plagues me any time I order at a café:
write a Perl regular expression that matches all common variations of spelling for "Geoffrey", e.g. "Jeffry", "Jeffery" etc.
Response:
To match all common variations of the name "Geoffrey" using Perl
regular expression, you can use the following pattern:
/^(G[eoa]offr(e|ey|y)|J[eo]ffr(e|ey|y))$/i
This pattern will match the following variations:
Geoffrey
Geoffry
Geoffery
Geofrey
Geofry
Jeffery
Jeffry
Jeffray
Jeffraye
Joffrey
Joffry
Joffery
It then provided a nice explanation of how this pattern is supposed to work. But just a few seconds' scrutiny will show that several of those variations (7/12 by my count) are actually not matched by its pattern. It doesn't even pick up "Jeffery", which was one of the cases in the prompt.
If you play around with GPT, you start to get a sense for the things that can trip it up. For instance, I find that it's often vulnerable to context reversals, where most of its training material deals with a question asked in one direction but the prompt runs in the opposite direction. Like so:
Please convert the following old Python 3 code to run in Python 2:
print("hello world")
GPT:
In Python 2, the print statement requires parentheses around the
arguments. Here is the Python 2 version of the code:
print("hello world")
This code will output "hello world" to the console when executed in
Python 2.
This is of course incorrect - in Python 2 print was a statement not a function, so the correct change would have been to remove the parentheses. But GPT has seen thousands upon thousands of examples of updating Python 2 to Python 3, and very few in the other direction, so it patterns its answer accordingly. The "old code" is thrown in to mislead it further.
(Disclaimer: GPT has a random element and this trick doesn't always work. On a few trials with similar prompts, I found it got the conversion right about 60% of the time. But even if it's right 90% of the time, I'd still need to vet its output 100% of the time.)
(As suggested by wizzwizz4 in comments, something with file= or end= might be a better example in Python.)
A problem format similar to the history-essay approach might be:
- Part A: Write a program to do [thing]
- Part B: Prompt GPT to write a program to do [thing], then compare with your own code from Part A. Discuss the major differences between the two versions and comment on which version is better.
Once you have a feel for GPT's weaknesses, you'll be in a better position to set questions for which it will occasionally stumble, or to provide examples where it has stumbled and ask students to explain what it did wrong. Once they understand that it's fallible, and have developed the habit of assessing its output with due suspicion, you've done your job.
The meta-problem here is that recycling old assessment material with minor variations is much, much less effort than coming up with new question styles appropriate to the new world, and lecturers who are suddenly required to update vast swathes of assessment material should be appropriately compensated and supported in that. But that'd be a separate question, perhaps best directed towards one's union.
