AI in computer science education
In this post, I will argue two points about AI and their implications for teaching:
- Unmanaged use of AI destroys students' incentive to learn.
- AI dramatically raises the ceiling for student outcomes in computer science.
And if studies are your thing (I take them all with a grain of salt), I will cite two that illustrate these points.
Unmanaged use of AI destroys the incentive to learn
At the start of the year, Anthropic ran a study with 52 mostly junior software engineers to analyse how AI affects performance and learning. The participants were divided into two groups: one had access to AI to complete a task, and the other had none. To no one's surprise, the group with AI access completed the task faster, while the group without AI performed better on a quiz testing knowledge of the task.
How AI assistance impacts coding speed and skill formation. Source: Anthropic.
This is not a surprise to anyone. As a software engineer, I live this reality daily. When I am under a time crunch, AI generates the code and I spend my time making sure nothing breaks instead of investing time to understand it. The one thing that reliably gets me to learn is working on projects with pedantic project owners who want to understand every design decision in the code review. Ultimately, this brings me to students, classrooms and my next point.
AI dramatically raises the ceiling for student outcomes in computer science
A study from the Technical University of Munich, published in Computers and Education: Artificial Intelligence, ran a similar comparison with 275 introductory programming students across three conditions: an AI tutor that gave hints but withheld full solutions, unrestricted ChatGPT, and no AI. Both AI groups scored far higher on the programming exercise itself (about 72% for ChatGPT and 58% for the tutor, versus 30% for the no-AI group), but this didn't translate into better understanding. The hint-based tutor edged out the other groups in conceptual understanding and, interestingly, increased students' intrinsic motivation, while students rated ChatGPT as easier and more helpful despite it offering no learning advantage.
I am proud to say that I have been working on code hints since December 2023. These hints are embedded in Agent Games, the coding education platform I am working on, with the idea that students receive an AI-generated hint only when they are truly stuck. There is no hint button, so students can't come to rely on it; a hint just shows up occasionally.
However, it's more than just hints. Programmers have the advantage of continuous feedback every time their code runs, and the only limitation for beginners is stack traces, which can look terrifying. This is where AI can help as well.
Honestly, I don't know whether, if I were teaching today, my students would do as well as they did in my final year of teaching (2022), before AI became a thing. However, I am confident that getting AI to work for our students, and not against them, is something we can figure out in the next couple of years.
Classroom implications
One big idea: create good incentives.
- The code no longer matters. The only thing that needs to be rewarded is demonstrated understanding of the code. If you are a teacher, just assume the code is AI-generated; the same goes if you are a code reviewer.
- Treat AI-generated code as a liability. If you submit something you don't understand, you lose a lot of marks. It's just like my job: I have a strong incentive to fire up the debugger and the Django shell to get to the bottom of how my code works, because I know a grilling is coming in the code review. The Agent Games platform flags all code that is pasted, or that shows a massive increase in complexity between submissions.
- Type code in manually. Yep, typing in code forces you to slow down and think about what you are doing. I strongly recommend it.
- Fail students for turning in AI work. I am not sure there is another way to send a clear message that the only way to pass, or to get good grades, is to genuinely understand the subject.