GPT in examination - experiment and lessons

Topic: GPT in examination - experiment and lessons

When: 29 January at 12.00-13.00

Where: Online

Prequel: Recording from lunch seminar 6 March 2024 at ai.lu.se

Speakers: 

• Julia Adlecreutz , PhD student, Automatic Control, Lund University 
• Felix Agner, PhD student, Automatic Control, Lund University

Spoken language: English

Abstract

At the department of automatic control, we teach our course "introcuction to automatic control" to most of the engineering students at LTH. With the giant improvements in large language models, we are of course interested in understanding how this will impact the control engineer of the future - both in terms of what tools they will need in their future work environment, and how our education should be shaped to provide them with these tools.

As an initial step towards understanding these questions, we wanted to answer the following question: How well can students handle the problems we test them on in our exams, with access to large language models?

In this talk, we will show the results from an experiment we performed in order to test this. We placed 8 engineering students in a room with access to GPT-4 through Microsoft copilot and asked them to take our exam. They had not taken the course before, but they had the required background courses. We will tell you quantitatively how they performed, but perhaps more importantly we will qualitatively discuss their interaction with the language model how they reached the answers they wrote down.