Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home. They would have said 'cheating', even with no concept of AI.
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
> The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
So true. I am aware of classes where everyone who didn't use AI cheated.
The simple reality is that if AI makes better answers than a student, and exam scores are normalized, then students who don't use it will fail as soon as a decent proportion of students do use it.
The common thinking of often a mental pattern of that intersects somewhere between laziness and comfort.
Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
> Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
I can't tell - are you suggesting these aren't good practices/traits to be learning when people are still in the "fundamentals of education/learning" stages of their lives?
I did all my basic differential and integral calculus studying by mind only. I don't do it that way in my career day to day now - nor could I without some serious practice. But the efforts I took in learning this way in undergrad made me a much stronger student and made me much more comfortable leveraging calculus in more application driven fields of study.
But in practice, having another human cheating for you was often unpractical: people don't usually like helping cheater, and simply trying to find an accomplice may get you in trouble. Because of that, it is relatively inefficient and therefore not a real problem and not a real impact on the final quality of the evaluation.
LLM is indeed just the same, except that finding an accomplice is now easy and without risk.
> Ask an examiner from 20 years ago the risk of allowing people to take exams in their own home.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
Not really. An open book exam still requires you to know which book to bring in, understand the concepts, and be able to reference them on the fly to answer questions. Basically, you need a reasonable grounding in the material to know where to start figuring out your answer.
What’s different with at-home exams is there’s nothing stopping your ringing your friend to ask for the answer, or looking it up on Google (now ChatGPT), or asking your parents who happen to be in the industry, if you want to go really old school!
I've had plenty of open book exams where the prof knew you would fail if you grabbed the book for more than a second. It's pretty much the same as the exams where you get to write your own cheatsheet: if you need it too much you are screwed.
The act of writing the cheat sheet is often enough to remember I find. It's yet another repetition of the material, just like doing labs and practice exams. And if you wrote the cheat sheet yourself, you also often know "where to look" for something specific, even if it's just to be sure you didn't remember something incorrectly and you really do only need to look at it for a few seconds.
So in my book (pun intended :P), allowing and actually encouraging a "cheat sheet" is a good thing. Open book is worse, as it's usually way too large and badly indexed. And who's gonna use an actual book in their actual job anyway?
Seriously. Kids are going to cheat. It's already easy enough to just throw the test material into the LLM and get a bunch of flash cards on relevant content and memorize that. I Wish I had AI in college.
Using a tool to help you study isn't cheating. Using a tool to take the test for you, without regard to your own skills or knowledge of the subject under test, is.
From watching slightly younger than college age kids adapt to the current world, I think you should be glad you did’t have access to LLMs during your learning years.
It’s too easy to slip from the idea that you’re just going to use the LLM to generate study materials into thinking that you’re just going to let the LLM do this homework assignment because your tired and then into a routine where ChatGPT is doing everything because you’ve come to rely on it. Then the students get slapped in the face with a sudden bad grade because the exams are in-person and they got all the way to the end of the semester with A-graded homework despite very little understanding of the material.
This is a very concerning statement given the implications of your post.
AI can be a tool for learning or a tool for passing. Only one of those things is beneficial for society and it's not the one short minded students in crunch time will, on average, care about.
I also wish I had AI in college. I would have used it to descramble the unintelligible utterances of the calculus lecturers who had minimal or no English language skills.
> It's already easy enough to just throw the test material into the LLM and get a bunch of flash cards on relevant content and memorize that
LLM summarisation is broken, so I wouldn't expect them to get very far with this (see this comment on lobste.rs: https://lobste.rs/c/je7ve5 )
Also, memorizing flashcards is actually, to some point, learning the material. There's a reason why Anki is popular for students.
Ultimately, however, this comes down to the 20th+21st century problem of "students learning only for the test", which we can see has critical problems that are well-known:
I remember reading something when I was studying for AWS certs (might’ve been from AWS itself): the goal of the certifying bodies is to make as much money as possible. For this to happen, the exam can’t be so hard that nobody takes it, but it can’t be so easy that everyone takes it and it loses its value.
Organizations have been coasting on their pre-Covid reputations for a while. Now it’s time for them to adjust the slider the other way.
I don't know about this part. Years ago, my friend in college was taking all kinds of Microsoft certification exams and passing them with near perfect score. Thing is, he had no clue about most of the topics he passed, he had never worked with those tech. He just spent a bunch of time collecting questions (which wasn't that hard to find) and memorizing the answers. They could've made it difficult enough so just rote memorization wouldn't work, but they didn't (don't know if it has changed now).
Companies had long figured out these certifications are just easy money. It is hard to resist the temptation to just charge hundreds of dollars for a test and add it as a "profit center"
I don't want to sound heartless, especially as I'm in the high-risk category myself, but I think it's important to recognise that while COVID hasn't gone away, it is no longer a pandemic.
It is now endemic instead, and needs to be managed as such.
My wife is a teacher of physics and math for an online highschool. Its very common for kids to go into the in person exam with a mark in the 80s and 90s and get a failing grade on the exam.
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
When I took quantum mechanics in grad school, I struggled through the weekly (and intense) homework sets. My TA was a hardass, I’d spend hours on some problem, several few pages of math work just for one problem, and make some dumb mistake in an integral somewhere, being off by a factor of 2 at the end and only getting 2 of 4 points.
It was painful, and I felt like a dumbass seeing the other kids regularly getting perfect scores.
Then the midterm came and I blew them all out of the water. I hadn’t realised they somehow had the solutions manual so just got perfect scores all along but clearly didn’t learn the material like I did.
I failed one exam in my final year of uni (marginally), but passed the module because of excellent coursework. I put in an order of magnitude more work into the coursework for that class than I did any other class because I knew I was going to struggle in the exam.
In all honesty I shouldn’t have passed that course but it is what it is - and as far as I was (and still am) concerned, it was a bolt on course that I am ok being limited in my knowledge of.
I don't know, I've known many people that struggle with exams even if they know the material and even more people that excel with exams that learn nothing. Falling back on any kind of exam is just a recipe for more rote learning and that doesn't create better people (although possibly better readers, which we need).
(Preface: I am not a teacher, and I understand this is a hot take). At the end of the day there's an unwillingness from every level of education (parents, teachers, administrators, school boards, etc) to fight against the assault on intelligence by tech.
I don't think kids should have access to the public internet until they're adults, and certainly should never have it in schools except in controlled environments. Schools could create a private networks of curated sites and software. Parents don't have to give their kids unfettered access to computers. It's entirely in the realm of possibility to use computers and information networks in schools, accessed by children, designed to make it impossible to cheat while maximizing their ability to learn in a safe environment.
We don't build it because we don't want to. Parents don't care enough, teachers are overworked, administrators are inept, and big tech wants to turn them into little consumers who don't have critical thinking and addicted to their software.
I see this line of argument more and more over the last decade and it makes me feel heartless for my opinion.
But if you know the material but cannot apply it in an examination then you either don't actually know the material or don't have the emotional (for lack of better term) control to apply it in critical situations. Both are valid reasons to be marked down.
It can be hard to prevent cheating in person too: A criminal enterprise was uncovered in 2019 in Sweden. They had targeted the local SAT variant (högskoleprovet).
Their end user equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers.
The answers had been provided by some team in a back office based on the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
Until quite recently, it was trivial to cheat on remotely proctored exams. All you had to do is spin up a VM, take the exam inside the VM, and use the host system to look up answers. I believe the main proctoring services now have crude VM checks. You can probably still use a KVM switch or a DP splitter and a buddy...
My point is that since it is so incredibly easy to cheat (despite countermeasures that are essentially theater), returning to in person exams is probably a good thing.
"We are doing what we can to hang on to relevancy as gatekeepers who already held way too much authority over a field". They are going to use AI on the job anyway.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Certifications have always been irrelevant for me, but that's only because my goal has always been what I'm capable of doing on my own AND (this one is a biggie) I was unbelievably fortunate to have several people in my career who trusted that I could get the job done.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Isn't that what the CPA and Bar exams, to use US analogs, do? They are an in-depth test or sets of tests that prove a person has a useful set of knowledge in a given domain.
I don’t think it will be too long before the pendulum swings back towards “real people who actually know the subject”. At that point, I might feel bad for everyone who coasted on AI.
I've had no end of problems with accountants regardless of their certifications, they operate in a domain with an incoherent body of contradictory and highly subjective rules yet make it out to be a science.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
One of the main issues I had when I took accounting was that you often couldn't figure out things from first principles because the "right" way was whatever the relevant financial accounting standards board said it was. But following that standard is what companies need to do--and therefore has value--even if it's arguably arbitrary (within some general framework).
Yeah ... that's kind of the point. The money doesn't exist, but the violence people will use if their money is misappropriated is very real. Accounting is loophole patch stacked on loophole patch for thousands and thousands of years.
It's not intellectually enriching, but like it has the weight of society going back forever with dire consequences when it fails. That's not nothing even if it's boring from a technological point of view.
I think of it sort of like git. Technically, any sort of distributed version control would have served our industry just fine. Git didn't need to win, but things are vastly simplified having basically one version control framework to rule them all.
Get back to me when AI is actually reliably correct about any technical field.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
At the end of the day the job market will correct itself accordingly which is what most people who bother going to university or collecting any certificate care about. And right now it is already looking bleak. https://accountancyage.com/2025/09/29/pwcs-graduate-glow-up-...
Might be time we start adapting the pipeline into employment and start revising the importance of some of these gatekeepers before more people fall into unnecessary debt.
Here is what happened. ACCA, one of several accountancy bodies in the UK, charge their students extraordinary sums of money to take their exams. When I took accountancy exams there were 9of 3 hour written exams, in a real building, with real invigilators. All of the bodies at the same time realised that they could charge the same amount, pay Pearson to administer an electronic test and make more money out of their students. It was a disgrace then and it is a disgrace now
AI has taken it to the next level. Previously, with many exams you would still have to know how to identify the concepts and related keywords in a word problem to even know what words to look for in the index of the books on hand before you could get to the right page to start cheating.
Some of the certification exams I had to take back in the day even came with their own little reference manual that everyone got and was free to use to look up concepts and equations like you would in the real world. The book wasn’t helpful if you didn’t know how to recognize the way to solve the problem and look it up, though.
AI changes that. Now you don’t need to know anything at all. You don’t even need to parse the question or even speak the same language. Copy the problem into ChatGPT with a prompt attached. Copy the answer into the solution box.
Anecdotally, the rise of ChatGPT has also normalized the concept of cheating among students. The common thinking is that everyone is using ChatGPT, therefore you’ll be left behind if you don’t cheat.
So true. I am aware of classes where everyone who didn't use AI cheated.
The simple reality is that if AI makes better answers than a student, and exam scores are normalized, then students who don't use it will fail as soon as a decent proportion of students do use it.
This never should've been done to begin with. Education isn't supposed to be a competition.
Is this the sort of thinking of “everyone needs to be able to do calculus in their heads with calculators around” or “you still need to write in the age of computers/printers” or something different?
I can't tell - are you suggesting these aren't good practices/traits to be learning when people are still in the "fundamentals of education/learning" stages of their lives?
I did all my basic differential and integral calculus studying by mind only. I don't do it that way in my career day to day now - nor could I without some serious practice. But the efforts I took in learning this way in undergrad made me a much stronger student and made me much more comfortable leveraging calculus in more application driven fields of study.
LLMs make this way easier but you can pay someone who gives private lessons in any subject and they can easily take an exam for you.
But in practice, having another human cheating for you was often unpractical: people don't usually like helping cheater, and simply trying to find an accomplice may get you in trouble. Because of that, it is relatively inefficient and therefore not a real problem and not a real impact on the final quality of the evaluation.
LLM is indeed just the same, except that finding an accomplice is now easy and without risk.
Isn't this like an "open-book" exam? We had them 50 years ago when I was doing my A-levels in the UK, and I always thought it was a good system. The trouble now is of course that you can ask the book to look up the answer, unless the question is very well thought out, which is hard. The open-book thing worked best IMHO for things like practical chemistry, where you needed the technique as well as the theory.
What’s different with at-home exams is there’s nothing stopping your ringing your friend to ask for the answer, or looking it up on Google (now ChatGPT), or asking your parents who happen to be in the industry, if you want to go really old school!
So in my book (pun intended :P), allowing and actually encouraging a "cheat sheet" is a good thing. Open book is worse, as it's usually way too large and badly indexed. And who's gonna use an actual book in their actual job anyway?
From watching slightly younger than college age kids adapt to the current world, I think you should be glad you did’t have access to LLMs during your learning years.
It’s too easy to slip from the idea that you’re just going to use the LLM to generate study materials into thinking that you’re just going to let the LLM do this homework assignment because your tired and then into a routine where ChatGPT is doing everything because you’ve come to rely on it. Then the students get slapped in the face with a sudden bad grade because the exams are in-person and they got all the way to the end of the semester with A-graded homework despite very little understanding of the material.
This is a very concerning statement given the implications of your post.
AI can be a tool for learning or a tool for passing. Only one of those things is beneficial for society and it's not the one short minded students in crunch time will, on average, care about.
LLM summarisation is broken, so I wouldn't expect them to get very far with this (see this comment on lobste.rs: https://lobste.rs/c/je7ve5 )
Also, memorizing flashcards is actually, to some point, learning the material. There's a reason why Anki is popular for students.
Ultimately, however, this comes down to the 20th+21st century problem of "students learning only for the test", which we can see has critical problems that are well-known:
https://matheducators.stackexchange.com/a/8203
https://www.youtube.com/watch?v=J6lyURyVz7k
There will be a lot of COVID-era qualifications that are treated with a hint of suspicion in the future.
Take a look at A-level scores: https://schoolsweek.co.uk/a-level-results-2024-future-exams-...
( direct link to graph: https://schoolsweek.co.uk/wp-content/uploads/2023/04/Overall... )
It's unfortunate for those affected either way. It was a difficult time when drastic measures needed to be taken at short notice.
It's right to go back to in-person testing if there is a problem keeping remote exams fair.
Organizations have been coasting on their pre-Covid reputations for a while. Now it’s time for them to adjust the slider the other way.
I don't know about this part. Years ago, my friend in college was taking all kinds of Microsoft certification exams and passing them with near perfect score. Thing is, he had no clue about most of the topics he passed, he had never worked with those tech. He just spent a bunch of time collecting questions (which wasn't that hard to find) and memorizing the answers. They could've made it difficult enough so just rote memorization wouldn't work, but they didn't (don't know if it has changed now).
Companies had long figured out these certifications are just easy money. It is hard to resist the temptation to just charge hundreds of dollars for a test and add it as a "profit center"
The pandemic isn't actually over, at least, not for disabled people.
It is now endemic instead, and needs to be managed as such.
https://en.wikipedia.org/wiki/Endemic_COVID-19
The web wasn't alwasy that useful for cheating on timed exams as it was essentially like being able to bring in a formula sheet.
LLM's changed this such that you can type in the question and get a fully correct answer in a lot of cases.
The only solution that I see in education is that in person exams start to represent a larger and larger portion of a students grade such that the mid term and final will be more than 50% of a students grade for most classes going forward due to the gratuitous use of llms by students.
When I took quantum mechanics in grad school, I struggled through the weekly (and intense) homework sets. My TA was a hardass, I’d spend hours on some problem, several few pages of math work just for one problem, and make some dumb mistake in an integral somewhere, being off by a factor of 2 at the end and only getting 2 of 4 points.
It was painful, and I felt like a dumbass seeing the other kids regularly getting perfect scores.
Then the midterm came and I blew them all out of the water. I hadn’t realised they somehow had the solutions manual so just got perfect scores all along but clearly didn’t learn the material like I did.
In all honesty I shouldn’t have passed that course but it is what it is - and as far as I was (and still am) concerned, it was a bolt on course that I am ok being limited in my knowledge of.
(Preface: I am not a teacher, and I understand this is a hot take). At the end of the day there's an unwillingness from every level of education (parents, teachers, administrators, school boards, etc) to fight against the assault on intelligence by tech.
I don't think kids should have access to the public internet until they're adults, and certainly should never have it in schools except in controlled environments. Schools could create a private networks of curated sites and software. Parents don't have to give their kids unfettered access to computers. It's entirely in the realm of possibility to use computers and information networks in schools, accessed by children, designed to make it impossible to cheat while maximizing their ability to learn in a safe environment.
We don't build it because we don't want to. Parents don't care enough, teachers are overworked, administrators are inept, and big tech wants to turn them into little consumers who don't have critical thinking and addicted to their software.
I see this line of argument more and more over the last decade and it makes me feel heartless for my opinion.
But if you know the material but cannot apply it in an examination then you either don't actually know the material or don't have the emotional (for lack of better term) control to apply it in critical situations. Both are valid reasons to be marked down.
My guess is the number of exceptional circumstances is about to explode...
Their end user equipment consisted of a modified mobile phone hidden somewhere private, a necklace that acts like a magnetic coil and small magnets that you place against the eardrum. Then the operation would call the phone while the customer was in the auditorium and give them the correct answers.
The answers had been provided by some team in a back office based on the test that they had obtained in realtime from some planted source taking the test at the same time, somehow.
Calculations must be getting accurate now. Not only questions about vocabulary or domain concepts.
Triviality is not a dimension of ethics as far as I have come to understand it.
This also applies to universities. The world has changed but they have not and they will make sure to try and stay relevant as much as they can to continue to take money.
Edit: looks like it will take a while for some people to accept that we are not going back from this. The cat is out of the bag and your certificates are increasingly irrelevant. Sorry if you spent a lot of money and time to get it.
Certifications are about low trust. With the advent of modern LLM tech, trust levels are probably not going up.
Nobody needs to hire someone who can use an LLM because if that is the skill they're looking for they can just use the LLM themselves.
So if you need to hire someone because the LLM isn't cutting it, then you'll by definition need to be hiring someone who isn't using an LLM. Someone who isn't just using an LLM to make you think that they aren't using an LLM.
How is that going to be done? Sounds like a job for certifications to me. Not today's certifications, but a much more in depth, in person, and gatekeepery certification.
My guess would be that certifications, unfortunately, will be significantly more relevant in the days of LLMs. Not less.
Much like how if you stop going gym you lose muscle mass, the same happens with knowledge and understanding with the brain.
My conclusion as a whole is that accountancy as a profession rarely delivers any actual value to their customers, where much of the job is compliance theater at best.
It's not intellectually enriching, but like it has the weight of society going back forever with dire consequences when it fails. That's not nothing even if it's boring from a technological point of view.
I think of it sort of like git. Technically, any sort of distributed version control would have served our industry just fine. Git didn't need to win, but things are vastly simplified having basically one version control framework to rule them all.
Accounting exams are gatekeeping, yes. The good kind of gatekeeping where you make sure the people doing the job are actually capable. And you have avenues to punish those who fail their clients.
> This also applies to universities
Eh. I’d say the actual academics are about 1/3 of the university experience. The rest is networking and teaching you how to think and solve problems on a more abstract level. I’d say the people who farm that (and particularly the abstract thinking part) out to AI are going to be the ones left at disadvantage in the future. You’re completely replaceable.
For exams and other tutorial like material* the LLMs have enough public training data for it to be good enough.
* all those vibe coded apps that are 95% boilerplate.
And no one is financing anything but LLMs at the moment.
Might be time we start adapting the pipeline into employment and start revising the importance of some of these gatekeepers before more people fall into unnecessary debt.