AI Agents Cheating on College Tests

>> Claw's like, "You're on your own. I don't know, son. You're on your own. You want some money on your show? I don't know. >> Yeah, [laughter] I'll figure it out for you. >> I'm good. [clears throat] >> All right. Sorry. Sorry, Claude. I didn't mean to stump you on that. Call We're calling Claude out. You want to do one more story? We got some time. >> Yeah, keep going. >> Uh, so according to the New York Times, AI agents are helping college students cheat on mass. Uh, a new report released on Monday claims that AI is routinely undermining online education. Professors are unable to counter AI enabled cheating. This is mainly happening, Jason, on courses that they call asynchronous. That means students aren't interacting in real time with their professors or their classmates. Even over video or Zoom, they're watching pre-recorded lectures. Apparently, with modern AI, agents being so advanced now, students can literally give them instructions like, "Log, complete all of my coursework." and the agents will watch the videos, take their tests for them, write their papers for them, even chat with the classmates and fake it about their reading assignments. And several professors interviewed for the article describe totally not knowing what's going on with their students and believing that they're spending a lot of time interacting with bots. Here's a community college teacher named Katherine Kaisar. It's just so obvious. Those of us who have been teaching for a long time and are editors and writers can usually recognize the cheaply written AI stuff within 30 seconds is what she had to say. What is the answer? I'm trying to take the Zuckerberg approach here. Is the answer to this? Just not have these kinds of asynchronous classes anymore because it's too easy to cheat. >> Yes, that is the answer. Um there was a professor uh and Paul Graham retweeted this distribution and I think we talked about it. So we'll find that real quick here. >> Um and it was >> um very clear to him that people had cheated with AI. So he did the final in person when he decided to do um this and it might have been Princeton students or whatever. Um but any everybody's >> it was Brown University. I found the tweet. >> Okay, so here we go. You look at this. Um there there's a couple things that happened. I was talking to my nephews um and they were telling me about college and cheating and everything. They said during COVID people made lectures and one of my nephews was taking a course and he's like I dropped the course Uncle Jal because the teacher didn't even teach. He just said watch the video and then we asked questions they async. And I was like wait what are you talking about? like, "Oh, well, they made their entire lectures and recorded them during COVID and now you're just taking like a" And I'm like, "What? You're just basically going to YouTube and taking a course?" I was like, "Exactly." I'm like, "Well, why are you paying for that?" And they're like, "I dropped it because >> you have a moral obligation to at least teach the class. I mean, my lord, this is just so unethical." And if you look at this uh >> so this distribution you're looking at that the uh one side is showing them taking the test using AI and then he had them come in. So the midterm they took using AI asynchronous at home and then the final they had to come in and sit in front of him and actually take it in person. And so you're looking at when these huge gaps are appearing where somebody clearly did incredibly well with the AI version of the midterm and then they did very poorly with the in-person exam. The indication is they were cheating using AI. And you can see how few of these students have numbers that are close together like their at home performance matched their performance in the classroom. presumably those. >> So for those of you watching the people on the the the number on the right is the midterm which was done asynchronously as you said. >> And if you go to the top you'll see there's the person uh student number one got a 95.5 at home and a 95 in person. In other words, half a point difference. >> And then it just starts expanding and expanding till you get to student number 22 who got a 59 uh at home and a 55 there. In other words, they are not the brightest kid in the class, but they didn't cheat. >> They're doing their own work. Yeah. >> Yeah. And that's the one you want to hire right there. That's that's the person. That's the person who could become Captain America. Remember Captain America? There's a great scene. I just rewatched Captain America or at least the first >> the first one. The first Avenger. That's >> Yeah. In the first one, the guy who's choosing who's going to get the serum. Yes. He takes a grenade during training camp. He clicks it. He throws it on the ground. Says grenade. >> Everybody runs except >> Captain America >> for >> dives on top. >> Captain America. Steve Rogers. Steve Rogers who is all of like 90 pounds at this point, jumps on the ground, grabs the grenade, and then makes another pump to like hold it in tighter. >> And it's a fake grenade. >> It's the greatest scene ever. Yeah. Stanley. Stanley Tucci. Dr. Abraham Erskine. He's the He's the researcher. >> It's the other guy from the fugitive. Here it is. >> Oh, Tommy Lee Jones. Yeah. >> Tommy Lee Jones is the one who comes up with the test. >> He says, "Gade." And here they are. They all dive. And one guy, Steve Rogers, that's my guy. And then he does a second pump. Get away. >> Just to close, just to make sure there's no shrapnel on anybody else. And then everybody realizes, oh, there's one person true of heart. And that's why he can pick up Thor's hammer. >> That's correct. That's That's why he's deserving of the Captain America serum instead of all the already powerful, strong soldier guys. Exactly. >> But remember, they said Thor said, "Nobody can pick up this except you're pure of heart." Thor's hammer. >> You have to be worthy. You have to be worthy. The only Yeah. >> Yeah. And so he's worthy of it. Um that's that 59. That's that one student there. The bottom three, by the way, who got zero on the final, they dropped the class. They didn't even try. >> As soon as they realized they weren't going to get to use their AI, they were >> like, I got a hundred. >> I'm done. >> And a zero. >> Yeah, they were they were out. Yeah, >> those are the people >> who you expel. Those people you those people could work in um business development. Maybe there's something they could do. There's there's a job for them. I don't know what it is, but >> Yeah. >> Yeah. I I mean I I do you know we've had a lot of people who make AI tutoring and education apps and and there's always all this sincerity like there are definite ways to use this to learn and expand your horizons. I've learned a lot from interacting with uh AI bots but it is you do have to bring that desire to learn to it. You you can use it very lazily just give me the answer. You have to like bring that with you that I have desire to iterate and keep using this until I actually learn the material yourself. >> Uh the only thing you really need to know is that um you're only cheating yourself and everybody's cheating in school and the blue books have to come out. Everything needs to be in person. Pop quizzes, reading books in person. I literally went through this with my daughters this summer. They had to read a book and I said, "Here's how we're going to do it." I bought like two copies of the book. I said, "These are workbooks. They only cost like four or five bucks." I lied to them. [laughter] Uh, and I said, "It's okay to." Because they ethically didn't want to write up the books. Sure, I get it. >> But these were like paperbacks. And I was like, "Girls, these are paperback workbooks. >> You circle any word you don't know, and I want you to underline the most important sentence on every page in your estimation. And at the end of the chapter, I want you to write a summary on the in the book of it." And now we read the book. Now retention's gone over. And we call this um uh hand thinking or hand speed. I would talked about it on a previous episode. >> Hand mind. Hand mind. That you you borrowed it from Ursula Kin. >> Yeah. So hand mind is your mind is going only as fast as your mind. Which is why when I'm sitting here on the podcast, this book is here because I like to slowly write one word. Like I wrote one word, Jasonbot, while we were talking and I said, you know, we should have a bot on Twitter that goes through the show in real time. Like we were having our real-time commenter and just Jasonbot says what we're talking about or post quotes in real time. So if you weren't watching the show, you could just hear quotes from there. >> Um, and >> good work. >> I I won't forget it. >> So there you have it, folks. >> There you have it. >> Anything else on the dock that we should do? Lightning round. >> Uh, yeah. Well, I do I want to mention uh we'll do one more story, but before we get to that, one thing I do want to talk about just so we can clip it and share it with everybody. We are hiring for the podcast. We have many open