Shortly after ChatGPT was introduced in November 2022, students figured out how to use the chatbot as an answering machine, allowing it to do their homework for them. Not surprisingly, this shortcut has harmed student learning and led to lower test scores.
AI enthusiasts were looking for a solution. Chatbots could easily be tasked with acting more like a human tutor, remembering answers, providing guidance, and guiding students with their questions. Khan Academy created one of these “Socratic method” chatbots, named it “Khanmigo” and launched it in 2023.
But a new study published last month suggests that creating an AI tutor is just one obstacle in helping children improve their math skills. Students have to actually want to use it.
Researchers at the University of Toronto followed students at 18 Tennessee colleges for two years, from 2024 to 2026. Almost all of the students tried Khanmigo first. And many tried to convince the AI assistant to spit out the answer. But when Khanmigo refused and instead asked questions or offered clues, students largely stopped using it.
“We observe how students actually used the AI tutor, and the answer is: not much,” Philip Oreopoulos and Nina Low, the researchers, wrote in their report. draft paperreleased by the National Bureau of Economic Research in August. “Asking for help to resolve one’s own confusion remained a choice, and most students refused it most of the time. »
The researchers were testing whether Khanmigo could improve the performance of low-performing students who were at least one level behind their peers. These students benefited from an additional period of mathematics catch-up integrated into their school day. Some were randomly assigned to use Khan Academy, with Khanmigo available as an AI assistant. The rest received their usual remedial math assignments, which often included online tools such as Waggle, IXL, Zearn, and DeltaMath.
Students who benefited from the Khan Academy intervention performed better in mathematics than those in regular remedial classes. But the gains were modest, and no larger than previous studies had found in Khan Academy workouts without an AI assistant.
In other words, Khan Academy seemed to help. But Khanmigo’s guardian added nothing.
Sal Khan, founder and CEO of Khan Academy, praised the study and wrote a detailed report. blog post about it, explaining the randomized controlled trial (RCT) the same way he explains a math problem in his videos. In an interview, Khan said the results confirmed what Khan Academy was seeing internally: Students weren’t engaging much with Khanmigo.
Khan said he did not regret releasing the AI tutor before knowing whether it would be effective. He is confident that the early release has “done no harm” to students, saying his organization has gone to great lengths to protect student data, secure chatbot conversations, prevent cheating, and minimize the risk of the bot hallucinating and giving incorrect instructions to students. “This allowed us to learn and hopefully make the new version even more useful,” he said.
The version of Khanmigo tested in Tennessee was released in 2023. Since then, Khan said, the company has changed how the AI assistant works. Khanmigo is now integrated directly into Khan Academy rather than being in a separate tab for students to click through. If students ask for help before solving a problem, Khanmigo is deliberately taciturn, offering little more than a hint and a pep talk. But when a student gets a problem wrong, the AI assistant automatically appears to offer help and solve the problem with the student. Schools can also choose to disable the AI assistant altogether.
Khan Academy is experimenting with other ways to encourage students to use the AI Tutor. Khan said he plans to give students “credit” for using Khanmigo after a mistake, allowing a robot-assisted recast to count as one of the five consecutive correct answers students need before moving on to the next skill.
It will be interesting to see if these changes will encourage students to use Khanmigo more. But usage and engagement are only one piece of the puzzle. We still need evidence that when students use an AI tutor regularly, they actually learn more. I’ll look.
Contact the editor Jill Barshay at 212-678-3595, jillbarshay.35 on Signal, or barshay@hechingerreport.org.
This story about Khanmigo, an AI mathematics teacher, was produced by The Hechinger reportan independent, nonprofit news organization that covers education. Register for Proof points and others Hechinger Newsletters.
The message The students did not get answers from Khanmigo. They also didn’t want his questions to appear first in the Hechinger report.
