Does artificial intelligence help students learn? It’s a question many teachers and parents want to answer as new educational technologies quickly spread into classrooms.
Normally, meta-analyses would be the right place to look, because they are supposed to bring together a scattered body of research and make sense of it by indicating where the preponderance of evidence lies. And indeed, there are 39, all reporting substantial benefits for AI in education.
But it appears that the studies synthesizing this evidence are themselves unreliable – and the peer review process has failed to identify problems.
“This is not trustworthy evidence,” said Justin Reich, an associate professor at the Massachusetts Institute of Technology and director of MIT’s Teaching Systems Lab, who organized a webinar on October 20 to discuss the problem of the growing number of positive meta-analyses on AI. “All these meta-analyses are wrong.”
A forensic verification of 14 of 39 meta-analyses revealed serious conceptual and statistical flaws. Appearing in journals from leading publishers including Springer Nature, Wiley, SAGE, and Taylor & Francis, these 14 articles have accumulated more than 3,000 citations and continue to accumulate 200 citations per month, according to Google Scholar, implying that the effectiveness of AI is established science. A meta-analysis was retractedalmost a year after its publication in May 2025 in a Nature journal after two Norwegian researchers filed a complaint.
The new audit was led by Patrick O’Neill, a community college professor at Ivy Tech’s campus in Bloomington, Indiana, who examined a sample of the underlying studies and the statistical methods used to combine their findings in meta-analyses. O’Neill discovered recurring patterns of errors, some of which he said could have easily been detected by simply reading the study summaries.
During the October 20 webinar, O’Neill described a constellation of “impossible statistics,” “crazy references” and “exaggerated conclusions.” Some of the underlying studies even cite references to papers published on generative AI before the technology was made public. Links to citations sometimes led to unrelated studies.
Many pooled studies didn’t even measure the extent to which AI stimulated learning. For example, nursing students who studied three hours a week with an instructor using AI were compared to their peers who had no study sessions (of course, they tested better). Students who played video games and answered AI-generated questions outperformed those who did neither. In another experiment, students who chatted with an AI robot reported feeling less anxious.
Beth Tipton, a meta-analysis expert and professor at Northwestern University who joined colleagues Reich and O’Neill in the critique, called the idea of conducting a very broad meta-analysis encompassing all the various uses of AI in education “crazy.”
Tipton compared it to a meta-analysis of mathematics education. Is mathematics teaching effective? It depends on how mathematics is taught. A thoughtful meta-analysis could narrow the question to a specific approach, such as whether spatial learning interventions improve math skills. Likewise, Tipton said, how AI is used should be specified before conducting a meta-analysis of it.
“Do we mean people using apps and (educational) programs that include AI, like Khan Academy? Do we mean giving students access to Claude?” asked Tipton.
A big concern in the current batch of AI meta-analyses, O’Neill said, is “publication bias.” It found that two-thirds of calls for papers on AI in education specifically called for research into its benefits, “apparently discouraging null or negative findings.” This means that meta-analyses effectively “manufacture evidence” by grouping together selected, positive findings, he said.
Rethink the numbers
O’Neill is currently conducting his own meta-analysis of AI, using appropriate statistical methods, and arrives at a much less optimistic picture. He found that the results of 44 well-designed experiments varied widely, from very helpful to very harmful for students. The results averaged about zero, according to O’Neill’s preliminary calculations. This does not mean that educators should conclude that AI is on average harmless. It depends on how it is used.
Susanna Loeb, a Stanford education professor who also participated in the meta-analysis critique webinar, pointed out that her team collected nearly 900 studies on AI in education and determined that only 20 of them were of high enough quality to be included in a recent study. summary of evidence for her AI Center for Education. His team has since identified 32 additional studies and will update this summary soon. But, she says, it is still too early to conduct a meta-analysis that would be useful to school leaders or teachers. Very little high-quality education takes place in elementary and secondary schools in the United States.
Northwestern’s Tipton worries that the rapidly growing field of meta-analysis has outgrown the number of qualified experts to review the work. More and more researchers are performing meta-analyses because they are quicker and less expensive than conducting experiments, and they can attract many citations for ambitious professors. The thirst for evidence on AI has made this topic a particular magnet for meta-analysis and given rise to a sort of cottage industry of AI meta-analysis.
Tipton said she receives requests to review meta-analysis articles every day, but she can’t accept them all. Verifying the underlying studies can take days, and peer review is unpaid volunteer work.
Medicine has research flaws of its own, Tipton said, but it has gatekeepers to help separate the wheat from the chaff. The Food and Drug Administration evaluates evidence before approving drugs, and insurers review research when deciding which treatments to cover. Education has no comparable guarantees.
A “no man’s land”
“In education, it’s kind of a no-man’s land,” Tipton said. “Anyone can sort of say they have evidence of anything and share it with anyone, and consumers, schools, teachers, parents have to try to wade through that and make sense of it. It’s incredibly frustrating.”
The researchers turned to philanthropic funders and the National Science Foundation to help support rigorous and reliable research syntheses. “Maybe our voluntary peer review model doesn’t work very well for landmark studies” on critical education topics, Reich said. “Peer review doesn’t work. Meta-analysis doesn’t work.”
Until truly reliable evidence emerges, researchers have advised teachers to cautiously experiment with AI, starting with small groups of students and monitoring the results. Even introducing AI into a lecture course of 300 students could be risky, O’Neill said.
In other words, teachers are left alone to experiment in their own classrooms. It is nonetheless encouraging to see education researchers denouncing flaws in their own field and attempting to correct them.
Contact the editorJill Barshayat 212-678-3595, jillbarshay.35 on Signal, or barshay@hechingerreport.org.
This story on the meta-analysis of AI in education was produced by The Hechinger reportan independent, nonprofit news organization that covers education. Register for Proof points and others Hechinger Newsletters.
The article 39 Meta-Analyses, 39 Positive Findings: Audit Reveals Serious Gaps in AI in Education Research appeared first in the Hechinger Report.
