AI tutoring study: What math mistake trials show

AI tutoring study: What math mistake trials show

A large AI tutoring study involving more than 6,000 middle school students in Hamilton County Schools found a result that is easy to oversimplify. Students using AI support moved more slowly and attempted fewer questions than students using standard computer-assisted learning. When they did attempt a question, however, they answered more accurately, especially after making a mistake. NBER reported the study two months ago.

That distinction matters. Getting the next problem right is not the same as remembering the skill a week later, and access to a tutor is not the same as using it well. Other randomized trials point in a similar direction, but they tested different tools, forms of feedback, age groups, and outcomes.

For students, parents, and teachers, the practical question is not simply whether AI tutoring “works.” It is: What kind of support was tested, compared with what, and did it improve immediate correctness, next-attempt success, delayed learning, or broader math achievement?

How to read the AI tutoring evidence

The studies discussed here measure several different outcomes:

• Immediate correctness: Whether a student fixes the answer currently on the screen.
• Next-attempt success: Whether the student answers correctly after receiving help with a mistake.
• Delayed learning: Whether the student performs better on an assessment later, including on material that was or was not practiced.
• Broader achievement: Whether assignment to an AI-supported program changes math performance over a term or school year.

These outcomes answer different questions. A helpful explanation after an error may improve the next attempt without producing a durable change in achievement. A practice program may raise test performance even if the AI adds little beyond the practice work itself.

The interventions also differ. NUMI tested AI support inside a computer-assisted learning platform, with students separately randomized to AI or computer-assisted-learning-only support and to mastery or non-mastery progression. Khanmigo was used during existing remedial math sessions and configured to coach students rather than provide answers. The LearnLM trial placed an expert tutor between the AI system and every student message.

Those differences are not technical footnotes. They define what the research can and cannot tell schools about a general-purpose chatbot.

What happens immediately after a student makes a mistake?

The clearest pattern concerns mistake remediation. In the NUMI experiment, AI support improved correctness on a student’s next attempt after an error and reduced the number of attempts needed to return to a correct answer. Students also spent more time on each question that received structured support. NBER described these findings two months ago.

The result is conditional, though. AI-supported students attempted fewer questions overall and progressed more slowly. “More accurate once they attempted” does not mean “completed more work” or “learned more overall.” It means the students who reached an attempt were more likely to answer correctly.

The UK study points in the same direction, while adding an important condition: human oversight. The exploratory randomized controlled trial included 165 students across five UK secondary schools. Students receiving supervised LearnLM tutoring had a 93.0% success rate on mistake remediation, compared with 91.2% for human tutors and 65.4% for static hints. alphaXiv summarized the Google and Eedi study last year.

Across these studies, the evidence supports a narrow conclusion. Interactive or structured feedback can be useful at the point when a student is stuck. It does not establish that every AI tutor improves learning broadly, or that a chatbot can replace a teacher or tutor.

Why mastery-based AI tutoring needs more than access

A mastery-based system generally asks students to demonstrate repeated success before moving on. That structure can make practice more deliberate, but it does not ensure that students will use the available help.

The NUMI study found that mastery progression increased the number of students reaching three correct answers in a row, but mastery by itself did not improve delayed learning. The most encouraging delayed-assessment result appeared when AI was embedded in the mastery workflow, with marginally significant gains concentrated on practiced Exercise 1 material. Students completed the delayed assessment one week after practice, and it included both practiced and unpracticed material. NBER reported the design and findings two months ago.

That result is suggestive, not a blanket endorsement of mastery-based AI tutoring. The study shows that the combination produced its most promising delayed result under a specific condition. It does not show that pushing students through a mastery sequence caused durable learning in every topic or situation.

The two-year Khanmigo trial makes the engagement problem more visible. In 18 Tennessee middle schools, students were randomly assigned to use Khan Academy with Khanmigo during existing daily remedial mathematics sessions. Khanmigo was configured to coach rather than give answers. NBER reported the trial two months ago.

Ninety-six percent of students tried Khanmigo at least once, but the median student messaged the tutor on only one-third of practice days. Students used it in just 17% of exercise sessions in which they made a mistake. Messages were usually bare answers or clicks on suggested prompts rather than substantive mathematical dialogue. NBER identified limited and shallow engagement as a central problem.

This creates a practical distinction between availability and productive use. A school can provide access to an AI tutor without ensuring that students pause after an error, explain their thinking, read the feedback, and try a related problem. The Khanmigo study authors describe engagement as the binding constraint. That is their interpretation of this trial, not a universal rule about every AI tutoring product.

For a teacher or parent, the useful question is not only whether a tool is available. It is whether the learning routine gives students a reason to use it at the right moment and requires them to do something with the explanation.

AI math tutor study results: real gains, different scales

The Khanmigo results show why assignment effects and active-use effects should not be confused. Assignment to Khanmigo raised math achievement by 1.3 national percentile ranks per term, equivalent to about 0.06 to 0.08 standard deviations over a school year. The implied effect of a full year of active participation reached 0.14 standard deviations. NBER reported these estimates two months ago.

The full-year figure is an implied effect of active participation, not necessarily the same as the randomized effect of assigning students to the program. The study also found that the gains resembled those from Khan Academy practice without AI assistance. That comparison complicates any claim that the chatbot itself was responsible for a distinctive achievement boost.

The NUMI trial measured a different kind of learning. Its delayed-test evidence was limited to the most favorable combination of AI and mastery-based practice, and the gains were concentrated on material students had practiced. That is narrower than a broad improvement in math achievement. NBER described the limitation two months ago.

A preliminary Ghana study offers a much larger result, but under very different conditions. Approximately 500 students received access to Rori, an AI math tutor used through WhatsApp. Students in the treatment group had a phone for one hour each week during study hall, while both treatment and control groups continued their normal classes with the same curriculum and classroom hours. The study reported an effect size of 0.36 and a marginal cost of approximately $5 per student. Stanford SCALE described the findings in 2024.

That result should not be placed beside a U.S. middle school trial as though the programs were interchangeable. The Ghana intervention involved a phone, a defined weekly study period, a WhatsApp-based tutor, and a particular school setting. The researchers also cautioned that the evidence covered only the first year of the intervention. The result is promising, but transfer to another country, curriculum, device environment, or age group would need to be tested.

Taken together, the AI tutoring learning outcomes are mixed in a useful way. Short-term correction is the most consistent benefit. Broader achievement gains are smaller or comparable to non-AI practice in one large school trial. Longer-term retention remains less settled.

What the study designs actually tested

Calling all of these tools “AI tutors” hides the main differences.

• NUMI: More than 6,000 middle school students used a computer-assisted learning platform. The experiment varied AI support, mastery progression, and math topic, making it possible to examine how instructional structure interacted with AI. The delayed assessment came one week after practice. NBER
• Khanmigo: Students used Khan Academy during existing remedial math sessions. The tutor was configured to coach rather than provide answers, but actual use was limited. The study followed students over two years and measured broader math achievement. NBER
• LearnLM and Eedi: Seventeen expert tutors reviewed every AI-drafted message before students saw it. Tutors approved 76.4% with no edits or only minimal edits. An audit of 3,617 messages found no harmful content and five factual errors. alphaXiv

The last example is especially important for interpreting safety claims. Students who received supervised LearnLM support performed at least as well as students chatting with human tutors on the measured outcomes. For novel problems, the LearnLM group had a 66.2% success rate, compared with 60.7% for human tutors alone, a difference of 5.5 percentage points. alphaXiv

Those results came from an exploratory study conducted by Google’s LearnLM team with Eedi, and every drafted message was reviewed before delivery. They should not be treated as evidence about an unsupervised, general-purpose chatbot. Human review was part of the intervention, not a side detail.

The same caution applies when comparing tools that use different kinds of feedback. A carefully checked message bank or a supervised tutoring system is not the same as unlimited conversation with a language model. The label “AI tutor” tells a reader very little about the actual learning experience.

Questions to ask before adopting an AI tutor

A school, teacher, or parent evaluating a tool can start with the evidence rather than the marketing language. Ask:

• Was the study randomized? Identify who received the AI intervention and what the comparison group received.
• Was use required or voluntary? A tool used during scheduled practice is being tested differently from one students may choose to open.
• What kind of feedback did students receive? Find out whether responses were open-ended, drawn from validated feedback, or reviewed by educators.
• Did humans review the AI output? The LearnLM safety findings came from a system with expert review of every message.
• What outcome improved? Separate current-answer correction from next-attempt success, delayed learning, and broader achievement.
• How often did students use the tool? A high percentage of students trying a product once does not show sustained engagement.
• Does the evidence match the intended setting? Check the age group, subject, curriculum, device access, language, and schedule.
• Was the measured benefit on practiced material or new material? A result limited to practiced exercises answers a different question from transfer to unfamiliar problems.
• What does the tool do after an error? The strongest recurring evidence here concerns support that helps a student reason through a mistake, not simply receive an answer.

The evidence does not justify a simple verdict for or against AI tutoring. It supports a more useful one: AI can help students respond to mistakes, especially when feedback is structured and targeted, but learning gains depend on what the system asks students to do, how often they use it, and how its output is checked.

Before relying on a tool, compare its research with the actual learning goal. If the goal is immediate correction, look for next-attempt evidence. If the goal is retention or achievement, look for delayed assessments and broader measures. Then verify the teacher’s or school’s rules for AI use and choose a system whose tested conditions resemble the students, subject, and support routine in which it will be used.