Khanmigo AI tutoring study results: Two studies differ

Khanmigo AI tutoring study results: Why two studies found different outcomes

AI tutoring can look successful while students are using it, then produce little improvement once the tool disappears. That tension sits at the center of the Khanmigo AI tutoring study results. Two working papers released three months ago by overlapping research teams examined different ways of using AI in middle school math. The Khanmigo trial found gains similar to Khan Academy practice without AI, while a related NUMI experiment found more promising, though still preliminary, evidence when AI support was tied to mistakes and a mastery-based workflow (NBER; NBER).

The difference matters to teachers, parents, tutors, and school leaders comparing AI math tools. “AI tutor” might mean a coach that offers hints, a chatbot that supplies answers, or a feedback tool that appears at a specific point in a required lesson. Those are different learning experiences. The studies do not establish one universal effect for AI tutoring, but they do offer a practical test: ask what students did with the tool, what they could do without it, and how the support fit into the work.

What the Khanmigo effectiveness study found

The Khanmigo effectiveness study followed more than 6,000 students in 18 Tennessee middle schools for two years. Students were randomly assigned to use Khan Academy with Khanmigo during existing daily remedial math sessions, with the AI tutor configured to coach rather than simply provide answers (NBER).

Assignment to the Khanmigo condition raised math achievement by about 1.3 national percentile ranks per term, equivalent to roughly 0.06 to 0.08 standard deviations over a school year (NBER). A standard deviation is a way to describe how spread out scores are. It should not be read as a direct grade increase. Here, it is a standardized measure researchers use to compare the size of an effect across assessments.

The paper also reports an implied effect of 0.14 standard deviations for a full year of active participation. That figure is an extrapolation from the study’s estimates, not a directly observed result for every student who used Khanmigo for a full year (NBER).

The more useful comparison is not whether students with access to Khanmigo improved at all. They did. The question is whether AI access produced more improvement than Khan Academy practice without AI assistance. The researchers report that the gains resembled those from the platform’s non-AI practice tools (NBER).

That makes the result more modest than a headline about AI raising achievement might suggest. It does not show that AI tutoring is useless. It shows that placing an AI coach in an existing remedial practice setting did not clearly outperform the platform’s regular practice experience in this study.

Why student engagement matters as much as access

The Khanmigo trial raises a practical question: Did students have enough meaningful interaction with the tutor for its coaching approach to affect learning?

Almost all students tried Khanmigo at least once, but that does not mean they used it consistently. Ninety-six percent used it at least once. The median student messaged Khanmigo on only one-third of practice days and in just 17% of exercise sessions in which the student made a mistake (NBER).

The messages were often limited, too. Students mostly entered bare answers or clicked suggested prompts instead of engaging in extended mathematical dialogue (NBER). That describes interaction behavior. It does not, by itself, prove that students lacked motivation, misunderstood the tutor, or learned nothing from a brief exchange.

The researchers describe engagement as the binding constraint. In practical terms, access alone may not be enough. A student who can ask for help but rarely does so, or who uses the tool to confirm an answer without examining the reasoning, is receiving a different intervention from a student who works through hints after an error (NBER).

For a classroom, that distinction changes what implementation questions deserve attention:

• When are students expected to use the tool?
• Does the system respond to an incorrect answer with a hint, an explanation, or a completed solution?
• Can a teacher see whether students are reading and applying the feedback?
• Is success measured while the tool is available, or after students must work independently?

The Khanmigo results therefore say as much about participation as they do about the technology. A tool cannot demonstrate its tutoring value if students rarely use the tutoring features in a substantive way.

How the NUMI experiment changed the learning situation

A related NBER working paper studied more than 6,000 middle school students in Hamilton County Schools using NUMI, a research-based computer-assisted learning platform. Students were randomized to AI or CAL-only support, mastery or non-mastery progression, and one of two math topics (NBER).

This created a different learning situation from the Khanmigo trial. NUMI was a mastery-based platform, and the study examined structured support around student errors. The design does not prove that NUMI is a superior model, but it helps show why a tool’s workflow matters.

Students who received AI support progressed more slowly and attempted fewer questions. They answered more accurately conditional on reaching an attempt, meaning accuracy was higher among the questions they actually tried (NBER). That does not necessarily mean they completed more work or achieved higher overall scores.

The clearest mechanism appeared after mistakes. AI support improved the likelihood that students would answer correctly on the next attempt, reduced the number of attempts needed to return to a correct answer, and increased time spent on questions receiving structured support (NBER).

One week later, students completed a delayed assessment covering practiced and unpracticed material (NBER). A delayed assessment asks whether learning remains after the practice session rather than measuring only what a student can do immediately with support.

The most encouraging delayed-test evidence appeared when AI was embedded in the mastery workflow, but the gains were marginally significant and concentrated on practiced material. Mastery progression by itself did not improve delayed learning (NBER). That is narrower than saying AI improved learning generally.

The NUMI findings suggest that structured AI help may add value over standard computer-assisted practice when it turns an error into a chance to reconsider the next step. They do not establish that every student learned more broadly, remembered the material longer, or benefited equally.

The comparison with Khanmigo remains useful. Khanmigo provided access during established remedial sessions, while NUMI tied support more closely to errors and a defined progression. Those differences in workflow and measurement may help explain why the results did not look the same.

What other AI tutoring studies add

A peer-reviewed randomized controlled trial published last year offers a sharper warning about assisted performance. The study took place at a large high school in Turkey and involved nearly 1,000 students across about 50 ninth-, 10th-, and 11th-grade classes. Students used either an unrestricted GPT-4 interface called GPT Base, a guarded version called GPT Tutor, or no generative AI during math practice. They then completed an exam without access to resources (PNAS).

During assisted practice, GPT Base raised performance by 48% relative to the no-AI control group, while GPT Tutor raised it by 127%. These were relative improvements in practice performance, not percentage-point increases in students’ overall math grades (PNAS).

The unassisted exam told a different story. Students in the GPT Base group performed 17% worse than students in the control group. The negative result was essentially eliminated in the GPT Tutor group, whose exam performance was statistically indistinguishable from the control group. The guarded tool still did not produce a positive advantage over no AI (PNAS).

The researchers suggest that students used the unrestricted system as a crutch. Reliability was also a concern: GPT Base gave a correct answer only 51% of the time, made logical errors 42% of the time, and made arithmetic errors 8% of the time (PNAS). The study cannot predict what every chatbot will do, but it demonstrates why a strong practice score is not enough.

A large ACM study published four months ago adds another distinction. Across two randomized trials involving 21,478 students working on 572 problems, AI-generated feedback improved short-term outcomes, especially for students with lower prior performance (ACM Learning @ Scale).

In the first trial, students receiving AI feedback were 16% more likely to correct their current answer and 7% more likely to succeed on the subsequent problem. The benefit was concentrated among students who had answered zero or one of their previous five problems correctly, while effects diminished for higher-performing students (ACM Learning @ Scale).

The second trial compared AI-generated feedback with expert teacher-written feedback and found comparable learning outcomes (ACM Learning @ Scale). That comparison is useful, but the study focused on short-term feedback outcomes. It does not settle whether students retained the skill weeks later or performed independently on a broader exam.

Taken together, these studies point to several design features rather than one master explanation. Hints may work differently from answers. Support embedded in a practice sequence may work differently from support students choose to open. Immediate correction may not equal delayed learning. A tool that helps students solve one problem may not help them solve the next problem alone.

How to judge an AI tutor’s evidence

Before accepting a vendor’s claim, identify which kind of performance the evidence measures.

What is measured What it can show What it cannot show by itself Assisted performance Whether students solve problems more successfully while AI is available Whether they can solve similar problems alone Immediate next-attempt performance Whether feedback helps students correct an error right away Whether the idea remains after practice Delayed learning Whether some learning remains after time has passed Whether the tool works across topics, students, or settings Unassisted exam performance Whether students can perform without the tool Whether the tool caused every difference in a single context

Feedback quality also needs checking. A PLOS One study found that ChatGPT-generated hints failed quality checks on 32% of problems. After researchers applied self-consistency, the failure rate fell to nearly 0% for algebra problems and 13% for statistics problems (PLOS One).

The same study found statistically significant learning gains for students who received ChatGPT hints, with results that were not statistically distinguishable from human-authored hints (PLOS One). That supports careful testing, not unsupervised use. Schools should ask who checks the content, how errors are corrected, and whether teachers can review representative exchanges.

The two NBER papers are working papers released three months ago. That makes them valuable early evidence, while also meaning readers should treat the findings as preliminary rather than settled conclusions (NBER; NBER).

Questions to ask before adopting an AI math tool

A teacher, parent, tutor, or school leader can use this research to ask three focused questions:

1. Does the tool give answers, or does it guide students toward answers with hints and explanations?
2. Is meaningful use built into the practice workflow, or is support available without a clear routine for using it?
3. Has the tool been evaluated both while AI is available and after it is removed?

Those questions cannot predict whether a product will work. They can show whether its evidence measures assisted performance, immediate correction, delayed learning, or independent achievement.

Most of the evidence considered here concerns middle- or high-school mathematics. It should not automatically be generalized to reading, science, college courses, younger learners, or adult education. Before choosing a tool, compare its design with the learning goal. If the goal is independent problem-solving, look for evidence that students can still solve problems after the chatbot is gone. That check separates a better-looking practice session from learning that lasts.