AI tutoring effectiveness in schools: What the evidence shows
Nearly every student in a two-year school trial tried an AI tutor at least once. That broad access did not produce a clear advantage over Khan Academy practice without AI assistance. The result should make schools more careful about claims surrounding AI tutoring effectiveness in schools, especially when a new tool is presented as a replacement for established instruction.
Generative AI has been promoted as a way to give every student a personal tutor. NBER described that promise two months ago. Federal policy has also made room for AI-enhanced tutoring and personalized learning. Guidance issued last year by the U.S. Department of Education says those uses may be supported with federal grant funds when they meet applicable legal and regulatory requirements.
The promise is moving faster than the proof. A review published two months ago by IES found only 20 rigorous education studies that produced causal evidence about AI’s effects. That does not mean AI tutoring has no value. It means schools should adopt it for a defined instructional purpose, test whether it improves learning, and stop treating access as evidence of effectiveness.
Does AI tutoring improve student learning? The Khanmigo school experiment
One of the strongest available real-world tests comes from a two-year cluster-randomized trial in 18 Tennessee middle schools. In the study by Philip Oreopoulos and Nina Low, randomly assigned students used Khan Academy with Khanmigo during existing daily remedial mathematics sessions. The tutor was configured to coach students rather than simply give answers (Oreopoulos and Low).
The study found that the assignment increased math achievement by about 1.3 national percentile ranks per term, or approximately 0.06 to 0.08 standard deviations over a school year (NBER). The implied effect of a full year of active participation reached 0.14 standard deviations, but that figure is an estimate based on active participation, not the observed result for every student (NBER).
The comparison that matters most is not between an AI tutor and no instruction. The gains resembled those from Khan Academy practice without AI assistance (NBER). That suggests the AI layer added little measurable advantage in this setting, even though the overall program produced positive gains.
This is not evidence that Khanmigo failed, nor does it settle the question for every subject, age group, or tutoring design. It does challenge a simpler sales pitch: adding a chatbot to learning software does not automatically make the software more effective. The instructional task, the way students use the tool, and the teacher’s role still matter.
Student engagement is the missing part of AI tutor effectiveness
The Khanmigo trial makes the difference between availability and use unusually clear. About 96% of students tried Khanmigo at least once, but the median student messaged it on only about one-third of the days they practiced (NBER). Students used the tutor in only 17% of exercise sessions in which they made a mistake (NBER), a moment when coaching might have helped.
Even when students opened the conversation, the exchanges were usually not extended mathematical discussions. Messages were mostly bare answers or clicks on suggested prompts (NBER). A student who asks for an answer, copies the response, and moves on has interacted with AI, but has not necessarily practiced the reasoning the lesson is meant to develop.
The researchers’ interpretation is appropriately cautious: engagement appears to be the binding constraint. Realizing the promise of AI tutoring requires meaningful use, not just giving students accounts (NBER).
That finding gives teachers something concrete to test during a pilot:
- Do students explain what they tried before requesting help?
- Does the tutor respond with a hint, question, or strategy instead of completing the problem?
- Do students revise their work after the exchange?
- Can students solve a similar problem without the tool?
- Are students using the tutor when they are confused, or only clicking through suggested prompts?
Those questions turn “student engagement with AI tutors” from a dashboard metric into an instructional question. A login count does not show whether learning occurred. A satisfaction survey does not show whether students can transfer the skill to independent work.
Personalization, in practical terms, is not delivered by credentials alone. It requires a routine that connects the tool to a learning goal. A teacher might ask students to write a prediction before opening the tutor, record the hint they received, and complete a similar problem without assistance. That process keeps the student’s reasoning visible and gives the teacher something to evaluate.
What broader generative AI research does and does not show
The wider research base is useful, but it should not be confused with direct evidence about school-based AI tutors. The IES review covers different ages, tools, subjects, learning settings, and outcomes. Most of the research was conducted in postsecondary settings, with causal studies more common in high school than in middle or elementary school settings (IES).
Several patterns are still relevant. Teacher-mediated and AI-augmented tutoring may promote learning when the system gives educators diagnostic information or suggests tailored instructional support. Some tools helped identify individual learning patterns, while others offered real-time instructional suggestions associated with greater mastery (IES).
Student-facing tools have produced mixed results. Simply giving students access to an AI tool did not improve exam scores in the studies summarized by IES, though it did improve homework performance (IES). Tutoring-specific chatbots that provided hints without giving away answers performed as well as traditional study methods on an exam, but not better (IES).
That distinction matters. Completing more homework is not the same as understanding more mathematics, remembering content, or applying a skill independently.
The caution becomes stronger with general-purpose chatbots. Students who used tools such as ChatGPT or Claude while studying were more likely to show lower exam performance, shallower learning processes, and weaker recall, even when they believed the tools were helpful (IES). In a separate research task, students who used a traditional search engine produced stronger reasoning and argumentation than students who used a general-purpose chatbot (IES).
That task was not a tutoring study, but it illustrates the underlying concern. A general chatbot may produce a polished response while reducing the student’s need to search, compare, question, and construct an argument. A tutoring tool is different when it is specifically designed to provide feedback or hints and is integrated into teacher-led instruction. Schools should not treat the two categories as interchangeable.
The fair position is not that AI never helps. The evidence points toward better results when tools are purposefully designed, instructionally integrated, and used to support rather than replace student thinking (IES). IES also cautions that AI is advancing faster than research can evaluate it, which makes careful local testing more useful than broad promises (IES).
Why policy cannot substitute for evidence
Last year, the U.S. Department of Education issued guidance describing how AI may be used across educational functions, including AI-enhanced high-impact tutoring and personalized learning (U.S. Department of Education). That guidance makes certain uses allowable under applicable federal programs. It does not prove that those uses improve student learning.
IES’s recommendations are narrower: use AI to support, not replace, student thinking; center human relationships in teaching; and vet tools with learning, privacy, and access in mind (IES). Those principles give schools a better standard than adoption for its own sake.
Before adopting or renewing an AI tutor, administrators and teachers should ask:
- What specific learning problem is the tool meant to solve? “Personalization” is too broad.
- When should students use it? A planned revision step serves a different purpose from an unrestricted chatbot.
- What cognitive work stays with the student? The tool should prompt explanation, comparison, revision, or reflection.
- What does the teacher do differently? A tool has a clearer role when it provides useful information for instruction.
- What outcome will be measured? Look for independent work, assessment results, retention, or transfer, not just logins or completed tasks.
- Was there a meaningful comparison group? Compare the tool with a similar amount of instructional time and a clearly described alternative.
- How are privacy, access, and oversight handled? These questions should be addressed before students begin using the system (IES; U.S. Department of Education).
Teacher training should be treated as a practical condition of a pilot, not an optional add-on. Teachers need time to define acceptable use, model productive questions, inspect student work, and decide when the tool should be removed from the task.
Parents can ask whether the school is using a tutoring-specific system or a general chatbot, what the tool is designed to support, how student use is monitored, and how the school knows whether learning improved. Those are reasonable questions while the evidence base remains limited.
Schools do not need to ban AI tutoring. They do need to stop mistaking novelty, access, or faster task completion for learning. The responsible standard is narrower: choose a clear instructional purpose, preserve student thinking, involve teachers and families, measure independent outcomes, and expand use only when local evidence earns that next step.