AI literacy in programming education: Two classroom studies

AI literacy in programming education: What two classroom studies reveal about using AI well

A restricted AI tool can fail to outperform unrestricted access, while a particular tutoring platform can produce a positive short-term result. Those findings may sound contradictory, but they answer different questions.

For instructors and students in programming, data analytics, and related courses, the practical question is not only whether AI is allowed. It is whether students can decide when to use it, check what it produces, explain the reasoning behind their work, and continue learning instead of handing the task over to a tool.

Two classroom studies published earlier this week offer a useful comparison. One tested a guided AI Study Coach against students’ own unrestricted AI tools in a master’s-level data-analytics course. The other tested Medly, an AI tutoring platform, against self-directed revision in GCSE science. Both are preprints, not settled evidence. Their subjects, age groups, interventions, and outcomes differ, but together they show why AI literacy in programming education cannot be reduced to choosing between “AI on” and “AI off.”

What the programming study actually tested

The programming study was a seven-week mixed-methods pilot in a master’s-level data-analytics course. The researchers randomly assigned 33 students to one of two conditions: a scaffolded AI Study Coach embedded in notebook-based laboratory sessions, or unrestricted use of AI tools selected by the students (arXiv).

The Coach was deliberately limited. It offered step-by-step hints rather than generating code, capped the number of hints available in a session, and required a short reflection at the end. The design rested on a reasonable educational expectation: guided help might build confidence while reducing over-reliance, giving the scaffolded group an advantage.

That expectation was not supported by assignment results. Performance did not differ between the Coach group and the unrestricted-use group (arXiv). This does not show that restrictions are useless in every programming course. It shows that, in this small pilot, restricting the tool did not create a measurable assignment advantage over allowing students to choose their own AI tools.

The student interviews add useful detail. Coach users reported greater confidence, but they struggled to manage the limited hint allowance. Students with unrestricted access were satisfied with the choices they made, yet expressed concern about how dependent they had become on AI (arXiv).

The strongest pattern was not tied neatly to either tool. Students who had developed personal rules for when to use AI performed better in both conditions. The paper also reports that students with the strongest understanding of how the models worked, all of whom were self-taught in this sample, used AI more deliberately and earned the highest scores (arXiv).

That is an observed relationship, not proof that personal rules caused higher scores. The sample was small, and the study was a pilot. Still, the result gives instructors a concrete point of focus: access rules may shape the learning environment, but students also need practice making decisions about the tool.

Guided versus unrestricted generative AI in programming education

The programming study does not establish that open access is better. It raises a different possibility: the design of the tool may matter less than the learner’s capacity to govern its use.

In interviews, students in both conditions identified awareness of their own reliance on AI as the most valuable outcome of the course (arXiv). That was a report about what interviewed students valued, not a direct measure of independent mastery. Even so, it points toward a skill that can be taught.

Students can be asked to pause at several points during a programming task:

• What have I tried before asking an AI tool for help?
• Am I requesting a hint, or am I asking the tool to complete the task?
• Can I explain why this code change works?
• What test case could expose an error?
• Which parts of the response have been checked against course materials or documentation?
• What does the course require me to disclose?

A restricted tool may make some of those decisions more visible. An unrestricted tool may better resemble the environment students encounter elsewhere. Neither arrangement automatically teaches judgment.

The separate Medly study illustrates why broad claims about AI tools are difficult to defend. It was a four-week, multisite randomized evaluation in GCSE Biology, Chemistry, and Physics classes in English secondary schools. Students using Medly were compared with students completing business-as-usual, self-directed revision (arXiv).

Of the 929 students who completed the baseline assessment, 644 completed post-testing (arXiv). In the study’s primary assignment-based analysis, students allocated to Medly had higher post-test attainment than students in the self-directed revision group. The reported standardized difference was Hedges’ g = 0.33, with a 95 percent confidence interval from 0.18 to 0.48 (arXiv).

Positive estimates appeared by subject as well: g = 0.31 in Physics, g = 0.32 in Chemistry, and g = 0.52 in Biology (arXiv). These are study results, not promises of similar gains in every science classroom or programming course. The comparison was with self-directed revision, not with every possible teacher-led lesson, tutoring model, or non-AI intervention.

The studies therefore cannot be treated as a simple verdict on AI. One examined master’s students completing programming and data-analytics assignments with different forms of AI access. The other examined secondary students using a specific tutoring platform for GCSE science revision. Their students, subjects, interventions, comparison groups, and outcome measures were different.

The Medly researchers describe their findings as preliminary. Attrition was 30.7 percent, the outcomes were curriculum-aligned rather than standardized, and the response to the process evaluation was limited (arXiv). Greater platform engagement was associated with higher attainment, but the researchers treat that post-randomization relationship as exploratory rather than causal (arXiv).

The careful reading is narrower than “AI works.” A particular platform was associated with higher post-test scores in a particular four-week setting, while a particular restricted Coach did not outperform unrestricted tools in a small programming pilot.

Why student reliance belongs in the lesson plan

The programming paper’s most useful contribution may be its focus on self-government. Students are already making choices about when to ask for help, how much help to request, and whether to verify the response. A course that addresses only permission and prohibition leaves those decisions largely unexamined.

That concern appears beyond programming. A RAND report published six months ago drew on a December 2025 survey of 1,214 young people ages 12 to 29. It found that increasing shares of middle school, high school, and college students reported using AI for homework during 2025. The report also described substantial ambiguity about how students should navigate AI use at school, alongside concern about its effect on critical thinking (RAND American Youth Panel).

The findings support explicit guidance, but they do not show that every student lacks instruction. They show uncertainty across a broad group of learners. For a programming class, that uncertainty can be addressed through small, repeated decisions rather than a single policy statement.

School and district guidance may not be consistent either. A RAND report published two years ago drew on a nationally representative survey of 1,020 teachers and 231 districts, as well as interviews with 11 district leaders. Teachers reported how they used AI, while district leaders reported whether and how their districts provided policies, guidance, and training (RAND).

For students and instructors, the practical implication is that an AI rule may come from several levels: the assignment, the instructor, the department, the institution, or the district. A general policy may not answer whether code generation, debugging assistance, brainstorming, or paraphrasing is allowed on a particular task.

Higher education adds another perspective. A survey of 1,217 participants across 76 countries found high familiarity with generative AI, prior use, and an intention to continue using it, especially for information retrieval and text paraphrasing (International Journal of Educational Technology in Higher Education). The study also found correlations between cultural dimensions and respondents’ views about benefits, academic dishonesty, and the need for ethical guidance (International Journal of Educational Technology in Higher Education).

That correlation does not show that culture alone explains those views. It does show why one universal policy may not fit every class, institution, or student group. Teaching students to use AI responsibly requires clear course expectations and room to discuss how those expectations apply.

How to assess AI-assisted student work

The programming researchers recommend assessment that grades the reasoning behind AI-assisted work, along with explicit instruction in AI literacy as a core skill (arXiv). That recommendation becomes more useful when translated into assignment design.

An instructor does not need to use every option in every course. The right choice depends on the learning objective, the course policy, and the time available for feedback. Possible evidence of learning includes:

• An initial attempt. Ask students to submit a first solution or brief plan before using an AI tool, when the assignment permits AI assistance.
• A revision record. Students can identify what the tool suggested, what they accepted or rejected, and why.
• An explanation of a code change. Require a short paragraph, annotation, or recorded explanation describing what a suggested change does and why it belongs in the program.
• Student-designed tests. Ask students to create test cases, including cases that could expose an incorrect assumption.
• Error analysis. Have students identify one incorrect, incomplete, or unsuitable part of an AI response and explain how they found the problem.
• Disclosure. If AI use is allowed, specify what students should report, such as the tool used, the purpose of the interaction, and the parts of the work affected.
• An unaided explanation. A brief explanation or no-tool exercise can ask students to describe an algorithm, trace a function, or make a small modification without AI assistance, when that fits the course goals and accessibility arrangements.

These are recommendations for showing process, not universal requirements. One course might allow AI for brainstorming but not code generation. Another might permit generated code if the student tests, explains, and discloses it. A foundational assessment might prohibit AI because its purpose is to measure unaided programming knowledge. Students should check the assignment prompt and ask the instructor before assuming that a general school policy answers the question.

A 2024 education volume argues for shifting attention from learning as a finished output toward learning as a process. It presents generative AI as a course and program design issue involving teaching, assessment, and institutional planning, rather than as evidence from a randomized classroom experiment (SpringerBriefs in Education).

That distinction matters. If the final code is the only evidence collected, an instructor may not know whether the student can explain it, test it, or modify it. Process-focused assessment does not make AI use automatically acceptable. It makes the learning target more visible.

How to evaluate AI tools before adopting them

A vendor demonstration or one positive study is not enough to establish that a tool belongs in a programming course. Instructors and programs can begin with a focused review:

• What subject and age group did the evidence study?
• Was the comparison group doing nothing, revising independently, or receiving another form of instruction?
• Was the outcome an assignment, a curriculum-aligned test, a standardized measure, or student confidence?
• How long did the intervention last?
• What did students have to do, and what did the tool do for them?
• Were students asked to explain, test, revise, or disclose their work?
• What limitations did the researchers report?
• Does the proposed classroom use match the evidence?

The Medly researchers propose teacher-led micro-randomized controlled trials, or micro-RCTs, as one response to the speed of technological change. Small, repeatable randomized comparisons can generate evidence that is replicated and updated as tools and classroom implementation change. The researchers do not present them as replacements for definitive evaluation. They describe them as part of a cumulative approach to evidence (arXiv).

For a programming instructor, a modest classroom comparison might examine two approved ways of using an AI hint system on similar practice tasks, provided the design is appropriate and the course policy allows it. The instructor could compare finished answers with explanations, testing decisions, and student reflections. Any such activity should follow applicable institutional review or permission requirements. The central point is straightforward: evaluate the tool’s effect on the learning goal, not just its popularity or fluent output.

What students and instructors should ask next

The current evidence does not support a blanket claim that AI helps or harms learning. The Medly preprint reports a preliminary, short-term result over self-directed revision in GCSE science. The programming preprint reports no assignment-performance advantage for a restricted Coach over unrestricted AI access in a small master’s-level pilot. Neither study establishes a general rule for every classroom.

The programming study does offer a focused argument for AI literacy in programming education. Its findings connect deliberate use, understanding of how models work, and personal rules for AI use with stronger performance within the pilot. That interpretation deserves attention, while the study’s small sample and preliminary status remain important limits.

Before using AI on an assignment, students should ask:

• Is AI permitted for this task?
• What kind of assistance is allowed?
• What must be disclosed?
• Will the reasoning, testing, or revision process be assessed?
• Can the final work be explained without relying on the tool?

Before adopting a tool, instructors and programs should ask:

• What evidence exists for this specific tool and subject?
• What did the comparison group do?
• What outcome was measured?
• What limitations affect the result?
• How will students demonstrate learning beyond the tool’s output?

The useful question is not simply, “Is AI allowed?” It is, “What evidence will show that learning happened?” Students should check the assignment rules and ask for clarification before using a tool. Instructors and programs should match adoption decisions to evidence from the same subject, learner group, and learning goal, then assess the process as carefully as the finished code.