Ai2's TutorMoments turns the 'helpful assistant' default into a scorable bug
Every URL the pipeline pulled into ranking for this issue — primary sources plus the supporting and contradicting findings each Researcher returned. Inline citations in the issue point back here.
Sources
TutorMoments: Do AI tutors know when to help and when to hold back? huggingface.co
References
Wharton (Knowledge@Wharton) knowledge.wharton.upenn.edu
Students with on-demand AI assistance in complex tasks (e.g., chess) achieved less than half the performance gains of those using controlled, restricted assistance.
ETH Zürich research collection (MathTutorBench / benchmark survey) research-collection.ethz.ch
A model’s ability to solve a math problem does not correlate with its ability to teach it; subject expertise and pedagogical quality often form a trade-off.
GlobalSociety.earth report on Khanmigo deployment globalsociety.earth
A study involving 69 undergraduate physics students found that while Khanmigo provided superior step-by-step guidance compared to Google searches, there were no statistically significant differences in learning outcomes between the groups.
ResearchGate: ‘Towards Valid Student Simulation with Large Language Models’ researchgate.net
Simple prompting strategies for student simulation frequently result in ‘hallucinated’ learning, where a model claims to understand a concept without a corresponding shift in its underlying epistemic state.
Allen Institute for AI blog (TutorMoments release) allenai.org
The dataset contains fewer successful human rigor moments (260) compared to scaffolding moments (738), and the scoring pipeline for ‘rigor pushes’ is currently less reliable than its scaffolding counterpart.
arXiv 2606.15766 (MathDial / tutoring dialogue dataset) arxiv.org
MathDial is specifically designed to address the ‘tell-tale’ problem, where models often reveal answers too early rather than guiding students through reasoning.
