First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement
Second, trying to incorporate past grades into their modelling is not a substitute for a randomized trial.
Third, the headline engagement number of 90% is for "engaging with the platform, via Module Review or Lesson Quizzes, at least once". I don't know why much of that couldn't just be attributed to novelty. Or even partly a professor with all sorts of enthusiasm for the platform.
Fourth, the "full dosage" effectiveness is measured based the final exam scores. Were these exam questions produced independently from the "Phosphor" materials? (e.g. by blinding?) Were they checked for direct overlap with those materials? The 0.7 sigma shift is 3 points on a 24 point exam; if even a few of the questions on that exam were very similar to those materials it could account for almost all of it. This is not clear to me from the manuscript.
If this was the case, then it's a question less of "is AI effective" vs. "did the students look at the materials". You could still argue that the AI platform got them to read, but that is a somewhat different statement than the AI helped them learn.
Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again.
That means their experiment design is partially caused by their results instead of the other way around, which is a bad situation to be in. Their statistical analysis is completely inadequate for dealing with this.
And the change in engagement suggests that there's strong selection involved. Their attempt to use midterm scores to control for selection effects is unconvincing. Why not control for whether students used the platform more when there were only multiple-choice questions? Those are the ones who self-selected out of using the AI grader.
I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid students to read books from the library just because you want to do research). Using engagement instead of results is a common choice of proxy to acquire more data, but it can become a vanity metric quickly.
> but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students.
This is definitionally impossible. Testing whether the tool benefits the students that get it is an attempt to benefit certain students at the expense of others. Get over this hangup if you want to do this sort of research.
It's indeed difficult to do good education research, and instructors prefer to just mess around without a control group instead. But the intellectually humble thing to do would be to admit that the experiment can't differentiate between a method that turns bad students into good ones and one that merely identifies good ones through selection effects. When more conscientious researchers get around to doing a proper randomized controlled trial on an intent-to-treat basis, it invariably turns out to have been the latter.
It feels to me that the venn diagram between "students that fully engaged with the material" and "students that learned well from the material" is going to basically be a circle for any teaching method.
The tricky question at the forefront of education research is, at least in my mind, trying to thread the needle between effective techniques that students don’t like, ineffective techniques that students do like, and poorly defined techniques that administrators like. And on top of all that student self-reports aren’t actually very reliable indicators of learning progress at all!
This first sentence is the best summary of educational problems I've ever read, thank you.
I think we've all had a few teachers who seem able to teach 2-10x more effectively than normal; they all seem to do it in different ways. I read recently the Gates Foundation felt they'd found nothing statistically significant in their billion+ dollars of teaching philanthropy.
The most plausible education improvement proposal I'm aware of is individual tutoring. This truly does seem to work well, and I think it's why there's so much interest in agentic tutoring. Perhaps we need a TeachBench to get some hill climbing done by the frontier labs.
Students don't like being told they're bad at something. Students don't like being told to do the thing they're bad at. They especially don't like having to constantly repeat this in a loop until they've caught up with the rest of the class, while other students get to enjoy their free time.
Yeah you'd need to compare this method with some classic method as a control. Otherwise "engagement" is just an indirect measure of how much the students studied.
Yeah, calling this an "effect size" is just nonsense, and it is alarming that educational software can get away with such poor statistical practice. I'm hoping this was just a student project.
This is a helpful explanation - am not a researcher so I have little idea how to run an unbiased, meaningful experiment (except that it takes a lot of effort and thought to run one). Useful analysis
Education research is hard hard hard. Getting clean studies, especially at large sample sizes, is extremely difficult, leading to a lot of ambiguity in results.
Thank you for the feedback! Maybe the following info will be helpful when considering our results:
1. Quiz completion is our deliberately conservative lower bound on reading compliance, and the 0.71 figure is not a claim that those 16 students each gained that much. The estimate is from a regression carried by the per-lesson slope, fit across the whole dosage distribution, and the underlying dosage-performance relationship is essentially unchanged whether or not zero-completion students are included (R² 0.091 vs 0.096). In other words, more Phosphor use is strongly associated with better performance across the whole range of usage - not just the group who completed all content.
The numbers in Table 1 show how dosage was distributed across the course. We report that across the class, the median percentage of lessons reached on Phosphor, including both students with an account and those who never logged in, was above 90%. Among platform users in particular, it was 96%.
We'd like to emphasize that for this pilot, the platform was presented to students as an entirely optional "study aid", and our adoption rates far exceed those reported in the past for optional interventions. It will be interesting to see how things go when we attach completion to the course grade, as we're thinking of doing in the fall. Past literature from interventions in college courses predicts that this will achieve far higher levels of engagement, bringing the high-dosage effect to a large proportion of the class.
2. We explicitly note this in Limitations; it's an observational study. We were unable to do an RCT for this course since it raised an ethics consideration - neither we nor the instructors wanted to deprive students entirely of a course material that could have been helpful for them. We'd love to run a randomized trial at some point though - one way to do this is a crossover, where we offer the treatment to one of two groups, then switch it over to the other midway through the trial, so that both get even treatment. Another possibility is randomly selecting students to get access to MCQ-only vs. CRQ-enabled quizzes. That being said, this mechanism of conditioning on past performance is well-known and relatively robust for observational studies of educational interventions.
3. The platform was created independently of the instructors of the course. The instructors designed their curriculum ahead of time (as had been taught for years of past offerings of the course), lectured in a conventional style, referenced the course's official textbook (Freedman, Pisani, Purves) and suggested homework problems from the textbook only. The instructional content was authored using material that every student in the course had access to, and did not feature exam questions that students were evaluated on after-the-fact.
Phosphor was not endorsed publicly by the instructors, and was spread primarily via student word-of-mouth. In fact, one instructor in this course initially believed the project would be "a waste of time" and refused to collaborate with us for the pilot. Despite this, 97% of the students in this instructor's section used the platform!
Engagement also persisted well across the full ten-week term, and two-thirds of Review attempts involved retries spaced a day or more apart — not a pattern typically produced by novelty effects.
4. Instructors wrote exams independently with their long-running FPP-based curriculum. Even if we steelman and suppose that "the platform just got students to engage" rather than truly learn, this is refuted by our result that the MCQ-only Module 2 had similar engagement but no dosage relationship. This strongly suggests that the CRQ format was a driver of the results.
As we mention in the paper, we agree that replication, especially across contexts, is a priority. For us in particular, this means not only across other courses, but across other institutions as well. And an RCT would certainly help lock in the causal claim.
Thanks for this. No idea why you're getting downvoted on this background.
Anything promising that helps students learn and learn more effectively is obviously incredibly valuable. My concern here is that I don't read anything trying to disambiguate students who just, you know, work harder, and therefore used whatever materials they had to hand including Phosphor, from those who work at the same level and chose not to use Phosphor.
We all recall from our days in undergrad that there are students who do nothing and slide by, a few who do nothing and ace everything, and students who outwork everyone else and outperform -- I think a tool that only helps grinders who were already going to grind is likely not the contribution you're hoping to make.
(ie changing the environment can lead to short term productivity gains because either participants are aware they are being watch, or it breaks up the monotony and makes people work a bit harder. )
I'm on record saying that a system like this with some extra hardware (i.e. a way for the LLM to have live understanding of the student's paper notebook or handout which are being written in with a plain old pencil) combines the best of both worlds - individual tutoring with approximately zero screen time which scales linearly with the number of students. The role of the teacher or professor then becomes a manager of the student - agentic tutor pairs, a referee when the student and model disagree, etc. and most importantly still being the human teacher you can just talk to in the human education process.
I'm convinced this is the future of education - models are there, we need the classroom tech to catch up. The alternative is obvious and quantified in the paper - students just use models to do their work for them and learn nothing.
I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking.
Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad.
I've written a bunch of tools around the remarkable, and it can be easily integrated here, although realtime integration is quite fussy. At this point, it's easiest to vibe code your own, if you're interested, start with https://github.com/ddvk/rmapi and look into pulling documents and optionally putting them back.
remarkable keeps inking data on a per-page basis in vector format, so there's a bit of work to be done to render the inking and then visually assessing it, but it's all pretty much "done" work.
Remarkable's cloud syncing tech is .. not great, and I'd speculate you'd get a better setup here with some sort of harness where you talk / text with an agent that has access to all the files, dates, OCRs, etc first before you consider trying to generate PDFs / pages and put them back onto the tablet.
> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad
What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer.
What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significant to any kind of development of anything.
I assumed that too, back when i thought that getting an e-ink pad would be cool.
Right now Amazon is quoting me $450 on a 10.3" reMarkable, but the price goes up for the 11.8" model and/or for bells and whistles. It looks like an 11" ipad is $450 and up (based on https://www.apple.com/ipad/compare/), although you need to buy the $80-100 iPencil separately.
So yeah - I was hoping that e-ink things would cost less, but they're still pretty expensive and certainly competitive with the iPad.
I feel like an eink b/w device will work better because of the display and lack of distractions like on a full blown iPad. Seems like sink would be similar to a sheet of paper but digital so it could be sent to an LLM or some
Other API. Just my two cents though, I don’t have a super strong opinion about it.
I would add that somewhere in there should be a spaced repetition algorithm.
Spaced repetition is very effective, but it's really really clunky to use. My unpopular opinion is that we all have Stockholm syndrome when it comes to creating "cards", and people talk about how valuable creating cards is; but I think it stucks, it takes a lot of time.
If AI is already teaching me math (let's say), it would be nice to tell the AI/app "quiz me on this periodically", and then the AI makes up a fresh polynomial to factor (or whatever) and presents that to you according to a spaced repetition algorithm.
Behind the scenes, the AI should have access to what has happened the last several times a specific topic has been quized, so the AI can watch to see that certain mistakes are resolved, and the AI might also know better how to correct the user if it has context about previous quizzes of that topic.
But the very act of making and organizing your card deck is part of the SRS! It “sucks” because you get no dopamine hit from a fresh desk, as the reward system is not yet in place.
Again, I really think this is a viewpoint we've talked ourselves into to help us feel better about how cumbersome creating the cards are.
I'm willing to grant that there is some value in choosing what to put in the cards, but most of the awkwardness around making cards is UI related. Nobody creates cards on their phone, or while they're walking (AI could do both of these) - people create cards sitting at their computer (like cavemen!) usually clicking through a clunky UI and managing thousands of cards with thousands of clicks. That sucks, and people probably wont realize it sucks until something better comes along.
Anki is great for studying, but the card creation experience sucks. To be specific: I found creating any custom card type immediately dropped me into the bowels of CSS. It felt like writing HTML by hand.
Is there any facility for re-using shared pieces?
I felt like it needed a static site generator type tool to move up a layer of abstraction and reduce the copying of chunks into my card. Is there one? Please mention if so.
I really never thought it would come up in a HN thread but I’m actually working on a modified version of Anki as a personal project (not quite ready yet though, but will open source probably in a few weeks) where improving the editing/curating/creating experience is a big focus. I’m trying to make it markdown-based too.
Just to pick your brain real fast, when creating new cards (from scratch?) what would a better experience look like for you?
I rarely make cards by hand anymore. I would recommend forking something like https://github.com/jasperket/clanki and editing it (perhaps with an agent) so that it works exactly to your liking.
> Spaced repetition is very effective, but it's really really clunky to use
Spaced repetition is just reviewing the same material periodically. It doesn't have to be a complicated system.
We already know how to learn and educate: spaced repetition (periodic review), and retrieval practice (frequent testing). This is how school used to be fore centuries; it's not sexy but it's effective.
A 'smart pen' that records the student's writing in some way, maybe? My first thought was a tablet that boots straight into a writing software but students should not be subjected to any amount of latency in their writing.
Practically, I think if you want the AI system to have a live view of what the student's doing you're going to have to replace one of either the tablet or the writing instrument. A wearable camera could work as well but there are issues with that.
there was a pen that used special paper to directly record your notes (15-20years ago)... should be possible nowadays to directly transfer this to a connected device and have it feed it to an llm.
They still exist, along with a bunch of different ones. I don't think it's going to be all that different compared to just writing notes in an e-book or on an iPad though. And for many people who learn in other ways, the iPad or similar is superior because you can copy in pictures, make diagrams, and use other ways of learning all in one spot. For me, honestly, something like OneNote (or especially Obsidian) is awesome, because it's super easy to tie in AI into mark down text.
The title is misleading. This isn't an AI tutor so much as a practice quiz platform with an AI autograder.
> constructed-response questions (CRQ) are graded by Claude Sonnet 4.6 against
instructor-defined, question-specific rubric criteria
> Crucially, LLMs make it feasible to grade formative CRQ against rubric criteria at scale, a capability that appears pedagogically significant rather than merely convenient.
They specifically call out that the "RAG chat assistant" part of Phosphor (the platform) wasn't used much.
I commend the effort here, but I don't think these results are particularly noteworthy. The conclusion is essentially that people who do practice quizzes will do better on exams.
The role of a tutor is to find your weakest point (or points) and give you personalized advice on how to improve on them. It is important that you can put trust on the tutor, as your weak point is likely due to a blind spot. When given criticism, it can be viewed as objective or subjective, i.e. a "question of taste", and thus the criticism would be viewed as invalid and not acted upon. As to whether the tutor should point your weakest point or some combination of weak point, it should depend on what helps you learn the most efficiently; it might be to focus on one aspect, or work in a more integrated fashion.
Last (although that may be first), they should consider what are really your learning objectives to tailor that advice.
Definitely not just grading. Tutoring is explaining and back and forth discussion to impart knowledge, in context and in response to specific difficulties/confusion the student is having.
There's a famous post by Erik Hoel that calls the human version of this Aristocratic Tutoring [1] (Scott Alexander is unconvinced [2]).
In the 1980s, a researcher called Benjamin Bloom claimed a z=2.0 (that is 2σ) advantage for a combination of mastery learning (don't move on the the next topic until you've mastered the current one) and 1-on-1 tutoring. Later replications show there is definitely something going on, but the effect size is much lower, for example around z=0.7 in a 2020 paper [3].
I'm still open on AI tutoring, though the Dartmouth results look impressive. Someone please try and replicate this.
There's a saying that AI helps the best students get better, and the worst ones get worse. (Anthropic sort-of agrees [4].) It'll be interesting to see how that turns out.
This is exciting because the effect size is so large. But as the author's acknowledged, selection bias is nearly impossible to control for in this non-randomized study:
> and lacks randomized controls. Self-selection is the central threat: students who complete more quizzes
may be more motivated or higher-performing generally
But this is still a strong result. I'm excited to see more in this space.
Apparently my definition of tutor is so demanding that it should not be expected at all from a tutor (even if provided with information on the tutees weaknesses, tutors don't use the information effectively). However, the point of these studies are to gauge large scale effects rather than high variance small sample size effects. There's a lot more that is very interesting.
Hmm, it might be better to just go straight to what leading countries currently do (Finland/Estonia/Japan/Singapore/etc). Other factors probably play a bigger role, like school funding, school autonomy, teacher professionalism (high education level + continuing education + good pay), free school meals/healthcare/transportation etc
One major limitation is the cost. It costs lots of money to train teachers and pay them good salaries. What leading countries are doing currently might not be sustainable for the long run. Cognitive abilities also different greatly from people to people in different discipline. There is no silver bullet.
America is like a trust fund baby given all the advantages and then the baby goes "fuck it, life is too hard, I am just going to do coke and die early”.
The long term sustainability in at least one of those leading countries (Japan) has been shown effective for at least the period from the end of WWII to now. The principle difficulty they have is not maintaining high levels of educational success but in having enough children to keep the schools open and the teachers fully employed. But, that is a different concern than successful educational outcomes.
The average level of capability and comprehension in fundamental disciplines for students completing primary is a direct result of some fundamental differences in the way they approach classroom organization: for one middle school and high school students do not change classrooms during a school day.
Teachers rotate rooms while students remain in place and this maintains a less disjointed, less distracted transition between classes. There are still very little to zero technological advances to the teaching methods: blackboards/whiteboards, overhead projectors, hand written paperwork, textbooks.
The reasoning is simple. The fundamentals which students in primary education are in need of learning do not change much. Last years' textbooks are perfectly useful and the cost of replacement is directly carried by each student. If they damage a book, they must replace it.
Salaries for teachers are not unusually high, but they are also not low. The building and administrative costs are kept low, for one, by there not being a significant non-instructor labor expense of janitorial and maintenance workers. Schools share the pool of municipal HVAC and other trades for serious infrastructure, but janitors? Nope. The students are required to actually clean up after themselves, and to actually clean the whole school. It makes a difference, and those avoidable labor costs can be directed to proper compensation for the teaching staff.
Anecdotal observation of the overall efficacy of this approach reaches areas not usually measured as 'educational success' but also includes one noteworthy artifact. The common clerk, shop keeper, cook, or gas tank delivery driver all know that i = sqrt(-1) and that a complex number is a pair of numbers, one of which is the coefficient of (i). Let that sink in. How many people graduate with a B.A. from western schools without that?
There is not a 'silver bullet' but there are a full compliment of approaches which when applied together, consistently, and persistently, yield excellent sustainable outcomes.
Back to the falling population problem. There are pluses and minuses from the choice to provide separate boys and girls middle and high schools; one of which is lower teenage pregnancy and it's inverse, lower adult birthrates. (It's not the only reason for the lower birthrates, but it's not having zero impact.) You cant win 'em all.
As far as I know, replications showed an effect size that's still pretty amazing by educational standards but nowhere near z=2.0 (I think z=0.6 is the current best guess).
Conflicted about this study. On one hand, LLMs have been incredible for my personal learnings of new concepts.
On the other, I'm sceptical of that it'll have "strong benefits" at scale; I'd be more in favor if the wording was "some"/"moderate". I reckon self-selection plays a huge part, as mentioned in the "Limitations" section of the paper.
I'd also caution against attaching the tool to grading. That means students have to put more effort into the course, which increases the chances that they will use LLMs to save time rather than make the investment.
> LLMs have been incredible for my personal learnings of new concepts.
Mind if I ask what did you learn and how you're using it?
The reason I'm asking is that I repeatedly felt excitement only to realize down the line that the explanations didn't actually translate into practical skills. I'm not sure it's even an AI problem, it's a "doing versus reading" problem. Same as with reading a pop-science article and thinking to myself that I learned something about physics or medicine or mathematics.
The article explicitly calls out selection bias (this is entirely based on 90% that opted into using the tutor, there was no control group), I wish the headline did as well. "Engaged students score 0.71 - 1.30 SD better in tests" sounds like a much simpler explanation.
I used to TA a graduate level CS math class at Georgia Tech. We regularly saw that the students who self-organized study groups did dramatically better in the course than average. One semester they told us to put everyone in study groups to see if it helped. The effect disappeared. Turns out that it was the self-selection of the most engaged students into a small group that mattered, not the study group itself.
If it's purely a correlation, then maybe those students would be more successful than average even without the study group. They're already the most motivated kids. Maybe they just do "motivated kid stuff" and would still outperform.
Yes, that’s what I think at this point. There is no effect of the study group except as a support group. (That’s all it was for me when I was a student and joined the self-organized study group.)
My self-organized study group was quite a mix of characters, and while I don't doubt that a couple of my friends's grades did benefit from having me there to help them out, I do definitely think that most of the stuff I remember learning there was outside the boundaries of what any of our parents would have approved :)
I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine.
Earlier I used Claude by giving it the course material and asking it to generate me exercises (our cpurse work went way over my head) and yeah i learned to differentiate a gradient or Jacobian, but it was very shallow - I knew the formulas, but not what they meant or how to apple them correctly. After I just filled glaring holes I had in Univariate Calculus by readong and doing, I actually started to understand something.
Lon story short, in my experience Learning with LLM’s is ok with very unfamiliar material that is not too complex (there’s obvious problems of LLM’s themselves being pretty ghastly with maths sometimes), but at least it os not better than the traditional method of just putting your nose on the grimd stone.
In the Matrix movies you can upload complex knowledge to a person in seconds. If LLMs could help with this even a little it would be useful. But alas, people like myself who are slow learners still get overwhelmed when trying to learn too much or too fast without rest breaks.
Do you have a larger study planned for the Fall? It definitely seems promising.
I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature.
But it's not clear that using Sonnet or any other LLM as a "grader" would result in the same improvement. For objective grading, you could be sure that the additional adaptive support is helping. For subjective things like writing style, literature, poetry, you end up with whatever Sonnet thinks is good (and randomly so).
It still could be better for students, but it's not obvious that it would be (or maybe not as strongly?).
Interesting article, wonder where we're going with this though, I find it's very difficult to keep LLMs on track and critical enough to be useful.
Just want to say that:
>In our deployment, student-reported reading completion
baselines for MATH 010 were approximately 15%, with instructors estimating 10%. Individual student
reports of reading compliance ranged from "literally no one does that" to "is this being recorded?"
Wow, putting effort into training material, thoughtfully designing it, and relating the material to the final exam, will increase performance on said exam.
So much AI so much wow.
Like seriously. Most university statistics courses suck big time, so literally any effort put into them will Improve the field.
I’m happy the authors want to improve education but they don’t seem to understand that preparing questionare style material is a confounding factor which could very well explain the better performance too… instead of cramming AI into the next thing. I’m generally opposed to AI on basic textbooks. You don’t want hallucinations imprinted on students who have no idea and can’t judge the quality of the generated text. Some things require effort, reading intro to statistics is one of them, and it’s for a reason, the effort IS the learning
While there's some skepticism in the thread, I'm not particularly surprised if this is true. Children who can get human tutoring do a lot better. An LLM that can answer questions and patiently explain likely offers some benefit.
What creeps me out about bringing LLM into early education is that it's a period where kids learn to socialize and cope with problems, and I do worry about forming substitute relationships with chatbots that are engineered for sycophancy / enablement. But I guess that's a problem either way, because almost every student will try an LLM at some point.
Even if the research is flawed I'm happy they are trying this. They are taking advantage of LLMs to have less rigid tests and also give feedback.
I think there is more potential applications possible with combining LLMs with reference/text books.
Like how about an assistant that points you to the correct books/chapter/paragraph for the concept you need to understand better for a project you are working on? Or clarify any confusion you are having?
Like a human tutor but infinitely patient and non-judgy + search engine.
This is super, but students will have access to AI during the test in real life, so it's ironically less realistic to remove it (thinking of the "... GPT-4 actually harmed subsequent performance by 17% when the tool was removed ..." part).
I'm more curious how students perform on the test with vs. without AI.
A lot of pessimism in the comments, but I am just happy that we are seeing some work towards bridging the 2 Sigma gap for regular education vs. elite private tutoring. I can't imagine that people assume it's the physical presence of the tutor that is making the difference, it has to come down to the personalisation and expertise which is exactly what AI can provide in a form. And yea it might not be "there" yet. But if we don't start trying and studying then it'll never get there.
Tell me if I am oversimplifying, but I never understood the noise about the two sigma problem. Like, of course if you have a private tutor to immediately answer any question that pops into your head at the immediate moment you get confused, you are going to learn vastly more efficiently than in a large classroom where once you get confused you are likely to stay confused. To say nothing of how the pace will likely either drag way behind what you'd like, or accelerate too fast ahead of it.
The environment is just obviously two sigma better. This just... seems obvious to me? In the same way that I will get stronger much faster if I have a physical trainer to tell me exactly what I am doing wrong when I do it? And it seems obviously unsolvable other than by getting everyone a private tutor (or AI..?).
The noice isn’t about the effect, it’s about trying to scale the outcome. We essentially have proof of a method that is significantly better than what is used with the broad population, but we cannot scale it to the broad population. That’s the “problem” in the two sigma problem.
And in general ongoing from “it’s obvious to me” to a quantized effect is always an effort. For instance I don’t believe you are correct in assuming that one would see a two sigma effect in physical exercise when comparing someone who attends a regular exercise course va someone who spends the same time with a personal trainer. Two sigma is a lot, and you won’t be lifting significantly heavier weights from hav a PT vs doing starting strength training. In my opinion you would most definitely see a two sigma effect from doing steroids though. But this is all pure speculation which underlines that part of the “noice” is about the documented aspect of the two sigma problem, where we have a body of data to work against not just personal assumptions. That said if anyone has studies on personal trainers va course work in fitness I’d love to see my world image challenged by data.
The “problem” is exactly as you’ve framed it: individual 1-1 tutoring unambiguously improves education outcomes by a significant (2 StDev’s in Blooms study) vs any other understood method but delivering this universally is (was?) infeasible.
Bloom looked at other methods used in concert to achieve similar improvements to “solve” the problem that could be delivered at scale.
LLMs may provide a new path to the 2-Sigma improvements without the same delivery problems.
Jk, but the skepticism is inevitable. I think we can be dubious about how AI mobilizes global capital while also appreciating tutoring as one of its best targeted use cases.
Shocking that a well executed AI tutor improves outcomes.
Hasn't computer assisted interactive learning already been proven for years? Why does there seem to be so much skepticism about enhancing it with AI?
Is this just something like, astoundingly slow adoption or poor execution? Being held back by paper textbook makers? Teachers unions dragging their feet?
How can interactive AI driven individually paced learning _not_ be obviously dramatically more effective?
Selection effects are extremely important in education. Dartmouth students have already had a large selection effect. If you try to apply this more broadly then it might not work.
Motivation is also a huge part of the problem. I'm wondering if the novelty of the AI tutoring gets more people to try it and whether it would wear off?
It's surprising to me that many students at Dartmouth don't read the textbook. You'd think college admissions would select for that?
It seems promising but, as they say, more research needed.
Lots of people in education will happily tell you how the past 15 years of tech integration has been a net negative.
There ARE technologies that have improved things, but so much high-cost useless tech has been shoved into every level of education that many educators are incredibly leery of new tech.
The issue is that while the underlying technology is useful, the way it gets integrated is frequently not. An administrator cuts a deal for a product they never have to use to an ed-tech giant for a huge amount. Because the ink is dry and a huge sum of money has been spent admins pressure educators to use the technology as much as possible regardless of outcome.
In that context it makes a lot more sense why there is pushback and FUD among educators.
I had a chance to use Google Classroom for a non-profit I was volunteering with, and wow it sucks. If that is what teachers and students have to contend with, yeah, I'd push back against any and all tech forced on me as well. It's all well intentioned, but the road to hell is paved with them.
Having significant roles in the design, funding, and implementation of various public and private school educational projects which integrated some minimal technology I have seen this firsthand as far back as it has existed.
The schools which had the GOOGLE Classroom crap 'sold' to administrators and forced upon the teachers and students were examples of absolutely abysmal failure at every role to which they were supposed to help.
GC is s complete waste of the schools limited budgets, a waste of instructors already over-committed time, and a complete failure to deliver improved educational outcomes for the students. They may have been more 'engaged' in the screens, but they learned less. In the end the GC framework becomes a leash, used by schools to monitor and invade the out-of-school aspects of student lives, and tilts the field more steeply where disparities of home income levels exist.
My particular programs' inclusions of technology were never for 'education' with the exception of elective CS and robotics classes. Otherwise the technology was for records management and course content preparation help for teachers.
One project's tech use case stands out distinct from that limit. A charter school project - instructors and students working to create video and online course materials documenting the conversion of a school program from single grade standard 'science' classes to multi-age, collaborative, hands-on STEM science programs.
But the tech use and coursework were more for documentation and dissemination of the conversion process than for the principle educational objectives.
Actual students did use adapted legacy Samsung android devices to record observations and collaborate among peers during the observations of field studies and workshop classes.
The handhelds checked out to students' on a 'per project' basis each day, so there was no 'individual content' on any of the devices.
The repurposing of these devices wasn't accomplished through 'locking down' an overcapable platform, but by building from the kernel up and only including the specific functionality as required to collect and collaborate - so, no web, no chat, no games, no distractions. The specific workshop content and objectives were available via the handheld, and at specific places in the progression the students were required to input observations and record still or videos. Those images were not shared among students and there was no incentive for shenanigans as all content was directly attached each student's workshop performance and we were clear in the start of the classes that because this was going to be part of the conversion process documentary, no people in any film.
we has a big library of already categorized required captures: Plants in various stages, equipment set-ups, insects in situ for ID, test-strip color comparisons, reagent container labels, etc. Some of the kids called them something from a scifi TV show, complete with "Beam me outta here" jokes.
So, technology can be very useful, but not when it comes with external motives and further widens socio-economic disparity. If the objective is MORE engaging content and MORE accurate records of student progress, then sure.
But, that usefulness is usually directly inverse to the cost and functionality of the devices: cheaper, limited capability tech will usually if not always outperform fancier, overcapable tech.
A $very-cheap refurbished Samsung Galaxy stripped down to whitelisted bluetooth and Wifi LAN, with an distributed storage layer and local-only browsing (Apache Cordova based app) can get alot done without adding tools not required for the specific needs of the student [or educator in many cases].
Campus server infrastructure de-minimus was a solved problem with bittorrent sync and each device only subscribing to the specific file-tree required for the specific workshop/class, and student submissions are only accessible to that student, and the course instructor(s) - eliminates the bullying and other misuses of technology.
We also were able to use a direct approach to resolve the personal phones in class problem, at least for the part f the campus participating in the STEM conversion (4th to 7th grades). Students had to turn off phones, lock them in a pouch, and hand that in to receive a handheld.
If a parent needed to reach the student in school, that's why the school front office had phones. Otherwise, student's were 'in class' and not to be disturbed.
But this was over ten years ago, and today's parents and students would probably throw a toddler sized tantrum at these rules.
Too bad the project got rug-pulled for funding in between years 1 and 2 for the documentation and distributable methods layer; but the school did get to keep teaching the hands-on STEM focus to completion of the then enrolled students - we baked in the costs for that into the first year at the insistence of the participating teachers and parents.
Honestly whether or not this was effective seems less important to me than the adoption numbers.
Text book reading in this course was 10-15% at baseline ... but this AI thing got 90% voluntary usage ungraded.
Even if its worse per-hour than a textbook, you're now teaching 6x as many students _something_ instead of teaching a small minority everything.
So really it just becomes an optimization problem at that point because most students are at least in the funnel/in the running to learn something.
The paper kind of proves this itself ... they tweaked the quize formats mid-semester and where able to iterate which you can't do on a textbook that nobody opens in the first place
I'd argue the results are even better: just reading a textbook doesn't really teach you much. You have to do exercises, but they're expensive to create and grade. LLMs with a proper harness (see paper) tackle both.
I don't want to learn from hallucinations where it will change its answers based on me questioning their teachings. I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions for example is not a wise decision.
Curious whether you were just bare asking it questions, or whether you provided it with lessons one by one with instruction that the lesson is the baseline truth etc
I am somewhat skeptical of this.
First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement
Second, trying to incorporate past grades into their modelling is not a substitute for a randomized trial.
Third, the headline engagement number of 90% is for "engaging with the platform, via Module Review or Lesson Quizzes, at least once". I don't know why much of that couldn't just be attributed to novelty. Or even partly a professor with all sorts of enthusiasm for the platform.
Fourth, the "full dosage" effectiveness is measured based the final exam scores. Were these exam questions produced independently from the "Phosphor" materials? (e.g. by blinding?) Were they checked for direct overlap with those materials? The 0.7 sigma shift is 3 points on a 24 point exam; if even a few of the questions on that exam were very similar to those materials it could account for almost all of it. This is not clear to me from the manuscript.
If this was the case, then it's a question less of "is AI effective" vs. "did the students look at the materials". You could still argue that the AI platform got them to read, but that is a somewhat different statement than the AI helped them learn.
Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again.
That means their experiment design is partially caused by their results instead of the other way around, which is a bad situation to be in. Their statistical analysis is completely inadequate for dealing with this.
And the change in engagement suggests that there's strong selection involved. Their attempt to use midterm scores to control for selection effects is unconvincing. Why not control for whether students used the platform more when there were only multiple-choice questions? Those are the ones who self-selected out of using the AI grader.
I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid students to read books from the library just because you want to do research). Using engagement instead of results is a common choice of proxy to acquire more data, but it can become a vanity metric quickly.
> but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students.
This is definitionally impossible. Testing whether the tool benefits the students that get it is an attempt to benefit certain students at the expense of others. Get over this hangup if you want to do this sort of research.
It's indeed difficult to do good education research, and instructors prefer to just mess around without a control group instead. But the intellectually humble thing to do would be to admit that the experiment can't differentiate between a method that turns bad students into good ones and one that merely identifies good ones through selection effects. When more conscientious researchers get around to doing a proper randomized controlled trial on an intent-to-treat basis, it invariably turns out to have been the latter.
[dead]
It feels to me that the venn diagram between "students that fully engaged with the material" and "students that learned well from the material" is going to basically be a circle for any teaching method.
The tricky question at the forefront of education research is, at least in my mind, trying to thread the needle between effective techniques that students don’t like, ineffective techniques that students do like, and poorly defined techniques that administrators like. And on top of all that student self-reports aren’t actually very reliable indicators of learning progress at all!
This first sentence is the best summary of educational problems I've ever read, thank you.
I think we've all had a few teachers who seem able to teach 2-10x more effectively than normal; they all seem to do it in different ways. I read recently the Gates Foundation felt they'd found nothing statistically significant in their billion+ dollars of teaching philanthropy.
The most plausible education improvement proposal I'm aware of is individual tutoring. This truly does seem to work well, and I think it's why there's so much interest in agentic tutoring. Perhaps we need a TeachBench to get some hill climbing done by the frontier labs.
>effective techniques that students don’t like
Do students not like Mastership techniques?
Students don't like being told they're bad at something. Students don't like being told to do the thing they're bad at. They especially don't like having to constantly repeat this in a loop until they've caught up with the rest of the class, while other students get to enjoy their free time.
Also, mastery learning doesn't look all that effective on standardized tests: https://www.jstor.org/stable/1170613
How do you do Mastership in a class environment? Do they leave half the class waiting for everyone else to catch up?
Yeah you'd need to compare this method with some classic method as a control. Otherwise "engagement" is just an indirect measure of how much the students studied.
Yeah, calling this an "effect size" is just nonsense, and it is alarming that educational software can get away with such poor statistical practice. I'm hoping this was just a student project.
This is a helpful explanation - am not a researcher so I have little idea how to run an unbiased, meaningful experiment (except that it takes a lot of effort and thought to run one). Useful analysis
Education research is hard hard hard. Getting clean studies, especially at large sample sizes, is extremely difficult, leading to a lot of ambiguity in results.
Thank you for the feedback! Maybe the following info will be helpful when considering our results:
1. Quiz completion is our deliberately conservative lower bound on reading compliance, and the 0.71 figure is not a claim that those 16 students each gained that much. The estimate is from a regression carried by the per-lesson slope, fit across the whole dosage distribution, and the underlying dosage-performance relationship is essentially unchanged whether or not zero-completion students are included (R² 0.091 vs 0.096). In other words, more Phosphor use is strongly associated with better performance across the whole range of usage - not just the group who completed all content.
The numbers in Table 1 show how dosage was distributed across the course. We report that across the class, the median percentage of lessons reached on Phosphor, including both students with an account and those who never logged in, was above 90%. Among platform users in particular, it was 96%.
We'd like to emphasize that for this pilot, the platform was presented to students as an entirely optional "study aid", and our adoption rates far exceed those reported in the past for optional interventions. It will be interesting to see how things go when we attach completion to the course grade, as we're thinking of doing in the fall. Past literature from interventions in college courses predicts that this will achieve far higher levels of engagement, bringing the high-dosage effect to a large proportion of the class.
2. We explicitly note this in Limitations; it's an observational study. We were unable to do an RCT for this course since it raised an ethics consideration - neither we nor the instructors wanted to deprive students entirely of a course material that could have been helpful for them. We'd love to run a randomized trial at some point though - one way to do this is a crossover, where we offer the treatment to one of two groups, then switch it over to the other midway through the trial, so that both get even treatment. Another possibility is randomly selecting students to get access to MCQ-only vs. CRQ-enabled quizzes. That being said, this mechanism of conditioning on past performance is well-known and relatively robust for observational studies of educational interventions.
3. The platform was created independently of the instructors of the course. The instructors designed their curriculum ahead of time (as had been taught for years of past offerings of the course), lectured in a conventional style, referenced the course's official textbook (Freedman, Pisani, Purves) and suggested homework problems from the textbook only. The instructional content was authored using material that every student in the course had access to, and did not feature exam questions that students were evaluated on after-the-fact.
Phosphor was not endorsed publicly by the instructors, and was spread primarily via student word-of-mouth. In fact, one instructor in this course initially believed the project would be "a waste of time" and refused to collaborate with us for the pilot. Despite this, 97% of the students in this instructor's section used the platform!
Engagement also persisted well across the full ten-week term, and two-thirds of Review attempts involved retries spaced a day or more apart — not a pattern typically produced by novelty effects.
4. Instructors wrote exams independently with their long-running FPP-based curriculum. Even if we steelman and suppose that "the platform just got students to engage" rather than truly learn, this is refuted by our result that the MCQ-only Module 2 had similar engagement but no dosage relationship. This strongly suggests that the CRQ format was a driver of the results.
As we mention in the paper, we agree that replication, especially across contexts, is a priority. For us in particular, this means not only across other courses, but across other institutions as well. And an RCT would certainly help lock in the causal claim.
Thanks for this. No idea why you're getting downvoted on this background.
Anything promising that helps students learn and learn more effectively is obviously incredibly valuable. My concern here is that I don't read anything trying to disambiguate students who just, you know, work harder, and therefore used whatever materials they had to hand including Phosphor, from those who work at the same level and chose not to use Phosphor.
We all recall from our days in undergrad that there are students who do nothing and slide by, a few who do nothing and ace everything, and students who outwork everyone else and outperform -- I think a tool that only helps grinders who were already going to grind is likely not the contribution you're hoping to make.
I'm not an expert, but how much of this is down to novelty, ie https://en.wikipedia.org/wiki/Hawthorne_effect ?
(ie changing the environment can lead to short term productivity gains because either participants are aware they are being watch, or it breaks up the monotony and makes people work a bit harder. )
I'm on record saying that a system like this with some extra hardware (i.e. a way for the LLM to have live understanding of the student's paper notebook or handout which are being written in with a plain old pencil) combines the best of both worlds - individual tutoring with approximately zero screen time which scales linearly with the number of students. The role of the teacher or professor then becomes a manager of the student - agentic tutor pairs, a referee when the student and model disagree, etc. and most importantly still being the human teacher you can just talk to in the human education process.
I'm convinced this is the future of education - models are there, we need the classroom tech to catch up. The alternative is obvious and quantified in the paper - students just use models to do their work for them and learn nothing.
I work in consulting and one of my projects is piloting an AI use case for a department within one of my clients. On a discovery call someone casually brought up that they bought a reMarkable notebook themselves and were wondering if it could be integrated into the use case. It really got me thinking.
Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad.
https://remarkable.com/
I've written a bunch of tools around the remarkable, and it can be easily integrated here, although realtime integration is quite fussy. At this point, it's easiest to vibe code your own, if you're interested, start with https://github.com/ddvk/rmapi and look into pulling documents and optionally putting them back.
remarkable keeps inking data on a per-page basis in vector format, so there's a bit of work to be done to render the inking and then visually assessing it, but it's all pretty much "done" work.
Remarkable's cloud syncing tech is .. not great, and I'd speculate you'd get a better setup here with some sort of harness where you talk / text with an agent that has access to all the files, dates, OCRs, etc first before you consider trying to generate PDFs / pages and put them back onto the tablet.
> Maybe reMarkable or something like it could help bridge a student's writing with an LLM without having to fall back to a laptop or ipad
What does “bridge a student’s writing” mean?? If this is a real argument it needs to be clearer.
What’s the functional difference between a Remarkable and an iPad? The former is less responsive, costs less, and has better battery life, right? I really don’t see how that’s significant to any kind of development of anything.
Are you talking about running a local model??
> costs less
I assumed that too, back when i thought that getting an e-ink pad would be cool.
Right now Amazon is quoting me $450 on a 10.3" reMarkable, but the price goes up for the 11.8" model and/or for bells and whistles. It looks like an 11" ipad is $450 and up (based on https://www.apple.com/ipad/compare/), although you need to buy the $80-100 iPencil separately.
So yeah - I was hoping that e-ink things would cost less, but they're still pretty expensive and certainly competitive with the iPad.
I feel like an eink b/w device will work better because of the display and lack of distractions like on a full blown iPad. Seems like sink would be similar to a sheet of paper but digital so it could be sent to an LLM or some Other API. Just my two cents though, I don’t have a super strong opinion about it.
I assume that the reMarkable here would mainly just be used to capture the writing. It could then be examined by an LLM asynchronously.
I would add that somewhere in there should be a spaced repetition algorithm.
Spaced repetition is very effective, but it's really really clunky to use. My unpopular opinion is that we all have Stockholm syndrome when it comes to creating "cards", and people talk about how valuable creating cards is; but I think it stucks, it takes a lot of time.
If AI is already teaching me math (let's say), it would be nice to tell the AI/app "quiz me on this periodically", and then the AI makes up a fresh polynomial to factor (or whatever) and presents that to you according to a spaced repetition algorithm.
Behind the scenes, the AI should have access to what has happened the last several times a specific topic has been quized, so the AI can watch to see that certain mistakes are resolved, and the AI might also know better how to correct the user if it has context about previous quizzes of that topic.
But the very act of making and organizing your card deck is part of the SRS! It “sucks” because you get no dopamine hit from a fresh desk, as the reward system is not yet in place.
Again, I really think this is a viewpoint we've talked ourselves into to help us feel better about how cumbersome creating the cards are.
I'm willing to grant that there is some value in choosing what to put in the cards, but most of the awkwardness around making cards is UI related. Nobody creates cards on their phone, or while they're walking (AI could do both of these) - people create cards sitting at their computer (like cavemen!) usually clicking through a clunky UI and managing thousands of cards with thousands of clicks. That sucks, and people probably wont realize it sucks until something better comes along.
> Nobody creates cards on their phone, or while they're walking
Wait, when are you doing it then? No wonder you think it sucks! Adopt some modern tools, yo. Use Anki, or vibe code your own app.
> Use Anki
Anki is great for studying, but the card creation experience sucks. To be specific: I found creating any custom card type immediately dropped me into the bowels of CSS. It felt like writing HTML by hand.
Is there any facility for re-using shared pieces?
I felt like it needed a static site generator type tool to move up a layer of abstraction and reduce the copying of chunks into my card. Is there one? Please mention if so.
I really never thought it would come up in a HN thread but I’m actually working on a modified version of Anki as a personal project (not quite ready yet though, but will open source probably in a few weeks) where improving the editing/curating/creating experience is a big focus. I’m trying to make it markdown-based too.
Just to pick your brain real fast, when creating new cards (from scratch?) what would a better experience look like for you?
I rarely make cards by hand anymore. I would recommend forking something like https://github.com/jasperket/clanki and editing it (perhaps with an agent) so that it works exactly to your liking.
Write a tool to reduce the friction.
I use an Emacs based SRS tool. I have a capture template to quickly make a card.
For anything tedious, it's critical to reduce the friction!
It's rather nerdy, but QuickTurtle9 released mdfc on GitHub, https://github.com/bttger/markdown-flashcards a couple of years ago
Using that as the input file format standard, AI can generate what you're looking for, Android app, Webapp, iOS app, pdf.
> I found creating any custom card type immediately dropped me into the bowels of CSS.
Ah, LaTeX syndrome.
> Spaced repetition is very effective, but it's really really clunky to use
Spaced repetition is just reviewing the same material periodically. It doesn't have to be a complicated system.
We already know how to learn and educate: spaced repetition (periodic review), and retrieval practice (frequent testing). This is how school used to be fore centuries; it's not sexy but it's effective.
A 'smart pen' that records the student's writing in some way, maybe? My first thought was a tablet that boots straight into a writing software but students should not be subjected to any amount of latency in their writing.
Practically, I think if you want the AI system to have a live view of what the student's doing you're going to have to replace one of either the tablet or the writing instrument. A wearable camera could work as well but there are issues with that.
there was a pen that used special paper to directly record your notes (15-20years ago)... should be possible nowadays to directly transfer this to a connected device and have it feed it to an llm.
and after looking it up, it appears they are still available: https://www.livescribe.com/landingpage/ls3_onenote/
They still exist, along with a bunch of different ones. I don't think it's going to be all that different compared to just writing notes in an e-book or on an iPad though. And for many people who learn in other ways, the iPad or similar is superior because you can copy in pictures, make diagrams, and use other ways of learning all in one spot. For me, honestly, something like OneNote (or especially Obsidian) is awesome, because it's super easy to tie in AI into mark down text.
That was likely what I was thinking of - I have vague memories of seeing an ad for this in Popular Mechanics or Popular Science in the 2000s.
Today I saw a demo of Remarkable turned into Voldemort's diary from Harry Potter - you write to it, and it writes back, in handwriting.
This has been around for a bit, not sure it writes back in handwriting.
https://github.com/awwaiid/ghostwriter
Installation not for the average user.
Here's the new demo and repo: https://github.com/MaximeRivest/Riddle It writes back in a cursive font
The title is misleading. This isn't an AI tutor so much as a practice quiz platform with an AI autograder.
> constructed-response questions (CRQ) are graded by Claude Sonnet 4.6 against instructor-defined, question-specific rubric criteria
> Crucially, LLMs make it feasible to grade formative CRQ against rubric criteria at scale, a capability that appears pedagogically significant rather than merely convenient.
They specifically call out that the "RAG chat assistant" part of Phosphor (the platform) wasn't used much.
I commend the effort here, but I don't think these results are particularly noteworthy. The conclusion is essentially that people who do practice quizzes will do better on exams.
> a practice quiz platform with an AI autograder.
What do you think tutoring is?
The role of a tutor is to find your weakest point (or points) and give you personalized advice on how to improve on them. It is important that you can put trust on the tutor, as your weak point is likely due to a blind spot. When given criticism, it can be viewed as objective or subjective, i.e. a "question of taste", and thus the criticism would be viewed as invalid and not acted upon. As to whether the tutor should point your weakest point or some combination of weak point, it should depend on what helps you learn the most efficiently; it might be to focus on one aspect, or work in a more integrated fashion. Last (although that may be first), they should consider what are really your learning objectives to tailor that advice.
Definitely not just grading. Tutoring is explaining and back and forth discussion to impart knowledge, in context and in response to specific difficulties/confusion the student is having.
What do you think tutoring is? Because it's not just extra quizzes
There's a famous post by Erik Hoel that calls the human version of this Aristocratic Tutoring [1] (Scott Alexander is unconvinced [2]).
In the 1980s, a researcher called Benjamin Bloom claimed a z=2.0 (that is 2σ) advantage for a combination of mastery learning (don't move on the the next topic until you've mastered the current one) and 1-on-1 tutoring. Later replications show there is definitely something going on, but the effect size is much lower, for example around z=0.7 in a 2020 paper [3].
I'm still open on AI tutoring, though the Dartmouth results look impressive. Someone please try and replicate this.
There's a saying that AI helps the best students get better, and the worst ones get worse. (Anthropic sort-of agrees [4].) It'll be interesting to see how that turns out.
[1] https://www.theintrinsicperspective.com/p/why-we-stopped-mak... [2] https://www.astralcodexten.com/p/contra-hoel-on-aristocratic... [3] https://www.nber.org/papers/w27476 [4] https://www.anthropic.com/research/AI-assistance-coding-skil...
This is exciting because the effect size is so large. But as the author's acknowledged, selection bias is nearly impossible to control for in this non-randomized study:
> and lacks randomized controls. Self-selection is the central threat: students who complete more quizzes may be more motivated or higher-performing generally
But this is still a strong result. I'm excited to see more in this space.
They tried to control for this. It's described in the first paragraph of section 4.
[flagged]
Yes! Very exciting to see this.
Bloom's Two Sigma Opportunity suggests that there's another SD improvement available: https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem
The story around Bloom's two sigma is a bit complex https://nintil.com/bloom-sigma/
Apparently my definition of tutor is so demanding that it should not be expected at all from a tutor (even if provided with information on the tutees weaknesses, tutors don't use the information effectively). However, the point of these studies are to gauge large scale effects rather than high variance small sample size effects. There's a lot more that is very interesting.
Beat me to it. The Wikipedia page for example looks like an advertisement for tutoring firms, and when you dig into the sources, it’s pretty oversold
Thank you for this and for parent’s comment - I know what rabbit hole I’m going down today.
Hmm, it might be better to just go straight to what leading countries currently do (Finland/Estonia/Japan/Singapore/etc). Other factors probably play a bigger role, like school funding, school autonomy, teacher professionalism (high education level + continuing education + good pay), free school meals/healthcare/transportation etc
One major limitation is the cost. It costs lots of money to train teachers and pay them good salaries. What leading countries are doing currently might not be sustainable for the long run. Cognitive abilities also different greatly from people to people in different discipline. There is no silver bullet.
America is like a trust fund baby given all the advantages and then the baby goes "fuck it, life is too hard, I am just going to do coke and die early”.
The long term sustainability in at least one of those leading countries (Japan) has been shown effective for at least the period from the end of WWII to now. The principle difficulty they have is not maintaining high levels of educational success but in having enough children to keep the schools open and the teachers fully employed. But, that is a different concern than successful educational outcomes.
The average level of capability and comprehension in fundamental disciplines for students completing primary is a direct result of some fundamental differences in the way they approach classroom organization: for one middle school and high school students do not change classrooms during a school day.
Teachers rotate rooms while students remain in place and this maintains a less disjointed, less distracted transition between classes. There are still very little to zero technological advances to the teaching methods: blackboards/whiteboards, overhead projectors, hand written paperwork, textbooks.
The reasoning is simple. The fundamentals which students in primary education are in need of learning do not change much. Last years' textbooks are perfectly useful and the cost of replacement is directly carried by each student. If they damage a book, they must replace it.
Salaries for teachers are not unusually high, but they are also not low. The building and administrative costs are kept low, for one, by there not being a significant non-instructor labor expense of janitorial and maintenance workers. Schools share the pool of municipal HVAC and other trades for serious infrastructure, but janitors? Nope. The students are required to actually clean up after themselves, and to actually clean the whole school. It makes a difference, and those avoidable labor costs can be directed to proper compensation for the teaching staff.
Anecdotal observation of the overall efficacy of this approach reaches areas not usually measured as 'educational success' but also includes one noteworthy artifact. The common clerk, shop keeper, cook, or gas tank delivery driver all know that i = sqrt(-1) and that a complex number is a pair of numbers, one of which is the coefficient of (i). Let that sink in. How many people graduate with a B.A. from western schools without that?
There is not a 'silver bullet' but there are a full compliment of approaches which when applied together, consistently, and persistently, yield excellent sustainable outcomes.
Back to the falling population problem. There are pluses and minuses from the choice to provide separate boys and girls middle and high schools; one of which is lower teenage pregnancy and it's inverse, lower adult birthrates. (It's not the only reason for the lower birthrates, but it's not having zero impact.) You cant win 'em all.
As far as I know, replications showed an effect size that's still pretty amazing by educational standards but nowhere near z=2.0 (I think z=0.6 is the current best guess).
Conflicted about this study. On one hand, LLMs have been incredible for my personal learnings of new concepts.
On the other, I'm sceptical of that it'll have "strong benefits" at scale; I'd be more in favor if the wording was "some"/"moderate". I reckon self-selection plays a huge part, as mentioned in the "Limitations" section of the paper.
I'd also caution against attaching the tool to grading. That means students have to put more effort into the course, which increases the chances that they will use LLMs to save time rather than make the investment.
> LLMs have been incredible for my personal learnings of new concepts.
Mind if I ask what did you learn and how you're using it?
The reason I'm asking is that I repeatedly felt excitement only to realize down the line that the explanations didn't actually translate into practical skills. I'm not sure it's even an AI problem, it's a "doing versus reading" problem. Same as with reading a pop-science article and thinking to myself that I learned something about physics or medicine or mathematics.
Various concepts when I joined new teams in domains I've never worked in before. And system design. So very practical, and where stakes were high.
The article explicitly calls out selection bias (this is entirely based on 90% that opted into using the tutor, there was no control group), I wish the headline did as well. "Engaged students score 0.71 - 1.30 SD better in tests" sounds like a much simpler explanation.
I used to TA a graduate level CS math class at Georgia Tech. We regularly saw that the students who self-organized study groups did dramatically better in the course than average. One semester they told us to put everyone in study groups to see if it helped. The effect disappeared. Turns out that it was the self-selection of the most engaged students into a small group that mattered, not the study group itself.
So there might be zero effect?
If it's purely a correlation, then maybe those students would be more successful than average even without the study group. They're already the most motivated kids. Maybe they just do "motivated kid stuff" and would still outperform.
Yes, that’s what I think at this point. There is no effect of the study group except as a support group. (That’s all it was for me when I was a student and joined the self-organized study group.)
My self-organized study group was quite a mix of characters, and while I don't doubt that a couple of my friends's grades did benefit from having me there to help them out, I do definitely think that most of the stuff I remember learning there was outside the boundaries of what any of our parents would have approved :)
"Full dosage of the Phosphor material is associated with an increase in final exam performance."
This sentence is accurate, but inevitably leads to the confusion you see in these comments.
I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine.
Earlier I used Claude by giving it the course material and asking it to generate me exercises (our cpurse work went way over my head) and yeah i learned to differentiate a gradient or Jacobian, but it was very shallow - I knew the formulas, but not what they meant or how to apple them correctly. After I just filled glaring holes I had in Univariate Calculus by readong and doing, I actually started to understand something.
Lon story short, in my experience Learning with LLM’s is ok with very unfamiliar material that is not too complex (there’s obvious problems of LLM’s themselves being pretty ghastly with maths sometimes), but at least it os not better than the traditional method of just putting your nose on the grimd stone.
In the Matrix movies you can upload complex knowledge to a person in seconds. If LLMs could help with this even a little it would be useful. But alas, people like myself who are slow learners still get overwhelmed when trying to learn too much or too fast without rest breaks.
Do you have a larger study planned for the Fall? It definitely seems promising.
I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature.
PS - I'd say this qualifies for Show HN, too!
Do you
They were using Sonnet 4.6 for some fre form responses so that could be applied to something subjective.
But it's not clear that using Sonnet or any other LLM as a "grader" would result in the same improvement. For objective grading, you could be sure that the additional adaptive support is helping. For subjective things like writing style, literature, poetry, you end up with whatever Sonnet thinks is good (and randomly so).
It still could be better for students, but it's not obvious that it would be (or maybe not as strongly?).
Interesting article, wonder where we're going with this though, I find it's very difficult to keep LLMs on track and critical enough to be useful.
Just want to say that:
>In our deployment, student-reported reading completion baselines for MATH 010 were approximately 15%, with instructors estimating 10%. Individual student reports of reading compliance ranged from "literally no one does that" to "is this being recorded?"
is hilarious
Wow, putting effort into training material, thoughtfully designing it, and relating the material to the final exam, will increase performance on said exam. So much AI so much wow. Like seriously. Most university statistics courses suck big time, so literally any effort put into them will Improve the field. I’m happy the authors want to improve education but they don’t seem to understand that preparing questionare style material is a confounding factor which could very well explain the better performance too… instead of cramming AI into the next thing. I’m generally opposed to AI on basic textbooks. You don’t want hallucinations imprinted on students who have no idea and can’t judge the quality of the generated text. Some things require effort, reading intro to statistics is one of them, and it’s for a reason, the effort IS the learning
Interesting, congrats.
Are you planning on opening access to Phosphor?
maybe they did already via the "formerly known as" comment in the paper?: https://www.spongium.org
While there's some skepticism in the thread, I'm not particularly surprised if this is true. Children who can get human tutoring do a lot better. An LLM that can answer questions and patiently explain likely offers some benefit.
What creeps me out about bringing LLM into early education is that it's a period where kids learn to socialize and cope with problems, and I do worry about forming substitute relationships with chatbots that are engineered for sycophancy / enablement. But I guess that's a problem either way, because almost every student will try an LLM at some point.
From my understanding the actual AI that was barely used. What was used was a quiz with an AI grader.
Even if the research is flawed I'm happy they are trying this. They are taking advantage of LLMs to have less rigid tests and also give feedback.
I think there is more potential applications possible with combining LLMs with reference/text books. Like how about an assistant that points you to the correct books/chapter/paragraph for the concept you need to understand better for a project you are working on? Or clarify any confusion you are having?
Like a human tutor but infinitely patient and non-judgy + search engine.
On a Dartmouth related note, I still can’t believe they abandoned Blitzmail for Office 365. What a loss.
This is super, but students will have access to AI during the test in real life, so it's ironically less realistic to remove it (thinking of the "... GPT-4 actually harmed subsequent performance by 17% when the tool was removed ..." part).
I'm more curious how students perform on the test with vs. without AI.
A lot of pessimism in the comments, but I am just happy that we are seeing some work towards bridging the 2 Sigma gap for regular education vs. elite private tutoring. I can't imagine that people assume it's the physical presence of the tutor that is making the difference, it has to come down to the personalisation and expertise which is exactly what AI can provide in a form. And yea it might not be "there" yet. But if we don't start trying and studying then it'll never get there.
Tell me if I am oversimplifying, but I never understood the noise about the two sigma problem. Like, of course if you have a private tutor to immediately answer any question that pops into your head at the immediate moment you get confused, you are going to learn vastly more efficiently than in a large classroom where once you get confused you are likely to stay confused. To say nothing of how the pace will likely either drag way behind what you'd like, or accelerate too fast ahead of it.
The environment is just obviously two sigma better. This just... seems obvious to me? In the same way that I will get stronger much faster if I have a physical trainer to tell me exactly what I am doing wrong when I do it? And it seems obviously unsolvable other than by getting everyone a private tutor (or AI..?).
Asking from a place of curiosity.
The noice isn’t about the effect, it’s about trying to scale the outcome. We essentially have proof of a method that is significantly better than what is used with the broad population, but we cannot scale it to the broad population. That’s the “problem” in the two sigma problem.
And in general ongoing from “it’s obvious to me” to a quantized effect is always an effort. For instance I don’t believe you are correct in assuming that one would see a two sigma effect in physical exercise when comparing someone who attends a regular exercise course va someone who spends the same time with a personal trainer. Two sigma is a lot, and you won’t be lifting significantly heavier weights from hav a PT vs doing starting strength training. In my opinion you would most definitely see a two sigma effect from doing steroids though. But this is all pure speculation which underlines that part of the “noice” is about the documented aspect of the two sigma problem, where we have a body of data to work against not just personal assumptions. That said if anyone has studies on personal trainers va course work in fitness I’d love to see my world image challenged by data.
The “problem” is exactly as you’ve framed it: individual 1-1 tutoring unambiguously improves education outcomes by a significant (2 StDev’s in Blooms study) vs any other understood method but delivering this universally is (was?) infeasible.
Bloom looked at other methods used in concert to achieve similar improvements to “solve” the problem that could be delivered at scale.
LLMs may provide a new path to the 2-Sigma improvements without the same delivery problems.
> the platform was adopted by 90.2% of enrolled students
Then it is "effectiveness", not "efficacy". Prefer simpler and more specific words when possible, to reduce effort for the reader.
In mice!
Jk, but the skepticism is inevitable. I think we can be dubious about how AI mobilizes global capital while also appreciating tutoring as one of its best targeted use cases.
Shocking that a well executed AI tutor improves outcomes.
Hasn't computer assisted interactive learning already been proven for years? Why does there seem to be so much skepticism about enhancing it with AI?
Is this just something like, astoundingly slow adoption or poor execution? Being held back by paper textbook makers? Teachers unions dragging their feet?
How can interactive AI driven individually paced learning _not_ be obviously dramatically more effective?
Selection effects are extremely important in education. Dartmouth students have already had a large selection effect. If you try to apply this more broadly then it might not work.
Motivation is also a huge part of the problem. I'm wondering if the novelty of the AI tutoring gets more people to try it and whether it would wear off?
It's surprising to me that many students at Dartmouth don't read the textbook. You'd think college admissions would select for that?
It seems promising but, as they say, more research needed.
its like anything else. benifits students that are already motivated to learn.
very few are actually motivated to learn and are just there to get a job or its just next thing that they have to do in life.
I fear even a lot of bright, motivated people will be so discouraged by AI doomers that they won’t bother trying to learn.
Lots of people in education will happily tell you how the past 15 years of tech integration has been a net negative.
There ARE technologies that have improved things, but so much high-cost useless tech has been shoved into every level of education that many educators are incredibly leery of new tech.
The issue is that while the underlying technology is useful, the way it gets integrated is frequently not. An administrator cuts a deal for a product they never have to use to an ed-tech giant for a huge amount. Because the ink is dry and a huge sum of money has been spent admins pressure educators to use the technology as much as possible regardless of outcome.
In that context it makes a lot more sense why there is pushback and FUD among educators.
I had a chance to use Google Classroom for a non-profit I was volunteering with, and wow it sucks. If that is what teachers and students have to contend with, yeah, I'd push back against any and all tech forced on me as well. It's all well intentioned, but the road to hell is paved with them.
This, ALL DAY LONG.
Having significant roles in the design, funding, and implementation of various public and private school educational projects which integrated some minimal technology I have seen this firsthand as far back as it has existed.
The schools which had the GOOGLE Classroom crap 'sold' to administrators and forced upon the teachers and students were examples of absolutely abysmal failure at every role to which they were supposed to help.
GC is s complete waste of the schools limited budgets, a waste of instructors already over-committed time, and a complete failure to deliver improved educational outcomes for the students. They may have been more 'engaged' in the screens, but they learned less. In the end the GC framework becomes a leash, used by schools to monitor and invade the out-of-school aspects of student lives, and tilts the field more steeply where disparities of home income levels exist.
My particular programs' inclusions of technology were never for 'education' with the exception of elective CS and robotics classes. Otherwise the technology was for records management and course content preparation help for teachers.
One project's tech use case stands out distinct from that limit. A charter school project - instructors and students working to create video and online course materials documenting the conversion of a school program from single grade standard 'science' classes to multi-age, collaborative, hands-on STEM science programs.
But the tech use and coursework were more for documentation and dissemination of the conversion process than for the principle educational objectives.
Actual students did use adapted legacy Samsung android devices to record observations and collaborate among peers during the observations of field studies and workshop classes.
The handhelds checked out to students' on a 'per project' basis each day, so there was no 'individual content' on any of the devices.
The repurposing of these devices wasn't accomplished through 'locking down' an overcapable platform, but by building from the kernel up and only including the specific functionality as required to collect and collaborate - so, no web, no chat, no games, no distractions. The specific workshop content and objectives were available via the handheld, and at specific places in the progression the students were required to input observations and record still or videos. Those images were not shared among students and there was no incentive for shenanigans as all content was directly attached each student's workshop performance and we were clear in the start of the classes that because this was going to be part of the conversion process documentary, no people in any film. we has a big library of already categorized required captures: Plants in various stages, equipment set-ups, insects in situ for ID, test-strip color comparisons, reagent container labels, etc. Some of the kids called them something from a scifi TV show, complete with "Beam me outta here" jokes.
So, technology can be very useful, but not when it comes with external motives and further widens socio-economic disparity. If the objective is MORE engaging content and MORE accurate records of student progress, then sure.
But, that usefulness is usually directly inverse to the cost and functionality of the devices: cheaper, limited capability tech will usually if not always outperform fancier, overcapable tech.
A $very-cheap refurbished Samsung Galaxy stripped down to whitelisted bluetooth and Wifi LAN, with an distributed storage layer and local-only browsing (Apache Cordova based app) can get alot done without adding tools not required for the specific needs of the student [or educator in many cases].
Campus server infrastructure de-minimus was a solved problem with bittorrent sync and each device only subscribing to the specific file-tree required for the specific workshop/class, and student submissions are only accessible to that student, and the course instructor(s) - eliminates the bullying and other misuses of technology.
We also were able to use a direct approach to resolve the personal phones in class problem, at least for the part f the campus participating in the STEM conversion (4th to 7th grades). Students had to turn off phones, lock them in a pouch, and hand that in to receive a handheld.
If a parent needed to reach the student in school, that's why the school front office had phones. Otherwise, student's were 'in class' and not to be disturbed.
But this was over ten years ago, and today's parents and students would probably throw a toddler sized tantrum at these rules.
Too bad the project got rug-pulled for funding in between years 1 and 2 for the documentation and distributable methods layer; but the school did get to keep teaching the hands-on STEM focus to completion of the then enrolled students - we baked in the costs for that into the first year at the insistence of the participating teachers and parents.
TLDR; don't let your school get SCROOGLED.
Honestly whether or not this was effective seems less important to me than the adoption numbers.
Text book reading in this course was 10-15% at baseline ... but this AI thing got 90% voluntary usage ungraded.
Even if its worse per-hour than a textbook, you're now teaching 6x as many students _something_ instead of teaching a small minority everything.
So really it just becomes an optimization problem at that point because most students are at least in the funnel/in the running to learn something.
The paper kind of proves this itself ... they tweaked the quize formats mid-semester and where able to iterate which you can't do on a textbook that nobody opens in the first place
I'd argue the results are even better: just reading a textbook doesn't really teach you much. You have to do exercises, but they're expensive to create and grade. LLMs with a proper harness (see paper) tackle both.
0.71 SD effect size is impressive but i would be more convinced if they ran the control group against a TA who actually showed up to office hours.
Very nicely typeset.
Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
Wikipedia doesn’t make much money but is still helpful. LLMs don’t need to make a whole bunch of money to be helpful.
People aren't paying trillions to train them to be helpful. They want to make quadrillions.
Ok
I don't want to learn from hallucinations where it will change its answers based on me questioning their teachings. I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions for example is not a wise decision.
Curious whether you were just bare asking it questions, or whether you provided it with lessons one by one with instruction that the lesson is the baseline truth etc
Are we talking about human teachers or LLMs here?
This is such a lazy response to every LLM criticism
Not sure if its education, but there is huge money in the college admissions process, e.g., SAT prep.
Not enough to cover labs' expenses.
[flagged]
[dead]
[flagged]
Why did you make a new account to spam AI comments?