This is, without exaggeration, probably the fiftieth blog post or long-form comment about how someone is using an LLM for "complex learning", and I'd just really like to see at least one of these to be accompanied by a statement saying what are the kinds of problems the author can now confidently solve that they couldn't before.
In my experience, LLMs are really good for taking up your time and making you feel like you're learning, in the same way that many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
If you ask an LLM to give you a 500-word summary of quantum physics, it'll give you an oversimplification that probably leans on a hodgepodge of pop-sci metaphors. And if you start drilling down, you risk drilling down on these ELI5 metaphors, which can get you farther away from truth.
I personally don't get this attitude. For one thing, nobody on earth can give you a 500-word summary of quantum physics in a way that lets you internalize even a tiny bit of it. That's not how you learn anything.
Here's how I do it: I open Baby Rudin (3rd ed.), second chapter, and read the main text - absolute brutality. I unpack almost every sentence with Claude/GPT until I finally get what's going on. No ELI5 nonsense, just examples and counterexamples galore while absorbing the techniques and the way of thinking in analysis/topology. How do I know I've learned the material? By solving every single problem in that chapter. Here's the thing, though: the problems in Rudin can be brutal and decoupled from what's in the text, so if you can handle them, you've definitely mastered the material. No 500-word summary of analysis here.
One thing I've begun to notice is that LLMs list of a bunch of interesting stuff and raise all these thing you consider but often sometimes you just want a more focused response, so these scatter responses kind of lead you to being overwhelmed and losing focus on what you really wanted to do. At least I've started noticing this.
Like I'd ask about some statical approach taken in a paper and suddenly i'm being bombard with all these potential pivots and things I really need to consider, I kind of just want to consider 1 thing at a time and come to things once I fixed the immediate issue. Sure I have no doubt these other pieces of information are useful but it's just not the most useful information I need right now.
This is less of a problem with coding agents more so putting learning related questions to an LLM, like is the method covered in this paper, yes no? instead I get an exhaustive but overwhelming and indirect response that contains part of the answer. I just wanted to know if it was worth my time going through the paper but now I'm being bombarded told all this tangential information, which is unclear to me if I need to consider right at this moment, it's really distracting.
Maybe this is something others adapted to but i've resorted speaking past it saying, "this is the question please stay on topic" or literally "one thing at a time please" and then it narrows in, but they really stretch your attention thin if you're not more aggressive with keeping them on topic. The smarter models are better, and if you use max compute it does a better job.
I think they can definitely be helpful for learning, but you got play an active role, you can just consume what it says like content.
I’ve found it helpful to ask LLMs specific questions about your knowledge gaps, or ask for concrete worked examples that illustrate the ideas you’re learning about.
In [0], I ask: „When applying Hidden Markov Models to POS tagging in NLP, what do the latent states and observations usually represent?” I then follow up with some specific questions and requests for walkthrough. You can’t see it from this conversations, but I have Wikipedia and a bunch of other resources open in separate pages, cross-reading, and I follow up with a handwritten toy implementation of a Viterbi-based POS tagger once my mental model crystallizes. This is very different from a 500-word summary of quantum physics, and I still had to put in effort (this is unescapable!), but I found the experience rewarding. Also note that this is relearning of a topic that was part of my uni curriculum but long forgotten.
In [1] and [2], I’m learning Spanish by reading García Lorca’s poems. Here again I’m going through the texts with a dictionary, and augmenting my learning with what a dictionary won’t tell me: given the usage of a word or phrase in this specific poem, is it something that could occur in everyday speech, or is it poetical?
100% agree. They are not good teachers. Nevertheless I've been using LLMs more and more to learn complex topics, but my learning is always anchored in something else:
- Getting through textbooks and lecture notes. LLMs have gotten very good at answering basic questions on quite advanced material (e.g. representation theory and quantum field theory). By asking a very specific question or even giving the LLM a screenshot, I can get unstuck a lot faster.
- Learning e.g. new python packages. Instead of hunting for examples on Stack Exchange, now I ask an LLM to write a minimal working example and then build off of that. By writing most of the remaining code myself and only using the LLM to answer questions, I've been able to learn new packages significantly faster.
In both cases, the LLM isn't providing the curriculum or guiding what I learn. The textbooks, papers and coding tasks are. But now I can pick these things up much more efficiently.
I think it depends on what you want to learn and what your definition of "learnt" is.
The other day I realised I had no idea how DNA and life works. I guess I studied it at high school (25 years ago), but maybe it didn't go into much detail or it just didn't click.
So I asked ChatGPT to explain it to me, I came up with my own mental model from it's explanation, told it that, then it corrected me where I misunderstood things. We went backwards and forwards for an hour, me asking questions, it correcting me, until I felt like I understood the whole picture.
Am I going to become a biologist and study the origins of life from that? Definatley not! But if my kids need help on their biology homework, I now understand the basics of it.
Can you say in all honesty that a read of the Wikipedia article for DNA would have been less helpful? It's less convenient perhaps, but definitely more authoritative.
>So I asked ChatGPT to explain it to me, I came up with my own mental model from it's explanation, told it that, then it corrected me where I misunderstood things. We went backwards and forwards for an hour, me asking questions, it correcting me, until I felt like I understood the whole picture.
If you're only checking your understanding against the one source you used to obtain it, how can you tell whether your understanding coincides with reality (or rather, with general scientific understanding), and not just with the source you read? And I'm not asking just about ChatGPT; the same question could apply to any source. Books are not exempt from containing errors.
Concrete high-profile example: in "Surely You're Joking, Mr. Feynman!", Feynman told of a ball which, in a Brazilian college-level physics book, was described as having a 40% higher acceleration that it would actually have in practice.
Turns out that the author had done a thought experiment but neglected to factor in the rotational inertia.
Because you can successfully use it to communicate with other people. That's the grounding truth. If you'd only ever read about it and never actually employed it then yes, you would not be able to make that claim.
It's not the same thing. A language is a closed system, it just requires that the participants agree on the rules in order to be successful. Biology is not a language, it's a field of study of phenomena that exist independently of minds. If you read a biology textbook (or an LLM's output about biology) and then relate to someone else what you've read, you and the other person can discuss the topic, and be no closer to ascertain whether what you two understand has any basis in reality.
Biology is in some sense a language, how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?
In any case I think you have an overly narrow definition of learning that we're unlikely to come to terms over.
>how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?
Those boundaries are completely imaginary and don't exist in reality. In reality there's no biology, there's only elementary particles interacting physically. Whether you agree with someone else to classify a phenomenon as biological or chemical, you're not refining your knowledge of the real world, you're just performing an organizational task.
But, say, how many times the tympanic ear evolved independently is a real phenomenon that can be investigated, and it's something that you're either correct, incorrect, or ignorant about. If I tell you it evolved five times, what more can either of us gleam about the real facts by just discussing this factoid back and forth, if neither of us has access to any additional knowledge or way to put this datum to the test?
He's talking about learning that makes you operational. If you can speak with someone, then you've learned language. If you can get a computer to do something, then you've learned some programming.
The equivalent for biology would be to grow a plant or a few plants and animals successfully. That's operational at a certain level, you could also be operational at a lower or higher level.
No, it doesn't necessarily need to be applicable, it just needs to be testable. The size of the Earth for example may not affect you directly in any way, but it's something you can verify to some degree of precision. But if you just read about it, can you really say that you know it? What if all you believe about the size and shape of the Earth is from what you've asked an LLM? Do you still know it?
EDIT: Perhaps not the best example, because the size and shape of the Earth are data that are repeated often enough that an LLM would be unlikely to quote it grossly incorrectly, but I think my point still comes across.
For majority of people, especially children, book is an authoritative source. Virtually no one outside of research fields and very excited enthusiasts will check sources in depth to see whether what is shown in the book is true or not.
So, I think the point OP is making is that most people don't really check sources while learning things 'the conventional way'.
We have to distinguish learning from, let's call it, recreational education. If you're learning a topic, you're not going to be using a single book. Not even schoolchildren do that. Regardless of whether you consider a book an authoritative source, if you read multiple books those books will have to agree with each other, and where they don't you'll experience some confusion until you can resolve the contradiction.
If you're only ingesting information from a single source, be that a book, a teacher, or an LLM, then you haven't really learned, because your knowledge base is unmoored. You can't learn history by studying Tolkien's mythos.
Self-consistency and consistency with your lived experience are good heuristics. Reality is self-consistent, so anything that doesn't add up indicated an error in the source or your understanding of it (or both).
EDIT:
I think it's a kind thing you need to tune yourself into. OTOH, I've observed many (most?) people seemingly being completely oblivious to self-consistency issues of their beliefs and mental models, or even texts they're reading or instructions they're following, and yet... somehow they're generally more successful at life because of it ¯\_(ツ)_/¯.
Reality is most likely self-consistent, but as we can only experience a tiny part of it it‘s impossible to tell. Also, parts of reality may appear to be contradictory with each other when some pieces are not known.
The very first thing I mention: self-consistency. It's the only thing you have if you don't have any empirical data. It's the only thing anyone has, really. QM scientists reading QM papers and experiment reports of other people, and talking with each other, are still relying on self-consistency to sniff their own (or other people's) mistakes.
Not the OP, but I don't think you can. The understanding someone manufacturing silicon has will be not be attainable by reading about it etc. Similar for QM, no replacement for doing (some of) the math yourself, be involved in experiments etc.
Or do you? Cause if you went back and forth with ChatGPT for an hour it definitely hallucinated and lied to you at some point. Maybe consider using something else like Brilliant.org if you want to learn a topic, yanno, so you don’t propagate whatever hallucination from ChatGPT to your kids.
This would be a valid point maybe 3 years ago, but most chatbots will now query and verify direct sources, especially in research mode.
This is very simple to validate and verify. You could argue it may find false primary sources.
You can condemn models for a variety of other things, but acting as if this is still reality shows a lack of understanding as to modern model capabilities
These days online courses like Brilliant and such are likely largely LLM generated. Are they actually vetted by experts before publication? Who knows. My money is on no, or at least, not until someone complains.
Its easy enough to prompt ChatGPT for primary sources when doing research to validate any claims its making.
You should actually do this. Ask ChatGPT to check what it said against sources. It does correct itself. The corrections are usually not major. Just mental shortcuts.
> I'd just really like to see at least one of these to be accompanied by a statement saying what are the kinds of problems the author can now confidently solve that they couldn't before.
I've spent a lot of time with LLMs for the last two years. Something I've tried, almost for decades, is to learn enough CUDA programming to be productive with it when needed. About 6 months ago, after again banging my head against it for weeks, something finally clicked and I feel like I've overcome the initial step of at least grokking the needed ideas so I know where to go next, and I can actually write + compile + use kernels made for my use cases. I won't claim to understand everything, but I couldn't do what I can today, before I learnt the things I now know.
~2 years ago, because of my very weak math foundation, I basically said "Well, CUDA looks really interesting and really fun, but it's too difficult, lets focus on other things", even after reading some starting resources and stuff. But, by asking countless of dumb questions to LLMs, forcing it to steer me in the right direction, when I'm otherwise just driving on the highway or what not, I finally feel like I have a grasp on something I earlier only dreamed about understanding, and I'm able to be productive with it now.
To be fair the entry barrier got a lot lower over the past ~5 years. Now you can write very good CUDA kernels with just a few lines of python DSL code. Zero cpp boilerplate and zero explicit compiler calls.
Stuff like Triton, nvidia warp (the language), numba, cupy jax/pallas and so many others really paved the way. You can start out really high-level, run a profiler and then dive deep into the bottlenecks.
TL,DR: Keep going, it's a great time to have fun with GPUs.
> To be fair the entry barrier got a lot lower over the past ~5 years. Now you can write very good CUDA kernels with just a few lines of python DSL code. Zero cpp boilerplate and zero explicit compiler calls.
Well, yeah, but what I've being doing is learning proper CUDA, not "Python-compiled-to-CUDA" (otherwise it'd take like a just a week to understand enough :P ) and that's looking more or less the same today (although bunch of more complicated stuff piled on top of the fundamentals) as it used to, AFAIK.
With that said, the environment is a lot simpler to setup today at least :)
I wouldn't call one proper CUDA and the other one some dumbed down version. Nvidia really seems to be pushing for these DSLs to be first class within the ecosystem. In some cases probably even more cutting edge than the nvcc frontend, since it's easier to do some experimenting on a new niche package than on the tool everyone relies on.
I believe more and more production code is running kernels which didn't originate from the traditional cuda cpp route.
Agree. Just pay attention to the follow up questions a learner is asking to see the progress. If the follow up is just "continue", "go on", "next" or a non-sequitur then it is smell of a stall. If it is challenging or filling a gap in the answer then it is progress. So production from the learner is the only signal of worth here not the quality of LLM response, the time spent or the ability of the learner to reproduce the facts given by the LLM.
I like to use them to generate me tutorial series for technical topics I want to learn or improve. So I start a prompt like this:
> I want you to create a tutorial series about X for me. The prime objective is that I improve in topic X so never provide a solution but guide and teach. (for programming never write code).
First create a question catalog to assess my current level.
Then I would ask it to structure the tutorial challenges in the following way:
- Goal
- Concept
- Instructions
I figured that if I don't need to read any additional material on the topic the LLM is giving me too much information and I need to change the prompt.
Works for me and I used this too learn topics I feel now comfortable with, like nushell, opencyper, elisp, boot loaders etc.
But maybe you don't consider this "complex"
Curious: Why not pick up a book or two on the topic, and use the LLM to help you through it [0]? To me, prompting the LLM without grounding it sounds like a sure shot way to end up learning "pop-sci", as GP puts it, instead of the actual science?
If today's top LLMs are reliable enough (without grounding) to academically learn "complex topics" from, may be I need to adjust my priors. I must say, I do find myself chatting about other topics (without the need for grounding) that I'm trying to "absorb" (not really learn), like Behavioural Psychology & Philosophy.
[0] Products like NotebookLM are built specifically for such usecases.
I use books to accompany my learning, but I really need something that _forces_ me to think and solve problems in that space on my own.
Books can sometimes give me the illusion of learning something, but then, when needed, I've notice that I haven't really learned it.
LLMs give me structure based on my current skill level. And basically always I accompany this with books, I love reading. It's a nice combination for me.
I would love to see the "depth" of your knowledge in those topics.
You literally proved the OP's point ... thinking you're learning. More like scratching the surface, with lots of invalid data while not being able to recognize what's invalid.
It's like with latest vector of attacks being spamming Github with malware injected in proper looking code in hope of AI to index it.
Then you paste the code because you don't understand it, but you take it as working and only doing what you've asked for.
This feels like an impossible assessment - yes, a model probably can't give you the education that a advanced/expert book on a topic will, but implying that having a verifiable goal is somehow fake learning feels like an intractable problem.
What level of evidence would be sufficient for you to accept that a model may be able to teach a concept?
I'm happy to take on this challenge with a topic of your choosing, but I don't believe there will be an evidence base that satisfies you that the knowledge is earned or deep enough.
I really wonder where this assessment is coming from? It's not that I use LLM written code for something (in those exercises at least), in fact I don't let the LLM write code (see my prompt example).
It's about guiding me in _doing_ exercises so I learn and I can evaluate if I learned something if I can apply the learning myself.
Dunno, I've been learning a lot of Rust in the past few days. Just dove right into a project and asked AI to teach me stuff on a need to know basis. I'm actually getting used to Rust by now.
Learning programming languages is quite trivial. Many years ago I counted 16 that I had used professionally, now they are more.
It takes way more time to master and be very comfortable with a language due to its ecosystem, though. Some languages are more likely to click with a person, yet underneath they are all the same (minus the functional languages that form their own group), e.g. some performance issues may require looking at the generate assembly code.
A programming language is a skill with an accessible source of irrefutable feedback. If the program doesn't work, then you did something wrong. What happens if you try to learn something less concrete or less testable, like quantum mechanics, as the GP suggested?
I agree with you. I'm also using AI to help me learn electronics so I can finally make some real stuff I can hold in my hands. Lots of hard engineering involved so I'm a lot less confident in my ability to spot the AI's own mistakes.
However, saying you can't learn "anything" is just too strong. I'm definitely managing to distill the AI's weights into my own brain.
It's true in a very real sense, though. Without calibration, how can you tell whether you're learning something real or something fictitious? Imagine trying to learn a language from a single teacher, without ever talking to or corresponding with a native speaker. How could you tell whether you're actually learning the language, or the teacher's unique dialect? Yes, you indeed "learned" something, it's just something that might not bear much resemblance to the real thing.
This made me realize all those benchmarks which unlike loosely acts as a person being awarded with a certificate after his doctorate is rather worthless, for you never know when it bullshit-ted.
Same. I setup a practice skill that has a curriculum/list of topics, teaching methodology, approach to drills, progress tracking and spaced repetition.
Additionally I asked it to also give me problems relevant in my business domain so that I learn how to directly apply the knowledge in a realistic scenario.
I am getting much more comfortable writing rust than I was barely two weeks ago. More than I was just reading tutorials.
AI is a super polarising topic. Even now there are people who are like “it’s all AI slop so it’s all useless” along with the “OMG AI!!! OMG we will have AGI soon!!!” people.
The truth, as always, is in between. There’s loads of people using it for useful things, learning, automation and getting good results. But it’s also wrong enough that you need to deploy it carefully sometimes.
Don’t worry about either group. Keep objective and use AI where it helps and do it yourself where you are better. That’s all.
> many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
As a tangent, I think the concept of pop science has wasted so much time of what could be considered brilliant minds. I can't believe how much YouTube people I consider really smart consume under the guise of "learning stuff." And the videos are always designed to be addicting and to entice you to watch other of their stuff, which makes sense, because theyre a business, not a school.
I'm guilty of wasting time on YouTube as much as anyone else (I like watching stand-up routines and Red Bull extreme sports) but I am never under the guise that I'm doing anything productive with my time. Its okay to have fun learning, but I always felt that entertainment and education should be kept separate. You gotta learn something intentionally, not just get it served to you via algorithm.
Note im talking about the educational "shorts" not the 60+ minute deep dives that are basically a college level lecture.
Well everyone needs some downtime, and then I think it's important to acknowledge that there is a range of quality within the category of "educational entertainment." You can learn a lot of interesting things from 3BlueOneBrown and Veritasium, even though neither compares to working through a textbook.
It's the kind of high-brow entertainment that makes you feel like you learn something.
For me, most "push" things are edutainment, whether that'd be Youtube videos or public-broadcaster television programs. Things you seek out yourself are not.
In a similar vein, there's "newstertainment" (news that makes you feel like it's important to watch, but actually changes nothing tangible about your life).
I’m a bit too old for the YouTube generation (can’t stand the fast cuts and permanently agitated voices), but am absolutely guilty of reading a lot of pop-sci books. I feel like they do a better job at teaching stuff for the most part, but probably also make you fall into the trap of believing you actually learnt something…
On the other hand, I’m fine with not being an expert on topics outside of my domain, as long as I retain some basic knowledge and fun party facts. So there’s that.
I do appreciate the literal college lectures on there. Hard to get the algorithm to surface them but they’re golden for learning. Of course I would be unshocked if the authenticity signals I’m tracking (AB.203 Lecture 3 video title, general hubbub and shuffling of chairs at start of video, university affiliated channel) are all faked by grifters 6 months from now.
In reality, I would just ask LLM to give me a technical explanation of quantum physics, and ask any vocab and equation I do not understand in the response.
My trick is to let agents make an University Course Curriculum based of books and papers. Then if I don't understand something, I ask AI to simplify the book/paper until I get it.
> many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
I really enjoyed and learn a lot of things from Karpathy's and Andrew Ng's video. sure many don't really teach you anything, but I could say many others are useful too. Maybe it depends on the way we're prompting as well? it seems useful for some like Terence's message that was shared few weeks ago
I would tend to agree in the case that someone is using LLMs as their primary source for learning. But I've found a lot of use in having a claude project containing the PDF of a textbook I'm working through so that I can ask it to clarify or help me through parts I find confusing. I would definitely say that's greatly accelerated my learning - or at least greatly accelerated the speed at which I integrate information from textbooks.
The test I'd like to see: take a problem set or task you couldn't solve beforehand, learn the topic this way, then try to solve it without the LLM in the loop
as ever the real problem is formulating questions, which requires clarity on the next discrete layer of missing comprehension, which requires self-reflection and genuine insight into your own mind.
I thought LLMs were a great tool for learning new topics - perhaps even complex ones. But overtime, I have had several frustrations with this. First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point. Second, as I dive deeper, I need a way to organize the information in a useful way as I begin to branch out in many different directions. I have tried to use the LLM to fix this by having it generate a web page with diagrams and organized information flow. It's an improvement, but I still run into the issues I described in my first pint - LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise. You can direct it do something like "use plain English and avoid LLM prose - provide only as much information as necessary to demonstrate the point", but it is once again only a marginal improvement.
And then I begin to think to myself that I should just read a book on the topic written by a trusted source who put a lot of effort into teaching the topic properly and presenting the information in a thoughtful way. So, I am back to books and mostly try to use LLMs to clarify certain questions or ideas I have.
It's much better to feed the book to the LLM and ask questions as you read along, instead of asking the LLM to basically write a custom book for you from scratch.
I have to agree with this. Completely relying on LLM for all your learning needs is a disaster. But being absolutely against use of LLMs isn't doing you any favors. This is where you don't have a formula but rely on you judgement and evidence of your having learnt something.
For example having an LLM summarize a dense topic and to find books so that you can filter faster and spend time reading those books works way better than having the LLM summarize the books or the topic (or even relying on second hand information). Another one is having the LLM quiz you on your topics of interest. With questions tailored to attack specific areas that you struggle with. Its wonderful at this, nothing I've used comes close to what an LLM can do here.
You define for yourself what your goals are, slowly refining them as you learn more, and use LLM as a tool. This ,I find works best for learning.
Make it your goal to teach a room full of other humans that topic. I guarantee you will know that material cold. I've done lots of technical training in my career and after teaching a class two or three times I find myself to be very competent in the topic.
It's long been the case that the best way to learn something is to teach something.
> It's long been the case that the best way to learn something is to teach something
Which is pretty unfortunate for those that want to learn. I used to enjoy writing documentation at work, it was my favorite part of the job. And it did feel like it benefited me more than it benefited all the people that were (or weren't) reading my documentation. Now I can't really justify spending much time on docmentation when LLM's can do it in a fraction of the time and it's "good enough"
I got started as a software engineer working in the nuclear industry in the 80s. We measured our documentation in inches not pages, and it was all written by hand. And I'll bet you the documentation in the nuclear industry is still written by hand and not by LLMs.
I have used LLMs for safety sensitive tech writing and so have people I know. It’s pretty good at it, and you usually have a 95% ready to go product at the end of it. The last 5% can pretty easily be filled in by technical experts with way less time spent battling the type writer.
Is it perfect? No. But it suffices most of the time in a pinch.
Can’t speak to the nuke industry, but it’s pretty good at aviation related things.
There is still value in experts distilling knowledge and crafting it to the audience. I'm a consultant in cybersecurity and someone asked me "give me a best practice framework for good policy hygiene". I'm sure an LLM could spit out some tips, but I've been on the industry 15 years and can write in 5 pages what an LLM wouldn't conceive of in that space.
Even with the latest models today, the hallucination rate is absurdly high on anything deeper than surface level knowledge or something that can be directly scraped from reddit.
And you notice when it's a topic you know well or something like software where you can immediately tell the options it's giving you don't exist on the page. Leading to the amusing statement "LLMs are bad at what I do but great at everything else".
I've (elsewhere) written about this diminishing return effect on LLM utility in relation to increasing expertise.
The question: what's the net positive gain of turning people who know nothing in a given field into sub-novices, while weighing actual experts down with work slop and marginal returns?
And I wonder what the true cost is of arming so many novices with that level of dangerous knowledge.
The fact that some people get genuine value from LLMs when learning doesn’t contradict the fact that they’re Dunning-Krueger “expertise” generators. The fact that the person learning from them is in charge of ensuring they aren’t full of shit, which they frequently are, is an inescapable flaw in this process. I honestly think that reduces the value of these things to just above what you can find out with a search engine with most topics. Hey, great. An improvement is an improvement right? Is it an improvement worth trillions of dollars and screwing over writers and artists worldwide? Fuck no.
It wasn’t even all that long ago. I noticed around the time ChatGPT 3.5 was released I saw a significant change in tone regarding LLMs and diffusion models. I think that’s when some serious astroturfing started.
I’m old enough that I worked my first IT summer job the same year slashdot was founded. I’ve seen a lot of tech tribalism form and dissipate, and this one didn’t feel organic. My gut says a lot of the us-vs-them tension originated in a deliberate campaign to cast AI boosters as the tech industry in-crowd, and ‘other’ the people not on-board. Who knows.
1000% astroturfing—and across social media. Too many posts of the same quality at the same time(s). These things ran in cycles, spinning up and dissipating just as suddenly.
E.g. the entire framing to combat complaints about shortcomings was, "It's not the tech. It's you. You're just not doing it right. Wrong setup, wrong workflow, add this to your .MD, use loops, etc". Every complaint was immediately met with this same treatment by a swarm of vague bro-bots that materialized from the ether. The core message? Always the human's fault.
And, don't get me started on the waves of newly minted expert AI creators, making recommendations without showing a single example of what they'd supposedly built.
I'm sure some bandwagon organic creators tried to cash in on the genre, but encouraging that was also part of the point.
I came across the socratic method recently, and have used it to learn a couple of topics that I was having trouble getting to stick. There are some SKILL.md's available for it. It works for concepts as opposed to facts, and causes the model to guide you to answers through your own reasoning, which is both much more engaging than reading a wall of LLM text and helps the information stick.
The "Socratic Method" (aka maieutic) skills annoy me, precisely because when you read them they are the kind of low-effort, low-expertise crap someone who over relies on AI would naively come up with when tasked with the problem of coming up with skills for learning. "Hey the Platonic dialogues are pretty cool and smart, let's do that".
The body of literature on learning theory, and beyond that on specific types of learning and specific mediums such as learning from text is so rich there are way more useful models to draw from. Believe it or not, prellm, researchers in the textual learning field had already demonstrated you can achieve performance equal or better than novice tutors using pretty basic computer aids that follow specific hint/pump interaction structures. Guiding an LLM to use these findings has evidence backing it and is way better than telling it "i guess be like socrates". The problem is, to realize there might be richer more effective and highly researched ways of tackling the problem beyond the first fart of a thought you had one afternoon requires the deep respect for expertise and specialization that precisely basically everyone in the AI space right now fundamentally lacks.
Ironically, LLMs are pretty good at finding stuff like that from vague descriptions. With that, I think the parent commenter meant Art Graesser's AutoTutor work based on the terminology used (hint/pump).
TLDR: Actual human tutoring sessions were recorded and analyzed and Socratic questioning was barely used at all. Instead the following pattern was observed:
Pump — "Uh huh?" "What else?" Costs nothing, so try it first.
Hint — points at the region of the answer. "What about the pumpkin's motion sideways?"
Prompt — fishes for one specific word, with the sentence frame supplied. "The pumpkin keeps moving forward at the same ___?"
Assertion — just says it. "It keeps the runner's horizontal velocity."
what a time to be alive! "if you're having trouble understanding what your robot tutor is trying to teach you, you can ask it to guide you to the concepts using your own reasoning. This is both much more engaging than reading a wall of the robot's text and helps the information stick."
Whats funny about LLMs is they are trained from books, but they are also trained to not output books, so how much of an LLM skews its output because a perfectly normal sentence could be a quote in like 300 different books?
i threw the entire sanderson cosmere into a RAG graph sorta deal just to see how it would do if i questioned an mcp server for it about a universe i know decently well. it was actually astoundingly good. was able to find easter eggs acrossed different books and answer dumb questions like "why is kaladin emo"
If you don't know why kaladin is emo, did you truly read the books lol. Every character has to deal with the stresses of war and most don't come equipped with good mental health to begin with, they're just normal people
i was more or less testing to see how complete of an answer it could come up by scanning the whole semantic graph so i threw questions at it that spanned multiple books
I also agree with this. LLMs are a great companion when reading a book to clarify things and dive into specific topics.
I'd imagine an application that uses LLMs will be created that better manages learning. It's just not clear what that UX is yet- it's obviously not just a chatbot
You need a non DRMd copy of the book. You don't have to feed it all at once, although with a 1M context limit it is doable. A few chapters at a time is enough, in my experience. An easy alternative is using NotebookLM (now Gemini Notebook), and that has worked brilliantly for me, but I haven't tested it for technical topics (for that I like the LLM to create graphs and e.g. interact with Mathematica, so I haven't tried it).
LLM is still too verbose, a real person Socratic conversation can interact a couple sentences at a time, not spew 1-3 windowfuls of low density bullet points.
I even wonder if this behavior is due to next-token prediction architectures, somehow.
This. This is exactly how I use them and I have had no issues so far. I read the book myself, then I point the LLM at it to ask questions about notions I might be struggling with.
LLMs can't "read the room" and infer how much context the audience already has, so they try include everything.
human conceptual thinking is very much a multi-dimensional graph, which relies on light "approximate" concepts that are "good enough". LLM AR token generation is extremely one dimensional and doesnt care about the "weight" of the concept behind a token.
LLMs hold billions of parameters in "mind" at once. humans hold like four "concepts".
This is the essential mismatch and the primary reason LLM conversation can be so painful and exhausting.
Explaining this and limiting "concepts" to four at a time tops is one of the very few AGENTS.md / system prompts I always use, and it has proven invaluable time and again.
Thinking traces show how effective this is at forcing the LLM to simplify its thinking.
[edit] Also, myself and nearly all of my peers are struggling to choke down the flaws of LLM tooling along with the benefits. the speed at which LLM adoption is being forced, without truly crafting them into quality tools first, is not ok, and not normal.
LLMs have stirred an inhumane hunger and fear. the tech is fine, but the way tech companies (creators and consumers) are behaving should be deeply questioned.
If "attention is all you need" then it's something we do indeed lack, in comparison to LLMs! But it's an interesting question: might machine cognition benefit from similar bottlenecks in an attention algorithm? Advancements like Kimi Linear seem to indicate that we're far from the finish line: https://arxiv.org/abs/2510.26692
>I really don't want to read anything generated by something like Opus 5 at this point.
Personally, I find that its generated prose tends to have an undue weight to it, almost as if every topic I ask about somehow bears a heavy burden, or is otherwise load-bearing, to use its parlance.
It has a sort of metronomic quality. It never slows down or speeds up or modulates its tone. It plods forward at a relentless pace and never has a light touch with anything.
I think this is one reason why LLM text is pretty exhausting to read for long stretches.
>It has a sort of metronomic quality. It never slows down or speeds up or modulates its tone.
It's possible that this quality you describe stems from the extensive training corpora utilized by the major AI labs. These almost certainly include work from the esteemed economist Jacob Silj:
I have a personal theory: LLMs are *fundamentally* handicapped at perceiving what's going on in the mind of the human (this can't be "innovated away") and that's at the root of what makes them suck at conversation.
Next time you're chatting with someone, notice how much understanding is shared without anything being said. E.g. the other person might share something deeply disappointing, and they can tell without you even saying anything whether you get what they're going through. This unspoken-yet-communicated information guides the conversation. Or as another example: humans can read the room -- you walk into a room and immediately adjust your demeanor based on what you see and sense.
LLMs are totally blind to things like this, and this adds an inescapable awkwardness to interacting with them. I don't believe they'll ever grow out of this. Which thankfully implies more long term demand for humans instead of robots. :)
Very well observed. I found one more thing: they fail to consider what a 3rd person might understand from your conversation, so when you ask it to dump stuff into a Documentation, they keep making references to facts you had previously discussed or to the train of thought, completely irrelevant to bystander.
100% -- this is the worst. Referencing all sorts of "words with made up contextual/analogous meanings" based on the conversation...outside of the conversation.
Does anyone have a read on if this is primarily a Claude issue, or if all LLMs do this?
I have had very similar experience! I wanted to learn Probablistic ML, checked out a couple of MOOCs, but didn't find any that were at my level - some were too advanced, some too beginner level. Claude was unable to one-shot a course, so I am now asking it to generate it module by module. But even here, it is not doing a very good job. I muddle through the concepts that it has written, do a whole bunch of back-and-forth, which tbh is exhausting, and then rewrite everything in my words so it actually makes sense to another human being.
> I get exhausted reading LLM prose
So much this! If I see one more sentence with the words "genuinely" juxtaposed with "load bearing" my head is going to explode!
i’m personally deriving a huge amount of value from the custom materials fable is assembling for me. for example i asked it to write a focused expository math paper on reed solomon to accompany an implementation module that it wrote for me. it’s remarkably useful to steer it to create graphs and diagrams of exactly how you like the material presented. or the bibliography researched and cross-linked with the body or the order you want your questions addressed.
it also researched vision correcting displays for me and i can finally put that idea to bed - i was never really going to pick up an optometry textbook tbh. plus it was able to pull together a bunch of geometric and physical context about light and the eye plugging exactly my personal knowledge gaps.
in general i suspect these materials might not be that interesting to others because they are so custom to my learning style and personal needs and preferences.
these are usually not one shot documents but rather many prompts deep before i get something I’m willing to sit down and read or study. but dramatically quicker than assembling it myself from primary sources. i wouldn’t say it matches master expositors but then they’re not available to write on any topic i happen to need right now.
plus I’ll just have a live voice discussion with the system when i go for a walk and there are still things bothering me on a topic. it takes a little patience but if i’m in the mood it’s amazing.
i generally find that it can help track down specific references if i suspect hallucinations. but especially on factual topics my experience so far has been extremely encouraging.
Good point. Same with me. Despite it being very tiresome, all the back and forth that I do with it really deepens my understanding of the topic. I have been on Opus so far, let me try Fable and see if it gets better. I haven’t tried voice either. Next time I go for a walk I’ll try that!
I never used office hours as a student, which I later regretted because it made me work longer and harder to perhaps achieve somewhat better understanding in some classes, but also I dropped every proof-based math course I ever took. Overall I think my education would have been stronger by attending office hours.
I view LLMs in education similarly to office hours. Some people abuse it to get homework answers without grappling with the material, but the optimal amount is not zero.
LLM certainly not a replacement for a book, where you get someone’s extended personal approach to a topic, thoughtfully organized, reviewed and edited, often times actual courses taught based on it, with answers checked and errata available online.
I completely agree. I have vibe coded what i would consider to be some pretty weird things in the name of learning facilitation.
Perhaps the best example has been a native macOS app that is a completely custom text editor with built-in debugger, lsp support, fuzzy finder, etc stuff you'd expect. Inside the same app is a library of books i can read within the app completely formatted and for every chapter/section of each book that is a quiz to take (LLM generated of course), a "recitation" tab where i am asked a question and say outloud my response to the AI to evaluate me on and then finally practice problems to do within the custom text editor (these are usually programming books). The reader also has ai re-write built in.
As neat as this is, and i worked through K&R like this, i have ultimately fallen back on "just read the damn book and go to the AI when you've got questions."
I've decided that the main thing I'm building is my own mental model. You can take notes, create docs, put graphs and websites together, but unless I'm just trying to generate some reference material the only real objective is to develop the understanding and intuitions inside my own brain.
So I have the LLM offer a very short explanation of something, and from there's it's just me asking questions. Anything that feels fuzzy or not fully internalized is something I poke at until I'm satisfied.
It really has helped me develop a sensitivity to what I understand vs what I don't, and the ability to drill into any part of it is amazing.
I do this too. I use Claude. I picked a voice I like. I go on a three mile walk. I will ask it questions about a topic that I want to learn about. If it starts telling me more than I want to hear right then, I will say "stop". It doesn't get offended. I then ask it something else. I find this very effective. I control it so it only explains to me what I want explained. If what it says sparks questions on a related topic I jump to a brand new topic. No personal tutor could keep up with this or adjust to exactly how I want to be addressed like Claude does. I'm very excited about the progress I'm making mastering new topics.
And yes, it is not that it is just presenting the facts. By me taking control of the direction the questions and answers go, I can flesh out my mental model. I won't retain every little thing it tells me. But I am much farther ahead than before.
Good to know other people who get migrane reading LLMs dense prose. I started reading books again recently, since everything online is polluted by LLM prose. What i realise, is that a human author, especially a teacher understands the learning pathways of new learners, they motivate the learning, and start from simplest concepts (a spherical cow), and then building all the complexities. This helps us to emphasize on most important concepts, while throwing away unnecessary complexities. While reading LLM prose is like reading a research article, that is written to an expert in the area, that talks about bleeding edge, with full of jargons, caveats, that just is not conducive to the learning process for a new learner.
Also they tend to assemble complex jargon in obtuse or meaningless ways, which makes reading and parsing and understanding much more difficult. Tends to reveal that LLMs fundamentally do not have "understanding", just likely word generation
To me, it's just Claude. The other models have their quirks but nothing is quite like Claude.
But even with Claude, it's it's really the prose getting in the way you can install the caveman plugin or tell it to use that "standard technical English" thing.
I wish I had something more methodical I could show. It's all subjective, but GLM-5.2 feels more human to me. Even GPT-5.6 Sol tends to be easier on the eyes for me (though the stereotype of it overengineering and no common sense are still true).
I tried using a new agent service recently and could tell immediately that it's powered by Claude due to the way it writes.
You can either install that skill or put the Rules section directly in your Global CLAUDE.md for Claude or Personalization setting for Codex and it should cut down the output verbosity by quite a fair bit.
The sycophancy is also a concern, it’s not really an impartial teacher, all its training is to suck up and maximize engagement rather than learning. The incentives are wrong.
I use the LLM to point me at relevant books and papers. But there is still a trust problem: I am trusting the LLM to point me at reliable, trustworthy sources.
I'm not sure I'm better off with humans though -- I'm not qualified to judge whether a source is a proper authority, not an I qualified to judge whether someone knows enough to point me to a reliable source.
It seems this is a fundamental epistemological problem to which there may never be an answer.
I have been using LLMs to help me turn my journals into interconnected notes and sometimes it is so confusing to read the notes that it doesn't resemble any human would write. Its like the models are getting stronger while also losing its touch to write human sounding sentences on complex topics.
> First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point.
Agreed.
I find Opus 5, and even Fable, to be overly wordy in eg PR descriptions and code comments.
However, I suspect that's more to do with what they are trained to do by default than LLMs in general. I have a little setup where I tell Claude to work together with Codex to tighten up prose and comments, and for me that produces much more palatable text that needs less human editing afterwards.
I think one problem is that books are not customizable, and many books are aimed at people with some certain knowledge. With LLMs, you can tell it what your knowledge level is and ask it to customize the answer for you. This is difficult to achieve with books.
> I really don't want to read anything generated by something like Opus 5 at this point.
Recently switched to OpenAI and I've gotta say Sol is so much better at writing than Claude. Opus has a distinctive sentence structure and Fable somehow manages to be even more obtuse. The personality of these models really does come through...
Claude code has a „fork“ feature where you can fork an existing conversation and keep talking in the fork and then you can go back to the original of the fork. You can fork as many times as you want and let LLMs write to a markdown file to keep important facts and learnings - also good for agents to do research without expanding the context window
While my advice is specific to learning about codebases, the way I do it is to have it generate mock data and put it in the local development environment, and give me some exploratory commands, and then ask away. It's a machine after all, so I don't have to read its preceding prose to understand whether it did tell me something, it can just repeat it however many times I ask it, and the hands on commands etc. give me something to actually try and implement.
I think there's an underrated difference between information generation and pedagogy here. LLMs are very good at producing more explanation, yet "more explanation" is often exactly what you don't need when learning something difficult
You can make something like the link below. I'd say it's a more than marginal improvement. It's still tiring, but it's much better according to my taste. I imagine everyone would have their own version of this for their own preferences.
It's a loop that uses adversarial review to check several dimensions of the writing:
> LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise
This is my biggest gripe with reading AI-generated text as well (ignoring the meta issue of whether it's worth taking the time to read something that an author didn't think was worth the time to write). It's gotten to the point that weird AI-style analogies just take me completely out of the text and kill my interest.
Even if you tell the LLM not to use LLM pros they (still) do it. If you feed the Wikipedia article on signs of AI writing and tell them to use none of those signs they will also (still) do it. I have tried (many times) to get an LLM to explain a concept to me, or a process, or an algorithm or what have you, and every time they cannot help themselves. Either they use LLM pros, or they get so verbose that it all just becomes noise and I spend more time filtering out unnecessary jargon than I do reading let alone learning anything.
For Gemini 3 this seems bit to be the case. If you use gems you get completely different personalities - so different that its almost scary. You can create gems which are really insulting, gaslighting or seemingly of a specific profession
I've found it helpfull to ask the LLM to generate a sylybus for the topic. treat the sylybus as a design doc for a price of software. i find they do much better when they have subtasks to focus on. they can do big picture and small picture, but they can't do both at the same time.
> First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point.
I’ve found the tone of Kimi K3 to be less obnoxious. Unfortunately it doesn’t wholly solve the issue, I don’t think any LLMs out there have a truly pleasant writing style, but at least not every assumption is “load bearing”.
Same. I usually just read the book along and ask questions on a specific part, rather than trying to get the LLM to produce an entire study guide for me. It seems to work better as a Q&A than a "teach me" advisor
Yeah, I am currently trying to work with Opus 5 to refresh myself on deep learning fundamentals, and... it's a mixed bag. I'm glad I already am familiar with the subject matter, as I can prompt for refinement and improvement. It is kinda following the Karpathy videos so far (a couple lessons in) but adding more math/derivations, which was what I asked for. It has trouble staying on topic, presenting information in a coherent/meaningful order, and providing all the context necessary to move through steps in its "course notes".
Like I said, I'm essentially continually prompting to refine the material. LLMs certainly continue to append, and never cut back. It just keeps spitting out additional content at me. So that's a bit annoying too. But I can basically get figure out what's going on with a few extra promps.
If youre curious what i've got so far... just be warned it is quite literally AI slop plus me continually prompting for clarification/cleanup etc. : https://github.com/cmoscardi/ai-for-ai
opus 5 doesn't follow instructions that great. probably needed that extra "creativity" to benchmax. if you are stuck on anthropic, try Opus 4.8 or Fable5 a try with the same prompts. very different results.
To get rid of the llm prose issue you can take a representative sample of its prose (say a question and its answer), rewrite the answer in the way you'd prefer, and add that to the system prompt; I did this to get my LLMs to compress down what they say, and now everything they say is very dense and to the point
anybody adds some prologue about what style of answers are prefered ? i know i often try to change the linguistic patterns because i too (unsurprisingly) am tired of llm prose.
>LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise.
This problem doesn't get talked about enough and is second only to the hallucination problem IMO.
AI produces so much noise to wade through in order to find signal, and the more expertise you have in a field the more that costs. That noise directly subtracts signifcantly from productivity gains.
And, I think the problem is directly related to the hallucination problem. It feels very much like an effort to kitchen sink the response in order to provide some value among possible hallucinations.
It also seems to be a byproduct of Gen AI operation. It just fundamentally doesn't understand what it's outputting, so doesn't know how to narrow down to the most salient bits.
It's just long. It just doesn't shut up. It's overly verbose. And you can't tell it to be concise or you degrade its quality.
If I ask what an integral is, the correct answer is that it is the continuos analog of a sum, generally used to calculate areas and volumes.
It should really be a single sentence, and then let me ask more about the terms I don't understand, and here's the beauty, in the previous one there can be only 5 terms I cannot know.
An LLM will vomit an entire page or more of explanation which isn't bad per se, but is an answer to something different: "give me a short introductory explanation to integrals". And that's not what I asked.
I call that vomit shotgun answers, text from which you have to filter out all the extra info the LLM wasn’t asked for. Luckily you can control that behavior and make it behave closer to what you want. Just ask the LLM how to ask for it.
The new Google Translate. They've made it slightly better at translating paragraphs of text but in many cases it's lost the basic function for translation: dictionary.
Try it out, fairly sure that if you out in 100 random words for 30 of them it will just refuse to translate them (it will copy paste the original word into the target language) or it will do silly things like use the target 4th dictionary definition instead of the primary one).
something to try that works really well is to put "explain this as if you're talking to a 5th grader" at the end of your request.... it just breaks down the text into more manageable sentences that can be understood by general audiences
I think AI is surprisingly good at this. I use voice mode while working out to learn complex topics, follow up with reading, and then go back to ask the LLM more questions. They excel at simplifying complex ideas and have endless patience. One hack I found is telling the LLM to test my knowledge by asking me questions—that gives me a clear idea of what to read next. Overall, they’re a great tool to use alongside traditional learning methods like reading books and working through practice problems.
This is on top of the issue of LLMs being a moron. like, I'm sure dumb people can learn stuff from them, but I'm sticking to books written by people who know what they're talking ahout and not vibed slop.
AI hater here. Too me this story is all to predictable. This is exactly how I thought using LLMs to learn would go. Or in simpler terms: well duh.
I'm actually going to make a prediction here as well. I think you will soon realize that using LLMs to clarify certain questions or ideas you have will turn out to have frustrations as well. And that you will soon direct those questions to either peers you know in real life or internet forums which are very likely to have a non-AI policy.
> I'm actually going to make a prediction here as well. I think you will soon realize that using LLMs to clarify certain questions or ideas you have will turn out to have frustrations as well.
Much more likely that people will believe themselves to be an expert in a subject after having had a conversation with Claude about it.
Wild because people wouldn't do that with a friend. Something about an LLM giving people info has this insidious stoleb valor to the LLMs work. Like that wasn't your thought, you just google searched it.
Specifically, you can set the tone and style to match what you like or used to after asking it to interview you, create an output, and then use that output as the project description or document to refer to
If the output can't be trusted, and you use another llm whose output can't be trusted to check the untrusted output of the first llm, then you're back where you started.
Yeah this seems to me similar to how the mortgage backed security risk concentration occurred leading up to the global financial crisis. Whereby the risk from exposure to low grade / risky single mortgages was eliminated via diversification but the diversification was simply packaging all of the risky MBS’s together and in no way diversified or de-risked the entire portfolio
I'm hoping that the Big Horrible Realization comes sooner rather than later, when we have less collective damage and pain riding on it. (Plus I'd feel personally vindicated.)
But the problem with LLMs is that they get the facts wrong. PhDs are PhDs because they’d look it up in an authoritative source, or actually find out through research and experimentation. The whole point is that facts aren’t a matter of opinion. The only people that argue over documented, findable facts are idiots that nobody should listen to.
Unfortunately, I don;t think that can ever be fixed. For an LLM to know that it is not hallucinating about something, it must know that that statement(s) is/are true. Which it cannot infer due to Godel's Incompleteness theorems.
Not really. Take hallucinations for example. If they are 1 in 100 (actually they are much rarer, but for the sake of argument), then the chances that 2 LLMs or even just 2 runs of the same LLM have the same hallucination is, well, a lot less than 1 in 100.
It's also relying on the assumption that the checking LLM only ever corrects wrong statements and never incorrectly "corrects" an already correct statement, which might not always be the case as well.
Are there any reproducible hallucinations on any of the currently available OAI/Anthropic models? I’m not aware of any.
And even if they are related - if Opus 4.8 always has a 1:100 chance of a specific hallucination - then running the same model twice does indeed dramatically reduce the odds of an error in the final output.
Even if your conditions are granted, different LLMs can easily consistently hold divergent "worldviews". LLM A might consistently state that fact P is true (i.e. it will never hallucinate a string that states or implies ¬P), while LLM B consistently states that fact ¬P is true. You don't have a way out of this with these two, and adding a third LLM is an additional confounding factor, not a solution. You can't settle questions of fact by survey, you need to test or use reliable sources.
If simply running things thrice-over was enough to stop "hallucinations" (and not incur other problems) we wouldn't be here talking about it today, it'd have been "solved" months or years ago.
Constant hallucinations. OpenAI:s latest on max settings. If you are to naively feed say, a short corpus of text to turn it into a parallel corpus in a few different languages, the original text gets subtly mangled and no longer matches the original. Say you have several hundred annotated sentences. Without hand-coding some regex to make sure that each sentence in the source column occurs in the original corpus you’re bound to get hallucinated sentences with an error rate that exceeds 1:100. Whatever you use as the output, JSON or XML, you will end up with columns that just repeat the original instead of translating it, especially for languages that are very close to each other or represent the same language.
Yes, LLMs can be SOTA for NLP, but you’re going to have to use them to write software or workflows that are more deterministic.
"Turtles all the way down" is a phrase of, I think, unknown origin (https://en.wikipedia.org/wiki/Turtles_all_the_way_down) about infinite regress or trying to patch up some bad theory by appealing to itself. Someone claims
that what holds the Earth in place is that it sits atop a giant turtle, and a skeptic asks what holds the turtle up, and the response is that it's turtles all the way down.
Personally I think this is a bad characterization of using LLMs to fix up LLMs because while you can never guarantee results this way (as the quoted line claims here, which is worthy of criticism), it is, in practice, useful to use LLMs on top of LLMs. And there's no infinite regress. Auto-mode in Claude Code, for example, seems to me like it's been successful at making the system more safe than --dangerously-bypass-permissions without prompting the user for permissions constantly.
There are certainly uses where it’s good enough, but you can never be 100% certain of correctness in the way that people claim you can by stacking N layers of these models.
What triggered my response was the “just review the output with another LLM and it’s perfectly correct”
I would have bet it’s a Terry Pratchett quote, and it kinda is:
“"The turtle moves," said Didactylos. "The turtle is a giant reptile that swims through space. It doesn't have to stand on anything. Swimming is what turtles do. The idea that it has to stand on another turtle, and that turtle has to stand on another turtle, is just silly. It's turtles all the way down, and that's a logical absurdity."
I read it as “if your LLM is being checked by another LLM, well then you need another LLM to check the checker. And can you really trust _that_ LLM? Probably should have an LLM to check the third one, and…”
It’s a reference to Bernard Shaw, who once said that if we ever created a truly artificial mind it would be inside a turtle’s shell. Sturgill Simpson covered the track on his seminal work, Xeno’s Paradox.
This guy remembers what I thought I was saying. In my defense I stole the whole thing from Stephen King’s It which I read … forty years ago, that can’t be accurate. Let me sort out my instruments and get back to you.
Agreed. Given how many significant errors LLMs make in my topic of expertise, despite my taking multiple error checking steps, the idea of catching 100% of hallucinations because you told the LLM to check itself is hilarious. It’s just a wild lack of insight: “I’m using the LLM to teach me something I don’t know about, I definitely have the knowledge base to spot any errors that might remain!”
Yeah. People with technical and/or tech business bonafides claiming that AI granted them expertise are so often taken at face value when they really shouldn’t be. Who told them that they were proficient — a chatbot? Someone who knows even less about the topic, so any expertise seems impressive? I’ll bet it wasn’t someone that actually knew what they were talking about. Even some tech reporters are tripping over themselves to be amazed, but don’t bother checking if they should be.
People don’t even have to be lying to be wrong about this stuff. Someone can learn enough about a topic to be halfway up Mt. Stupid in no time flat, and in doing so, think they not only truly understand the topic at hand, but might be particularly adept because they were such quick studies. People that know less are impressed, because why wouldn’t they be? Anybody that knows more than them sounds like an expert. And people that know what they’re talking about cringe at the overconfidence, and probably try not to engage: who wants to have to prove that someone’s boundless confidence is entirely baseless? Most of the time, they think the actual expert is full of shit because they think they’re the expert. It’s incredible how many times I’ve had people in tech confidently, even smugly “explain” design concepts and strategies to me that they did not actually understand, knowing I was an experienced, degree-holding designer… and they didn’t even have a chatbot’s lips on their ass telling them how smart and insightful they were.
Completely agree. While I didn't set things up to have AI review its output in a loop, my experience trying to build a specific acoustic testing rig with Opus 5 also aligns with the other "it's turtles all the way down" comment.
Opus 5 first built me a detailed plan, but a couple important details were either obviously wrong or felt unnecessary. I went back and forth asking for sources and more information probably like 4 times and every time it did the "in looking at things in more detail it appears my previous advice was incorrect" spiel. It just became exhausting at some point because it feels like it really lays bare how LLMs are just minimizing that loss function but don't actually "understand" anything. It was really useful as a search engine (it correlated some highly relevant source docs), but I just couldn't trust it to believe it was actually done at any step.
A second agent reviewing it adversarially resolves some context rot. Whatever they trained these LLMs on will just infinitely double down so I break it with 1 layer of checking and then a judge who looks at facts, since the checker is adversarial.
I know it sounds silly but 1 layer ends uo being way worse than 2.
I could certainly envision a scenario whereby review would increase reliability but not how it would every guarantee 100%, there is a pretty big logical gap there.
In my experience it depends on how much in detail you want to go. Chip manufacturing is a really opaque industry, so in this particular case LLMs might not even have the training data. However, using it for a high-level introduction into something is usually pretty safe from hallucinations.
I don't do animations, but I have an answer. You research a topic well enough to be able to understand if the result is OK or not. Usually it means figuring out some sort of testing.
I'm researching causal inference right now, and my main goal was to make sure I understand how to test estimation on synthetic data.
Basically, it's the same way it works with people. If you delegate a task that you don't understand, and you can't have a credibility proof (i.e. doctors, lawyers), then you research a topic well enough to be able to (1) define the task and (2) verify the end result.
Yep, that's impossible. The hard truth is that most people this lost to LLM psychosis cannot understand that fact. It's better to treat it like someone in a cult, arguing the facts isn't going to help if they refuse to accept them.
I thought after reading the title that the text was about learning something, yet the actual text seems to be about having a system do something for me.
I've had success using the socratic method. I give Claude some topic (say, how the intricacies of the bond market works, the content from which are screenshots of pages from a textbook), and then I go on a walk chatting with it in voice mode. Claude is the expert, I am the student. Claude asks me questions, leading me to an answer logically. I come back with questions, and we back and forth. LLMs arent like they were in '23-'25. I'm almost always skeptical its going to lie, and almost always wrong.
In particular:
- I limit it/encourage it to give me single sentence questions
- I sometimes will ask it to tell me a motivating, human-grounded story, when we're starting a new concept: claude responds "Maya is a bond portfolio manager, and her boss has asked her to quickly price in what happened if yields go down. She knows her bond's average duration, a measure in time, but she doesn't have a percentage, which is what her manager wants. How can she give him a percentage number with just a duration figure and the proposed new yield?"
- I'll often ask claude to let me work through it, to derive the thing myself, often resulting in a string of thoughts with "yes/no" trailers, to get the LLM to reply yes or no only, and avoid derailing my train of thought. If yes, my train of thought keeps going. If no, I've got something wrong.
- I'll sometimes stop and have it craft an artifact. I typically say "build me a Brilliant.org-style interactive demo of the topic", especially when we get into the realm of looking at the actual maths of a thing (for which prose and dialog is not optimal by itself AFAICT)
- I'll do this while I'm traveling, while I'm walking, while I'm doing chores.
It's a stupid word soup on second read, but hey, it works, and that's all that matters with "stochasty the parrot."
"""
In this project, I require a socratically delivered line of conversation. Here's the typical structure to the conversation. I ask some question. You need to factor and reason about how to conduct and deliver a conversation. Best practices would be to limit terminology, or assess with the user whether they have a firm grasp on terminology before you use it. You must be very strict about this, it's unacceptable to just introduce a new concept, actor, phrase or other complication into the conversation without first labeling who what or why it exists for the conversation.
Conversation structure needs to be front-loaded with a brief interview for the user, "you understand X?", "whats your understanding of Y?".
Conversation structure then needs to proceed with single-sentence questions from the agent. User replies with an answer. Sometimes the agent needs to correct the user, but only ever do so with yet another question.
"""
^^ these are the instructions I have installed at the root of a "project".
Keep in mind, this is claude opus 5 low effort we're talking about, in the "projects" area of the mobile app. Here's the process I use to set up its knowledge:
1. I take screenshots of the textbook on my iPhone, and upload a chapter at a time.
2. I have it summarize the chapter into markdown by analyzing screenshots. You could probably achieve this simpler, if you just had the textbook in PDF.
3. I walk with my boy Clau-crates.
I've done this for a couple of weeks and haven't seen it revert back into its typical context-dumping behavior.
On second read, there's probably some clean up I could do. Thanks for making me pull it out and look at it. Things that could probably be improved:
- tell it to cross-check resources online to further ground itself
- use simple, short sentences (long sentences make the brain blur a bit)
- not be sycophantic (it seems like project mode has discarded with my root-level anti-sycophancy prompt)
What’s everyone’s opinion on learning new tech things in this day and age?
My opinion swings between positive and depressing vision of the future.
I still learn new stuff, but I’m afraid it won’t have any value in a year or so.
For example, I’m pretty good at optimizing low level stuff, but right now you can just ask LLMs to do so and they are pretty good at it. They will profile the code and suggest reasonable options like 90% of the time.
They're amazing at it, provided you keep asking the right questions.
Trust me when I say that in the hands of someone who doesn't have your experience, the LLMs would not be getting the results you get.
You might think what you're doing is trivial, it may be sessions that flow roughly, "Instrument this, okay this part is slow, profile this part, OK read the profile output and suggest a better approach".
But your experience will be steering it in the right direction, and you're probably unaware of just how much your experience is doing that guiding, as the LLM shoots off at 100mph, you feel like it's taking you with it, but you will be guiding it a lot more than you realise, and that's where learning and experience comes in, even if you're no longer operating at the lowest depth, your knowledge of that layer will be helping.
If nothing else, the experience to know when something is actually slow is a skill in itself. If a function takes 200ms, sometimes that's as quick as it can realistically go, and sometimes that's literally a million times slower than it could be, and there's actual skill and experience wrapped up in knowing what "slow" looks like.
So there's this 11 year old issue in a forgotten ffmpeg plugin and I fixed it with deepseek by putting it into a self-testing loop. Probably would have taken me a few weeks to even understand the initial code to begin with. I haven't done C work in a long ass time and have zero knowledge of even what sub pixel sampling means.
With DS4 took me a few days and a couple of dollars. And by few days I mean I checked on it a few minutes every half hour or so a few times.
I don't understand the code it wrote but it's been in production for a while now and no issues so it's good. Ended up speeding up our video processing pipeline by 20-30%.
So while I do agree that it is cool you were able to do this, I‘ll say:
Sending in the patches but refusing to take responsibility for them is a surefire way to contribute to maintainer burnout. Please don’t do this. Either commit to fixing something and driving the PR to merge, or abstain from it entirely.
The bottleneck isn’t the speed of coding, and what you’re doing here is actively worsening the situation.
Except the steering itself is also disappearing, the same prompts from just 6 months ago now need much less steering, AIs are learning to even ask back in certain cases to persuade people with no experience towards the most likely correct choice
Generally speaking, the pattern is that people are overestimating how much "work replacement" will happen, and underestimating how much "work shifting" will happen.
What is fascinating is how you can witness it at so many levels of organization. One example: Employer executive get enamored with moving from labor to capital. They believe that by using LLMs, they can replace a lot of workers. At my place of employment, we have people that are surprised they can't file a Jira ticket describing a product ask, and have it kick off an implementation. You can build the skill to attempt that, but invariably you'll get back questions like "what do you mean by <x>" and "what do you want to do in this case, a, b, or c?"; questions that a product person or an exec are not well suited to answer.
In the past, programmers did that kind of interpretation and judgment call. So then you're in a quandary; who should do that work? Work that previously, you never imagined was an inherent part of what the replaceable code monkeys do at your beck and call?
And then, how do you hire for that? How do you find the training for the people that are experienced enough with... something... to know what a cohesive error response is, or what kind of telemetry strategy is best for that particular product and organization, what collection of product asks are incredibly complicated for what they're asking and can deliver 95% of the benefits at 5% of the work if we just do this instead, and whether you want to aim more towards thick or thin clients?
Who are those people? Wait, those are programmers? Wait, there's this whole collection of inherently human skills that we devalued, by not appreciating they were always quietly doing that for us in the past?
That's just one example. There's a repeating pattern of discovering where the work truly is, work that was embedded in manual patterns we might not have to involve ourselves with anymore, but is yet still essential. So the nature of our jobs changes massively, but the overall level of employment does not.
At least, not in the medium to long term. There is a lot of painful churn we have to suffer through first.
I agree with you in the broad strokes, but replacement by LLMs isn't the only way the level of employment can be reduced. If LLMs can make it so that two programmers can do the work that used to require five, that can result in a very large reduction in employment, even while having (some) programmers is still essential.
Only if there is no further additional demand created by the now lower cost of doing the work. Take lighting as an example. As it has progressed from burning expensive candles to now leds, our demand for lighting has continued to increase.
What sources do you think are good for this data for the recent past? My brief initial search is turning up a lot of contradictory data, probably because the sources are defining "programmers" differently from each other.
I can only speak for myself, so I hope this resonates with you.
I wasn’t even really concerned with optimizing low level code before LLMs and that wasn’t why I was hired either.
However following that low level thread: We can look at the reasonable options and immediately know if they’re reasonable or nonsense. Why? We know the code. Now zoom a level out, where I think our expertise really lies.
Building a complex system isn’t easy. There are customers with requirements, there are budgets, SLAs etc. Sometimes one customer needs X and one needs Y. Our expertise is taking all of this in, and producing something that balances all the different variables. It’s knowing that we’ll expect X events a second so we’ll need Y to ensure we can tolerate failure.
Is it possible LLMs will be able to do all of that too? Maybe. But then why would our customers need the enterprises they pay for?
I sometimes get vague ideas for solving maths/science problems. They never pan out but i can talk in detail on group theory and advanced maths and science topics due to investigating such vague ideas over the years. These days the LLM shoots the ideas down instantly and honestly correctly, i know enough to know "yeah that's right, oh well" and move on. Which actually takes away a huge avenue of learning. I'm pretty torn on the outcome of this honestly.
I'm not 'wasting time' but I'm also not really learning.
I felt this. I had a feeling for a long time that spider solitaire was somehow related to knot theory (legal moves being Reidemeister move equivalents and untangling). Llm shot this idea down very quickly! It was freeing actually.
Both of you are being overly dismissive of yourselves. Without either a proof or a program that proves the statement in question, I wouldn't trust the LLM. I wouldn't even trust if it produced citations, because I've already come across LLMs citing sources to back up statements without being able to find that information in the source (not saying it wasn't there, just that I was not able to find it).
There's a big gap between being able to ask questions about something and understanding something.
The more things you understand, the higher the chance you'll spot a situation to use them in the future.
I think the best innovations come from times when someone is uniquely able to combine two of their previous experiences together. The more experiences you have in your back pocket the more combinations you have access to and the more likely you'll have a unique combination when the right problem comes along.
There’s still an immense value in training the brain to learn and be able to approach new problems with the sort of procedural thinking that LLMs enable. We can explore topics that we are curious about and develop that sort of “muscle” to continue asking questions when we have them. I have no fear that when those bigger (and existential) problems arise we’ll be well equipped to keep asking questions and figuring out ways to solve them.
I'm choosing to (mostly) switch off the news and continue learning things I find interesting anyway. Maybe the world will punish me for it at some point, but I guess I'll have to deal with that when it happens. The alternative is too depressing otherwise.
So you think building a warp drive is pointless because 99% of work, per your judgement, will be done by AI? Is your contribution meaningless and artifact useless? I wouldn't think so.
It depends strongly on the prompt. Like recently I was also doing some performance optimizations, and if I did not mention profiling none of the LLMs even profiled the code; they merely read the code and assumed, based on their own analysis of big-O time complexity. It is after I explicitly asked for profiling that the LLM started actually profiling.
> I still learn new stuff, but I’m afraid it won’t have any value in a year or so.
This is silly. This would be like arguing that encyclopedias made knowing things pointless. I learn new stuff for me.
Professionally, it's important to know enough to know if you're going in the correct direction. Practically, tokens are going to continue to cost money and knowledge can save you tokens.
I mean, part of the reason the LLM can do that is because you know enough to direct the LLM to do so and verify the results to some degree, right? It's good to learn new things because:
1. It satisfies you curiosity (and curiosity is always valuable)
2. You can better utilize the LLM to expedite something you now have knowledge about
3. You still improve as an engineer/programmer/prompter/whatever
I still think it's very important not to outsource everything to AI because there is a lot of value in learning and doing things yourself which is an important part of life.
I have staff ranging from 10 years of IR experience to right out of college.
I can tell you that there is an enormous gap in ability between them despite them both using LLMs for daily IR work.
The reasons aren’t complicated. The senior responders have tacit knowledge of how breaches evolve and what to look for which gives them a much better framework for where to employ the LLM.
The juniors will normally start from “here are some logs, look for weird” which is fine but leads to tunnel vision and a lack of confidence in their reporting.
I don’t mandate that anyone do work with or without an LLM. I hire seniors based on experience and juniors based on interest. But my experience has so far been that our best up and comers focusing more on learning the technologies instead of leaving those details to the LLM are developing their intuition and understanding faster and in a more robust manner.
The biggest thing I've learned from doing stuff like this is that there are no shortcuts. At some point or another, to truly learn something deeply, you've got to dig in to the boring details and do things the hard way. LLMs can help with this...but I find it's usually tempting to try and just offload the boring stuff to them, which doesn't work.
Having an initial higher level understanding across the domain is extremely useful to contextualize the deeper stuff. I think boring details is a very leaky characterization, but I'll continue with it.
In my experience it's infinitely easier and faster to learn deep, "boring" things when you understand how they relate to your shallow and wide understanding of all of the related components.
The LLM is merely a tool. And you can use it for domain discovery that enables efficient deep learning at an unprecedented rate or you can develop a cursory understanding of a topic and think yourself an expert.
I’ve found that if I’m not struggling I’m not learning. If stuff is coming fast and easy that’s a sign that what I’m doing is not stretching existing skills enough.
It’s true of most things. Running, dieting, weightlifting being uncomfortable is a sign of progress.
This is also why apps like Duolingo don't work for most users. They want to turn language learning into a fun, effortless game but effortless is fundamentally incompatible with learning.
Learning is uncomfortable. Reading a difficult (for you) text in a language you don't understand is exhausting and confusing. But that is where improvement happens.
Its even worse since Duolingo added a life system (not sure if they still use it), where you were only allowed to make 3 mistakes before having to recharge your energy. If you get everything right, you are not learning, you SHOULD be making mistakes constantly. That shows you're actually being challenged.
> The biggest thing I've learned from doing stuff like this is that there are no shortcuts. At some point or another, to truly learn something deeply, you've got to dig in to the boring details and do things the hard way.
This is the only path to mastery, or understanding if one prefers. There are no shortcuts to a person achieving deep understanding (a.k.a. "Aha!" moments).
Can a tool such as GenAI be beneficial to someone who already has done the work to understand? Absolutely. But it cannot infuse mastery into a person simply by its use.
Only the time and effort a person devotes can do that.
Quite - you still have to do the work yourself. I think LLMs are best placed to act as an eager tutor that doesn't mind discussing a topic ad nauseam until you're certain you understand it.
yes i've found that there are a few topics that i've really been able probably 10x my understanding of using LLMs, in particular in getting me over hoops that are hard to navigate when solo, BUT I have to be really careful for it to not just show me the answer all the time.
I’ve been using LLMs to create readable rewrites of RFCs and specs that interest me. It is not precise enough for implementation use, but it has increased my understanding of the underlying RFC.
Another useful approach has been asking Codex to implement complex things, like a Kademlia DHT or BitTorrent client in a literate style with the explicit purpose to increase understanding by reviewing the source code.
The title is not representing what the post is about. “Use LLM to learn complex topics” here actually means that the author asks an agent to describe the problem area, and then implement a simple web-based simulation game, and by playing that game, the author actually learns about the topic and its constraints. They use chip making as an example.
Its fun but is it really effective ? I mean I checked the LLM one and I came out more confused about a topic I already know about, I find the best way to to learn using LLMs is to just generate an example try to somewhat get a mental model of how it works and then ground my understanding with traditional documentation and resources, its an iteration of a technique I used to do in college where I would read the textbook questions first to understand what is important and then read the chapter
I assume different ways of learning work for different people. For me personally, it's taking a piece of paper and drawing the diagram of how things work together; of if it's some math, then, again, using the pen and paper to follow the text. I can very much accept that for some people playing the simulation is a good way to touch the new problem space. I can easily imagine that for some topics, let's say, traffic signal automation, a careful simulation game will probably give more information than reading papers or manuals.
Game-based learning, described by Comenius, works if someone else prepares “a game” for you. E.g. like a dungeon master. :)
Otherwise you probably get more confused as you have mentioned.
On the other side, Peter Diamandis describes a situation where a bunch of kids were given a internet-connected computer and they had no teacher. Instead of it there was a “grandma” that checked kids from time to time.
After that there was a knowledge test that revealed “no teacher” approach was more efficient.
Thanks! The original title was "How I use LLMs to learn...", but somehow HN removed the "How" part. I even removed the initial post thinking it was a typo on my end and tried to post again, but I stumbled upon the same behavior.
I only use LLMs for explaining things if the subject is one I’m deeply familiar with and can independently and easily verify the facts. Doing so with new knowledge areas is very risky for obvious reasons.
For example, “explain how the code in this file works,” I am familiar with the overall codebase, I know the purpose of the file, and I can read it or write tests to verify if I suspect what it’s telling me isn’t correct. Or, if it’s really important, I can overcome my introvertedness and ask the team member who wrote it…but that’s a last resort nowadays, which I am very thankful for. In 99% of cases since at least Claude 4.2 days, Claude and Codex have been very accurate. Gemini on the other hand messes up more frequently and sometimes does weird things like try to delete files it’s not familiar with, at least the 3.6 flash model I’ve been using lately does this. But, code explanations are still good for the most part.
There is something called tacit knowledge that cannot be easily learned through LLMs. LLMs are currently very good for research and theoretical knowledge. For example, learning how to make mechanical watches involves a lot of hands on, tacit knowledge. That skill would also be transferable to building other precision instruments. I don’t think we can currently learn these kinds of skills easily with the help of LLMs.
Even if you know everything about bicycles their mechanics, components, and how they work you still have to learn how to actually build, repair, or ride one through practice. Knowing, understanding, learning, and practicing are completely different things.
There are also niche areas of expertise that can take years to develop,not just to the point where you know the terminology and jargon, but where you understand the nuances of the field, can recognize the "unknown unknowns", and eventually have the ability to push the boundaries of existing knowledge. Maybe that is what we should really call learning: not simply acquiring information, but developing enough understanding and practical experience to contribute something new to the field.
Ironically, on the same front page of HN, there is a post about Andrew Wiles and this. I don’t think I would be able to comprehend Fermat’s Last Theorem, the Poincaré Conjecture, or Gödel’s Incompleteness Theorems, even with the availability of LLMs.
For learning, I find LLM's helpful as a knoledgeable "tutor" or "friend" during the reading of a complex paper or book. Before, when I got stuck, I spend lot's of time going around different books/pages/papers until I was able to find a simple enough explanation that allowed me to "catch up". Now, I can ask the LLM and then continue from there.
Learning from a book is still the best... Although I have been told that learning a complete subject from a book is now an inproductive use of my time... Perhaps they are right, but I still do it.
My main issue with using AI as a learning platform is that unlike documentation, books, Youtube videos, there is not really a process of having someone "review the learning material". For example, I can always read the review of some book or ciriculum, the comments under a video, or if it's some for of open source documentation you can check the PRs and verify to some extent it's claim. With AI I can't really say what it has halucinated, because I am learning a new thing, I don't have that benefit of previously reviewed material.
This is (exactly) why I very strongly tell people not to teach themselves with an LLM. Particularly from the ground up. If you do not understand the domain, you cannot learn from the model because you won't know what questions to ask and it certainly isn't going to answer all of them for you.
Of course, and that is what I do, but it becomes and additional mental and time consuming effort. As an dumb example, if I am learning a new programming language and trying to grasp some concept, I'll check the docs, find the section and read it. I trust that the source in the docs has been already vetted by other devs and the authors. But if I ask the AI the same thing, I then need to verify it's claims usually by asking it to check if the info is true, and then possibly opening the source link (lucky for me Claude provides the links in the desktop app as footnotes). It's not a question about AI, it's about the trust I have of this tool, it builds over time, but as soon as the AI makes an assumption or a hallucination we are back down to square one.
I assume this will become less of an issue in the future as there is more trust between the AI tools and me.
Wrote about this awhile ago, and it hasn’t changed:
> Time and time again, when talking to people who rely on ChatGPT, Claude, Perplexity, and other general AI tools, I hear them say, “AI is incredible. It handles nearly everything I throw at them.”
> “What does it fumble with?” I’ll ask.
> “Well, it still gets things wrong when it comes to my line of work.”
"Complex topics" in this case means reading 22 AI-generated paragraphs that supposedly cover the entire chip manufacturing process. If the author seriously thinks this level of detail is complex then they have psychosis.
Imo the podcast and the video are better served as background material for some other task. The low bandwidth becomes an advantage because it's often ok if you miss out on some parts due to lack of attention.
Indeed it's often a waste of time to just focus on talking people fully if you want to learn fast, reading and especially deliberate practice are better for that. But if you don't have the time, energy or focus, then listening to interviews in the background can be useful supplementally
One of the most exciting things about LLMs for me is to have someone with which I can discuss about technical/scientific/engineering things. Since usually I do not have a human at hand (or do not want to annoy people around me).
Especially conversations that involve formulae can get really interesting and insightful.
Of course, you have to be careful with the answers. Especially when the discussions get longer. But usually I see that it is time to stop or to start a new session when the formulae do not make so much sense anymore or when the LLM repeats itself.
But with enough caution, LLMs are really a not-so-bad intellectual sparring partner for discussing ideas and insights.
Check the shorter "Advice on upskilling" or "The Math Academy way" for well researched approach.
So what works for me
- open a project in ChatGPT/Notebook LM
- dump all the relevant and highly cited materials (textbooks, papers)
- dump the advice on upskilling text or a short summary I've written for the LLM
- create "Learning Goals", that contain what I want to learn, and how to estimate is my level good enough
1) Ask it to create a learning path from the materials, following the approach. Give that to an adversarial LLM for cross check. (just for sanity check)
2) Ask it to create an "entry test" to check what I do know and what I don't
3) Iterate step by step on each module/submodule from the learning path that intermingles the approach of: small theory step + small practical task + small test. Log what's missing/wrong in my dept log. Give the dept log at the end of the session to the LLM to incorporate/create another test/task.
What I have found useful in this approach is that it will generate a lot of practical tests/tasks for me and it will explain a concept in many ways until I understand it. Also it finds some prerequisites I might miss, but based on my tests and debt log unexpected things I thought I understood surface.
So with the limits of LLM and while building a mental map of the relevant parts it's usually enough to spot the hallucinations, but if you apply structured approaches these are minimal. And it's super good, because the number of practice tests and explanations is endless.
The interfaces are a bit clunky, but current multimodal LLMs are ok with images or even hand writing.
I’ve written a skill that I basically feed what I’m looking to do, some ideas I had for accomplishing it and any other details like tech stack, etc.
The skill then riffs with me, judging my ideas and suggesting alternatives. We go back and forth until something useful comes out of it. This process isn’t unlike how I do normal development.
However, once agreed it breaks the work into “steps”. It then creates a tutorial for me, for those steps, explaining each line, why each change happens etc. I can then ask questions, muse about an alternative idea etc. Then I do the steps, and I’ve learned and gotten what I wanted to get done.
This has been how I’ve been learning Godot and making a game for the past month or so. I didn’t go in blind, I started with a course from GDQuest so I could feel confident guiding the tutorials. I will say though, having a tutor to bounce ideas off of has been really useful.
I still try to figure it out myself, consult the docs, discord etc. But if I’m stumped I’ll run my tutor skill and have some fun.
I’ve started doing a similar thing after reading a post on hn about manually applying the code so that you actually understand it.
I have done this for all my work this week and it works quite well.
For one it lets you actually query the LLM as to why, their plans give a high level not every single change and it allows you to correct it as you go and the plan will change.
There have been numerous approaches doing things akin to that. I remember my university working on such a project. Or at least a similar one, representing a code base as a street network with traffic showing how data flows through the system.
I too, would like to have such a tool for viewing larger projects where the flow can be cumbersome to reason about. One issue I guess would be finding the right level of abstraction in the representation.
The author says it's "100% accurate and free of hallucinations" but I am sceptical. Although I don't know much about chip production, when I ask LLMs about advanced topics like memory order LLMs tend to hallucinate more than entry-level questions. But the point is that I didn't found that LLM hallucinated before knowing it deeper. It's possible that LLM hallucinates but you don't find out because you are just learning it.
Buildings are a fantastically complicated and interesting complex problem space. I found my architecture studio students using it to ask technical and code questions about the buildings they were tasked with designing. I observed that the inaccurate information it was returning was compounding...
To be clear, I say "inaccurate" rather than "wrong" in this case because even if the information it returns is factually correct to the question being asked, students don't have an understanding of the complexity of the interdependent tectonic, regulatory, and spatial / experiential factors of a building sophisticated enough to ask their questions of the specificity and nuance necessary to get a good output that addresses the entire problem.
Anyway - with the students still learning to ask questions the right way, and the conditionally-incorrect facts making their learning more complicated rather than less, I hit on a strategy for them to use LLM's that seemed to help much better.
I suggested that instead of ask the LLM for the factual answer, or even better for the facts and an explanation, that they ask it to direct them to the proper place in the source material to find the answer themselves. Then, to treat it like a lab partner. IE:
Hey Claude I'm looking for "x."
Claude: "look at foo, bar."
Thank you - chapter (foo) part (bar) table (goo) says "car." However I notice that footnote (hoo) says there's an exception if "dar." Which is what I have. Walk me through this exception...
It seemed to have good results as a guide to understanding the disparate bodies of knowledge that they will eventually have to keep together in their heads and work synthetically and non-linearly through, rather than just as an external source of blindly trusted authority.
I've been using them by reading some docs/wiki/tutorial, then when I think I understand something trying to do a rough explanation to the LLM and ask if I'm right. I'm usually making some analogy to something I already understand a little. I'm usually partially right but missing some key bits at the first pass. I go back and forward asking for explanations of various bits or asking for resources around the area I'm not understanding. Often times just discovering the relevant name for the area of study opens lots of doors. I basically use it like I would talk to a knowledgeable and patient teacher.
As for how useful it is to understand thins, I believe it's still useful and hope it will continue to be.
The little tool it outputted is nice, but click around the stages and the text is not high quality at all. The snippy titles, abbrievated explanations, I wish a few more iterations and thought was put into the actual main textual content. Especially for 'complex' stuff
I use it by telling it my background, giving it a rough timeline and asking it to create a learning timeline, save progress along the way, and git push / pull periodically so I can use the same thing on both Linux and Mac. I tell it for each phase in the learning timeline, present me information, then challenge me on it. If it's code, it challenges me with a coding challenge, where I use it in an IDE plugin. If I'm learning something that isn't strictly code, then I ask it to give me info, then challenge me with questions and grill til I get it right. I ask it to save what it think I struggled with, so that later we can drill it again and I can also review it in an .md file.
Does anyone else use Claude like this?
It's sped up my learning by 10x. I struggled with 'just reading a book.' Take kubernetes. I hemmed and hawed and spent years periodically reading some dry book or blog or official doc, falling asleep, and forgetting while I got busy. Now I'm aggressively working with it, almost like I'm addicted to a gamification, of getting through our learning timeline, and I'm excited to move forward as quickly as possible and pass its tests.
It's like a fake teacher, because I can also ask it to drill into a topic or re-explain itself if it made no sense.
The only thing that worries me is, sometimes I'll say something like, "Um, are you sure about that?", and it'll apologize and correct itself. I barely challenged it!
These animations are great. I have learnt so many things from LLM's, cross checking things is easy enough, but the hallucinations are really not much of a thing any more (in my experience). When you deep dive on things it does seem to get very wordy sometimes (this is claude anyway). Its taught me flutter and dart without opening a book (with the occasional reference page), set me straight on monads finally, refined some linear algebra, various bits of history, and philosophy, I'm learning spinors at the moment. Is some of it wrong - maybe, but it's not like my brain is 100% accurate any way, and when I need accuracy I look up references. It is fantastic getting a broad overview of a subject you don't know or a precis of a current subject, and its so much faster.
Hallucinations bring to question what you think you've learned. That's going to cost long-term if you labor under mis-apprehensions until you maybe figure out you learned something wrong.
where's it mentioned on that link? I can't seem to find hallucinations (don't worry found it).
Yes, sometimes its wrong, most times its right, cross checking is fairly easy, not using it because of the possibility its wrong seems a baby with the bath water thing.
I learnt a lot of functional programming from it, stuff I've always wanted to learn, but just didn't have the time and really the sources can be difficult, it really explained things well, and as someone else said in this thread, you can ask questions over and over until you understand, asking a person that (if you can get an expert) would drive them nuts. Maybe my experience isn't typical, its hard to tell, everyone reports something different.
TBH I was expecting something different. When I use LLM for learning I ask it to make questions for me. It's genius in these tasks. I moderate it to not kill me with difficulty but otherwise it's friendly to help me learn and understand
I use LLMs to learn deep technical concepts. I really like them because I can spend countless hours a day understanding things and building an investigation file with all my findings. I code examples and test the findings. It has helped me understand basically anything.
I'm using LLMs right now to build a terminal browser, a GUI browser, and a PyTorch/LibTorch replacement. It's really fun to be able to learn and make progress this way. It's like reading multiple interactive books, where every concept can be explained again and again until I understand it.
If you're using LLMs to learn or for research, and at some point you don't end up engaging with an actual resource (books, papers, lectures, web pages, etc) then you're playing yourself.
Sorta. If its response is grounded in actual material, and you're thorough, it's not so risky. As with all learning, trusting one source is a risk in itself. Hell, I didn't even trust my physics textbooks in college. Physics.
This looks very interesting, something I was also trying to do with my learning.
One thing I wonder is, do you mentally 'fight back' monotonicity of your interactive tool? All seem to be in 3D space, with low-poly, like in a factory moving through the belt and giving you an information + textual description to read more
But sometimes you want to visualize the charts, or graph of simulations, or maybe even the parts of an item in the rocket.
I am very interested in figuring out how people use LLMs for learning. I definitely have the knowledge, but it is severely autistic in a way.
OTOH, I am curious if there's a "practical value" to this exercise? If the LLM already contains the information and implementation knowledge to implement the networking stack inside an FPGA by itself, what value do I gain by learning about HDL, TCP, the bespoke Xillinx tooling, reading the documentation, reading papers on the implementation and going through every bit of details and theory.
I feel like there's a meta skill that is more worthwhile for "practical value".
Gamifying the presentation could make topics more accessible to others. For me the overhead wouldn't help with my own learning. Also I've been burned by just learning things mechanistically (e.g., coding, applying algebraic rules), so I'm leery of learning just by making flashcards or models of the topic.
I find LLM's do great for learning when I ask what are the principles, how the main applications work, what are the key drawbacks, where are the growth plates in the field, etc. - the kind of thing a good advisor points to. Sometimes I have to ask it explicitly to use topological order of topics and show relations, which often highlights the gradient changes in the learning curve. For pruning, it's surprisingly good applying philosophical heuristics - Occam's razor, or Derrida's differance (the difference that makes a difference), etc.
And finally, no learning is effective without problem sets, and for those LLM's at times get me over blocking issues.
The degenerate case is memorizing the glib phrases regurgitated back to me; they're helpful and functional enough to get me into real trouble!
I just gave a simple prompt to Kimi k3 when it came out, and then just using it and asking it to improve based on my own taste/gaps etc so it’s a long drawn out process. I’m doing that for the first one, will iteratively keep improving it myself as I run through it. Maybe will post on HN after that.
I usually learn compelx topics using the image generator of an LLM, capabilities like GPT-Image2 or Nano-Banana.
I give it like a complex paper -> turn into visualzation or a poster, then ask quesitons and it helps me understand someitmes a very complex paper rather easily. The jump that nanobanana / gpt-image-2 had done is pretty wild.
The main bottleneck as an engineer is no longer writing or testing code. It is how long it takes to understand complex systems. This is a really nice approach that, if you have the tokens and the patience, feels like I great way to learn something and I think we'll see more and more stuff like this.
I had a similar realization a few months back and am working on a tool that generates "mermaid walkthroughs". It is 1000% less pretty but it is fast and is pretty good at explaining how services work or what a code review does or just as a way for your agent to explain some decision to you.
I wish there was a LLM tool to explore a topic recursively, as a tree or a mindmap. You would start with some high level concept (say "cryptography") then dig further and further to more specific topics.
I think that would be a way more natural way to explore than being stuck on the classic linear output of a LLM.
Those animations are... series of steps, each with a description? How different is this from a plain text list? It's not that you can see forks or that the layout of the assembly line gives you more information.
If anything, a plain text list is easier to scroll backwards and forward?
> In plan mode (using CC, or OpenCode) I ask a model to build the foundational knowledge for X topic. I ask it to review the accuracy of the knowledge base it built in the previous step.
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
It depends what you mean by “learn”. To have a superficial idea of the high level concepts of a step-by-step process, sure, a 3d visualization like the ones this author makes can work. But it stops there.
But AI will be better than you at those topics as well, and when someone needs an expert in that topic take a guess who will they approach in such scenario.
I don't think we are even that far when the complexity AI can handle surpasses 99.999% of what humans can handle, where AI make e.g. physics discoveries beyond the grasp of most humans and it will have to "dumb it down" when talking with humans -even physicists- but not with other AIs
One concern though: "100% accurate and free of hallucinations" is doing a lot of work. A second LLM pass can catch some mistakes, but it can also confidently agree with the first one
For everyone who thinks you can’t learn with LLMs, Dr. Cat Hicks, psychological scientist and author of the recent book “The Psychology of Software Teams[1],” who worked at Google and founded the Developer Success Lab at Pluralsight, has written two skills called learning-opportunities[2] and learning-goal[3] that use validated learning science to help you learn while using LLMs.
Cat also has an awesome podcast with her wife, Ashley Juavinett, Phd, called Change, Technically.[4]
I encourage everyone to check out her work! She’s dedicated her life to helping software developers get the support they need inside organizations to be seen as humans, not just robots.
Something that I have realized recently is that it has become so easy to get an answer to almost any question with the help of chatbots that its almost unnecessary to spend any effort thinking about the problem or the solution. I feel like before when I had to spend time researching a problem to find an answer I learned so many things around the topic itself which helped me understand the problem itself better and gained a deeper understanding. Today it feels like you can have an answer to the most complex questions you might have, yet you gain a superficial understanding of the topic and might forget about it quickly.
Personally I'm excited about these sorts of experiments. We all learn in different ways, and these sorts of techniques allow us to create "on-demand" syllabuses and lessons that fit our learning style and learning level.
It's not perfect, but I'm optimistic this will be a useful way to teach/learn in the future.
And, to be clear, I think this will be best utilized within a group/community setting. I don't think it will replace teachers or classrooms.
having llm audit all my work/code, and generate review pages (as a teacher) of before/after with working examples is incredibly useful. Its something no course can do for me, even a teacher wouldn't have enough patience to go through each one of my mistakes.
having things defined/have correct solution to compare for review is useful, and keeps llm on track. Don't think i would trust llm if it were reviewing it all on its own
This guy is severely milking it now. If learning means building an inaccurate and incomplete understanding of the topic then go hog wild. Otherwise https://news.ycombinator.com/item?id=49209049 sums up my feelings about the author's attitude.
For background, in 2020 the International Maritime Organization changed fuel standards globally. The intent was to reduce pollution, which is great for human health. But, it turns out some of that pollution was supporting cloud formation and creating a cooling layer globally. And temperatures started to climb when the pollution was removed.
I haven't looked through how it's doing the calculations yet but this kind of visualization is full of possibilities if done right.
And since the details of ship tracks are a newer topic for me, I'm curious about Gell-Mann Amnesia. Next I'm going to apply this to a topic I understand well and will report back. Stay tuned! (will be tomorrow, bookmark or reply to this thread)
Really neat idea, I think it is one of the best ways to exploit the combined building and explaining capabilities of LLMs.
I am currently building an app/game to explain friends and family concepts around wealth management and wealth building. Games are a great way to hide complexity while still including it in the « guide » you are making.
i looked at the animations, they look cool, and i don't think i will enjoy learning things that way. as someone else said, there's a lot of content already produced on these topics. i also think the level at which these animations are playing, they are actually hiding the 'complexity' of these topics.
Very cool, I like the visual learning nature of this and the auto play once starting. The game graphics are engaging which counts for a lot these days, I feel my attention span suffering after using agents for the past year.
I've been working on a similar process of pushing to github pages, but focused more on having "practice sessions" with coding blocks to test content. Using webassembly and mock servers to mock backend endpoints Here's one I built to build a full stack llm chat system in the browser.
I tend to ask the LLM for a single HTML page explanation, with a pedagogical approach. Something about dropping the word pedagogical leads to a more structured outcome, but I haven't quite figured it out why yet.
My favorite way to learn infra topics at work right now is asking for a humorous analogy involving monkeys and bananas. I tend to remember the result, and it gives me reference points for new topics.
LLMs can help you understand a language, but they can't replace learning the vocabulary. Words and phrases still need to be learned the old-fashioned way: repetition
I don’t think I really agree with the author’s approach here, but I will say LLMs have been a huge help to me as I’ve been reviewing linear algebra and diving into signal processing. Anything in a textbook that I don’t fully grasp or am confused about, I just take a snapshot or copy paste then ask a model to derive it or explain it in different terms.
It reduces friction a ton, but at the end of the day I’m not skipping anything.
haha, I did the same to study new topic and then realized how personalized education will turn out in the near future. The only thing left is keeping a curious mind and ask a lot of questions !!
The selected topic (chip manufacturing) is being presented at a shallow level at best. Go read Wikipedia’s article [0] and see how deep and broad the topic is. It also skips quite a few steps, and utterly glosses over how insane of an accomplishment EUV is.
This is my biggest societal issue with LLMs: they allow you to think that you’ve “learned” a topic because you read a lot of technical terms. I don’t think it bodes well for the future.
YouTube has so many truly wonderful videos on chip production. I admire your approach but it seems like a lot of people are in this ai maxxing phase where they reach for ai for everything despite their being ready, high quality things already available for free
I'm also not sure you can really learn chip production from widely available public information. It's a hugely complex industry where the details tend to shape larger strategies. For example, you can't really understand the relationship between Micron and TSMC without some awareness of the trade-offs of memory processes for peripheral transistors.
YouTube is just a big dump of information. Having a structured way to learning along with interacting helps you learn. Otherwise you're just binging information
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
How do you know if you're learning this for the first time? Very risky to learn from LLMs. I've done it, but you have to keep your wits about you. Lots of "oh of course you're right - what I just told you was completely wrong".
i've asked llms to write a presentation for me on a given topic. for whatever reason, that finds the hidden layer responsible for explaining things well.
That seems like a terrible way to learn. It’s a neat animation but cmon, there’s like educational TV programs from the 80s that explain this so well, in Germany there’s “Sendung mit der Maus”, not sure if they have a segment on chip manufacturing. But in these clips you can at least see the real stuff instead of some half wrong animation, just let the LLM write a few paragraphs for you or better find an ACM article or book on the subject, probably still takes less time than coming up with that animation…
God does everything have to be productized and glorified as if you’ve invented a new way of learning. Read some books!
I had an internal company assessment I needed to pass before end of our fiscal year. The study material consisted of 10 ppt decks about 80 slides each (so around 800 total). I had an AI read all the decks and compose a study guide with quizzes along the way. It came up with a 100page word doc that I used in place of the decks to prepare. It worked very well for this including, like you said, quizzing me over various sections.
(Yes I confirmed it was ok to use AI with the material)
This animation is worse than useless. I've taught a lot of students. This is not learning, it's stamp collecting.
You're just memorizing a nonsensical recipe. What are the constraints? Why do we do X rather than Y? How does a particular thing scale? etc.
All you're doing is fooling yourself into thinking that you've acquired some knowledge. When in reality you haven't even learned the basic mental model to reason about this stuff.
You've learned something when you have a mental model that makes correct predictions. Until then you've memorized it at best, and as with most memorized things it will decay exponentially and will be gone from your memory soon enough.
> I personally find the style used by LLMs to explain things difficult to follow. It's just too simplistic
I also struggle with LLMs explaining things, but for the opposite reason.
I consistently have problems to get short, precise but plain/simple answers.
Instead I'm overwhelmed with walls of texts, often filled with jargon that is a mixture of imprecise and unneeded.
The style at which I learn better is by asking about stuff interactively. I ask you what something is, you give me a 3-4 sentences top answer. Then I explore and dig into the topic from your answer on the things I want to know better.
i had Bolt create an interactive app teaching me Oberon language but despite several attempts to fix issues, there were still glitches in unexpected places.
So the experience was... meh.
Then I went back to a book written by humans (Eric Nikitin's "Realm of Oberon").
I am greatly disturbed by the idea that any of the demonstrations involve "complex topics". Whether it was the silicon workflow, LLMs, or EUV, I saw nothing complex. Are these supposed to be freshman undergrad or even high school level explainers? I suppose I didn't see any glaring errors on a quick glance, but definitely lots of details were glossed over.
Perhaps basic special relativity could be done this way, or simple derivatives, but definitely not general relativity or integrals, much less PDEs. I guess I was hoping for a 3B1B type output. Oh well.
This is, without exaggeration, probably the fiftieth blog post or long-form comment about how someone is using an LLM for "complex learning", and I'd just really like to see at least one of these to be accompanied by a statement saying what are the kinds of problems the author can now confidently solve that they couldn't before.
In my experience, LLMs are really good for taking up your time and making you feel like you're learning, in the same way that many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
If you ask an LLM to give you a 500-word summary of quantum physics, it'll give you an oversimplification that probably leans on a hodgepodge of pop-sci metaphors. And if you start drilling down, you risk drilling down on these ELI5 metaphors, which can get you farther away from truth.
I personally don't get this attitude. For one thing, nobody on earth can give you a 500-word summary of quantum physics in a way that lets you internalize even a tiny bit of it. That's not how you learn anything.
Here's how I do it: I open Baby Rudin (3rd ed.), second chapter, and read the main text - absolute brutality. I unpack almost every sentence with Claude/GPT until I finally get what's going on. No ELI5 nonsense, just examples and counterexamples galore while absorbing the techniques and the way of thinking in analysis/topology. How do I know I've learned the material? By solving every single problem in that chapter. Here's the thing, though: the problems in Rudin can be brutal and decoupled from what's in the text, so if you can handle them, you've definitely mastered the material. No 500-word summary of analysis here.
RE: On agents taking up your time
One thing I've begun to notice is that LLMs list of a bunch of interesting stuff and raise all these thing you consider but often sometimes you just want a more focused response, so these scatter responses kind of lead you to being overwhelmed and losing focus on what you really wanted to do. At least I've started noticing this.
Like I'd ask about some statical approach taken in a paper and suddenly i'm being bombard with all these potential pivots and things I really need to consider, I kind of just want to consider 1 thing at a time and come to things once I fixed the immediate issue. Sure I have no doubt these other pieces of information are useful but it's just not the most useful information I need right now.
This is less of a problem with coding agents more so putting learning related questions to an LLM, like is the method covered in this paper, yes no? instead I get an exhaustive but overwhelming and indirect response that contains part of the answer. I just wanted to know if it was worth my time going through the paper but now I'm being bombarded told all this tangential information, which is unclear to me if I need to consider right at this moment, it's really distracting.
Maybe this is something others adapted to but i've resorted speaking past it saying, "this is the question please stay on topic" or literally "one thing at a time please" and then it narrows in, but they really stretch your attention thin if you're not more aggressive with keeping them on topic. The smarter models are better, and if you use max compute it does a better job.
I think they can definitely be helpful for learning, but you got play an active role, you can just consume what it says like content.
I’ve found it helpful to ask LLMs specific questions about your knowledge gaps, or ask for concrete worked examples that illustrate the ideas you’re learning about.
In [0], I ask: „When applying Hidden Markov Models to POS tagging in NLP, what do the latent states and observations usually represent?” I then follow up with some specific questions and requests for walkthrough. You can’t see it from this conversations, but I have Wikipedia and a bunch of other resources open in separate pages, cross-reading, and I follow up with a handwritten toy implementation of a Viterbi-based POS tagger once my mental model crystallizes. This is very different from a 500-word summary of quantum physics, and I still had to put in effort (this is unescapable!), but I found the experience rewarding. Also note that this is relearning of a topic that was part of my uni curriculum but long forgotten.
In [1] and [2], I’m learning Spanish by reading García Lorca’s poems. Here again I’m going through the texts with a dictionary, and augmenting my learning with what a dictionary won’t tell me: given the usage of a word or phrase in this specific poem, is it something that could occur in everyday speech, or is it poetical?
[0]: https://chatgpt.com/share/6a743bc7-d0dc-83eb-acc9-8f2faaffc4...
[1]: https://chatgpt.com/share/6a731be6-26bc-83eb-8d21-c965da5364...
[2]: https://chatgpt.com/share/6a731c01-5604-83eb-a5b8-cd1295d0ef...
100% agree. They are not good teachers. Nevertheless I've been using LLMs more and more to learn complex topics, but my learning is always anchored in something else:
- Getting through textbooks and lecture notes. LLMs have gotten very good at answering basic questions on quite advanced material (e.g. representation theory and quantum field theory). By asking a very specific question or even giving the LLM a screenshot, I can get unstuck a lot faster.
- Learning e.g. new python packages. Instead of hunting for examples on Stack Exchange, now I ask an LLM to write a minimal working example and then build off of that. By writing most of the remaining code myself and only using the LLM to answer questions, I've been able to learn new packages significantly faster.
In both cases, the LLM isn't providing the curriculum or guiding what I learn. The textbooks, papers and coding tasks are. But now I can pick these things up much more efficiently.
I think it depends on what you want to learn and what your definition of "learnt" is.
The other day I realised I had no idea how DNA and life works. I guess I studied it at high school (25 years ago), but maybe it didn't go into much detail or it just didn't click.
So I asked ChatGPT to explain it to me, I came up with my own mental model from it's explanation, told it that, then it corrected me where I misunderstood things. We went backwards and forwards for an hour, me asking questions, it correcting me, until I felt like I understood the whole picture.
Am I going to become a biologist and study the origins of life from that? Definatley not! But if my kids need help on their biology homework, I now understand the basics of it.
Can you say in all honesty that a read of the Wikipedia article for DNA would have been less helpful? It's less convenient perhaps, but definitely more authoritative.
>So I asked ChatGPT to explain it to me, I came up with my own mental model from it's explanation, told it that, then it corrected me where I misunderstood things. We went backwards and forwards for an hour, me asking questions, it correcting me, until I felt like I understood the whole picture.
If you're only checking your understanding against the one source you used to obtain it, how can you tell whether your understanding coincides with reality (or rather, with general scientific understanding), and not just with the source you read? And I'm not asking just about ChatGPT; the same question could apply to any source. Books are not exempt from containing errors.
Concrete high-profile example: in "Surely You're Joking, Mr. Feynman!", Feynman told of a ball which, in a Brazilian college-level physics book, was described as having a 40% higher acceleration that it would actually have in practice.
Turns out that the author had done a thought experiment but neglected to factor in the rotational inertia.
How can you really say you you've learnt the alphabet if you haven't read the document in which the first use of the letter Y appeared?
Because you can successfully use it to communicate with other people. That's the grounding truth. If you'd only ever read about it and never actually employed it then yes, you would not be able to make that claim.
That's exactly what OP was doing, or does teaching kids not qualify somehow?
It's not the same thing. A language is a closed system, it just requires that the participants agree on the rules in order to be successful. Biology is not a language, it's a field of study of phenomena that exist independently of minds. If you read a biology textbook (or an LLM's output about biology) and then relate to someone else what you've read, you and the other person can discuss the topic, and be no closer to ascertain whether what you two understand has any basis in reality.
Biology is in some sense a language, how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?
In any case I think you have an overly narrow definition of learning that we're unlikely to come to terms over.
>how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?
Those boundaries are completely imaginary and don't exist in reality. In reality there's no biology, there's only elementary particles interacting physically. Whether you agree with someone else to classify a phenomenon as biological or chemical, you're not refining your knowledge of the real world, you're just performing an organizational task.
But, say, how many times the tympanic ear evolved independently is a real phenomenon that can be investigated, and it's something that you're either correct, incorrect, or ignorant about. If I tell you it evolved five times, what more can either of us gleam about the real facts by just discussing this factoid back and forth, if neither of us has access to any additional knowledge or way to put this datum to the test?
He's talking about learning that makes you operational. If you can speak with someone, then you've learned language. If you can get a computer to do something, then you've learned some programming.
The equivalent for biology would be to grow a plant or a few plants and animals successfully. That's operational at a certain level, you could also be operational at a lower or higher level.
No, it doesn't necessarily need to be applicable, it just needs to be testable. The size of the Earth for example may not affect you directly in any way, but it's something you can verify to some degree of precision. But if you just read about it, can you really say that you know it? What if all you believe about the size and shape of the Earth is from what you've asked an LLM? Do you still know it?
EDIT: Perhaps not the best example, because the size and shape of the Earth are data that are repeated often enough that an LLM would be unlikely to quote it grossly incorrectly, but I think my point still comes across.
Same question applies if you only check it against your children's biology books
Uh, yeah. That's what I said. Did you have a point or...?
For majority of people, especially children, book is an authoritative source. Virtually no one outside of research fields and very excited enthusiasts will check sources in depth to see whether what is shown in the book is true or not.
So, I think the point OP is making is that most people don't really check sources while learning things 'the conventional way'.
We have to distinguish learning from, let's call it, recreational education. If you're learning a topic, you're not going to be using a single book. Not even schoolchildren do that. Regardless of whether you consider a book an authoritative source, if you read multiple books those books will have to agree with each other, and where they don't you'll experience some confusion until you can resolve the contradiction.
If you're only ingesting information from a single source, be that a book, a teacher, or an LLM, then you haven't really learned, because your knowledge base is unmoored. You can't learn history by studying Tolkien's mythos.
Self-consistency and consistency with your lived experience are good heuristics. Reality is self-consistent, so anything that doesn't add up indicated an error in the source or your understanding of it (or both).
EDIT:
I think it's a kind thing you need to tune yourself into. OTOH, I've observed many (most?) people seemingly being completely oblivious to self-consistency issues of their beliefs and mental models, or even texts they're reading or instructions they're following, and yet... somehow they're generally more successful at life because of it ¯\_(ツ)_/¯.
Reality is most likely self-consistent, but as we can only experience a tiny part of it it‘s impossible to tell. Also, parts of reality may appear to be contradictory with each other when some pieces are not known.
How do you do that with topics not directly experimenceable, like quantum mechanics or, as in this case, silicon manufacture?
The very first thing I mention: self-consistency. It's the only thing you have if you don't have any empirical data. It's the only thing anyone has, really. QM scientists reading QM papers and experiment reports of other people, and talking with each other, are still relying on self-consistency to sniff their own (or other people's) mistakes.
Not the OP, but I don't think you can. The understanding someone manufacturing silicon has will be not be attainable by reading about it etc. Similar for QM, no replacement for doing (some of) the math yourself, be involved in experiments etc.
> consistency with your lived experience
Aka confirmation bias.
We like explanations that fit what we expect, even if they're completely wrong.
Or do you? Cause if you went back and forth with ChatGPT for an hour it definitely hallucinated and lied to you at some point. Maybe consider using something else like Brilliant.org if you want to learn a topic, yanno, so you don’t propagate whatever hallucination from ChatGPT to your kids.
This would be a valid point maybe 3 years ago, but most chatbots will now query and verify direct sources, especially in research mode.
This is very simple to validate and verify. You could argue it may find false primary sources.
You can condemn models for a variety of other things, but acting as if this is still reality shows a lack of understanding as to modern model capabilities
These days online courses like Brilliant and such are likely largely LLM generated. Are they actually vetted by experts before publication? Who knows. My money is on no, or at least, not until someone complains.
Its easy enough to prompt ChatGPT for primary sources when doing research to validate any claims its making.
You should actually do this. Ask ChatGPT to check what it said against sources. It does correct itself. The corrections are usually not major. Just mental shortcuts.
> I'd just really like to see at least one of these to be accompanied by a statement saying what are the kinds of problems the author can now confidently solve that they couldn't before.
I've spent a lot of time with LLMs for the last two years. Something I've tried, almost for decades, is to learn enough CUDA programming to be productive with it when needed. About 6 months ago, after again banging my head against it for weeks, something finally clicked and I feel like I've overcome the initial step of at least grokking the needed ideas so I know where to go next, and I can actually write + compile + use kernels made for my use cases. I won't claim to understand everything, but I couldn't do what I can today, before I learnt the things I now know.
~2 years ago, because of my very weak math foundation, I basically said "Well, CUDA looks really interesting and really fun, but it's too difficult, lets focus on other things", even after reading some starting resources and stuff. But, by asking countless of dumb questions to LLMs, forcing it to steer me in the right direction, when I'm otherwise just driving on the highway or what not, I finally feel like I have a grasp on something I earlier only dreamed about understanding, and I'm able to be productive with it now.
To be fair the entry barrier got a lot lower over the past ~5 years. Now you can write very good CUDA kernels with just a few lines of python DSL code. Zero cpp boilerplate and zero explicit compiler calls.
Stuff like Triton, nvidia warp (the language), numba, cupy jax/pallas and so many others really paved the way. You can start out really high-level, run a profiler and then dive deep into the bottlenecks.
TL,DR: Keep going, it's a great time to have fun with GPUs.
> To be fair the entry barrier got a lot lower over the past ~5 years. Now you can write very good CUDA kernels with just a few lines of python DSL code. Zero cpp boilerplate and zero explicit compiler calls.
Well, yeah, but what I've being doing is learning proper CUDA, not "Python-compiled-to-CUDA" (otherwise it'd take like a just a week to understand enough :P ) and that's looking more or less the same today (although bunch of more complicated stuff piled on top of the fundamentals) as it used to, AFAIK.
With that said, the environment is a lot simpler to setup today at least :)
I wouldn't call one proper CUDA and the other one some dumbed down version. Nvidia really seems to be pushing for these DSLs to be first class within the ecosystem. In some cases probably even more cutting edge than the nvcc frontend, since it's easier to do some experimenting on a new niche package than on the tool everyone relies on.
I believe more and more production code is running kernels which didn't originate from the traditional cuda cpp route.
Agree. Just pay attention to the follow up questions a learner is asking to see the progress. If the follow up is just "continue", "go on", "next" or a non-sequitur then it is smell of a stall. If it is challenging or filling a gap in the answer then it is progress. So production from the learner is the only signal of worth here not the quality of LLM response, the time spent or the ability of the learner to reproduce the facts given by the LLM.
I like to use them to generate me tutorial series for technical topics I want to learn or improve. So I start a prompt like this:
> I want you to create a tutorial series about X for me. The prime objective is that I improve in topic X so never provide a solution but guide and teach. (for programming never write code). First create a question catalog to assess my current level.
Then I would ask it to structure the tutorial challenges in the following way: - Goal - Concept - Instructions
I figured that if I don't need to read any additional material on the topic the LLM is giving me too much information and I need to change the prompt. Works for me and I used this too learn topics I feel now comfortable with, like nushell, opencyper, elisp, boot loaders etc. But maybe you don't consider this "complex"
I think the key difference here is that you're using the LLM to create exercises, not to replace the learning material
_Doing_ is probably one of the best ways to learn something.
"The best way is to learn"
From the hilarious "There are 3 ways to learn. (Knowledge Fight Animated)"
https://www.youtube.com/watch?v=NsqZZiWDHAQ
Curious: Why not pick up a book or two on the topic, and use the LLM to help you through it [0]? To me, prompting the LLM without grounding it sounds like a sure shot way to end up learning "pop-sci", as GP puts it, instead of the actual science?
If today's top LLMs are reliable enough (without grounding) to academically learn "complex topics" from, may be I need to adjust my priors. I must say, I do find myself chatting about other topics (without the need for grounding) that I'm trying to "absorb" (not really learn), like Behavioural Psychology & Philosophy.
[0] Products like NotebookLM are built specifically for such usecases.
I use books to accompany my learning, but I really need something that _forces_ me to think and solve problems in that space on my own. Books can sometimes give me the illusion of learning something, but then, when needed, I've notice that I haven't really learned it.
LLMs give me structure based on my current skill level. And basically always I accompany this with books, I love reading. It's a nice combination for me.
I would love to see the "depth" of your knowledge in those topics.
You literally proved the OP's point ... thinking you're learning. More like scratching the surface, with lots of invalid data while not being able to recognize what's invalid.
It's like with latest vector of attacks being spamming Github with malware injected in proper looking code in hope of AI to index it.
Then you paste the code because you don't understand it, but you take it as working and only doing what you've asked for.
This feels like an impossible assessment - yes, a model probably can't give you the education that a advanced/expert book on a topic will, but implying that having a verifiable goal is somehow fake learning feels like an intractable problem.
What level of evidence would be sufficient for you to accept that a model may be able to teach a concept?
I'm happy to take on this challenge with a topic of your choosing, but I don't believe there will be an evidence base that satisfies you that the knowledge is earned or deep enough.
I really wonder where this assessment is coming from? It's not that I use LLM written code for something (in those exercises at least), in fact I don't let the LLM write code (see my prompt example).
It's about guiding me in _doing_ exercises so I learn and I can evaluate if I learned something if I can apply the learning myself.
> don't really teach you anything
Dunno, I've been learning a lot of Rust in the past few days. Just dove right into a project and asked AI to teach me stuff on a need to know basis. I'm actually getting used to Rust by now.
Learning programming languages is quite trivial. Many years ago I counted 16 that I had used professionally, now they are more.
It takes way more time to master and be very comfortable with a language due to its ecosystem, though. Some languages are more likely to click with a person, yet underneath they are all the same (minus the functional languages that form their own group), e.g. some performance issues may require looking at the generate assembly code.
A programming language is a skill with an accessible source of irrefutable feedback. If the program doesn't work, then you did something wrong. What happens if you try to learn something less concrete or less testable, like quantum mechanics, as the GP suggested?
I agree with you. I'm also using AI to help me learn electronics so I can finally make some real stuff I can hold in my hands. Lots of hard engineering involved so I'm a lot less confident in my ability to spot the AI's own mistakes.
However, saying you can't learn "anything" is just too strong. I'm definitely managing to distill the AI's weights into my own brain.
It's true in a very real sense, though. Without calibration, how can you tell whether you're learning something real or something fictitious? Imagine trying to learn a language from a single teacher, without ever talking to or corresponding with a native speaker. How could you tell whether you're actually learning the language, or the teacher's unique dialect? Yes, you indeed "learned" something, it's just something that might not bear much resemblance to the real thing.
This made me realize all those benchmarks which unlike loosely acts as a person being awarded with a certificate after his doctorate is rather worthless, for you never know when it bullshit-ted.
Same. I setup a practice skill that has a curriculum/list of topics, teaching methodology, approach to drills, progress tracking and spaced repetition.
Additionally I asked it to also give me problems relevant in my business domain so that I learn how to directly apply the knowledge in a realistic scenario.
I am getting much more comfortable writing rust than I was barely two weeks ago. More than I was just reading tutorials.
AI is a super polarising topic. Even now there are people who are like “it’s all AI slop so it’s all useless” along with the “OMG AI!!! OMG we will have AGI soon!!!” people.
The truth, as always, is in between. There’s loads of people using it for useful things, learning, automation and getting good results. But it’s also wrong enough that you need to deploy it carefully sometimes.
Don’t worry about either group. Keep objective and use AI where it helps and do it yourself where you are better. That’s all.
> many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
As a tangent, I think the concept of pop science has wasted so much time of what could be considered brilliant minds. I can't believe how much YouTube people I consider really smart consume under the guise of "learning stuff." And the videos are always designed to be addicting and to entice you to watch other of their stuff, which makes sense, because theyre a business, not a school.
I'm guilty of wasting time on YouTube as much as anyone else (I like watching stand-up routines and Red Bull extreme sports) but I am never under the guise that I'm doing anything productive with my time. Its okay to have fun learning, but I always felt that entertainment and education should be kept separate. You gotta learn something intentionally, not just get it served to you via algorithm.
Note im talking about the educational "shorts" not the 60+ minute deep dives that are basically a college level lecture.
Well everyone needs some downtime, and then I think it's important to acknowledge that there is a range of quality within the category of "educational entertainment." You can learn a lot of interesting things from 3BlueOneBrown and Veritasium, even though neither compares to working through a textbook.
I really like the term "edutainment" for this.
It's the kind of high-brow entertainment that makes you feel like you learn something.
For me, most "push" things are edutainment, whether that'd be Youtube videos or public-broadcaster television programs. Things you seek out yourself are not.
In a similar vein, there's "newstertainment" (news that makes you feel like it's important to watch, but actually changes nothing tangible about your life).
I’m a bit too old for the YouTube generation (can’t stand the fast cuts and permanently agitated voices), but am absolutely guilty of reading a lot of pop-sci books. I feel like they do a better job at teaching stuff for the most part, but probably also make you fall into the trap of believing you actually learnt something…
On the other hand, I’m fine with not being an expert on topics outside of my domain, as long as I retain some basic knowledge and fun party facts. So there’s that.
I can't watch Mark Rober's content because of this. He dilutes his remarkable engineering stunts into ADHD internet memes.
I do appreciate the literal college lectures on there. Hard to get the algorithm to surface them but they’re golden for learning. Of course I would be unshocked if the authenticity signals I’m tracking (AB.203 Lecture 3 video title, general hubbub and shuffling of chairs at start of video, university affiliated channel) are all faked by grifters 6 months from now.
In reality, I would just ask LLM to give me a technical explanation of quantum physics, and ask any vocab and equation I do not understand in the response.
I like to rephrase things in my own words when I read new concepts. The LLM can tell me if my version is totally off.
My trick is to let agents make an University Course Curriculum based of books and papers. Then if I don't understand something, I ask AI to simplify the book/paper until I get it.
> many of the popular educational videos on YouTube are fun to watch and don't really teach you anything.
I really enjoyed and learn a lot of things from Karpathy's and Andrew Ng's video. sure many don't really teach you anything, but I could say many others are useful too. Maybe it depends on the way we're prompting as well? it seems useful for some like Terence's message that was shared few weeks ago
https://news.ycombinator.com/item?id=49010345
I would tend to agree in the case that someone is using LLMs as their primary source for learning. But I've found a lot of use in having a claude project containing the PDF of a textbook I'm working through so that I can ask it to clarify or help me through parts I find confusing. I would definitely say that's greatly accelerated my learning - or at least greatly accelerated the speed at which I integrate information from textbooks.
The test I'd like to see: take a problem set or task you couldn't solve beforehand, learn the topic this way, then try to solve it without the LLM in the loop
as ever the real problem is formulating questions, which requires clarity on the next discrete layer of missing comprehension, which requires self-reflection and genuine insight into your own mind.
I thought LLMs were a great tool for learning new topics - perhaps even complex ones. But overtime, I have had several frustrations with this. First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point. Second, as I dive deeper, I need a way to organize the information in a useful way as I begin to branch out in many different directions. I have tried to use the LLM to fix this by having it generate a web page with diagrams and organized information flow. It's an improvement, but I still run into the issues I described in my first pint - LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise. You can direct it do something like "use plain English and avoid LLM prose - provide only as much information as necessary to demonstrate the point", but it is once again only a marginal improvement.
And then I begin to think to myself that I should just read a book on the topic written by a trusted source who put a lot of effort into teaching the topic properly and presenting the information in a thoughtful way. So, I am back to books and mostly try to use LLMs to clarify certain questions or ideas I have.
It's much better to feed the book to the LLM and ask questions as you read along, instead of asking the LLM to basically write a custom book for you from scratch.
I have to agree with this. Completely relying on LLM for all your learning needs is a disaster. But being absolutely against use of LLMs isn't doing you any favors. This is where you don't have a formula but rely on you judgement and evidence of your having learnt something.
For example having an LLM summarize a dense topic and to find books so that you can filter faster and spend time reading those books works way better than having the LLM summarize the books or the topic (or even relying on second hand information). Another one is having the LLM quiz you on your topics of interest. With questions tailored to attack specific areas that you struggle with. Its wonderful at this, nothing I've used comes close to what an LLM can do here.
You define for yourself what your goals are, slowly refining them as you learn more, and use LLM as a tool. This ,I find works best for learning.
Make it your goal to teach a room full of other humans that topic. I guarantee you will know that material cold. I've done lots of technical training in my career and after teaching a class two or three times I find myself to be very competent in the topic.
It's long been the case that the best way to learn something is to teach something.
> It's long been the case that the best way to learn something is to teach something
Which is pretty unfortunate for those that want to learn. I used to enjoy writing documentation at work, it was my favorite part of the job. And it did feel like it benefited me more than it benefited all the people that were (or weren't) reading my documentation. Now I can't really justify spending much time on docmentation when LLM's can do it in a fraction of the time and it's "good enough"
I got started as a software engineer working in the nuclear industry in the 80s. We measured our documentation in inches not pages, and it was all written by hand. And I'll bet you the documentation in the nuclear industry is still written by hand and not by LLMs.
I sure hope those docs aren’t written by LLMs for many reasons!
i wonder if people who use llm's for business critical documentation would have the same opinion, and if so why?
I have used LLMs for safety sensitive tech writing and so have people I know. It’s pretty good at it, and you usually have a 95% ready to go product at the end of it. The last 5% can pretty easily be filled in by technical experts with way less time spent battling the type writer.
Is it perfect? No. But it suffices most of the time in a pinch.
Can’t speak to the nuke industry, but it’s pretty good at aviation related things.
There is still value in experts distilling knowledge and crafting it to the audience. I'm a consultant in cybersecurity and someone asked me "give me a best practice framework for good policy hygiene". I'm sure an LLM could spit out some tips, but I've been on the industry 15 years and can write in 5 pages what an LLM wouldn't conceive of in that space.
> Another one is having the LLM quiz you on your topics of interest
This is a great idea. I'm going to try it.
Even with the latest models today, the hallucination rate is absurdly high on anything deeper than surface level knowledge or something that can be directly scraped from reddit.
And you notice when it's a topic you know well or something like software where you can immediately tell the options it's giving you don't exist on the page. Leading to the amusing statement "LLMs are bad at what I do but great at everything else".
I've (elsewhere) written about this diminishing return effect on LLM utility in relation to increasing expertise.
The question: what's the net positive gain of turning people who know nothing in a given field into sub-novices, while weighing actual experts down with work slop and marginal returns?
And I wonder what the true cost is of arming so many novices with that level of dangerous knowledge.
The fact that some people get genuine value from LLMs when learning doesn’t contradict the fact that they’re Dunning-Krueger “expertise” generators. The fact that the person learning from them is in charge of ensuring they aren’t full of shit, which they frequently are, is an inescapable flaw in this process. I honestly think that reduces the value of these things to just above what you can find out with a search engine with most topics. Hey, great. An improvement is an improvement right? Is it an improvement worth trillions of dollars and screwing over writers and artists worldwide? Fuck no.
Sounds about right.
Tangentially, but related: I'm old enough to remember when the spirit of your comment was pervasive on HN.
It wasn’t even all that long ago. I noticed around the time ChatGPT 3.5 was released I saw a significant change in tone regarding LLMs and diffusion models. I think that’s when some serious astroturfing started.
I’m old enough that I worked my first IT summer job the same year slashdot was founded. I’ve seen a lot of tech tribalism form and dissipate, and this one didn’t feel organic. My gut says a lot of the us-vs-them tension originated in a deliberate campaign to cast AI boosters as the tech industry in-crowd, and ‘other’ the people not on-board. Who knows.
1000% astroturfing—and across social media. Too many posts of the same quality at the same time(s). These things ran in cycles, spinning up and dissipating just as suddenly.
E.g. the entire framing to combat complaints about shortcomings was, "It's not the tech. It's you. You're just not doing it right. Wrong setup, wrong workflow, add this to your .MD, use loops, etc". Every complaint was immediately met with this same treatment by a swarm of vague bro-bots that materialized from the ether. The core message? Always the human's fault.
And, don't get me started on the waves of newly minted expert AI creators, making recommendations without showing a single example of what they'd supposedly built.
I'm sure some bandwagon organic creators tried to cash in on the genre, but encouraging that was also part of the point.
I think the true cost will be some catastrophic failures.
Just hoping folks don’t get hurt due to people not understanding what they’re doing with these things but believing they’re competent.
If that is the case, I seriously doubt the blame would ever touch the people that oversold this shit.
I came across the socratic method recently, and have used it to learn a couple of topics that I was having trouble getting to stick. There are some SKILL.md's available for it. It works for concepts as opposed to facts, and causes the model to guide you to answers through your own reasoning, which is both much more engaging than reading a wall of LLM text and helps the information stick.
> I came across the socratic method recently
This statement would out you as someone who didn't attend an elite school.
Can you link the Skills.md?
The "Socratic Method" (aka maieutic) skills annoy me, precisely because when you read them they are the kind of low-effort, low-expertise crap someone who over relies on AI would naively come up with when tasked with the problem of coming up with skills for learning. "Hey the Platonic dialogues are pretty cool and smart, let's do that".
The body of literature on learning theory, and beyond that on specific types of learning and specific mediums such as learning from text is so rich there are way more useful models to draw from. Believe it or not, prellm, researchers in the textual learning field had already demonstrated you can achieve performance equal or better than novice tutors using pretty basic computer aids that follow specific hint/pump interaction structures. Guiding an LLM to use these findings has evidence backing it and is way better than telling it "i guess be like socrates". The problem is, to realize there might be richer more effective and highly researched ways of tackling the problem beyond the first fart of a thought you had one afternoon requires the deep respect for expertise and specialization that precisely basically everyone in the AI space right now fundamentally lacks.
Can you provide some references to the research you're referring to? Sounds interesting but you haven't really provided enough information to find it.
Ironically, LLMs are pretty good at finding stuff like that from vague descriptions. With that, I think the parent commenter meant Art Graesser's AutoTutor work based on the terminology used (hint/pump).
TLDR: Actual human tutoring sessions were recorded and analyzed and Socratic questioning was barely used at all. Instead the following pattern was observed:
Pump — "Uh huh?" "What else?" Costs nothing, so try it first.
Hint — points at the region of the answer. "What about the pumpkin's motion sideways?"
Prompt — fishes for one specific word, with the sentence frame supplied. "The pumpkin keeps moving forward at the same ___?"
Assertion — just says it. "It keeps the runner's horizontal velocity."
That is the point – you can't just provide the references in a short post here.
You need much more time and guidance.
I was so annoyed I made a Socratic wrapper based on predefined curriculum:
https://adaptive.bounded.cc
Trying to diagrams/animations didn't yield good results even with frontier models. But pure text, any model does a decent job.
I didn't expect this to be hit by hundreds of request per minute.
So it may be very slow or become unavailable, back end can't handle that, no caching whatsoever.
what a time to be alive! "if you're having trouble understanding what your robot tutor is trying to teach you, you can ask it to guide you to the concepts using your own reasoning. This is both much more engaging than reading a wall of the robot's text and helps the information stick."
Whats funny about LLMs is they are trained from books, but they are also trained to not output books, so how much of an LLM skews its output because a perfectly normal sentence could be a quote in like 300 different books?
i threw the entire sanderson cosmere into a RAG graph sorta deal just to see how it would do if i questioned an mcp server for it about a universe i know decently well. it was actually astoundingly good. was able to find easter eggs acrossed different books and answer dumb questions like "why is kaladin emo"
If you don't know why kaladin is emo, did you truly read the books lol. Every character has to deal with the stresses of war and most don't come equipped with good mental health to begin with, they're just normal people
i was more or less testing to see how complete of an answer it could come up by scanning the whole semantic graph so i threw questions at it that spanned multiple books
Sanderson is 100% in the training data
damn it good point
I also agree with this. LLMs are a great companion when reading a book to clarify things and dive into specific topics.
I'd imagine an application that uses LLMs will be created that better manages learning. It's just not clear what that UX is yet- it's obviously not just a chatbot
dumb question, but what is the best way to feed the book to the llm?
i run into context window limits, or practical limitations of digitizing the book
You need a non DRMd copy of the book. You don't have to feed it all at once, although with a 1M context limit it is doable. A few chapters at a time is enough, in my experience. An easy alternative is using NotebookLM (now Gemini Notebook), and that has worked brilliantly for me, but I haven't tested it for technical topics (for that I like the LLM to create graphs and e.g. interact with Mathematica, so I haven't tried it).
Hoping you get answer to this. I have the same question.
I prefer to have the LLM ask me the questions
This. Ask it to quiz you if you are feeling it.
LLM is still too verbose, a real person Socratic conversation can interact a couple sentences at a time, not spew 1-3 windowfuls of low density bullet points.
I even wonder if this behavior is due to next-token prediction architectures, somehow.
I don't understand some previous complaints. It's dense and verbose seem at ends to me.
I know you probably don't consider it dense but wondering if someone can shed insight.
I find them like empty calories, like programming youtube tutorials. They maximize for feeling learnt instead of steady progress
There are ways to ground an LLM to be concise, and Socratic (method of inquiry, back and forth dialogue)
This. This is exactly how I use them and I have had no issues so far. I read the book myself, then I point the LLM at it to ask questions about notions I might be struggling with.
I'm sorry, but why not just read the fucking book if you're interested?
LLMs can't "read the room" and infer how much context the audience already has, so they try include everything.
human conceptual thinking is very much a multi-dimensional graph, which relies on light "approximate" concepts that are "good enough". LLM AR token generation is extremely one dimensional and doesnt care about the "weight" of the concept behind a token.
LLMs hold billions of parameters in "mind" at once. humans hold like four "concepts".
This is the essential mismatch and the primary reason LLM conversation can be so painful and exhausting.
Explaining this and limiting "concepts" to four at a time tops is one of the very few AGENTS.md / system prompts I always use, and it has proven invaluable time and again.
Thinking traces show how effective this is at forcing the LLM to simplify its thinking.
[edit] Also, myself and nearly all of my peers are struggling to choke down the flaws of LLM tooling along with the benefits. the speed at which LLM adoption is being forced, without truly crafting them into quality tools first, is not ok, and not normal.
LLMs have stirred an inhumane hunger and fear. the tech is fine, but the way tech companies (creators and consumers) are behaving should be deeply questioned.
it's NOT normal. it's not ok.
I’m curious where you get the estimate that humans hold “like four” parameters in their mind at once?
Cowan (2001) is an oft-cited paper proposing three to five "chunks" of capacity in human attention: https://www.cambridge.org/core/journals/behavioral-and-brain...
The full PDF is worth a read (Figure 1 may be of interest to many here): https://www.cambridge.org/core/services/aop-cambridge-core/c...
If "attention is all you need" then it's something we do indeed lack, in comparison to LLMs! But it's an interesting question: might machine cognition benefit from similar bottlenecks in an attention algorithm? Advancements like Kimi Linear seem to indicate that we're far from the finish line: https://arxiv.org/abs/2510.26692
updated the comment. i meant four "concepts". i dont reason about my own thinking in terms of parameters.
Would you be willing to provide an example (even a contrived one) of how this "four at a time" prompt changes the LLM's behavior?
I just want to understand more.
Also, would you be willing to share the actual text of it that you put in AGENTS.md?
>I really don't want to read anything generated by something like Opus 5 at this point.
Personally, I find that its generated prose tends to have an undue weight to it, almost as if every topic I ask about somehow bears a heavy burden, or is otherwise load-bearing, to use its parlance.
Quite puzzling, really.
It has a sort of metronomic quality. It never slows down or speeds up or modulates its tone. It plods forward at a relentless pace and never has a light touch with anything.
I think this is one reason why LLM text is pretty exhausting to read for long stretches.
>It has a sort of metronomic quality. It never slows down or speeds up or modulates its tone.
It's possible that this quality you describe stems from the extensive training corpora utilized by the major AI labs. These almost certainly include work from the esteemed economist Jacob Silj:
https://www.youtube.com/watch?v=Poc1upTejD8
Yes!
I have a personal theory: LLMs are *fundamentally* handicapped at perceiving what's going on in the mind of the human (this can't be "innovated away") and that's at the root of what makes them suck at conversation.
Next time you're chatting with someone, notice how much understanding is shared without anything being said. E.g. the other person might share something deeply disappointing, and they can tell without you even saying anything whether you get what they're going through. This unspoken-yet-communicated information guides the conversation. Or as another example: humans can read the room -- you walk into a room and immediately adjust your demeanor based on what you see and sense.
LLMs are totally blind to things like this, and this adds an inescapable awkwardness to interacting with them. I don't believe they'll ever grow out of this. Which thankfully implies more long term demand for humans instead of robots. :)
Very well observed. I found one more thing: they fail to consider what a 3rd person might understand from your conversation, so when you ask it to dump stuff into a Documentation, they keep making references to facts you had previously discussed or to the train of thought, completely irrelevant to bystander.
100% -- this is the worst. Referencing all sorts of "words with made up contextual/analogous meanings" based on the conversation...outside of the conversation.
Does anyone have a read on if this is primarily a Claude issue, or if all LLMs do this?
Tech guy discovers conversations with humans.
Even over phone calls you get a sense so unless it's in timing it isnt demeanor either
No, they're just trained to impress the C-suite motherfuggers with dense vocab.
I have had very similar experience! I wanted to learn Probablistic ML, checked out a couple of MOOCs, but didn't find any that were at my level - some were too advanced, some too beginner level. Claude was unable to one-shot a course, so I am now asking it to generate it module by module. But even here, it is not doing a very good job. I muddle through the concepts that it has written, do a whole bunch of back-and-forth, which tbh is exhausting, and then rewrite everything in my words so it actually makes sense to another human being.
> I get exhausted reading LLM prose
So much this! If I see one more sentence with the words "genuinely" juxtaposed with "load bearing" my head is going to explode!
btw, I am building the tutorial here for anybody interested in this topic: https://github.com/avilay/learn-probml
i’m personally deriving a huge amount of value from the custom materials fable is assembling for me. for example i asked it to write a focused expository math paper on reed solomon to accompany an implementation module that it wrote for me. it’s remarkably useful to steer it to create graphs and diagrams of exactly how you like the material presented. or the bibliography researched and cross-linked with the body or the order you want your questions addressed.
it also researched vision correcting displays for me and i can finally put that idea to bed - i was never really going to pick up an optometry textbook tbh. plus it was able to pull together a bunch of geometric and physical context about light and the eye plugging exactly my personal knowledge gaps.
in general i suspect these materials might not be that interesting to others because they are so custom to my learning style and personal needs and preferences.
these are usually not one shot documents but rather many prompts deep before i get something I’m willing to sit down and read or study. but dramatically quicker than assembling it myself from primary sources. i wouldn’t say it matches master expositors but then they’re not available to write on any topic i happen to need right now.
plus I’ll just have a live voice discussion with the system when i go for a walk and there are still things bothering me on a topic. it takes a little patience but if i’m in the mood it’s amazing.
i generally find that it can help track down specific references if i suspect hallucinations. but especially on factual topics my experience so far has been extremely encouraging.
Good point. Same with me. Despite it being very tiresome, all the back and forth that I do with it really deepens my understanding of the topic. I have been on Opus so far, let me try Fable and see if it gets better. I haven’t tried voice either. Next time I go for a walk I’ll try that!
I never used office hours as a student, which I later regretted because it made me work longer and harder to perhaps achieve somewhat better understanding in some classes, but also I dropped every proof-based math course I ever took. Overall I think my education would have been stronger by attending office hours.
I view LLMs in education similarly to office hours. Some people abuse it to get homework answers without grappling with the material, but the optimal amount is not zero.
LLM certainly not a replacement for a book, where you get someone’s extended personal approach to a topic, thoughtfully organized, reviewed and edited, often times actual courses taught based on it, with answers checked and errata available online.
I completely agree. I have vibe coded what i would consider to be some pretty weird things in the name of learning facilitation.
Perhaps the best example has been a native macOS app that is a completely custom text editor with built-in debugger, lsp support, fuzzy finder, etc stuff you'd expect. Inside the same app is a library of books i can read within the app completely formatted and for every chapter/section of each book that is a quiz to take (LLM generated of course), a "recitation" tab where i am asked a question and say outloud my response to the AI to evaluate me on and then finally practice problems to do within the custom text editor (these are usually programming books). The reader also has ai re-write built in.
As neat as this is, and i worked through K&R like this, i have ultimately fallen back on "just read the damn book and go to the AI when you've got questions."
I've decided that the main thing I'm building is my own mental model. You can take notes, create docs, put graphs and websites together, but unless I'm just trying to generate some reference material the only real objective is to develop the understanding and intuitions inside my own brain.
So I have the LLM offer a very short explanation of something, and from there's it's just me asking questions. Anything that feels fuzzy or not fully internalized is something I poke at until I'm satisfied.
It really has helped me develop a sensitivity to what I understand vs what I don't, and the ability to drill into any part of it is amazing.
I do this too. I use Claude. I picked a voice I like. I go on a three mile walk. I will ask it questions about a topic that I want to learn about. If it starts telling me more than I want to hear right then, I will say "stop". It doesn't get offended. I then ask it something else. I find this very effective. I control it so it only explains to me what I want explained. If what it says sparks questions on a related topic I jump to a brand new topic. No personal tutor could keep up with this or adjust to exactly how I want to be addressed like Claude does. I'm very excited about the progress I'm making mastering new topics.
And yes, it is not that it is just presenting the facts. By me taking control of the direction the questions and answers go, I can flesh out my mental model. I won't retain every little thing it tells me. But I am much farther ahead than before.
Can you say what topics you’re using this method to learn? I think it would be more effective for some than others.
Good to know other people who get migrane reading LLMs dense prose. I started reading books again recently, since everything online is polluted by LLM prose. What i realise, is that a human author, especially a teacher understands the learning pathways of new learners, they motivate the learning, and start from simplest concepts (a spherical cow), and then building all the complexities. This helps us to emphasize on most important concepts, while throwing away unnecessary complexities. While reading LLM prose is like reading a research article, that is written to an expert in the area, that talks about bleeding edge, with full of jargons, caveats, that just is not conducive to the learning process for a new learner.
Also they tend to assemble complex jargon in obtuse or meaningless ways, which makes reading and parsing and understanding much more difficult. Tends to reveal that LLMs fundamentally do not have "understanding", just likely word generation
To me, it's just Claude. The other models have their quirks but nothing is quite like Claude.
But even with Claude, it's it's really the prose getting in the way you can install the caveman plugin or tell it to use that "standard technical English" thing.
Can you elaborate more on juxtaposing Claude's terrible prose with other LLMs?
Any more detail you can share? Do the others feel more "human"? Are there any that are particularly digestible/human-friendly?
I've been wondering for a while if this is just Claude because I mostly use Claude, so this is very telling.
I wish I had something more methodical I could show. It's all subjective, but GLM-5.2 feels more human to me. Even GPT-5.6 Sol tends to be easier on the eyes for me (though the stereotype of it overengineering and no common sense are still true).
I tried using a new agent service recently and could tell immediately that it's powered by Claude due to the way it writes.
I had the same problem - Opus models past 4.7 tend to inflate output tokens for no good reasons, introduce innumerable jargons and is a pain to read. Then, I cam across this: https://github.com/ayghri/i-have-adhd/blob/main/skills/i-hav...
You can either install that skill or put the Rules section directly in your Global CLAUDE.md for Claude or Personalization setting for Codex and it should cut down the output verbosity by quite a fair bit.
The sycophancy is also a concern, it’s not really an impartial teacher, all its training is to suck up and maximize engagement rather than learning. The incentives are wrong.
> all its training is to suck up and maximize engagement rather than learning
That's speculative, isn't it
No, that's a direct consequence (intended or not) of how RLHF works.
I use the LLM to point me at relevant books and papers. But there is still a trust problem: I am trusting the LLM to point me at reliable, trustworthy sources.
I'm not sure I'm better off with humans though -- I'm not qualified to judge whether a source is a proper authority, not an I qualified to judge whether someone knows enough to point me to a reliable source.
It seems this is a fundamental epistemological problem to which there may never be an answer.
I have the exact same experience, so reassuring to know I'm not the only person who feels this way.
I will say, opus 5 is an egregiously bad case of this, but other LLMs have this too, just less bad.
I have been using LLMs to help me turn my journals into interconnected notes and sometimes it is so confusing to read the notes that it doesn't resemble any human would write. Its like the models are getting stronger while also losing its touch to write human sounding sentences on complex topics.
> First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point.
Agreed.
I find Opus 5, and even Fable, to be overly wordy in eg PR descriptions and code comments.
However, I suspect that's more to do with what they are trained to do by default than LLMs in general. I have a little setup where I tell Claude to work together with Codex to tighten up prose and comments, and for me that produces much more palatable text that needs less human editing afterwards.
I think one problem is that books are not customizable, and many books are aimed at people with some certain knowledge. With LLMs, you can tell it what your knowledge level is and ask it to customize the answer for you. This is difficult to achieve with books.
> generate a web page with diagrams and organized information flow.
Sounds like you'd be just as well off link-surfing Wikipedia?
> I really don't want to read anything generated by something like Opus 5 at this point.
Recently switched to OpenAI and I've gotta say Sol is so much better at writing than Claude. Opus has a distinctive sentence structure and Fable somehow manages to be even more obtuse. The personality of these models really does come through...
Claude code has a „fork“ feature where you can fork an existing conversation and keep talking in the fork and then you can go back to the original of the fork. You can fork as many times as you want and let LLMs write to a markdown file to keep important facts and learnings - also good for agents to do research without expanding the context window
> I get exhausted reading LLM prose.
While my advice is specific to learning about codebases, the way I do it is to have it generate mock data and put it in the local development environment, and give me some exploratory commands, and then ask away. It's a machine after all, so I don't have to read its preceding prose to understand whether it did tell me something, it can just repeat it however many times I ask it, and the hands on commands etc. give me something to actually try and implement.
I think there's an underrated difference between information generation and pedagogy here. LLMs are very good at producing more explanation, yet "more explanation" is often exactly what you don't need when learning something difficult
You can make something like the link below. I'd say it's a more than marginal improvement. It's still tiring, but it's much better according to my taste. I imagine everyone would have their own version of this for their own preferences.
It's a loop that uses adversarial review to check several dimensions of the writing:
https://github.com/Vibecodelicious/llm-conductor/blob/main/w...
> LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise
This is my biggest gripe with reading AI-generated text as well (ignoring the meta issue of whether it's worth taking the time to read something that an author didn't think was worth the time to write). It's gotten to the point that weird AI-style analogies just take me completely out of the text and kill my interest.
And I can usually tolerate a lot of purple prose.
Have you tried using the caveman skill ? :D Might be worth a try if you dont like long prose https://github.com/JuliusBrussee/caveman
Even if you tell the LLM not to use LLM pros they (still) do it. If you feed the Wikipedia article on signs of AI writing and tell them to use none of those signs they will also (still) do it. I have tried (many times) to get an LLM to explain a concept to me, or a process, or an algorithm or what have you, and every time they cannot help themselves. Either they use LLM pros, or they get so verbose that it all just becomes noise and I spend more time filtering out unnecessary jargon than I do reading let alone learning anything.
I ask them to use Simple English and a jargon of the domain. This seems to work best for me.
For Gemini 3 this seems bit to be the case. If you use gems you get completely different personalities - so different that its almost scary. You can create gems which are really insulting, gaslighting or seemingly of a specific profession
I've found it helpfull to ask the LLM to generate a sylybus for the topic. treat the sylybus as a design doc for a price of software. i find they do much better when they have subtasks to focus on. they can do big picture and small picture, but they can't do both at the same time.
> First, I get exhausted reading LLM prose. I really don't want to read anything generated by something like Opus 5 at this point.
I’ve found the tone of Kimi K3 to be less obnoxious. Unfortunately it doesn’t wholly solve the issue, I don’t think any LLMs out there have a truly pleasant writing style, but at least not every assumption is “load bearing”.
Same. I usually just read the book along and ask questions on a specific part, rather than trying to get the LLM to produce an entire study guide for me. It seems to work better as a Q&A than a "teach me" advisor
Yeah, I am currently trying to work with Opus 5 to refresh myself on deep learning fundamentals, and... it's a mixed bag. I'm glad I already am familiar with the subject matter, as I can prompt for refinement and improvement. It is kinda following the Karpathy videos so far (a couple lessons in) but adding more math/derivations, which was what I asked for. It has trouble staying on topic, presenting information in a coherent/meaningful order, and providing all the context necessary to move through steps in its "course notes".
Like I said, I'm essentially continually prompting to refine the material. LLMs certainly continue to append, and never cut back. It just keeps spitting out additional content at me. So that's a bit annoying too. But I can basically get figure out what's going on with a few extra promps.
If youre curious what i've got so far... just be warned it is quite literally AI slop plus me continually prompting for clarification/cleanup etc. : https://github.com/cmoscardi/ai-for-ai
opus 5 doesn't follow instructions that great. probably needed that extra "creativity" to benchmax. if you are stuck on anthropic, try Opus 4.8 or Fable5 a try with the same prompts. very different results.
To get rid of the llm prose issue you can take a representative sample of its prose (say a question and its answer), rewrite the answer in the way you'd prefer, and add that to the system prompt; I did this to get my LLMs to compress down what they say, and now everything they say is very dense and to the point
I have been using this tool for the past few months that was posted on here: https://github.com/devenjarvis/lathe
It generates tutorials for you, and serves a webpage that lets you complete them. It does a remarkable job.
It still has a bit of the LLM prose problem, but it does help you fine tune the ‘voice’ it uses.
anybody adds some prologue about what style of answers are prefered ? i know i often try to change the linguistic patterns because i too (unsurprisingly) am tired of llm prose.
I've heard people promoting LLM to use ASD-STE100 Simplified Technical English with some improvement.
So you want it to teach you and take notes for you?
> want to read anything generated by something like Opus 5 at this point
I've stopped using CC because of it. I find it insufferable.
Opus 5's language is just terrible
"hey LLM, I'm trying to learn about ___. I already know ___. My favorite authors are ___. Suggest some reading material."
It annoys me that the default AI mode is so tedious and longwinded. I read a lot of nonfiction - the house style of AI is basically marketing copy.
>LLM prose is annoyingly dense, and the useful information gets lost in a bunch of noise.
This problem doesn't get talked about enough and is second only to the hallucination problem IMO.
AI produces so much noise to wade through in order to find signal, and the more expertise you have in a field the more that costs. That noise directly subtracts signifcantly from productivity gains.
And, I think the problem is directly related to the hallucination problem. It feels very much like an effort to kitchen sink the response in order to provide some value among possible hallucinations.
It also seems to be a byproduct of Gen AI operation. It just fundamentally doesn't understand what it's outputting, so doesn't know how to narrow down to the most salient bits.
> LLM prose is annoyingly dense
It's just long. It just doesn't shut up. It's overly verbose. And you can't tell it to be concise or you degrade its quality.
If I ask what an integral is, the correct answer is that it is the continuos analog of a sum, generally used to calculate areas and volumes.
It should really be a single sentence, and then let me ask more about the terms I don't understand, and here's the beauty, in the previous one there can be only 5 terms I cannot know.
An LLM will vomit an entire page or more of explanation which isn't bad per se, but is an answer to something different: "give me a short introductory explanation to integrals". And that's not what I asked.
I call that vomit shotgun answers, text from which you have to filter out all the extra info the LLM wasn’t asked for. Luckily you can control that behavior and make it behave closer to what you want. Just ask the LLM how to ask for it.
The new Google Translate. They've made it slightly better at translating paragraphs of text but in many cases it's lost the basic function for translation: dictionary.
Try it out, fairly sure that if you out in 100 random words for 30 of them it will just refuse to translate them (it will copy paste the original word into the target language) or it will do silly things like use the target 4th dictionary definition instead of the primary one).
Its aligned in getting you interested and frustrated about a topic enough to go to a primary source you would have never looked at
LLMs help me refine my search. If I want to dive deeper into any topic, I can yield a strong list of primary sources relatively quickly.
something to try that works really well is to put "explain this as if you're talking to a 5th grader" at the end of your request.... it just breaks down the text into more manageable sentences that can be understood by general audiences
You can instruct it to be terse and even take on a specific voice if the standard prose bothers you.
I think AI is surprisingly good at this. I use voice mode while working out to learn complex topics, follow up with reading, and then go back to ask the LLM more questions. They excel at simplifying complex ideas and have endless patience. One hack I found is telling the LLM to test my knowledge by asking me questions—that gives me a clear idea of what to read next. Overall, they’re a great tool to use alongside traditional learning methods like reading books and working through practice problems.
This is on top of the issue of LLMs being a moron. like, I'm sure dumb people can learn stuff from them, but I'm sticking to books written by people who know what they're talking ahout and not vibed slop.
AI hater here. Too me this story is all to predictable. This is exactly how I thought using LLMs to learn would go. Or in simpler terms: well duh.
I'm actually going to make a prediction here as well. I think you will soon realize that using LLMs to clarify certain questions or ideas you have will turn out to have frustrations as well. And that you will soon direct those questions to either peers you know in real life or internet forums which are very likely to have a non-AI policy.
> I'm actually going to make a prediction here as well. I think you will soon realize that using LLMs to clarify certain questions or ideas you have will turn out to have frustrations as well.
Much more likely that people will believe themselves to be an expert in a subject after having had a conversation with Claude about it.
Wild because people wouldn't do that with a friend. Something about an LLM giving people info has this insidious stoleb valor to the LLMs work. Like that wasn't your thought, you just google searched it.
> First, I get exhausted reading LLM prose.
Often "be concise, to the point." is enough, but you can also paste it some stuff you like as an example text and ask to do style transfer.
You can change the prose that it outputs to anything you like. ask the llm how.
Specifically, you can set the tone and style to match what you like or used to after asking it to interview you, create an output, and then use that output as the project description or document to refer to
This feels like the weirdest complaint to me. Just tell it to shorten its response.
Literally saying "one sentence response" solves most of this problem.
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
I'm not sure I follow how this is actually guaranteed? The fact-checking process mentioned just seems to involve asking AI to review its own work.
All these LLM-as-review hype pieces don’t acknowledge that it’s turtles all the way down
What do you mean by this?
If the output can't be trusted, and you use another llm whose output can't be trusted to check the untrusted output of the first llm, then you're back where you started.
Yeah this seems to me similar to how the mortgage backed security risk concentration occurred leading up to the global financial crisis. Whereby the risk from exposure to low grade / risky single mortgages was eliminated via diversification but the diversification was simply packaging all of the risky MBS’s together and in no way diversified or de-risked the entire portfolio
I'm hoping that the Big Horrible Realization comes sooner rather than later, when we have less collective damage and pain riding on it. (Plus I'd feel personally vindicated.)
I don't see it. To me it's like having e.g. 3 drunk PhDs arguing between each other to settle on truthful answers to questions.
But the problem with LLMs is that they get the facts wrong. PhDs are PhDs because they’d look it up in an authoritative source, or actually find out through research and experimentation. The whole point is that facts aren’t a matter of opinion. The only people that argue over documented, findable facts are idiots that nobody should listen to.
Unfortunately, I don;t think that can ever be fixed. For an LLM to know that it is not hallucinating about something, it must know that that statement(s) is/are true. Which it cannot infer due to Godel's Incompleteness theorems.
No you don’t get it, I asked it specifically to make sure it’s accurate
Not really. Take hallucinations for example. If they are 1 in 100 (actually they are much rarer, but for the sake of argument), then the chances that 2 LLMs or even just 2 runs of the same LLM have the same hallucination is, well, a lot less than 1 in 100.
That rests on a false-assumption that the errors are statistically independent events, and have nothing to do with the shared nature of the judges.
It's also relying on the assumption that the checking LLM only ever corrects wrong statements and never incorrectly "corrects" an already correct statement, which might not always be the case as well.
Are there any reproducible hallucinations on any of the currently available OAI/Anthropic models? I’m not aware of any.
And even if they are related - if Opus 4.8 always has a 1:100 chance of a specific hallucination - then running the same model twice does indeed dramatically reduce the odds of an error in the final output.
Even if your conditions are granted, different LLMs can easily consistently hold divergent "worldviews". LLM A might consistently state that fact P is true (i.e. it will never hallucinate a string that states or implies ¬P), while LLM B consistently states that fact ¬P is true. You don't have a way out of this with these two, and adding a third LLM is an additional confounding factor, not a solution. You can't settle questions of fact by survey, you need to test or use reliable sources.
If simply running things thrice-over was enough to stop "hallucinations" (and not incur other problems) we wouldn't be here talking about it today, it'd have been "solved" months or years ago.
Constant hallucinations. OpenAI:s latest on max settings. If you are to naively feed say, a short corpus of text to turn it into a parallel corpus in a few different languages, the original text gets subtly mangled and no longer matches the original. Say you have several hundred annotated sentences. Without hand-coding some regex to make sure that each sentence in the source column occurs in the original corpus you’re bound to get hallucinated sentences with an error rate that exceeds 1:100. Whatever you use as the output, JSON or XML, you will end up with columns that just repeat the original instead of translating it, especially for languages that are very close to each other or represent the same language.
Yes, LLMs can be SOTA for NLP, but you’re going to have to use them to write software or workflows that are more deterministic.
"Turtles all the way down" is a phrase of, I think, unknown origin (https://en.wikipedia.org/wiki/Turtles_all_the_way_down) about infinite regress or trying to patch up some bad theory by appealing to itself. Someone claims that what holds the Earth in place is that it sits atop a giant turtle, and a skeptic asks what holds the turtle up, and the response is that it's turtles all the way down.
Personally I think this is a bad characterization of using LLMs to fix up LLMs because while you can never guarantee results this way (as the quoted line claims here, which is worthy of criticism), it is, in practice, useful to use LLMs on top of LLMs. And there's no infinite regress. Auto-mode in Claude Code, for example, seems to me like it's been successful at making the system more safe than --dangerously-bypass-permissions without prompting the user for permissions constantly.
There are certainly uses where it’s good enough, but you can never be 100% certain of correctness in the way that people claim you can by stacking N layers of these models.
What triggered my response was the “just review the output with another LLM and it’s perfectly correct”
I would have bet it’s a Terry Pratchett quote, and it kinda is:
“"The turtle moves," said Didactylos. "The turtle is a giant reptile that swims through space. It doesn't have to stand on anything. Swimming is what turtles do. The idea that it has to stand on another turtle, and that turtle has to stand on another turtle, is just silly. It's turtles all the way down, and that's a logical absurdity."
Small Gods, 1992
I read it as “if your LLM is being checked by another LLM, well then you need another LLM to check the checker. And can you really trust _that_ LLM? Probably should have an LLM to check the third one, and…”
Other comments have answered in the concrete what was meant, but to answer in the abstract, in case you’re unfamiliar with the expression:
https://en.wikipedia.org/wiki/Turtles_all_the_way_down
It’s a reference to Bernard Shaw, who once said that if we ever created a truly artificial mind it would be inside a turtle’s shell. Sturgill Simpson covered the track on his seminal work, Xeno’s Paradox.
Might be a reference to the story at the beginning of A Brief History of Time, attributed to Bertrand Russell’s audience member.
Full story in the book
This guy remembers what I thought I was saying. In my defense I stole the whole thing from Stephen King’s It which I read … forty years ago, that can’t be accurate. Let me sort out my instruments and get back to you.
an LLM tells me there is no evidence that Bernard Shaw ever said any such thing :)
It's Sturgill Simpson all the way down: https://en.wikipedia.org/wiki/Turtles_All_the_Way_Down_(song...
In my defense, I was suffering from white line fever.
Well sure, it was George Bernard Shaw, not the old CNN anchor. Also, turtles are tight lipped by nature.
Agreed. Given how many significant errors LLMs make in my topic of expertise, despite my taking multiple error checking steps, the idea of catching 100% of hallucinations because you told the LLM to check itself is hilarious. It’s just a wild lack of insight: “I’m using the LLM to teach me something I don’t know about, I definitely have the knowledge base to spot any errors that might remain!”
Yeah. People with technical and/or tech business bonafides claiming that AI granted them expertise are so often taken at face value when they really shouldn’t be. Who told them that they were proficient — a chatbot? Someone who knows even less about the topic, so any expertise seems impressive? I’ll bet it wasn’t someone that actually knew what they were talking about. Even some tech reporters are tripping over themselves to be amazed, but don’t bother checking if they should be.
People don’t even have to be lying to be wrong about this stuff. Someone can learn enough about a topic to be halfway up Mt. Stupid in no time flat, and in doing so, think they not only truly understand the topic at hand, but might be particularly adept because they were such quick studies. People that know less are impressed, because why wouldn’t they be? Anybody that knows more than them sounds like an expert. And people that know what they’re talking about cringe at the overconfidence, and probably try not to engage: who wants to have to prove that someone’s boundless confidence is entirely baseless? Most of the time, they think the actual expert is full of shit because they think they’re the expert. It’s incredible how many times I’ve had people in tech confidently, even smugly “explain” design concepts and strategies to me that they did not actually understand, knowing I was an experienced, degree-holding designer… and they didn’t even have a chatbot’s lips on their ass telling them how smart and insightful they were.
Completely agree. While I didn't set things up to have AI review its output in a loop, my experience trying to build a specific acoustic testing rig with Opus 5 also aligns with the other "it's turtles all the way down" comment.
Opus 5 first built me a detailed plan, but a couple important details were either obviously wrong or felt unnecessary. I went back and forth asking for sources and more information probably like 4 times and every time it did the "in looking at things in more detail it appears my previous advice was incorrect" spiel. It just became exhausting at some point because it feels like it really lays bare how LLMs are just minimizing that loss function but don't actually "understand" anything. It was really useful as a search engine (it correlated some highly relevant source docs), but I just couldn't trust it to believe it was actually done at any step.
A second agent reviewing it adversarially resolves some context rot. Whatever they trained these LLMs on will just infinitely double down so I break it with 1 layer of checking and then a judge who looks at facts, since the checker is adversarial.
I know it sounds silly but 1 layer ends uo being way worse than 2.
I could certainly envision a scenario whereby review would increase reliability but not how it would every guarantee 100%, there is a pretty big logical gap there.
In my experience it depends on how much in detail you want to go. Chip manufacturing is a really opaque industry, so in this particular case LLMs might not even have the training data. However, using it for a high-level introduction into something is usually pretty safe from hallucinations.
even if you say use RAG or something to a source you can trust, there's no guarantee the agent will still use exactly what the source has.
i can't even get agents to remember core instructions like "use jq instead of writing a python script to parse some json"..
The "100% accurate and free of hallucinations" claim should probably be replaced with something much weaker
I don't do animations, but I have an answer. You research a topic well enough to be able to understand if the result is OK or not. Usually it means figuring out some sort of testing.
I'm researching causal inference right now, and my main goal was to make sure I understand how to test estimation on synthetic data.
Basically, it's the same way it works with people. If you delegate a task that you don't understand, and you can't have a credibility proof (i.e. doctors, lawyers), then you research a topic well enough to be able to (1) define the task and (2) verify the end result.
You can add "make no mistakes" to the end of the prompt and achieve the same result while burning less tokens.
Yep, that's impossible. The hard truth is that most people this lost to LLM psychosis cannot understand that fact. It's better to treat it like someone in a cult, arguing the facts isn't going to help if they refuse to accept them.
I thought after reading the title that the text was about learning something, yet the actual text seems to be about having a system do something for me.
I've had success using the socratic method. I give Claude some topic (say, how the intricacies of the bond market works, the content from which are screenshots of pages from a textbook), and then I go on a walk chatting with it in voice mode. Claude is the expert, I am the student. Claude asks me questions, leading me to an answer logically. I come back with questions, and we back and forth. LLMs arent like they were in '23-'25. I'm almost always skeptical its going to lie, and almost always wrong.
In particular:
- I limit it/encourage it to give me single sentence questions
- I sometimes will ask it to tell me a motivating, human-grounded story, when we're starting a new concept: claude responds "Maya is a bond portfolio manager, and her boss has asked her to quickly price in what happened if yields go down. She knows her bond's average duration, a measure in time, but she doesn't have a percentage, which is what her manager wants. How can she give him a percentage number with just a duration figure and the proposed new yield?"
- I'll often ask claude to let me work through it, to derive the thing myself, often resulting in a string of thoughts with "yes/no" trailers, to get the LLM to reply yes or no only, and avoid derailing my train of thought. If yes, my train of thought keeps going. If no, I've got something wrong.
- I'll sometimes stop and have it craft an artifact. I typically say "build me a Brilliant.org-style interactive demo of the topic", especially when we get into the realm of looking at the actual maths of a thing (for which prose and dialog is not optimal by itself AFAICT)
- I'll do this while I'm traveling, while I'm walking, while I'm doing chores.
It's so much fun.
can you share ane example of one of your prompts? I have tried to convert some text into the socratic method and haven't succeeded yet.
It's a stupid word soup on second read, but hey, it works, and that's all that matters with "stochasty the parrot."
""" In this project, I require a socratically delivered line of conversation. Here's the typical structure to the conversation. I ask some question. You need to factor and reason about how to conduct and deliver a conversation. Best practices would be to limit terminology, or assess with the user whether they have a firm grasp on terminology before you use it. You must be very strict about this, it's unacceptable to just introduce a new concept, actor, phrase or other complication into the conversation without first labeling who what or why it exists for the conversation.
Conversation structure needs to be front-loaded with a brief interview for the user, "you understand X?", "whats your understanding of Y?".
Conversation structure then needs to proceed with single-sentence questions from the agent. User replies with an answer. Sometimes the agent needs to correct the user, but only ever do so with yet another question. """
^^ these are the instructions I have installed at the root of a "project".
Keep in mind, this is claude opus 5 low effort we're talking about, in the "projects" area of the mobile app. Here's the process I use to set up its knowledge:
1. I take screenshots of the textbook on my iPhone, and upload a chapter at a time.
2. I have it summarize the chapter into markdown by analyzing screenshots. You could probably achieve this simpler, if you just had the textbook in PDF.
3. I walk with my boy Clau-crates.
I've done this for a couple of weeks and haven't seen it revert back into its typical context-dumping behavior.
On second read, there's probably some clean up I could do. Thanks for making me pull it out and look at it. Things that could probably be improved:
- tell it to cross-check resources online to further ground itself
- use simple, short sentences (long sentences make the brain blur a bit)
- not be sycophantic (it seems like project mode has discarded with my root-level anti-sycophancy prompt)
What’s everyone’s opinion on learning new tech things in this day and age? My opinion swings between positive and depressing vision of the future.
I still learn new stuff, but I’m afraid it won’t have any value in a year or so.
For example, I’m pretty good at optimizing low level stuff, but right now you can just ask LLMs to do so and they are pretty good at it. They will profile the code and suggest reasonable options like 90% of the time.
They're amazing at it, provided you keep asking the right questions.
Trust me when I say that in the hands of someone who doesn't have your experience, the LLMs would not be getting the results you get.
You might think what you're doing is trivial, it may be sessions that flow roughly, "Instrument this, okay this part is slow, profile this part, OK read the profile output and suggest a better approach".
But your experience will be steering it in the right direction, and you're probably unaware of just how much your experience is doing that guiding, as the LLM shoots off at 100mph, you feel like it's taking you with it, but you will be guiding it a lot more than you realise, and that's where learning and experience comes in, even if you're no longer operating at the lowest depth, your knowledge of that layer will be helping.
If nothing else, the experience to know when something is actually slow is a skill in itself. If a function takes 200ms, sometimes that's as quick as it can realistically go, and sometimes that's literally a million times slower than it could be, and there's actual skill and experience wrapped up in knowing what "slow" looks like.
> Trust me when I say that in the hands of someone who doesn't have your experience, the LLMs would not be getting the results you get.
https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/23049
So there's this 11 year old issue in a forgotten ffmpeg plugin and I fixed it with deepseek by putting it into a self-testing loop. Probably would have taken me a few weeks to even understand the initial code to begin with. I haven't done C work in a long ass time and have zero knowledge of even what sub pixel sampling means.
With DS4 took me a few days and a couple of dollars. And by few days I mean I checked on it a few minutes every half hour or so a few times.
I don't understand the code it wrote but it's been in production for a while now and no issues so it's good. Ended up speeding up our video processing pipeline by 20-30%.
So while I do agree that it is cool you were able to do this, I‘ll say:
Sending in the patches but refusing to take responsibility for them is a surefire way to contribute to maintainer burnout. Please don’t do this. Either commit to fixing something and driving the PR to merge, or abstain from it entirely.
The bottleneck isn’t the speed of coding, and what you’re doing here is actively worsening the situation.
So what should newcomers do who haven't yet gained enough expertise to ask LLMs the right questions?
Newcomers have to do exactly the same what they did years and years ago. Practice the craft and gain experience by doing things.
And the craft is loose term, it can mean anything you like to get better at.
“asking the right questions” is built on years of experience doing the things now being offloaded to AI
“asking the right questions” is also a moving target with each model release
People simply underestimate the value of doing the work and think that the end result is all that matters
> “asking the right questions” is built on years of experience doing the things now being offloaded to AI
https://en.wiktionary.org/wiki/eat_one%27s_seed_corn#English
Except the steering itself is also disappearing, the same prompts from just 6 months ago now need much less steering, AIs are learning to even ask back in certain cases to persuade people with no experience towards the most likely correct choice
Generally speaking, the pattern is that people are overestimating how much "work replacement" will happen, and underestimating how much "work shifting" will happen.
What is fascinating is how you can witness it at so many levels of organization. One example: Employer executive get enamored with moving from labor to capital. They believe that by using LLMs, they can replace a lot of workers. At my place of employment, we have people that are surprised they can't file a Jira ticket describing a product ask, and have it kick off an implementation. You can build the skill to attempt that, but invariably you'll get back questions like "what do you mean by <x>" and "what do you want to do in this case, a, b, or c?"; questions that a product person or an exec are not well suited to answer.
In the past, programmers did that kind of interpretation and judgment call. So then you're in a quandary; who should do that work? Work that previously, you never imagined was an inherent part of what the replaceable code monkeys do at your beck and call?
And then, how do you hire for that? How do you find the training for the people that are experienced enough with... something... to know what a cohesive error response is, or what kind of telemetry strategy is best for that particular product and organization, what collection of product asks are incredibly complicated for what they're asking and can deliver 95% of the benefits at 5% of the work if we just do this instead, and whether you want to aim more towards thick or thin clients?
Who are those people? Wait, those are programmers? Wait, there's this whole collection of inherently human skills that we devalued, by not appreciating they were always quietly doing that for us in the past?
That's just one example. There's a repeating pattern of discovering where the work truly is, work that was embedded in manual patterns we might not have to involve ourselves with anymore, but is yet still essential. So the nature of our jobs changes massively, but the overall level of employment does not.
At least, not in the medium to long term. There is a lot of painful churn we have to suffer through first.
I agree with you in the broad strokes, but replacement by LLMs isn't the only way the level of employment can be reduced. If LLMs can make it so that two programmers can do the work that used to require five, that can result in a very large reduction in employment, even while having (some) programmers is still essential.
Only if there is no further additional demand created by the now lower cost of doing the work. Take lighting as an example. As it has progressed from burning expensive candles to now leds, our demand for lighting has continued to increase.
On the bet that layoffs will result in programmers ending up at 40% of the staffing level they are now, I will take the over.
What sources do you think are good for this data for the recent past? My brief initial search is turning up a lot of contradictory data, probably because the sources are defining "programmers" differently from each other.
I can only speak for myself, so I hope this resonates with you.
I wasn’t even really concerned with optimizing low level code before LLMs and that wasn’t why I was hired either.
However following that low level thread: We can look at the reasonable options and immediately know if they’re reasonable or nonsense. Why? We know the code. Now zoom a level out, where I think our expertise really lies.
Building a complex system isn’t easy. There are customers with requirements, there are budgets, SLAs etc. Sometimes one customer needs X and one needs Y. Our expertise is taking all of this in, and producing something that balances all the different variables. It’s knowing that we’ll expect X events a second so we’ll need Y to ensure we can tolerate failure.
Is it possible LLMs will be able to do all of that too? Maybe. But then why would our customers need the enterprises they pay for?
> I still learn new stuff, but I’m afraid it won’t have any value in a year or so.
I have stuff to do now, the value of the knowledge in a year or two isn't important if it solves the issues I have today.
I sometimes get vague ideas for solving maths/science problems. They never pan out but i can talk in detail on group theory and advanced maths and science topics due to investigating such vague ideas over the years. These days the LLM shoots the ideas down instantly and honestly correctly, i know enough to know "yeah that's right, oh well" and move on. Which actually takes away a huge avenue of learning. I'm pretty torn on the outcome of this honestly.
I'm not 'wasting time' but I'm also not really learning.
I felt this. I had a feeling for a long time that spider solitaire was somehow related to knot theory (legal moves being Reidemeister move equivalents and untangling). Llm shot this idea down very quickly! It was freeing actually.
Both of you are being overly dismissive of yourselves. Without either a proof or a program that proves the statement in question, I wouldn't trust the LLM. I wouldn't even trust if it produced citations, because I've already come across LLMs citing sources to back up statements without being able to find that information in the source (not saying it wasn't there, just that I was not able to find it).
There's a big gap between being able to ask questions about something and understanding something.
The more things you understand, the higher the chance you'll spot a situation to use them in the future.
I think the best innovations come from times when someone is uniquely able to combine two of their previous experiences together. The more experiences you have in your back pocket the more combinations you have access to and the more likely you'll have a unique combination when the right problem comes along.
Not good at subjects you have no previously learnt knowledge.
They tell you have "hit the nail on the head" when you really haven't.
They tell you have had a "great insight" when you are really haven't.
They give you the illusion of learning and progress but essentially give you faulty preconceptions will trip you up further down the road.
You can ask the LLM to be more critical and less sycophantic but that only gets you so far:
They want you to continue using, being dependent on and feeding data into the LLM--your independence isn't a priority.
There’s still an immense value in training the brain to learn and be able to approach new problems with the sort of procedural thinking that LLMs enable. We can explore topics that we are curious about and develop that sort of “muscle” to continue asking questions when we have them. I have no fear that when those bigger (and existential) problems arise we’ll be well equipped to keep asking questions and figuring out ways to solve them.
One possible value in learning new things is developing a habit of learning.
In particular it might be valuable to be in the habit of learning things that one is bad at doing.
Or not.
I'm choosing to (mostly) switch off the news and continue learning things I find interesting anyway. Maybe the world will punish me for it at some point, but I guess I'll have to deal with that when it happens. The alternative is too depressing otherwise.
So you think building a warp drive is pointless because 99% of work, per your judgement, will be done by AI? Is your contribution meaningless and artifact useless? I wouldn't think so.
It depends strongly on the prompt. Like recently I was also doing some performance optimizations, and if I did not mention profiling none of the LLMs even profiled the code; they merely read the code and assumed, based on their own analysis of big-O time complexity. It is after I explicitly asked for profiling that the LLM started actually profiling.
> I still learn new stuff, but I’m afraid it won’t have any value in a year or so.
This is silly. This would be like arguing that encyclopedias made knowing things pointless. I learn new stuff for me.
Professionally, it's important to know enough to know if you're going in the correct direction. Practically, tokens are going to continue to cost money and knowledge can save you tokens.
I mean, part of the reason the LLM can do that is because you know enough to direct the LLM to do so and verify the results to some degree, right? It's good to learn new things because:
1. It satisfies you curiosity (and curiosity is always valuable)
2. You can better utilize the LLM to expedite something you now have knowledge about
3. You still improve as an engineer/programmer/prompter/whatever
I still think it's very important not to outsource everything to AI because there is a lot of value in learning and doing things yourself which is an important part of life.
I don’t agree with the implication that there has to be a practical reason to learn new things. I enjoy learning new technologies because it’s fun.
The way learning new stuff rewires your brain cannot really be predicted, but the effects are positive.
I decided to learn watchmaking instead
I have staff ranging from 10 years of IR experience to right out of college.
I can tell you that there is an enormous gap in ability between them despite them both using LLMs for daily IR work.
The reasons aren’t complicated. The senior responders have tacit knowledge of how breaches evolve and what to look for which gives them a much better framework for where to employ the LLM.
The juniors will normally start from “here are some logs, look for weird” which is fine but leads to tunnel vision and a lack of confidence in their reporting.
I don’t mandate that anyone do work with or without an LLM. I hire seniors based on experience and juniors based on interest. But my experience has so far been that our best up and comers focusing more on learning the technologies instead of leaving those details to the LLM are developing their intuition and understanding faster and in a more robust manner.
> In plan mode (using CC, or OpenCode) I ask a model to build the foundational knowledge for X topic.
Makes sense.
> I ask it to review the accuracy of the knowledge base it built in the previous step.
Ooookay that sounds good.
> I proceed asking it to build a simulation of that topic in a low-poly, Rollercoaster Tycoon-like animation.
wat.
Author likes this kind of animation. I have something similar like this, but without the animation step.
The main idea is you can do any style you want or like to learn.
It's such a surprising and delightful turn, I love it!
The biggest thing I've learned from doing stuff like this is that there are no shortcuts. At some point or another, to truly learn something deeply, you've got to dig in to the boring details and do things the hard way. LLMs can help with this...but I find it's usually tempting to try and just offload the boring stuff to them, which doesn't work.
Having an initial higher level understanding across the domain is extremely useful to contextualize the deeper stuff. I think boring details is a very leaky characterization, but I'll continue with it.
In my experience it's infinitely easier and faster to learn deep, "boring" things when you understand how they relate to your shallow and wide understanding of all of the related components.
The LLM is merely a tool. And you can use it for domain discovery that enables efficient deep learning at an unprecedented rate or you can develop a cursory understanding of a topic and think yourself an expert.
I’ve found that if I’m not struggling I’m not learning. If stuff is coming fast and easy that’s a sign that what I’m doing is not stretching existing skills enough.
It’s true of most things. Running, dieting, weightlifting being uncomfortable is a sign of progress.
This is also why apps like Duolingo don't work for most users. They want to turn language learning into a fun, effortless game but effortless is fundamentally incompatible with learning.
Learning is uncomfortable. Reading a difficult (for you) text in a language you don't understand is exhausting and confusing. But that is where improvement happens.
Its even worse since Duolingo added a life system (not sure if they still use it), where you were only allowed to make 3 mistakes before having to recharge your energy. If you get everything right, you are not learning, you SHOULD be making mistakes constantly. That shows you're actually being challenged.
> If you get everything right, you are not learning.
One of the teachers on the college I went to had a note on his door with "If you understand everything you're doing - you're not learning anything.
> The biggest thing I've learned from doing stuff like this is that there are no shortcuts. At some point or another, to truly learn something deeply, you've got to dig in to the boring details and do things the hard way.
This is the only path to mastery, or understanding if one prefers. There are no shortcuts to a person achieving deep understanding (a.k.a. "Aha!" moments).
Can a tool such as GenAI be beneficial to someone who already has done the work to understand? Absolutely. But it cannot infuse mastery into a person simply by its use.
Only the time and effort a person devotes can do that.
Quite - you still have to do the work yourself. I think LLMs are best placed to act as an eager tutor that doesn't mind discussing a topic ad nauseam until you're certain you understand it.
yes i've found that there are a few topics that i've really been able probably 10x my understanding of using LLMs, in particular in getting me over hoops that are hard to navigate when solo, BUT I have to be really careful for it to not just show me the answer all the time.
I’ve been using LLMs to create readable rewrites of RFCs and specs that interest me. It is not precise enough for implementation use, but it has increased my understanding of the underlying RFC.
Another useful approach has been asking Codex to implement complex things, like a Kademlia DHT or BitTorrent client in a literate style with the explicit purpose to increase understanding by reviewing the source code.
Examples: https://rickcarlino.com/notes/note-dump-and-ai-summaries/ind...
https://github.com/RickCarlino/tiny-bt
The title is not representing what the post is about. “Use LLM to learn complex topics” here actually means that the author asks an agent to describe the problem area, and then implement a simple web-based simulation game, and by playing that game, the author actually learns about the topic and its constraints. They use chip making as an example.
That's actually a fun way to learn processes!
Its fun but is it really effective ? I mean I checked the LLM one and I came out more confused about a topic I already know about, I find the best way to to learn using LLMs is to just generate an example try to somewhat get a mental model of how it works and then ground my understanding with traditional documentation and resources, its an iteration of a technique I used to do in college where I would read the textbook questions first to understand what is important and then read the chapter
I assume different ways of learning work for different people. For me personally, it's taking a piece of paper and drawing the diagram of how things work together; of if it's some math, then, again, using the pen and paper to follow the text. I can very much accept that for some people playing the simulation is a good way to touch the new problem space. I can easily imagine that for some topics, let's say, traffic signal automation, a careful simulation game will probably give more information than reading papers or manuals.
> a careful simulation game
Totally agree, unfortunately careful simulation games are very rare
Game-based learning, described by Comenius, works if someone else prepares “a game” for you. E.g. like a dungeon master. :)
Otherwise you probably get more confused as you have mentioned.
On the other side, Peter Diamandis describes a situation where a bunch of kids were given a internet-connected computer and they had no teacher. Instead of it there was a “grandma” that checked kids from time to time.
After that there was a knowledge test that revealed “no teacher” approach was more efficient.
But it was a group, not an individual activity…
Thanks! The original title was "How I use LLMs to learn...", but somehow HN removed the "How" part. I even removed the initial post thinking it was a typo on my end and tried to post again, but I stumbled upon the same behavior.
HN likes to remove “How” and “Why” from titles. You need to edit them back right after you post it
I only use LLMs for explaining things if the subject is one I’m deeply familiar with and can independently and easily verify the facts. Doing so with new knowledge areas is very risky for obvious reasons.
For example, “explain how the code in this file works,” I am familiar with the overall codebase, I know the purpose of the file, and I can read it or write tests to verify if I suspect what it’s telling me isn’t correct. Or, if it’s really important, I can overcome my introvertedness and ask the team member who wrote it…but that’s a last resort nowadays, which I am very thankful for. In 99% of cases since at least Claude 4.2 days, Claude and Codex have been very accurate. Gemini on the other hand messes up more frequently and sometimes does weird things like try to delete files it’s not familiar with, at least the 3.6 flash model I’ve been using lately does this. But, code explanations are still good for the most part.
100% agreed.
There is something called tacit knowledge that cannot be easily learned through LLMs. LLMs are currently very good for research and theoretical knowledge. For example, learning how to make mechanical watches involves a lot of hands on, tacit knowledge. That skill would also be transferable to building other precision instruments. I don’t think we can currently learn these kinds of skills easily with the help of LLMs.
Even if you know everything about bicycles their mechanics, components, and how they work you still have to learn how to actually build, repair, or ride one through practice. Knowing, understanding, learning, and practicing are completely different things.
There are also niche areas of expertise that can take years to develop,not just to the point where you know the terminology and jargon, but where you understand the nuances of the field, can recognize the "unknown unknowns", and eventually have the ability to push the boundaries of existing knowledge. Maybe that is what we should really call learning: not simply acquiring information, but developing enough understanding and practical experience to contribute something new to the field.
Ironically, on the same front page of HN, there is a post about Andrew Wiles and this. I don’t think I would be able to comprehend Fermat’s Last Theorem, the Poincaré Conjecture, or Gödel’s Incompleteness Theorems, even with the availability of LLMs.
For learning, I find LLM's helpful as a knoledgeable "tutor" or "friend" during the reading of a complex paper or book. Before, when I got stuck, I spend lot's of time going around different books/pages/papers until I was able to find a simple enough explanation that allowed me to "catch up". Now, I can ask the LLM and then continue from there.
Learning from a book is still the best... Although I have been told that learning a complete subject from a book is now an inproductive use of my time... Perhaps they are right, but I still do it.
Learning is an unproductive use of your time??? No one knows how LLMs will change things, but that sounds like terrible advice.
My main issue with using AI as a learning platform is that unlike documentation, books, Youtube videos, there is not really a process of having someone "review the learning material". For example, I can always read the review of some book or ciriculum, the comments under a video, or if it's some for of open source documentation you can check the PRs and verify to some extent it's claim. With AI I can't really say what it has halucinated, because I am learning a new thing, I don't have that benefit of previously reviewed material.
This is (exactly) why I very strongly tell people not to teach themselves with an LLM. Particularly from the ground up. If you do not understand the domain, you cannot learn from the model because you won't know what questions to ask and it certainly isn't going to answer all of them for you.
You can ask the LLM to connect the dots to the sources before displaying it to you so you can see the proof in line.
You can ask the LLM how to do this. Start with a topic you know well to get the mechanism working and trust it well.
Of course, and that is what I do, but it becomes and additional mental and time consuming effort. As an dumb example, if I am learning a new programming language and trying to grasp some concept, I'll check the docs, find the section and read it. I trust that the source in the docs has been already vetted by other devs and the authors. But if I ask the AI the same thing, I then need to verify it's claims usually by asking it to check if the info is true, and then possibly opening the source link (lucky for me Claude provides the links in the desktop app as footnotes). It's not a question about AI, it's about the trust I have of this tool, it builds over time, but as soon as the AI makes an assumption or a hallucination we are back down to square one.
I assume this will become less of an issue in the future as there is more trust between the AI tools and me.
> What you get is a beautiful animation that is 100% accurate and free of hallucinations
How does he know?
Wrote about this awhile ago, and it hasn’t changed:
> Time and time again, when talking to people who rely on ChatGPT, Claude, Perplexity, and other general AI tools, I hear them say, “AI is incredible. It handles nearly everything I throw at them.”
> “What does it fumble with?” I’ll ask.
> “Well, it still gets things wrong when it comes to my line of work.”
https://www.dbreunig.com/2025/04/08/on-ai-observational-comi...
It's the kind of overstated unjustified claim an LLM would make.
I thought of the "does he know" meme when I saw this.
Gell-Mann Amnesia. Recognises the failings in their own field of discourse, assumes all true about anything else, same source.
"Complex topics" in this case means reading 22 AI-generated paragraphs that supposedly cover the entire chip manufacturing process. If the author seriously thinks this level of detail is complex then they have psychosis.
I guess we all learn in different ways. I prefer reading, and can fairly fast find the articles that make sense to me.
Colleagues often suggest podcasts and videos - I very, very rarely listen to them or see them.
The bandwidth is too low. It's not efficient and ultimately I'm bored.
This is a nice project, it looks cute. I watched some of the pages But I want more than that, more information, and faster - still a Wiki fan.
Also, step number 2 in the flow: have the LLM check itself... Naah, I don't believe that.
But you're not the only using gen ai like that. Take care.
Imo the podcast and the video are better served as background material for some other task. The low bandwidth becomes an advantage because it's often ok if you miss out on some parts due to lack of attention.
Indeed it's often a waste of time to just focus on talking people fully if you want to learn fast, reading and especially deliberate practice are better for that. But if you don't have the time, energy or focus, then listening to interviews in the background can be useful supplementally
One of the most exciting things about LLMs for me is to have someone with which I can discuss about technical/scientific/engineering things. Since usually I do not have a human at hand (or do not want to annoy people around me). Especially conversations that involve formulae can get really interesting and insightful.
Of course, you have to be careful with the answers. Especially when the discussions get longer. But usually I see that it is time to stop or to start a new session when the formulae do not make so much sense anymore or when the LLM repeats itself.
But with enough caution, LLMs are really a not-so-bad intellectual sparring partner for discussing ideas and insights.
Yes, there are many misconceptions on "learning", but LLM can be useful.
The approach that works for me is using Justin's skycak methods he mentions in his books:
https://www.justinmath.com/books/
Check the shorter "Advice on upskilling" or "The Math Academy way" for well researched approach.
So what works for me
- open a project in ChatGPT/Notebook LM - dump all the relevant and highly cited materials (textbooks, papers) - dump the advice on upskilling text or a short summary I've written for the LLM
- create "Learning Goals", that contain what I want to learn, and how to estimate is my level good enough
1) Ask it to create a learning path from the materials, following the approach. Give that to an adversarial LLM for cross check. (just for sanity check)
2) Ask it to create an "entry test" to check what I do know and what I don't
3) Iterate step by step on each module/submodule from the learning path that intermingles the approach of: small theory step + small practical task + small test. Log what's missing/wrong in my dept log. Give the dept log at the end of the session to the LLM to incorporate/create another test/task.
What I have found useful in this approach is that it will generate a lot of practical tests/tasks for me and it will explain a concept in many ways until I understand it. Also it finds some prerequisites I might miss, but based on my tests and debt log unexpected things I thought I understood surface.
So with the limits of LLM and while building a mental map of the relevant parts it's usually enough to spot the hallucinations, but if you apply structured approaches these are minimal. And it's super good, because the number of practice tests and explanations is endless.
The interfaces are a bit clunky, but current multimodal LLMs are ok with images or even hand writing.
I will recommend that structured approach.
I’ve written a skill that I basically feed what I’m looking to do, some ideas I had for accomplishing it and any other details like tech stack, etc.
The skill then riffs with me, judging my ideas and suggesting alternatives. We go back and forth until something useful comes out of it. This process isn’t unlike how I do normal development.
However, once agreed it breaks the work into “steps”. It then creates a tutorial for me, for those steps, explaining each line, why each change happens etc. I can then ask questions, muse about an alternative idea etc. Then I do the steps, and I’ve learned and gotten what I wanted to get done.
This has been how I’ve been learning Godot and making a game for the past month or so. I didn’t go in blind, I started with a course from GDQuest so I could feel confident guiding the tutorials. I will say though, having a tutor to bounce ideas off of has been really useful.
I still try to figure it out myself, consult the docs, discord etc. But if I’m stumped I’ll run my tutor skill and have some fun.
I’ve started doing a similar thing after reading a post on hn about manually applying the code so that you actually understand it.
I have done this for all my work this week and it works quite well.
For one it lets you actually query the LLM as to why, their plans give a high level not every single change and it allows you to correct it as you go and the plan will change.
I would prefer to view code this way - a flow of data between things... a project I'd really like to work on. Has anyone done anything similar?
There have been numerous approaches doing things akin to that. I remember my university working on such a project. Or at least a similar one, representing a code base as a street network with traffic showing how data flows through the system.
I too, would like to have such a tool for viewing larger projects where the flow can be cumbersome to reason about. One issue I guess would be finding the right level of abstraction in the representation.
The author says it's "100% accurate and free of hallucinations" but I am sceptical. Although I don't know much about chip production, when I ask LLMs about advanced topics like memory order LLMs tend to hallucinate more than entry-level questions. But the point is that I didn't found that LLM hallucinated before knowing it deeper. It's possible that LLM hallucinates but you don't find out because you are just learning it.
Buildings are a fantastically complicated and interesting complex problem space. I found my architecture studio students using it to ask technical and code questions about the buildings they were tasked with designing. I observed that the inaccurate information it was returning was compounding...
To be clear, I say "inaccurate" rather than "wrong" in this case because even if the information it returns is factually correct to the question being asked, students don't have an understanding of the complexity of the interdependent tectonic, regulatory, and spatial / experiential factors of a building sophisticated enough to ask their questions of the specificity and nuance necessary to get a good output that addresses the entire problem.
Anyway - with the students still learning to ask questions the right way, and the conditionally-incorrect facts making their learning more complicated rather than less, I hit on a strategy for them to use LLM's that seemed to help much better.
I suggested that instead of ask the LLM for the factual answer, or even better for the facts and an explanation, that they ask it to direct them to the proper place in the source material to find the answer themselves. Then, to treat it like a lab partner. IE:
Hey Claude I'm looking for "x."
Claude: "look at foo, bar."
Thank you - chapter (foo) part (bar) table (goo) says "car." However I notice that footnote (hoo) says there's an exception if "dar." Which is what I have. Walk me through this exception...
It seemed to have good results as a guide to understanding the disparate bodies of knowledge that they will eventually have to keep together in their heads and work synthetically and non-linearly through, rather than just as an external source of blindly trusted authority.
I've been using them by reading some docs/wiki/tutorial, then when I think I understand something trying to do a rough explanation to the LLM and ask if I'm right. I'm usually making some analogy to something I already understand a little. I'm usually partially right but missing some key bits at the first pass. I go back and forward asking for explanations of various bits or asking for resources around the area I'm not understanding. Often times just discovering the relevant name for the area of study opens lots of doors. I basically use it like I would talk to a knowledgeable and patient teacher.
As for how useful it is to understand thins, I believe it's still useful and hope it will continue to be.
The little tool it outputted is nice, but click around the stages and the text is not high quality at all. The snippy titles, abbrievated explanations, I wish a few more iterations and thought was put into the actual main textual content. Especially for 'complex' stuff
I use it by telling it my background, giving it a rough timeline and asking it to create a learning timeline, save progress along the way, and git push / pull periodically so I can use the same thing on both Linux and Mac. I tell it for each phase in the learning timeline, present me information, then challenge me on it. If it's code, it challenges me with a coding challenge, where I use it in an IDE plugin. If I'm learning something that isn't strictly code, then I ask it to give me info, then challenge me with questions and grill til I get it right. I ask it to save what it think I struggled with, so that later we can drill it again and I can also review it in an .md file.
Does anyone else use Claude like this?
It's sped up my learning by 10x. I struggled with 'just reading a book.' Take kubernetes. I hemmed and hawed and spent years periodically reading some dry book or blog or official doc, falling asleep, and forgetting while I got busy. Now I'm aggressively working with it, almost like I'm addicted to a gamification, of getting through our learning timeline, and I'm excited to move forward as quickly as possible and pass its tests.
It's like a fake teacher, because I can also ask it to drill into a topic or re-explain itself if it made no sense.
The only thing that worries me is, sometimes I'll say something like, "Um, are you sure about that?", and it'll apologize and correct itself. I barely challenged it!
These animations are great. I have learnt so many things from LLM's, cross checking things is easy enough, but the hallucinations are really not much of a thing any more (in my experience). When you deep dive on things it does seem to get very wordy sometimes (this is claude anyway). Its taught me flutter and dart without opening a book (with the occasional reference page), set me straight on monads finally, refined some linear algebra, various bits of history, and philosophy, I'm learning spinors at the moment. Is some of it wrong - maybe, but it's not like my brain is 100% accurate any way, and when I need accuracy I look up references. It is fantastic getting a broad overview of a subject you don't know or a precis of a current subject, and its so much faster.
It's not close to zero. https://artificialanalysis.ai/?omniscience=omniscience-hallu....
Hallucinations bring to question what you think you've learned. That's going to cost long-term if you labor under mis-apprehensions until you maybe figure out you learned something wrong.
where's it mentioned on that link? I can't seem to find hallucinations (don't worry found it).
Yes, sometimes its wrong, most times its right, cross checking is fairly easy, not using it because of the possibility its wrong seems a baby with the bath water thing.
I learnt a lot of functional programming from it, stuff I've always wanted to learn, but just didn't have the time and really the sources can be difficult, it really explained things well, and as someone else said in this thread, you can ask questions over and over until you understand, asking a person that (if you can get an expert) would drive them nuts. Maybe my experience isn't typical, its hard to tell, everyone reports something different.
TBH I was expecting something different. When I use LLM for learning I ask it to make questions for me. It's genius in these tasks. I moderate it to not kill me with difficulty but otherwise it's friendly to help me learn and understand
My high hopes were quickly dashed at the step that involved turning a complex topic into rollercoaster tycoon.
I use LLMs to learn deep technical concepts. I really like them because I can spend countless hours a day understanding things and building an investigation file with all my findings. I code examples and test the findings. It has helped me understand basically anything.
I'm using LLMs right now to build a terminal browser, a GUI browser, and a PyTorch/LibTorch replacement. It's really fun to be able to learn and make progress this way. It's like reading multiple interactive books, where every concept can be explained again and again until I understand it.
If you're using LLMs to learn or for research, and at some point you don't end up engaging with an actual resource (books, papers, lectures, web pages, etc) then you're playing yourself.
Sorta. If its response is grounded in actual material, and you're thorough, it's not so risky. As with all learning, trusting one source is a risk in itself. Hell, I didn't even trust my physics textbooks in college. Physics.
This looks very interesting, something I was also trying to do with my learning.
One thing I wonder is, do you mentally 'fight back' monotonicity of your interactive tool? All seem to be in 3D space, with low-poly, like in a factory moving through the belt and giving you an information + textual description to read more
But sometimes you want to visualize the charts, or graph of simulations, or maybe even the parts of an item in the rocket.
I am very interested in figuring out how people use LLMs for learning. I definitely have the knowledge, but it is severely autistic in a way.
OTOH, I am curious if there's a "practical value" to this exercise? If the LLM already contains the information and implementation knowledge to implement the networking stack inside an FPGA by itself, what value do I gain by learning about HDL, TCP, the bespoke Xillinx tooling, reading the documentation, reading papers on the implementation and going through every bit of details and theory. I feel like there's a meta skill that is more worthwhile for "practical value".
Gamifying the presentation could make topics more accessible to others. For me the overhead wouldn't help with my own learning. Also I've been burned by just learning things mechanistically (e.g., coding, applying algebraic rules), so I'm leery of learning just by making flashcards or models of the topic.
I find LLM's do great for learning when I ask what are the principles, how the main applications work, what are the key drawbacks, where are the growth plates in the field, etc. - the kind of thing a good advisor points to. Sometimes I have to ask it explicitly to use topological order of topics and show relations, which often highlights the gradient changes in the learning curve. For pruning, it's surprisingly good applying philosophical heuristics - Occam's razor, or Derrida's differance (the difference that makes a difference), etc.
And finally, no learning is effective without problem sets, and for those LLM's at times get me over blocking issues.
The degenerate case is memorizing the glib phrases regurgitated back to me; they're helpful and functional enough to get me into real trouble!
I’ve been doing something similar (browser only interactive courses served from GitHub pages) to teach me topics from beginner to advanced
LLMs and systems intersection - https://kernelspace.naigap.com
Distributed systems - https://byzantine.play.naigap.com
These are high quality at first glance. the visualizations and exercises are helpful.
What is your process for creating these resources?
I just gave a simple prompt to Kimi k3 when it came out, and then just using it and asking it to improve based on my own taste/gaps etc so it’s a long drawn out process. I’m doing that for the first one, will iteratively keep improving it myself as I run through it. Maybe will post on HN after that.
I usually learn compelx topics using the image generator of an LLM, capabilities like GPT-Image2 or Nano-Banana. I give it like a complex paper -> turn into visualzation or a poster, then ask quesitons and it helps me understand someitmes a very complex paper rather easily. The jump that nanobanana / gpt-image-2 had done is pretty wild.
The main bottleneck as an engineer is no longer writing or testing code. It is how long it takes to understand complex systems. This is a really nice approach that, if you have the tokens and the patience, feels like I great way to learn something and I think we'll see more and more stuff like this.
I had a similar realization a few months back and am working on a tool that generates "mermaid walkthroughs". It is 1000% less pretty but it is fast and is pretty good at explaining how services work or what a code review does or just as a way for your agent to explain some decision to you.
https://github.com/scottrogowski/ariel
I wish there was a LLM tool to explore a topic recursively, as a tree or a mindmap. You would start with some high level concept (say "cryptography") then dig further and further to more specific topics.
I think that would be a way more natural way to explore than being stuck on the classic linear output of a LLM.
Those animations are... series of steps, each with a description? How different is this from a plain text list? It's not that you can see forks or that the layout of the assembly line gives you more information. If anything, a plain text list is easier to scroll backwards and forward?
> In plan mode (using CC, or OpenCode) I ask a model to build the foundational knowledge for X topic. I ask it to review the accuracy of the knowledge base it built in the previous step.
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
How do you make that leap?
It depends what you mean by “learn”. To have a superficial idea of the high level concepts of a step-by-step process, sure, a 3d visualization like the ones this author makes can work. But it stops there.
This looks like the most convoluted and token heavy way to consume a bulleted list.
But AI will be better than you at those topics as well, and when someone needs an expert in that topic take a guess who will they approach in such scenario.
I don't think we are even that far when the complexity AI can handle surpasses 99.999% of what humans can handle, where AI make e.g. physics discoveries beyond the grasp of most humans and it will have to "dumb it down" when talking with humans -even physicists- but not with other AIs
One concern though: "100% accurate and free of hallucinations" is doing a lot of work. A second LLM pass can catch some mistakes, but it can also confidently agree with the first one
For everyone who thinks you can’t learn with LLMs, Dr. Cat Hicks, psychological scientist and author of the recent book “The Psychology of Software Teams[1],” who worked at Google and founded the Developer Success Lab at Pluralsight, has written two skills called learning-opportunities[2] and learning-goal[3] that use validated learning science to help you learn while using LLMs.
Cat also has an awesome podcast with her wife, Ashley Juavinett, Phd, called Change, Technically.[4]
I encourage everyone to check out her work! She’s dedicated her life to helping software developers get the support they need inside organizations to be seen as humans, not just robots.
1: https://www.drcathicks.com#book 2: https://github.com/DrCatHicks/learning-opportunities 3: https://github.com/DrCatHicks/learning-goal 4: https://www.changetechnically.fyi
Something that I have realized recently is that it has become so easy to get an answer to almost any question with the help of chatbots that its almost unnecessary to spend any effort thinking about the problem or the solution. I feel like before when I had to spend time researching a problem to find an answer I learned so many things around the topic itself which helped me understand the problem itself better and gained a deeper understanding. Today it feels like you can have an answer to the most complex questions you might have, yet you gain a superficial understanding of the topic and might forget about it quickly.
The back-and-forth questioning seems especially useful for exposing gaps in your mental model rather than just producing a summary.
Personally I'm excited about these sorts of experiments. We all learn in different ways, and these sorts of techniques allow us to create "on-demand" syllabuses and lessons that fit our learning style and learning level.
It's not perfect, but I'm optimistic this will be a useful way to teach/learn in the future.
And, to be clear, I think this will be best utilized within a group/community setting. I don't think it will replace teachers or classrooms.
having llm audit all my work/code, and generate review pages (as a teacher) of before/after with working examples is incredibly useful. Its something no course can do for me, even a teacher wouldn't have enough patience to go through each one of my mistakes.
having things defined/have correct solution to compare for review is useful, and keeps llm on track. Don't think i would trust llm if it were reviewing it all on its own
This is a great share, gonna come back to it.
Also worth mentioning that Matt Pocock has a /teach skill that creates interactive, learning sites for learning a new skill.
I'm all for this and I'm keen to try it. I certainly don't want to take away from sharing another neat use case for learning.
But
> What you get is a beautiful animation that is 100% accurate and free of hallucinations
100% free of hallucinations when you're not an expert that can check it is impossible. LLM hallucinations are an unsolved problem.
This guy is severely milking it now. If learning means building an inaccurate and incomplete understanding of the topic then go hog wild. Otherwise https://news.ycombinator.com/item?id=49209049 sums up my feelings about the author's attitude.
I ran this on ship tracks to see the global warming result. It built a toy climate model for cloud forcing: https://ship-tracks-tycoon.netlify.app
For background, in 2020 the International Maritime Organization changed fuel standards globally. The intent was to reduce pollution, which is great for human health. But, it turns out some of that pollution was supporting cloud formation and creating a cooling layer globally. And temperatures started to climb when the pollution was removed.
I haven't looked through how it's doing the calculations yet but this kind of visualization is full of possibilities if done right.
Thanks for making this guide!
Here's the code and publications/sources on Github for anyone to poke through: https://github.com/titojankowski/ship-tracks-tycoon
And since the details of ship tracks are a newer topic for me, I'm curious about Gell-Mann Amnesia. Next I'm going to apply this to a topic I understand well and will report back. Stay tuned! (will be tomorrow, bookmark or reply to this thread)
That's it? It doesnt feel right. This feels good enough for a casual dinner table conversation and not much more.
Really neat idea, I think it is one of the best ways to exploit the combined building and explaining capabilities of LLMs. I am currently building an app/game to explain friends and family concepts around wealth management and wealth building. Games are a great way to hide complexity while still including it in the « guide » you are making.
> I ask it to review the accuracy of the knowledge base it built in the previous step.
> [I ask it to build an interactive thing]
> I then push it to a new repo and enable GitHub Pages for it.
Congratulations. You are an echo chamber for LLMs. Use it to create, and verify, and post to then be scraped and trained on again.
i looked at the animations, they look cool, and i don't think i will enjoy learning things that way. as someone else said, there's a lot of content already produced on these topics. i also think the level at which these animations are playing, they are actually hiding the 'complexity' of these topics.
Very cool, I like the visual learning nature of this and the auto play once starting. The game graphics are engaging which counts for a lot these days, I feel my attention span suffering after using agents for the past year.
I've been working on a similar process of pushing to github pages, but focused more on having "practice sessions" with coding blocks to test content. Using webassembly and mock servers to mock backend endpoints Here's one I built to build a full stack llm chat system in the browser.
https://model-systems-labs.github.io/latent/llm-systems/less...
Open engine factory, stuff is flowing opposite direction of the conveyor belt it’s on. 100% accurate!
I tend to ask the LLM for a single HTML page explanation, with a pedagogical approach. Something about dropping the word pedagogical leads to a more structured outcome, but I haven't quite figured it out why yet.
This is cool project !
My favorite way to learn infra topics at work right now is asking for a humorous analogy involving monkeys and bananas. I tend to remember the result, and it gives me reference points for new topics.
Oh, I do similar, but using single page websites.
Last month, I read The Prince and had it make a text adventure campaign for me.
For a lot of other topics, I often just ask it to create a simple python example that I can run.
LLMs can help you understand a language, but they can't replace learning the vocabulary. Words and phrases still need to be learned the old-fashioned way: repetition
this is pretty cool! it feels very good for getting a brief summary/overview of something and getting a beginner's level grasp.
AKA - How I skim read a lot and learn nothing of value.
I don’t think I really agree with the author’s approach here, but I will say LLMs have been a huge help to me as I’ve been reviewing linear algebra and diving into signal processing. Anything in a textbook that I don’t fully grasp or am confused about, I just take a snapshot or copy paste then ask a model to derive it or explain it in different terms.
It reduces friction a ton, but at the end of the day I’m not skipping anything.
haha, I did the same to study new topic and then realized how personalized education will turn out in the near future. The only thing left is keeping a curious mind and ask a lot of questions !!
The selected topic (chip manufacturing) is being presented at a shallow level at best. Go read Wikipedia’s article [0] and see how deep and broad the topic is. It also skips quite a few steps, and utterly glosses over how insane of an accomplishment EUV is.
This is my biggest societal issue with LLMs: they allow you to think that you’ve “learned” a topic because you read a lot of technical terms. I don’t think it bodes well for the future.
0: https://en.wikipedia.org/wiki/Semiconductor_device_fabricati...
YouTube has so many truly wonderful videos on chip production. I admire your approach but it seems like a lot of people are in this ai maxxing phase where they reach for ai for everything despite their being ready, high quality things already available for free
I'm also not sure you can really learn chip production from widely available public information. It's a hugely complex industry where the details tend to shape larger strategies. For example, you can't really understand the relationship between Micron and TSMC without some awareness of the trade-offs of memory processes for peripheral transistors.
YouTube is just a big dump of information. Having a structured way to learning along with interacting helps you learn. Otherwise you're just binging information
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
How do you know if you're learning this for the first time? Very risky to learn from LLMs. I've done it, but you have to keep your wits about you. Lots of "oh of course you're right - what I just told you was completely wrong".
Building learning artefacts is under appreciated.
Inthink this was always the best way to learn. But it used to require immense work for a teacher.
My compiler course is a great example - program in plug in a stage of a compiler.
These exercises can be made on demand and incredibly easy now.
i've asked llms to write a presentation for me on a given topic. for whatever reason, that finds the hidden layer responsible for explaining things well.
Too over engineered, few text paragraphs would suffice to understand the topic.
The biggest win for AI dev efficiency is cutting down what gets loaded into context. Semantically matching tasks to the top tools helps a lot.
Is there a way we could turn this into a reusable skill?
That seems like a terrible way to learn. It’s a neat animation but cmon, there’s like educational TV programs from the 80s that explain this so well, in Germany there’s “Sendung mit der Maus”, not sure if they have a segment on chip manufacturing. But in these clips you can at least see the real stuff instead of some half wrong animation, just let the LLM write a few paragraphs for you or better find an ACM article or book on the subject, probably still takes less time than coming up with that animation…
God does everything have to be productized and glorified as if you’ve invented a new way of learning. Read some books!
anyone using LLM to learn a new language? got any useful prompts for this?
not all knowledge can easily be translated into a pipeline alike game imho
The self importance this author has is sickening. No wonder he's chronically talking to chatbots.
Another, possibly easier, way is to ask an LLM to give you a quiz on a topic, and then discuss your answers with it.
Surprisingly effective.
I had an internal company assessment I needed to pass before end of our fiscal year. The study material consisted of 10 ppt decks about 80 slides each (so around 800 total). I had an AI read all the decks and compose a study guide with quizzes along the way. It came up with a 100page word doc that I used in place of the decks to prepare. It worked very well for this including, like you said, quizzing me over various sections.
(Yes I confirmed it was ok to use AI with the material)
I love this idea!
Kids use agents for exam training, rather than cheating. Which is great as long as the LLM does not hallucinate
This animation is worse than useless. I've taught a lot of students. This is not learning, it's stamp collecting.
You're just memorizing a nonsensical recipe. What are the constraints? Why do we do X rather than Y? How does a particular thing scale? etc.
All you're doing is fooling yourself into thinking that you've acquired some knowledge. When in reality you haven't even learned the basic mental model to reason about this stuff.
You've learned something when you have a mental model that makes correct predictions. Until then you've memorized it at best, and as with most memorized things it will decay exponentially and will be gone from your memory soon enough.
> I personally find the style used by LLMs to explain things difficult to follow. It's just too simplistic
I also struggle with LLMs explaining things, but for the opposite reason.
I consistently have problems to get short, precise but plain/simple answers.
Instead I'm overwhelmed with walls of texts, often filled with jargon that is a mixture of imprecise and unneeded.
The style at which I learn better is by asking about stuff interactively. I ask you what something is, you give me a 3-4 sentences top answer. Then I explore and dig into the topic from your answer on the things I want to know better.
"What you get is a beautiful animation that is 100% accurate and free of hallucinations."
Do you know it is free of hallucinations because you crossed checked it with the source material or because you told the LLM "don't hallucinate"
I recently used Opus to create a learning plan in ClickUp for every Tailscale feature and it worked very well.
i had Bolt create an interactive app teaching me Oberon language but despite several attempts to fix issues, there were still glitches in unexpected places. So the experience was... meh.
Then I went back to a book written by humans (Eric Nikitin's "Realm of Oberon").
> What you get is a beautiful animation that is 100% accurate and free of hallucinations.
I am pretty sure this person is
a. Ignorant of the fact, that AI is designed to give just answers, they do not care whether they’re true or not
b. Have never Cross-checked Information given by AI with reliable sources
I mean, yeah, fun project. But that whole game could be like 15 mins of prompting.
Gell-Mann Amnesia: the blog post
once again. state and persistence are the issue. not generic llm inference.
His other project, https://aivestor.tech/, seems like a huge grift.
Or is it just an unproven attempt? Seems like a pretty straightforward value proposition. Caveat emptor.
If you know anything about investing, you know it is highly likely to be either A a grift or B an innocent, but naive attempt.
I don't know anything about investing, but I know a lot about naivete. :)
What does this have to do with the topic?
I am greatly disturbed by the idea that any of the demonstrations involve "complex topics". Whether it was the silicon workflow, LLMs, or EUV, I saw nothing complex. Are these supposed to be freshman undergrad or even high school level explainers? I suppose I didn't see any glaring errors on a quick glance, but definitely lots of details were glossed over.
Perhaps basic special relativity could be done this way, or simple derivatives, but definitely not general relativity or integrals, much less PDEs. I guess I was hoping for a 3B1B type output. Oh well.
Rejoice for your chance to brag