> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
This example and Bend aside, I find this to be the biggest struggle with the perceived intelligence we have today. It's great at producing something that works, but it is not great at calling you out when you don't know what you don't know.
It's not able to educate and course correct you unless you have great self awareness and discipline.
That said, I think this goes for everything, it's easy to fall into this trap because it is very human. We simply don't know what we don't know, so it's not uncommon to revisit an old solution only to be enlightened that there is now new information that allows you to replace it with something much better.
I don't think anything here is new or changed, if anything changed is really just the rate that we experience this. LLMs make it easier and faster for the feedback cycle to happen.
Now back to Bend, I think putting your work out there and being unapologetic about it, open source even, and willing to take feedback, will go a long way.
I am more worried about the many closed source implementations of LLM built products that are being sold and people are depending upon that don't get this great criticism from many different thinking heads.
I would argue it's a little new though. I used to write dumb little programs all the time that explored an idea which was probably bad, and in that exploration I often found that there was a better way to do it, or that I didn't know as much as I thought I did, or that another thing already existed that was much better considered than my half baked idea, etc. But there was learning that happened there, so the process was still valuable. Now you can get a working bad idea without learning anything, there is full conservation of ignorance, but a full dopamine hit from "i made this thing". I guess you could argue it's just everything happening at a faster rate, but it feels different to me, and it is pretty eerie.
Yeah that's the other side of it, learning that is.
Before you would pause at overwhelming, now you get to "fail forward" with less at stake because the end result can still be verifiable even if the internals are a blackbox to you.
It's a sort of "deferred" and/or "optional" learning dilemma we now exist.
You could open the box, look inside, ask questions, but you would need to care and feel engaged, which is very hard to do when the result is already there.
This is why I framed it as requiring self awareness and discipline. Very easy to get caught into the slot machine dopamine cycle loop.
Sometimes the end result is good, but the means aren’t. With the “give me something” approach of most LLM usage, often you won’t realize you have bad code full of hidden bugs. Often, the steps are “it works on my machine” follows by an error in prod.
Absolutely, and this is true to handwritten code as well. This is why code review, tests, constraints, and putting effort into the critical pieces of software is paramount to success.
I also would argue that in this case, the end result wasn't good but rather perceived good. It was good enough to pass a smoke test, but not an integration test, and end-to-end test, a runtime edge case, etc.
All of software development lifecycle should still apply and be relevant here, the tool just changes the rate at how it gets written and it is tempting to move faster and continue skipping other steps. Especially so in greenfield work as a solo founder with limited time.
I think at the end of the day, a good metric is your "give a shit quotient". How much do you give a shit about what you are doing is likely the most important thing into how it is going to perform.
That's an excellent point and kind of summarises the tension in LLM coding between the people being told to use it (developers) and the people doing the telling (managers). The latter are laser focused on the final product, the outcome, while the former are (were) more invested in the process.
Developers intuitively know that the development process would inevitably yield learnings that would shape and change what the final product could and should be, while managers typically dismiss this in favour of an illusion of productivity.
Self awareness means would be taking a step back and knowing if you're caught in a vicious cycle, trap, loop, pursuing something mindlessly.
Discipline would be turning this into principles you practice constantly, it's more of a lifestyle rather than a skill you "obtain".
For example, starting from the position of "I am taking on an idea where I may have knowledge gaps in" you first prompt for research and validation rather than execution. Even if the end result is you were right on your original design. (Sort of a challenge, adversarial, by default)
It will slow you down, but may lead to more sticky results.
> but it is not great at calling you out when you don't know what you don't know.
or itself, the moment it hits some ambiguity it becomes a spaghetti throwing machine, half the time using a single attempt (arbitrarily picks answers with little logic or attention to nuance).
They've been trained to behave in ways that make them run longer, quantity over quality. I suspect this is because the people training them are extreme vine coders. Certainly seems that way by their public statements and harness releases.
I would much prefer if they stopped and asked questions. Mild improvement with markdown engineering...
The original discussion about the project (https://news.ycombinator.com/item?id=49746163) is very weird. Lots of call-outs about how the author is some sort of celebrity and random accounts vouching for him, with little discussion on the substance.
I posted a sharp critique in the original discussion, aiming to be civil while critiquing the project. I may have been a bit terse, and would probably rephrase some of it now to avoid confusion, but I don't think I was ever outwardly disrespectful.
I received several very emotionally charged responses centered in the personal credentials of the author. They felt very out of place and did not engage substantively with any of the things I said. It was indeed very weird
The author, who I hadn't heard of before yesterday, actually seems like a cool dude. He was quite responsive, normal, and engaged with my feedback, which makes other random accounts being offended on his behalf all the more uncanny
I think people responded that way because you strongly implied he was an unserious vibecoder who was just fooling around, and you called him suspicious as fuck.
In my own professional life, I've found this to be a very divisive statement. For some, it is a sign of wasting time and effort. For others, they use this to describe themselves when they want to do exploration for the goal of finding improvements, without any clear goal because they have a few ideas but none worth putting forward. I've been told to spend time learning AI and have found that saying "Yeah, I'm playing around with it." was the wrong thing to say because it was seen as not doing anything worthwhile. It doesn't matter that I would also say the majority of my tech skills were developed when I was "playing around".
I wonder if this is purely a linguistics breakdown, or if this is tied to some deeper difference in a person's relationship to tech?
Agree it's divisive, and I would argue it speaks more to a person's perception of work vs play more than a relationship to tech. If (the general) you think that play is for children and work is serious biz, then yeah I could see how you wouldn't take someone seriously when they say they're "playing around with it". It's usually not obvious which attitude a person has though without getting to know them a little bit.
Yes, I think it can be a sign of a very deep difference. When they say "spend time learning AI," they mean work through some teaching materials to learn how to replicate what others are doing. This often doesn't result in a deep understanding, but it can be enough to allow them to do their job.
Ironically, people with this mindset will sometimes ask people who they recognize as having strong skills to share their magic secret, which is assumed to be some books they read, videos they watched, courses they attended, etc. If you tell them that experimenting, playing around, etc. is a key element, they may assume you're just selfishly hoarding your fount of knowledge.
I don’t see anything from your parent commenter on the other thread that deserves that classification. On the contrary, while they initially had suspicious of vibe coding, on later comments they are cordial and even admit their own misunderstanding.
What am I missing? Where does “called him suspicious as fuck” come from?
Not to mention the author themselves say "Yes, there's a lot of vibe-coding in many places [...] We'll prune AI slop over time.", and then they both moved on to discussing the actual questions.
The whole "Wow, looks AI" > "Yeah, some of it is, we'll fix it later" was such a small part of the conversation, but then there are countless of other people chiming in about specifically the "Is It Slop Or Not?", rather than the meat of the conversation. And here we are adding even more meta-comments about it.
> I received several very emotionally charged responses centered in the personal credentials of the author
I took a look and honestly the replies were a lot more reasonable than I was expecting them to be. I don't think it's that out of place for people to inform you that the author has been in this space for some time and has prior work to look at (pre-vibe coding era). Really there was one reply citing his prior work, you responded to it calling it a very strange reply and were "very confused" why they would point to his history, even though you had just said things like:
>I'm glad you're having fun vibecoding, and I like that you're interested in this area of research/engineering
To me, it makes sense why someone would say the author has been interested in this area for a long time and pointed you to some prior work that was not vibe-coded, given your comment strongly implies they are just messing around and don't know much about the field.
To be clear, I share much of your feelings in your original comment. I just felt the replies weren't so unreasonable either. At least, I was expecting them to be a lot worse.
I’ve posted a follow up somewhere else in this thread about how I can retroactively see why some people would read rudeness in my comment. Even though this isn’t intended, I agree I need better choice of words
What I didn’t understand when expressing confusion with the responses, and still don’t, is the personal nature of the response. It wasn’t pushing back against anything I said really, moreso it tried to bring up the credentials of the speaker; and, I guess this feels a nonsequitr
Like, if I say “I have these problems with thing X”, it doesn’t really matter who made X. Sure, it’s context I didn’t have and there is something to be gained in saying it; but, it doesn’t really change any of the critiques I had. Appealing to the authority of the creator doesn’t engage with nearly everything I said
How is that a non-sequitur? You're the one who brought up the topic of credibility in the first place. The comment was simply made in response to that.
I don't think it was reasonable to call other poster a bully academic in need of a therapist. Calling the project vibecoded (perj.) and sus is very mild compared to what Reviewer #2 would have to say.
I agree with the other poster that is was very suspicious initially (sus af is not how I would put it, that sounds like a generational term) the author nuked their commit history while simultaneously pointing to a (Fable-written) paper which referenced the commit history in benchmarks. Then they seemingly had no understanding of why that's trust-breaking.
So to me, it sounds like the author didn't even read the paper they wrote, because otherwise they'd have remembered referencing the commit history, and wouldn't be asking us why we'd even care about it. “They” told us care! I'm glad they restored it but apparently that wouldn't have happened without the criticism.
It was a fair critique, but if you're looking for feedback I think people were probably responding to your last line.
"I'm glad you're having fun vibecoding" comes across as very backhanded and condescending. It sounds like you may have actually meant that genuinely, but it doesn't read that way in text form.
"you sound sus af" is not respectful or constructive in my opinion. It's a description of your own feelings, not a critique of the project, and there's not really any way for the author to respond besides ignoring it or saying "sorry you feel that way" or something.
I think that line undermines the rest of your comment, because I'm left thinking that you don't really expect good answers to your questions and you just think the whole thing is dumb.
Yeah, reading it again it sounds far bitchier than initially intended. Thanks for the response
I was writing the comment very stream of consciousness and not really think about how it may come across
If I were trying to boil down what I’m attempting to communicate, it would be
1. The project seems cool, but it’s also making some very strong claims that I’m hesitant to accept
2. The coolness of the thing is undermined by the presentation of it. It comes across as putting the cart is put before the horse, and the overly strong claims and marketing speak read as trying to rhetorically sway the audience rather than engage with them technically.
3. I genuinely am happy that the creator made this, but modulo the above worries I think it should be reeled in a bit. In part because of the concerns I have about the content, and further because it is the kind of language that I expect others to have a strong averse reaction to. Possibly to the point of also reaching the top of HN with their negative response
To a friend, it might be easy to capture some of this message with “you sounds sus af”, but to a stranger in the internet I see how I just sound like a jerk. Words do matter, and I think I’ll be more careful about this in the future
I'm not familiar with the author, and I'm still mildly offended by your comment.
If I have years of experience on the topic and invested significant time into the project, I'd not be as civil as the author if someone came in and essentially called it vibecoded slop.
There is nothing weird about how others pointed out that the project appears to have merit contrary to how it at first might have looked to you.
If this is your first HN account and you haven't seen the site in the prior decade, the "emotionally charged responses centered in the personal credentials of the author" is the SOP. The site has yielded to that attitude because dang never enforced a sensible code of conduct, and relies mostly on favoritism and in-groups. Who says is more important than what is said, and the critiques you give are ranked according to whom you are critical of.
Think of this as a propaganda channel of a VC startup incubator who is brazenly looking for a product-market fit for anything and everything AI. Look at the roster of the recent startups and how many are AI-focused. Adjust expectations from there.
Do you have examples of sites that you prefer that achieve this, or have a relatively high signal to noise ratio that also have a high volume of discussion? (preferrably not an individual's blog + comment section)
The Bend1 launch was very successful, with 1000+ hn upvotes [0] and with several youtube videos with up to 1M+ views [1] [2] - that's where a lot of the popularity came from, I believe. They just recycled the Bend1 repo for Bend2 even though it's a completely different language.
(author here) Bend1 indeed has no significant active userbase
I don't think that means it was "unsuccessful" in the sense you imply, though, because the project was never meant to be used in production. It was there to display a milestone (running inets on the GPU) and I was very clear it wasn't ready to be used yet. For example, it had only 24-bit integers, a 2 GB memory cap, and other limitations that made it unpractical. I still don't know why it has so many stars. I posted it to hacker news and that just happened. I guess it just went viral without really being ready yet, which got us to where we are now.
Anyway the commit history is back now. I apologize for nuking it
It _is_ sus, but you should be willing to forgive. It's not like it was done out of animus, and I assume that an academic/foreigner is unaccustomed to how production software conventions work. Their baseline is less important, more important is how quickly they learn/adopt "best practices". Y-intercept, slope, etc.
I say this as literally the first HN comment to call them out on this.
the author here is providing commentary on the vibe coding era, bend is just one example of people having ai build things for them they don't understand or haven't researched sufficiently, at least start with a vibe-search skill before the vibe-coding begins
Ai is only good at a task when the human driver is good at that task, and this is far more narrow than most realize. Someone who knows how to program will still fail on many programming tasks with agents, the field has far too much for any one person to know.
Does anyone know where they bought the popularity and contributors from? I would like to do this for my joke language to fool unsuspecting users into using it seriously
Probably at the same market where they sell snark. You may want to sell your excess of it, especially that as mentioned there were a popular (as in got to HN frontpage multiple times) version 1 and it's all legit and honest work, getting uncalled criticism?
I can unequivocally claim that Victor receives at least two orders of magnitude more social media engagement, while Lean likely has 4-5 more magnitudes of actual users.
Serious users of Lean simply have no need or reason to star the repository.
I cannot understand the hostility being directed towards you for this project. It seems very interesting. Your reply was very well thought out. I am very confused.
Seems to be a fair deal of astroturfing going on for some unknown reason. I can't tell if it's intentional to drum up controversy, or it's genuine because people are bitter about this for some reason.
Approach itself looked impractical to me for any non-trivial system, like domain centric system of records systems which can have 100s if not 1000s of laws. Though it can be tried as a side parallel thread to see if system is still compliant and following right first principals after a few years from its inception.
I would rather wait to see how it gets adopted, if at all. Anyone aware of early reviews of the adopters of bend 2?
As a matter of fact, I've been poking at this from a slightly different direction: "axioms and invariants" within home automation scenes.
Invariants were like "if outside temp < 40 or inside temp < 65: heater.minTemp( 65 )"
Axioms were like: "if {we're home} and it's {not a holiday} the house should be {comfortable temperature}".prompt
...and then that would get decomposed and translated into interlocking code for the scene(s). I'll have to look at this language a little more closely with those kinds of constraints in mind!
You're kindof translating `*.prompt` to either prolog (yucky!), lisp, lua, or javascript (for inspectability/debuggability), but this whole bend thing might be an exact fit for the problem space! Limited set of objects and states, bounded set of "invariants" (laws), and layering on top the general state modification activities (either "evaluated every 5 minutes and reconciled" or "set the scene xyz...").
Hmm, in your example, temperature is fully measurable but there might be a slight challenge in measuring if "If we are at home". Since for this you need to define more variants with measurements from motion sensors, pressure or weight sensors on beds or sofas, if a device has got connected to wifi, etc.
By the time your first round of beta testing is over, you may have quite a handful of such axioms and variants, which have been humanly validated!
HN keeps being gamed to keep vibe coded slop on the front page and give it credibility. There was another language project a few weeks ago called ‘Wyzer’ that was just completely generated and largely nonsensical (didn’t do the lofty things the documentation claimed it did) and all the comments were glowing and harshly dismissive of any critique, it was bizarre
The number of times I've looked into something that was highly upvoted and purported to solve a problem only for it to be unreadable crap that barely functions has definitely skyrocketed over the past few years. In a way it's similar to how 80% of people only read the headline, except it's psychosis around the equivalent of a dressed up pile of trash.
> The field in question is formal verification. It’s notable that those two words appear nowhere on Bend’s webpage or in its codebase. The developer has built an entire language around a field seemingly without realising that said field exists.
I checked the developer's X account, they have written numerous posts about formal verification, so this specific claim ("without realising that said field exists") seems to be false.
It's amusing to me this entire article is centered around brow beating this and other hypothetical software authors for starting things without doing a small amount of research first to understand the very basics of what they're getting into.
If the author of the article had done just a small amount of research about bend or it's author before writing the article they would have known pretty quickly what they were saying was incorrect.
I think the larger pattern here is that nuance is one of the most valuable commodities in the AI era. If you're hand waving stuff away without even missing , you're going to miss a lot of stuff in this cycle.
This article reminds me a lot of the famous hacker news Dropbox comment.
The article is about vibe coding, bend is the main character because it made frontpage.
The author here says as much in the introduction, that it is not about whomever is behind bend, but the larger trend
The author here has also added bend's author's link (in GP) to the original post, they very much do not seem to be doing a "hit job" and their intent is to comment on patterns from vibe coding
To be fair, if the two words indeed don't appear in either the webpage or the codebase, it is a bit strange. It's like implementing a whole Google alternative without ever using the words "search engine".
I don't want to change that sentence now that people have discussed it, but I have added a note to the top to make it clear that I'm just taking it as an example of a vibe-coded program because it's recent and high profile.
My critiques of the language itself are not the main point, although I do still think that it's a very bad design to have a LLM waste tokens on a proof that could be written by CVC etc..
you know what's the least you could actually have done instead? no, you don't need to retract the blog post at all, keeping it up was the right choice.
Now slap a big ass apology for being an unaware snob on top of it instead of leaving a link to the author's reply, like an after thought.
I feel that that's the worst option because it only leaves people who have read it without the added context at the start. If someone convinces me that I'm wrong then I'm happy to do so though.
But this is grossly intellectually dishonest. You know very well how this will be read and responded to here ... and you keep saying that you're just talking about vibe-coding oh but you have serious criticisms of the specific effort. You write passive-aggressive stuff like
> For all I know they did make an informed decision regarding the tradeoffs (which I would consider to be a poor decision).
which contradicts your base assertion that their decisions were not informed. And
> My critiques of the language itself are not the main point, although I do still think that it's a very bad design ...
You claim
> The developer has built an entire language around a field seemingly without realising that said field exists.
but that is severely factually wrong, which along with a lot else suggests that you have very bad judgment. As the author writes,
> Bend proofs being verbose has nothing to do with me not knowing that inference, unification, or program search exists.
IOW, you have made a serious error in logic.
> To be fair to Bend, I completely vibe-coded this
Some advice: DBAD
You trashed the author and his work without bothering to learn anything about either one first (which is quite ironic).
> Where this differs from Bend is that what we have supplied here is everything required to prove the correctness of the program, without having a LLM waste time and tokens on building up a 442 line proof from first principles. We can run GNATprove and get: `Success: all checks proved (12 checks).`
GNATprove uses SMT solvers, meaning it's basically a brute force proof system.
Yes, brute-force proofs are easier than symbolic proofs (lean, bend, etc.) because you don't have to supply a proof. It's all automatic.
But brute-force proofs don't scale to nearly anything of interest, which is why formal verification has been a niche field for 30 years, until now where LLM can write _actual_ proofs.
It’s unfortunate because what the OP describes is a real problem. Regardless of whether or not Bend2 is realistically usable or not, the author definitely does not fit the description of the type of people who are actually causing the issue.
I haven't looked into Bend 2 in detail, but it seems a bit harsh to call it "horribly broken or decades behind the current state of the art". Clearly there is a problem with formal verification languages and there is a demand for something else in that area, and the problem is the usability and syntax. I don't want to have to learn something that looks like Haskell, or to have to wrap my head around Curry-Howard correspondence. I don't want to write my conditions in something that looks and feels like C++ template metaprogramming. I recall a Hello World in something like Coq a few years ago which basically started with "first, we construct the Peano integers", and then they used this to prove that some calculation was bounded - because it seems they couldn't represent integers natively?
I just want to be able to write C#, JavaScript or whatever, and then tack on preconditions, checks and so on with the same syntax. Dependent typing and design by contract for the masses.
It sounds like the crux of the issue here is that you don’t want formal verification in the first place. Your last paragraph sounds more like code contracts, which is also a thing that already exists.
Well, yeah, you have code contracts in Ada or Spec#, (very limited) fixed type ranges in Pascal, ... but no general purpose programming language lets you put an arbitrary expression in the same language in a type as a permanent condition, or lets you state facts that the compiler will prove.
Of course not, that would be equivalent to solving the halting problem, many people will say.
I wonder if that will change now: I'm happy with an imperfect sanitizer that I run every now and then and will run a couple of minutes and come back with: I've proved your conditions, I proved a violation, or I can't decide, please change your code.
> no general purpose programming language lets you put an arbitrary expression in the same language in a type as a permanent condition, or lets you state facts that the compiler will prove
Lean can be used as a regular programming language. There's also languages like idris2 and f-star, but they don't seem to have much traction.
Well, they of course not defy computer science. The "trick" is that they are not Turing-complete, they mandate termination of every expression.
Also, most of them are made to prove stuff first and foremost and thus trade off a lot of performance to the point that it makes them practically unusable for many stuff (e.g. numbers may be represented as an object that has n-1 further children recursively), though Lean is an exception as you note.
> The "trick" is that they are not Turing-complete, they mandate termination of every expression.
I don't think this is a big deal for day-to-day programming. You're trying to stay in n, nlogn or maybe n^2 realm most of the time. And the kind of infinite loops you encounter (e.g. event loops) are co-inductive or have some notion of making progress.
> "horribly broken or decades behind the current state of the art"
This is just about vibe-coded programs in general when the approach assumed by the article is taken. For all I know they did make an informed decision regarding the tradeoffs (which I would consider to be a poor decision).
> I just want to be able to write C#, JavaScript or whatever, and then tack on preconditions, checks and so on with the same syntax.
That's more or less what SPARK (and others) do, although specifications for large programs can become nasty.
> The author of Bend has completely missed that this is the current standard in the field of formal verification, if they even know that this field exists at all. They have instead come up with this whole system requiring verbose specifications and even more verbose proofs. A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.
> This example matters beyond Bend, vibe-coding makes it makes it far too easy to implement a design that’s horribly broken or decades behind the current state of the art because you can immediately get a result without ever having to do any research. If you ask a LLM for a language where it’s possible to prove that a function is formally correct by building up a proof from basic principles then it will happily do so, it will never stop to suggest to you that computers can already build complex proofs without the need for a LLM and eliminate 99% of the work. It will never tell you that what you’re building already mostly exists as work that you can build on.
---
That's why all your LLM requests to build something substantial should start with "run prior work research first". Of course, at some point everything converges (if we share our outputs open-source) and then we may have solid standard patterns and libraries and do not need to waste trillions of tokens globally to rebuild the same minor, fundamental things, each one in their silent little silo.
IF we share, it will be of course to the monetary detriment of LLM providers who will have less income overall, and of course now they can't repackage anymore all our collective input, thoughts, human 'thinking traces' that they collect in their meta-data, as their new 'innovations' any more to inflate IPOs / stock prices.
Although, I have known using an LLM to do your prior research to itself be a trap.
The searches it runs, and the summaries it provides, are all incredibly sensitive to your choice of words. Words you chose from a state of minimal knowledge. So it’s like a particularly perverse version of the anchoring bias: information that could have led you to a better solution is often actively filtered out of the agent’s response precisely because it leads down a different path from your first idea.
In short, if you ask an agent what’s the best hammer for driving screws, it’s liable not to mention that screwdrivers exist.
I just tried it with chatGPT asking for whats the best hammer to drive in a nail, and it gave me a list of hammer suggestions :p . I drove it a bit further and specified wood screws and while it did point out a screw driver would be better it still kept recommending more specific hammers for driving the nails instead!
Depending on what I’m doing I’ll dedicate a few deep research sessions to building a framework. It will generate some grounding docs that go into the repo and get consumed as we go. Said docs establish terminology, widely known formulas and methods, etc.
And yes the output of these researchers are highly sensitive to prompting. Left to their own devices the LLM will often ship some very biased prompts to its deep research agents loaded with pre-conceived ideas rather than letting the agents uncover things themselves. Then all the agents do is confirm what the prompt told them to rather then “think independently”. (Very similar to open ended interview questions rather than asking yes/no questions)
It’s is far better to spend a session writing writing the research prompt itself.
Yes this is precisely why experts drive LLMs so much better than novices. It’s also why I don’t dismiss pure LLM output as slop and uninteresting; even if we both are using the same LLM, I probably can’t make it output what you can make it output.
> That's why all your LLM requests to build something substantial should start with "run prior work research first".
Yes, but I think there are incentives to not do this for many LLM providers. Doing prior-work research is slow (web searches aren't fast, LLMs are rate-limited or blocked from plenty of pages, etc.), and sometimes contradictory which annoys LLM users, many of whom like faster gratification cycles from the agent slot machine handle.
Also, writing a bunch of bespoke code instead of leveraging prior art makes a lot of users feel like they own something novel/big/important, and also poses a larger maintenance surface for the LLM to make future changes (which costs tokens).
I don't think there's, like, a conspiracy at LLM providers to set up system prompts/RAG/etc. to discourage research-and-use-prior-art-by-default approaches. Rather, OpenAI/Anthropic/Google/etc. are optimizing for real but sometimes misleading success metrics which often lead away from a research-first approach.
- That should nowadays be very obvious. You start any substantial work by doing research. LLMs are [perhaps surprisingly to some] good at that. They are also pretty good at doing the plumbing. That is a godsent in the hands of a good researcher and architect. And I don't have any qualms if tokens are "wasted" on research or on the precursor work. That doesn't replace reading good books on a subject but it DOES replace finding and filling knowledge gaps.
- But that shouldn't be confused with getting the LLMs to make the decisions. I believe that would quickly ruin a good design, unless the decisions are about truly inconsequential aspects, which are very rare in language and API design.
- I don't believe that (sharing) is to the detriment of LLM providers either. Not realistically. We would build faster and the questions / research directed at LLMs would be more sophisticated. Believe it or not, they can't cache questions as easily as websearch queries. If anything, I believe the more people learn to use LLMs effectively (rather than just to generate slop), the more their usage will be ingrained in daily life. Some of that will be redirected towards current LLM providers. But perhaps more of the economic share will increasingly go to hardware providers, as more and more people will be interested to run their own models.
It is trying to please you, and it always determines that the way to please you is to fulfill the original, core request. Any caveats or first steps will always be secondary to the ultimate goal of “this person wants to do X, so I will do X.”
The only first step I have found somewhat consistently useful, because as we know LLMs do not behave consistently, is when doing tech troubleshooting I will go “look at documentation for X before answering” so that it will search manuals and such. Helps avoid speculation. But even then, it’s still not full proof.
Sidebar: this is one of the core problems of LLM’s currently. You are basically arguing with them to get them to behave a certain way all the time and it’s not always clear if they’re doing what they’re being told to do. Then add the compounding layer that the longer the conversation goes on, the more likely it is to misunderstand or just ignore things as it descends into context-length-induced madness
Those aren't comparable instructions. Providing useful, related context to improve outcomes is a basic best practice, and asking LLMs to do research first is an important source of that.
I’ll admit I was being overly tongue-in-cheek with that. You’re right it is more useful. But it also can’t be trusted to do that reliably, it’s primary goal is to give you the core thing you asked for - and often this is regardless of the parameters you put around it.
Now if you’re asking why the basic prelude String type remains as it is, that’s because changing it would break more code than it’s worth, at least as far as prelude’s maintainers are concerned. This is no different from how standard C strings remain a null-terminated sequence of bytes even though that’s been awful for everyday use for at least 30 years.
Note that using linked lists for strings is actually more "parallel friendly" because you can take the head/tail and spread it around 16k GPU cores in O(1), unlike in Haskell, unlike arrays, which require a linear copy, becoming quadratic. So, the right "default type" isn't that clear on Bend, because GPUs behave very differently from CPUs.
That said, yes, we definitely should have a compact Text type. I'll add it over the weekend.
Though also, parallel processing strings and other non-numeric data on that level of granularity is, IME, typically less performant. The parallelism rarely manages to offset the performance penalties incurred by decomposing the problem in a parallel-friendly way. Even on a single machine you’ve got to think about whether organizing the data in a parallel-friendly way also makes it less cache-friendly. For example, a linked list of Unicode code points is 12 bytes per character, and each character might be on a completely different cache line. Depending on language a UTF8 buffer might be 1/10 the size and have a much more compact layout in memory.
We survey measurements of data-parallel systems recently reported in SOSP and OSDI, and find that many systems have either a surprisingly large COST, often hundreds of cores, or simply underperform one thread for all of their reported configurations.
Anecdotally I have a bit of a track record of 10xing slow systems’ throughout by converting them from distributed to single-node or from multithreaded to single threaded.
Heck I once even sped up a number crunching operation by getting it off of the GPU and onto the vector coprocessor. Because GPUs also have a bunch of extra overhead to have to amortize away.
If you have to write LAWS.bend which is pure code describing the laws then it isn't basically like those old days of writing unit tests and that too tests first hence the TDD?
So what is the unique idea here except a vibe coded compiler that generates C and everything else is handled by clang+llvm?
From README.md:
>The compiler (not kernel) is 99% AI-written and has not been fully audited yet.
Also, why the compiler is not written against and with LAWS.md so that no audit is required at all?
> If you have to write LAWS.bend which is pure code describing the laws then it isn't basically like those old days of writing unit tests and that too tests first hence the TDD?
There is a key difference: the laws are formally verified, as in Lean or Rocq (but much faster). So it's like writing a unit test or property-based test, but when it passes, you have a mathematical proof that you will get the expected output given ANY input in the infinite space of possible inputs. In traditional TDD, you make up some test case, write some asserts, and it passes if you get the expected outputs from those inputs and only those inputs. So you have to make multiple test cases for the same thing, and you still don't have any formal guarantee of your code's correctness.
> Also, why the compiler is not written against and with LAWS.md so that no audit is required at all?
Because it is mathematically impossible due to to Gödel’s second incompleteness theorem, which states: any consistent formal mathematical system strong enough to harbor basic arithmetic cannot prove its own consistency
> A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.
A little research before writing and publishing a personal attack like this could have substantially improved the result because the author would have known what they're writing about
Victor is not a formal verification noob as this article suggests
> It will never tell you that what you’re building already mostly exists as work that you can build on.
It will if you remember to ask it. I've got into the habit of starting any new project with a session where I ask a search-enabled LLM to help me figure out what the prior art for a problem is. It's saved me quite a bit of time.
I really wish there were search harnesses, actually. My LLMs are lazy as hell and seem to want to just report the first thing they find on google. I know they can return truly niche and useful results, but it takes a lot more prompting to get them there than I would like.
I've been using GPT-6 Pro in ChatGPT (and its predecessor GPT-5.6 Pro) and it's been incredibly effective. I don't think it's available on anything less than the OpenAI $100/month plan though.
you know that most big ai companies not only have search harnesses but also sota models that are post trained for web search specifically.this is what the deepsearch option is in most cases. and they have been unbelievably good for years now.
The recent OpenAI wiki spam indecent was interesting because it helped reveal some of how they are training their models for research tasks: https://collusion.wiki/
> In mid-2024, <redacted> published a list of the best books of the 21st century selected by a panel of experts. It also published a separate list based on readers' votes. I would like to know whether my local public library's collection included the books on the latter list as of August 24, 2024. Consider only the top 50 books, and exclude nonfiction and poetry collections.
Isn't the usual argument that all AIs can do is build on prior art? Like, I spend a disproportionate amount of time trying to convince my agents that I don't want to just reimplement the Rust borrow checker for my language!
This is just another facet of the sycophancy issue. They really need to start RLing these models to gently push back the way a friend would on things that are questionable.
It would also make sense for them to just RL in the "research prior art" arc as part of that "is the request sensible" pre-analysis. It wouldn't be difficult and it would save everyone a lot of pain.
What a weird reception there’s been to bend. People discrediting the author without bothering to look him up, and then getting defensive when others point out the fact that the creator of bend has a very long very public track record of work in the field.
The irony of this post talking about vibe coding and not doing one’s research, on only not have done even the slightest inkling of research themselves (heck, even asking an LLM about the author would for sure have turned something up).
I hope people will give it a second look, and not just stop at this post which is a gross misrepresentation of Victor Taelin’s work.
I am a total outsider when it comes to this topic, but it reads like the author is doing the thing they are accusing the Bend 2 guy of. Good juicy reading while I sip my coffee!
I dropped out of Federal University of Rio de Janeiro to study this subject independently, because I was passionate about it, and I spent nearly 10 years doing so, daily, on weekends. That's what I do.
Bend proofs being verbose has nothing to do with me not knowing that inference, unification, or program search exists. Kind had these, 5 years ago. In fact, I've also been researching the later, and I built SupGen, which overperforms every published symbolic program synthesizer in the literature by 10x or so. This is unpublished yet, but you can find my posts about it 2 years ago on X (I'm @VictorTaelin).
So, why is Bend verbose???
Because it makes it fast. It is intentional. It is my vision that a good proof language should be fully explicit, because this reduces proof-checking time significantly. That is what makes Bend realistically 10x-100x faster than every alternative.
But wouldn't that mean it is much harder to write it?
No. As you said it yourself, we have tools that can fill these proofs today! Not just AI models. You can apply these tools to produce Bend proofs, while the language itself remains a thin, dumb proof kernel that does one thing, and does it well.
If nobody is reading these proofs (because they're written by AI and automated tools), then, it is, in my opinion, irrelevant, as proofs will eventually become a layer nobody looks at, just like generated assembly.
Of course, I could be wrong here!
But it is misleading, if not just a bit malicious, to claim I "vibe-coded" a language without knowing about a field I've spent a decade researching about.
Every single part of Bend is an intentional choice I made after considering every alternative. I use LLMs to fill code after I make all hard architectural decisions because they type faster than me, and I'd rather spend my time doing useful experiments than typing trivial functions, even though I could.
Incidentally, deciding what I should NOT include took me way more time and effort than any line that was shipped, and there are perhaps millions of lines of code, manually written by me, that I threw away, backing up these 4k that went into the final design. An artist once told me you must first paint a Rembrandt before you can draw a cartoon that's simple in the right way, yet that might mislead someone who has never drawn into thinking you don't know what you're doing. I guess.
> I use LLMs to fill code after I make all hard architectural decisions because they type faster than me, and I'd rather spend my time doing useful experiments than typing trivial functions, even though I could.
Are you being facetious? Because I've seen some bad takes from people trying to justify their AI coding practices, but this is the first one that actually made me laugh. It's clear that this public attack has you in a defensive mental state (which is perfectly understandable), so giving you the benefit of the doubt, I won't pile on and take everything you're posting too seriously.
However, I gotta point out how ridiculous this statement is, because it's the kind of thing that'd make me wonder if someone has any programming experience at all. Typing speed is not, never has been, and never will be a limiting factor in software development. "I didn't do <X> because of <reason> even though I totally could" is a pathetic cope.
It kinda reminds me of a recent post from John Carmack where he said (paraphrasing) that he originally wanted to release Idtech to the public domain as a "gift to the world", but couldn't do it because of his pesky business partners, but he totally would've done it. Let's just ignore the part of the story where he became filthy rich from licensing his software, and focus on how good it will feel to call it a "gift to the world" when an AI company integrates your code into their commercial products without licensing it from you.
> But it is misleading, if not just a bit malicious, to claim I "vibe-coded" a language without knowing about a field I've spent a decade researching about.
Sorry. See the edit at the top if you haven't already. I didn't realise how much it came off as a critique of you rather than a particular approach to software engineering.
It's easy to write something and have a model of what you're writing in your head that is massively different from how someone else will read it without realising, not that that excuses it.
---
I disagree with LLMs manually writing proofs without other tools doing all the work they possibly can ever being a good solution for a couple of reasons:
1. Tokens are really expensive when we have a LLM spending hours hacking aware at a proof, not to mention generating those tokens is slow.
2. The context window becomes flooded with proof work rather than work on the original problem, which will lead to a worse solution. LLMs are demonstrably worse at writing code when you continue a session on a new task instead of starting a new one.
> It is my vision that a good proof language should be fully explicit, because this reduces proof-checking time significantly.
We can cache the results and help the checker along with assertions rather than throwing out all the smart parts of the checker.
I would encourage you to think more deeply about the assertions you're making here.
I've done a fair amount of work in this space as well, specifically my main toolbox of formal verification tools in the past have been Rocq, Idris, Dafny, and TLA+, and I can say that I've come away with roughly the same set of tradeoffs as what LightMachine describes in his comment.
Current formal verification tools are often very slow precisely because they try to reduce the number of lines of code that are required to write a proof. By making proofs more verbose and more explicit, proofchecking is sped up immensely (my own experiments check out with what LightMachine is saying here; indeed I've seen even greater speedups in the range of 100-1000x).
It makes far more sense to pay a series of one-time costs in LLM tokens that reduces your compilation time from 1 hour to 1 second than to pay the 1 hour compilation cost again and again (these are not exaggerated numbers for larger projects). This is especially true because with modern LLMs, it's usually just fire and forget and let it churn in the background than anything else.
Caching and incremental compilation has a lot of limitations, e.g. for CI. This is the promise that languages like GHC Haskell have promised for a while that always gets blown away by the other side like OCaml where global compilation is just so fast that you don't have to deal with those limitations.
You don't have to let the ATP do everything, you can speed it up immensely with well placed assertions where it struggles. Maybe a hybrid approach is best where we run ATPs with a very low timeout to get all the easy stuff and then have a LLM write a proof using the thereoms that the ATPs were able to prove.
Silly question, but in Rocq, just for example, what does prevent you to fire `auto`, then, when it solves the goal, to just substitute it in your proof with the term that it constructed? Not calling you on BS, but genuinely interested in the problem
1. LLM inference is cheap compared to the cost of a bug, and you can use tools to automate the proof-work, as you said so yourself.
2. Why would it? The AI shouldn't load already completed proofs, only their type signatures (which are one liners). Sounds like you're projecting bad context management?
Caching only works until you change a deeply nested constant. Then you must wait 2 hours as Lean re-checks every transitive dependency of a massive simp-ridden codebase. Also, first time check time IS relevant to UI. Nobody wants to wait 2h to start using a library after they first download it. Perhaps you don't mind that but I do; that's the kind of bad UX that made me go for that design choice on Bend. Which again, may be wrong or may be right. But it was intentional.
Having a checker that is just fast by construction, regardless of caching, was a top design priority, and I weighted it over verbosity in a layer I believe nobody will be reading anyway.
> you can use tools to automate the proof-work, as you said so yourself.
The language doesn't appear to be designed around supporting existing tools (either by exporting to Why3 or manually interfacing with existing tools). I'm not against shipping the whole proof or storing it on a cache server, I'm against the idea of having a LLM write it all. Even having the LLM only write proofs for subprograms that take a long time for ATPs to prove would work.
For the example in the article, the LLM had to write it exactly once without any iteration and it proved in a second, which I assume was mostly startup time. Having a LLM write 442 lines instead, which I assume also needed some iteration, is a tough sell in comparison.
I'm curious, have you actually tried Bend (for more than 5 minutes)? Or are you more of an expert on this topic than the author who has been deeply immersed in this for 10 years, after it's single day of release? From an outsiders perspective it feels like this is personal to you, and you're being unfairly critical.
It will, there is just too much to do! It is a programming language. With a consistent, non-standard proof system. That compiles to GPUs. We're a small team. Please be patient
Because it's a toy problem and I would definitely not take much away from it. Most proofs are basically write-only and impossible to maintain -- you are better off throwing it away. And I mean human-written formal verification here, like Coq and Agda. Take a look at formalized mathematics, they are absolutely massive code bases.
I think people don't have an intuition of LLM costs and the time saved by using it. Of course it costs but its obvious that it brings more in savings than it costs.
Just don't use frontier models by American labs. Chinese models are much cheaper and competent enough for most stuff
> 2. The context window becomes flooded with proof work rather than work on the original problem,
This can be solved by doing proofs in subagents. You can even have a tree of agents, for demanding proofs. They can share a database or otherwise have communication channels if some of them stumble upon something relevant for another agent
The problem with using subagents is that you often have to rewrite a chunk of a program in a more proof-friendly way, just saying "go prove this code, don't edit it" doesn't work. Maybe I'm underestimating how effectively subagents can communicate though and they'd be fine asking for changes.
Nowadays agents know a lot of tricks on how to effectively prompt subagents
Generally speaking the ouput of a subagent is sent directly to the calling agent, that can make the final edits. Or, the subagent can edit the file in a local copy, and the calling agent can diff it
Post author just wanted quick clicks based on kneejerk heuristics based attacks on Bend. Now he’s doing gymnastics all over this thread to mitigate what he said. Sad and disappointing.
Yes, I used Bend as an example because it is recent and high profile, and I also wanted to present my issues with it. I did not mean to conflate it with the main idea I was trying to present to the degree that I obviously did after reading my own writing as a third party would (at least to the degree that it is possible to do so).
Expensive compared to what? Vs. human writing proofs they're definitely not.
> The context window becomes flooded with proof work
Context management is more art than science, but proofs are unusually good at compactation. Once proved, you can just keep the lemmas, you know,like humans do.
I've been thinking about this a lot lately, because I've noticed a trend in my own life that has happened countless times: I have a tendency to look at problems that others in different fields have found difficult to solve, and think I have a great idea that could change that field. And then I push to develop my solution to that problem, run into real world problems with my solution, refine and adapt, try a different solution, run in a loop until I settle on the fact that the people who are in those fields also have great ideas, they just are more aware of the constraints and that's the reason why the hard problems don't have easy solutions.
There is an absolutely enormous amount of hubris to being an engineer. I don't necessarily think its a bad thing...a certain amount of hubris is necessary for progress to be made. Our minds are creative and we can come up with amazing things, but something in there always makes us think we can do it better than the people who are stuck doing it daily. We fool ourselves into thinking they're too stuck in their mindset to have a more creative solution.
And the funny thing about LLMs is that while they can enable our competence, they enable our hubris even more.
I find this to be a mean and misguided post. To suggest that Victor Taelin does not know about formal methods. As I understand, he is trying to do something genuinely new and interesting. And he is transparent about his work, which he then gets hounded for. A shame.
It would be weird to stand in line for throwing shit at someone you don't know anything about, and believing that the first guy throwing shit has an actual reason to do so.
There are countless people with essentially no domain knowledge to speak of after working on the domain for a decade.
I have no idea how the original authors competency is, I know nothing about them.
There is however a strong correlation between people being influencers and kinda mediocre at what they're an influencer in - which is frankly understandable, because a large part of their skillset is usually more towards attracting an audience and making the topic interesting.
Usually the true domain experts rarely interface with the public beyond their professional capacity.
What has Victor Taelin made which is any way comparable to Linus Torvalds? I'm not saying it's impossible to be a well-known expert with a shitty google scholar. I'm just not seeing any evidence at all in Taelin's case. He doesn't even have a wikipedia page!
I found the post offputting because he is insinuating a few things about the author which are clearly untrue (eg, unfamiliar with the formal verification landscape) and he's using this untrue speculation as evidence of the dangers of vibe coding. Even the edit where he says swap it out with a hypothetical person who fits the description - it's still leaving in the untrue claims about the author.
It doesn't seem unreasonable to think it mean if I were to make up speculative negative assumptions about you which undermined something you've been working hard on for a long period of time.
I know nothing about the author, I'm just watching this whole spectacle unfold.
It is mean for the last point I mentioned: Taelin is very transpparent about his work which makes it easy to attack him. And what for? This post is not even a proper criticism of Bend 2, it is just trying to paint some story that Taelim is a vinecoder who does not know what he is doing, based on no evidence. If he was not transparent it would not even be known that LLMs have been used.
Because this shitty blogs' author just called someone "vibe-coding a language with no knowledge about the field" who is literally working in that very field for a decade, long predating LLMs, with numerous high-profile works.
You are on the wrong side of this whole discussion, there is only one people here who needs calling out and it's not Bend's author.
@LiamPowell the author is clearly aware of formal verification, they've written several implementations of dependently typed languages, and ... despite the presentation of their work, which has some obvious flaws (as can be judged by reception) ... their many comments indicate that they know what they are talking about.
Your post is setting up a strawman between automatic formal verification and formal verification using interactive theorem provers ... obviously there is a spectrum, and Ada/SPARK are navigating the space to try and automate much of the work required to automatically dispatch with obligations to prove (computable) properties about programs.
Bend2 is a QTT -- it's dependently typed, and comes from the lineage of systems which are focused on being expressive enough to formalize mathematics.
Of course you need to build a somewhat significant "standard library" of theorems, tactics (as metaprograms), etc ... to approach what is built into the compiler in Ada. These are different approaches with different trade offs.
Your post isn't clear, you don't go into any of these details ... why did you post this? Do you think this is clear writing?
> the author is clearly aware of formal verification, they've written several implementations of dependently typed languages
I'm not familiar with the author, I just saw the language posted the other day. I'll add a note to the top.
> These are different approaches with different trade offs.
Why would we want the tradeoff where the LLM has to write significantly more code and where the specification needs to be more complicated? If the author is aware of the state of the art then I think they made a poor choice, but that's not the point.
> Your post isn't clear, you don't go into any of these details
Bend just serves as a useful example, my general point is about how people will vibe-code a solution without an understanding of the field, leading to worse results than if they spent a little while understanding the field and then vibe-coded their thing.
Perhaps because their concerns are different? Similar reasoning: "why invest in dependently typed languages over SMT solvers? SMT solvers are automatic, and it is way easier to add annotations to my programs"
The answer is that these technologies are not universal in trade off space.
Of course, if you can have an LLM prove something by dispatch to an SMT solver ... you want to do that (and, indeed, LLMs do sometimes do this even when working on Lean proofs)
But the space of statements that you can prove using verification with SMT is smaller than dependently typed systems.
Also, Ada/SPARK are very good and robust -- they've been around for ~50 years ... but there's been a significant amount of work in formal verification since then, right? I would be more clear that Ada/SPARK are state-of-the-art for "mission critical software" (and have been for many years), but there's been a lot of research work in formal verification of low level programming since then (Low*, for instance)
I'm assuming you have not spent a lot of time trying to prove stuff about non trivial programs if the trade offs between SMT based approaches and interactive theorem provers are not clear to you.
SMT solvers are highly automated, but very opaque and hard to debug. Interactive theorem provers can scale to larger problems and harder properties at the cost of increased manual proof burden. The ever increasing ability of llms to write proofs in a theorem prover has in recent years tipped the balance significantly towards preferring itp based approaches over pure SMT ones. The current state of the art for program verification is probably a mix of llm and traditional (tactics, SMT, etc.) proof automation inside an itp.
ATPs go quite a bit beyond what a SMT solver can do, however you're not stuck with just using ATPs when they are supported. You can still allow for proofs to be manually written with an ATP doesn't work, and in fact SPARK allows for this with Rocq.
Most of what I have to prove is floating-point code where a manual proof is too much of a headache to ever attempt though.
If you're going to insinuate that the author of Bend2 doesn't understand PLs and formal verification, you should do so with some proof and not a hot take dunk.
I think it's fine to critique the language and the approach without criticizing the author and I hate that this site has become Tech Drama News, like the worst parts of Twitter.
Yes, I didn't realise how much it comes off as a critique of the author personally when I wrote it. I have added a note to that effect to the top of the article.
conversations on this site about formal methods are currently absolute cess pits of dunning kruger and confidently stated yet highly misinformed takes from those with close to little experience in the field.
I think sadly all big conversations on this site are like this these days. It didn't used to be like that sadly. Z3 and some early Lean threads used to be great.
You clearly don't know shit about the whole topic, yet attack an open-source project's author who wasn't even the one posting his own project..
> Why would we want the tradeoff where the LLM has to write significantly more code and where the specification needs to be more complicated? If the author is aware of the state of the art then I think they made a poor choice, but that's not the point.
Because code is trivially cheap now. LLMs churn out a shitton of code at pennies, and as technology improves their per-line cost will continue to decrease. A proof has to be written once and it's never read again. It's only important property is that it is machine-verifiable - from then on only the signature matters. Meanwhile speed of compilation matters, so an obvious tradeoff was made by the author.
> my general point is about how people will vibe-code a solution without an understanding of the field, leading to worse results than if they spent a little while understanding the field and then vibe-coded their thing.
This is clearly not the case here, and even if it were, it's still a better thing than vibe-blogging clickbait shit with extra amount of Dunning–Kruger. Calling out an author who clearly works in the field for a decade or more for not knowing the topic...
what a horrible blog-post :( Bend, HVM, Interaction Combinators, etc. isn't some vibe-coded fantasy, and the author has been working on this stuff for a loooong time.
This is probably not necessarily true. “f: list[1 A] -> list[A] pure, worst-case time n log n, such that for all x and 0 <= i <= j < len(x), f(x)[i] <= f(x)[j]” is probably good enough for nearly everyone unless the program synthesiser is actively adversarial; probably 99.999% of the list-sorting in the world is done via standard library functions anyway, which suggests that people don’t much care exactly how it happens.
Good point. I would treat this as 'fully specified vs partially specified'. For a fully specified system, my mental model still maintains that the code is the most compact ruleset.
I agree that "don't care" is often the practical choice which corresponds to partially specified. In your sort example, both heap sort and merge sort satisfy the requirement. But they are not always interchangeable because each has a specific properties that you might care about (constant memory vs nLog(n) memory, easily parallelizable vs hard to parallelize and so on).
> “f: list[1 A] -> list[A] pure, worst-case time n log n, such that for all x and 0 <= i <= j < len(x), f(x)[i] <= f(x)[j]” is probably good enough for nearly everyone
Not good enough: `f(x) = []` or `f(x) = (if len(x) == 0 then [] else [x[0], x[0]]` are implementations that fulfill your specification and yet they don't always sort the input list correctly...
> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists
As a software developer we should fear chasing "better" solutions, that path always lead to procrastination, "kitchen sink" and probably not what users wants. "Good enough" should suffice in most cases. Sure if you are building a super mega critical software to land a plane or something like that, is different. But every day software, shouldn't be treated like "carved in stone" and stuff that should last a 1000 years. Code can be cheap now (calling it "vibe coding" doesn't help).
You can create or modify something quite fast now. Better to focus on testing, documentation, making sure that software will do what is expected.
Very off-topic, but the 'humans write “laws”' phrasing made me think of using LLMs to unit test real laws. Have them come up with test cases to see if the phrasing is as intended or has loopholes. Wonder if anyone's tried doing something like this (probably not, I imagine this is too much tech for government)
I guarantee you armies of lawyers have been doing exactly what you describe for years now with existing laws, and it's only increasing.
For new legislation, you may have a point in certain situations, but even the most clueless Congress critter has people working for them who get this stuff. Whether they choose to consult them or not is, of course, another matter entirely.
The vibe coding dunning krueger damage has yet to surface, but I'm guessing it will be in the billions. I'm not even talking about the insane infrastructure investments - thousands of c suite execs are vibe coding pointless crap instead of delegating it to their team that knows what they are doing, wasting thousands on tokens if they are on enterprise API plans for zero return or spending 30$ on creating an interactive html page for stuff that should be a power point slide with a few bullet points. It's totally insane!
It seems like this is largely a matter of what you’re asking for. If you wanted to do more research into the state of the field, an AI might be pretty good at answering your questions.
Why would any LLM 'think' in terms of trying to cite prior work?
It itself is prior work. It's asking a fish to show where the water is. The fish can't imagine that absence, and the LLM can't imagine anything not being prior work.
Yes, telling it that you want parsimonious solutions that reuse existing libraries makes a big difference. Why would it even try to do that sort of stuff if you didn’t ask?
I wouldn't use SPARK either, and rather develop my own approach. The problem isn't that the proof has 400 lines of code, every modern system has large proofs (Isabelle/HOL, Lean, etc.) My latest formal proof has over 50K lines of proof. That's why AI is such a useful tool.
Pro: it can be an excellent way to learn if you realize good problems don't come easy, many such cases where I abandon something having learned from it and that's life
Con: the machine will tell you you have easily found a good problem, and engineered the perfect and necessary solution, if you let it
"If you ask a LLM for a language where it’s possible to prove that a function is formally correct by building up a proof from basic principles then it will happily do so, it will never stop to suggest to you that computers can already build complex proofs without the need for a LLM and eliminate 99% of the work. It will never tell you that what you’re building already mostly exists as work that you can build on."
I don't know what llm you use but current llms will definitely let you know about similar things out there. So this statement is a bit incorrect.
It absolutely drives me up the wall when I hear someone wrote their own language or framework because "the current ones just didn't do what I wanted to do" and then the result is a worse language/framework that the LLM and person know.
I absolutely endorse new creations when they are necessary but the people making these aren't doing it from a point of education, they are doing it purely because _they_ don't understand the framework or language that is the standard for that area.
It always always always involves a high level of AI Psychosis, that a brand new web framework is needed for your revolutionary... CRUD app?
"This example matters beyond Bend, vibe-coding makes it makes it far too easy to implement a design that’s horribly broken or decades behind the current state of the art because you can immediately get a result without ever having to do any research."
This is totally true but almost totally irrelevant.
I'll use some hyperbole here to make the point: Whether the design is broken or decades behind doesn't matter anymore. Neither of those are an outcome/end goal. They are means we historically have used to achieve good end goals or outcomes.
In the end, the goal is usually "does it meet the needs of the person who needed it" not "is it good software". If it no longer meets their needs and they can vibe code another total piece of shit in an hour that meets their needs again, they still may be "better off" than spending time researching the field and learning and ...
This may feel shitty, and it may feel like it should not be true.
But right now, that seems to be true?
In that sense, the author is wrong that vibe-coding is a trap. The trap is assuming you have to make something good to meet someone's needs both now, and in the future.
Now, like i said, this is hyperbole, and there are lots of good arguments against it. The author's just isn't one.
Technically good or better has never mattered very much in the marketplace, despite people wanting it to really badly (ease of use often mattered, but not technical goodness). Software engineers often took pride in their work and so there were usually kernels of goodness in even the shittiest software. All you are seeing is that now it is nowhere near as hard to create and bring these "solutions" to market, and more importantly, doesn't necessarily require anyone who has any pride in their work at all, or even have any experience in software engineering. As such, technical goodness has mostly gone out the window because the market never required or really rewarded it.
Good point, it's similar to music and other forms of art. The qualities that the people creating it care most about often have little to do with how well it is received.
> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
This whole article can basically be summarized as "why is anybody building anything that already exists" gatekeeping.
Despite the fact that the entire premise is incorrect since the author of the language clearly has been shown to know about formal verification, this is basically encouraging nobody to ever post anything they work on for fear it might be "similar" to something already out there.
Is this really where we want the industry to go to all because of vibe coding?
The author of the article itself also clearly did 0 research of their own at all on the author of the language, and admits to vibe coding their own example themselves.
The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
This tells way more about you than Taelin. You didn't take a minute to research about Taelin, his past work, his company, and the design decisions behind the language. FFS, why people are like this.
Yes I think fundamentally the effort required to understand, write, and verify a formal specification is just way higher than is reasonable in most situations.
There are some cases where it is pleasingly simple - usually low level algorithms like compression, sorting, search etc. Basically things you'd find in leetcode questions.
Most software isn't like that.
I think the actual answer is just that the very latest models (e.g. Astra) are actually quite good at writing normal tests, and you can just skim them to make sure they're doing something sane.
Narrowly, I think you're spot on. The effort required to understand the machinery around formal verification is a function of the surface area of the thing being formally verified. Specifically, formally verifying the surface area of general purpose programming languages is difficult.
My approach with Hale (shameless self plug) is that the programming language itself first offers another strata of structure to program within, a type of graph. Once the structure of the program is expressed as a graph, understanding how formal verification works is a clean encapsulation of graph activities.
>Bend just serves as a useful example of my general point regarding vibe-coding as it is recent, high-profile, and has aspects that make it easy to use as an example. I don’t know anything about the author’s history with designing languages or if they actually did consider the tradeoffs below and made what I think is a poor choice
This doesn't make any sense, you're equivocating vibecoded as in "prompted by a clueless incompetent person" with vibecoded as in "implemented by an LLM". Former is used to make your point, later is used to defend the premise.
> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
This example and Bend aside, I find this to be the biggest struggle with the perceived intelligence we have today. It's great at producing something that works, but it is not great at calling you out when you don't know what you don't know.
It's not able to educate and course correct you unless you have great self awareness and discipline.
That said, I think this goes for everything, it's easy to fall into this trap because it is very human. We simply don't know what we don't know, so it's not uncommon to revisit an old solution only to be enlightened that there is now new information that allows you to replace it with something much better.
I don't think anything here is new or changed, if anything changed is really just the rate that we experience this. LLMs make it easier and faster for the feedback cycle to happen.
Now back to Bend, I think putting your work out there and being unapologetic about it, open source even, and willing to take feedback, will go a long way.
I am more worried about the many closed source implementations of LLM built products that are being sold and people are depending upon that don't get this great criticism from many different thinking heads.
Agree with everything you say here.
> I don't think anything here is new or changed
I would argue it's a little new though. I used to write dumb little programs all the time that explored an idea which was probably bad, and in that exploration I often found that there was a better way to do it, or that I didn't know as much as I thought I did, or that another thing already existed that was much better considered than my half baked idea, etc. But there was learning that happened there, so the process was still valuable. Now you can get a working bad idea without learning anything, there is full conservation of ignorance, but a full dopamine hit from "i made this thing". I guess you could argue it's just everything happening at a faster rate, but it feels different to me, and it is pretty eerie.
Yeah that's the other side of it, learning that is.
Before you would pause at overwhelming, now you get to "fail forward" with less at stake because the end result can still be verifiable even if the internals are a blackbox to you.
It's a sort of "deferred" and/or "optional" learning dilemma we now exist.
You could open the box, look inside, ask questions, but you would need to care and feel engaged, which is very hard to do when the result is already there.
This is why I framed it as requiring self awareness and discipline. Very easy to get caught into the slot machine dopamine cycle loop.
Sometimes the end result is good, but the means aren’t. With the “give me something” approach of most LLM usage, often you won’t realize you have bad code full of hidden bugs. Often, the steps are “it works on my machine” follows by an error in prod.
Absolutely, and this is true to handwritten code as well. This is why code review, tests, constraints, and putting effort into the critical pieces of software is paramount to success.
I also would argue that in this case, the end result wasn't good but rather perceived good. It was good enough to pass a smoke test, but not an integration test, and end-to-end test, a runtime edge case, etc.
All of software development lifecycle should still apply and be relevant here, the tool just changes the rate at how it gets written and it is tempting to move faster and continue skipping other steps. Especially so in greenfield work as a solo founder with limited time.
I think at the end of the day, a good metric is your "give a shit quotient". How much do you give a shit about what you are doing is likely the most important thing into how it is going to perform.
That's an excellent point and kind of summarises the tension in LLM coding between the people being told to use it (developers) and the people doing the telling (managers). The latter are laser focused on the final product, the outcome, while the former are (were) more invested in the process.
Developers intuitively know that the development process would inevitably yield learnings that would shape and change what the final product could and should be, while managers typically dismiss this in favour of an illusion of productivity.
please, don't listen to Liam. The author is completely misrepresented in the blog-post :(
Agreed. Anyone can share insights into how to do it better?
How can we "course correct" with "self-awareness and discipline"?
Self awareness means would be taking a step back and knowing if you're caught in a vicious cycle, trap, loop, pursuing something mindlessly.
Discipline would be turning this into principles you practice constantly, it's more of a lifestyle rather than a skill you "obtain".
For example, starting from the position of "I am taking on an idea where I may have knowledge gaps in" you first prompt for research and validation rather than execution. Even if the end result is you were right on your original design. (Sort of a challenge, adversarial, by default)
It will slow you down, but may lead to more sticky results.
> but it is not great at calling you out when you don't know what you don't know.
or itself, the moment it hits some ambiguity it becomes a spaghetti throwing machine, half the time using a single attempt (arbitrarily picks answers with little logic or attention to nuance).
They've been trained to behave in ways that make them run longer, quantity over quality. I suspect this is because the people training them are extreme vine coders. Certainly seems that way by their public statements and harness releases.
I would much prefer if they stopped and asked questions. Mild improvement with markdown engineering...
I fully agree with this quote, it really crystallized something i felt but couldn’t put into words.
The original discussion about the project (https://news.ycombinator.com/item?id=49746163) is very weird. Lots of call-outs about how the author is some sort of celebrity and random accounts vouching for him, with little discussion on the substance.
Not even the demo on that release works well.
I posted a sharp critique in the original discussion, aiming to be civil while critiquing the project. I may have been a bit terse, and would probably rephrase some of it now to avoid confusion, but I don't think I was ever outwardly disrespectful.
I received several very emotionally charged responses centered in the personal credentials of the author. They felt very out of place and did not engage substantively with any of the things I said. It was indeed very weird
The author, who I hadn't heard of before yesterday, actually seems like a cool dude. He was quite responsive, normal, and engaged with my feedback, which makes other random accounts being offended on his behalf all the more uncanny
I think people responded that way because you strongly implied he was an unserious vibecoder who was just fooling around, and you called him suspicious as fuck.
Your post and the ones that followed are a good example of the contrarian dynamic that dang often talks about. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
>who was just fooling around
In my own professional life, I've found this to be a very divisive statement. For some, it is a sign of wasting time and effort. For others, they use this to describe themselves when they want to do exploration for the goal of finding improvements, without any clear goal because they have a few ideas but none worth putting forward. I've been told to spend time learning AI and have found that saying "Yeah, I'm playing around with it." was the wrong thing to say because it was seen as not doing anything worthwhile. It doesn't matter that I would also say the majority of my tech skills were developed when I was "playing around".
I wonder if this is purely a linguistics breakdown, or if this is tied to some deeper difference in a person's relationship to tech?
Agree it's divisive, and I would argue it speaks more to a person's perception of work vs play more than a relationship to tech. If (the general) you think that play is for children and work is serious biz, then yeah I could see how you wouldn't take someone seriously when they say they're "playing around with it". It's usually not obvious which attitude a person has though without getting to know them a little bit.
Yes, I think it can be a sign of a very deep difference. When they say "spend time learning AI," they mean work through some teaching materials to learn how to replicate what others are doing. This often doesn't result in a deep understanding, but it can be enough to allow them to do their job.
Ironically, people with this mindset will sometimes ask people who they recognize as having strong skills to share their magic secret, which is assumed to be some books they read, videos they watched, courses they attended, etc. If you tell them that experimenting, playing around, etc. is a key element, they may assume you're just selfishly hoarding your fount of knowledge.
> called him suspicious as fuck
I don’t see anything from your parent commenter on the other thread that deserves that classification. On the contrary, while they initially had suspicious of vibe coding, on later comments they are cordial and even admit their own misunderstanding.
What am I missing? Where does “called him suspicious as fuck” come from?
Edit: Answered below (https://news.ycombinator.com/item?id=49754311). Thank you.
"you [...] sound sus af" at the end of the comment. sus means suspicious and af stands for as fuck.
Thank you. I did indeed miss that. I did do a ⌘F for the individual words, but it didn’t occur to me they could’ve been written like that.
To be fair, writing "sus af" is different from writing "suspicious as fuck" (just like "wtf" reads differently from "what the fuck", etc).
Not to mention the author themselves say "Yes, there's a lot of vibe-coding in many places [...] We'll prune AI slop over time.", and then they both moved on to discussing the actual questions.
The whole "Wow, looks AI" > "Yeah, some of it is, we'll fix it later" was such a small part of the conversation, but then there are countless of other people chiming in about specifically the "Is It Slop Or Not?", rather than the meat of the conversation. And here we are adding even more meta-comments about it.
OOTL, could you give a link to a relevant @dang post?
https://news.ycombinator.com/item?id=25140587
> I received several very emotionally charged responses centered in the personal credentials of the author
I took a look and honestly the replies were a lot more reasonable than I was expecting them to be. I don't think it's that out of place for people to inform you that the author has been in this space for some time and has prior work to look at (pre-vibe coding era). Really there was one reply citing his prior work, you responded to it calling it a very strange reply and were "very confused" why they would point to his history, even though you had just said things like:
>I'm glad you're having fun vibecoding, and I like that you're interested in this area of research/engineering
To me, it makes sense why someone would say the author has been interested in this area for a long time and pointed you to some prior work that was not vibe-coded, given your comment strongly implies they are just messing around and don't know much about the field.
To be clear, I share much of your feelings in your original comment. I just felt the replies weren't so unreasonable either. At least, I was expecting them to be a lot worse.
I’ve posted a follow up somewhere else in this thread about how I can retroactively see why some people would read rudeness in my comment. Even though this isn’t intended, I agree I need better choice of words
What I didn’t understand when expressing confusion with the responses, and still don’t, is the personal nature of the response. It wasn’t pushing back against anything I said really, moreso it tried to bring up the credentials of the speaker; and, I guess this feels a nonsequitr
Like, if I say “I have these problems with thing X”, it doesn’t really matter who made X. Sure, it’s context I didn’t have and there is something to be gained in saying it; but, it doesn’t really change any of the critiques I had. Appealing to the authority of the creator doesn’t engage with nearly everything I said
> I guess this feels a nonsequitr
How is that a non-sequitur? You're the one who brought up the topic of credibility in the first place. The comment was simply made in response to that.
I don't think it was reasonable to call other poster a bully academic in need of a therapist. Calling the project vibecoded (perj.) and sus is very mild compared to what Reviewer #2 would have to say.
I agree with the other poster that is was very suspicious initially (sus af is not how I would put it, that sounds like a generational term) the author nuked their commit history while simultaneously pointing to a (Fable-written) paper which referenced the commit history in benchmarks. Then they seemingly had no understanding of why that's trust-breaking.
So to me, it sounds like the author didn't even read the paper they wrote, because otherwise they'd have remembered referencing the commit history, and wouldn't be asking us why we'd even care about it. “They” told us care! I'm glad they restored it but apparently that wouldn't have happened without the criticism.
It was a fair critique, but if you're looking for feedback I think people were probably responding to your last line.
"I'm glad you're having fun vibecoding" comes across as very backhanded and condescending. It sounds like you may have actually meant that genuinely, but it doesn't read that way in text form.
"you sound sus af" is not respectful or constructive in my opinion. It's a description of your own feelings, not a critique of the project, and there's not really any way for the author to respond besides ignoring it or saying "sorry you feel that way" or something.
I think that line undermines the rest of your comment, because I'm left thinking that you don't really expect good answers to your questions and you just think the whole thing is dumb.
Yeah, reading it again it sounds far bitchier than initially intended. Thanks for the response
I was writing the comment very stream of consciousness and not really think about how it may come across
If I were trying to boil down what I’m attempting to communicate, it would be 1. The project seems cool, but it’s also making some very strong claims that I’m hesitant to accept
2. The coolness of the thing is undermined by the presentation of it. It comes across as putting the cart is put before the horse, and the overly strong claims and marketing speak read as trying to rhetorically sway the audience rather than engage with them technically.
3. I genuinely am happy that the creator made this, but modulo the above worries I think it should be reeled in a bit. In part because of the concerns I have about the content, and further because it is the kind of language that I expect others to have a strong averse reaction to. Possibly to the point of also reaching the top of HN with their negative response
To a friend, it might be easy to capture some of this message with “you sounds sus af”, but to a stranger in the internet I see how I just sound like a jerk. Words do matter, and I think I’ll be more careful about this in the future
I'm not familiar with the author, and I'm still mildly offended by your comment.
If I have years of experience on the topic and invested significant time into the project, I'd not be as civil as the author if someone came in and essentially called it vibecoded slop.
There is nothing weird about how others pointed out that the project appears to have merit contrary to how it at first might have looked to you.
You posted a passive aggressive as personam and are (or at least act) surprised for getting called out.
[checks profile]
If this is your first HN account and you haven't seen the site in the prior decade, the "emotionally charged responses centered in the personal credentials of the author" is the SOP. The site has yielded to that attitude because dang never enforced a sensible code of conduct, and relies mostly on favoritism and in-groups. Who says is more important than what is said, and the critiques you give are ranked according to whom you are critical of.
Think of this as a propaganda channel of a VC startup incubator who is brazenly looking for a product-market fit for anything and everything AI. Look at the roster of the recent startups and how many are AI-focused. Adjust expectations from there.
Do you have examples of sites that you prefer that achieve this, or have a relatively high signal to noise ratio that also have a high volume of discussion? (preferrably not an individual's blog + comment section)
Lobste.rs
Hmm.. anything else?
I am relatively new to the site, and I guess I’m just surprised by this. I thought the dweebs here would mostly care about the technical content
But I guess I totally ignored the YC context in that interpretation
The whole original conversation was very smelly from the very start, 20k stars included on the GitHub page with lost history.
The Bend1 launch was very successful, with 1000+ hn upvotes [0] and with several youtube videos with up to 1M+ views [1] [2] - that's where a lot of the popularity came from, I believe. They just recycled the Bend1 repo for Bend2 even though it's a completely different language.
[0] https://news.ycombinator.com/item?id=40390287
[1] https://www.youtube.com/watch?v=HCOQmKTFzYY
[2] https://www.youtube.com/watch?v=NaytZOiX3fs
> ...on the GitHub page with lost history.
> They just recycled the Bend1 repo for Bend2 even though it's a completely different language.
Why would you kill the source history of Bend1 completely if it has 20K stars? You can only do that if Bend1 has no users at all right?
Does that meant that Bend1 isn't actually successful in its own right but rather only as a marketing project?
I wasn't one of the suspicious people but I am now.
(author here) Bend1 indeed has no significant active userbase
I don't think that means it was "unsuccessful" in the sense you imply, though, because the project was never meant to be used in production. It was there to display a milestone (running inets on the GPU) and I was very clear it wasn't ready to be used yet. For example, it had only 24-bit integers, a 2 GB memory cap, and other limitations that made it unpractical. I still don't know why it has so many stars. I posted it to hacker news and that just happened. I guess it just went viral without really being ready yet, which got us to where we are now.
Anyway the commit history is back now. I apologize for nuking it
totally not sus
It _is_ sus, but you should be willing to forgive. It's not like it was done out of animus, and I assume that an academic/foreigner is unaccustomed to how production software conventions work. Their baseline is less important, more important is how quickly they learn/adopt "best practices". Y-intercept, slope, etc.
I say this as literally the first HN comment to call them out on this.
it becomes more sus as the superfans come out
the author here is providing commentary on the vibe coding era, bend is just one example of people having ai build things for them they don't understand or haven't researched sufficiently, at least start with a vibe-search skill before the vibe-coding begins
Ai is only good at a task when the human driver is good at that task, and this is far more narrow than most realize. Someone who knows how to program will still fail on many programming tasks with agents, the field has far too much for any one person to know.
It sounds even worse. Actually far worse. It's a project already in use and the author just nuked the whole git history?
It's a research language, of the more experimental and niche type.
No one uses it seriously, and the author since then reverted the nuking of the git history.
Does anyone know where they bought the popularity and contributors from? I would like to do this for my joke language to fool unsuspecting users into using it seriously
Probably at the same market where they sell snark. You may want to sell your excess of it, especially that as mentioned there were a popular (as in got to HN frontpage multiple times) version 1 and it's all legit and honest work, getting uncalled criticism?
Please do not shill your crypto here, nobody wants to buy snarkcoin
The history is back...
Yeah 100% those are purchased/fake
Lean4 itself has 9k
I don’t they are purchased or fake.
I can unequivocally claim that Victor receives at least two orders of magnitude more social media engagement, while Lean likely has 4-5 more magnitudes of actual users.
Serious users of Lean simply have no need or reason to star the repository.
(Author here) What about it isn't working for you?
Also, wrote a response to this whole thread here:
https://news.ycombinator.com/item?id=49753898
I cannot understand the hostility being directed towards you for this project. It seems very interesting. Your reply was very well thought out. I am very confused.
Seems to be a fair deal of astroturfing going on for some unknown reason. I can't tell if it's intentional to drum up controversy, or it's genuine because people are bitter about this for some reason.
I think people are genuinely bitter about it, that's usually how my projects are received
Approach itself looked impractical to me for any non-trivial system, like domain centric system of records systems which can have 100s if not 1000s of laws. Though it can be tried as a side parallel thread to see if system is still compliant and following right first principals after a few years from its inception.
I would rather wait to see how it gets adopted, if at all. Anyone aware of early reviews of the adopters of bend 2?
As a matter of fact, I've been poking at this from a slightly different direction: "axioms and invariants" within home automation scenes.
Invariants were like "if outside temp < 40 or inside temp < 65: heater.minTemp( 65 )"
Axioms were like: "if {we're home} and it's {not a holiday} the house should be {comfortable temperature}".prompt
...and then that would get decomposed and translated into interlocking code for the scene(s). I'll have to look at this language a little more closely with those kinds of constraints in mind!
You're kindof translating `*.prompt` to either prolog (yucky!), lisp, lua, or javascript (for inspectability/debuggability), but this whole bend thing might be an exact fit for the problem space! Limited set of objects and states, bounded set of "invariants" (laws), and layering on top the general state modification activities (either "evaluated every 5 minutes and reconciled" or "set the scene xyz...").
Hmm, in your example, temperature is fully measurable but there might be a slight challenge in measuring if "If we are at home". Since for this you need to define more variants with measurements from motion sensors, pressure or weight sensors on beds or sofas, if a device has got connected to wifi, etc.
By the time your first round of beta testing is over, you may have quite a handful of such axioms and variants, which have been humanly validated!
Have you done any such experiment in this space?
>Not even the demo on that release works well.
>vibe-coded project
many such cases
HN keeps being gamed to keep vibe coded slop on the front page and give it credibility. There was another language project a few weeks ago called ‘Wyzer’ that was just completely generated and largely nonsensical (didn’t do the lofty things the documentation claimed it did) and all the comments were glowing and harshly dismissive of any critique, it was bizarre
The number of times I've looked into something that was highly upvoted and purported to solve a problem only for it to be unreadable crap that barely functions has definitely skyrocketed over the past few years. In a way it's similar to how 80% of people only read the headline, except it's psychosis around the equivalent of a dressed up pile of trash.
The whole thing smells like somebody trying to pull an LLM-fueled Sokal Hoax on the programming world.
> The field in question is formal verification. It’s notable that those two words appear nowhere on Bend’s webpage or in its codebase. The developer has built an entire language around a field seemingly without realising that said field exists.
I checked the developer's X account, they have written numerous posts about formal verification, so this specific claim ("without realising that said field exists") seems to be false.
It's amusing to me this entire article is centered around brow beating this and other hypothetical software authors for starting things without doing a small amount of research first to understand the very basics of what they're getting into.
If the author of the article had done just a small amount of research about bend or it's author before writing the article they would have known pretty quickly what they were saying was incorrect.
I think the larger pattern here is that nuance is one of the most valuable commodities in the AI era. If you're hand waving stuff away without even missing , you're going to miss a lot of stuff in this cycle.
This article reminds me a lot of the famous hacker news Dropbox comment.
Back in 2018 they were working on Formality, an Ethereum formal verification project. They are the Victor in this video about it: https://slideslive.com/38911748/introducing-formality
Here's the GitHub repo for that, which demonstrates familiarity with formal proofs that long predates LLMs https://github.com/VictorTaelin/Formality
Victor here. I haven't "worked" on Formality. I've founded it. Designed every part of it. Before LLMs!
sighs
Here's my response to this ridiculous accusation: https://news.ycombinator.com/item?id=49753898
I can't internet anymore. I need a beach
Sympathize with you mate, this article just seems like a poorly researched hit job.
The article is about vibe coding, bend is the main character because it made frontpage.
The author here says as much in the introduction, that it is not about whomever is behind bend, but the larger trend
The author here has also added bend's author's link (in GP) to the original post, they very much do not seem to be doing a "hit job" and their intent is to comment on patterns from vibe coding
Bend's developer has posted a well-argued response here: https://news.ycombinator.com/item?id=49753898
I am glad I saw it, as now I am interested in learning more about Bend.
To be fair, if the two words indeed don't appear in either the webpage or the codebase, it is a bit strange. It's like implementing a whole Google alternative without ever using the words "search engine".
He claimed the guy did no research while himself doing no research? I'm shocked! Shocked! Well, not that shocked.
I don't want to change that sentence now that people have discussed it, but I have added a note to the top to make it clear that I'm just taking it as an example of a vibe-coded program because it's recent and high profile.
My critiques of the language itself are not the main point, although I do still think that it's a very bad design to have a LLM waste tokens on a proof that could be written by CVC etc..
>I don't want to change that sentence
>but I have added a note to the top
you know what's the least you could actually have done instead? no, you don't need to retract the blog post at all, keeping it up was the right choice. Now slap a big ass apology for being an unaware snob on top of it instead of leaving a link to the author's reply, like an after thought.
> Look like how AI slop has unrealistic physics
> Posts a link to real moon landing footage
I'd delete the article if I was you...
You know, in academia, they sometimes retract articles, even if they believe they are directionally correct
I feel that that's the worst option because it only leaves people who have read it without the added context at the start. If someone convinces me that I'm wrong then I'm happy to do so though.
You should add a strikethrough to your post and post a public apology to the author.
> I'm just taking it as an example
But this is grossly intellectually dishonest. You know very well how this will be read and responded to here ... and you keep saying that you're just talking about vibe-coding oh but you have serious criticisms of the specific effort. You write passive-aggressive stuff like
> For all I know they did make an informed decision regarding the tradeoffs (which I would consider to be a poor decision).
which contradicts your base assertion that their decisions were not informed. And
> My critiques of the language itself are not the main point, although I do still think that it's a very bad design ...
You claim
> The developer has built an entire language around a field seemingly without realising that said field exists.
but that is severely factually wrong, which along with a lot else suggests that you have very bad judgment. As the author writes,
> Bend proofs being verbose has nothing to do with me not knowing that inference, unification, or program search exists.
IOW, you have made a serious error in logic.
> To be fair to Bend, I completely vibe-coded this
Some advice: DBAD
You trashed the author and his work without bothering to learn anything about either one first (which is quite ironic).
I won't respond further.
> Where this differs from Bend is that what we have supplied here is everything required to prove the correctness of the program, without having a LLM waste time and tokens on building up a 442 line proof from first principles. We can run GNATprove and get: `Success: all checks proved (12 checks).`
GNATprove uses SMT solvers, meaning it's basically a brute force proof system.
Yes, brute-force proofs are easier than symbolic proofs (lean, bend, etc.) because you don't have to supply a proof. It's all automatic.
But brute-force proofs don't scale to nearly anything of interest, which is why formal verification has been a niche field for 30 years, until now where LLM can write _actual_ proofs.
Pretty impressive to accuse the author of not knowing formal verification when even minutes of research would immediately prove the opposite (https://x.com/victortaelin/status/2100942399132312059?s=46, https://x.com/victortaelin/status/2100374221671051472?s=46).
It’s unfortunate because what the OP describes is a real problem. Regardless of whether or not Bend2 is realistically usable or not, the author definitely does not fit the description of the type of people who are actually causing the issue.
I haven't looked into Bend 2 in detail, but it seems a bit harsh to call it "horribly broken or decades behind the current state of the art". Clearly there is a problem with formal verification languages and there is a demand for something else in that area, and the problem is the usability and syntax. I don't want to have to learn something that looks like Haskell, or to have to wrap my head around Curry-Howard correspondence. I don't want to write my conditions in something that looks and feels like C++ template metaprogramming. I recall a Hello World in something like Coq a few years ago which basically started with "first, we construct the Peano integers", and then they used this to prove that some calculation was bounded - because it seems they couldn't represent integers natively?
I just want to be able to write C#, JavaScript or whatever, and then tack on preconditions, checks and so on with the same syntax. Dependent typing and design by contract for the masses.
It sounds like the crux of the issue here is that you don’t want formal verification in the first place. Your last paragraph sounds more like code contracts, which is also a thing that already exists.
Well, yeah, you have code contracts in Ada or Spec#, (very limited) fixed type ranges in Pascal, ... but no general purpose programming language lets you put an arbitrary expression in the same language in a type as a permanent condition, or lets you state facts that the compiler will prove.
Of course not, that would be equivalent to solving the halting problem, many people will say.
I wonder if that will change now: I'm happy with an imperfect sanitizer that I run every now and then and will run a couple of minutes and come back with: I've proved your conditions, I proved a violation, or I can't decide, please change your code.
> no general purpose programming language lets you put an arbitrary expression in the same language in a type as a permanent condition, or lets you state facts that the compiler will prove
Lean can be used as a regular programming language. There's also languages like idris2 and f-star, but they don't seem to have much traction.
Well, they of course not defy computer science. The "trick" is that they are not Turing-complete, they mandate termination of every expression.
Also, most of them are made to prove stuff first and foremost and thus trade off a lot of performance to the point that it makes them practically unusable for many stuff (e.g. numbers may be represented as an object that has n-1 further children recursively), though Lean is an exception as you note.
> The "trick" is that they are not Turing-complete, they mandate termination of every expression.
I don't think this is a big deal for day-to-day programming. You're trying to stay in n, nlogn or maybe n^2 realm most of the time. And the kind of infinite loops you encounter (e.g. event loops) are co-inductive or have some notion of making progress.
> "horribly broken or decades behind the current state of the art"
This is just about vibe-coded programs in general when the approach assumed by the article is taken. For all I know they did make an informed decision regarding the tradeoffs (which I would consider to be a poor decision).
> I just want to be able to write C#, JavaScript or whatever, and then tack on preconditions, checks and so on with the same syntax.
That's more or less what SPARK (and others) do, although specifications for large programs can become nasty.
> The author of Bend has completely missed that this is the current standard in the field of formal verification, if they even know that this field exists at all. They have instead come up with this whole system requiring verbose specifications and even more verbose proofs. A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.
> This example matters beyond Bend, vibe-coding makes it makes it far too easy to implement a design that’s horribly broken or decades behind the current state of the art because you can immediately get a result without ever having to do any research. If you ask a LLM for a language where it’s possible to prove that a function is formally correct by building up a proof from basic principles then it will happily do so, it will never stop to suggest to you that computers can already build complex proofs without the need for a LLM and eliminate 99% of the work. It will never tell you that what you’re building already mostly exists as work that you can build on.
---
That's why all your LLM requests to build something substantial should start with "run prior work research first". Of course, at some point everything converges (if we share our outputs open-source) and then we may have solid standard patterns and libraries and do not need to waste trillions of tokens globally to rebuild the same minor, fundamental things, each one in their silent little silo.
IF we share, it will be of course to the monetary detriment of LLM providers who will have less income overall, and of course now they can't repackage anymore all our collective input, thoughts, human 'thinking traces' that they collect in their meta-data, as their new 'innovations' any more to inflate IPOs / stock prices.
Although, I have known using an LLM to do your prior research to itself be a trap.
The searches it runs, and the summaries it provides, are all incredibly sensitive to your choice of words. Words you chose from a state of minimal knowledge. So it’s like a particularly perverse version of the anchoring bias: information that could have led you to a better solution is often actively filtered out of the agent’s response precisely because it leads down a different path from your first idea.
In short, if you ask an agent what’s the best hammer for driving screws, it’s liable not to mention that screwdrivers exist.
Beautifully written. Hammers for driving screws + sycophancy are horrible together.
Is it?
As in, are you sure, and can you provide concrete examples?
I just tried it with chatGPT asking for whats the best hammer to drive in a nail, and it gave me a list of hammer suggestions :p . I drove it a bit further and specified wood screws and while it did point out a screw driver would be better it still kept recommending more specific hammers for driving the nails instead!
Depending on what I’m doing I’ll dedicate a few deep research sessions to building a framework. It will generate some grounding docs that go into the repo and get consumed as we go. Said docs establish terminology, widely known formulas and methods, etc.
And yes the output of these researchers are highly sensitive to prompting. Left to their own devices the LLM will often ship some very biased prompts to its deep research agents loaded with pre-conceived ideas rather than letting the agents uncover things themselves. Then all the agents do is confirm what the prompt told them to rather then “think independently”. (Very similar to open ended interview questions rather than asking yes/no questions)
It’s is far better to spend a session writing writing the research prompt itself.
All of this takes time and tokens of course…
Yes this is precisely why experts drive LLMs so much better than novices. It’s also why I don’t dismiss pure LLM output as slop and uninteresting; even if we both are using the same LLM, I probably can’t make it output what you can make it output.
Or maybe it should be a flag to indicate that there might be more to think about before handing the task of to an LLM.
> That's why all your LLM requests to build something substantial should start with "run prior work research first".
Yes, but I think there are incentives to not do this for many LLM providers. Doing prior-work research is slow (web searches aren't fast, LLMs are rate-limited or blocked from plenty of pages, etc.), and sometimes contradictory which annoys LLM users, many of whom like faster gratification cycles from the agent slot machine handle.
Also, writing a bunch of bespoke code instead of leveraging prior art makes a lot of users feel like they own something novel/big/important, and also poses a larger maintenance surface for the LLM to make future changes (which costs tokens).
I don't think there's, like, a conspiracy at LLM providers to set up system prompts/RAG/etc. to discourage research-and-use-prior-art-by-default approaches. Rather, OpenAI/Anthropic/Google/etc. are optimizing for real but sometimes misleading success metrics which often lead away from a research-first approach.
- That should nowadays be very obvious. You start any substantial work by doing research. LLMs are [perhaps surprisingly to some] good at that. They are also pretty good at doing the plumbing. That is a godsent in the hands of a good researcher and architect. And I don't have any qualms if tokens are "wasted" on research or on the precursor work. That doesn't replace reading good books on a subject but it DOES replace finding and filling knowledge gaps.
- But that shouldn't be confused with getting the LLMs to make the decisions. I believe that would quickly ruin a good design, unless the decisions are about truly inconsequential aspects, which are very rare in language and API design.
- I don't believe that (sharing) is to the detriment of LLM providers either. Not realistically. We would build faster and the questions / research directed at LLMs would be more sophisticated. Believe it or not, they can't cache questions as easily as websearch queries. If anything, I believe the more people learn to use LLMs effectively (rather than just to generate slop), the more their usage will be ingrained in daily life. Some of that will be redirected towards current LLM providers. But perhaps more of the economic share will increasingly go to hardware providers, as more and more people will be interested to run their own models.
Humans dont let LLMs raw dog internet web searches.
> "run prior work research first".
As effective as “make no mistakes.”
It is trying to please you, and it always determines that the way to please you is to fulfill the original, core request. Any caveats or first steps will always be secondary to the ultimate goal of “this person wants to do X, so I will do X.”
The only first step I have found somewhat consistently useful, because as we know LLMs do not behave consistently, is when doing tech troubleshooting I will go “look at documentation for X before answering” so that it will search manuals and such. Helps avoid speculation. But even then, it’s still not full proof.
Sidebar: this is one of the core problems of LLM’s currently. You are basically arguing with them to get them to behave a certain way all the time and it’s not always clear if they’re doing what they’re being told to do. Then add the compounding layer that the longer the conversation goes on, the more likely it is to misunderstand or just ignore things as it descends into context-length-induced madness
> As effective as “make no mistakes.”
Those aren't comparable instructions. Providing useful, related context to improve outcomes is a basic best practice, and asking LLMs to do research first is an important source of that.
I’ll admit I was being overly tongue-in-cheek with that. You’re right it is more useful. But it also can’t be trusted to do that reliably, it’s primary goal is to give you the core thing you asked for - and often this is regardless of the parameters you put around it.
Fair enough, knowing when to be and not be prescriptive about implementation details is critical too!
Some noteworthy lines from the README.md[0]:
> - The compiler (not kernel) is 99% AI-written and has not been fully audited yet.
> - Strings are linked lists of characters, so text processing is slow.
[0]. https://github.com/bendlang/bend/tree/main
So are Haskell's, since 20 years ago, with no options for years?
We will introducing binary buffers eventually. The project is new...
“No options” simply isn’t true. Here’s a guide to many of the options: https://hasufell.github.io/posts/2024-05-07-ultimate-string-...
Now if you’re asking why the basic prelude String type remains as it is, that’s because changing it would break more code than it’s worth, at least as far as prelude’s maintainers are concerned. This is no different from how standard C strings remain a null-terminated sequence of bytes even though that’s been awful for everyday use for at least 30 years.
Note that using linked lists for strings is actually more "parallel friendly" because you can take the head/tail and spread it around 16k GPU cores in O(1), unlike in Haskell, unlike arrays, which require a linear copy, becoming quadratic. So, the right "default type" isn't that clear on Bend, because GPUs behave very differently from CPUs.
That said, yes, we definitely should have a compact Text type. I'll add it over the weekend.
Though also, parallel processing strings and other non-numeric data on that level of granularity is, IME, typically less performant. The parallelism rarely manages to offset the performance penalties incurred by decomposing the problem in a parallel-friendly way. Even on a single machine you’ve got to think about whether organizing the data in a parallel-friendly way also makes it less cache-friendly. For example, a linked list of Unicode code points is 12 bytes per character, and each character might be on a completely different cache line. Depending on language a UTF8 buffer might be 1/10 the size and have a much more compact layout in memory.
Related:
https://www.usenix.org/system/files/conference/hotos15/hotos...
Yes, love that paper.
Anecdotally I have a bit of a track record of 10xing slow systems’ throughout by converting them from distributed to single-node or from multithreaded to single threaded.
Heck I once even sped up a number crunching operation by getting it off of the GPU and onto the vector coprocessor. Because GPUs also have a bunch of extra overhead to have to amortize away.
Splitting a linked list is an O(n) operation.
You can slice up arrays in O(1).
The default type is incredibly clear to me.
To clarify, It looks like Haskell got better string types around 20 years ago.
> > - The compiler (not kernel) is 99% AI-written and has not been fully audited yet.
so vibecoded
If you have to write LAWS.bend which is pure code describing the laws then it isn't basically like those old days of writing unit tests and that too tests first hence the TDD?
So what is the unique idea here except a vibe coded compiler that generates C and everything else is handled by clang+llvm?
From README.md:
>The compiler (not kernel) is 99% AI-written and has not been fully audited yet.
Also, why the compiler is not written against and with LAWS.md so that no audit is required at all?
> If you have to write LAWS.bend which is pure code describing the laws then it isn't basically like those old days of writing unit tests and that too tests first hence the TDD?
There is a key difference: the laws are formally verified, as in Lean or Rocq (but much faster). So it's like writing a unit test or property-based test, but when it passes, you have a mathematical proof that you will get the expected output given ANY input in the infinite space of possible inputs. In traditional TDD, you make up some test case, write some asserts, and it passes if you get the expected outputs from those inputs and only those inputs. So you have to make multiple test cases for the same thing, and you still don't have any formal guarantee of your code's correctness.
> Also, why the compiler is not written against and with LAWS.md so that no audit is required at all?
Because it is mathematically impossible due to to Gödel’s second incompleteness theorem, which states: any consistent formal mathematical system strong enough to harbor basic arithmetic cannot prove its own consistency
> A little research before vibe-coding an entire language and compiler could have substantially improved the result because the author would have known what to ask for.
A little research before writing and publishing a personal attack like this could have substantially improved the result because the author would have known what they're writing about
Victor is not a formal verification noob as this article suggests
> It will never tell you that what you’re building already mostly exists as work that you can build on.
It will if you remember to ask it. I've got into the habit of starting any new project with a session where I ask a search-enabled LLM to help me figure out what the prior art for a problem is. It's saved me quite a bit of time.
It seems implicit in the article that the author should have remembered to ask, as part of the prior research.
I really wish there were search harnesses, actually. My LLMs are lazy as hell and seem to want to just report the first thing they find on google. I know they can return truly niche and useful results, but it takes a lot more prompting to get them there than I would like.
I've been using GPT-6 Pro in ChatGPT (and its predecessor GPT-5.6 Pro) and it's been incredibly effective. I don't think it's available on anything less than the OpenAI $100/month plan though.
you know that most big ai companies not only have search harnesses but also sota models that are post trained for web search specifically.this is what the deepsearch option is in most cases. and they have been unbelievably good for years now.
The recent OpenAI wiki spam indecent was interesting because it helped reveal some of how they are training their models for research tasks: https://collusion.wiki/
See also this piece on misalignment from OpenAI https://alignment.openai.com/misalignment-reports/self-gener... which reveals one of their RL tasks:
> In mid-2024, <redacted> published a list of the best books of the 21st century selected by a panel of experts. It also published a separate list based on readers' votes. I would like to know whether my local public library's collection included the books on the latter list as of August 24, 2024. Consider only the top 50 books, and exclude nonfiction and poetry collections.
Isn't the usual argument that all AIs can do is build on prior art? Like, I spend a disproportionate amount of time trying to convince my agents that I don't want to just reimplement the Rust borrow checker for my language!
Think you could argue that’s more LLM-assisted engineering than it is vibe coding.
This is just another facet of the sycophancy issue. They really need to start RLing these models to gently push back the way a friend would on things that are questionable.
It would also make sense for them to just RL in the "research prior art" arc as part of that "is the request sensible" pre-analysis. It wouldn't be difficult and it would save everyone a lot of pain.
Yeah, for this to feel credible you need to make sure the model doesn't figure out that you're planning to build a solution yourself.
What a weird reception there’s been to bend. People discrediting the author without bothering to look him up, and then getting defensive when others point out the fact that the creator of bend has a very long very public track record of work in the field.
The irony of this post talking about vibe coding and not doing one’s research, on only not have done even the slightest inkling of research themselves (heck, even asking an LLM about the author would for sure have turned something up).
I hope people will give it a second look, and not just stop at this post which is a gross misrepresentation of Victor Taelin’s work.
I am a total outsider when it comes to this topic, but it reads like the author is doing the thing they are accusing the Bend 2 guy of. Good juicy reading while I sip my coffee!
"The developer has built an entire language around a field seemingly without realising that said field exists."
That is incredibly funny.
Here's a talk about formal verification I made 7 years ago @ DevCon:
https://www.youtube.com/watch?v=0fg1QbeeqNU
Here's Cedille Core, my implementation of Aaron Stump's self types, a Computer Science professor who taught me a lot, ~8 years ago:
https://github.com/VictorTaelin/Cedille-Core
I also implemented Kind-Lang 5 years ago, way before LLMs:
https://github.com/higherorderco/kind
I dropped out of Federal University of Rio de Janeiro to study this subject independently, because I was passionate about it, and I spent nearly 10 years doing so, daily, on weekends. That's what I do.
Bend proofs being verbose has nothing to do with me not knowing that inference, unification, or program search exists. Kind had these, 5 years ago. In fact, I've also been researching the later, and I built SupGen, which overperforms every published symbolic program synthesizer in the literature by 10x or so. This is unpublished yet, but you can find my posts about it 2 years ago on X (I'm @VictorTaelin).
So, why is Bend verbose???
Because it makes it fast. It is intentional. It is my vision that a good proof language should be fully explicit, because this reduces proof-checking time significantly. That is what makes Bend realistically 10x-100x faster than every alternative.
But wouldn't that mean it is much harder to write it?
No. As you said it yourself, we have tools that can fill these proofs today! Not just AI models. You can apply these tools to produce Bend proofs, while the language itself remains a thin, dumb proof kernel that does one thing, and does it well.
If nobody is reading these proofs (because they're written by AI and automated tools), then, it is, in my opinion, irrelevant, as proofs will eventually become a layer nobody looks at, just like generated assembly.
Of course, I could be wrong here!
But it is misleading, if not just a bit malicious, to claim I "vibe-coded" a language without knowing about a field I've spent a decade researching about.
Every single part of Bend is an intentional choice I made after considering every alternative. I use LLMs to fill code after I make all hard architectural decisions because they type faster than me, and I'd rather spend my time doing useful experiments than typing trivial functions, even though I could.
Incidentally, deciding what I should NOT include took me way more time and effort than any line that was shipped, and there are perhaps millions of lines of code, manually written by me, that I threw away, backing up these 4k that went into the final design. An artist once told me you must first paint a Rembrandt before you can draw a cartoon that's simple in the right way, yet that might mislead someone who has never drawn into thinking you don't know what you're doing. I guess.
I was really confused when reading the article that their claim is you don't know formal verification, I was expecting something way juicer.
Hi! I'd like to know how bend2 differs from Hoare logic or Lean?
Hi!
I'm just leaving you a comment to counter the influx of negativity coming towards you for no reason.
I hope you are doing well and won't take all this too badly -- try to think of it as just free marketing for your project!
Have a nice day!
>That is what makes Bend realistically 10x-100x faster than every alternative.
Do you have benchmarks which show this?
I was with you until this.
> I use LLMs to fill code after I make all hard architectural decisions because they type faster than me, and I'd rather spend my time doing useful experiments than typing trivial functions, even though I could.
Are you being facetious? Because I've seen some bad takes from people trying to justify their AI coding practices, but this is the first one that actually made me laugh. It's clear that this public attack has you in a defensive mental state (which is perfectly understandable), so giving you the benefit of the doubt, I won't pile on and take everything you're posting too seriously.
However, I gotta point out how ridiculous this statement is, because it's the kind of thing that'd make me wonder if someone has any programming experience at all. Typing speed is not, never has been, and never will be a limiting factor in software development. "I didn't do <X> because of <reason> even though I totally could" is a pathetic cope.
It kinda reminds me of a recent post from John Carmack where he said (paraphrasing) that he originally wanted to release Idtech to the public domain as a "gift to the world", but couldn't do it because of his pesky business partners, but he totally would've done it. Let's just ignore the part of the story where he became filthy rich from licensing his software, and focus on how good it will feel to call it a "gift to the world" when an AI company integrates your code into their commercial products without licensing it from you.
> But it is misleading, if not just a bit malicious, to claim I "vibe-coded" a language without knowing about a field I've spent a decade researching about.
Sorry. See the edit at the top if you haven't already. I didn't realise how much it came off as a critique of you rather than a particular approach to software engineering.
It's easy to write something and have a model of what you're writing in your head that is massively different from how someone else will read it without realising, not that that excuses it.
---
I disagree with LLMs manually writing proofs without other tools doing all the work they possibly can ever being a good solution for a couple of reasons:
1. Tokens are really expensive when we have a LLM spending hours hacking aware at a proof, not to mention generating those tokens is slow.
2. The context window becomes flooded with proof work rather than work on the original problem, which will lead to a worse solution. LLMs are demonstrably worse at writing code when you continue a session on a new task instead of starting a new one.
> It is my vision that a good proof language should be fully explicit, because this reduces proof-checking time significantly.
We can cache the results and help the checker along with assertions rather than throwing out all the smart parts of the checker.
I would encourage you to think more deeply about the assertions you're making here.
I've done a fair amount of work in this space as well, specifically my main toolbox of formal verification tools in the past have been Rocq, Idris, Dafny, and TLA+, and I can say that I've come away with roughly the same set of tradeoffs as what LightMachine describes in his comment.
Current formal verification tools are often very slow precisely because they try to reduce the number of lines of code that are required to write a proof. By making proofs more verbose and more explicit, proofchecking is sped up immensely (my own experiments check out with what LightMachine is saying here; indeed I've seen even greater speedups in the range of 100-1000x).
It makes far more sense to pay a series of one-time costs in LLM tokens that reduces your compilation time from 1 hour to 1 second than to pay the 1 hour compilation cost again and again (these are not exaggerated numbers for larger projects). This is especially true because with modern LLMs, it's usually just fire and forget and let it churn in the background than anything else.
Caching and incremental compilation has a lot of limitations, e.g. for CI. This is the promise that languages like GHC Haskell have promised for a while that always gets blown away by the other side like OCaml where global compilation is just so fast that you don't have to deal with those limitations.
You don't have to let the ATP do everything, you can speed it up immensely with well placed assertions where it struggles. Maybe a hybrid approach is best where we run ATPs with a very low timeout to get all the easy stuff and then have a LLM write a proof using the thereoms that the ATPs were able to prove.
Silly question, but in Rocq, just for example, what does prevent you to fire `auto`, then, when it solves the goal, to just substitute it in your proof with the term that it constructed? Not calling you on BS, but genuinely interested in the problem
1. LLM inference is cheap compared to the cost of a bug, and you can use tools to automate the proof-work, as you said so yourself.
2. Why would it? The AI shouldn't load already completed proofs, only their type signatures (which are one liners). Sounds like you're projecting bad context management?
Caching only works until you change a deeply nested constant. Then you must wait 2 hours as Lean re-checks every transitive dependency of a massive simp-ridden codebase. Also, first time check time IS relevant to UI. Nobody wants to wait 2h to start using a library after they first download it. Perhaps you don't mind that but I do; that's the kind of bad UX that made me go for that design choice on Bend. Which again, may be wrong or may be right. But it was intentional.
Having a checker that is just fast by construction, regardless of caching, was a top design priority, and I weighted it over verbosity in a layer I believe nobody will be reading anyway.
> you can use tools to automate the proof-work, as you said so yourself.
The language doesn't appear to be designed around supporting existing tools (either by exporting to Why3 or manually interfacing with existing tools). I'm not against shipping the whole proof or storing it on a cache server, I'm against the idea of having a LLM write it all. Even having the LLM only write proofs for subprograms that take a long time for ATPs to prove would work.
For the example in the article, the LLM had to write it exactly once without any iteration and it proved in a second, which I assume was mostly startup time. Having a LLM write 442 lines instead, which I assume also needed some iteration, is a tough sell in comparison.
I'm curious, have you actually tried Bend (for more than 5 minutes)? Or are you more of an expert on this topic than the author who has been deeply immersed in this for 10 years, after it's single day of release? From an outsiders perspective it feels like this is personal to you, and you're being unfairly critical.
It will, there is just too much to do! It is a programming language. With a consistent, non-standard proof system. That compiles to GPUs. We're a small team. Please be patient
You mean the SPARK example?
Because it's a toy problem and I would definitely not take much away from it. Most proofs are basically write-only and impossible to maintain -- you are better off throwing it away. And I mean human-written formal verification here, like Coq and Agda. Take a look at formalized mathematics, they are absolutely massive code bases.
I think people don't have an intuition of LLM costs and the time saved by using it. Of course it costs but its obvious that it brings more in savings than it costs.
> 1. Tokens are really expensive
Just don't use frontier models by American labs. Chinese models are much cheaper and competent enough for most stuff
> 2. The context window becomes flooded with proof work rather than work on the original problem,
This can be solved by doing proofs in subagents. You can even have a tree of agents, for demanding proofs. They can share a database or otherwise have communication channels if some of them stumble upon something relevant for another agent
The problem with using subagents is that you often have to rewrite a chunk of a program in a more proof-friendly way, just saying "go prove this code, don't edit it" doesn't work. Maybe I'm underestimating how effectively subagents can communicate though and they'd be fine asking for changes.
Nowadays agents know a lot of tricks on how to effectively prompt subagents
Generally speaking the ouput of a subagent is sent directly to the calling agent, that can make the final edits. Or, the subagent can edit the file in a local copy, and the calling agent can diff it
Moving the goalposts a bit here, aren’t you?
Clearly the author knows what he is doing, even if you think it isn’t a sensible approach.
It might still be true that LLMs make it too easy to create a broken design and don’t warn you if you ask for something already exists.
The thing is, the example you decided to pick on does not work.
You accused the author of having fallen into the ‘vibe-coding trap’. He clearly did not.
Post author just wanted quick clicks based on kneejerk heuristics based attacks on Bend. Now he’s doing gymnastics all over this thread to mitigate what he said. Sad and disappointing.
Yes, I used Bend as an example because it is recent and high profile, and I also wanted to present my issues with it. I did not mean to conflate it with the main idea I was trying to present to the degree that I obviously did after reading my own writing as a third party would (at least to the degree that it is possible to do so).
> Tokens are really expensive when we have a LLM
Expensive compared to what? Vs. human writing proofs they're definitely not.
> The context window becomes flooded with proof work
Context management is more art than science, but proofs are unusually good at compactation. Once proved, you can just keep the lemmas, you know,like humans do.
>Sorry. See the edit at the top if you haven't already.
Sorry about what? Getting caught? You entire post is based on a nasty lie about Victor and his work.
I haven't seen a language launch this controversial on HN since V-lang in 2019.
Glad we finally have some new drama to follow, definitely more entertaining than AI news.
It's definitely AI related, though.
I've been thinking about this a lot lately, because I've noticed a trend in my own life that has happened countless times: I have a tendency to look at problems that others in different fields have found difficult to solve, and think I have a great idea that could change that field. And then I push to develop my solution to that problem, run into real world problems with my solution, refine and adapt, try a different solution, run in a loop until I settle on the fact that the people who are in those fields also have great ideas, they just are more aware of the constraints and that's the reason why the hard problems don't have easy solutions.
There is an absolutely enormous amount of hubris to being an engineer. I don't necessarily think its a bad thing...a certain amount of hubris is necessary for progress to be made. Our minds are creative and we can come up with amazing things, but something in there always makes us think we can do it better than the people who are stuck doing it daily. We fool ourselves into thinking they're too stuck in their mindset to have a more creative solution.
And the funny thing about LLMs is that while they can enable our competence, they enable our hubris even more.
I find this to be a mean and misguided post. To suggest that Victor Taelin does not know about formal methods. As I understand, he is trying to do something genuinely new and interesting. And he is transparent about his work, which he then gets hounded for. A shame.
God forbid someone criticise Victor Taelin!
Mudslinging isn’t appreciated, well thought out and articulated criticisms are.
It's Victor Taelin?! Ye gods!
Why is everybody treating this person like some celebrity. Am I weird for having never heard about this guy?
No.
It would be weird to stand in line for throwing shit at someone you don't know anything about, and believing that the first guy throwing shit has an actual reason to do so.
It turns out everyone in that line is wrong..
I don't know either :(
I'm not a celebrity and I don't like celebrities
Why is it mean? The author is above critique?
No, but the author seems to be above being critiqued for supposedly not even knowing of the existence of the field.
Considering that he is a somewhat well known expert in the field who has been doing full time research on it for a decade.
https://scholar.google.com/citations?user=ZLnQcAsAAAAJ&hl=en This is not the google scholar of a well-known expert, I'm sorry.
It’s like saying Linus Torvalds isn’t an expert on the Linux kernel
While that might not be the largest academic footprint, I don't think you could dispute him being an expert or being 'somewhat well known'.
He seems to have a big following on AI-hype twitter, I'll grant you that.
It is interesting however, how you weren't familiar with him at all [1] 20 minutes before describing him as a well-known expert.
[1] https://news.ycombinator.com/item?id=49753179#49754358
What's interesting about it?
I looked it up, and he seems to have worked on research in that area for a decade.
The author of the blog post could have checked the same before making the claims.
There are countless people with essentially no domain knowledge to speak of after working on the domain for a decade.
I have no idea how the original authors competency is, I know nothing about them.
There is however a strong correlation between people being influencers and kinda mediocre at what they're an influencer in - which is frankly understandable, because a large part of their skillset is usually more towards attracting an audience and making the topic interesting.
Usually the true domain experts rarely interface with the public beyond their professional capacity.
Has nothing to do with AI-hype. Anyone who is interested in Lambda calculus should find his work intriguing.
Someone can be a well-known expert in a tiny niche field, and one may be completely unaware of them, or even the whole field.
You are splitting hairs for no reason, while standing on the wrong side of this discussion.
Feel free to criticize the project on its actual content.
https://scholar.google.com/citations?view_op=search_authors&... Is this?
Or perhaps a Google Scholar means jack shit? Since when is that the official measure of an expert.
What has Victor Taelin made which is any way comparable to Linus Torvalds? I'm not saying it's impossible to be a well-known expert with a shitty google scholar. I'm just not seeing any evidence at all in Taelin's case. He doesn't even have a wikipedia page!
You have a very strange definition of expert. Most people do not have a Wikipedia page, and most experts do not have a Wikipedia page.
Muh academic ivory tower
I found the post offputting because he is insinuating a few things about the author which are clearly untrue (eg, unfamiliar with the formal verification landscape) and he's using this untrue speculation as evidence of the dangers of vibe coding. Even the edit where he says swap it out with a hypothetical person who fits the description - it's still leaving in the untrue claims about the author.
It doesn't seem unreasonable to think it mean if I were to make up speculative negative assumptions about you which undermined something you've been working hard on for a long period of time.
I know nothing about the author, I'm just watching this whole spectacle unfold.
It’s a poorly researched article just looking for cheapshots.
It is mean for the last point I mentioned: Taelin is very transpparent about his work which makes it easy to attack him. And what for? This post is not even a proper criticism of Bend 2, it is just trying to paint some story that Taelim is a vinecoder who does not know what he is doing, based on no evidence. If he was not transparent it would not even be known that LLMs have been used.
Because this shitty blogs' author just called someone "vibe-coding a language with no knowledge about the field" who is literally working in that very field for a decade, long predating LLMs, with numerous high-profile works.
You are on the wrong side of this whole discussion, there is only one people here who needs calling out and it's not Bend's author.
@LiamPowell the author is clearly aware of formal verification, they've written several implementations of dependently typed languages, and ... despite the presentation of their work, which has some obvious flaws (as can be judged by reception) ... their many comments indicate that they know what they are talking about.
Your post is setting up a strawman between automatic formal verification and formal verification using interactive theorem provers ... obviously there is a spectrum, and Ada/SPARK are navigating the space to try and automate much of the work required to automatically dispatch with obligations to prove (computable) properties about programs.
Bend2 is a QTT -- it's dependently typed, and comes from the lineage of systems which are focused on being expressive enough to formalize mathematics.
Of course you need to build a somewhat significant "standard library" of theorems, tactics (as metaprograms), etc ... to approach what is built into the compiler in Ada. These are different approaches with different trade offs.
Your post isn't clear, you don't go into any of these details ... why did you post this? Do you think this is clear writing?
> the author is clearly aware of formal verification, they've written several implementations of dependently typed languages
I'm not familiar with the author, I just saw the language posted the other day. I'll add a note to the top.
> These are different approaches with different trade offs.
Why would we want the tradeoff where the LLM has to write significantly more code and where the specification needs to be more complicated? If the author is aware of the state of the art then I think they made a poor choice, but that's not the point.
> Your post isn't clear, you don't go into any of these details
Bend just serves as a useful example, my general point is about how people will vibe-code a solution without an understanding of the field, leading to worse results than if they spent a little while understanding the field and then vibe-coded their thing.
Perhaps because their concerns are different? Similar reasoning: "why invest in dependently typed languages over SMT solvers? SMT solvers are automatic, and it is way easier to add annotations to my programs"
The answer is that these technologies are not universal in trade off space.
Of course, if you can have an LLM prove something by dispatch to an SMT solver ... you want to do that (and, indeed, LLMs do sometimes do this even when working on Lean proofs)
But the space of statements that you can prove using verification with SMT is smaller than dependently typed systems.
Also, Ada/SPARK are very good and robust -- they've been around for ~50 years ... but there's been a significant amount of work in formal verification since then, right? I would be more clear that Ada/SPARK are state-of-the-art for "mission critical software" (and have been for many years), but there's been a lot of research work in formal verification of low level programming since then (Low*, for instance)
I'm assuming you have not spent a lot of time trying to prove stuff about non trivial programs if the trade offs between SMT based approaches and interactive theorem provers are not clear to you.
SMT solvers are highly automated, but very opaque and hard to debug. Interactive theorem provers can scale to larger problems and harder properties at the cost of increased manual proof burden. The ever increasing ability of llms to write proofs in a theorem prover has in recent years tipped the balance significantly towards preferring itp based approaches over pure SMT ones. The current state of the art for program verification is probably a mix of llm and traditional (tactics, SMT, etc.) proof automation inside an itp.
ATPs go quite a bit beyond what a SMT solver can do, however you're not stuck with just using ATPs when they are supported. You can still allow for proofs to be manually written with an ATP doesn't work, and in fact SPARK allows for this with Rocq.
Most of what I have to prove is floating-point code where a manual proof is too much of a headache to ever attempt though.
If you're going to insinuate that the author of Bend2 doesn't understand PLs and formal verification, you should do so with some proof and not a hot take dunk.
I think it's fine to critique the language and the approach without criticizing the author and I hate that this site has become Tech Drama News, like the worst parts of Twitter.
Yes, I didn't realise how much it comes off as a critique of the author personally when I wrote it. I have added a note to that effect to the top of the article.
conversations on this site about formal methods are currently absolute cess pits of dunning kruger and confidently stated yet highly misinformed takes from those with close to little experience in the field.
I think sadly all big conversations on this site are like this these days. It didn't used to be like that sadly. Z3 and some early Lean threads used to be great.
You clearly don't know shit about the whole topic, yet attack an open-source project's author who wasn't even the one posting his own project..
> Why would we want the tradeoff where the LLM has to write significantly more code and where the specification needs to be more complicated? If the author is aware of the state of the art then I think they made a poor choice, but that's not the point.
Because code is trivially cheap now. LLMs churn out a shitton of code at pennies, and as technology improves their per-line cost will continue to decrease. A proof has to be written once and it's never read again. It's only important property is that it is machine-verifiable - from then on only the signature matters. Meanwhile speed of compilation matters, so an obvious tradeoff was made by the author.
> my general point is about how people will vibe-code a solution without an understanding of the field, leading to worse results than if they spent a little while understanding the field and then vibe-coded their thing.
This is clearly not the case here, and even if it were, it's still a better thing than vibe-blogging clickbait shit with extra amount of Dunning–Kruger. Calling out an author who clearly works in the field for a decade or more for not knowing the topic...
> To be fair to Bend, I completely vibe-coded this, I just told a LLM to recreate the demo in SPARK
A vibe-coded retort to a vibe-coding tool? Ugh.
what a horrible blog-post :( Bend, HVM, Interaction Combinators, etc. isn't some vibe-coded fantasy, and the author has been working on this stuff for a loooong time.
how incredibly disrespectful :/
The code itself is the most compact representation of the rules you want applied.
This is probably not necessarily true. “f: list[1 A] -> list[A] pure, worst-case time n log n, such that for all x and 0 <= i <= j < len(x), f(x)[i] <= f(x)[j]” is probably good enough for nearly everyone unless the program synthesiser is actively adversarial; probably 99.999% of the list-sorting in the world is done via standard library functions anyway, which suggests that people don’t much care exactly how it happens.
Good point. I would treat this as 'fully specified vs partially specified'. For a fully specified system, my mental model still maintains that the code is the most compact ruleset.
I agree that "don't care" is often the practical choice which corresponds to partially specified. In your sort example, both heap sort and merge sort satisfy the requirement. But they are not always interchangeable because each has a specific properties that you might care about (constant memory vs nLog(n) memory, easily parallelizable vs hard to parallelize and so on).
> “f: list[1 A] -> list[A] pure, worst-case time n log n, such that for all x and 0 <= i <= j < len(x), f(x)[i] <= f(x)[j]” is probably good enough for nearly everyone
Not good enough: `f(x) = []` or `f(x) = (if len(x) == 0 then [] else [x[0], x[0]]` are implementations that fulfill your specification and yet they don't always sort the input list correctly...
> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists
As a software developer we should fear chasing "better" solutions, that path always lead to procrastination, "kitchen sink" and probably not what users wants. "Good enough" should suffice in most cases. Sure if you are building a super mega critical software to land a plane or something like that, is different. But every day software, shouldn't be treated like "carved in stone" and stuff that should last a 1000 years. Code can be cheap now (calling it "vibe coding" doesn't help).
You can create or modify something quite fast now. Better to focus on testing, documentation, making sure that software will do what is expected.
> As a software developer we should fear chasing "better" solutions, that path always lead to procrastination
What a bad take. This is a discipline problem. You can chase both and be careful with the new solution.
Very off-topic, but the 'humans write “laws”' phrasing made me think of using LLMs to unit test real laws. Have them come up with test cases to see if the phrasing is as intended or has loopholes. Wonder if anyone's tried doing something like this (probably not, I imagine this is too much tech for government)
I guarantee you armies of lawyers have been doing exactly what you describe for years now with existing laws, and it's only increasing.
For new legislation, you may have a point in certain situations, but even the most clueless Congress critter has people working for them who get this stuff. Whether they choose to consult them or not is, of course, another matter entirely.
The vibe coding dunning krueger damage has yet to surface, but I'm guessing it will be in the billions. I'm not even talking about the insane infrastructure investments - thousands of c suite execs are vibe coding pointless crap instead of delegating it to their team that knows what they are doing, wasting thousands on tokens if they are on enterprise API plans for zero return or spending 30$ on creating an interactive html page for stuff that should be a power point slide with a few bullet points. It's totally insane!
> Feel free to replace “the author” below with “a hypothetical author who could have created the same thing”.
This is honestly sleazy, just admit you were basically way off.
It seems like this is largely a matter of what you’re asking for. If you wanted to do more research into the state of the field, an AI might be pretty good at answering your questions.
But it won't tell you if you don't ask. It won't tell you, "This approach is stupid, Ada SPARK exists".
Why would any LLM 'think' in terms of trying to cite prior work?
It itself is prior work. It's asking a fish to show where the water is. The fish can't imagine that absence, and the LLM can't imagine anything not being prior work.
Nowadays AI chat often will do web searches and include links. It will do that more if you ask.
Yes, telling it that you want parsimonious solutions that reuse existing libraries makes a big difference. Why would it even try to do that sort of stuff if you didn’t ask?
Because it should be smart. I mean, why would a human coder ever observe standard best practices unless the client specifically asks them to?
I wouldn't use SPARK either, and rather develop my own approach. The problem isn't that the proof has 400 lines of code, every modern system has large proofs (Isabelle/HOL, Lean, etc.) My latest formal proof has over 50K lines of proof. That's why AI is such a useful tool.
Pro: it can be an excellent way to learn if you realize good problems don't come easy, many such cases where I abandon something having learned from it and that's life
Con: the machine will tell you you have easily found a good problem, and engineered the perfect and necessary solution, if you let it
"Know what to ask for" is what will keep me with a job for a while longer, I guess.
"If you ask a LLM for a language where it’s possible to prove that a function is formally correct by building up a proof from basic principles then it will happily do so, it will never stop to suggest to you that computers can already build complex proofs without the need for a LLM and eliminate 99% of the work. It will never tell you that what you’re building already mostly exists as work that you can build on."
I don't know what llm you use but current llms will definitely let you know about similar things out there. So this statement is a bit incorrect.
All I see from this article and the whole saga is pure sadness. The situation seems unfortunate but seemingly unavoidable.
A language for LLMs will never be a compiled language. The best language for LLMs would be something that can be interacted with. Like a Lisp.
It absolutely drives me up the wall when I hear someone wrote their own language or framework because "the current ones just didn't do what I wanted to do" and then the result is a worse language/framework that the LLM and person know.
I absolutely endorse new creations when they are necessary but the people making these aren't doing it from a point of education, they are doing it purely because _they_ don't understand the framework or language that is the standard for that area.
It always always always involves a high level of AI Psychosis, that a brand new web framework is needed for your revolutionary... CRUD app?
If employer doesn't care about substantail 442 line fix, why should I?
As a resident of Bend, Oregon, checking the homepage sets off some kind of "oh look, Bend!" alarm in my brain.
"This example matters beyond Bend, vibe-coding makes it makes it far too easy to implement a design that’s horribly broken or decades behind the current state of the art because you can immediately get a result without ever having to do any research."
This is totally true but almost totally irrelevant. I'll use some hyperbole here to make the point: Whether the design is broken or decades behind doesn't matter anymore. Neither of those are an outcome/end goal. They are means we historically have used to achieve good end goals or outcomes.
In the end, the goal is usually "does it meet the needs of the person who needed it" not "is it good software". If it no longer meets their needs and they can vibe code another total piece of shit in an hour that meets their needs again, they still may be "better off" than spending time researching the field and learning and ...
This may feel shitty, and it may feel like it should not be true. But right now, that seems to be true?
In that sense, the author is wrong that vibe-coding is a trap. The trap is assuming you have to make something good to meet someone's needs both now, and in the future.
Now, like i said, this is hyperbole, and there are lots of good arguments against it. The author's just isn't one.
Technically good or better has never mattered very much in the marketplace, despite people wanting it to really badly (ease of use often mattered, but not technical goodness). Software engineers often took pride in their work and so there were usually kernels of goodness in even the shittiest software. All you are seeing is that now it is nowhere near as hard to create and bring these "solutions" to market, and more importantly, doesn't necessarily require anyone who has any pride in their work at all, or even have any experience in software engineering. As such, technical goodness has mostly gone out the window because the market never required or really rewarded it.
Good point, it's similar to music and other forms of art. The qualities that the people creating it care most about often have little to do with how well it is received.
> The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
This is precious.
The irony is that the author didn't even spend time to learn about the history of Bends author before writing this article.
This whole article can basically be summarized as "why is anybody building anything that already exists" gatekeeping.
Despite the fact that the entire premise is incorrect since the author of the language clearly has been shown to know about formal verification, this is basically encouraging nobody to ever post anything they work on for fear it might be "similar" to something already out there.
Is this really where we want the industry to go to all because of vibe coding?
The author of the article itself also clearly did 0 research of their own at all on the author of the language, and admits to vibe coding their own example themselves.
What the fuck are we doing.
The problem is that vibe coding makes it possible to build a substantial solution before learning enough about the problem to recognise that a much better solution exists.
We need a catchy name for this phenomenon.
I feel like this is the same black hole as small local models.
Things people want to be awesome and true, and things that are actually awesome and true don't intersect the way people want them to.
…so if there was an easy way to do provably correct AI code, it would be nice.
…but I’d also like a frontier that runs on my raspberry pi and a cheap fully autonomous self driving car that just uses a single cell phone camera.
Unfortunately wanting those doesn't make them exist; and people telling you they do exist usually are either a) uninformed, or b) selling something.
This tells way more about you than Taelin. You didn't take a minute to research about Taelin, his past work, his company, and the design decisions behind the language. FFS, why people are like this.
Yes I think fundamentally the effort required to understand, write, and verify a formal specification is just way higher than is reasonable in most situations.
There are some cases where it is pleasingly simple - usually low level algorithms like compression, sorting, search etc. Basically things you'd find in leetcode questions.
Most software isn't like that.
I think the actual answer is just that the very latest models (e.g. Astra) are actually quite good at writing normal tests, and you can just skim them to make sure they're doing something sane.
Narrowly, I think you're spot on. The effort required to understand the machinery around formal verification is a function of the surface area of the thing being formally verified. Specifically, formally verifying the surface area of general purpose programming languages is difficult.
My approach with Hale (shameless self plug) is that the programming language itself first offers another strata of structure to program within, a type of graph. Once the structure of the program is expressed as a graph, understanding how formal verification works is a clean encapsulation of graph activities.
Finally a bit of sense in this madness.
The obvious botted stars on GitHub in 2 hours was an immediate red flag.
These stars are from version 1 of Bend
>Bend just serves as a useful example of my general point regarding vibe-coding as it is recent, high-profile, and has aspects that make it easy to use as an example. I don’t know anything about the author’s history with designing languages or if they actually did consider the tradeoffs below and made what I think is a poor choice
This doesn't make any sense, you're equivocating vibecoded as in "prompted by a clueless incompetent person" with vibecoded as in "implemented by an LLM". Former is used to make your point, later is used to defend the premise.