Some conjecture on my part, but I imagine that the bugs being found are not the kind you can just press a button on the LLM to fix anyway, but are instead the sort of bugs which require the reviewer to deeply consider design tradeoffs, of which I believe there are simply more in a DB or programming language codebase, for example. Stuff I am finding in review is often like "this thing isn't like the other thing, which has this behavior, which is wrong if the aim is X, but sort of correct if the aim is Y, and totally perfect is the goal is Z and the use case is J, but not I. Unless it's Q, in which case you probably need to reconsider some fundamental decision making from 12 months ago and have some conversations with people about how users desire it to work in the first place, and then set about understanding the tradeoffs inherent to that design."
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
I like the way you put this. I agree this is where the frontier of software development is right now. You can have LLM chug and bug fix all day long but in the end most real issues are in this layer of intent. I don't think this problem is particular to DB/programming language codebase either - it happens a lot in product design it just doesn't surface in code review. "Everything is working perfectly" yet the resulting workflow or UI is not truly making the user's life easier, or has grown grotesque with additions/modifications.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
This is why code is going to be automated, and the software engineer is a link between organizational intent and the code. This is what I’m working on.
The difference is that instead of spending half your time understanding the business requirements and half your time writing code, you'll spend half your time understanding the business requirements, 5% of your time typing these requirements into a chatbot, and the rest of your time doing... ????
Some of the rest of the time can be spent making it look like your ratios are the same but in reality you just take the saved time and do whatever you want with it. The decision making and reviews aren’t happening any faster anyway so just take it easy and coast a bit
I guarantee there will not actually be saved time - the bullshit will increase to more than compensate, as everyone actually fixing things will be drowned out by bullshitters just pretending to fix things and creating more actual problems.
And imagine you had an LLM ingest the entire corpus of comments and discussion threads for all of Postgres' history, and asked it to make a judgment call on the tradeoffs - or even on how to present the tradeoffs for discussion.
How much of its judgment would be driven by the data vs. its own internal knowledge? How much might be driven by old vs. new discussions that might not represent new directions of the project? How much might be driven by over-matching against how bugs in less mission-critical code paths were handled?
There's no way to quantify the answers to these questions. Which is not to say this applies to every problem out there! But human judgment is still a bottleneck for design questions on widely, disparately-used complex systems.
You nailed it! I often got bug reports from AI that would touch far more code to fix than what I am willing to touch, and the likelihood of that large piece of new code (which I honestly would not be able to review thoroughly) will likely contain more bugs than the one I am fixing.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
I remember an LWN article on this question a few weeks ago. The debate was going on and they weren’t sure if it was that there were more bugs or just that because of AI they suddenly knew about more bugs than usual.
Either way, unfortunate but a very understandable decision.
I'm inclined to think the latter - all sorts of software, particularly long-standing open source Unix software written in C, is riddled with bugs that sometimes don't get discovered until years later. Most ordinary Unix system software like Postgres is written in such a way that it's not possible to be confident that the code is free of subtle bugs, and we should assume that it is not free of subtle bugs.
QA is becoming the blocker for projects in an AI era, and the real source of value. AIs can lay down a good base of testing but I see even frontier models still need help with testing because they are basically as bad as humans in terms of testing only what comes to mind, if not even a touch worse. Though they are really good at blasting through problems once they are found.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
> Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
I don’t think I’ve worked at a place that wasn’t trying to automate away QA. Perhaps Delivery Hero pre-acquisition, since we had 2,5 QAs for 4 devs in my team.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
Not to mention the blast radius of any small mistake in payment processing, tax filing, or any other financial or regulatory issue, is just so massive compared to making some ai slop bar special flyer. Joe's bait shack is still just going to pay for square.
There will be a ton of smaller Joe’s bait shacks, all competing against each other in a race to the bottom - and with relatively limited blast radius per shop.
Not surprising. I've been running AI reviews through old codebases and finding all sorts of interesting things. AI is significantly raising the quality bar and there is a lot of backfilling to do.
I have seen people discussing the fact that AI can literally find an infinite number of bugs. There are always more bugs to be found, whether they would have ever affected any actual real world case or been noticed or not.
> whether they would have ever affected any actual real world case or been noticed or not.
That's what I was thinking as well. Wouldn't the majority of the bugs not also have existed in Postgres 18? The version that everyone is using atm...
There is a similar thing with AI detecting vulnerabilities. It's not that the vulnerability isn't real, but just that in most cases for these vulnerabilities to exist there also needs to be a very specific set of circumstances as well making the likelihood of the vulnerability to exist in the wild quite low.
Doesn't that just imply the existing version of Postgres also probably has more bugs than humans can fix? Just release it. Software is never perfect and always a work in progress. I'm guessing the bulk of these bugs are probably stupid edge cases that just get development bogged down trying to plug so many holes at once.
Indeed, but I for one very much appreciate Postgres’ stability and quality.
As a lead contributor explains in the article, if they were to ship some of these broken features, they wouldn’t be able to fix them until v20, rather than patching them on-the-go. Sounds like a Very Bad Idea given many installations stay behind for some years.
I have a unique perspective on this as I've been working on rewriting and rearchitecting Postgres to solve a lot of the long standing issues Postgres has[0][1].
If I had to root cause why we're finding so many bugs now, I would attribute it to the limited Postgres test suite. AI is merely uncovering bugs that already existed. For the most part, Postgres relies on individual regression tests to test each feature. While regression tests are great for making sure features don't break (yes, SELECT 'Hello, World' LIKE '%Hello%' still returns true), they actually aren't that great hitting the edge cases (passing a string that's over 2MB to a specific function will trigger a buffer overflow and crash your DB). The main way Postgres has prevented bugs in the past has been by minimizing risky changes and having lots of human review. As we're learning now, there were lots of gaps that were not caught by the existing process.
While the Postgres team is skeptical of AI, I've had lots of success in using AI to scale more advanced forms of testing such as Deterministic Simulation Testing, differential coverage guided fuzzing, and formal verification. To put it in perspective, in the process of doing testing for pgrust, I've uncovered ~50 different bugs in core Postgres. The bugs I'm finding aren't through AI review, but are instead through using AI to do more advanced testing than regression testing. All of the techniques I'm using existed existed pre-AI, but aren't used in meaningful capacity as part of the core Postgres development.
Right now I'm going all in on formal verification. I wrote a compiler to translate C and Rust functions to Lean and am then having an LLM write a proof that the two functions are identical. So far I've been able to verify 10.5% of the codebases behave identically (10.5% = 67k/640k LOC of Postgres). I'm finding details as minute as a parsing function where Postgres trimmed spaces using isspace() and pgrust only trimmed ascii whitespaces. These aren't cases you would find with traditional regression testing or human review. The thoroughness of the types of bugs I'm finding is giving me confidence that that this is catching more bugs than any sort of manual testing could. I don't currently see any reason that I won't be able to scale up the formal verification to cover 100% of the codebase. The only issue is it takes a long time to do.
> What does "rebuild" mean? Translation from C to Rust?
We translated from C to Rust, rearchitected several components and are working on rearchitecting more. For instance, we replaced the process-per-connection model with a thread-per-connection model and added a columnar format + new query executor that's hundreds of times faster than Postgres for OLAP workloads.
Seeing how Fable handled Linux kernel bugs, where out of 82 bugs reported only 6-8 were actually bugs (not already fixed or fixed in newest version or straight up made up) I'm skeptical till humans actually sift through this pile.
I tried that this summer. I encountered this doom loop where the review agent would find bugs, the coding agent would fix them in some overengineered way that introduced new bugs, the review agent found them, the coding agent made more while fixing them, repeat until insanity.
That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Same experience. It's especially bad with Claude and anything with non-code text, where it will happily write up endless Claude-ese comments that break any rules in the repo and are constantly incorrect in small ways, catch them in the review, rewrite specific lines in ways that are still wrong despite still having the review in its context, and repeat twenty times in a row.
The coding loop and the review loop are orthogonal and probably should remain so. Not least because it's probably worth the money for more intelligent models for code reviews where it's a more focused task while coding loops are all over the place.
Weirdly enough I've found the opposite, using too smart a review model gets you overfit instead of benefit. The coding side can be more complex in a way that benefits from fable/astra, but the review side sometimes not so much.
Training data is filled with shortcuts, conveniences, bugs, hacks, hacks that fix other hacks, hot fixes, patches, mappings that map the hack to hack to a better solution and all types of other surprises.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
From the article: "He was also at pains to point out that the reverts were not down to 'vibe coding' in development, as the technique was not available when most of the new features were written."
I wonder when people will finally feel comfortable not putting vibe coding in scare quotes. Maybe when it accounts for 50% of all new software, instead of only 10%.
Why read the article when you can read the headline and make up your own?
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
Good! We need to delay all software releases until security audits of the code by AI/fuzzers/etc. Otherwise people ship buggy software, the hackers find the exploits, and we're compromised as soon as we upgrade. This is why we absolutely must have un-nerfed security models. And if you want to stop having your personal information leaked by compromised companies, we need to make this process mandatory.
This is the happy path for LLM assisted coding. Obviously consumes less tokens being more constrained and focused so it's not what gets the most marketing but having the second loop of code reviews seems worth investment even if the coding loop is entirely manual or you're in full LLM autopilot mode
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
They have been doing Postgres for a long time, and I believe they have been significant contributors from before AI coding.
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.
Some conjecture on my part, but I imagine that the bugs being found are not the kind you can just press a button on the LLM to fix anyway, but are instead the sort of bugs which require the reviewer to deeply consider design tradeoffs, of which I believe there are simply more in a DB or programming language codebase, for example. Stuff I am finding in review is often like "this thing isn't like the other thing, which has this behavior, which is wrong if the aim is X, but sort of correct if the aim is Y, and totally perfect is the goal is Z and the use case is J, but not I. Unless it's Q, in which case you probably need to reconsider some fundamental decision making from 12 months ago and have some conversations with people about how users desire it to work in the first place, and then set about understanding the tradeoffs inherent to that design."
ETA: This is extremely generous to the LLMs, btw. If I got an LLM to pop something this concise out I would fully commit to finding a different profession tomorrow, but for now I am for better or worse stuck doing prompting some non-zero number of times in order to get all of this assembled context, and then need to use my human powers to decide whether or not more humans indeed need to discuss it.
I like the way you put this. I agree this is where the frontier of software development is right now. You can have LLM chug and bug fix all day long but in the end most real issues are in this layer of intent. I don't think this problem is particular to DB/programming language codebase either - it happens a lot in product design it just doesn't surface in code review. "Everything is working perfectly" yet the resulting workflow or UI is not truly making the user's life easier, or has grown grotesque with additions/modifications.
I will say, if the problem is purely code you occasionally see "sparks of AGI" where the model will step back and say "hold on, there is a massive simplification we can make here...". This started happening with Fable-class models. But rarely if ever will the model push for a huge simplification at the product level.
Yes > real issues are in this layer of intent
This is why code is going to be automated, and the software engineer is a link between organizational intent and the code. This is what I’m working on.
How does this differ from what I'm doing today? Want to say any more on the topic? This is a bit vague.
The difference is that instead of spending half your time understanding the business requirements and half your time writing code, you'll spend half your time understanding the business requirements, 5% of your time typing these requirements into a chatbot, and the rest of your time doing... ????
Some of the rest of the time can be spent making it look like your ratios are the same but in reality you just take the saved time and do whatever you want with it. The decision making and reviews aren’t happening any faster anyway so just take it easy and coast a bit
I guarantee there will not actually be saved time - the bullshit will increase to more than compensate, as everyone actually fixing things will be drowned out by bullshitters just pretending to fix things and creating more actual problems.
And imagine you had an LLM ingest the entire corpus of comments and discussion threads for all of Postgres' history, and asked it to make a judgment call on the tradeoffs - or even on how to present the tradeoffs for discussion.
How much of its judgment would be driven by the data vs. its own internal knowledge? How much might be driven by old vs. new discussions that might not represent new directions of the project? How much might be driven by over-matching against how bugs in less mission-critical code paths were handled?
There's no way to quantify the answers to these questions. Which is not to say this applies to every problem out there! But human judgment is still a bottleneck for design questions on widely, disparately-used complex systems.
You nailed it! I often got bug reports from AI that would touch far more code to fix than what I am willing to touch, and the likelihood of that large piece of new code (which I honestly would not be able to review thoroughly) will likely contain more bugs than the one I am fixing.
So in the end I often ignore these bug reports after evaluating their actual impact on the product.
This is one angle of the productivity story. You don't have any sort of plausible deniability covering all these bugs, test cases etc. when they are comprehensively discoverable by tooling. It's like having typing bugs on a strongly typed language with static analysis, you just can't make a lot of excuses.
So that's one measure of extra time that will be spent on every single feature, where in the past things would've been rolled out in a more barebones state. It's better overall, but is it going to be perceived as being more productive?
If its more "bugs than humans can fix", its also likely "more bugs than humans can triage and verify as legitimate bugs".
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
More bugs than humans can fix, but not more bugs than humans created.
More hypothetical bugs, perhaps?
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.
"What even 'IS' a bug really?" HN topics from 2036
I remember an LWN article on this question a few weeks ago. The debate was going on and they weren’t sure if it was that there were more bugs or just that because of AI they suddenly knew about more bugs than usual.
Either way, unfortunate but a very understandable decision.
I'm inclined to think the latter - all sorts of software, particularly long-standing open source Unix software written in C, is riddled with bugs that sometimes don't get discovered until years later. Most ordinary Unix system software like Postgres is written in such a way that it's not possible to be confident that the code is free of subtle bugs, and we should assume that it is not free of subtle bugs.
Yeah, I don't understand how this can even be a question when we're finding an unprecedented number of bugs that are decades old.
I think the concern was the bugs in the new features specifically but still I suspect you’re right it’s just that AI finding them “early”.
https://lwn.net/Articles/1092003/
Determinable for a code base, with enough tokens.
QA is becoming the blocker for projects in an AI era, and the real source of value. AIs can lay down a good base of testing but I see even frontier models still need help with testing because they are basically as bad as humans in terms of testing only what comes to mind, if not even a touch worse. Though they are really good at blasting through problems once they are found.
Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services may be happy to instantiate some front office system for themselves via AI prompt, but they aren't going to want to QA that post-AI any more than they want to now. Once we all settle down, SaaS will manage more code than before, and do more QA, and Joe Bob's etc. will be even less interested in trying to replicate the work with AI because it will represent an even larger QA load than it does today, relative to the size of their business.
Or, to put it in the context of this article, even if you can prompt a database into being today, which you can, you can't prompt something into existence that has been tested and QA'd like Postgres, and you would be well advised to ponder that gap between your prompt and billions of computer hours of field testing very carefully before you decide where to store your company's most important data.
> Ironically, despite the predictions that AI will murder SaaS, I think the result is they end up even more entrenched, because Job Bob's Bait Shack And Tax Filing Services
OK, so this is slightly different since it's more about how runtime behavior than design-time behavior, but... I'm in a SaaS workplace, and I've been trying to seed the following idea:
1. The very cheapest/fastest end of the market will be eaten up by Job Bob's Bait Shack stuff. Cheap wrappers around a prompt that goes "Pretend you're a computer that obeys the following rules" jank. A race to the bottom we can't win.
2. Most of our (desirable) customers will (eventually) realize they can't tolerate systems which casually or unexpectedly violate known business rules. That leads to angry consumers, junk data, broken workflows, and lawsuits.
3. When our product can capture and enforce those rules with confidence, that's both an important differentiator and also a kind of moat/stickiness.
4. Therefore there is a strategic business justification for Engineering to fence-off where we use LLMs in production. When we use them for design, it's to create better deterministic systems than before.
I don’t think I’ve worked at a place that wasn’t trying to automate away QA. Perhaps Delivery Hero pre-acquisition, since we had 2,5 QAs for 4 devs in my team.
So… are we gonna use the extra productivity (and money) from coding agents to hire QA people? Oh boy, is that highly unlikely.
But I like your take. I wish it were so, perhaps we wouldn’t have had Tahoe.
Not to mention the blast radius of any small mistake in payment processing, tax filing, or any other financial or regulatory issue, is just so massive compared to making some ai slop bar special flyer. Joe's bait shack is still just going to pay for square.
There will be a ton of smaller Joe’s bait shacks, all competing against each other in a race to the bottom - and with relatively limited blast radius per shop.
Not surprising. I've been running AI reviews through old codebases and finding all sorts of interesting things. AI is significantly raising the quality bar and there is a lot of backfilling to do.
I have seen people discussing the fact that AI can literally find an infinite number of bugs. There are always more bugs to be found, whether they would have ever affected any actual real world case or been noticed or not.
> There are always more bugs to be found
A common misunderstanding.
The # of bugs lurking in a codebase is finite. At some point, the only way to find more is introduce fresh bugs through new code.
What we're seeing now is a transition phase where pre-existing bugs are found (and hopefully fixed) at an increased rate.
> whether they would have ever affected any actual real world case or been noticed or not.
That's what I was thinking as well. Wouldn't the majority of the bugs not also have existed in Postgres 18? The version that everyone is using atm...
There is a similar thing with AI detecting vulnerabilities. It's not that the vulnerability isn't real, but just that in most cases for these vulnerabilities to exist there also needs to be a very specific set of circumstances as well making the likelihood of the vulnerability to exist in the wild quite low.
Doesn't that just imply the existing version of Postgres also probably has more bugs than humans can fix? Just release it. Software is never perfect and always a work in progress. I'm guessing the bulk of these bugs are probably stupid edge cases that just get development bogged down trying to plug so many holes at once.
Indeed, but I for one very much appreciate Postgres’ stability and quality.
As a lead contributor explains in the article, if they were to ship some of these broken features, they wouldn’t be able to fix them until v20, rather than patching them on-the-go. Sounds like a Very Bad Idea given many installations stay behind for some years.
I have a unique perspective on this as I've been working on rewriting and rearchitecting Postgres to solve a lot of the long standing issues Postgres has[0][1].
If I had to root cause why we're finding so many bugs now, I would attribute it to the limited Postgres test suite. AI is merely uncovering bugs that already existed. For the most part, Postgres relies on individual regression tests to test each feature. While regression tests are great for making sure features don't break (yes, SELECT 'Hello, World' LIKE '%Hello%' still returns true), they actually aren't that great hitting the edge cases (passing a string that's over 2MB to a specific function will trigger a buffer overflow and crash your DB). The main way Postgres has prevented bugs in the past has been by minimizing risky changes and having lots of human review. As we're learning now, there were lots of gaps that were not caught by the existing process.
While the Postgres team is skeptical of AI, I've had lots of success in using AI to scale more advanced forms of testing such as Deterministic Simulation Testing, differential coverage guided fuzzing, and formal verification. To put it in perspective, in the process of doing testing for pgrust, I've uncovered ~50 different bugs in core Postgres. The bugs I'm finding aren't through AI review, but are instead through using AI to do more advanced testing than regression testing. All of the techniques I'm using existed existed pre-AI, but aren't used in meaningful capacity as part of the core Postgres development.
Right now I'm going all in on formal verification. I wrote a compiler to translate C and Rust functions to Lean and am then having an LLM write a proof that the two functions are identical. So far I've been able to verify 10.5% of the codebases behave identically (10.5% = 67k/640k LOC of Postgres). I'm finding details as minute as a parsing function where Postgres trimmed spaces using isspace() and pgrust only trimmed ascii whitespaces. These aren't cases you would find with traditional regression testing or human review. The thoroughness of the types of bugs I'm finding is giving me confidence that that this is catching more bugs than any sort of manual testing could. I don't currently see any reason that I won't be able to scale up the formal verification to cover 100% of the codebase. The only issue is it takes a long time to do.
[0] https://github.com/malisper/pgrust
[1] https://malisper.me/the-four-horsemen-behind-thousands-of-po...
What does "rebuild" mean? Translation from C to Rust?
You look at Postgres C functions and prove they are identical, but the license is AGPL without mentioning Postgres authors. Consider me confused.
> What does "rebuild" mean? Translation from C to Rust?
We translated from C to Rust, rearchitected several components and are working on rearchitecting more. For instance, we replaced the process-per-connection model with a thread-per-connection model and added a columnar format + new query executor that's hundreds of times faster than Postgres for OLAP workloads.
> without mentioning Postgres authors
We have a NOTICE file which references the Postgres license here: https://github.com/malisper/pgrust/blob/main/NOTICE
Seeing how Fable handled Linux kernel bugs, where out of 82 bugs reported only 6-8 were actually bugs (not already fixed or fixed in newest version or straight up made up) I'm skeptical till humans actually sift through this pile.
I guess with LLM as the reviewer, it is going to take a lot longer to be Zero Defect, unless they use LLM as the coder, too.
I tried that this summer. I encountered this doom loop where the review agent would find bugs, the coding agent would fix them in some overengineered way that introduced new bugs, the review agent found them, the coding agent made more while fixing them, repeat until insanity.
Not every time. But often enough.
FWIW gpt-6.1-sol and opus 5.5 are much, much less prone to this, if you include some coaching around complexity, YAGNI, that kind of thing
Agents understand red/green/refactor. Tell them to follow it, and use a coverage tool.
That is how you end up with a test suite that now takes 10x as long, unless you occasionally prompt the agent to consolidate and group tests.
There are still a lot of issues with coding that require intervention. Just recently, I had Astra detect a minor bug in an HTTP client that resulted in it deciding that the best approach would be to spend 1800 LOC on implementing its own from scratch. Some other time, Opus 5.5 was trying to tell me that test runs on a small to mid-sized application would take another three weeks until they would be fully finished.
The only time I really get tests which take crazy time is when I'm doing mobile dev, because emulator testing on my system is slow.
Y'know, I had a teammate who had been claiming this up until it became clear that the horrific spike in production defects that we were struggling with was mostly attributable to them.
I'm not going to say that's always what happens. But I will say that you can't find problems you aren't watching for.
> But I will say that you can't find problems you aren't watching for
I can't, but Claude almost regularly comes across "BTW things". I catch them because I read the lead agent's transcript, and have worker agents always add a retro section to their report, so I'm almost always seeing attention-worthy things pop up.
And I’m definitely not saying don’t do that.
But think of Claude as being like a particularly keen outsourced developer: great at the tactical stuff, but sometimes lacks common sense in weird and surprising ways, so you’ve still got to keep an eye on the work product.
And then you discover it’s added tests for insane things that shouldn’t be getting tests
That's not my experience. Agent usually follows the pattern correctly by creating a relevant test and then making it pass. Where strange tests are more likely to fall in is when agent writes the test after the testee.
Then you remove them and move on with your life? We're still software developers at the end of the day, my delete key works better than ever.
Also maybe I'm a curmudgeon but I generally at least mark-one-eyeball all the tests it's generating and tell it when it's being extra dumb.
Same experience. It's especially bad with Claude and anything with non-code text, where it will happily write up endless Claude-ese comments that break any rules in the repo and are constantly incorrect in small ways, catch them in the review, rewrite specific lines in ways that are still wrong despite still having the review in its context, and repeat twenty times in a row.
The coding loop and the review loop are orthogonal and probably should remain so. Not least because it's probably worth the money for more intelligent models for code reviews where it's a more focused task while coding loops are all over the place.
Weirdly enough I've found the opposite, using too smart a review model gets you overfit instead of benefit. The coding side can be more complex in a way that benefits from fable/astra, but the review side sometimes not so much.
Training data is filled with shortcuts, conveniences, bugs, hacks, hacks that fix other hacks, hot fixes, patches, mappings that map the hack to hack to a better solution and all types of other surprises.
That's the cost of trying to scale so quickly. You dont filter out the crap.
A nice side effect is that the best way to fix it is more AI.
It's a endless money loop.
From the article: "He was also at pains to point out that the reverts were not down to 'vibe coding' in development, as the technique was not available when most of the new features were written."
I wonder when people will finally feel comfortable not putting vibe coding in scare quotes. Maybe when it accounts for 50% of all new software, instead of only 10%.
Journalists are in general slower to do this kind of thing than technologists.
The article mentioned the exact opposite.
Why read the article when you can read the headline and make up your own?
I think these type of new techniques will ultimately create more jobs for humans. If we think the answer is just “AI” everyone will create their own Postgres with a lot more bugs and vulnerabilities.
Using AI to find bugs (rather than write them) is a great use case IMO. Even if you have skilled reviewers, more eyes are almost always better.
Just don’t post the unedited AI “bug report” as an issue please. A human should make sure it’s actually a real problem before dumping it on devs.
Good! We need to delay all software releases until security audits of the code by AI/fuzzers/etc. Otherwise people ship buggy software, the hackers find the exploits, and we're compromised as soon as we upgrade. This is why we absolutely must have un-nerfed security models. And if you want to stop having your personal information leaked by compromised companies, we need to make this process mandatory.
This is the happy path for LLM assisted coding. Obviously consumes less tokens being more constrained and focused so it's not what gets the most marketing but having the second loop of code reviews seems worth investment even if the coding loop is entirely manual or you're in full LLM autopilot mode
From the article:
"Explaining the number of reverts — features whose rollouts have now been postponed to next year's release, PostgreSQL 20 — Peter Eisentraut, veep at EDB, a PostgreSQL support and consultancy, said the reason could be at least in part attributed to the LLM code reviews used by developers."
Naturally, EDB is an AI shop (https://www.enterprisedb.com):
"Agents need direct, governed access to data at the source. EDB Postgres AI is built for that."
Of course, nothing ever has worked in PostgreSQL before AI!
They have been doing Postgres for a long time, and I believe they have been significant contributors from before AI coding.
Their marketing has to mention AI. Everyone has to in a business like that. I would not take it as evidence they are slop coders or hyping AI just on that basis.
If by "they" you mean Eisentraut, I know who he is. That is precisely why I'm disappointed.