Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
> products with a real userbase that couldn't exist or be scaled pre-AI?
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
> I'm pretty sure we would find an overall increase in actually useful apps
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
>Why do you assume "we not seen an improvements in products"? Who is "we"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
> The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Depends on which targets and how granular the caching is.
- if you touch package.json that might invalidate everything
- touch core and you'll invalidate everything
- touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Agentic Coding has been a huge strain on CI,
I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
> where sub agent orchestration is done through agent to agent messaging
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?
Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
Here's my constant question:
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
Last year people were asking "if AI is so great then where are the new apps?". Then data for 2026 came out and now the IOS app store has a 84% percent year-over-year increase in new app submissions.
For the question where are the alternative OSes? Here is one that I've seen. There's probably more - https://www.reddit.com/r/ClaudeAI/comments/1wfpydl/i_asked_c...
For that other stuff you mentioned like the right click menu. Those huge corporate projects suffer more from layers of institutional dysfunction and will be very very slow to show any improvement. Their dysfunction can't be solved with just faster coding.
Using AI to build more features is easier than using AI to improve existing projects. People will gradually figure out how to do latter too, it'll just take longer.
That's just volume though. I also know, and have no trouble believing, that github is going down partly due to the weight of all the vibecoded pushes. But that does not necessarily translate to people's needs and wants being covered, for all we know iOS just has a plague of unused POCs.
Do you have actual productive examples? As in, products with a real userbase that couldn't exist or be scaled pre-AI? Genuinely asking, I might have missed some large hits. The closest I can remember was bun rewrite kerfuffle, which seemed more a marketing action than anything.
> That's just volume though
If we sorted through all those new apps and ignored all the crap, I'm pretty sure we would find an overall increase in actually useful apps. I just shipped a new app myself, and I think it's useful, and I wouldn't have finished mine without AI assistance.
> products with a real userbase that couldn't exist or be scaled pre-AI?
Coding hasn't been the bottleneck for product creation for a while. So I don't think there are going to be many examples that fit that exact criteria. Any app idea was 'possible' before, it just took more developer hours to do it. So now teams are doing more things that were weren't worth the effort before, but are a lot more feasible with AI assistance.
> I'm pretty sure we would find an overall increase in actually useful apps
I think you're probably right but at what point does it become diminishing returns? I have observed the app market to be overly saturated for years and rarely download something other than a mobile-banking update.
What you're saying wouldn't be hard to prove, you're acting as if we aren't aware of usage statistics. We can even tell if MAU (monthly active users) are increasing or not for particular apps. So we are aware of them and what do they actually say?
The follow on question is if it's making us all so much more productive, where is the increased revenue? As far as I can tell, it's mostly the AI labs seeing that, not everyone using them (modulo small founders building new things and doing okay, I think)
Isn’t cloudflare a pretty clear answer to these? They are launching more products than ever, some of them clearly vibe coded, and their revenue is exploding.
The increased revenue is in startups, like the recent couple YC batches.
Do you have a source for that? Are they raising more money or receiving more money for AI-oriented products or are they actually making more money on consumer/B2B end products?
AI doesn’t make a responsive right-click menu a higher business priority. But we are seeing a ton of small custom projects that are as easy to dismiss as they are to abandon.
Surely it makes the backlog clear faster to the point where you reach the non-priority stuff? But I'll bite, what business priorities are being created at 100x?
The same reason it took many years for corperate america to get a real productivity increase from computers and the internet, all the old ways of doing things had to be redone. I think the largest companies are least equipped to take advantage of AI productivity gains. Agile no longer makes sense, Org charts as no longer make sense, etc...
In my experience everyone is just rebuilding the same wheel over and over.
A lot of people may be more empowered to create things now with less up front effort but it doesn’t lead to having better ideas or more actual system architects.
All of the work work at my company has been internal facing. People are rightly leery of exposing things made by an LLM to the public.
Why do you assume "we not seen an improvements in products"? Who is "we"? Have you done an exhaustive (or even half-assed) analysis or is this just your "vibe"?
>Why do you assume "we not seen an improvements in products"? Who is "we"?
Have you? As in, can you name products you use daily that have been transformed night and day?
I am not talking about the existence of minor improvement, mind you. I am denying the existence of anything that is minimally consistent with the level of discourse that surrounds development itself (throwing away practices, checks and balances, hitting constant roadblocks due to the new speed, etc).
A true 100x requires no analysis. If we could suddenly build and improve houses at 100x I would know because I would be typing this from my 5th living room.
Well you mention Android, iPhone, Nintendo, and Microsoft. Those are the biggest companies, and many of them don't even use the best coding tools. So they are not going to be the ones improving.
On the other hand, look at the YC companies. I think the last batch or two is growing revenue faster than any batches ever before. Those are the companies that are being sped up a huge amount by AI.
We 100% have. I've seen significant improvements in many of the services I use.
Not doubting you, care to share what those improvements were?
Drop a couple names, I'd like to know who is leveraging AI well
In my case it's not the CI that's the bottleneck. It's the human testing side. Does it work, sure. But does it actually do the thing we want (and more importantly) does it do it in a way our customers will understand and actually like?
I feel no one talks about this, and yet it is glaringly obvious. Are we even solving the right problem? No one cares. Push code. Number go up.
I feel like if anything LLMs have reduced coding/software to "throw as much as possible at the wall and see what sticks". It feels quite shortsighted and wasteful, especially considering that humanity needs to get better about how it produces and consumes energy. It's sort of bleak.
hasn't improved software either. it's made the overall quality worse if anything.
to me the last part if the most important part. product management hasn't been automated at all, and is more important than ever. you can't just keep adding features to make a great product
Think of it as a layered problem. If the bottom layer (CI) cannot keep up with the output of agents, then solving problems at a higher layer - like user experience checks - will be exponentially slower and less reliable. Kind of like how optimizing tight inner loops makes your whole program faster.
It's true because it usually makes no sense to QA test a build where the test suite has not yet passed.
Uh, no. If stage X is the bottleneck, it's immaterial how much you speed up stage Y.
You're assuming QA reviews won't be fully automated, and triggered from a CI pipeline.
We can assume lots of things, but at some point the rubber must hit the road.
What exactly are you using to back up this claim?
> QA reviews [...] fully automated
Clearly you have to replace obsolete human testers with agentic AI testers, duh.
At some point, with all this velocity, human users become the bottleneck, unable to keep up with and learn all the changes and new features. Luckily, there's a simple solution: just replace the human users with agentic AI users.
Maybe we can also replace the customers with agentic consumers.
> The forceful executive, Henry Ford II, and the leader of the automobile workers union, Walter Reuther, both saw many examples of advanced machinery operating at the plant. The words they exchanged brilliantly encapsulated the paradox of automation:
> Henry Ford II: Walter, how are you going to get those robots to pay your union dues?
> Walter Reuther: Henry, how are you going to get them to buy your cars?
https://quoteinvestigator.com/2011/11/16/robots-buy-cars/
I think you're being sarcastic, but there is actual truth behind what you're saying.
Because every developer is now a slop cannon by default, by default users will experience churn and whiplash, and things will break all over the place. As you point out, this is bad. It's also impossible to fix without deploying agents on the QA side. Like it or hate it, agentic testing is inevitable to protect users from the churn and noise caused by the slop cannon. I don't think that replaces test engineers at all - if anything it makes the job more fun. If you've ever had to keep playwright tests in sync with the target manually, and kept the CI environment up to speed with toolchain changes, you know what I mean.
Whether the "slop cannon by default" situation could have been avoided in the first place, is another question... But we're here now and there's no going back. Might as well deal with it as best as we can.
TLDR: it's not all bad :)
> Moving our workloads off GitHub Actions to third-party runners with faster CPUs, higher-performance storage, and better cache infrastructure gave us faster machines to run the same pipeline on
Yeah, was not surprised to read this. Actions is convenient if you already use GitHub, but it can also be pretty slow. Given reliability is also a major issue with GitHub these days I expect to see more orgs moving to different pipelines
I wonder how much faster GitHub actions could be if they weren’t running on Azure. Because Azure is either slow or very expensive.
I think its actually because GitHub isn't solely running on Azure. They are currently using a mix of on prem, AWS, and Azure. The Azure migration has been a challenge with Azure running into capacity issues due to AI load.
We switched our Actions workload to blacksmith.sh (not affiliated) and have been pretty happy with how fast and inexpensive they are. I wouldn't be surprised to see this trend continue.
Good thing Linear has been finished for a long time doesn't need more features, so AI coding can go slow. Oh damn, it's busy becoming the next Jira :(
I’m conflicted about Linear’s progression. I dislike some of the features but on the whole they’ve managed to keep the software pleasant to use, it doesn’t feel to me that it is drifting towards Jira territory, rather, it feels like it is losing the carefully considered product design because now code is cheap to generate. I’m not worried about it turning into Jira but it has lost its soul. Still a great product.
I haven't used Linear in any serious capacity, but I feel like there's plenty of market in "Jira that doesn't feel like it hates its users" and "Jira but we care about performance".
Not saying that they're shooting for either of those segments, but someone should. Plenty of enterprise orgs that need (or are convinced they need) Jira's featureset.
All software becomes popular because it's not the old behemoth, but it's new, fast, and simple. All software then grows and grows until it becomes the new behemoth, and the circle of life begins anew.
If you can pay the setup cost, Bazel will get you build times ~10s with a warm cache for even massive projects.
I lead a bazel conversion for a pretty complex piece of software written in 5+ programming languages and shipping native binaries to all 3 major OSes a few years ago, and it took multiple years to get it done.
For a less complex project (1 programming language, still shipping to all 3 major OSes), with my knowledge and agents I got the bazel conversion done in 2 weeks.
The setup cost for bazel just went down by a lot, and I don't think the industry as a whole is aware of that yet.
Were any of those using JS/TS? I've seen Bazel perform wonders with many compiled languages, but I'm not impressed with the JS ecosystem.
Nothing substantial, no. I've never personally experienced builds to be so slow to begin investigating bazel as an option for JS/TS.
Tsgo, oxlint, caching dependencies etc. what linear outlined in their blog post would be more impactful for the average TS project I've worked on.
The entire industry, including its outputs that LLMs are trained on, hasn’t reconsidered what’s easy vs. hard or fast vs. slow. LLMs consistently recommend against code changes because they will take “a weekend”. No, Claude. You will do the work and it will take 20 minutes.
Bazel seems to have a lot of tradeoffs, from setup time of the sandbox for each task, to ergonomics that lead folks to maintain parallel 'normal' tooling.
Plus, 'with a warm cache' is doing heavy lifting, what's the real cache hit rate for a week of development? Investing in improving the cold build and frequent actions is still important with bazel or any incremental builder.
I'm not sure it's useful to talk about bazel broadly, it's actual performance and behavior comes down to the rules you use. You can configure bazel like turbo/nx and cache tsc/vitest/eslint on each package.json module, and get course cached units that are evicted on every change, or you can use gazelle and target per-file actions which are only invalidated when their dependencies change. But that trades off batching unless you use workers.
Most PRs only touch a handful of targets so cache hits are extremely high in practice.
Depends on which targets and how granular the caching is. - if you touch package.json that might invalidate everything - touch core and you'll invalidate everything - touch one file in api and you may run all api tests (see gazelle)
I ran an experiment where I migrated a package to bazel then replayed a weeks worth of changes and it saved 20%. That's nothing to scoff at, but not the headline numbers you see after a full hot build.
Can't comment on Bazel specifically, but having worked with both nx and turbo, the bottleneck was usually network and disk IOPS rarely compute.
Even fully cached outputs needs to fetched and read from a remote server[1]. A step n-1 outout fetched from remote cache server need to written to disk and then again read by step n[3] - all disk I/O and network bound operations.
10s may be achievable/realistic goal in the Java/C++ world where Bazel normally seen. In TS eco-system most people would be over the moon to get into ballpark of 1-2m for a decently large monorepo.
We should define Build more clearly here, if you mean running just transpile/compile steps or the full series of steps that includes tests (as the linear post here is talking about). It is hard to see even a small sub-set of a large suite of test that require a virtual DOM or a real browser can run in 10s or less.
[1] Typical for say managed CI setup .
[3] Common run-of-the-mill frontend + backend stacks in different languages etc.
If you don't need to rebuild anything, bazel can fetch only the final artifact (not the intermediates) from the remote cache.
Also, if you have persistent CI workers with a persistent bazel instance, you save on some network roundtrips, but that's obviously harder to set up and make bulletproof.
Typically you need the intermediates to compute if you need the next one , so you cannot skip to final until you have the intermediaries.
The final asset/artifact is rarely small either. even best optimized artifacts can be few hundred MB docker image or more commonly multiple image layers running GBs in size .
each step is a network pull then recompute cache if stale and keep going till end .
Bazel can even skip the final artefact!
well for test runs you kinda need the binary to run, but you're correct, if the job is just "does this build" no download necessary.
My take: it seems like systems should become smaller, more isolated, and contract-oriented.
I have been a long time proponent of monoliths, but it seems like agents would be happier with smaller, more isolated services. The more isolated, the better. Contracts between the service components only. Then it can iterate internally as long as it satisfies the contract. If it needs to, it can version the contract and keep iterating.
Agentic Coding has been a huge strain on CI, I have been using Bazel to improve our build times and ultimately building customized runners to improve our CI
Anyways great blog post from linear team a lot to learn from it
This still surprises me.
Mostly around, at least the majority of 'stuff' I've worked on (both before and after the rise of coding agents) myself or others took enough time to make sure that anything done locally, if you run the tests locally, you're at least 90% of the way there as far as what CI/CD does.
I suppose the flipside being, most of those projects had less churn (i.e. one person was working on a service at a time, and we had good contracts between services.) Also, Our local boxes were way better than our CI boxes, so there was incentive to run locally versus waiting 2-10x the time for CI to run...
I call this moment "the Great CI Bottleneck of 2026". Here are my thoughts on it: https://dagger.io/blog/the-great-ci-bottleneck-of-2026/
I wonder if the GitHub actions outages we keep seeing is due to themselves making self-hosted runners paid, hence bringing broke/cheap users back from hosted runners to their garbage infra.Meanwhile I'm running my Codeberg Actions on the free Oracle ARM machine 2 cores 12GB ram (previously 4 cores 24GB) and way more reliable. Keep winning bozos.
Edit: After second thought, I guess "alternative runner" providers still have to pay the self-hosted tax. So M$ actually saves and makes money by not scaling their infra and driving people to alternative providers they can tax freely. Actual geniuses.
They abandoned (or at least postponed) the idea of charging for self-hosted runners usage.
Actions management plane is the most problematic part of the platform (in regards to scale), so they are basically losing money with self-hosted runners.
Scaling of VMs shouldn't be a problem, since that is practically stateless, whereas there are lots of operations performed for every job/workflow.
Speed up CI and the next bottleneck just moves to deploy and rollback, which do not scale the same way.
(Disclosure: I'm a cofounder of RWX) For anybody wanting to solve similar problems and considering Bazel, take a look RWX. It's built around the same concepts of content-based caching and graph-based task execution, but it's far more runtime agnostic and easier to adopt. https://rwx.com
vitest load balancing on test duration times instead of by file would be a nice performance win
We actually tried this and (for us) it doesn't lead to enough of a gain that's worth the extra complexity.
I just feel apps like linear are increasingly getting in the way of full send agentic development where sub agent orchestration is done through agent to agent messaging, work trees, on demand git restructuring and epoch specific coordination plains, often .md files. The smaller the human component of total product development gets, the more this may be the case.
Question here:
How do you expect to see the history/record of what the agents did and why? Is it enough to see it in PRs? Do you expect tickets that have the design and history? How are you thinking of agents being able to historically resolve reasoning/why/decisions made in earlier passes?Genuine open question here. My assumption is that a GH or Linear or Jira is still useful as a decision store. It may as well be a custom app over Postgres, but it seems like something is needed to store this and for observability. A GH/Linear/Jira is nice if only because of standard APIs and integration points (whatever you build would likely end up duplicating a subset of those).
Linear is a key part of my prompting technique. I write specifications in Linear, and the history I generate in it becomes a critical source of context for my agents. Prompting my agents has become “look at XYZ-124 in Linear and ask me any questions you have.”
My recent winnowwallet.com still has build times of over 10 minutes. I'm convinced the agent really wants build times around 5m to move at a quick pace. Also to not drive me insane. It took a ton of work to get it down from 45 minutes because my tests launch a full version of the app, and walk it through major usecases while recording video and screenshots. I then use AI to qa this. It also generates its webpage this way. All from CI/CD
I cannot believe we are in 2026 and CI/CD hasn't evolved enough to even consider hot updates. Burn your CI/CD pipelines to the ground and start over again without any of the slop. Don't let anyone who calls themselves DevOps Engineers design it again, only people who are System Administrators.
Maybe I'm alone in this, but as someone who is in tech, I don't know what CI is, and I don't think it's unreasonable to expect it to have an expansion within the article the first time you use it...
AI is in the cultural zeitgeist, but you gotta expand most other things at least once.
So if you don't know what CI is, what does "I am someone in tech", mean? Marques Brownlee probably doesn't know CI is, but he's also in tech. Because if you by mean "I am in tech" that "I am a software engineer", and you don't know what CI is, then boy; I'd be worried for you.
CI is really only significant on the programming side of things. Networking, systems administration, hardware, etc often don’t have that workflow.
I see you have not discovered infrastructure-as-code (IaC). We have CI for our infra.
sysadmins CI is hoping the machine still boots after an update. Unless you are lucky you got two identical machines to have a test env.
Also the case of 'learning that someone did updates directly on AWS console instead of terraform and losing 0.5 or more days cleaning up the resulting mess'
I am a Site Reliability Engineer, I have been for close to a decade, and this is not as common an abbreviation as you think it is. Also, my entire job isn't in question because I didn't memorize shorthand for a term that I don't use in my day-to-day.
Really appreciate getting downvoted to hell for saying "We can't throw readers a simple bone?"
All I'm asking for is CI(Continuous Integration) in the first paragraph.
What do you think about this? https://claude.ai/share/54eef6b8-b702-4f3e-968c-b19184e951f4
I didn't say I don't know what Continuous Integration was. I said I didn't know what "CI" stood for.
Cannot stress enough, this is like, basic communication/writing that they teach in elementary school.
But I'm so happy you're able to get off on your feelings of superiority that you always automatically know what those two letters stand for. I hope that translates to meaningful happiness and fulfillment in your life.
"AI is in the cultural zeitgeist"
Within the engineering tech sphere, CI/CD have also been terms that have been standard for at least a decade now.
You would probably be rejected from most interviews at the first stage if you didn't vaguely know what they mean at this point.
Linears entire product is tailored towards software engineers/engineers in general or people who work alongside engineers, so its not surprise their posts have a bit of assumed knowledge.
I know what a CD (Compact Disk) is.
(Continuous Integration, and Continuous Deployment, are fine if someone tells me what the letters literally stand for, but sorry for not being so tech-brained that I haven't memorized every possible TLA that's ever been used in tech).
CI -> Continuous Integration
Its best thought of as the testing systems that are run as part of pull request review / merge to main / build processes.
Thank you
Man, what a weird thing to complain about.
Most days I feel like I must be the dumbest person on HN. I don't understand what 80%+ of submissions are about. But if it sounds interesting, I'll dig into it a bit and learn a few things along the way.
I recommend making a habit of googling or asking AI about terms you run into. As someone who is in tech, you will hear tech jargon your whole career that will not be explained if it's considered standard terminology for the audience being spoken to. It is better to learn to educate yourself than expect others to go out of their way.
It's wild to me that people think asking for
"Continuous Integration (CI)"
in the first line of a paragraph of a long article is "going out of your way". It used to be the standard expectation for using shorthand.