If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Most automated analysis isn't dependent on just behavior, but rather suspicious things in the code itself. You have a popular open source package with files that exist on pypi but not github then that's a big flag, or if a similar package suddenly has some base64encoded garbage that runs through an obfuscated exec call. In other words the simple fact that the project has obfuscated code is enough to flag for further attention.
That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.
Cat and mouse like the sibling says ;) Like if you start changing system time, I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this. Of course I'll run the code that does this through "my obfuscator".
The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
Publishing publicly then applying cooldowns in projects is much easier tgan establishing a new standard for pre-release security testing versions that works across ecosystems and gains zooling support.
Releasing secure software is the right thing to do. We only release insecure software because we need the eyeballs to make it secure.
If we believe the point made above that the many eyeballs are not that important then releasing before we have done everything to make the software as secure as possible is irresponsible.
"We only release insecure software because we need the eyeballs to make it secure." No one is doing this on purpose, they believe the software to be secure when they release it... You shouldn't gate the release of a package until every single offsec research shop in the world has had a chance to look at it
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
When you're running from a bear on a hiking trip you just have to be faster than your friend. So just set your cooldown slightly longer than everyone else's cooldowns. The cooldown will give security researchers some time to scan the packages so it's still good.
I agree, it’s just the wrong approach. As a user, there’s no way to know if a package has been audited during the cooldown by some generous cybersecurity firm before you pull it in, it’s just wishful thinking. Minimizing your dependencies is a more effective strategy against supply chain attacks.
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
I don't think that HNers understand the recent supply chain attacks very well at all. I also don't think they realize the tests the SCA/package providers do to all the major packages.
Almost all these attacks try to reach out to external sites to steal your data. That is exceptionally hard to hide in any meaningful way.
You can make it much less suspicious. In particular, if you can compromise the package publishing process, and not just pushes to main, you can add your malicious code to binary artifacts, not to the source code.
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
They are literally solving the same problem, it’s just that distro packages operate on a lower level and thus receive more scrutiny. There have been plenty of examples of poisoned Linux packages, both at the source level and at the package level.
A distro package manager provides access to a small set of packages that the distro thinks you might like. A language package manager provides access to the full set of packages. The language level package manager is solving a more ambitious problem.
Distro package managers don't solve the problem they just punt on it, saying "you can add an unofficial source but you're on your own to maintain security". I agree with GP that the comparison is tiresome.
The difference comes from the fact distro maintainers choose the software they package, while language maintainers largely don't.
For example, how do you prevent somebody from phishing/typosquatting users with a package named similarly to a popular one? For distro maintainers the answer is simple - don't package it. Debian is unlikely to add a "f1refox" package. Language maintainers don't have that luxury.
Oh what?? Most package managers do try to prevent typo squatting!! Your insistence that there’s something fundamentally different between different package managers is just weird! Debian, as npm, lets you package and distribute software. The differences between them are related to policy preferences but the basic idea is the same!
I think that’s a difference in implementation details, not a difference in what it fundamentally is.
I’m not arguing that there aren’t differences between the two, I’m arguing that they are fundamentally the same solution (gather all of the software in one location with) to same problem (how can I safely download some software).
They are very fundamentally different solutions: the entire premise of distribution packaging is that you’re relying on some distribution maintainer’s discretion. It’s not an open index.
I agree it's tiresome that language package managers continue to be an embarrassment.
But actually your comparison is incorrect, distros are nothing like app stores. App stores are full of the same junk and malware as language package managers. Distros are vetted collections of real software that use modern best practices to secure the software supply chain.
Being unable to rotate the same weak RSA key for over a decade because it was too operationally cumbersome[1] does not strike me as “modern best practices.” I also don’t think any Linux distribution package maintainer would make a legally binding claim to have “vetted” packages: the filter is reputational, not a strong guarantee that the package is not malicious. The xz incident demonstrates that tidily: the only reason it was caught it because it was noisy; the package itself was presumed to be safe.
(This is, to be clear, fine. But we should be clear-headed about what package distribution mechanisms actually provide in terms of guarantees.)
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
> I’m convinced that updating so much more often is worse, not better.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
For most things in life that hit us periodically, we try to buffer them into manageable amounts so that when we address them, we do them all at once, benefiting from “economy of scale”, instead of having to constantly work on addressing everything that comes our way immediately. With package updates, however, another consideration is security: the longer you wait to release a security update, the more users may be vulnerable to it.
There’s no right answer. Every case is different, trying to impose a single rule for everyone just simplifies things beyond what is reasonable.
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
In reality, one massive update is too big to digest and it never happens. So you're stuck on out-of-date packages having lost the ability to update.
That's never a problem until it suddenly is. Company is put at significant risk (courts want to reason by analogy and "engineers skipped maintenance and endangered people" is an easy one) and nobody is to blame since nobody owned the task.
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
> The reality is that each update is its own potential security issue
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
As a previous sysadmin that does stuff in tangently related things to security (not the security person that makes these rules) I agree. The rules they come out with to address issues in operating systems that haven't been deployed in 10 years blows my mind.
"Ya, Windows something ancient had an issue with WevDAV 2 decades ago, but that is not a reason to block the http DELETE verb at the WAF"
I keep getting things like "denial of service when the SCSS or whatever parse is given malicious input”. Great, that's part of the build chain, all of its inputs are stuff we control. Why do we care.
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
> Dependabot now waits until a new release has been available on its registry for at least three days before opening a version update pull request
So you exploit on Tuesday 12pm, dependabot opens a PR on Friday 12pm, people merge it, and your trojan's timer is set to go off over the weekend when nobody is patching.
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
This seems to be primarily an issue with a few specific package management solutions that have suffered SCA vulnerabilities recently, not generaly across the board.
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
I don't disagree with you that freely distributed software on conveniently licensed terms is going to be the go-to stance for the majority of solo and non-commercial developers.
I just believe I could arrange the universe such that I get to have my cake (commercial licensing) and eat it too (with default open source licensing).
It is my experience that corporations do pay handsomely for software they use, even SaaS ones as the cost of doing business. Open source communities need mechanisms for funding that are consistent and low friction.
This is why the software language/repository/platform itself needs to facilitate license tracking and billing for alternative commercial licenses, to make it easy for corporations.
A successful new language effort that provides this facility need not be an enforcer except to say if an enterprise is willing and signed up to pay for any dependencies it uses, it is obligated to pay for all of them with something like AGPL 3 as the poison pill they have to swallow otherwise if they distribute or serve from any copyleft software.
Having simple, consistent rules that vendors and consumers have to follow with no rugpulls will be important for market acceptance. Having voluntary compliance with license terms will also be important to not turn people off from the ecosystem and to let them kick the tires. If software vendors want to distribute only unencumbered free and open source, then god bless 'em, they should be able to do it.
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Most automated analysis isn't dependent on just behavior, but rather suspicious things in the code itself. You have a popular open source package with files that exist on pypi but not github then that's a big flag, or if a similar package suddenly has some base64encoded garbage that runs through an obfuscated exec call. In other words the simple fact that the project has obfuscated code is enough to flag for further attention.
That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.
Cat and mouse like the sibling says ;) Like if you start changing system time, I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this. Of course I'll run the code that does this through "my obfuscator".
The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.
ok, but your package still contains obfuscated code that we can read
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
You can simulate accelerated time in the sandbox used to evaluate the runtime behavior of the package being analyzed.
This would increase the malware surface area and make it much easier for automated security scanners to detect. So win-win for everyone.
If it is about a few highly specialized firms finding the vulns we should let them do it before publication and we do not need cooldowns.
If it is not about that and we still subscribe to Linus's law then cooldowns will just postpone the problem.
Publishing publicly then applying cooldowns in projects is much easier tgan establishing a new standard for pre-release security testing versions that works across ecosystems and gains zooling support.
Releasing secure software is the right thing to do. We only release insecure software because we need the eyeballs to make it secure.
If we believe the point made above that the many eyeballs are not that important then releasing before we have done everything to make the software as secure as possible is irresponsible.
"We only release insecure software because we need the eyeballs to make it secure." No one is doing this on purpose, they believe the software to be secure when they release it... You shouldn't gate the release of a package until every single offsec research shop in the world has had a chance to look at it
Exactly this. The firms can follow the standard practices of the industry without needing to injection further process into the projects.
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
Do you have an example of those things you're alleging?
All package malware related news I see are related to users being affected by it (then security firms do their analysis whatever) ...
Analysis and detection are not the same.
If you currently use a 7 day cooldown on a 0 day default. You can just use a 10 day cooldown on a 3 day default. But don't tell anyone...
Only a few of the recent supply chain attacks were discovered by users noticing weird behavior.
The majority were noticed by maintainers or third party groups noticing things like releases not tied to a source tag, many rapid releases, etc.
Cooldowns won’t stop everything, but it makes a malicious release significantly more likely to be noticed
When you're running from a bear on a hiking trip you just have to be faster than your friend. So just set your cooldown slightly longer than everyone else's cooldowns. The cooldown will give security researchers some time to scan the packages so it's still good.
The goal is to give time for automated scanners ran by cybersecurity companies to flag malware before it gets installed on real users.
I agree, it’s just the wrong approach. As a user, there’s no way to know if a package has been audited during the cooldown by some generous cybersecurity firm before you pull it in, it’s just wishful thinking. Minimizing your dependencies is a more effective strategy against supply chain attacks.
Hence by writing your own code with its own set of vulns to be detected.
Well, there’s always woodworking.
Easy, then you just delay your project’s dependency updates just a little more than everyone else
The cooldowns should probably be randomised.
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
There are plenty of ways to notice a malicious release without observing it running.
Build provenance, maintainer alerts on new releases, tying releases to specific git tags, etc all help.
Every single one now will be more sneaky, and we’ll be operating on a 3-day cooldown for no reason.
How exactly does that work?
I don't think that HNers understand the recent supply chain attacks very well at all. I also don't think they realize the tests the SCA/package providers do to all the major packages.
Almost all these attacks try to reach out to external sites to steal your data. That is exceptionally hard to hide in any meaningful way.
OK, so it looks like you've still added suspicious code to your package.
You can make it much less suspicious. In particular, if you can compromise the package publishing process, and not just pushes to main, you can add your malicious code to binary artifacts, not to the source code.
You really think it has zero benefit whatsoever? Nothing malicious will be caught?
Pretty much. These tools are effective now only because the malware doesn’t have to avoid being detected at all to be successful.
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
There was a story about two men and a tiger
The men see the tiger, one scrambles to run and the other starts putting on their shoes
"Why are you putting on shoes? You'll never outrun the tiger"
"I don't need to, I just need to outrun you"
Watching language package managers reinvent everything distribution package managers have been doing since the 90s has been as fun as watching crypto people reinvent financial regulation.
The publishing topology is pretty fundamentally different: the entire power (and danger) of language package managers is that anybody can publish, not just a privileged few.
(This cuts both ways: I’d say that distribution package managers have learned valuable lessons about what users actually want from language package managers. Learning is a good thing.)
This comparison is tiresome. Distro package managers are curated, language package managers are not. They're serving completely different use cases; the former is the App Store, the latter is the web.
Give it two years.
They are literally solving the same problem, it’s just that distro packages operate on a lower level and thus receive more scrutiny. There have been plenty of examples of poisoned Linux packages, both at the source level and at the package level.
A distro package manager provides access to a small set of packages that the distro thinks you might like. A language package manager provides access to the full set of packages. The language level package manager is solving a more ambitious problem.
Distro package managers don't solve the problem they just punt on it, saying "you can add an unofficial source but you're on your own to maintain security". I agree with GP that the comparison is tiresome.
Debian stable has over 69,000 packages, which is more then some languages have…
69,000 sounds like a lot, but it's still missing maybe 30% of the packages I want to install.
It’s not about quantity but target audience and use case. They solve different problems.
I don’t understand how they aren’t both generalized to “distributes software with versioning at a large scale”
The difference comes from the fact distro maintainers choose the software they package, while language maintainers largely don't.
For example, how do you prevent somebody from phishing/typosquatting users with a package named similarly to a popular one? For distro maintainers the answer is simple - don't package it. Debian is unlikely to add a "f1refox" package. Language maintainers don't have that luxury.
Oh what?? Most package managers do try to prevent typo squatting!! Your insistence that there’s something fundamentally different between different package managers is just weird! Debian, as npm, lets you package and distribute software. The differences between them are related to policy preferences but the basic idea is the same!
I think that’s a difference in implementation details, not a difference in what it fundamentally is.
I’m not arguing that there aren’t differences between the two, I’m arguing that they are fundamentally the same solution (gather all of the software in one location with) to same problem (how can I safely download some software).
They are very fundamentally different solutions: the entire premise of distribution packaging is that you’re relying on some distribution maintainer’s discretion. It’s not an open index.
Because they both have very different management processes (centralized vs distributed) and release cycles. They solve different problems.
> They are literally solving the same problem
No they’re not. Distro packages cater to end users and have very different release cycles and maintenance processes.
Distro packages are managed top-down (pushed by maintainers), while language packages are managed bottom-up (pushed by authors), so to say.
I agree it's tiresome that language package managers continue to be an embarrassment.
But actually your comparison is incorrect, distros are nothing like app stores. App stores are full of the same junk and malware as language package managers. Distros are vetted collections of real software that use modern best practices to secure the software supply chain.
Being unable to rotate the same weak RSA key for over a decade because it was too operationally cumbersome[1] does not strike me as “modern best practices.” I also don’t think any Linux distribution package maintainer would make a legally binding claim to have “vetted” packages: the filter is reputational, not a strong guarantee that the package is not malicious. The xz incident demonstrates that tidily: the only reason it was caught it because it was noisy; the package itself was presumed to be safe.
(This is, to be clear, fine. But we should be clear-headed about what package distribution mechanisms actually provide in terms of guarantees.)
[1]: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1461834
I really hate dependabot making generic security people at work so pushy about updates updates updates. They seem to just be dogmatic about whatever dependabot says, forcing churn even when the documented issues are clearly not relevant. I’m not sure how to handle it politically. I’m convinced that updating so much more often is worse, not better.
> I’m convinced that updating so much more often is worse, not better.
The issue of cooldowns aside (which is about delaying updates, not reducing their frequency): you're going to have the same set of problems when you update, whether you do it frequently or infrequently. The difference is that if you update frequently, you'll have a smaller set of updates (so it's easier to debug) and you'll have more opportunity to report issues upstream and fix them in a timely fashion.
It's the same underlying problem as CI and build time. Most people abandoned the concept of projects that take so long to build you can only do testing once a week, because CI that runs on every PR provides a much better experience. This is the same lesson applied to updates.
For most things in life that hit us periodically, we try to buffer them into manageable amounts so that when we address them, we do them all at once, benefiting from “economy of scale”, instead of having to constantly work on addressing everything that comes our way immediately. With package updates, however, another consideration is security: the longer you wait to release a security update, the more users may be vulnerable to it.
There’s no right answer. Every case is different, trying to impose a single rule for everyone just simplifies things beyond what is reasonable.
My logical read of the situation is that I end up making fewer overall changes if I end up upgrading a dependency once, not thrice, to a specific version. And the changes are their own source of risk.
In reality, one massive update is too big to digest and it never happens. So you're stuck on out-of-date packages having lost the ability to update.
That's never a problem until it suddenly is. Company is put at significant risk (courts want to reason by analogy and "engineers skipped maintenance and endangered people" is an easy one) and nobody is to blame since nobody owned the task.
I don't know if all ecosystems are as bad as node is, but the node ecosystem has terrible issue severity ranking which makes infosec squeamish for no reason.
Every week or so there's a new High+ "vulnerability" that gets published against our dependencies and I have to go look at it to confirm that it's yet another case of "it's possible for someone to give this dev-only tool a bad regex that would cause the test runner to OOM on that branch".
I'm in a similar camp, I dislike how often third-party package updates get pushed out, especially given the lack of serious inspection.
The reality is that each update is its own potential security issue and with supply chain attacks being all too frequent, it's not a panacea.
> The reality is that each update is its own potential security issue
Even beyond security issues: each update is a new opportunity for breakage, not only from bugs in the third-party package, but also from unexpected dependencies on the third-party package's behavior.
As a sysadmin I'm in the same boat. I've unfortunately never worked with security folks that seemed to have any sysadmin or dev experience. Whether or not this is universal, idk, and I have no idea what they are teaching in these security courses. But I'm tired of security teams telling me "you need to implement these 230 group policies this quarter" or whatever. They don't seem to grasp the externalities of a request like that and how much fucking work it is to vet, test, deploy, monitor, verify, etc. 80% of the time, they don't even know what they do or if it's even impactful for us.
As a previous sysadmin that does stuff in tangently related things to security (not the security person that makes these rules) I agree. The rules they come out with to address issues in operating systems that haven't been deployed in 10 years blows my mind.
"Ya, Windows something ancient had an issue with WevDAV 2 decades ago, but that is not a reason to block the http DELETE verb at the WAF"
I feel the same. So so so many "regular expression denial-of-service" issues at my last job that just didn't seem serious or often reachable.
I keep getting things like "denial of service when the SCSS or whatever parse is given malicious input”. Great, that's part of the build chain, all of its inputs are stuff we control. Why do we care.
All of my services are internal, yet I am continually getting red alerts that there is a novel denial-of-service that needs to be patched immediately.
I’ve mainly handled it by pushing my team to be extremely conservative about what dependencies we take, especially if they pull in scads and scads of transitive dependencies.
This elegantly mitigates three problems in one go: update churn, dependency hell, and supply chain attack surface.
It also, frankly, tends to make the code easier to understand. I’m not a huge NIH person but I do have to say that a lot of packages these days tend to encourage ways of doing things that are unnecessarily complex. More than once I’ve replaced a dependency with homegrown code and reduced LOC in the same commit.
> Dependabot now waits until a new release has been available on its registry for at least three days before opening a version update pull request
So you exploit on Tuesday 12pm, dependabot opens a PR on Friday 12pm, people merge it, and your trojan's timer is set to go off over the weekend when nobody is patching.
But updates to broken packages are still allowed: if a new version is pushed within the three days, it does not reset the cool-down. You just get a pull request to update to a known-bad version instead.
This makes me think whether npm (and other registries) should apply security requirements based on ecosystem impact. Example a package having millions of downloads can have special security measures enforced.
What would be a security measure that should only be selectively enforced?
There are few ideas which come to my mind, some might be far fetched, taking NPM as an example.
- Restricting packages with similar names as of popular packages restrict expres because express is a popular package.
- Imposing stricter 2FA checks on accounts of authors of these packages.
- Making sure that published packages don't have vulnerabilities and clear npm audit.
- Alerts in case these packages contain a dependency which is new / relatively new.
Pre-publish to official security orgs. Does not get released into the wild until k of N auditors agree.
Higher cost (“Mythos” vs static code analysis) vulnerability scanning prior to successful merge to main branch or deployment as an artifact. As risk increases (popular code->greater exposure potential), increase automated, programmatic scrutiny on subject code to lower residual risk.
(application security and vulnerability management is a component of my work in financial services, thoughts and opinions always my own)
This seems to be primarily an issue with a few specific package management solutions that have suffered SCA vulnerabilities recently, not generaly across the board.
It’s foolish to feel safe because your package management solution hasn’t been attacked yet.
The attack vector is generalized.
"We don't call 'em 0days any more, now we call 'em 3days"
More like 3-per-day amirite
> The default applies only to version updates. Security updates still open immediately, so critical fixes are never delayed.
does this require a real vulnerability report, or CVE? if the package is compromised would they just be able to push a false "critical update" that bypasses this wait?
Requires a GitHub security advisory and
> Only advisories reviewed by GitHub trigger alerts.
From https://docs.github.com/en/code-security/concepts/supply-cha...
So it forces everyone to use more GitHub stuff?
Maybe I'm misunderstanding, but this means I now need to submit to GitHub Security Advisor to get my security fix out ASAP?
The idiocy of cooldowns speaks for itself.
What a state of things where we have to fear installing software, and rely on vendors to scan things ahead of time, because our supply chain is such a mess and our tooling is so incapable of (and uninterested in) protecting us.
You cannot call it a supply chain, if you have zero contractual relationships with the authors of the solutions you are using.
[1] https://news.ycombinator.com/item?id=44434355
Oh that one really makes you think doesn’t it.
I mean, that’s just arguing over whether or not the definition of “supply” implies “compensation”, which isn’t very interesting imho.
The grandparent’s point remains the same, the software ecosystem and its supply chain or however you want to call it is a hot mess.
Traditionally the term "supply chain" has implied a buyer/seller relationship
I think that’s up to debate, and my point is that debating whether free software counts as “supply” or not is really not that interesting.
So what?
What would a solution to this look like?
What would it take to not fear installing software? This isn't a npm problem, its a computing problem in general. Spaces like this are generally pretty against any sort of restrictions or limitations being put on computers under the name of safety (see Manifest v3)
For libraries, I like the Gnu Affero Public License[1]. If you run the library in software with that license, you have to publish all the source of the entire project that incorporates it.
No corporation could tolerate this, though, so the library vendor can negotiate a commercial license of their software for appropriate fees.
That said, corporations are not going to want to negotiate fees with 100's of vendors over constantly fluctuating dependencies in their software.
This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
Similar to Amazon's Dynamo API, whatever the next big language/ecosystem is needs to be designed around _billing_ and automatic license management for # of deployments, seats, call volumes, etc.
[1] https://web.archive.org/web/20260712154038/https://www.gnu.o...
> This is why the next big language/software ecosystem needs to integrate payments to vendors in their repository system. That way, commercial license management can occur between the ecosystem owners and the corporate customers and all the vendors get paid their fair share.
I don't think this idea is going to go anywhere.
If a package is available for free, on convenient licensing terms, developers will use it.
If you make them pay, many developers will prefer to just build it themselves. Coding agents make that easier than ever.
Buying a package involves a lot more paperwork – it needs to go through procurement – and introduces new risks, e.g. what if the vendor increases their prices
There are potential exceptions – software with really advanced algorithms (e.g. solvers for optimisation problems); safety critical software; software needing regulatory certification (e.g. there are some Australian government APIs they won't let you call unless you've hired an auditor to certify the software you are calling them with, and the relevant government agency has approved the auditor's report) – but those exceptions are relatively rare, and the existing solutions are arguably adequate to handle them
I also think it is different for packaged SaaS applications [0] because there the buyer isn't a developer, it is someone non-technical, and "use a coding agent to build it yourself" isn't within their comfort zone or risk appetite (at least, not yet).
[0] conflict of interest disclaimer: work for a SaaS vendor
I don't disagree with you that freely distributed software on conveniently licensed terms is going to be the go-to stance for the majority of solo and non-commercial developers.
I just believe I could arrange the universe such that I get to have my cake (commercial licensing) and eat it too (with default open source licensing).
It is my experience that corporations do pay handsomely for software they use, even SaaS ones as the cost of doing business. Open source communities need mechanisms for funding that are consistent and low friction.
This is why the software language/repository/platform itself needs to facilitate license tracking and billing for alternative commercial licenses, to make it easy for corporations.
A successful new language effort that provides this facility need not be an enforcer except to say if an enterprise is willing and signed up to pay for any dependencies it uses, it is obligated to pay for all of them with something like AGPL 3 as the poison pill they have to swallow otherwise if they distribute or serve from any copyleft software.
Having simple, consistent rules that vendors and consumers have to follow with no rugpulls will be important for market acceptance. Having voluntary compliance with license terms will also be important to not turn people off from the ecosystem and to let them kick the tires. If software vendors want to distribute only unencumbered free and open source, then god bless 'em, they should be able to do it.
Manifest v3's actual motive was so shamelessly transparent that most of us just don't allow the "safety" argument for it to really be entertained. I don't have a suspension of disbelief rich enough to pretend I don't know.
> What would a solution to this look like?
Sandboxing and auditing built into the software from the start. Browser Extensions solved this ages ago.
No way to prevent this says only package manager where this regularly happens.
Every major package manager is just as fucked.