Hypothesis: a big accelerant of these rapid repository compromise (from Red hat to GitHub to Amazon to small startups) might be GitHub+dependabot automatic dependency updates.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
Yep, this is the thesis behind them. I wish people engaged more fully with this argument: it’s possible to believe that security vendors won’t do a good job of upholding their side of the bargain, but I’ve yet to see anybody argue that rather than making a faulty universalization argument against cooldowns.
If this is the idea, why don't we let the dedicated security firms and/or automated scanners find the vulnerabilities before the release?
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
The idea that a package can be updated and with a deploy at the right time could be live on your servers in prod 10 minutes later has always been crazy, and the last years have just reinforced that.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
Yea, all the new advice around using dependency cooldowns only works if _someone_ is installing these things before you and finding the vulnerabilities.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
Using dependency cooldowns is not a free-rider problem. There's a real tradeoff here – ppl are trading their time preference for security.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
This is how a chunk of people function anyway. There are plenty of people that choose to not install "point zero" release for software of a certain importance, assuming with any major changes there are often bugs that come along with it.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
not really, there are a number of security companies doing analysis of any new packages looking for supply chain attacks, so if you wait a couple of days, till their analysis is complete, you're reducing the risk of hitting a compromised package.
> It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
> Except there is lots to gain from being the first to write about the new malware on some registry
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
I think it's more some security company writing about a vulnerability they discovered in this module or a worm/backdoor and not the company that wrote the software. The security company gets publicity and potentially gets more biz for security consulting.
We’re not talking about security researchers here:
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
>We’re not talking about security researchers here:
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
you have 1.0 installed. you enable 7 day cooldowns. an exploit is discovered in 1.0, and 1.1 is immediately released to fix the exploit. do you sit on 1.0 for 7 days?
So, the threat actor now, after making the compromise, just needs to announce that the previous version has a 0-day, and folks need to install the latest version? I love the idea of a cool down, but it can still be thwarted. I would just hope folks that are trying to patch a 0-day take extra caution to vet the new version. I wouldn't be opposed to a --cooldown 0 doing a side by side diff. I may not know what's going on in the code, but a 0-day shouldn't be a ton of new code either.
> A version whose source does not expose created_at, such as older gem servers, historical entries from before the v2 cutover, or private registries still on the v1 format, is treated as outside the window and stays resolvable.
How is that not an easy exploit to circumvent the cooldown?
Most gems in Ruby/Rails projects come from rubygems, so if they were published long ago, any exploits should have already been found hopefully. Any old gems that would attempt to release a new compromised version would now get a created_at timestamp and the cooldown applies.
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
RubyGems.org backfilled older releases with `created_at` fields, so theoretically you could still do the cooldown with very old gems (though I don't know why you would). It's only private / alternative gem servers that may not provide `created_at` fields.
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.
Hypothesis: a big accelerant of these rapid repository compromise (from Red hat to GitHub to Amazon to small startups) might be GitHub+dependabot automatic dependency updates.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
Aren't we back to the drawing board once everyone uses this?
I think the idea is that dedicated security firms and/or automated scanners will discover exploits in the cooldown period.
Yep, this is the thesis behind them. I wish people engaged more fully with this argument: it’s possible to believe that security vendors won’t do a good job of upholding their side of the bargain, but I’ve yet to see anybody argue that rather than making a faulty universalization argument against cooldowns.
If this is the idea, why don't we let the dedicated security firms and/or automated scanners find the vulnerabilities before the release?
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
i am not sure what the benefits of your proposal are compared to the "cooldown period" way.
the releases will be delayed for the same time period, but you increase the amount of coordination required significantly and reduce user agency.
The point is to allow the automated scanners a chance to run.
Every security company and their cousin wants to be the one to find the next big dependency malware.
The idea that a package can be updated and with a deploy at the right time could be live on your servers in prod 10 minutes later has always been crazy, and the last years have just reinforced that.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
Yea, all the new advice around using dependency cooldowns only works if _someone_ is installing these things before you and finding the vulnerabilities.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
Using dependency cooldowns is not a free-rider problem. There's a real tradeoff here – ppl are trading their time preference for security.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
This is how a chunk of people function anyway. There are plenty of people that choose to not install "point zero" release for software of a certain importance, assuming with any major changes there are often bugs that come along with it.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
not really, there are a number of security companies doing analysis of any new packages looking for supply chain attacks, so if you wait a couple of days, till their analysis is complete, you're reducing the risk of hitting a compromised package.
It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
> It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
> Except there is lots to gain from being the first to write about the new malware on some registry
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
I think it's more some security company writing about a vulnerability they discovered in this module or a worm/backdoor and not the company that wrote the software. The security company gets publicity and potentially gets more biz for security consulting.
security researchers not the ones shipping the faulty code.
We’re not talking about security researchers here:
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
(Emphasis mine)
>We’re not talking about security researchers here:
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
you have 1.0 installed. you enable 7 day cooldowns. an exploit is discovered in 1.0, and 1.1 is immediately released to fix the exploit. do you sit on 1.0 for 7 days?
it specifically addresses this in the "The escape hatch" section...
So, the threat actor now, after making the compromise, just needs to announce that the previous version has a 0-day, and folks need to install the latest version? I love the idea of a cool down, but it can still be thwarted. I would just hope folks that are trying to patch a 0-day take extra caution to vet the new version. I wouldn't be opposed to a --cooldown 0 doing a side by side diff. I may not know what's going on in the code, but a 0-day shouldn't be a ton of new code either.
Security updates bypass the cooldown.
> A version whose source does not expose created_at, such as older gem servers, historical entries from before the v2 cutover, or private registries still on the v1 format, is treated as outside the window and stays resolvable.
How is that not an easy exploit to circumvent the cooldown?
Most gems in Ruby/Rails projects come from rubygems, so if they were published long ago, any exploits should have already been found hopefully. Any old gems that would attempt to release a new compromised version would now get a created_at timestamp and the cooldown applies.
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
RubyGems.org backfilled older releases with `created_at` fields, so theoretically you could still do the cooldown with very old gems (though I don't know why you would). It's only private / alternative gem servers that may not provide `created_at` fields.
Can you in your own gem depend on gems from another server? Or does it need to be configured on the client?
If not, and the current defacto standard gem server doesn't accept v1 anymore, we're good I suppose?
Meanwhile ruby is dropping ranks.
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.