Instead of post-processing the binary to add the application (non-SELF) schema, you could run database migrations before servicing requests. Thus, every time you start the process, the app creates and/or upgrades its own schema.
The SELF upgrade (heh, self upgrade) and rollback processes could benefit from some... fancier... footwork.
Your example has a new binary copying old data into it, but then you have to move the new binary to the deployed location. Which means an outage through stop service, data migration, replace file, start service.
What if the upgrade process was more like... write the new SELF data into the old binary, send SIGHUP, and then the service fork+execs itself, while doing haproxy-like zero downtime FD handover?
Replacing the SELF data in the existing file is safe right now, because you can't mmap segments into memory. But if you do end up figuring out some clever BLOB alignment mmap stuff, you could do the SELF upgrade like a data migration! INSERT segments/symbols, fork+exec, and the data migration cleans out the old code. :-D
Updating the SELF schema to allow multiple sets of segments and symbols would allow for this upgrade trick, but could do other fancy things... thin multi-arch binaries where only the code segments differ.
BLOB alignment should also mean more efficient static asset serving and a bunch of other niceties... definitely worthy of investigation.
However -- very strong however -- as fun as this is, I would never, ever, ever allow an internet-facing service binary to be self-writable. :-)
> We can collapse not only a complete distribution but all the state for every application into a single file, alleviating the need for /var/ or /tmp/ or /home/ or any other filesystem. The program can store its own state in the same file it is running from, and it can do so transactionally.
On the one hand: I don't think I want that. Including static content with the binary makes sense, certainly. However, storing writable run-time data there feels messy; I prefer a read only binary which is handed a writable state directory (it is worth saying that I've spent a lot of time with nix and other immutable distros).
On the other hand: This is the coolest, most fun thing I've seen in a good while, and I absolutely want to see it taken 1000% further. Who cares about perfectly operationalized immutable deployments when the hacker spirit is in the air?
I'll bet you could use this to run with another thing APE does: fat binaries. If program text lives in a database, what's one more row? Just
SELECT text FROM executable WHERE arch = $(uname -m)
and off we go:)
Edit: actually on further consideration this feels perfect for smalltalk; you can put the VM and image in a single file
Is it just me or does this create a huge potential security vulnerability where it’s a lot easier to trick the application into mutating itself. There’s also the alternate problem where if the file is placed in a privileged location, you won’t be able to store any state. And the final problem that each user needs their own copy of the application if it’s a multi-user application.
The biggest concern for me would be the security angle - if there would be some way to seal the executable itself and descriptor tables so that an application can be guaranteed to never touch that and only ever modify the other “runtime” tables. Not doable with raw sqlite though since it has no kind of ACL mechanism, but would be a neat extension so that the interpreter handed the handle to the process directly with the privileged tables cordoned off from writing.
Simple -- you just add a custom SQLite VFS that ensures particular SQLite pages are mapped into underlying OS pages that are appropriately mprotect()ed. Try to modify the executable pages and you crash (W^X). Or you know, don't try to use a hack like this where security matters.
SQLite's unix VFS is actually using a mixture of mmap and write() by default[1] and you'd need to combine that with mseal() and some more pieces to actually pull it off. It would probably be possible.
Between this and actually portable executable I'm not convinced someone hasn't made a PNG thats a spreadsheet, or an audio file the somehow renders DOOM across the room. HN amazes me with the absolutely cursed ideas of implementing a minecraft in pure css (or showing whatever other nightmares one can do with CSS). It shows the most incredible creativity in what one can do with the freedom of arranging bits however one wants. I'm in love with all these cursed projects and hope they never stop.
PoC || GTFO has an issue that is a PDF that is also a valid NES rom which will render the md5sum of the PDF itself, and other crazy tricks of that type over the years.
I wonder if the interpreter could create a mount namespace and mount virtual filesystems backed by the SQLite database itself, so you wouldn’t need source changes to self-contain (ha!) file accesses.
This for some reason reminds me of OS/400 libraries. Basically on AS/400 everything is an object, libraries are basically like DB tables but are first class OS objects (like files in unix). You might want to read up on it, they took the concept incredibly far and it’s of course still a part of i series to this day. You basically can use SQL right on the command line.
This is deranged, and perilously close to dumb, which makes it one of the best things I’ve seen on hacker news this year.
Absolutely wonderful stuff.
Instead of post-processing the binary to add the application (non-SELF) schema, you could run database migrations before servicing requests. Thus, every time you start the process, the app creates and/or upgrades its own schema.
The SELF upgrade (heh, self upgrade) and rollback processes could benefit from some... fancier... footwork.
Your example has a new binary copying old data into it, but then you have to move the new binary to the deployed location. Which means an outage through stop service, data migration, replace file, start service.
What if the upgrade process was more like... write the new SELF data into the old binary, send SIGHUP, and then the service fork+execs itself, while doing haproxy-like zero downtime FD handover?
Replacing the SELF data in the existing file is safe right now, because you can't mmap segments into memory. But if you do end up figuring out some clever BLOB alignment mmap stuff, you could do the SELF upgrade like a data migration! INSERT segments/symbols, fork+exec, and the data migration cleans out the old code. :-D
Updating the SELF schema to allow multiple sets of segments and symbols would allow for this upgrade trick, but could do other fancy things... thin multi-arch binaries where only the code segments differ.
BLOB alignment should also mean more efficient static asset serving and a bunch of other niceties... definitely worthy of investigation.
However -- very strong however -- as fun as this is, I would never, ever, ever allow an internet-facing service binary to be self-writable. :-)
> We can collapse not only a complete distribution but all the state for every application into a single file, alleviating the need for /var/ or /tmp/ or /home/ or any other filesystem. The program can store its own state in the same file it is running from, and it can do so transactionally.
On the one hand: I don't think I want that. Including static content with the binary makes sense, certainly. However, storing writable run-time data there feels messy; I prefer a read only binary which is handed a writable state directory (it is worth saying that I've spent a lot of time with nix and other immutable distros).
On the other hand: This is the coolest, most fun thing I've seen in a good while, and I absolutely want to see it taken 1000% further. Who cares about perfectly operationalized immutable deployments when the hacker spirit is in the air?
I'll bet you could use this to run with another thing APE does: fat binaries. If program text lives in a database, what's one more row? Just
and off we go:)Edit: actually on further consideration this feels perfect for smalltalk; you can put the VM and image in a single file
Is it just me or does this create a huge potential security vulnerability where it’s a lot easier to trick the application into mutating itself. There’s also the alternate problem where if the file is placed in a privileged location, you won’t be able to store any state. And the final problem that each user needs their own copy of the application if it’s a multi-user application.
The biggest concern for me would be the security angle - if there would be some way to seal the executable itself and descriptor tables so that an application can be guaranteed to never touch that and only ever modify the other “runtime” tables. Not doable with raw sqlite though since it has no kind of ACL mechanism, but would be a neat extension so that the interpreter handed the handle to the process directly with the privileged tables cordoned off from writing.
Simple -- you just add a custom SQLite VFS that ensures particular SQLite pages are mapped into underlying OS pages that are appropriately mprotect()ed. Try to modify the executable pages and you crash (W^X). Or you know, don't try to use a hack like this where security matters.
SQLite's unix VFS is actually using a mixture of mmap and write() by default[1] and you'd need to combine that with mseal() and some more pieces to actually pull it off. It would probably be possible.
(There's prior art here; although done differently: https://sqlite.org/src/file/ext/misc/appendvfs.c).
[1]: https://sqlite.org/mmap.html
Between this and actually portable executable I'm not convinced someone hasn't made a PNG thats a spreadsheet, or an audio file the somehow renders DOOM across the room. HN amazes me with the absolutely cursed ideas of implementing a minecraft in pure css (or showing whatever other nightmares one can do with CSS). It shows the most incredible creativity in what one can do with the freedom of arranging bits however one wants. I'm in love with all these cursed projects and hope they never stop.
PoC || GTFO has an issue that is a PDF that is also a valid NES rom which will render the md5sum of the PDF itself, and other crazy tricks of that type over the years.
https://dl.packetstormsecurity.net/mag/pocgtfo/pocorgtfo14.p...
so the executable is now a database you can query with sql. next step is someone writing an ORM against a binary and i will simply retire.
I wonder if the interpreter could create a mount namespace and mount virtual filesystems backed by the SQLite database itself, so you wouldn’t need source changes to self-contain (ha!) file accesses.
Fantastic ideas. The SQL Injection to ACE pipeline is incredible.
This for some reason reminds me of OS/400 libraries. Basically on AS/400 everything is an object, libraries are basically like DB tables but are first class OS objects (like files in unix). You might want to read up on it, they took the concept incredibly far and it’s of course still a part of i series to this day. You basically can use SQL right on the command line.
extremely creative stuff!