This might be a very dumb question, but if the process is being run under KVM to catch `int 0x03` then couldn't you also use KVM to catch `syscall` and execute the original binary as-is? I don't understand what value the instruction rewriting is providing here.
Yes, that seems unneccessary. The overhead of trapping and rewriting every syscall instruction once can't be (much) greater than that required for rewriting them at the start either.
Even if you disallow executing anything outside of the .text section, you still need the syscall trap to protect against adversarial code which hides the instruction inside an immediate value:
foo: mov eax, 0xc3050f ;return a perfectly harmless constant
ret
...
call foo+1
(this could be detected if the tracing went by control flow instead of linearly from the top, but what if it's called through a function pointer?)
You either have a writing style that is uncannily similar to what an LLM generates, or this article was substantially written by an LLM. I don't know what it is about the style, but I just find it a bit exhausting, like an overfit on "engaging writing" that strips away sincerity.
Name sounds very likely not an English speaker. And the one reply here to a top-level comment is extremely obvious. I think it's unfortunate that people who write English poorly feel the need to do it, but I get it at least. The person behind this probably has a real interest and knowledge in the space but feels they can't communicate it without assistance.
It is too bad, though. People bad at English will themselves be reading this forever now and think this is the way real people write, speak, or are supposed to.
It's many things. The relentless ethusiasm about everything. Prefacing any answer to a question with an affirmation that it was a good question first. And yes, sorry, pedants of the web who feel witch-hunted because you knew how to employ keyboard shortcuts and used em-dashes in 2015 and have the receipts to prove it -- you never used 17 in the span of a single page. I think that was the first I can remember using ever and I had to contrive a way to do it where a semi-colon wouldn't clearly work better.
You mentioned SECCOMP_RET_TRACE, but there is also SECCOMP_RET_TRAP[1] which appears to perform better. There is also KVM. Both of these are options for gVisor: <https://github.com/google/gvisor>
I assume this would break observability through existing methods, right? If you were to strace a process that has been patched, would you see regular syscall data (as if it wasnt patched) or would your syscall replacement appear along the way?
Good question. I didn't cover this in the post — the binary doesn't run on the host kernel directly. It runs inside a lightweight KVM-based VM with no operating system. The shim is the only thing handling syscalls inside the guest. So strace on the host wouldn't see anything — no syscalls reach the host kernel from the guest. From the host side, the only visible activity is the hypervisor process making syscalls on behalf of the guest.
Inside the guest, there's no kernel to attach strace to — the shim IS the syscall handler. But we do have full observability: every syscall that hits the shim is logged to a trace ring buffer with the syscall number, arguments, and TSC timestamp. It's more complete than strace in some ways — you see denied calls too, with the policy verdict, and there's no observer overhead because the logging is part of the dispatch path.
So existing tools don't work, but you get something arguably better: a complete, tamper-proof record of every syscall the process attempted, including the ones that were denied before they could execute.
I'll publish a follow-on tomorrow that details how we load and execute this rewritten binary and what the VMM architecture looks like.
I mean just distributing the regular compiled x86_64 binary and then running it as a normal executable on the client side but just using that syscall shim so it is safe.
If you think about the fundamentals involved here, what you actually need is for the OS to refuse to implement any syscalls, and not share an address space.
A process is already a hermetically sealed sandbox. Running untrusted code in a process is safe. But then the kernel comes along and pokes holes in your sandbox without your permission.
On Linux you should be able to turn off the holes by using seccomp.
This is the kind of foundation that I would feel comfortable running agents on. It’s not the whole solution of course (yes agent, you’re allowed to delete this email but not that email can’t be solved at this level)… let me know when you tackle that next :-)
This might be a very dumb question, but if the process is being run under KVM to catch `int 0x03` then couldn't you also use KVM to catch `syscall` and execute the original binary as-is? I don't understand what value the instruction rewriting is providing here.
Yes, that seems unneccessary. The overhead of trapping and rewriting every syscall instruction once can't be (much) greater than that required for rewriting them at the start either.
Even if you disallow executing anything outside of the .text section, you still need the syscall trap to protect against adversarial code which hides the instruction inside an immediate value:
(this could be detected if the tracing went by control flow instead of linearly from the top, but what if it's called through a function pointer?)You either have a writing style that is uncannily similar to what an LLM generates, or this article was substantially written by an LLM. I don't know what it is about the style, but I just find it a bit exhausting, like an overfit on "engaging writing" that strips away sincerity.
Name sounds very likely not an English speaker. And the one reply here to a top-level comment is extremely obvious. I think it's unfortunate that people who write English poorly feel the need to do it, but I get it at least. The person behind this probably has a real interest and knowledge in the space but feels they can't communicate it without assistance.
It is too bad, though. People bad at English will themselves be reading this forever now and think this is the way real people write, speak, or are supposed to.
It's many things. The relentless ethusiasm about everything. Prefacing any answer to a question with an affirmation that it was a good question first. And yes, sorry, pedants of the web who feel witch-hunted because you knew how to employ keyboard shortcuts and used em-dashes in 2015 and have the receipts to prove it -- you never used 17 in the span of a single page. I think that was the first I can remember using ever and I had to contrive a way to do it where a semi-colon wouldn't clearly work better.
It’s clearly LLM written but the idea was interesting enough that I read it. I suspect based on username the writer is cleaning up their voice.
I think the idea of sharing the raw prompt traces is good. Then I can feed that to an LLM and get the original information prior to expansion.
You mentioned SECCOMP_RET_TRACE, but there is also SECCOMP_RET_TRAP[1] which appears to perform better. There is also KVM. Both of these are options for gVisor: <https://github.com/google/gvisor>
[1] <https://github.com/google/gvisor/blob/master/pkg/sentry/plat...>
There's also SECCOMP_RET_USER_NOTIF, which is typically used by container runtimes for their sandboxing.
SECCOMP_RET_USER_NOTIF seems to involve sending a struct over an fd on each syscall. Do they really use it? Performance ought to suffer.
Also gVisor (aka runsc) is a container runtime as well. And it doesn't gatekeep syscalls but chooses to re-implement them in userland.
I assume this would break observability through existing methods, right? If you were to strace a process that has been patched, would you see regular syscall data (as if it wasnt patched) or would your syscall replacement appear along the way?
Good question. I didn't cover this in the post — the binary doesn't run on the host kernel directly. It runs inside a lightweight KVM-based VM with no operating system. The shim is the only thing handling syscalls inside the guest. So strace on the host wouldn't see anything — no syscalls reach the host kernel from the guest. From the host side, the only visible activity is the hypervisor process making syscalls on behalf of the guest.
Inside the guest, there's no kernel to attach strace to — the shim IS the syscall handler. But we do have full observability: every syscall that hits the shim is logged to a trace ring buffer with the syscall number, arguments, and TSC timestamp. It's more complete than strace in some ways — you see denied calls too, with the policy verdict, and there's no observer overhead because the logging is part of the dispatch path.
So existing tools don't work, but you get something arguably better: a complete, tamper-proof record of every syscall the process attempted, including the ones that were denied before they could execute. I'll publish a follow-on tomorrow that details how we load and execute this rewritten binary and what the VMM architecture looks like.
Really informative writing thank you.
How secure does this make a binary? For example would you be able to run untrusted binary code inside a browser using a method like this?
Then can websites just use C++ instead of javascript for example?
They already can use C++ if they want to. Emscripten? Jslinux?
I mean just distributing the regular compiled x86_64 binary and then running it as a normal executable on the client side but just using that syscall shim so it is safe.
If you think about the fundamentals involved here, what you actually need is for the OS to refuse to implement any syscalls, and not share an address space.
A process is already a hermetically sealed sandbox. Running untrusted code in a process is safe. But then the kernel comes along and pokes holes in your sandbox without your permission.
On Linux you should be able to turn off the holes by using seccomp.
Love the detailed write up, thanks!
This is the kind of foundation that I would feel comfortable running agents on. It’s not the whole solution of course (yes agent, you’re allowed to delete this email but not that email can’t be solved at this level)… let me know when you tackle that next :-)
Hah, I've been looking into something amusingly similar to track mmap syscalls for a process :)
> It can’t detect the interception
What's stopping the process from reading its own memory and seeing that the syscall was patched?
I've been thinking of making a kernel patch that disables eBPF for certain processes as a privacy tool. Everyone is using eBPF now.
What about int 80h?