Having worked on a number of message brokers, but not really fucked with append-only log streams, what's the actually selling point here?
Feels like the simplicity wins are made by only solving one half of the problem space (only writes), and it introduces new problems of its own (like how do we truncate the append-only log once readers are done with it?)
There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don't truncate the append-only log unless specified in its settings via a particular cleanup policy (e.g. based on message expiry or its allowed size).
Fun write-up. Good stuff, very nice contribution to to compio of compio-ws. Looks like the Iggy switched to thread-per-core compio back in July.
https://github.com/apache/iggy/pull/1960
One thing I saw mentioned in the 0.5.0 release notes (which happened shortly after migrating to compio): trying to switch to a Sans-IO model design (but maybe just for clients?). It feels like there's similar ownership concerns that might arise, as seen here with the poll-vs-completion challenges faced here. A lot of good learnings to be had, as we go forth to faster better io, eager to keep tuning in & hearing more. Thanks for the write up!
https://iggy.apache.org/blogs/2025/08/10/release-0.5.0/#what...https://www.firezone.dev/blog/sans-io
Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :)
Sans-IO is for clients only, and we'd like to introduce it one day to SDK to support the different runtimes as well as the blocking, sync client.
Krishna is a high quality engineer, pretty happy to see the good work he is doing on Iggy. Pretty excited with the future of the project as well.
Thank you, and yes, 100% agreed about Krishna's work! :)
Having worked on a number of message brokers, but not really fucked with append-only log streams, what's the actually selling point here?
Feels like the simplicity wins are made by only solving one half of the problem space (only writes), and it introduces new problems of its own (like how do we truncate the append-only log once readers are done with it?)
There's a core difference between the message queue and the stream, and one of them is that the append-only log acts as a simple database from which multiple independent consumers can read records at the same time; hence, you don't truncate the append-only log unless specified in its settings via a particular cleanup policy (e.g. based on message expiry or its allowed size).
Fun write-up. Good stuff, very nice contribution to to compio of compio-ws. Looks like the Iggy switched to thread-per-core compio back in July. https://github.com/apache/iggy/pull/1960
One thing I saw mentioned in the 0.5.0 release notes (which happened shortly after migrating to compio): trying to switch to a Sans-IO model design (but maybe just for clients?). It feels like there's similar ownership concerns that might arise, as seen here with the poll-vs-completion challenges faced here. A lot of good learnings to be had, as we go forth to faster better io, eager to keep tuning in & hearing more. Thanks for the write up! https://iggy.apache.org/blogs/2025/08/10/release-0.5.0/#what... https://www.firezone.dev/blog/sans-io
Thanks! Actually, the mentioned PR was just an internal merge; the one to the master branch is this one https://github.com/apache/iggy/pull/2299 which happened last week :)
Sans-IO is for clients only, and we'd like to introduce it one day to SDK to support the different runtimes as well as the blocking, sync client.