Interesting deep dive into how Adobe built a streaming ingestion layer on top of Apache Iceberg to handle massive volumes of Experience Platform data, addressing challenges like the small‑file problem and commit bottlenecks with asynchronous writes and compaction. All stuff I've had to deal with in the past.
Good nuggets on they partition tables by time, stage writes in separate ingestion and reporting tables, and tune snapshot and metadata handling to deliver a lakehouse‑style pipeline that scales without melting the object store.
Interesting deep dive into how Adobe built a streaming ingestion layer on top of Apache Iceberg to handle massive volumes of Experience Platform data, addressing challenges like the small‑file problem and commit bottlenecks with asynchronous writes and compaction. All stuff I've had to deal with in the past.
Good nuggets on they partition tables by time, stage writes in separate ingestion and reporting tables, and tune snapshot and metadata handling to deliver a lakehouse‑style pipeline that scales without melting the object store.