> Because AI agents are stateful, they frequently pause for seconds or hours while waiting on a database lookup, an external software tool, or human feedback. To make our accelerators more power-efficient during those pauses, our new Agent Compute Engine software automatically suspends idle agent sessions and shifts working memory off the chip in milliseconds, cutting operational infrastructure costs
Nice. I wonder how they manage to move kv cache from working memory to disk so quickly to enable speed ups when pauses are a few seconds at most
Off chip does not have to mean to disk? I would assume a staged approach. First offload to CPU RAM, and then to disk (if not used within a few seconds), and then to network storage (so it can maybe be resumed by other machines). And some request routing that makes use of cache affinity based on session/user.
> Because AI agents are stateful, they frequently pause for seconds or hours while waiting on a database lookup, an external software tool, or human feedback. To make our accelerators more power-efficient during those pauses, our new Agent Compute Engine software automatically suspends idle agent sessions and shifts working memory off the chip in milliseconds, cutting operational infrastructure costs
Nice. I wonder how they manage to move kv cache from working memory to disk so quickly to enable speed ups when pauses are a few seconds at most
Off chip does not have to mean to disk? I would assume a staged approach. First offload to CPU RAM, and then to disk (if not used within a few seconds), and then to network storage (so it can maybe be resumed by other machines). And some request routing that makes use of cache affinity based on session/user.
Seems like a big deal, but there's nothing about availability. They might say more during the broadcast later today.
More bad news for Nvidia today.