Hey OP, this is absolutely awesome and I am very much on your creative wavelength! I'd love to get in contact with you if you have a moment, as I've built an adjacent 3D visualizer that might complement the rendering portion of your system by providing actual glyph-level rendering of each file instead of block-based representations (glyph3d.dev). I have a similar "trace" feature that shows files pre/post edit from CC at the moment, so there is some definitely interesting overlap.
Little rough is putting it generously haha, and thanks for checking it out anyway!
This is a standard three.js implementation with just some fancy shaders and loading primitives. If you looked at the IDE demo, that's basically the test bed I use for rendering. The library is available in NPM (I can update to latest this evening if you'd like to pull it that way) and it's meant to be composable, as are the app tools and field abstractions.
My hope is that it means import as a dep, call into the bootstrap, and then you can place a "glyph field" node wherever ya want and stream text and buffer updates to it.
A lot of people want a use case. One I think might be cool is some kind of spatial/represented comparison: let's see how two different models interact with the codebase (for the same problem), what they touched, and what they did. Or the same model, but averaged across 100 runs, so we can see how much variance there really is per task. Something along those lines sounds interesting to me.
I think the issue is, on the surface this is very much a "watch this non-deterministic actvity" but I can see the value in it. This is self promotional but I am working on a "Brain" (https://github.com/gitsense/pi-brains) for the Pi (https://github.com/earendil-works/pi) coding agent and I can actually see this being quite useful to quickly tell if you did make an agent smarter or not.
There is obviously different ways you can do this, but I can see being able to quickly visualize changes based on additional knowledge/constraints can be quite useful.
One of the rules shows how you can inject information when you know an agent wants to read a certain file. So if you have skills related to a certain file/directory, you can inject the information when you need to.
So using the OPs visualization tool, if you find the agent wandering you could create a new rule to guide the agent and compare it with and with guidance.
This is really cool! I’m becoming convinced the optimal UI to engage with agents, long term is going to be something spatial. No idea shape that even takes, though I really feel what you’ve made might be Xerox PARC days in terms of metaphor maturity, but there’s some real new seeds of “obvious in retrospect” ideas here. Thanks for conceiving of and building this!
As a once-off, not much because the treemap or graph look arbitrary so it's just eye candy. But a small fast treemap in your IDE would quickly become something you depend on, just like minimaps turned out to be really useful to enough people that they become a default in vscode.
My original motivation is to compare the task-solving ability of LLMs by visualizing the agent's trajectory. This offers an alternative way to inspect the capabilities of LLMs and agent systems.
> Anyways, besides that, the README does clearly state "The Problem" and "The Idea" which outlines exactly why the author built this particular project.
I asked about the use case _because_ the readme states a problem but does not offer a solution to that problem as far as I could see.
I agree with all of this, and gource is an excellent example, but this is still "what" and builds no case for a "why". Not even an anecdote like "I told the agent this and it completely misunderstood and did the other thing so I built this which you can use to undo the work after the fact."
Because it’s fun and not everything has to exist for a reason other than that maybe? I built something in the same vein, and the way is because it’s neat. It serves no real purpose.
Tried exploring a small project i built with CC, but i don't see anything in the tree/terrain view (The edits/reads/writes do show up in the timeline). The project itself doesn't exist on my drive anymore, is that a requirement?
Haven't tried it yet, but I think we need something in that direction.
The terminal "Read file: xyz" mentions are not really followable.
It would be nice to easily see where the LLM is taking info from.
this is cool! i started tracking file changes across sessions with file hunk method on a binlog.. to see hot paths, or do partial reverts where files werent committed across my sessions. not for visual, but operational for the harness i use. approach lets you see the whole rollups on a period of days or minutes. think there are many use cases where you can run a side agent to check things / progress analyze risky areas as you go etc etcs .. all file change lookups are 3-5ms. script here: https://github.com/geeewhy/haicue-brew/tree/main/scripts
i wonder if i would be able to visualize with mindwalk
That's great but im not sure what the use cases are. Did you have something in mind when building this to help the conception process ? I dont feel like i need to know what the agent did, never opened the session json. But there could be gold in there perhaps.
My original motivation is to compare the task-solving ability of LLMs by visualizing the agent's trajectory. This offers an alternative way to inspect the capabilities of LLMs and agent systems.
This reminds me of that community epsiode where they get the VR system and the dean has to walk through a maze, climb up several things so that they can go to a filing cabinet and then retrive a file. Yes, its cool to see and watch but it seems to be adding more friction than reducing it. Like who is going to spend that much time watching what their agents did when there are far quicker and efficient ways of scanning through changes and organizing code with better ergonomics?
Standard CLI version control tools like git diff, git log, or any built-in IDE diff viewer. Scrolling through a text summary of changes is inherently faster and more ergonomic than generating a 3D spatial visualization and watching an agent physically navigate through it.
I absolutely hate using diff view except when I'm debugging. Reading (or even skimming) 2 text files at once demands a lot of concentration. Now following activity in visual displays like these also takes concentration, and the process can certainly be refined further, but they can help you get a quick overview of what's going on in the same way as a profiling graph.
(Though I disagree, not sure why this comment was flagged? vouched for it)
If you have a big codebase, text diff is not faster to see what was done here and there, then a visual overview. I don't know about this project, but experimented with other tools of this kind and am building one myself. With text diff - you have to parse lots of text and you cannot see on one glance that, oh, these changes are in the rendering, this is DB - and here was done something with the server connection. If done right, I can immediately or very quickly see what was changed - and also decide if the agent (or human) did work in the right spots or gone off track. Now doing this right is not easy, but reading lot's of text is definitely slower then seeing, oh in this area there was a change.
gource seems to be on of those tools that people forget about. I don’t know if it’s the name or just simply that it’s the kind of thing you run once and think “that’s nice - what do I do with it?”. It’s not something you run regularly so it slowly disappears from memory.
I think the chord diagram in the video is not informative. Generally speaking I suggest starting with the problem you are trying to shed light on, and devise the visualization accordingly. For example, if you are shipping a mobile app and want to keep your package small, the treemap in the video would help you understand how the space is allocated.
When it comes to coding sessions my problems are memory and inter-agent communication, for which I'm using https://ctx.rs/, and tracking, for which I'm testing https://usegitai.com/
I do not see what there is to be gained from replaying a session.
Much appreciated. That said, I think we may have different use cases. My goal is to solve the problem of inspecting and evaluating how LLMs and agents perform tasks. This project could draw inspiration from Anthropic's research: https://www.anthropic.com/research/global-workspace. It's just the beginning, and there are lots of things to improve. :)
Hey OP, this is absolutely awesome and I am very much on your creative wavelength! I'd love to get in contact with you if you have a moment, as I've built an adjacent 3D visualizer that might complement the rendering portion of your system by providing actual glyph-level rendering of each file instead of block-based representations (glyph3d.dev). I have a similar "trace" feature that shows files pre/post edit from CC at the moment, so there is some definitely interesting overlap.
Well done creating A Cool Thing!
Glyph3D looks cool! I'll look into it and give it a try as soon as I can.
Cheers to ya! The rendering library is separate from the app itself, so I can help push that around if ya want a better handle to play with it.
looks cool but a little rough. If you had another 3d scene how would you integrate this?
Little rough is putting it generously haha, and thanks for checking it out anyway!
This is a standard three.js implementation with just some fancy shaders and loading primitives. If you looked at the IDE demo, that's basically the test bed I use for rendering. The library is available in NPM (I can update to latest this evening if you'd like to pull it that way) and it's meant to be composable, as are the app tools and field abstractions.
My hope is that it means import as a dep, call into the bootstrap, and then you can place a "glyph field" node wherever ya want and stream text and buffer updates to it.
A lot of people want a use case. One I think might be cool is some kind of spatial/represented comparison: let's see how two different models interact with the codebase (for the same problem), what they touched, and what they did. Or the same model, but averaged across 100 runs, so we can see how much variance there really is per task. Something along those lines sounds interesting to me.
> A lot of people want a use case.
I think the issue is, on the surface this is very much a "watch this non-deterministic actvity" but I can see the value in it. This is self promotional but I am working on a "Brain" (https://github.com/gitsense/pi-brains) for the Pi (https://github.com/earendil-works/pi) coding agent and I can actually see this being quite useful to quickly tell if you did make an agent smarter or not.
There is obviously different ways you can do this, but I can see being able to quickly visualize changes based on additional knowledge/constraints can be quite useful.
I'd advise you to put a video demo at the top of your repo, similar to OP, to convince or interest me.
There are some additional things I need to implement first but I do have a hands on repo that you actually try at
https://github.com/gitsense/gsc-rules-demos
One of the rules shows how you can inject information when you know an agent wants to read a certain file. So if you have skills related to a certain file/directory, you can inject the information when you need to.
So using the OPs visualization tool, if you find the agent wandering you could create a new rule to guide the agent and compare it with and with guidance.
Thanks bro, you see the value of this project!
This is really cool! I’m becoming convinced the optimal UI to engage with agents, long term is going to be something spatial. No idea shape that even takes, though I really feel what you’ve made might be Xerox PARC days in terms of metaphor maturity, but there’s some real new seeds of “obvious in retrospect” ideas here. Thanks for conceiving of and building this!
Thanks for your thought!
Very cool project!
Other than being aesthetically pleasing (which it doubtlessly is), what's the use case for this?
As a once-off, not much because the treemap or graph look arbitrary so it's just eye candy. But a small fast treemap in your IDE would quickly become something you depend on, just like minimaps turned out to be really useful to enough people that they become a default in vscode.
My original motivation is to compare the task-solving ability of LLMs by visualizing the agent's trajectory. This offers an alternative way to inspect the capabilities of LLMs and agent systems.
2019, in a comment about gource:
> It's pretty, but I can't think of anything I'd use it as a "tool" for. - https://news.ycombinator.com/item?id=21560013
Seems any time this sort of software comes up on, people can't let the use case be "this is aesthetically pleasing".
Anyways, besides that, the README does clearly state "The Problem" and "The Idea" which outlines exactly why the author built this particular project.
> Anyways, besides that, the README does clearly state "The Problem" and "The Idea" which outlines exactly why the author built this particular project.
I asked about the use case _because_ the readme states a problem but does not offer a solution to that problem as far as I could see.
"The agent's understanding of the task becomes a shape you can see at a glance."
Use case is to monitor agents activity.
I agree with all of this, and gource is an excellent example, but this is still "what" and builds no case for a "why". Not even an anecdote like "I told the agent this and it completely misunderstood and did the other thing so I built this which you can use to undo the work after the fact."
Because it’s fun and not everything has to exist for a reason other than that maybe? I built something in the same vein, and the way is because it’s neat. It serves no real purpose.
Tried exploring a small project i built with CC, but i don't see anything in the tree/terrain view (The edits/reads/writes do show up in the timeline). The project itself doesn't exist on my drive anymore, is that a requirement?
Haven't tried it yet, but I think we need something in that direction. The terminal "Read file: xyz" mentions are not really followable. It would be nice to easily see where the LLM is taking info from.
very fun! sent me off into a rabbit hole building something similar tuned to my own projects
Curious, have you thought of utilizing your tool within performance diagnostic testing?
Any license for it? I think technically we can't even run it legally at the moment?
MIT. I've added it. Have fun!
Many thanks!
I know I sound childish, but I'm very excited to see our UIs catching up to sci-fi movies. Very cool work, I'll check it out today
this is cool! i started tracking file changes across sessions with file hunk method on a binlog.. to see hot paths, or do partial reverts where files werent committed across my sessions. not for visual, but operational for the harness i use. approach lets you see the whole rollups on a period of days or minutes. think there are many use cases where you can run a side agent to check things / progress analyze risky areas as you go etc etcs .. all file change lookups are 3-5ms. script here: https://github.com/geeewhy/haicue-brew/tree/main/scripts
i wonder if i would be able to visualize with mindwalk
Very nice, but needs pi.dev / rcarmo/piclaw support :)
Thanks, added to my backlog~
First thought after reading “3D map of your codebase” was that now we will be able to code in Johnny Mnemonic style :D
I tested it, history feature is per agent session, a cumulative history of all my agents would be nice.
Very glad to see this, I've been dreaming about spatial representations for code for a long time.
That's great but im not sure what the use cases are. Did you have something in mind when building this to help the conception process ? I dont feel like i need to know what the agent did, never opened the session json. But there could be gold in there perhaps.
My original motivation is to compare the task-solving ability of LLMs by visualizing the agent's trajectory. This offers an alternative way to inspect the capabilities of LLMs and agent systems.
This can help you evaluate agentic capabilites of various llm's. How they explore codebase.
Heads up, I get a 404 on the 30 sec demo link.
This reminds me of that community epsiode where they get the VR system and the dean has to walk through a maze, climb up several things so that they can go to a filing cabinet and then retrive a file. Yes, its cool to see and watch but it seems to be adding more friction than reducing it. Like who is going to spend that much time watching what their agents did when there are far quicker and efficient ways of scanning through changes and organizing code with better ergonomics?
"when there are far quicker and efficient ways of scanning through changes and organizing code with better ergonomics?"
Like what?
Standard CLI version control tools like git diff, git log, or any built-in IDE diff viewer. Scrolling through a text summary of changes is inherently faster and more ergonomic than generating a 3D spatial visualization and watching an agent physically navigate through it.
Counterpoint: it's not
I absolutely hate using diff view except when I'm debugging. Reading (or even skimming) 2 text files at once demands a lot of concentration. Now following activity in visual displays like these also takes concentration, and the process can certainly be refined further, but they can help you get a quick overview of what's going on in the same way as a profiling graph.
(Though I disagree, not sure why this comment was flagged? vouched for it)
If you have a big codebase, text diff is not faster to see what was done here and there, then a visual overview. I don't know about this project, but experimented with other tools of this kind and am building one myself. With text diff - you have to parse lots of text and you cannot see on one glance that, oh, these changes are in the rendering, this is DB - and here was done something with the server connection. If done right, I can immediately or very quickly see what was changed - and also decide if the agent (or human) did work in the right spots or gone off track. Now doing this right is not easy, but reading lot's of text is definitely slower then seeing, oh in this area there was a change.
Sort of like gource?
Wow, it's my first time learning it. The visualization seems similar.
gource seems to be on of those tools that people forget about. I don’t know if it’s the name or just simply that it’s the kind of thing you run once and think “that’s nice - what do I do with it?”. It’s not something you run regularly so it slowly disappears from memory.
what visualization library is used -- looks cool -- maybe we could adapt it to other use cases that can benefit from a graph visualization
This: https://www.npmjs.com/package/three?activeTab=readme
according to packages: https://github.com/cosmtrek/mindwalk/blob/master/web/package...
This is freakin cool.. worked in first go.
Thanks OP.
I think the chord diagram in the video is not informative. Generally speaking I suggest starting with the problem you are trying to shed light on, and devise the visualization accordingly. For example, if you are shipping a mobile app and want to keep your package small, the treemap in the video would help you understand how the space is allocated.
* https://en.wikipedia.org/wiki/Chord_diagram_(information_vis...
* https://en.wikipedia.org/wiki/Treemapping
When it comes to coding sessions my problems are memory and inter-agent communication, for which I'm using https://ctx.rs/, and tracking, for which I'm testing https://usegitai.com/
I do not see what there is to be gained from replaying a session.
Much appreciated. That said, I think we may have different use cases. My goal is to solve the problem of inspecting and evaluating how LLMs and agents perform tasks. This project could draw inspiration from Anthropic's research: https://www.anthropic.com/research/global-workspace. It's just the beginning, and there are lots of things to improve. :)
Beautiful!
Completely unrelated, but the name and the visual similarities triggered a memory for me: https://amiga.abime.net/games/view/mind-walker#screenshots
This would have been a better aesthetic by far.
"Claude, can you copy gource and put my name on it, no slop lol"
This seems like a different project to gource
amazing!
Nice. Ask your slopservant to make a video, please.
Is slopservant acceptable as a substitute to those who consider clanker any degree of an offensive slur? Nice!
https://news.ycombinator.com/item?id=48608570#48618398
"Clanker" is considered a slur
https://news.ycombinator.com/item?id=47598511#47601098
how clanker took off
Good idea!