This looks cool and the mechanism looks plausible. I found the experience of trying to understand whether the claims here are legit to be aggravating.
First of all, the whole readme section about benchmarks appears to be Claude/Codex written. What a slog to read this.
Second, they claim success on SWE-Bench Verified, but it's only on 50 tasks, not making clear how these tasks are chosen. I know from experience that you can keep selecting sets of tasks until you get a result. Also, this was run 1x, and the lift they show actually only has a p val of 0.22.
There's a famous book "how to lie with statistics". I don't think the continual posts about benchmark results here are purposeful lies, but I think it's just so easy to fool yourself (and I've been burned, most recently building http://pellmell.ai ). Also I don't think this post is the most egregious example.
we are running them on a recurring basis, so will keep on updating that 50 number. and hence as it's currently running that section gets updated by claude only.
HN commenters shouldn't have to be Sherlock Holmes!
Extraordinary claims should be supported by evidence. You could have easily posted the seed, the 50 questions, and the results for claude and graft on each, but didn't.
I also see Graft got 16/16 on one update and 2/14 on another. Further evidence that each group of questions was selected somehow, not random.
Love the idea, I was thinking about this problem a few weeks ago, but never got anywhere with it. I'm intrigued by the latency savings in particular, that sounds great.
One concern I have is that right now each session gets fresh "eyes" on the problem. Right now I find I get a lot of mileage out of a combination of long-running sessions and fresh ones. I worry with a single generated concept graph that gets only incremental refreshes will become stale slowly, and in subtle ways that are hard to detect. That could lead to semantic drift in the graph from reality, and every new session will take the drifted form as gospel. Have you run any long tests (weeks or longer) on this to make sure that this doesn't happen? My understanding is SWE Bench is only a point-in-time evaluation.
Also the graph is stored in the repo, right? How mergeable is it? I know I wouldn't want to do conflict resolution on that myself, and even Opus struggles to keep all the references correct (especially when comments are involved) when there's an B->C, A->B symbol rename.
For the staleness specifically we are using hooks on claude code and codex, where on every turn, or post edit or pre-tool use, we sync the graph. so that graph never goes stale.
We have run tests on DeepSWE as well which are long running tasks, we got 20% better accuracy on the tasks where sonnet 5 failed. didn't want to post that numbers yet as I think we can do better on DeepSWE and on a cheaper model like gpt-5.6-luna or grok-4.6
The graph is pre-filling the context that does the sync, right? Doesn't that create a bias towards what's already present? If anything hallucinated does make it in at some point, won't the sync just reinforce that hallucination if some new component touches the bad info, just like a long session rabbit-holing? New sessions, in contrast, have no contextual assumptions and often catch those problems.
Opus 5 specifically seems to be affected by a severe case of turbo-encabulationitis, almost as if it was trained to spit out as complex of sentences as it can.
And may your deity of choice help you if you decide to venture on subjects which you don't have a deep understanding of, as half the sentences it generates will be (barely) cohesive.
> almost as if it was trained to spit out as complex of sentences as it can.
It probably was, at least inadvertently.
Very easy to imagine that one of the post-training step (RLHF, DPO etc) reinforced the "sounds clever" behaviour.
I intuitively and perhaps naively thought that Claude using the LSP server would negate a lot of grep use. Is this tool solving the same problem or something else?
LSP is for when you already know the symbol and want an exact answer, like go-to-definition or find-references. graft is for the step before that: you ask "where does auth happen" and it hands the agent the right files and lines to start from, so it reads a lot less to get oriented.
so graft solves for making claude code understand your codebase every session.
And I feel you can save a lot more tokens and also increase accuracy by just switching to new claude session without worrying about your codebase context. (doesn't solve for the session context yet, but we are still figuring out how can we solve for that. Maybe hooks that can manage your CLAUDE.md)
We are using hooks to direct the coding agent to use graft grep instead of just grep, so this should ideally not happen.
Also, for java projects we do support a more deeper graph, a few of our contributors are working on making it better for java projects, let us know if you have any ideas there.
We made this for X audience initially but got a lot of praise for it on reddit, so decided to put in the README. Sorry it made it more difficult for you.
In our tests, graft found the right code about twice as often as graphify (MRR 0.73 vs 0.38, recall@10 54% vs 20%). The difference is simple: graft searches inside the actual code, while graphify only looks at names and file paths.
You feel it while working too. graft hands the agent the exact file and line it needs for each question, so Claude keeps using graft.
Same is true for any other cli tools, claude never actually uses them as it's trained to use grep. but for graft as we set the directive to use graft at the start of the session and before the turn, claude just knows graft exists and also get the relevant context without it going and calling tools it was not trained on.
Thanks, that makes sense. The “Claude actually uses it” part is especially interesting. I actually find it better this approach don’t add another skill Claude might ignore, but make better retrieval the default.
Every task, your coding agent starts blind. Before it changes anything, it re-explores the repo: grep a term, open a file, follow an import, back out, try again. It is rebuilding a picture of a codebase it mapped an hour ago and threw away.
That rediscovery burns most of a run's tool calls, tokens, and latency, and it is pure overhead"
The author of this article brings up a very interesting problem -- that, at least as far as using LLM's as coders/coding assistants go, eventually context runs out and context related to the underlying codebase does too. This in turn burns tokens and in turn, wastes energy resources.
Historically (well, in the past couple of years!), a bunch of solutions have been proposed to address this problem (i.e., take abstracts/subsets/maps of code, write them to different databases and persistent storage methods, bring them back in when the LLM requires it, etc., etc.)...
But there's no really good solution to this problem (although, arguably Graft goes a lot farther than past tools and should be commended for that!) because the problem seems to lie in separate parts, across several problem domains:
1) LLM context window size -- limited. Anything that future LLM's do to make context windows larger will help ameliorate this problem.
2) Lack of a good way to represent a codebase to an LLM for training other than text.
In other words, first we need some kind of way to map codebases into Tensors rather than text (i.e., a higher-level "map" of the code) then train future LLM's on those code-specific Tensors.
3) Arguably, programming languages themselves share some of the blame...
Programming languages have historically been written so that an arbitrary corpus of text represents and can be interpreted and/or compiled into a computer program.
That is, while tools for mapping codebases exist, tools for directly training LLM's on those specific created "code maps" as Tensors, do not, do not seem to, or at least I'm currently unaware of any!
(Anyway, just thinking aloud...)
Graft looks good, and looks like it has made some serious inroads to solving the problem...
We should have moved past storing code in files, using the filesystem as the symbol database of programs, a long time ago. There was a lot of interesting research toward this in Haskell, for example, but also Academia in general. There’d be lots of value in using things like SQLite for example, or just ecosystem-specific containers that know about the layout and can present it to an IDE or LLM or a runtime in whatever shape is best suited to the task.
for your point #2: people are trying out to give coding agents LSP support, not sure though how well it'll work but it gives coding agents a better idea of the language in which the code is written.
This looks cool and the mechanism looks plausible. I found the experience of trying to understand whether the claims here are legit to be aggravating.
First of all, the whole readme section about benchmarks appears to be Claude/Codex written. What a slog to read this.
Second, they claim success on SWE-Bench Verified, but it's only on 50 tasks, not making clear how these tasks are chosen. I know from experience that you can keep selecting sets of tasks until you get a result. Also, this was run 1x, and the lift they show actually only has a p val of 0.22.
There's a famous book "how to lie with statistics". I don't think the continual posts about benchmark results here are purposeful lies, but I think it's just so easy to fool yourself (and I've been burned, most recently building http://pellmell.ai ). Also I don't think this post is the most egregious example.
we are running them on a recurring basis, so will keep on updating that 50 number. and hence as it's currently running that section gets updated by claude only.
Wow, this is really weird. I looked at the github and see you published benchmark updates in these sizes of N:
9 → 20 → 36 → 50
These are super arbitrary numbers of benchmarks to run (9, 11, 16, 14). I see a few explanations for this:
> Claude dropped "pathological trials" for you before publishing. This would make your results quite dishonest.
> You chose different sized task groups (almost certainly not the case).
> You excluded timed out trials, or something (also would be dishonest).
Hopefully there's a better explanation here!
that's just when our claude limit's were about to exhaust :/ as we have to spin up a whole claude session to test it.
but you could 100% be Sherlock Holmes!
HN commenters shouldn't have to be Sherlock Holmes!
Extraordinary claims should be supported by evidence. You could have easily posted the seed, the 50 questions, and the results for claude and graft on each, but didn't.
I also see Graft got 16/16 on one update and 2/14 on another. Further evidence that each group of questions was selected somehow, not random.
Love the idea, I was thinking about this problem a few weeks ago, but never got anywhere with it. I'm intrigued by the latency savings in particular, that sounds great.
One concern I have is that right now each session gets fresh "eyes" on the problem. Right now I find I get a lot of mileage out of a combination of long-running sessions and fresh ones. I worry with a single generated concept graph that gets only incremental refreshes will become stale slowly, and in subtle ways that are hard to detect. That could lead to semantic drift in the graph from reality, and every new session will take the drifted form as gospel. Have you run any long tests (weeks or longer) on this to make sure that this doesn't happen? My understanding is SWE Bench is only a point-in-time evaluation.
Also the graph is stored in the repo, right? How mergeable is it? I know I wouldn't want to do conflict resolution on that myself, and even Opus struggles to keep all the references correct (especially when comments are involved) when there's an B->C, A->B symbol rename.
For the staleness specifically we are using hooks on claude code and codex, where on every turn, or post edit or pre-tool use, we sync the graph. so that graph never goes stale.
We have run tests on DeepSWE as well which are long running tasks, we got 20% better accuracy on the tasks where sonnet 5 failed. didn't want to post that numbers yet as I think we can do better on DeepSWE and on a cheaper model like gpt-5.6-luna or grok-4.6
The graph is pre-filling the context that does the sync, right? Doesn't that create a bias towards what's already present? If anything hallucinated does make it in at some point, won't the sync just reinforce that hallucination if some new component touches the bad info, just like a long session rabbit-holing? New sessions, in contrast, have no contextual assumptions and often catch those problems.
I'm glad Claude is so recognisable, it lets me bounce right off the empty calorie language very efficiently.
Maybe this thing is great, but it cannot be determined with this presentation.
Opus 5 specifically seems to be affected by a severe case of turbo-encabulationitis, almost as if it was trained to spit out as complex of sentences as it can.
And may your deity of choice help you if you decide to venture on subjects which you don't have a deep understanding of, as half the sentences it generates will be (barely) cohesive.
> almost as if it was trained to spit out as complex of sentences as it can.
It probably was, at least inadvertently. Very easy to imagine that one of the post-training step (RLHF, DPO etc) reinforced the "sounds clever" behaviour.
"It tested well with the focus group..."
You can actually tell from the very first real sentence in the README:
> Efficiency is a 162-run controlled benchmark (same agent, same file tools, only the context differs).
It's so bad that I can tell it's Claude, specifically Opus 4.8/5.0, from the first 6 words.
Opus 5 is very paranoid on giving proofs for every thing it claimed. so I let it keep this one line.
I intuitively and perhaps naively thought that Claude using the LSP server would negate a lot of grep use. Is this tool solving the same problem or something else?
LSP is for when you already know the symbol and want an exact answer, like go-to-definition or find-references. graft is for the step before that: you ask "where does auth happen" and it hands the agent the right files and lines to start from, so it reads a lot less to get oriented.
so graft solves for making claude code understand your codebase every session.
And I feel you can save a lot more tokens and also increase accuracy by just switching to new claude session without worrying about your codebase context. (doesn't solve for the session context yet, but we are still figuring out how can we solve for that. Maybe hooks that can manage your CLAUDE.md)
Does Claude's grep still prepend the relative path of the file before _every_ single line? Because that is nasty, especially in java projects.
We are using hooks to direct the coding agent to use graft grep instead of just grep, so this should ideally not happen.
Also, for java projects we do support a more deeper graph, a few of our contributors are working on making it better for java projects, let us know if you have any ideas there.
Relative path? You sweet summer child. It prepends the absolute path for me all the time.
Why these animations in the github README? why? It just made understanding anything more difficult
We made this for X audience initially but got a lot of praise for it on reddit, so decided to put in the README. Sorry it made it more difficult for you.
interesting, thanks for sharing!
What's the benchmark against graphify?
In our tests, graft found the right code about twice as often as graphify (MRR 0.73 vs 0.38, recall@10 54% vs 20%). The difference is simple: graft searches inside the actual code, while graphify only looks at names and file paths.
You feel it while working too. graft hands the agent the exact file and line it needs for each question, so Claude keeps using graft.
Same is true for any other cli tools, claude never actually uses them as it's trained to use grep. but for graft as we set the directive to use graft at the start of the session and before the turn, claude just knows graft exists and also get the relevant context without it going and calling tools it was not trained on.
Thanks, that makes sense. The “Claude actually uses it” part is especially interesting. I actually find it better this approach don’t add another skill Claude might ignore, but make better retrieval the default.
Is this still cheaper when stale?
we use claude hooks. so that, post edits it auto updates and before using the graft tools also we check if it's already in sync.
Hooks also solve for the issue of the LLMs not calling our CLI tools instead of Grep.
I'm thinking more in terms of the case where my coworkers' agents aren't working via graft, and changes are made without the post-edit update hooks.
if the changes are on your local when you pull those changes from the remote, graft will be auto-synced before graft tool use.
Oh, of course. Brilliant.
>"The problem:
Every task, your coding agent starts blind. Before it changes anything, it re-explores the repo: grep a term, open a file, follow an import, back out, try again. It is rebuilding a picture of a codebase it mapped an hour ago and threw away.
That rediscovery burns most of a run's tool calls, tokens, and latency, and it is pure overhead"
The author of this article brings up a very interesting problem -- that, at least as far as using LLM's as coders/coding assistants go, eventually context runs out and context related to the underlying codebase does too. This in turn burns tokens and in turn, wastes energy resources.
Historically (well, in the past couple of years!), a bunch of solutions have been proposed to address this problem (i.e., take abstracts/subsets/maps of code, write them to different databases and persistent storage methods, bring them back in when the LLM requires it, etc., etc.)...
But there's no really good solution to this problem (although, arguably Graft goes a lot farther than past tools and should be commended for that!) because the problem seems to lie in separate parts, across several problem domains:
1) LLM context window size -- limited. Anything that future LLM's do to make context windows larger will help ameliorate this problem.
2) Lack of a good way to represent a codebase to an LLM for training other than text.
In other words, first we need some kind of way to map codebases into Tensors rather than text (i.e., a higher-level "map" of the code) then train future LLM's on those code-specific Tensors.
3) Arguably, programming languages themselves share some of the blame...
Programming languages have historically been written so that an arbitrary corpus of text represents and can be interpreted and/or compiled into a computer program.
That is, while tools for mapping codebases exist, tools for directly training LLM's on those specific created "code maps" as Tensors, do not, do not seem to, or at least I'm currently unaware of any!
(Anyway, just thinking aloud...)
Graft looks good, and looks like it has made some serious inroads to solving the problem...
We should have moved past storing code in files, using the filesystem as the symbol database of programs, a long time ago. There was a lot of interesting research toward this in Haskell, for example, but also Academia in general. There’d be lots of value in using things like SQLite for example, or just ecosystem-specific containers that know about the layout and can present it to an IDE or LLM or a runtime in whatever shape is best suited to the task.
for your point #2: people are trying out to give coding agents LSP support, not sure though how well it'll work but it gives coding agents a better idea of the language in which the code is written.
slop README. If you couldn't be bothered to write it, don't expect people to read it.
Feedback taken, actually as we have been iterating on this everyday, the README got updated by the coding agents in this process.