i would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
I've been working on a pi extension to do this, after frustration with getting the best current local models to stay on track, and just to deal with their relative slowness; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
A SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone (both Go/ebiten).
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
I just can’t get excited about any of these meta-frameworks.
Doesn’t everyone get by now that any advance just gets rolled in to Claude and Codex a few months later, then the downstream competitors a few months after that?
indeed, but I also value the freedom of using whichever provider I want and not get locked into Claude Code alone for instance. I want to use both Opus, DeepSeek and GPT 6 at the same time, not just the 2/3 models Anthropic offers.
What I am trying to build is something that gives you absolute control, agnostic of the subscription/API you will use.
I agree with this. Why get locked in at the harness layer. I want my workspace to be independent of my coding agents and models so I can use multiple and switch as I wish
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
I like your analogy. The main problem though is context and keeping it clean as much as possible as long a parallelization. This is what drove to build this tool: having control of everything that the LLMs will do, controlling all with one main planner that orchestrates the rest. This way we can have cheaper LLMs with a short context window used (less intelligence degradation) while still obtaining the same objective.
And again, you can have a clear picture of everything structured as tasks.
until we can get to rely on huge swarms of agents (tasks) being directed on the planner alone I don't see how we can get a better framework.
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Yeah it's exactly what I've seen as well. Ordewell is close to your high end case: a frontier model builds the plan once, then it's fixed, the executing model can't reinterpret it.
Won't get you to 4B, but should help a small model that only has to execute, not plan and execute at once.
I thought about it but where the tool shines are large undefined tasks, which are complex to quantify and test (not as easy as implementing a simple bug fix that you can test directly). Even if I were to find such test dataset it would probably require a lot of money to reach statistically significant results.
Either way, the framework is inspired a lot from Matt Pocock, and is pretty much established.
In my experience the details matter a lot. Too rigid a plan and the agents get stuck and then thrash endlessly on work they manufacture for themselves, too little they get lost, the integration strategy is in part forced by cost (e.g. using Claude Code subscription discounts by using their TUI), etc.
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model
I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
To me that's not the embarrassing part... unmaintainable would be. Using AI for coding and writing doc is standard practice now, not using it would be insane.
FWIW, replying to people with AI generated replies is (at least to me) extremely disrespectful. Please don't do it without consent from the other party. If you expect someone (who isn't a bot) to make the effort to read what you wrote then please make the effort to write it yourself.
How does this compare with Matt Pocock's /grill-me -> /to-spec -> /to-tickets ? I see that Ordewell recommends model and effort, but is the workflow overall the same?
Yeah absolutelly. I just wanted that exact framework in a simple controllable UI, where I could also plug in any of my subscriptions/API keys such as Claude Code or Codex at the same time.
With Ordewell you get all layed out as tasks that are easily manageble (Matt's specs) in a UI.
i would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
I've been working on a pi extension to do this, after frustration with getting the best current local models to stay on track, and just to deal with their relative slowness; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
A SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone (both Go/ebiten).
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
- https://github.com/Dicklesworthstone/beads_rust
I just can’t get excited about any of these meta-frameworks.
Doesn’t everyone get by now that any advance just gets rolled in to Claude and Codex a few months later, then the downstream competitors a few months after that?
indeed, but I also value the freedom of using whichever provider I want and not get locked into Claude Code alone for instance. I want to use both Opus, DeepSeek and GPT 6 at the same time, not just the 2/3 models Anthropic offers. What I am trying to build is something that gives you absolute control, agnostic of the subscription/API you will use.
I agree with this. Why get locked in at the harness layer. I want my workspace to be independent of my coding agents and models so I can use multiple and switch as I wish
Btw you can configure Claude code to use any provider or model you want
Sure, as you could on OpenCode/PI, but most of us use subscriptions for which you are forced to use the relative UI
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
I like your analogy. The main problem though is context and keeping it clean as much as possible as long a parallelization. This is what drove to build this tool: having control of everything that the LLMs will do, controlling all with one main planner that orchestrates the rest. This way we can have cheaper LLMs with a short context window used (less intelligence degradation) while still obtaining the same objective. And again, you can have a clear picture of everything structured as tasks. until we can get to rely on huge swarms of agents (tasks) being directed on the planner alone I don't see how we can get a better framework.
Have you looked at AWS's AI-DLC setup?
https://github.com/awslabs/aidlc-workflows
very interesting, the concept is the same. I see they flagged Opus as the suggested planner, which makes a lot of sense. Thanks
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Yeah it's exactly what I've seen as well. Ordewell is close to your high end case: a frontier model builds the plan once, then it's fixed, the executing model can't reinterpret it. Won't get you to 4B, but should help a small model that only has to execute, not plan and execute at once.
Have you quantified the performance on any particular benchmark?
I thought about it but where the tool shines are large undefined tasks, which are complex to quantify and test (not as easy as implementing a simple bug fix that you can test directly). Even if I were to find such test dataset it would probably require a lot of money to reach statistically significant results. Either way, the framework is inspired a lot from Matt Pocock, and is pretty much established.
In my experience the details matter a lot. Too rigid a plan and the agents get stuck and then thrash endlessly on work they manufacture for themselves, too little they get lost, the integration strategy is in part forced by cost (e.g. using Claude Code subscription discounts by using their TUI), etc.
[flagged]
What sort of determinism do you have in mind?
[flagged]
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
I opened the page looking for a differenciator, but everything about the project is AI-written (including author replies in these comments).
To me that's not the embarrassing part... unmaintainable would be. Using AI for coding and writing doc is standard practice now, not using it would be insane.
so why did it get upvotes and a place on coveted frontpage?
Fair, and I won't pretend otherwise. I heavily used AI to help write the docs and code. The thing is though... I designed it and I stand behind it.
FWIW, replying to people with AI generated replies is (at least to me) extremely disrespectful. Please don't do it without consent from the other party. If you expect someone (who isn't a bot) to make the effort to read what you wrote then please make the effort to write it yourself.
I agree with you, I did not reply to anybody with an LLM. I employed AI for building
But why write these comments with AI?
I replied to everyone on my own.
Complete brainrot lol
have you tried https://github.com/highflame-ai/codeoid
nope but I will check it out, thanks
[dead]
How does this compare with Matt Pocock's /grill-me -> /to-spec -> /to-tickets ? I see that Ordewell recommends model and effort, but is the workflow overall the same?
Yeah absolutelly. I just wanted that exact framework in a simple controllable UI, where I could also plug in any of my subscriptions/API keys such as Claude Code or Codex at the same time. With Ordewell you get all layed out as tasks that are easily manageble (Matt's specs) in a UI.
kind of amazing how much thought /grill-me gets. have you looked inside those skills? grill-me is literally 4 sentences!
Absolutelly, it is brilliant! I use it daily myself from Ordewell
[flagged]
Did you try some of the other things like OpenSpec before you built this?
[dead]