This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.
The cheap, fast and smart-enough LLM space has been wildly neglected. Jev is one of the few players in the space truly targeting that space. And for a lot of people it is the first time they are asking "what could I build if llms were interaction-speed fast?". The answers are cool, the problem is that Jev is not, I think, smart-enough yet to have that many applications, but it's smart enough that you can start to see what they will look like.
A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.
That being said, one thing having been unrealistic 10 years ago and just about possible today doesn't mean that it's going to change the world the same way another technically related, previously-impossible thing did. The Jev hype gives me a bit of the "you're still early to crypto" vibes of some later altcoins. I really like the idea, I think it's going to open up possibilities for using classifiers where we wouldn't or couldn't have trained one before. I'm crossing my fingers for an open weights version to drop. But it's still just a classifier, people have built similar things before Jev, the one thing that really stands out about it is their ability to generate hype.
We have a bad universal classifier now (via Jev). 0->1, one might say.
A bad universal classifier does suggest a good one later. And that is exactly what I would call "theoretically doable"
That said, I don't think that Jev is a magic breakthrough or anything. I think it is just a particularly good narrative with an easy way to try it out.
For what it does, it classifies, orchestrates, operates and delegates tasks exceedingly well for its size and weight. It's ridiculously cheap and efficient, but if you can only see progress in terms of raw cognitive power then you'll surely miss how interesting this is.
It is impressive, but all the hype and fake demos are selling it as a model that is as smart as frontier reasoning LLMs in the decisions it makes yet much cheaper and much faster, which is not true.
I think that misses the point of Jev being ridiculously efficient while maintaining adequate intelligence for automation tasks. We have to train our minds to filter out branding and marketing.
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
There is a lot of room for a lot of different models. For many use cases, intelligence beats out all.
For me in my day job, having extremely fast low quality decision makers over noisy inputs is very valuable. I work in security and having something that can help triage alerts, classify items and group things together is extremely valuable. It doesn't need to be perfect. Just being able to take a set of inputs from deterministic tooling and to be make general priority classifications goes a long way on helping humans look at the most important items first.
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
Super cool to see it do the whole game. I spent my fable budget building something similar this week but only drove it to Brock. I like the "current focus" framing too.
Yeah I would have expected it to only decide which button to press, not something abstract like the choice of "go east to lavender town" for the goal of "in lavender town, climb the pokemon tower"
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
Hm wondering what a first pass optimal setup might be - jev for overworld navigation, escalate to sonnet for easy battles, opus for medium difficulty battles, and fable for gym bosses could probably have jev also manage all the escalation / de-escalation to different models.
It's connected to the ROM of the actual game, so it can see a lot of things. It has multiple tools available, including being able to move to coordinates.
I wish jev took in images so we could do this generically for any game, without memhacks. I'm sure that's coming.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.
This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.
This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking up but probably won't start building around it yet.
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions? We already went through the "look this AI can play pokemon terribly" phase like a decade ago.
The cheap, fast and smart-enough LLM space has been wildly neglected. Jev is one of the few players in the space truly targeting that space. And for a lot of people it is the first time they are asking "what could I build if llms were interaction-speed fast?". The answers are cool, the problem is that Jev is not, I think, smart-enough yet to have that many applications, but it's smart enough that you can start to see what they will look like.
A pre-trained universal classifier that can replace specifically-trained ones would have been considered just as much science fiction in the 2010's as the capabilities of modern LLMs. I'm not sure Jev is actually there yet, but at least it sounds theoretically doable today.
That being said, one thing having been unrealistic 10 years ago and just about possible today doesn't mean that it's going to change the world the same way another technically related, previously-impossible thing did. The Jev hype gives me a bit of the "you're still early to crypto" vibes of some later altcoins. I really like the idea, I think it's going to open up possibilities for using classifiers where we wouldn't or couldn't have trained one before. I'm crossing my fingers for an open weights version to drop. But it's still just a classifier, people have built similar things before Jev, the one thing that really stands out about it is their ability to generate hype.
> but at least it sounds theoretically doable today
why
We have a bad universal classifier now (via Jev). 0->1, one might say.
A bad universal classifier does suggest a good one later. And that is exactly what I would call "theoretically doable"
That said, I don't think that Jev is a magic breakthrough or anything. I think it is just a particularly good narrative with an easy way to try it out.
For what it does, it classifies, orchestrates, operates and delegates tasks exceedingly well for its size and weight. It's ridiculously cheap and efficient, but if you can only see progress in terms of raw cognitive power then you'll surely miss how interesting this is.
It is impressive, but all the hype and fake demos are selling it as a model that is as smart as frontier reasoning LLMs in the decisions it makes yet much cheaper and much faster, which is not true.
Nothing fake here and fully open source if you wanna take a peek. It does make a bunch of mistakes, often. But it eventually recovers!
https://github.com/christianmat/jev-pokemon
I think that misses the point of Jev being ridiculously efficient while maintaining adequate intelligence for automation tasks. We have to train our minds to filter out branding and marketing.
I agree it's overhyped, but the transition to a general purpose classifier (vs a narrow scope classifier) is new and noteworthy.
Ie the famous "Hotdog" clip from Silicon Valley [0]
https://www.youtube.com/watch?v=ACmydtFDTGs
Maybe noteworthy but definitely not new. The category of zero-shot classification has been around for a while.
Example (2022):
https://developers.openai.com/cookbook/examples/zero-shot_cl...
Making decisions quickly, cheaply and without having to train your own model.
Math.random can make poor decisions quickly and cheaply
Benchmark it against jev and you'll have your answer.
I mean,
> get stuck in strange loops of going in and out of the same door to no end
Math.random is statistically unlikely to do this.
The exact message I sent my friend this morning:
> the most interesting thing about this jev stuff
> is that people are seemingly like
> completely disinterested in how smart it actually is
> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheap
After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.
There is a lot of room for a lot of different models. For many use cases, intelligence beats out all.
For me in my day job, having extremely fast low quality decision makers over noisy inputs is very valuable. I work in security and having something that can help triage alerts, classify items and group things together is extremely valuable. It doesn't need to be perfect. Just being able to take a set of inputs from deterministic tooling and to be make general priority classifications goes a long way on helping humans look at the most important items first.
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
then it would cost human lives, no more fun i made this thing with no real effort vibes
Actually, yes, that would be at least interesting
People need to be live streaming their AI more!
Great work!
Super cool to see it do the whole game. I spent my fable budget building something similar this week but only drove it to Brock. I like the "current focus" framing too.
with such a fat harness, this is more like watching a walk thru play the game.
Yeah I would have expected it to only decide which button to press, not something abstract like the choice of "go east to lavender town" for the goal of "in lavender town, climb the pokemon tower"
That was how I first implemented it. Jev sadly never left Pallet Town.
That's actually fun to watch. Did you experiment with nicknaming before you turned it off? I'd be a little curious to see how it behaves.
Jev can't come up with original text, but I did consider giving it a list of hilarious names.
Sure it can, just ask it for next char or "done" in a loop
I just took your advice and added it to the list of Jev decisions. Watch it name its next Pokemon!
True!
Maybe letter by letter spelling?
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
Frontier reasoning models do pretty well in Pokemon: https://github.com/benchflow-ai/pokemon-gym
The interesting thing here imo is the cost and latency. So far we're at 4 badges for less than $0.5
I'm not sure if you're asking about whether using Fable makes playing the game possible or if you're just curious about Jev + LLM interactions.
However, https://x.com/TynanSylvester/status/2096965749369720970 Astra was able to beat RimWorld. So LLMs are definitely able to drive these sorts of games to completion with their current abilities.
Hm wondering what a first pass optimal setup might be - jev for overworld navigation, escalate to sonnet for easy battles, opus for medium difficulty battles, and fable for gym bosses could probably have jev also manage all the escalation / de-escalation to different models.
>but not fast enough to play Doom yet sadly
Did I miss something? I thought one of the demo videos was it doing pretty decent at the first level of Doom?
In my experience it was too slow to do an FPS with 30 ticks per second reliably. It gets killed too fast.
What's not clear to me on the video is whether jev is doing the button presses for controlling the character to move around.
The "Jev calls" counter only seems to increment at junction points like battles, conversation prompts, menus etc.
Is something else moving the character around?
It's connected to the ROM of the actual game, so it can see a lot of things. It has multiple tools available, including being able to move to coordinates.
All in the OSS repo if you wanna play around with it: https://github.com/christianmat/jev-pokemon
Seems like it got stuck on a Ghost enemy.
watched it stuck at rocket hideout for 10 mins.... let me check 1hr later to see if it can find a way out
It’s currently stuck at an elevator and deciding to teach Pokemon various TMs and HMs instead of progressing… pretty hilarious!
it made it through!
Considering it just made Charizard forget its only fire-type move "Ember" to learn "Counter", I note no signs of intelligence.
Rookie mistake clearly..
Is Frigade going to use Jev to do object detection?
Working on it :)
Nice. When I saw your connection to Frigade, I knew there was a connection haha
I wish jev took in images so we could do this generically for any game, without memhacks. I'm sure that's coming.
You could front this with an image -> text model but that would be much lower quality vs latency, and the whole point of doing it with a decision model is remove the latency.
Games are a really interesting testing ground for robotics; if we can solve game playing (incl 3d) we could embody "system one" intelligence into robots that have something emulating general reflexes without needing to fine tune.
Agreed this would be super cool and I do see that coming in the future. But Jev-level latency just isn't there yet with full images.