Well, part of its brain, that is. And right now, it's controlling a simulated Mercedes-Benz on my computer without any training.
Yes. Zero backpropagation. Zero reinforcement learning.
I downloaded the fly's connectome, filtered it down to the motor pathways, and built an artificial retina so it can see the world through a car’s dashcam (just 900 pixels total, black and white!).
The result? It uses just 1,555 FLOPs per frame. Despite the circuit having zero concept of a “car” or a “road” to begin with, it seems to (very primitively) keep a lane, turn, and dodge stuff it’s never explicitly seen before.
Code + writeup + demos at the link (addresses the limitations too, of course!): siddaarth.com/fly-by-wire
Really curious to hear your thoughts and learn what you would have done differently.
This is cool. Are you considering doing a real world experiment? Obviously not with a real car, but maybe something like a small object that the fly neurons can steer through a maze or something?
I was thinking exactly this! Either an RC car, or more interestingly, a drone. I’ve only tested it on simpler obstacles and environments so far, but who knows what more it can do?
You could use a real car too! (As long as low speed / open lot), I just did an experiment like this last weekend, a similar stack (build off of Openpilot/Comma 4) could work with the fly model, will email you some details if you're curious
True RC car is definitely the next logical step :) Yeah as long as you set a low enough speed limit, do it in an empty parking lot, have a human always ready to brake, I think it should be relatively safe haha
Well, part of its brain, that is. And right now, it's controlling a simulated Mercedes-Benz on my computer without any training.
Yes. Zero backpropagation. Zero reinforcement learning.
I downloaded the fly's connectome, filtered it down to the motor pathways, and built an artificial retina so it can see the world through a car’s dashcam (just 900 pixels total, black and white!).
The result? It uses just 1,555 FLOPs per frame. Despite the circuit having zero concept of a “car” or a “road” to begin with, it seems to (very primitively) keep a lane, turn, and dodge stuff it’s never explicitly seen before.
Code + writeup + demos at the link (addresses the limitations too, of course!): siddaarth.com/fly-by-wire
Really curious to hear your thoughts and learn what you would have done differently.
This is cool. Are you considering doing a real world experiment? Obviously not with a real car, but maybe something like a small object that the fly neurons can steer through a maze or something?
I was thinking exactly this! Either an RC car, or more interestingly, a drone. I’ve only tested it on simpler obstacles and environments so far, but who knows what more it can do?
You could use a real car too! (As long as low speed / open lot), I just did an experiment like this last weekend, a similar stack (build off of Openpilot/Comma 4) could work with the fly model, will email you some details if you're curious
Wild that you did this just last weekend. I think with the proper manual overrides this could be a great experiment. After the RC car, I guess :)
True RC car is definitely the next logical step :) Yeah as long as you set a low enough speed limit, do it in an empty parking lot, have a human always ready to brake, I think it should be relatively safe haha