Not with any amount of temporal cohesion. This is using neural rendering that can stay consistent across arbitrary lights and their movement. But the authors chose to go proprietary, hopefully we’ll get some open alternatives in the near future.
This isn’t my work - just sharing the approach. I agree this should have been an open and reproducible publication. It won’t be long for this to become a normal feature in neural rendering engines though, there’s (IMO) no breakthrough novel idea in this pipeline.
I would like to implement this feature in software.
Now I cannot legally do so. Not even in free software.
We have to wait 20 years for this shit to expire. And by that point half of the people here will be retired and technology will have moved on, but this one important technique and feature will still be locked away.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
I would do it illegally right now if I had the hardware resources. The article is 3 extremely vague pages but from the photo description it looks pretty simple.
You make a diffuse light only version of the image with one trained NN, estimate a depth map from it, take the delta from diffuse to original image and reverse raytrace it with Mitsuba to get the environment map, use differentiable rendering to optimize the environment map and make a new point light map.
You're left with a 3D environment where you can move the lights around and call Mitsuba again to get a physics based render. That's the whole innovation. I guess the secret sauce is in the optimization algorithm, but it could just be basic gradient descent.
Patents don't restrict you from writing or publishing software. They might restrict you from compiling and running it, but in practice they're irrelevant for free software where you're distributing source anyway.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
Only if you think that progress comes from divine inspiration in the ineffable depths of the minds of great men, as opposed to smart people applying themselves to problems using the tools they have available. A lot of new ideas aren't so much someone figuring out how to do something, as someone deciding that something is worth doing. Amazon's infamous patent on "1-click purchasing" wasn't some kind of flying car that the world would have been forever robbed of, had Jeff Bezos not been incentivized to disclose his monumental trade secret.
> figuring out how to do something, as someone deciding that something is worth doing.
I do agree that there's a logical causal chain to any idea, and that causal graph could maybe be considered as some pre-existing "truth" built grown from all the rules/constraints that exits. But, I don't think being able to explore those "undiscovered" vertices is completely about "worth doing". If you get a chance to work in a field of experts, you'll see that some are significantly better at predicting/finding those vertices. More than that, you'll find that ideas will be proposed, but their fellow, collective, experts will claim it will not work. Either way, they see the path clearer than anyone else, and put in the effort for it. To me, and the patent system, that effort is worth something and deserves to be protected. The alternative is they put in the effort and everyone just copies their work, so less motivation to put in that, sometimes expensive, effort.
Or outside imagery: 3D printing, only took off after the patents expired. Which is now a huge industry. What could have been would that patent or with it expiring earlier?
Were you going to do in your basement all the computation necessary for the research? This is something that can't reasonably be accomplished by a hobbyist on their own. If it's going to be done, it's got to be financed by some larger entity.
Do you want that to be publicly-financed, or owned by some corporation with deep pockets?
That's pretty close to magick. Kudos!
Am I mistaken in thinking this was already doable with diffusion models?
Not with any amount of temporal cohesion. This is using neural rendering that can stay consistent across arbitrary lights and their movement. But the authors chose to go proprietary, hopefully we’ll get some open alternatives in the near future.
Neat! And the video was pretty interesting!
Wait ... patent pending? Research done with public money? Yuck! Downvoted immediately.
This isn’t my work - just sharing the approach. I agree this should have been an open and reproducible publication. It won’t be long for this to become a normal feature in neural rendering engines though, there’s (IMO) no breakthrough novel idea in this pipeline.
> patent pending
Ugh, more reason to want AI to automate research. No more patents.
This is going to be used to shake down teams that had no idea these folks happened to do this off in their own corner of the world.
> Ugh, more reason to want AI to automate research. No more patents.
Why would AI research be any less patented than non-AI research?
It can’t be copy writable.
Could you expand on that?
I would like to implement this feature in software.
Now I cannot legally do so. Not even in free software.
We have to wait 20 years for this shit to expire. And by that point half of the people here will be retired and technology will have moved on, but this one important technique and feature will still be locked away.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
I would do it illegally right now if I had the hardware resources. The article is 3 extremely vague pages but from the photo description it looks pretty simple.
You make a diffuse light only version of the image with one trained NN, estimate a depth map from it, take the delta from diffuse to original image and reverse raytrace it with Mitsuba to get the environment map, use differentiable rendering to optimize the environment map and make a new point light map.
You're left with a 3D environment where you can move the lights around and call Mitsuba again to get a physics based render. That's the whole innovation. I guess the secret sauce is in the optimization algorithm, but it could just be basic gradient descent.
Patents don't restrict you from writing or publishing software. They might restrict you from compiling and running it, but in practice they're irrelevant for free software where you're distributing source anyway.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
Isn't the alternative that everything stays secret/proprietary, forever?
Only if you think that progress comes from divine inspiration in the ineffable depths of the minds of great men, as opposed to smart people applying themselves to problems using the tools they have available. A lot of new ideas aren't so much someone figuring out how to do something, as someone deciding that something is worth doing. Amazon's infamous patent on "1-click purchasing" wasn't some kind of flying car that the world would have been forever robbed of, had Jeff Bezos not been incentivized to disclose his monumental trade secret.
tl;dr NO
> figuring out how to do something, as someone deciding that something is worth doing.
I do agree that there's a logical causal chain to any idea, and that causal graph could maybe be considered as some pre-existing "truth" built grown from all the rules/constraints that exits. But, I don't think being able to explore those "undiscovered" vertices is completely about "worth doing". If you get a chance to work in a field of experts, you'll see that some are significantly better at predicting/finding those vertices. More than that, you'll find that ideas will be proposed, but their fellow, collective, experts will claim it will not work. Either way, they see the path clearer than anyone else, and put in the effort for it. To me, and the patent system, that effort is worth something and deserves to be protected. The alternative is they put in the effort and everyone just copies their work, so less motivation to put in that, sometimes expensive, effort.
Or outside imagery: 3D printing, only took off after the patents expired. Which is now a huge industry. What could have been would that patent or with it expiring earlier?
Were you going to do in your basement all the computation necessary for the research? This is something that can't reasonably be accomplished by a hobbyist on their own. If it's going to be done, it's got to be financed by some larger entity.
Do you want that to be publicly-financed, or owned by some corporation with deep pockets?