I'm making a game engine that uses rollback netcode for its multiplayer architecture. As far as I can tell, no physics engine supports incremental rollback thus far. This means the entire physics engine state has to be snapshotted every frame, which basically means it's infeasible to have large worlds with rollback netcode. I've made a physics engine which only snapshots the changes, and so now I think you can have large worlds, as long as most of the world is static. I think that's true in most cases, like when you're walking around a big spaceship for example, all the walls, tables, control panels etc don't really move. I wrote up a bit of a post to describe some of the cool things I discovered while making my own physics engine.
This physics engine is a deterministic physics engine. It has to be to make sure that when it rolls back and resimulates forward, it gets the same answer on all machines.
The determinism is partly possible because WebAssembly is deterministic (except for a few known cases https://github.com/WebAssembly/design/blob/main/Nondetermini...), and partly because I’m making sure to use my own trigonometric functions, and the entire game simulation is executed single threaded with a known order of execution.
If you meant to ask how does it compare to non deterministic physics engines, I’m sure they might be faster on the physics but would be slower on the rollback, and I think on most reasonably-sized games the slow rollback would dominate and so they would be slower overall. But, you wouldn’t make a rollback netcode game with that size of world anyway, at least maybe not until now, so it’s a bit of a false comparison. They’re good at their different use cases.
Have you considered the opportunity of using delta compression on snapshots? Like the internal state of the physics simulation, most of the gamestate itself don't change between frames. Using delta compression on the whole structure is doable.
I was also curious about this, and I don't think the other replies understood what was being suggested.
If I understood correctly, the aim of the engine is to lower the in-memory size of the history of game states, by only snapshotting the delta. I'm also curious what would happen if, instead, you'd just run any deterministic snapshottable physics engine, and delta compressed the history on the fly. I think this is how, for example, Braid works.
Might be that it doesn't work, that running the delta check on two big enough snapshots would be too slow, and that's what this engine fixes. But would love to hear if it was considered.
The thing the author is trying to solve for here is reducing the amount of CPU used on the client when it rolls back the simulation and re-simulates to keep server authority.
He does this by only rolling back and re-simulating only a subset of the world, greatly reducing the amount of CPU required. It's cool that he's approaching this from the point of view of adding support for it in the physics engine itself, vs. making it something that the game has to do themselves.
Delta compression is an unrelated technique which reduce the amount of bandwidth sent from server to client, by sending only the differences between the snapshot at baseline frame n and the current snapshot frame m on the server.
Just want to clear this up for anybody trying to follow along. Bringing in delta compression is an unrelated thing (but somewhat similar conceptually). It might confuse people to talk about these things at the same time, if they're really just trying to understand what the author is doing in the article.
No, in this context I meant delta-compression in the case of local roll-back.
The entire gamestate has to be rolled back when using this style of netcode, regardless of bandwidth, reducing the size of snapshots in memory can also reduce make it faster to rebuild.
Actually, it doesn’t roll back the entire game state. Everything is stored in slot maps, and if there are 1000 slots and only 10 changed, it really does only rollback those 10 only and it does not touch anything else. Slot maps are great!
How do you accurately resimulate if you don't rollback everything to the previous state? E.g. a giant wall moved from point A to B but it was accurately predicted (no rollback). Don't you need to roll the wall back to accurately simulate everything around it?
EDIT: I misunderstood the previous comment, I think you are rolling back everything that changed, and not rolling back objects which were "static" in that timeframe.
Yes, sounds like you get it, Giant Wall changed so it gets rolled back to point A, then it resimulates, which might mean Giant Wall might push some other things this time around that it didn't before during the predicted simulation. It kind of uses a transaction log copy-on-write kind of mechanism, so there will be a log entry for previous value of the Giant Wall slot and it will restore that into its old slot. It's a bit more advanced than that but that's the gist.
Yes thank you, you understand perfectly and thank you for explaining. Also, I LOVE your series of blog posts, thank you for making them!
Just to add to the general discussion for everyone following along - rollback netcode only sends inputs around, not state, so it doesn't really have much to send. I think I'm doing about 1.5 KB per second. When you point your mouse it sends that data in 10 bytes. There's not a lot to delta compress.
One cool trick you could try (although you are probably doing it already) is to include all inputs for some long period (like 1-2 seconds!) in every input packet the client sends to the server.
This way if one input packet gets lost, the very next one getting through will have all the inputs for the last 1-2 seconds, and this greatly improves how well your game will play under packet loss.
When you do this, you can even do an encoding from left -> right for all inputs, and actually, sort of delta encode inputs within the packet! Inputs don't change that much, so you can even get smart with the encoding and optimize it down to basically nothing.
No need to resend inputs with seq IDs <= the last one acknowledged by the server, right? 1-2 seconds sounds like overkill. Unless the server updates themselves are lower than 0.5-1 Hz, but what kind of game is that? A very sparse world where chances of misprediction are very low?
I’ve found that sometimes p2p connections will have large lag spikes when communicating across the country (and on WiFi networks, that’s likely the true culprit).
I would see rare bursts of packet delay for ~1 second, that would quickly resolve. In a rollback game where inputs are predicted well, often times this would be unnoticeable.
I send up to 2s worth of input history every frame to handle these lag spikes. I also confirm inputs received, so in practice usually players are only sending a handful of recent unconfirmed inputs (with the 2s buffer available if unconfirmed inputs pile up due to lag).
My guess is their 2s window is for similar reasons, as buffer for rare connection issues. Even if lag spikes are incredibly rare, they need to be handled for a reliable player experience
Ah yes, I have heard of this method but the idea of sorting/delta encoding is new to me and now I’m reconsidering!
Maybe some people might find it interesting - I’m relaying packets from peer-to-peer using Cloudflare Realtime, which is like a one-to-many broadcast system for WebRTC. Each peer sends their input packet to Cloudflare, then Cloudflare forwards that on to the 10 other players (for example). It’s cool because (a) 10x less upload bandwidth from the peer (b) people IP addresses are not revealed to their peers and (c) Cloudflare is in 400 datacenters around the world so it adds minimal latency. Cloudflare Realtime is a really cool system that maybe more web game developers should look into!
Unfortunately, I’m paying for all the bandwidth that goes through Cloudflare Realtime and so I have perhaps over optimised on minimising bandwidth by sending only one input per packet. The other part of my equation is I’m getting my server to broadcast authoritative batches of inputs every 100ms or so via TCP, so if a packet gets lost, every peer will eventually receive the input but it might be a bit slow, and it will cause a big rollback that might be noticeable.
Reading your comment makes me think it might not be as expensive as I thought, and maybe I can play around with how long of an input period I resend for. Perhaps there is a better balance to strike between cost and reliability. So thanks for bringing this up!
If you want to get fancy you can track the last received input frame on the server and send it down to the client, then the client only needs to send inputs since the last asked input frame the server has seen.
This looks great - how much of it is open source, could I host it all myself? wouldn't be interested in investing any time learning if I'm locked in to hosting on some else's hardware.
I would LOVE to one day make a source-available standalone license for Easel that you can run on your own server! I just have a bunch of problems to solve to get there before it’s ready.
1. The people who really love Easel right now are teenagers who have mastered Scratch and are looking for more. Expert game devs who come from Unity or other programming languages find it’s missing some capability they really want and tend to get frustrated right now. I’ve been solving so many issues every month, like we got multiple cameras the other month for example, but there’s so much left. If I made a standalone version now, no one would buy it because it doesn’t yet do what they need, and if you did I would probably be refunding you quite soon. We might need another 6-12 months to be ready for experts. Sign up to the mailing list and we can keep you updated.
2. I would be willing to go as far as source available to licensees but there’s are some practical issues with fully open source. First, it makes it easier to cheat at all Easel games. Easel is client authoritative for certain things. Why? So I can keep the server costs within what I can actually afford because I am one person. Maybe once I can afford 30 servers we can make it all server authoritative and I won’t worry about this anymore. Second, I expect a net decrease in my productivity if I have to argue and defend my code, and become the BDFL to the open internet all the time in an open source environment. I’ve been really really productive just working by myself.
As a long-time web/app developer getting into game dev, it feels like I'm entering "the big leagues" of software engineering. Tougher problems, more problems, more _interesting_ problems, and problems without prebuilt solutions. Much more fun than making yet another dashboard.
Combining player control, multiplayer, non-player control, and physics is one of the tougher problems. I got it handled (enough) for my project, but I'd be very interested to read the source if Easel's physics engine gets open-sourced.
I’m in the same boat. Been learning 3d game dev past few years seriously after dabbling for half a decade. I even released my first few tiny 3d games in the last 6 months and they’ve made hundreds of dollars! That alone was a dream come true.
I picked game dev specifically because I wanted to build some things I had envisioned and found it challenging. And in the beginning each new concept within 3d modeling, optimisation, shaders, physics, lighting, shadows & rendering felt intractable and unmasterable.
Now, I have a basic working understanding of nearly everything that goes into traditional 3d game dev. Except the very cutting edge stuff. And have mastered things I was struggling with 2 years ago.
And recently I felt something that scared me. It was the feeling that within 2 years, I’ll have lost the excitement and challenges that learning game dev has brought me with these past few years. And I could see how to someone experienced this was as boring as full stack web dev was for me.
Fwiw, one case where I've wanted rollback has been input fusion over interface devices with diverse latencies. You might have 10 ms for a keypress, 100 ms for optical tracking, and 1000 ms for speech. So given click+"the red one"(spoken), you might start running click+"the one in front"(default), and almost a second later rollback and rerun with "the red one". Or for real example, keypress event handling might branch on optical "pressed where on the keycap" and "by which finger", which won't become available for several frames.
Let me know when you have enough demand to make it multiple full-time jobs. I’ve been making notes for a few years now about all the best patterns and principles for designing complex systems and your language + engine more or less hits all the right notes.
Declarative state and reactivity, lexical lifetimes and ownership, etc.
Really curious how you set it all up and what prior art was your primary inspiration.
Wow, I’m amazed someone noticed this. I’d be interested to know more about your background and interests. How much have you been looking at programming language design vs just designing for complex systems in general?
I ask because yes, I have put in a great deal of time and effort into the programming language design, and to me, I think that is the greater achievement, more than the automatic multiplayer. But the benefits of automatic multiplayer are easy for the general population to understand and the improvements programming language design is hard to convey and so people don’t normally get it. The fact that you can see what I’ve been trying to do so quickly shows you must be coming from a place which has developed that discernment for you.
While there are many inspirations, like I love coding using React for example, my primary inspiration is the last game I made, I released all the modding tools along with them and lots of non-coders loved it. The modding was JSON, which might sound primitive, but it was actually a hierarchical declarative domain specific language and it seemed to really work for people intuitively.
Easel was born from me spending 2 years trying to make an imperative programming language in a similar shape as that declarative one. I wanted it to be just as easy, but infinitely more powerful. It took a lot of iteration to merge the declarative and imperative styles into one language. There is so much to it - lexical lifetimes and ownership, reactivity like you noticed, but also weaving in concurrency and asynchronous programming seamlessly took time as well.
I really wish this is more the part of Easel that would stand out and be talked about more because I think it’s the coolest part.
The export basically creates a page with an HTML IFRAME in it that embeds the hosted version of your game on easel.games so that all the multiplayer and leaderboards continue to work.
What you're describing isn't uploading the game, then, but uploading a stub with a transclusion of the game. I'm not the same commenter, but surely that doesn't answer (what I see as) most of the implications of the original question?
Okay, in that case you can upload a stub with a transclusion of the game. If there is enough demand I would definitely work on a standalone export system!
Vendor lock-in with a monthly subscription where the price can change at any moment or the company may disappear along with your product. I wouldn't build my business on that.
I would love to make a license where you could export and then you can run it forever yourself on your own server! It’s not because I don’t want to do these things, it’s just I’m one person and I never get the time to do everything I would like to do.
Hi everyone,
I'm making a game engine that uses rollback netcode for its multiplayer architecture. As far as I can tell, no physics engine supports incremental rollback thus far. This means the entire physics engine state has to be snapshotted every frame, which basically means it's infeasible to have large worlds with rollback netcode. I've made a physics engine which only snapshots the changes, and so now I think you can have large worlds, as long as most of the world is static. I think that's true in most cases, like when you're walking around a big spaceship for example, all the walls, tables, control panels etc don't really move. I wrote up a bit of a post to describe some of the cool things I discovered while making my own physics engine.
How does it compare with deterministic physics engines, given that their appeal for multiplayer is that they can perform rollback?
This physics engine is a deterministic physics engine. It has to be to make sure that when it rolls back and resimulates forward, it gets the same answer on all machines.
The determinism is partly possible because WebAssembly is deterministic (except for a few known cases https://github.com/WebAssembly/design/blob/main/Nondetermini...), and partly because I’m making sure to use my own trigonometric functions, and the entire game simulation is executed single threaded with a known order of execution.
If you meant to ask how does it compare to non deterministic physics engines, I’m sure they might be faster on the physics but would be slower on the rollback, and I think on most reasonably-sized games the slow rollback would dominate and so they would be slower overall. But, you wouldn’t make a rollback netcode game with that size of world anyway, at least maybe not until now, so it’s a bit of a false comparison. They’re good at their different use cases.
How big is the state that you want to rollback?
Have you considered the opportunity of using delta compression on snapshots? Like the internal state of the physics simulation, most of the gamestate itself don't change between frames. Using delta compression on the whole structure is doable.
I was also curious about this, and I don't think the other replies understood what was being suggested.
If I understood correctly, the aim of the engine is to lower the in-memory size of the history of game states, by only snapshotting the delta. I'm also curious what would happen if, instead, you'd just run any deterministic snapshottable physics engine, and delta compressed the history on the fly. I think this is how, for example, Braid works.
Might be that it doesn't work, that running the delta check on two big enough snapshots would be too slow, and that's what this engine fixes. But would love to hear if it was considered.
The thing the author is trying to solve for here is reducing the amount of CPU used on the client when it rolls back the simulation and re-simulates to keep server authority.
He does this by only rolling back and re-simulating only a subset of the world, greatly reducing the amount of CPU required. It's cool that he's approaching this from the point of view of adding support for it in the physics engine itself, vs. making it something that the game has to do themselves.
Delta compression is an unrelated technique which reduce the amount of bandwidth sent from server to client, by sending only the differences between the snapshot at baseline frame n and the current snapshot frame m on the server.
Just want to clear this up for anybody trying to follow along. Bringing in delta compression is an unrelated thing (but somewhat similar conceptually). It might confuse people to talk about these things at the same time, if they're really just trying to understand what the author is doing in the article.
cheers
- Glenn
No, in this context I meant delta-compression in the case of local roll-back.
The entire gamestate has to be rolled back when using this style of netcode, regardless of bandwidth, reducing the size of snapshots in memory can also reduce make it faster to rebuild.
Actually, it doesn’t roll back the entire game state. Everything is stored in slot maps, and if there are 1000 slots and only 10 changed, it really does only rollback those 10 only and it does not touch anything else. Slot maps are great!
How do you accurately resimulate if you don't rollback everything to the previous state? E.g. a giant wall moved from point A to B but it was accurately predicted (no rollback). Don't you need to roll the wall back to accurately simulate everything around it?
EDIT: I misunderstood the previous comment, I think you are rolling back everything that changed, and not rolling back objects which were "static" in that timeframe.
Yes, sounds like you get it, Giant Wall changed so it gets rolled back to point A, then it resimulates, which might mean Giant Wall might push some other things this time around that it didn't before during the predicted simulation. It kind of uses a transaction log copy-on-write kind of mechanism, so there will be a log entry for previous value of the Giant Wall slot and it will restore that into its old slot. It's a bit more advanced than that but that's the gist.
Yes thank you, you understand perfectly and thank you for explaining. Also, I LOVE your series of blog posts, thank you for making them!
Just to add to the general discussion for everyone following along - rollback netcode only sends inputs around, not state, so it doesn't really have much to send. I think I'm doing about 1.5 KB per second. When you point your mouse it sends that data in 10 bytes. There's not a lot to delta compress.
One cool trick you could try (although you are probably doing it already) is to include all inputs for some long period (like 1-2 seconds!) in every input packet the client sends to the server.
This way if one input packet gets lost, the very next one getting through will have all the inputs for the last 1-2 seconds, and this greatly improves how well your game will play under packet loss.
When you do this, you can even do an encoding from left -> right for all inputs, and actually, sort of delta encode inputs within the packet! Inputs don't change that much, so you can even get smart with the encoding and optimize it down to basically nothing.
No need to resend inputs with seq IDs <= the last one acknowledged by the server, right? 1-2 seconds sounds like overkill. Unless the server updates themselves are lower than 0.5-1 Hz, but what kind of game is that? A very sparse world where chances of misprediction are very low?
I’ve found that sometimes p2p connections will have large lag spikes when communicating across the country (and on WiFi networks, that’s likely the true culprit).
I would see rare bursts of packet delay for ~1 second, that would quickly resolve. In a rollback game where inputs are predicted well, often times this would be unnoticeable.
I send up to 2s worth of input history every frame to handle these lag spikes. I also confirm inputs received, so in practice usually players are only sending a handful of recent unconfirmed inputs (with the 2s buffer available if unconfirmed inputs pile up due to lag).
My guess is their 2s window is for similar reasons, as buffer for rare connection issues. Even if lag spikes are incredibly rare, they need to be handled for a reliable player experience
Ah yes, I have heard of this method but the idea of sorting/delta encoding is new to me and now I’m reconsidering!
Maybe some people might find it interesting - I’m relaying packets from peer-to-peer using Cloudflare Realtime, which is like a one-to-many broadcast system for WebRTC. Each peer sends their input packet to Cloudflare, then Cloudflare forwards that on to the 10 other players (for example). It’s cool because (a) 10x less upload bandwidth from the peer (b) people IP addresses are not revealed to their peers and (c) Cloudflare is in 400 datacenters around the world so it adds minimal latency. Cloudflare Realtime is a really cool system that maybe more web game developers should look into!
Unfortunately, I’m paying for all the bandwidth that goes through Cloudflare Realtime and so I have perhaps over optimised on minimising bandwidth by sending only one input per packet. The other part of my equation is I’m getting my server to broadcast authoritative batches of inputs every 100ms or so via TCP, so if a packet gets lost, every peer will eventually receive the input but it might be a bit slow, and it will cause a big rollback that might be noticeable.
Reading your comment makes me think it might not be as expensive as I thought, and maybe I can play around with how long of an input period I resend for. Perhaps there is a better balance to strike between cost and reliability. So thanks for bringing this up!
If you want to get fancy you can track the last received input frame on the server and send it down to the client, then the client only needs to send inputs since the last asked input frame the server has seen.
I think this is a cool idea. Well done!
Thank you so much!
This looks great - how much of it is open source, could I host it all myself? wouldn't be interested in investing any time learning if I'm locked in to hosting on some else's hardware.
I would LOVE to one day make a source-available standalone license for Easel that you can run on your own server! I just have a bunch of problems to solve to get there before it’s ready.
1. The people who really love Easel right now are teenagers who have mastered Scratch and are looking for more. Expert game devs who come from Unity or other programming languages find it’s missing some capability they really want and tend to get frustrated right now. I’ve been solving so many issues every month, like we got multiple cameras the other month for example, but there’s so much left. If I made a standalone version now, no one would buy it because it doesn’t yet do what they need, and if you did I would probably be refunding you quite soon. We might need another 6-12 months to be ready for experts. Sign up to the mailing list and we can keep you updated.
2. I would be willing to go as far as source available to licensees but there’s are some practical issues with fully open source. First, it makes it easier to cheat at all Easel games. Easel is client authoritative for certain things. Why? So I can keep the server costs within what I can actually afford because I am one person. Maybe once I can afford 30 servers we can make it all server authoritative and I won’t worry about this anymore. Second, I expect a net decrease in my productivity if I have to argue and defend my code, and become the BDFL to the open internet all the time in an open source environment. I’ve been really really productive just working by myself.
As a long-time web/app developer getting into game dev, it feels like I'm entering "the big leagues" of software engineering. Tougher problems, more problems, more _interesting_ problems, and problems without prebuilt solutions. Much more fun than making yet another dashboard.
Combining player control, multiplayer, non-player control, and physics is one of the tougher problems. I got it handled (enough) for my project, but I'd be very interested to read the source if Easel's physics engine gets open-sourced.
I’m in the same boat. Been learning 3d game dev past few years seriously after dabbling for half a decade. I even released my first few tiny 3d games in the last 6 months and they’ve made hundreds of dollars! That alone was a dream come true.
I picked game dev specifically because I wanted to build some things I had envisioned and found it challenging. And in the beginning each new concept within 3d modeling, optimisation, shaders, physics, lighting, shadows & rendering felt intractable and unmasterable.
Now, I have a basic working understanding of nearly everything that goes into traditional 3d game dev. Except the very cutting edge stuff. And have mastered things I was struggling with 2 years ago.
And recently I felt something that scared me. It was the feeling that within 2 years, I’ll have lost the excitement and challenges that learning game dev has brought me with these past few years. And I could see how to someone experienced this was as boring as full stack web dev was for me.
Fwiw, one case where I've wanted rollback has been input fusion over interface devices with diverse latencies. You might have 10 ms for a keypress, 100 ms for optical tracking, and 1000 ms for speech. So given click+"the red one"(spoken), you might start running click+"the one in front"(default), and almost a second later rollback and rerun with "the red one". Or for real example, keypress event handling might branch on optical "pressed where on the keycap" and "by which finger", which won't become available for several frames.
That is a cool idea! Could be a great application of rollback!
Is this a demo? A hobby? How long have you been at this?
I’ve been working on this for four years. I’m trying to make this my full time job!
Let me know when you have enough demand to make it multiple full-time jobs. I’ve been making notes for a few years now about all the best patterns and principles for designing complex systems and your language + engine more or less hits all the right notes.
Declarative state and reactivity, lexical lifetimes and ownership, etc. Really curious how you set it all up and what prior art was your primary inspiration.
Wow, I’m amazed someone noticed this. I’d be interested to know more about your background and interests. How much have you been looking at programming language design vs just designing for complex systems in general?
I ask because yes, I have put in a great deal of time and effort into the programming language design, and to me, I think that is the greater achievement, more than the automatic multiplayer. But the benefits of automatic multiplayer are easy for the general population to understand and the improvements programming language design is hard to convey and so people don’t normally get it. The fact that you can see what I’ve been trying to do so quickly shows you must be coming from a place which has developed that discernment for you.
While there are many inspirations, like I love coding using React for example, my primary inspiration is the last game I made, I released all the modding tools along with them and lots of non-coders loved it. The modding was JSON, which might sound primitive, but it was actually a hierarchical declarative domain specific language and it seemed to really work for people intuitively.
Easel was born from me spending 2 years trying to make an imperative programming language in a similar shape as that declarative one. I wanted it to be just as easy, but infinitely more powerful. It took a lot of iteration to merge the declarative and imperative styles into one language. There is so much to it - lexical lifetimes and ownership, reactivity like you noticed, but also weaving in concurrency and asynchronous programming seamlessly took time as well.
I really wish this is more the part of Easel that would stand out and be talked about more because I think it’s the coolest part.
Does it run Doom?
Can the games only be played on the easel site, or can I upload to e.g. itch.io as well (probably with the limitation of only singleplayer?)
You can upload to itch.io yes: https://easel.games/docs/learn/publishing/export
The export basically creates a page with an HTML IFRAME in it that embeds the hosted version of your game on easel.games so that all the multiplayer and leaderboards continue to work.
Thanks for your interest!
What you're describing isn't uploading the game, then, but uploading a stub with a transclusion of the game. I'm not the same commenter, but surely that doesn't answer (what I see as) most of the implications of the original question?
Okay, in that case you can upload a stub with a transclusion of the game. If there is enough demand I would definitely work on a standalone export system!
Vendor lock-in with a monthly subscription where the price can change at any moment or the company may disappear along with your product. I wouldn't build my business on that.
I would love to make a license where you could export and then you can run it forever yourself on your own server! It’s not because I don’t want to do these things, it’s just I’m one person and I never get the time to do everything I would like to do.