One of the things that impressed me in Quake (the first one) was the demo recording system. The system was deterministic enough that it could record your inputs/the game state and just play them back to get a gameplay video. Especially given that Quake had state of the art graphics at the time, and video playback on computers otherwise was a low-res, resource intensive affair at the time, it was way cool.
It always surprised me how few games had that feature - though a few important ones, like StarCraft, did - and it only became rarer over the years.
It wasn't really that much to do with determinism. Quake uses a client-server network model all the time, even when you're only playing a local single-player game. What the demo recording system does is capture all of the network packets that are being sent from the server to the client. When playing back a demo, all the game has to do is run a client and replay the packets that it originally received from the server. It's a very elegant system that naturally flows out of the rather forward-looking decision to build the entire engine around a robust networking model.
I don't see why it makes a difference for this purpose that you're replaying network packets or controller inputs or any other interface to the game engine.
The important thing is that there is some well-defined interface. I guess designing for networked multiplayer does probably necessitate that, but if the engine isn't deterministic it still isn't going to work.
There was a twitter thread years ago (which appears to be long gone) about how the SNES Pilot Wings pre-game demo was just a recording of controller inputs. For cartridges manufactured later in the game's life, a plane in the demo crashes rather than landing gracefully, due to a revised version of a chip in the cartridge. The inputs for the demo were never re-recorded, so the behaviour was off.
It does make quite a big difference. The network packets received from the server in Quake will tell you exactly what state the game is in at any point in time. They contain information about the position and state of every entity and their motion, compressed via delta encoding. That means there's very little room for misinterpretation on the client side that would lead to de-sync issues. In fact clients have quite a lot of freedom in how they want to represent said game state, and can for example add animation interpolation to smoothen things out.
The example you mention of demo playback de-syncing when the circumstances slightly change, that is exactly what you get when you only record inputs from the player. Doom actually did this too for its networking model and demo playback system. That relies much more on the engine being deterministic and the runtime environment behaving consistently, because each client that replays those inputs has to run the exact same game simulation, in order for the resulting game states to match.
Demo files work, but I'm talking about spectating live. The "Watch" tab was removed and the ability to just browse and spectate the top games currently being played.
I'm sure the technology still exists in the engine, but it's no longer the key feature it once was. HLTV/GOTV was launched with some fanfare back in the day.
Spectators don't watch the game on the same server that's hosting the game. The host server sends the traffic to a 'relay' on a delay, which spectators then connect to. Similarly for the HTTP streamed games, the game server is writing the data for spectators on a delay.
That's not the kind of nondeterminism that would cause replay divergence. The PRNG seed is stored in the replay (if it wasn't, almost every game would diverge very quickly. And since the multiplayer works the same basic way, the game would basically not function at all).
I'm pretty sure it's because it's in fact 'just' a cool side effect to a common network architecture optimisation from the time where you could'nt send the 'state' of the entire game even with only delta modifiers and so you make the game detertministic to only synchronize inputs :) an exemple article I remember : https://www.gamedeveloper.com/programming/1500-archers-on-a-...
The main downside which probably caused the diseapearance is that any patch to the game will make the replay file unusable.
Also at the time (not sure for quake) there was often fixed framerate, today the upsides of using delta time based frame calculation AND multithreading/multi platform target probably make it harded to stay deterministic (specialy for game where you want to optimize input latency)
I think it's more the patching thing that made "collect and replay inputs" less common.
Networked games have a "tickrate", just for the networking/state aspect. For example, Counter-Strike 2 has a 64Hz tickrate by default. They also typically have a fixed time interval for physics engines. Both of these should be completely independent of framerate, because that's jittery and unpredictable.
Fun fact, overwatch must have done a similar things because they would let you play back games up until some release when you could no longer replay them unless you'd saved the render.
I think if I remember right there were also funny moments where things didn't look right after patches?
If memory serves well, that worked by replaying network packets, which is what some other games do as well, the problem with that approach is that for live service games unlike old games that were often "set in stone", the protocol always changes, so it's a huge maintenance burden. You either need to add conversion tools, keep maintaining backwards compatibility with older protocol versions, or you accept that replays quickly become outdated.
Checking in as a random indie developer who still prioritises determinism in my engine. I don't understand why so many games/engines sacrifice it when it has so much utility.
I think if it were as simple as "remember the RNG seed", game developers would do it every time. But determinism also means, for instance, running the physics engine at a deterministic timestep regardless of the frame rate, to avoid differences in accumulated error or collision detection. And that's something that needs designing in from day one.
> running the physics engine at a deterministic timestep
As well as using special library versions of floating-point functions which don't behave the same across different processors I suppose, if you want to be safe.
Eg cr-libm[1] or more modern alternatives like core-math[2].
It's desirable for arcade games, which have fixed hardware including the display. There's no possibility of upgrading for better framerate, and the game can be designed so slowdown is rare or non-existent. Tying the physics to the framerate gives you very low and very consistent input latency with minimum developer effort.
Determinism isn't essential to achieve record/playback experiences. You could just record the transform of the player at some fixed interval and then replay it with interpolation. This is arguably more "deterministic" in a strategic sense of shipping a viable product.
The player is just one entity, you'd need to do the same to any other non-trivial entity. And you couldn't use fixed intervals and naive interpolation, otherwise you'd have entities clipping the ground when bouncing etc.
Bungies Marathon series (1994) had the same recording system, as other commenters mentioned due to networking multiplayer.
What's totally insane is that the modern engine rewrite Aleph One can also play back such old recordings, for M2 Durandal (1995) and Infinity (1996) at least.
Be nice if they fixed whatever bug they have on the Switch 2 that means every replay I previously downloaded[0] is unplayable and further that every time I now try and download a replay, the game crashes.
But then it'd also be nice if they fixed the "game crashes randomly when joining games" bug too.
(To give them credit, it doesn't now take 5 minutes after waking the Switch 2 before Rocket League reconnects me to the Epic servers like it did a couple of months ago...)
[0] Also the stupidly low limit on how many you can download - it's my storage cost, not yours, wtf.
Replays are very common in fighting games as well, rollback netcode gets you most of the way to a replay system already (replaying game state from inputs is a core requirement for online play)
I worked on this for a pretty big game. We recorded the network traffic and played it back and simulated the game - so same problem with patches. It also has the awkward side effect of exposing a metric crap ton of “join in progress” style bugs because our game didn’t support JiP.
Quake1 was my first love. From the old DOS version to the GLQuake to grappling hooks on Trinicom. I was amazed not only by said demo system but by QuakeC, and how heavily it was used, especially in college communities. I remember MIT and UWisc both being unreasonably productive modders in said language.
As a kid, I couldn't wait to see what came next. Sadly, Q1 was rather one of a kind, and it was many years until anything else like it showed up.
I had a puzzle game were all of the solutions it would show were playbacks of my keypresses as I solved it myself. As the puzzles got more difficult it got harder and harder to record a solution without having pauses to think about what to do next.
I would expect pausing to bring a game’s CPU/GPU usage down to near-zero, which won’t happen if the game keeps redundantly rendering the exact same frame. A game engine can optimize this by special casing time scale zero to simply render a single textured quad under the pause menu (which is probably what one of the commenters in TFA referred to).
The strangest pause bug I know is in Mario Sunshine: pausing will misalign the collision logic (which runs 4 times per frame) and the main game loop. So certain specific physics interactions will behave differently depending on how many times the game has been paused modulo 4.
Against the Storm (and excellent rouguelite city-builder) does this in a really cool way. Pausing is a core mechanic of the game, and you frequently pause while you place building or things like that - and all the visual animations stop (fire, rain, trees swaying, etc).
But when you find a broken ancient seal in the forest, the giant creepy eyeball moving around in it keeps moving even when you pause the game, which helps emphasise how other-worldly it is.
I find it confusing: for me a clear indicator the game is paused is all animations also pausing. Some games do not pause in menu’s, for example. And some do, but not when in a multiplayer session
Torch flames and trees swaying in the wind do not affect gameplay at all - they're most likely done in a shader and I think it's easier to keep updating a time uniform than to add extra conditions everywhere :D
That's usually because the system that runs those things is independent of the timing of the main game loop. And then when someone finally gets around to implementing the pause screen, they still run even with the main game time stopped. And you look at it and think "eh, you know what - looks cool - we'll leave it".
You still have to be careful about how you implement "gameplay", though. For example if at any point you read the 'system clock' to do time-based stuff like animations or physics, then when you unpause you suddenly will have a couple minutes of advance in a place where you expect fractions of a second.
While the game is paused, if a player were to click on the "level up" buttons for their skills, each click actually advanced the game by 1 frame - so it was possible for people to die etc. during a pause screen.
I only know pausing games is funky because the highest my playstation fans ever go is pausing some games. Quite weird pausing is not just a feature of the game engine or runtime, especially as the menu and settings system seem to be totally separate in most cases anyways.
When I present TLA+ [0], I am referencing game pauses (pause buffer / item duplication Legend of Zelda exploit, Dark Souls menuing to cancel animations) and deliberate crashes as mechanics to exploit games, as those are still valid actions that can be modeled, and not doing that allows such exploits to happen.
A system is only correct relative to the transition system you wrote down. If the real system admits extra transitions that you care about (pause, crash, re-entry, partial commits), and you didn't model them, then you proved correctness of the wrong system.
Early versions of Unreal Engine had these animated procedural textures that would produce sparks, fire effects, etc. The odd part is that when you paused the game, the animated textures would still animate. Presumably, the game would pause its physics engine or set the timestep to 0, but the texture updater didn't pause. I suspect it was part of the core render loop and each new iteration of the texture was some sort of filtered version of the previous frame's texture. Arguably a very early version of GPU physics.
Modern games can have the same issue. Even taking a capture of the exact graphics commands and repeating them, you'll sometimes see animated physics effects like smoke and raindrops. They're doing the work on the GPU where it's not necessarily tied to any traditional physics timestep.
When I first played the NES the pause feature impressed me even more than did the graphics. Apparently Atari already had the feature on the 5200 console, but even as late as 1988 it felt like magic to hit a button, go and eat dinner, and an hour later resume my game with another press of the button.
> when it was time to ship, we’d read the [Technical Requirements Checklists] and have to go back and add a special pause for when you unplug the controller
article confirms my early theory I formed when reading the title about why would pause be complicated
I was suspicious of those random game developers getting quoted, and this is the pinned post of the one giving this cute silly story about slowing down game speed time:
"Announcing TORMENT HEXUS, a match-3 roguelike about putting technofascist CEOs on the wrong side of skyscraper windows!
[...]
And remember: they SHOULD be afraid of us.
#indiedev #indiegame"
Eh, I remember the myriad of both "Torture Bin Laden" as well as "Torture George Bush" flash games in the early/mid 2000s. I think it's very on brand for indie developers.
I find the notion odd that this is even a problem to be solved.
It suggests a level of control way below what I would ordinarily consider required for game development.
I have made maybe around 50 games, and I think the level of control of time has only ever gone up. Starting at move one step when I say, to move a non-integer amount when I say, to (when network stuff comes into play) return to time X and then move forward y amount.
The console cert ones are interesting but all the others are just Unity/Gamemaker/Unreal not allowing the developers to write normal code? The nonzero timestep thing is very strange
Games are multithreaded and have loads of objects running everywhere. If you're using anything that's not a custom game engine, there really isn't a single main() function that you can plop an if statement like that into.
Slowing down time applies it universally. Otherwise you're going to need that condition to every single object in the game.
The slowing down thing sounds like a hack needed for engines that don’t give you control over the main loop.
I haven’t tried this yet, but for a custom engine I would introduce a second delta time that is set to 0 in the paused state. Multiplying with the paused-dt „bakes in“ the pause without having to sprinkle ifs everywhere. Multiplying with the conventional dt makes the thing happen even when paused (debug camera, UI animations).
Unity does this. You get scaledDeltaTime (when you set game speed to 0, it'll return 0) and unscaledDeltaTime (returns time between frames ignoring game speed). Pauseable logic uses the former. Pause menus use the latter.
It depends on how your timers are implemented. If they are implemented as a "rendez-vous" absolute date (as in UTC for instance - not in local time unless you want to add "eastern eggs" related to daylight saving time...), this will cause bugs. If you implement your own in-game monotonic clock that you can stop, it should be ok.
>If they are implemented as a "rendez-vous" absolute date
Do people actually do that? What's the plan for when the user sleeps their machine? All the events just inexplicably happen all at once when they wake it?
I've implemented timers that had timeouts using unix timestamps, but only for multiplayer - when a player's attempt to connect to the server times out, etc.
Inside the game loop, we would keep the global tick counter that incremented on every tick, and timeouts would be based on that rather than on UTC.
The tick counter was updated only when the game logic was actually running. Our approach to pausing was to not run the functions that handled frame updates or physics updates, and to only run the rendering functions.
Generally we would never care about actual world time other than for some timeouts like for network (as the time passes for everyone), or for easter eggs like changing the tree models for Christmas or so.
I don't think anyone serious would implement event timers based on real time.
For my game (custom engine) I had a way to stop the game clock from advancing, while the input and draw loop kept running. It would also put the game into the "pause" input state during which only the resume button would be active.
Live migration boils down to copy memory over the network, stream the page faults till you converge enough, and resume execution on the other host. It’s not a hard problem but a precise and tedious one.
Pausing a game might involve a lot of GPU contexts to freeze, network resources to pause, storage streams to pause, input handling, sound, etc. Add to that physics engine that may be tied deeply in the system and you end up with a hard problem.
What a VM does is not the role of the hypervisor, thus it can apply its hammer that works in pretty much all cases, and VMs are pretty much all the same. On the other hand, all games are bespoke with custom plugins and custom integrations, which make them the opposite of "generic pause implementation".
It always surprised me how few games had that feature - though a few important ones, like StarCraft, did - and it only became rarer over the years.
There was a twitter thread years ago (which appears to be long gone) about how the SNES Pilot Wings pre-game demo was just a recording of controller inputs. For cartridges manufactured later in the game's life, a plane in the demo crashes rather than landing gracefully, due to a revised version of a chip in the cartridge. The inputs for the demo were never re-recorded, so the behaviour was off.
The example you mention of demo playback de-syncing when the circumstances slightly change, that is exactly what you get when you only record inputs from the player. Doom actually did this too for its networking model and demo playback system. That relies much more on the engine being deterministic and the runtime environment behaving consistently, because each client that replays those inputs has to run the exact same game simulation, in order for the resulting game states to match.
This used to be a promoted feature in CS, with "HLTV/GOTV", but sadly disappeared when they moved to CS2.
Spectating in-client is such as powerful way to learn what people are doing that you can't always see even from a recording from their perspective.
Halo 3's in-engine replay system was the high water mark of gaming for me.
https://github.com/saul/demofile-net
I'm sure the technology still exists in the engine, but it's no longer the key feature it once was. HLTV/GOTV was launched with some fanfare back in the day.
Ah, the good old days of watching live competition of quake through the game itself, chatting with others basically through the game console.
Pretty cool system.
The game engine, Source, is also using client-server architecture
https://news.ycombinator.com/item?id=21920508
The main downside which probably caused the diseapearance is that any patch to the game will make the replay file unusable. Also at the time (not sure for quake) there was often fixed framerate, today the upsides of using delta time based frame calculation AND multithreading/multi platform target probably make it harded to stay deterministic (specialy for game where you want to optimize input latency)
Networked games have a "tickrate", just for the networking/state aspect. For example, Counter-Strike 2 has a 64Hz tickrate by default. They also typically have a fixed time interval for physics engines. Both of these should be completely independent of framerate, because that's jittery and unpredictable.
I think if I remember right there were also funny moments where things didn't look right after patches?
The bigger problem is that floating point math isn't deterministic. So replays need to save key frames to avoid drift.
Quake used fixed point math.
I guess floats are still mostly deterministic if you use the exact same machine code on every PC.
Warcraft 3 replays couldn't jump in time, just forward very fast. HoN could do that. It was amazing.
For a few months they even made ALL replays searchable on a website. Every game of HoN played globally.
Thank you for still prioritizing it.
As well as using special library versions of floating-point functions which don't behave the same across different processors I suppose, if you want to be safe.
Eg cr-libm[1] or more modern alternatives like core-math[2].
[1]: https://github.com/SixTrack/crlibm
[2]: https://core-math.gitlabpages.inria.fr/
There's no scenario in which that's desirable.
And yet even Rockstar gets it wrong. (GTA V has several framerate dependent bugs)
The only place where that doesn't matter is fixed hardware - i.e. old generation consoles, before they started to make "pro" upgrades.
What's totally insane is that the modern engine rewrite Aleph One can also play back such old recordings, for M2 Durandal (1995) and Infinity (1996) at least.
But then it'd also be nice if they fixed the "game crashes randomly when joining games" bug too.
(To give them credit, it doesn't now take 5 minutes after waking the Switch 2 before Rocket League reconnects me to the Epic servers like it did a couple of months ago...)
[0] Also the stupidly low limit on how many you can download - it's my storage cost, not yours, wtf.
[0] https://www.youtube.com/watch?v=yDUdGvgmKIw
[1] https://donadigo.com/tmx1
As a kid, I couldn't wait to see what came next. Sadly, Q1 was rather one of a kind, and it was many years until anything else like it showed up.
Like torch flames and trees swaying in the wind.
But when you find a broken ancient seal in the forest, the giant creepy eyeball moving around in it keeps moving even when you pause the game, which helps emphasise how other-worldly it is.
switch(game_state):
You still have to be careful about how you implement "gameplay", though. For example if at any point you read the 'system clock' to do time-based stuff like animations or physics, then when you unpause you suddenly will have a couple minutes of advance in a place where you expect fractions of a second.While the game is paused, if a player were to click on the "level up" buttons for their skills, each click actually advanced the game by 1 frame - so it was possible for people to die etc. during a pause screen.
A system is only correct relative to the transition system you wrote down. If the real system admits extra transitions that you care about (pause, crash, re-entry, partial commits), and you didn't model them, then you proved correctness of the wrong system.
[0] https://lamport.azurewebsites.net/video/videos.html
Modern games can have the same issue. Even taking a capture of the exact graphics commands and repeating them, you'll sometimes see animated physics effects like smoke and raindrops. They're doing the work on the GPU where it's not necessarily tied to any traditional physics timestep.
article confirms my early theory I formed when reading the title about why would pause be complicated
"Announcing TORMENT HEXUS, a match-3 roguelike about putting technofascist CEOs on the wrong side of skyscraper windows!
[...]
And remember: they SHOULD be afraid of us. #indiedev #indiegame"
Weird times.
It suggests a level of control way below what I would ordinarily consider required for game development.
I have made maybe around 50 games, and I think the level of control of time has only ever gone up. Starting at move one step when I say, to move a non-integer amount when I say, to (when network stuff comes into play) return to time X and then move forward y amount.
The best approach would be using something like if(game_is_paused) return; in the game loops.
Slowing down time applies it universally. Otherwise you're going to need that condition to every single object in the game.
I haven’t tried this yet, but for a custom engine I would introduce a second delta time that is set to 0 in the paused state. Multiplying with the paused-dt „bakes in“ the pause without having to sprinkle ifs everywhere. Multiplying with the conventional dt makes the thing happen even when paused (debug camera, UI animations).
Do people actually do that? What's the plan for when the user sleeps their machine? All the events just inexplicably happen all at once when they wake it?
Inside the game loop, we would keep the global tick counter that incremented on every tick, and timeouts would be based on that rather than on UTC.
The tick counter was updated only when the game logic was actually running. Our approach to pausing was to not run the functions that handled frame updates or physics updates, and to only run the rendering functions.
Generally we would never care about actual world time other than for some timeouts like for network (as the time passes for everyone), or for easter eggs like changing the tree models for Christmas or so.
I don't think anyone serious would implement event timers based on real time.
Live migration boils down to copy memory over the network, stream the page faults till you converge enough, and resume execution on the other host. It’s not a hard problem but a precise and tedious one.
Pausing a game might involve a lot of GPU contexts to freeze, network resources to pause, storage streams to pause, input handling, sound, etc. Add to that physics engine that may be tied deeply in the system and you end up with a hard problem.
What a VM does is not the role of the hypervisor, thus it can apply its hammer that works in pretty much all cases, and VMs are pretty much all the same. On the other hand, all games are bespoke with custom plugins and custom integrations, which make them the opposite of "generic pause implementation".