Why using a custom game engine

Sam

, 2023-10-17

On the home page of this website I state that “I make games with my own game engine”. The reason for “why making games” is explained in the first post. The reason for doing this with a custom engine is the topic of today’s post. This is a bit more of a developer-oriented post, but for people interested in game development in general this might still be worth a read.

To be clear, in my understanding the term custom engine can refer to a wide range of “configurations”. In general, I would consider anything that is not relying on an existing off-the-shelf-engine to be a custom engine. That could be a combination of SDL, Raylib and other libraries or something that is built completely from scratch.

Disclaimer

While do I have a decent amount of experience in software developement, my journey into game and engine developement has only been a little bit over a year so far. And my opinions might reflect this lack of experience somewhat.

So, initially this whole idea about creating games was sparked while I was working on a small Vulkan renderer. Before that I actually had already worked a bit with a few different engines, over the time of the past few years. It was always fun, but nothing has ever lead to this “desire” of creating my own games. Working on and using my own renderer, however, produced a kind of deep satisfaction that you sometimes get as a programmer. This feeling of having created something nice, something interesting, something complete, by myself. And then noticing: Now adding only input and sound would already be enough to go much further, creating full games, using the same process. These thoughts and that experience somehow completely capture me.

What I mean by explaining this is: Working with a custom engine has been an integral part of my whole journey into game development, since the very beginning. I almost certainly would not have chosen to do game development if it wasn’t for using and writing a custom engine. Therefore, I am heavily biased towards this approach.

Now you, dear reader, might be wondering: What really is so awesome about writing and using a custom engine? There are many ways to answer this question, and different people describe it differently (surprise, I am not the only person advocating for this). Personally, I came up with the following three underlying reasons. Let me explain.

There is a lot of interesting stuff to learn

I think this is the most obvious point and something that many (indie) game developers will acknowledge, even if they are mostly opposed to writing a custom engine for a serious project.

While there is also a lot to learn when using an existing engine, there will be many things that are either specific to the engine or that are conceptually on a rather high level, where many important and interesting details get abstracted away.

When developing a custom engine you will need to learn at least the following:

• Computer graphics math
• A range of algorithms
• Graphics APIs
• Operating system APIs
• Creating resource/asset pipelines
• Creating builds that package the whole project into an executable that runs on any PC
• And many other smaller things

In addition to that, as it is always the case with larger code bases, you will need to think about software architecture. Even if the main concern is developing a game and not to become proficient in any of those areas, having a deeper understanding of them will make you a better (game) developer. I believe this is true for any kind of development, from desktop application development to website development, from mobile apps to HPC computing from embedded systems to machine learning. Having a better understanding of the underlying systems, API’s and protocols, whatever that may be, will help you to develop a better solution. In the case of games I am therefore convinced, that I will be able to make better games

I want fewer dependencies

One of the most fundamental things I have learned to value during my time as a software developer so far is the following: For every dependency you avoid, you avoid a range of potential and actual problems. Some dependencies are necessary or provide enough value that they should not be avoided. However, in many cases it is very valuable to either write a piece of code by yourself or copy only the required part of a dependency into the project (depending on license and source-availability of course). It would require a whole post on its own to go into the reasons for this view. But, to put it quickly: Controlling ¹ a piece of code has many advantages when building a stable and coherent system, and it allows removing road blockers faster.

You see, using an existing engine is alredy a huge dependency on itself in the first place. Of course, you get a lot of value out of that dependency, no question. But, apart from the lack of control over the system, you are also stuck with that specific engine (which some indie developers only just seemed to realize with the debacle over the license changes in the Unity engine last year). Furthermore, it can be very difficult or ugly if you want to replace certain pieces of an engine. Sure, you may always include an additional library to replace a feature in the engine. But now you lost the convenience and integration that the engine gives you with its native features, and that often increases friction.

For some developers this might all sound a bit farfetched, but there are also experienced games developers that do advocate along the same lines:

But just keep in mind, you don’t have any control over [3rd party game engines and libraries]. It’s nice to use them, it will speed you up. But some of them don’t even provide you source. So if you run into a bug, you often… you [will] often […] lose time, going: “Is this bug on my end? Or am I using it wrong? Or is it a bug on their end?” […] If you have the source, at least you can dig into theirs, but you will spend a lot of time, trying to find out where that bug is happening

Source: Timothy Cain (Fallout, Fallout 2, Pillars of Eternity, The Outer Worlds)

So, like five years later, when we’re finishing up the game, it’s like: Okay this particular thing isn’t working. We need to change this way that [our custom] engine works, to do something else. And we have the ability to do that, right? Which you don’t get (or it’s very difficult) […] if you license Unreal or Unity or something. First, you have to even have a source license (which a lot of people don’t have). And then, you’re trying to modify something that is a great deal more complex than what it would be if you only did the things that your game needs, right? Because solving all problems for everyone is a lot more complicated.

Source: Jonathan Blow (Braid, The Witness)

Along with that, it seems to me like there is an inverse correlation between the amount of joy you experience through programming and the amount of dependencies in a given project. For many game developers it is probably more important to have a lot of features and a well tested system at hand. For me personally, enjoying the process of writing and designing code, systems and architecture has a much higher priority. And that leads us to the third point.

It is more fun

As mentioned above, I have tried a few different engines. Never for anything big, just for a few visualizations, playing with procedural generation and similar stuff. But those experiments often left a feeling of an unfullfilled developer experience:

• For Editor-based engines, there are many mouse clicks involved while programming. For a keyboard-first user (I use neovim, by the way), this does feel like a “drag”. Also, some of those engines are really heavy applications, and it is often very noticable in differnt parts of the workflow. A custom engine will by default feel much more lightweight and faster for development.
• The engine does determine a big part of the control flow of the program. How the main-loop looks, when which hooks/plugins/entity-lifecycle-methods are called, what datastructure the scenegraph uses etc. That means we, the developers, often have to adjust to whatever style (or architecture, or convention, or whatever else) it wants us to use. And that style is something that has to fit thousands and thousands of other games. For my specific game, on the other hand, I often might want to do things differently and have more control over what is happening when and how.

Drawing on an example that touches both of those points: Currently my games take only a few seconds to rebuild, including the whole engine. That makes the change-build-test cycle very short and comfortable. But even if I would get to the point where I find the cycle-times no longer acceptable, I could start to work with hot-code reloading or incremental compilation (or, what I already did, adding an auto-reload feature of configuration files).

Also, with a custom engine I am completely free to choose my preferred programming language to write my game in. I love the fact, that I do not need to use the “most productive”, “easiest to learn”, “most powerfull” or “most performant” language of the day. I can just use the language that I love (which is Nim, because it is productive, easy to learn, powerfull, performant and also pragmatic ;).

In short, I get to make many decisions that can make my life easier (or harder). This kind of freedom also leads to high level of satisfaction during development.

Those reasons are of course strongly related to my subjective preferences. That is fine since I am not trying to make an objective argument for writing and using a custom game engine. I try to explain to myself and to the vast emptiness of the internet why my preferences are what they are.

Acknowledging the difficulties

Now, of course not all is bright and green pastures over here in custom-engine-land. We have to acknowledge the issues that arise when writing a custom engine:

• It is a lot of work. Even if we ignore the time required to aqucire the knowledge for different areas, there are thousands of lines of codes that need to be written. Algorithms, API-integrations (e.g. Steam, PhysiX, VR, etc.), OS- and driver specific workarounds etc. often need to be implemented “by hand”. Sometimes external libraries can help, but integrating those into the engine takes time as well. Especially when writing an engine for the first time, the required amount of hours is considerable. You have no experience and code to rely on or reuse directly.
• Switching genre or core mechanics tends to be very costly. A jump-n-run, an isometric RPG, a first-person shooter or a car racing game all need very different systems. A ready-made engine supports most of those systems out of the box or can integrate them via plugins. Making one kind of game first and then something completely different for the next game is easily possible. Not so with a custom engine: Switching to a different genre or even just another camera perspective might require the implementation of a bunch of systems, algorithms or features that do not yet exists.
• Custom engines have no support. There is neither a substack nor a forum nor paid support. You need to solve most problems on your own (which is great, for programmers, I would say ;).

Those difficulties can be overcome by being carefull when selecting scope and genre. At least as long as I am not aiming for “financial success”. But there are actually quite a few indie successfull indie games that were using a custom engine. And in some cases, the creators state, that using a custom engine was crucial for the realization of the game. Let’s take a look at a few examples in the next part of this post.

Examples of success

So, you might say: “These are fine points and all, but, be honest Sam, releasing real games based on a custom engine is just not something viable. This is just a dream”. To which I would say: Your right, this is my dream ;) But also: You are wrong! :P There are many example of people in very small teams not only releasing real games, based on a custom engine, but also having massive commercial success. Obviously, those people usually have a ton of experience and with most (or all) hits there is some degree of luck involved. I think two of the most famous examples are Stardew Valley and Minecraft, which are both well known not only in the PC-gaming community but also among many casual players. Another example is Zachtronics who is known by many developers due to their programmning-y, puzzl-y style of games. They released a few games that were built with a custom engine and mostly done by a single person.

But there are plenty more examples. There is a really nice small study of games that use custom engines, from AAA studios to single-man projects. The post can be found here.

So, to sum up: Building games with a custom engine is certainly a hard enterprise, and it is, at this point in time, still a dream for me, personally. However, I am working on letting this dream become reality in some form or another. And even if, in two or three years, I am going look back and have to say that it does not work this way, I am sure that I will still be glad to have learned so many things along the way.

That is all for now. I hope I was able to present some interesting arguments or point-of-views to you, dear reader, and hope you had a good time.

Thanks for spending your time and reading this,
Sam