At the time of writing this, I've only been exposed to commercial, production quality, large-scale game engines for a short period of time. Before this, I had only been exposed to much smaller-scale games and engines, many of which were my own. On these smaller scale projects, it is pretty common to not really have a clear boundary between "game" and "engine". The codebase might have tasks that can traditionally be seen as gameplay code or engine code, but they might be mixed together in ways that one might not see in a larger scale engine for a relatively larger studio.
I've always been interested in developing my own engine technologies and using those to make games, so I have recently begun thinking about how I would want an engine to be structured, and what qualities I would want to preserve in a game engine, even in one that needs to scale to large projects. This post is a collection of my thoughts on some of my desirable principles behind the structure of a game engine, my reasoning behind those principles, and their associated tradeoffs.
Something I've found myself appreciating overtime is the ability of a codebase to be self-contained; that is, completely robust to external changes. This is difficult, if possible, to eliminate completely, because there are always tools that a codebase relies upon that cannot be removed, ultimately: A compiler, or an operating system, for example. There are other dependencies that are more work to eliminate (though not as impossible as they are initially assumed to be) as well; for example, it's quite rare for a project to not link with the C runtime library.
There's something not only satisfying but useful in having a project that can build by itself without any external build dependencies. Not only does it become trivial to build the project on a new development machine, or have other developers test a program without needing to make a build for them, the program also maintains complete robustness against, for example, differences in library versions. A great practical example of this quality in practice is widespread use of some of the more popular single-header or local code libraries used nowadays, like the stb libraries, or Dear ImGui. Even if these libraries are updated, as long as the version of a codebase that uses them has been made to work and integrated successfully within a codebase, that codebase will continue to work, because the library that works is just locally stored as part of the codebase, as if the owner of the codebase wrote that code themselves. The complexity of library versioning completely disappears, which avoids an entire class of very high complexity problems.
This is a quality that I think is useful to maintain with engine code, specifically one with multiple projects in flight. For every project, a copy of the engine code should be stored locally. There should never be a case where the engine can update, and it breaks a project without manual integration. The user of the engine should control when their project integrates updated engine code, and how that integration takes place.
As an aside, this also outlines the requirement of the engine code having minimized use of build dependencies, to ensure that the build process, and setup work required for building, is as seamless as possible. This is especially important for an engine that can be licensed to individuals outside of an internal team, to ensure that working with the engine is as frictionless as possible.
The first principle provides the engine with a very important quality: Each project has the full ability to modify and manipulate the engine in whatever way it needs. Because any given project has a completely local copy of the engine source code, and because the project does not depend on a shared library of any kind, the project's codebase is free to do anything with the engine code on which it runs.
Within this structure, some specific feature or phenomenon within a particular game can be explored to its fullest extent without relying on generic APIs provided by the engine. While I have not tested this in practice, I suspect that this would also improve the amount of time required for a newcomer to the engine to ramp-up and begin implementing useful features, because there aren't oddly shaped constraints by which that programmer must work within; the programmer has a much larger degree of freedom to do very specific kinds of programming in a very straightforward way.
To explain this principle, I first will explain a visual model of API genericism on which I ground my statements. Imagine, within this visual model, that a circle is some action that can be referred to by an API. For any given circle, it can also have a number of children, which represents an API's ability to differentiate between several different actions with variation in user input. This forms a tree structure, within which each level corresponds directly with different levels of genericism.
An API is a streamlined way to refer to a number of specific traversals of the tree, but an API only supports the children of the node at which it points. This implies that as an API moves closer to the tree's root, it supports more but requires more information to determine the correct path:
By definition, the root of this tree is the most generic API possible, because all subtrees can be specified by it, and there is no parent. The point of an API is to provide the ability to refer to specific actions, and therefore, despite the modern software engineering culture that has arbitrarily valued the ability of an API to be generic, this API is both useless and meaningless. It specifies nothing, and it provides no ability to the user, because if this API is used, no knowledge has been gained about which actions to take.
Something I have observed in mature game engines is that, as they begin to support a variety of game styles, their interfaces become more generic. This can be intuitively understood in the tree model; each action within a project can be represented as a different path of traversal on the tree. In order to support such traversals, the API must exist at a higher level, such that it is possible to traverse from the API to the specific final action.
The problem is this: As an API becomes more generic, it begins to require more input from the user, because the transformation that must happen is from the API boundary to the specific instructions that a computer must follow (the traversal of the tree). As an API moves higher in the tree, the number of choices that API must make increases, which requires an additional input from the user of the API. So, even if a game engine starts with the intention of providing a simple, specific, and comfortable high-level API for a particular project's gameplay code, as the engine supports more projects, this API inevitably devolves into a more generic API, which makes it less useful of an API overtime, because the entire purpose of providing an API in the first place was to make certain outcomes easily accessible with a lesser number of steps.
It's also worth noting that the traversal of this tree isn't free; this is a process that must happen somewhere, be it within the minds of the programmers themselves, the computer of the programmer, or the computer of the user. That is, as an API becomes more generic, the amount of work needed to resolve it into a specific set of steps for a machine to follow increases.
Following from this, I think an important point on API design is hidden within this argument: The art of making a good API is understanding the user's intent in using that API, and exposing knobs for the user of that API to allow them the level of control they desire, while hiding away knobs that the user does not care about. The more generic an API becomes, the less this purpose is fulfilled, because it increase the number of steps between the user and the reasonably sized subset of actions that they care about for their particular problem.
An important aside here is also that this model implies this tradeoff being a property of genericism itself, not a property of a particular implementation of a generic API. This is to say that I am not criticizing specifically an engine like Unity or Unreal, but rather that the very manner in which they have approached the problem is flawed in this way.
So, what does this mean in the context of game engines? My point is primarily this: Game-specific code is the place where ideas come to life, and to ensure that the exploration of these ideas is done to the fullest extent possible, the process of exploring those ideas must have as little friction as possible. This is to say that the engine should be as close to the perfect service to the game as possible; it should be as close to the magical API from the sky that solves all of the game's problems as possible.
Now, in reality, this isn't realistically obtainable, but understanding the ideal engine API for a game is important, so that game engines can at least get closer to it.
So, how does this work when it comes to supporting a wide variety of games? Well, this is built off of an important quality that I've already mentioned: Self-containment. Because each project is self-contained, each project's codebase actually provides the perfect environment for achieving the very specific engine features required by a game's design. Does this imply that engine programming is necessary for the fulfillment of a specific game project? Yes, but that's true in practice anyways. In cases where it can't be done, like in the case of an indie developer using Unity (or something like this), there are a few things that occur in the absence of engine programming: Engine APIs become more generic (which I've already explained has serious tradeoffs associated with it), and some designs simply cannot be fulfilled within the engine without a more gameplay-oriented programmer becoming intimately familiar with the generic engine APIs, which is a system just as complex, if not more, than just having the ability to write engine code in the first place. I can't fail to re-iterate, either, that the navigation of a more generic API doesn't only have complexity tradeoffs, but performance tradeoffs; therefore, even if a result identical to its straightforward engine implementation counterpart is achieved through a more generic API, it is inevitably slower and less performant.
The best way to approach this problem, then, is to leverage a subset of engine functionality that is applicable to the set of games that the engine supports. The obvious problem is the fact that it is not well-defined what this engine functionality subset even looks like. So, to solve this problem in practice, the way forward would be to implement a basic subset, and as projects ship, features that are identifiable as useful within the subset of engine functionality are "promoted" to engine code. Certain features that are game-specific and implemented for a design's unique requirements simply won't be promoted. This is not a problem, because if those requirements are not unique, then the game (at least in the respect of the concerned functionality) itself probably isn't either, and therefore the game is less interesting than it could be, in which case I would question the point of making the game in the first place.
Implementing a common subset of features useful for a particular set of games is a useful first step in approaching a good amount of support for a wide variety of games without compromising the engine API's integrity as a service to game code. However, within the aforementioned tree model, any useful API inevitably traverses the tree to some degree, meaning that other possible paths throughout the tree are necessarily made impossible through the API. In a case where one of these restricted paths should be explored, it is critical that the engine is still leveraged as much as possible. For example, if a certain game's design requires a set of extremely unique rendering features that do not fit within the common subset implemented by the engine's 3D renderer, then that game should not be forced to work with that 3D renderer. That being said, that same game might have very standard user-interface or audio requirements. In this example, that game should not be forced away from using the engine's user-interface code because it cannot use the engine's 3D renderer code. The only way to achieve this property is with clean separations between different pieces of engine code; that is, it is achieved through modularity, and decoupling different engine modules.
I think that, within a practical implementation to this theoretical game engine design, there would be automated paths through which modules of the engine could work together; for example, code could be generated that automatically provides the proper definitions and callbacks for the user-interface code to use the engine's 3D renderer. However, it should always be possible within this implementation to toss that aside and hook in something new to the user-interface code. While I am using rendering and user-interface modules as examples, I maintain that this quality of modularity should be consistent across all pieces of the engine.
In designing the above four principles as an outline for a game engine, I was trying to maintain some of the most notable benefits I see in smaller scale games and engines (flexibility, the ability to write specific solutions to specific problems, taking advantage of understanding context), and trying to maintain some of the most notably benefits of larger scaler games and engines (code reuse and ability to accomplish a wide variety of things), while also trying to find solutions for some of the problems with both cases.
While I'm happy with this general outline, I am definitely open to the possibility of being wrong about different points, or accepting the idea that I have missed some very important tradeoffs. I very much hope to discuss my thoughts with others, so if you have any, be sure to contact me.