Monday, June 21, 2010

Custom Engine Development

The Zombie Engine Deconstructed

It's been a long time since I last worked with high-performance real-time 3D graphics. I began to work professionally in this field in 2003, as a graphics programmer in Tragnarion Studios, a startup company based in Palma de Mallorca, Spain.

I worked in Tragnarion for four entire years, and for the most part of it, I worked with our in-house technology, a graphics engine and game framework based on open source technology, that later on was released to the public under the name of The Zombie Engine.

The development of The Zombie Engine was one big learning experience for all those who were involved in it, with many successes as well as many mistakes, which in the end rendered the technology as unfitting to the requirements of the project for which it had been developed.

But I'd like to share some insights on the key lessons I learned from the development of The Zombie Engine, for the benefit of any other company or developer who might be considering taking a similar path or, god forbid, use The Zombie Engine in their own development. Something that, by the way, I've considered myself, although under very specific conditions and constraints.

So to begin with, let me share a bit of history.


An unlikely choice


Just like any other startup company, the first decision we had to make in Tragnarion was where to start in the development of our project. There was already a lot of competition in the FPS genre, and more coming our way in the next years, so picking the right technology for the job was a key decision that couldn't be made lightly.

So our choice, and this was a heavy weight we carried with us for the years to come, was to create our own games technology, what later would become the Zombie Engine.

In the games industry, you're not a programmer until you've tried to create your own game engine, renderer, framework or whatever. It's been said that developing your own technology helps you learn more about the foundations of game programming. I can't say it's false, because I actually learned a lot from the experience of trying to do it myself, but I'll will give you a bit of advice: if you're thinking of actually creating games technology from scratch, DON'T DO IT. Unless what you want is so basic or custom that no existing technology, free or commercial, will get you there without a significant time investment.

Of course, it may happen that you, or your company, is interested in developing your own private game technology for strategic or commercial reasons. In this case, it's your choice to do so, and of course some companies have been widely succesful in this purpose, but still, I would advise you to do it with care and thought. And most importantly, with experience in the field, and clear goals in mind. Otherwise, your project might be doomed from the start.

The Zombie Engine was an evolution of another open source engine, The Nebula Device. This was developed by the german developer Radon Labs and by the time we heard of it, back in 2003, and had a number of valuable points. It was completely free and it included some of the basic commodities we were looking for: it was highly modular, extensible, and well-written. It included some basic tools, and modules already developed for graphics, animation, input, sound, network and other such low-level systems.


The building blocks

Basically, the Nebula Device as we received it was an engine, not a framework. Let me explain what I mean by this. It included a toolbox of services that included:

  • An object system, complete with classes, instancing, serialization and script interface
  • Libraries for math, strings, data containers,
  • System classes for file system, logging, time, etc.
  • Resource management for graphics, sounds, fonts, etc.
  • A renderer, including hierarchical scene graph, animation, particle systems, programmable shaders supporting effects like shadows, lighting, post-processing, etc.
  • scene exporter for 3DStudio Max and Maya.

This might look quite complete from the standpoint of a new developer, and it certainly is. There is a lot that can be done with these systems, and they save you many hours of development, when you know that every time you need a certain type of container, a math class, or managing some graphic resource, the basics are there.

We certainly appreciated the fact of having all of these, and if you're thinking of developing your own game engine, it is definitely a good starting point to have all of these aspects figured out. You should consider getting yourself an existing library for math, strings, containers for all your basic needs, and having a ready-to-use solution for objects, resources, scripting, graphics, input, sound and network, you'll certainly gain a huge amount of autonomy.

For starters, if all you want to create is a fairly simple game, prototype or concept, these might be the only things you actually need to start working. At the beginning of many game projects, concepts are difficult to visualize, and having a fast and easy way to implement simple stuff is always helpful. Say you need to test an idea for a special type of camera that has some specific behavior. You don't need a fancy engine to test something so simple, so you can create a prototype for this in only a few hours, if some basic systems and libraries are available.

Also, you can even create an entire game using only these things if your game is simplistic from the standpoint of technology – for example a quiz game, or a music game. For these types of games, the strength of the product lies on the content, not on the technology – and so you could create a custom framework for this kind of game making use of engine features directly, without using a complex framework.

And this kind of low-level engines are also ideal to create casual or low-performance games for portable consoles, smart phones, or social networks. This is assuming of course your platform is compatible with your engine (I've seen it done with the iPhone) but the general idea is that if the cost of creating an entire engine, framework and toolset for a specific project is bigger than the advantages, you're good to go with such a simple engine.

But if your project starts to be just a little bit more ambitious, what you are missing from using such an engine more than surpasses what you already have.


The missing link

The minute you want to create anything more complicated that a brick or quiz game, developing some kind of game framework, and toolset, becomes a necessity.

Now that I've presented what Nebula was when we adopted it, let me summarize what it wasn't, and with that mean, what we had to build ourselves.

  • A level editor.
  • An entity system.
  • A spatial system.
  • A level system.
  • Physics and collision systems.
  • An interaction system: triggers, waypoints, vehicles, items, weapons, etc.
  • An actor system: actions, controllers, AI, memory, perception, decision, navigation, etc.

If these look like a lot of work and complexity, it's because they are. Designing and implementing these modules is what took the bulk of the more than three years we worked on the Zombie Engine. And this is just a high-level overview of what this worked represented, not mentioning the huge improvements we had to make on existing subsystems, including, not exhaustively: a highly optimized terrain renderer, a material system, dynamic graphic effects, container spaces, music, video, complete inner (in-game) and outer (editor) GUI, game modes and objectives, plus an overwhelming number of other features, tools and utilities integrated in both the editor and the art exporter. And all of this, without mentioning the system that made everything so much more interesting and complicated: networking and multiplayer.


Lessons learned the hard way

I can't stress enough the importance of having clear goals when it comes to game development. Games are one of the fields of software development with the longest and most unstable of development cycles. Changes in design, requirements and deadlines have a profound impact on technological decisions, which are arguably the most expensive in terms of risk.

I only could learn this valuable lesson by making the mistake of taking the path of custom development without having a clear end in mind, for lack of experience, or criteria. But you're invited to learn from my mistakes. If you're seriously considering developing your own game technology, for a number of reasons, or getting your company involved in doing so, think again. The key question here is: why? What do you need that is not already available out there? What are the costs and the benefits? What do you gain from doing it yourself instead of buying it from someone else?

I'm not going to advise anybody agains developing game engines or other middleware. On the contrary, it's an important strategic decision for a company, and specialization is one really great decision to make.

But if your business, at least for now, is to make games, I'm sure there is a lot you can get from existing solutions that will save you a lot of time, money and effort. And most importantly, you'll build your product on the shoulders of giants, so that you don't have to repeat the same mistakes over and over, and focus on shaping your ideas using the product of someone else's hard work.

If, eventually, you, or your company, are still interested in building your own technology for the sake of productivity, innovation, or commerciality, by all means do so. Then, you could be building the next Unreal, the next Source, or the next Unity, so do not underestimate your chances at making a difference in the middleware market.

But first, make sure to know your lessons. Don't create the next Zombie Engine.


In the next few posts, I'll be covering different aspects of the development of an in-house engine, based on my experience with The Zombie Engine, and others.

No comments: