Entering virtual environments

It's been awhile since I last wrote in this blog, and now I'd like to introduce what I've been working on during the last few years, after doing the one-week game project.

Shortly after that, in November 2010, I got back to work as a programmer for Blit Software, including mobile, web and desktop development. In all of these cases, I worked with applications that provided a web-based approach to retrieve data from a webservice. Then I worked with Unity3D on some augmented-reality game for mobile platforms: Drop Defender.

After a year, I moved on to worked at Digital Chocolate (Barcelona) as Lead Game Programmer, first in Galaxy Life: Pocket Adventures, then as Lead Tech Programmer in Battle of Heroes: Land of Immortals, an exciting project. In both, I got to work with a web-based online game that would perform a client simulation but would retrieve and update its progress with a remove server using web endpoints.

In both of these positions, I've had the opportunity to work with web services, which was a new kind of paradigm for me, even though these days it is a rather standard way of storing and accessing remote data. You build a front end that manages the user interaction and most of the logic, but then you get some significant data from a service, through an asynchronous HTTP connection.

Some time ago, while working at Tragnarion Studios, I got in touch with Squeak, an environment built on top of a Smalltalk virtual machine, an old project dating back to the 70s. I found this kind of environment enlightening, so wonderfully interactive, that I found it surprising that there was no other modern project with the same kind of interactive, edit-from-inside development environment. On top of Squeak, another team developer the Croquet Project (and then the Cobalt Project, now actively developed).

Finally, another project I got to work in Tragnarion Studios involved a multiplayer experience where multiple players would share a common experience across a distributed environment. That's when I learned how to work with replicated variables, remote procedure calls, and predicted, interpolated and remotely controlled behaviors. A deliciously interactive environment would replicate changes to its state across all players sharing the same simulation across a local, or global network. And that considering delays and all sorts of unpredictable network situations, is quite a feat.

Let me summarize these kinds of scenarios:

• a client front end providing user interaction that will get its content from a server
• a interactive environment for editing data and behavior
• a distributed simulation shared among a number of player over a network

Now what I've started to think is: what if we took the interesting pieces of each of these approaches to a distributed interactive experience, and put them together with cutting edge technology, over multiple platforms? That's of course what I've been working on, and I'd like to share with you the interesting findings of this approach.

Of course the development of such an environment has been highly incremental, for basically each piece is built on top of the previous one. First we build (1) an interactive environment we can download from a website, then we add (2) some edition capabilities, then we (3) share it with multiple clients. It is this combination that brings us the full power of collaborative virtual environment.

(to be continued)

The one week game project: aftermath

The week of my OWGP is over, and it's time to wrap-up and summarize the lessons learned while conducting this experiment. First of all, I would like to explain what the idea of the project was all about.

Motivation

During my years working in the game industry, the most disturbing pattern I noticed was one that has to do with planning, prototyping and delivery. Basically, these steps only existed as a good idea but they were never truly developed.

Planning was just a way of casually stating what kind of game we were doing and some general rules about the directions, but the whole project was rapidly driven off course by means of improvisation, unexpected requirements and general trashing that came from upper management, or simply from the last person to join the team.

Prototyping didn't exist, or at least not intentionally. Turns out everything we were doing was treated as a prototype, because everything was open to tweaking and experimentation.

Finally, delivery was just a requirement to fulfill, but had nothing to do with the actual development schedule. Significant changes were made during the beta stage, introducing new bugs and cases, and the final version released was just the one that crashed the least, and included the features that had made it through the chaotic development cycle with as fewer bugs as possible.

In fact, I was actually able to ship only one game, and reaching gold was a nightmare of last minute changes, unforeseen features and overtime.

Unfortunately, these things are usual in the games industry. Not all studios like it, but most accept it, and even exploit it, as part of their culture. And these things are the cancer of an already troublesome industry, leading projects to delays, overbudget, cancellations, and teams to frustration, turnover and burnout.

What I want to prove is that planning and prototyping can happen in very short development iterations, where features can be tried and tested. And with a consistent application of these, and some work discipline on the part of managers, producers and designers, it is possible to actually deliver on time and on budget.

This is why I decided to conduct this small-scale experiment, where I would play all the roles, and feel the urge to delay and trash, but restraining myself from doing so because of the short time schedule. And I also had to keep myself from the programmer urge to overengineer and aim toward perfection, basically because there was no time for that. I had only five days to get the job done (hit a fictious beta stage, let's say) and that had to be it.

Postmortem

So let me summarize briefly the day-by-day development of this project:

• Day 1. I created a small scale simulation of the game idea, without using any actual resources, only abstract progammer art (cubes and spheres) representing the game entities (the stage, the props, the player, the enemies and the projectiles). I could get a simple game like this done in less than a day work, given that I had the technology for that (I used the Zombie Demo framework). Also, the game was a simple shoot'em up retro-style game, so I guess design was not such a big issue here.
• Day 2. This was the moment to start creating an environment for the game. I decided to use resources I had from an old project I worked with using the Zombie Engine. Problem is, I didn't want to use the Zombie Engine itself, because the complexity of defining and managing entities was too big and I wanted to keep such a small project small. So I tried to load only the data I needed and go with it. By the end of that day, it was obvious that I couldn't do that in just a few days, and still get the game done.
• Day 3. This is where I had to be practical and came up with an interesting idea. The only thing I knew for sure of the Zombie Engine is that graphics and edition worked really well. It was gameplay I couldn't really grasp. So I had this idea to create a level for my game that was made only of "dumb" entities, that is, entities with no logic or behavior. Then, I would get access to each of these entities and let my game logic update their state. I didn't even think this was going to work, but it was worth a shot, and by the end of the day, I could move my non-animated player through the level, so I decided to go this way.
• Day 4. After finding this indirect way to use Zombie to my advantage, I kept on developing the game but instead of abstract graphics, I used to entities in the level, and had to write the gameplay logic to move and animate graphics, and let game objects interact with each other (shoot, etc.) By the end of the day, I had my player running up and down, shooting forward, and enemies attacking. Unfortunately, I had only one day left to finish it off and still had some important things to figure out.
• Day 5. This was the day I had reserved for the finishing touches and I still had the important task of defining end game conditions, and restraining the movements of the player (I didn't want to use the collision system in zombie) so that's pretty much what I ended up doing. By the end of the day, the game ran more or less ok in my test level, but not so much in an actual game level. But this was something for the beta stage to work on, and I had already proven my point.

The result of this experiment has been enlightening. I've realized how easy it can be to stray from your original concept when faced with technical limitations, but also that short iterations lead to practical solutions, and having a clearly defined timespan, you don't get to trash or change your mind, so you just work with what you have. If it's not perfect the first time, you will do better next time. But if you deliver, you'll most likely be able to deliver again.

As Seth Godin states it: "You don't make a living out of being creative. You make a living because you ship." Being creative is something that happens only in the first stages of a project, but the kind of changes and redirections I've had to suffer (and many other developers for that matter) because of the hierarchical nature of development in the games industry are proof that poorly understood creativity leads to projects that are finished late, over budget and with a hidden cost of lost human motivation and satisfaction.

These factors may not usually be taken into account, but they are part of the process of creating games as well. And what I wanted to prove, myself included, is that it is possible to think of games development in a more practical and imperfect way, but one that ultimately leads to shipping, and getting to try again because of that.

So maybe I'll try again, and see what comes up next time.

I can't really publish the resulting product, because it's too shameful, and because some of the resources it uses are copyrighted by a former employer. But if you're interested, the code can be found in the SVN address.

The One Week Game Project

Please take a look at this video. It's 6 minutues long but you'll get the point rather quickly. In fact, if you're seriously interested in game development, you won't be able to stop looking at it. Then come back.

Welcome back!

I love these challenges. Creating a game (or anything else, for that matter) in under a specific, and usually rather limited, amount of time, is the kind of experience that proves who is a hardcore developer and who isn't. By forcing yourself to work under severe time costraints demands the kind of discipline, pragmatism and openmindedness that many experienced developers should practice once in a while. The people who is able to get away with a usable product at the end of a limited time are usually the ones that, when working in an actual project, lead the way for the rest to actually deliver a finished and successful product, even if it's not as polished as you would've liked.

These are the people who actually ship products, often in budget and in time. The rest are the architecture astronauts.

Having been an architecture astronaut myself more times than I like to admit (and having worked with some hardcore ones too), I've realized how important it is to stay sharp, to be practical, and to work not towards creating some perfect but ultimately impossible piece of art that never gets done, but towards creating an actual product, imperfect as it may be.

Creating a game (or any product for that matter) in a ridiculously short time is fun, but most importantly, it lets realize how little time you actually need to build stuff. When you don't have time to try many different architectures, or the best way to actually implement that so that the High Gods of Game Code are pleased, you take short cuts. You make compromises. You write ugly code that gets the job done. And most importantly, you make actual progress, fast.

So why am I telling you about this? Because I've decided to take one of these challenges myself, for the sake of my own self-advancement, professional growth and actualization.

The point is to code a game, in a week. My One Week Game Project is about building a 3d game from scratch, assuming (here's the catch) that I already have the resources and low-level engine I need, which in my case will be the low-level engine from the Zombie Engine and the Nebula Device. And the point of it is not to create a game that I can sell anybody, but to see what I can do in such a short time span. I've chosen a classic Shoot'em-up type of game but with 3D graphics that should be fun to code and has no relevant design decisions because it is such an established format.

And then I'll come back here and write about the results and what I've learned from the experience. And depending on the results, I may even release the game, or create a demo video for it.

I hope to tell you about this soon enough, otherwise it'll mean I failed. But I'm anthusiastic about the idea. And it's just a week anyway. The ball starts rolling on monday.

References:

• Ludum Dare: http://www.ludumdare.com/compo/
• How to Prototype a Game in Under 7 Days:
  Tips and Tricks from 4 Grad Students Who Made Over 50 Games in 1 Semester
• Someone else who tried to do the same thing this year: http://oneweek.ludoko.com/

The Lost Art of Prototyping

Innovation in games is all about new ideas. And new ideas, like new technology, fall into the realm of the unknown. And until you haven't experienced the unknown (that is, known it) you don't know what works, and what doesn't. This is what prototyping games is for.

Prototyping is something that doesn't receive much attention in actual game development. Surely, people talks about it, and some prototype stage is usually planned, but what usually happens is that upper management is expecting incremental releases of a product. And no matter how much you explain what prototyping means to them, they can't just be handed a demo showing a bunch of untextured cubes, "just because this is the layout of the level we're testing".

The reality is that companies don't usually pay much attention to prototyping, and yet it is one of the fundamental parts of the learning process for a developer, be it a designer, artist or specially a programmer. It is the process where the developer gets to know with some detail a specific part of the project at hand, tries different possibilities and gets data from all of them. Then, hopefully, some makes a decision, be it the developer or someone else.

Prototypes are not expected to be pretty. They are not expected to be optimal, fast or robust. Unless prettiness or speed or whatever is what you're actually testing. But generally speaking, prototypes are created to test a concept, then throw them away.

Wait a minute, throw them away? This is the part I've never understood about prototyping, or at least my idea of what prototyping should mean. Once you create your working prototype (or several) you can actually use them as the basis for more tests, or other prototypes. After all, if a prototype has been correctly designed, it should be fairly independent and isolate from the rest of the code, shouldn't it?

That's exactly the problem with most prototypes – they are created as part of the project where they belong, and creating a prototype under those circumstances breaks the idea that you're trying a new concept. What you're doing is work, actual work. And doing actual work without enough information (which is what prototypes are meant to provide) is not only a bad idea, but a waste of time.

So how does prototyping work?

Whatever your technology, the absolute requirement for prototyping is that it should be possible outside the environment where the project is being developed. Or at least, as far away from it as possible. The reason for that is that is that your prototype may interfere with other parts of the system, and there may be a lot of noise in the information obtained from that process.

So for example, if you want to try a layout for your game level, this is how you do it: you create the level in your editor, using just simple shapes and no other graphics or resources. Then place your players or spawn points, and you test it. This way there is no overhead in the level that may distract you, and you're not messing with actual assets, or a level that is half built.

For coders, prototyping is more like a dark science. The ability to isolate the conditions and data that allow prototyping a new feature is only acquired through trial-and-error. Say you want to create a new camera mode, and need to test it before you implement the whole thing. You would need an empty level to create and define your camera, and a piece of code that would load that level, and nothing else, to test your new camera.
But is this always available? Do modern engines or technologies provide support for this kind of development cycle, this trial-and-error approach?

Some of them are certainly designed with prototyping in mind – Unity3D comes to mind. Others are a pain to prototype due to the amount of dependencies in the code – UnrealEngine3. And for not so professional solutions, such engines as Irrlicht or Ogre3D allow fast prototyping with a minimal set of resources.

I've been creating a demo framework for the Zombie Engine with prototyping in mind. The idea here is that adding some complex mechanics to the whole engine and game framework may be difficult enough, which is why a simple demo environment should be enough to test many different things – new materials, lighting or rendering modes, cameras, controls, etc. I've been able in just a few hours to prototype several simple game cameras using this, and the results are excellent. With just minimal code, a few ugly resources and the bare minimum that is needed to run the engine, I've been able to test new features and experiment with control modes.

And what's more important, whenever I find a bug, or need to try something new on the same part of the game, I can just go back to the prototype (which should always be kept safe) and modify it to extend, fix or improve the feature, without the risk of breaking production code.

Go for fast code prototyping, by all means. If I could find happiness in a brick-textured torus, so can you.

Custom Engine Development (2)

The Zombie Engine Deconstructed

The Basics

Covering math library, utility classes, system classes, debugging tools, object system, resource management.

When it comes to game development, most of the initial enthusiasm is usually focused on the visual part. Admittedly, graphics are the most attractive part of games, and usually the one that actually sells the product to publishers and customers alike. The results of graphics development are visible, fancy, and can be shown with pride to strangers.

This is probably why graphics renderers are the piece of game technology you'll find most usually covered in forums, projects and articles over the Internet. I know because I wanted to get into games development mainly because of this area. And this excess of enthusiasm led me to neglect a more fundamental building block of game development, the ones you may not even know about, unless you're seriously working in actual games.

Possibly, the reason is that these basics are usually assumed. Most development tools and environments include some libraries covering file management, math, strings, containers and other such classes and utilities. Standard libraries like the STL (for C++) provide you with a readily available toolbox of types, definitions and classes, covering a wide range of programming needs. This is why many times you can just jump into programming concepts, demos or simple games without having to give any of these much of a thought.

But are these really all you need?

(Example) Say you're creating a simple puzzle game, one where an image is split into square pieces that are shuffled randomly, and the point of the game is to move them back to their place by moving the pieces in a specific way.

This is a fairly simple exercise for you to make, and it is, in fact, one of the first attempts at game programming I would advise an unexperienced developer to try: No 3D, no complex interaction, no multiple processes, no internal logic. A simple problem and a simple set of rules and victory conditions.

Game complexity aside, just by creating such a simple example you would start noticing the convenience of having more than just a few standard classes to help you with. Surely, you would have to write the game logic, and some way to load and process the image, display it on the screen, and receive and process the point-and-click input from your mouse.

In order to program such a game, you could just allocate memory dynamically using standard memory methods, read the image from a file using standard file access functions, use a standard system timer, and create all required custom classes for the state of the game, including custom methods to save it to file.

There are a lot of different reasons why you don't want to use directly the standard classes and libraries, and I'm sure you could find even more of these.

• Code sanity: game code that uses classes, macros or functions with meaningful names and well-defined behavior is easier to write, read, test and maintain. WAY easier.
• Reusability: creating classes and functions specific for use in games makes it very convenient for repeated use, in the same or different projects, reducing redundancy.
• Encapsulation: you really don't want to think about memory leaks, resource referencing or empty pointers, when you are writing a game, so hiding these inside classes hides the complexity and exposes cleaner interfaces.
• Platform independence: this means your time, file, device and other such classes are abstracted from the actual, system-dependent classes, making code both portable and clearer.

If you're a programmer, I'm sure you could keep filling this list yourself, but the point I'm trying to make is: no matter how simple your project, make sure your basic stuff is in place before you even start creating a concept. There are things you will need eventually when writing your game, and having to think about them when you're already developing actual game systems or features is going to be painful.

And now, I'm going to summarize what these basics are, so that you can check if you have them in your code weaponry, before you even start thinking about triangles.

Math
Numeric Types, Vectors, Matrices, Quaternions, Points, Lines, Planes, Angles, Transforms, etc.

A complete and efficient math library is a must for any game engine worth that name. Practically every text book and demo framework will provide you with one that you can extend and complete as you need, and there are many such libraries available for free at different sites.
One such library should provide, at the bare minimum, structures for 2D and 3D math: and all required operations, conversions and tests. Also, it is highly desirable that it provides structures to represent standard 3D objects, such as boxes, spheres, etc. including their transformations and interaction tests. Collision detection is a complex module to build, but having a readily usable set of shapes and tests is a useful thing to have.

Another important feature, specially to support portability, multiplatform, and compatibility with third-party libraries, is some encapsulation of numeric types and operations for special data architectures (eg. SIMD), optimized implementations (fixed point, double precision), etc.

It is highly tempting to skip adopting a math library, using instead the one provided by another module. For example, many graphics and physics libraries come with their own math library included. My advice is to choose a math library that suits your needs, and stick to it. If you need to interface with an external module, prefer always converting data rather than creating dependencies with a library that is not under your control. The internal representation of numeric data in a math-intensive system (eg. physics) is probably optimized for some internal reasons.

Also, relying on the math library provided by an external API (eg. DirectX) will make your code immediately dependent on the use of that API, which is a bad thing in general. Specific APIs and middleware may come and go, but your maths should be as stable and reliable as possible.

Utils
Strings, Containers, Streams, Packing, User data

I've always found it funny that every engine I've seen (and Nebula was no exception) includes its own class definition for strings, arrays, dictionaries, and all such basic structures. Because the point of these articles is to avoid reinventing the wheel (unless you want to do it with another shape) I would try to use whatever library you can find that gets the job done and forget about it. You won't probably come up with a more optimized version of a hash table, so don't bother writing one yourself, unless you really need some very specific and custom feature.

Aside of these basic data types and containers, you should start thinking about defining your abstract stream interface, that you can later on use along with your file, network or memory system to read and write streams of arbitrary data types. This way, you'll be able to define the interfaces for resource loading and object serialization (see below) to use generic streams instead of files or memory, which will simplify your methods and make your life easier.

Think of packing, compressing and encoding as part of this basic utility library. The reason why is that you want to have a way of packing and compressing resources as soon as possible, so that you don't have to optimize for disk space, bandwidth or access speed later on. The Zombie Engine failed to provide at an early stage a proper resource packing system, instead relying on resource files (levels, models, textures, sounds, etc) being loaded from raw files on the hard drive. Don't make the same mistake, and work with packaged resources from the very beginning.

Finally, I'd create some simple way of providing text-based user data to your game in some way, in a standard, or user-defined form. In the Zombie Engine, and other projects where I've worked, such a user-editable data format was used to describe such diverse things as materials, render pipelines, character states, game actions settings, user-defined data templates, GUI definitions or localized texts. XML is a good choice because of its hierarchical nature, human readability and wide tool support. But it has some other problems, so you can choose other ways of doing the same thing, such as C-like structured files, INI files a-la windows, or whatever makes you happy. I would make sure the same format is used consistently across the whole project, specially if these data files are going to be processed into optimized binary data, which will simplify the process of data conversion and packing, and your personal sanity.

System
Memory, Timer, Devices, Files, IO

These are several low-level systems that depend heavily on the specific operating system, and there are many subtle changes from one platform to another.

You need to account for memory management from the very beginning, or you'll be in trouble later. If there are going to be different memory areas, or you want to have a budget to allocate resources, you need that information available as soon as possible in the project. Games use to manage memory in an intensively dynamic way, which easily leads to fragmentation, and potential memory leaks if you don't pay attention. Having a proper memory manager in place before you create a single game object will make your life easier, will let you know where exactly your memory is being allocated, and will prevent errors later on.

Using time, access to external devices, managing files, and the all-important user input are some of the utilities that depend on how each specific system handles it, and you shouldn't have to worry whether a click came from a mouse button, or a multitouch screen. To allow proper portability, the internal working of these needs to be not only completely hidden from the client application, but also be able to provide access with consistent data format and ranges across all systems. So for example, proper file management would encapsulate file paths using some virtual file system that provides a common namespace for all possible implementation, and this includes hiding completely things like whether the system uses slashes or backslashes to separate path components.

When I talk about input here I mean raw input, that is, collecting input events from any input device your platform supports, and enqueuing them for later processing. Input handling, such as converting mouse clicks or key presses into actual events for the application, is a higher-level functionality. If you're able to collect and store the state of all keys, buttons and screen interactions, using a consistent representation (eg. a normalized coordinate system, to abstract the actual resolution) you'll have everything you need to start defining input handlers for your game.

I'm skipping here two important parts of modern games that are just too complex to handle as part of your basic utility library, although they arguably belong in this category: networking and multithreading. I'll talk about these in another post, for they deserve some individual attention each.

Debugging
Asserts, Logging, Profiling, Memory, Console

Although debugging, just like testing, is usually seen as something that needs to be taken care of only close to the end of a project iteration, having a functional and useful debugging system in place from the very first moment can make the difference between a succesful project and a project that may not fail but will painful, if not impossible, to maintain.

These utilities fall in the category of Risk Management, and you should look at them as the fire extinguishers of your engine. If everything goes as expected, these remain in place untouched, but the moment awkward things happen, or when you start having a lot of data loaded, you definitely need these.

Asserts and Logging are well known mechanisms to monitor the behavior of a program, and to catch unexpected, or unsupported, conditions. But here I need to make an important note though: although they are related in the sense that they provide useful information about what's happenning, and the state of the game, they are not to be mistaken, or used interchangeably.

One of the lessons you can learn from the Zombie Engine is how not to use Asserts as a way to complain about input data. Asserts are supposed to check for internal system conditions: allocated memory, array indices in range, stuff like that. They are not supposed to check for the validity of data, which is something you need to check anyway, and probably Log it if it's not valid. But you can't Assert something like whether a given file exists, which is not a condition, but a data dependency. In the Zombie Engine, there are lots of different asserts that check for data consistency, such as:

n_assert(this->parent->IsA(kernelServer->FindClass("ngeometrynode")));

Profiling and Memory inspection are your other best new friends when it comes to performance and data size. As important as it is to log what's happening at any moment during an execution, it's just as important to know where is the most time spent, and how much memory is being used, and where. This kind of information will be critical to identify bottlenecks, and to track where all your memory is being sucked up. Remember that a game project usually is the result of conflicting parts trying to get more and more space and time out of what a frame allows: artists want bigger textures, while sound designers want better quality, but both require space. Budgeting frame time and memory space is an integral part of the design of a game. And Profiling and Memory managemtn will be your main tools in sticking to these budgets. You can even Log when these budgets are exceeded on either area, and you certainly should.

There are many ways in which you can define these, but one particularly effective is to define “watchers” which are arbitrary variables that you can use from different modules to track any kind of profiling you'd like to do, and then some ways to keep them in memory. The Zombie Engine uses these from nebula2, and also adds a visual inspector of such “watched” variables so that you can have an onscreen representation to track variations in these values, which are data-dependent.

Finally, the Console will be your way to interact with your low-level systems for most of the time. Until you can have some visual inspection of the state of the game simulation, or ways to enable or disable different options, you should be able to do this from an onscreen console, like the one used in many engines out there. If your engine is script-enabled, your console could allow any script command to be input, thus allowing some closer access to the engine internals.
The Zombie Engine inherited the console from nebula2, and it remains one of the most useful ways to interact with the core engine. But in order to make it really useful, you need another fundamental part of the system, as discussed below.

Objects
Metaclasses, Namespace, Life cycle (instancing, releasing, reference counting), serialization, signaling, scripting

An object system, for those of you who haven't heard the name before, is a way to describe the relevant objects in your engine (and in your game) and manage them as if they all belong to the same space of names and access. Ok, this may not be a very formal definition, but what I try to describe is that all relevant objects in your game (objects in the OOP sense) need to be described in a standard way. If you're like to work with Smalltalk or some other pure OOP environment, you don't need to worry, because you already have this. If you're into good old C++, you want a way to describe the interface of your objects, supported commands and properties, namespaces, saving and restoring, and other such things. If you don't have one of these object systems, you'll have to emulate all this functionality on your own.

If you're just creating some casual game, you can skip this if you want. But if you're into serious game development, you need an object system. By that I mean a way to describe what your game objects, resources and system look from the outside. This includes how they are named, created, referenced, persisted and restored, accessed and intercommunicated. These may look like a lot of things, and they are. But this is one basic need to you need to cover in modern game engines.

The object system in Zombie is inherited and extended from the one in nebula2. It defines metaclasses holding a set of commands, a kernel that acts as an object manager and is responsible for creating and destroying the objects, and putting them in a named hierarchy, which similar to a folder system, where nodes are objects belonging to this object systems.

Finally, there are objects that are managed by their own managers, including the all-important Entities, that not only are objects themselves, but are an aggregation of multiple components (which are objects of its own), and belong to a different namespace, namely the Entity namespace, where they are identified using a unique Id instead of a name. More details about this in a future article on Game Entities.

Resources
Referencing, Loading, Memory

If you're going to work with meshes, textures, sounds, animations, or any other such system-specific data, you need resources. These are objects (in the above sense, or not) that contain data that applies only to a specific system. For example, TGA files are to be used as textures, while MP3 files contain sound samples. These resources will contain basically all data that is not generated inside the game (or editing) framework itself. All art and sound assets are typical resources, and all of them are managed in a similar way.

You want to be able to reference your resources from your code, or your game objects, in a consistent way, access and load them, and track how much memory each kind of resource is eating up. A Resource Manager should provide these and other functionalities, and allow the client application to just claim and dismiss resources as they are needed.

Tools
Resource processing: conversion, packing, exporting, viewing

Resources are not always generated in the most efficient way for the game runtime to access or use them. Many times you'll organized your data in folders so that you can easily access and edit it from your authoring tools, but this may not be in the most efficient organization or format for the game.

This is why having tools to collect, convert, optimize, pack, and do other operations with your resources is a strong “should”. And if you're generating data packages for versions of the game on multiple platforms, it is simply a must. Fortunately, you don't always need to create these tools, there are conversion tools for many different formats, but you probably want to automate the process using a data build script or a similarly automated process.

Finally, having a way to preview your resources is a necessity, specially when they start behaving in all the wrong ways in your application. During debugging, you'll want to make sure that the data itself is correct, so that you can discard that being the source of a bug. Specific tools to check and preview resource data should be always on your toolbox.

...

And this will be all for this “basics” post. You may have noticed a few things you didn't think were actually required to start building a game, but you'll have a problem if you don't have them from the very beginning. Which is why, my original advice remains: don't try to build a game engine from scratch unless you have to.

And if you have to, now you need what fundamental pieces you need, and may be interested in looking for them in standard libraries rather than making the effort to write them yourself completely. Trust me, it's not worth it.

In the next post, I'll be covering the first major system, which is not graphics but Input. Stay tuned!

Custom Engine Development

The Zombie Engine Deconstructed

It's been a long time since I last worked with high-performance real-time 3D graphics. I began to work professionally in this field in 2003, as a graphics programmer in Tragnarion Studios, a startup company based in Palma de Mallorca, Spain.

I worked in Tragnarion for four entire years, and for the most part of it, I worked with our in-house technology, a graphics engine and game framework based on open source technology, that later on was released to the public under the name of The Zombie Engine.

The development of The Zombie Engine was one big learning experience for all those who were involved in it, with many successes as well as many mistakes, which in the end rendered the technology as unfitting to the requirements of the project for which it had been developed.

But I'd like to share some insights on the key lessons I learned from the development of The Zombie Engine, for the benefit of any other company or developer who might be considering taking a similar path or, god forbid, use The Zombie Engine in their own development. Something that, by the way, I've considered myself, although under very specific conditions and constraints.

So to begin with, let me share a bit of history.


An unlikely choice

Just like any other startup company, the first decision we had to make in Tragnarion was where to start in the development of our project. There was already a lot of competition in the FPS genre, and more coming our way in the next years, so picking the right technology for the job was a key decision that couldn't be made lightly.

So our choice, and this was a heavy weight we carried with us for the years to come, was to create our own games technology, what later would become the Zombie Engine.

In the games industry, you're not a programmer until you've tried to create your own game engine, renderer, framework or whatever. It's been said that developing your own technology helps you learn more about the foundations of game programming. I can't say it's false, because I actually learned a lot from the experience of trying to do it myself, but I'll will give you a bit of advice: if you're thinking of actually creating games technology from scratch, DON'T DO IT. Unless what you want is so basic or custom that no existing technology, free or commercial, will get you there without a significant time investment.

Of course, it may happen that you, or your company, is interested in developing your own private game technology for strategic or commercial reasons. In this case, it's your choice to do so, and of course some companies have been widely succesful in this purpose, but still, I would advise you to do it with care and thought. And most importantly, with experience in the field, and clear goals in mind. Otherwise, your project might be doomed from the start.

The Zombie Engine was an evolution of another open source engine, The Nebula Device. This was developed by the german developer Radon Labs and by the time we heard of it, back in 2003, and had a number of valuable points. It was completely free and it included some of the basic commodities we were looking for: it was highly modular, extensible, and well-written. It included some basic tools, and modules already developed for graphics, animation, input, sound, network and other such low-level systems.


The building blocks

Basically, the Nebula Device as we received it was an engine, not a framework. Let me explain what I mean by this. It included a toolbox of services that included:

• An object system, complete with classes, instancing, serialization and script interface
• Libraries for math, strings, data containers,
• System classes for file system, logging, time, etc.
• Resource management for graphics, sounds, fonts, etc.
• A renderer, including hierarchical scene graph, animation, particle systems, programmable shaders supporting effects like shadows, lighting, post-processing, etc.
• scene exporter for 3DStudio Max and Maya.

This might look quite complete from the standpoint of a new developer, and it certainly is. There is a lot that can be done with these systems, and they save you many hours of development, when you know that every time you need a certain type of container, a math class, or managing some graphic resource, the basics are there.

We certainly appreciated the fact of having all of these, and if you're thinking of developing your own game engine, it is definitely a good starting point to have all of these aspects figured out. You should consider getting yourself an existing library for math, strings, containers for all your basic needs, and having a ready-to-use solution for objects, resources, scripting, graphics, input, sound and network, you'll certainly gain a huge amount of autonomy.

For starters, if all you want to create is a fairly simple game, prototype or concept, these might be the only things you actually need to start working. At the beginning of many game projects, concepts are difficult to visualize, and having a fast and easy way to implement simple stuff is always helpful. Say you need to test an idea for a special type of camera that has some specific behavior. You don't need a fancy engine to test something so simple, so you can create a prototype for this in only a few hours, if some basic systems and libraries are available.

Also, you can even create an entire game using only these things if your game is simplistic from the standpoint of technology – for example a quiz game, or a music game. For these types of games, the strength of the product lies on the content, not on the technology – and so you could create a custom framework for this kind of game making use of engine features directly, without using a complex framework.

And this kind of low-level engines are also ideal to create casual or low-performance games for portable consoles, smart phones, or social networks. This is assuming of course your platform is compatible with your engine (I've seen it done with the iPhone) but the general idea is that if the cost of creating an entire engine, framework and toolset for a specific project is bigger than the advantages, you're good to go with such a simple engine.

But if your project starts to be just a little bit more ambitious, what you are missing from using such an engine more than surpasses what you already have.


The missing link

The minute you want to create anything more complicated that a brick or quiz game, developing some kind of game framework, and toolset, becomes a necessity.

Now that I've presented what Nebula was when we adopted it, let me summarize what it wasn't, and with that mean, what we had to build ourselves.

• A level editor.
• An entity system.
• A spatial system.
• A level system.
• Physics and collision systems.
• An interaction system: triggers, waypoints, vehicles, items, weapons, etc.
• An actor system: actions, controllers, AI, memory, perception, decision, navigation, etc.

If these look like a lot of work and complexity, it's because they are. Designing and implementing these modules is what took the bulk of the more than three years we worked on the Zombie Engine. And this is just a high-level overview of what this worked represented, not mentioning the huge improvements we had to make on existing subsystems, including, not exhaustively: a highly optimized terrain renderer, a material system, dynamic graphic effects, container spaces, music, video, complete inner (in-game) and outer (editor) GUI, game modes and objectives, plus an overwhelming number of other features, tools and utilities integrated in both the editor and the art exporter. And all of this, without mentioning the system that made everything so much more interesting and complicated: networking and multiplayer.


Lessons learned the hard way

I can't stress enough the importance of having clear goals when it comes to game development. Games are one of the fields of software development with the longest and most unstable of development cycles. Changes in design, requirements and deadlines have a profound impact on technological decisions, which are arguably the most expensive in terms of risk.

I only could learn this valuable lesson by making the mistake of taking the path of custom development without having a clear end in mind, for lack of experience, or criteria. But you're invited to learn from my mistakes. If you're seriously considering developing your own game technology, for a number of reasons, or getting your company involved in doing so, think again. The key question here is: why? What do you need that is not already available out there? What are the costs and the benefits? What do you gain from doing it yourself instead of buying it from someone else?

I'm not going to advise anybody agains developing game engines or other middleware. On the contrary, it's an important strategic decision for a company, and specialization is one really great decision to make.

But if your business, at least for now, is to make games, I'm sure there is a lot you can get from existing solutions that will save you a lot of time, money and effort. And most importantly, you'll build your product on the shoulders of giants, so that you don't have to repeat the same mistakes over and over, and focus on shaping your ideas using the product of someone else's hard work.

If, eventually, you, or your company, are still interested in building your own technology for the sake of productivity, innovation, or commerciality, by all means do so. Then, you could be building the next Unreal, the next Source, or the next Unity, so do not underestimate your chances at making a difference in the middleware market.

But first, make sure to know your lessons. Don't create the next Zombie Engine.


In the next few posts, I'll be covering different aspects of the development of an in-house engine, based on my experience with The Zombie Engine, and others.

The unfinished swan

It's been a while since I last wrote in here, so before I start rambling about technology (which I plan to do soon) I'm linking here one of those game concepts that really get to you: one set in an entirely colorless world where you get oriented by spilling ink on the walls and floors. A really, really interesting turn.

The Unfinished Swan - Tech Demo 9/2008 from Ian Dallas on Vimeo.