Introduction

I have previously written an article about the The making of Darkfall – One developer’s perspective where I talk about how we started our company and got from idea to finished game. This article explains how Darkfall was made from a technical perspective. I will be going pretty deep into the programming rabbit hole, so the intended audience is programmers and people that have a good understanding of systems design. Ideally you should also understand the concept of game engines having a game loop that maintains a steady heartbeat. One run of the game loop is a tick.

When we started working on Darkfall there were no mmo engines that could be licensed, or even engines that had been proven on the scale we needed for Darkfall, so on top of the craziness of wanting to make an mmo, we set out to make our own technology to run the game.

My role in the creation of Darkfall was being one of the co-founders of the company and a programmer. I worked on the early design and implementation of the game logic framework, some of the java/C++ native communication, and I was the main programmer on the worldbuilder tool. I also worked on the java-side of the animation logic, and lots of other game logic bits and pieces.

We had an excellent team working on Darkfall, and in my opinion we made great game from a technical standpoint. Even with the limited resources we had, we made an engine that could handle some of the largest pvp battles ever seen in an mmo.

I don’t want to take credit for the architecture design for Darkfall. To be honest it’s been so long I can’t even remember who came up with all the specifics, except that it was the lead programmer Kjetil Helland who came up with the main ideas for the modular design and using a message system to communicate between modules. Then Kjetil and I fleshed out more of the system. Eventually other people got involved in the design and it evolved. What we ended up with is something unique that I want to share with you. 🙂

What is Darkfall?

Darkfall is a massively multiplayer online (mmo) game with a strong focus on player-versus-player interaction. One of the most popular forms of interaction within this genre is fighting, and Darkfall is aimed at encompassing this interest, amongst many others.

Darkfall allows players to undertake a wide range of activities as a solo experience, but the game comes into it’s own when players organize themselves into parties, clans and alliances in order to fight over control and ownership of resources and cities.

Character progression within Darkfall is skill-based, which means that whatever activity you perform will improve your characters ability in that specific action. Using your sword in combat increases the sword skill, letting loose a volley of arrows from your bow increases archery skills and so on. This in itself differentiates Darkfall from most other current MMOs which tend to utilize a level-based model.

Darkfall was conceived as Ultima Online on steroids, with more races, more skills and more content, but also more directed gameplay by embracing players natural urge to want to fight each other in games.

 

Engine overview

The engine that Darkfall sits on top is called Spenefett. Spenefett is a norwegian word that means “teat cream” and is a cream you rub on cows teats to keep them soft and prevent infections. It is a silly name, and one we chose because.. uh.. reasons. We just thought it was funny. 🙂

Spenefett is a multi-language / polyglot engine that uses C++ and Java. C++ is used for rendering, physics and network while Java is used for all the game logic. Spenefett does not use a separate scripting language like lua. All the game logic is done in Java.

 

Darkfall Top-level breakdown

 

Basically, the core of the game engine runs in native C++. Anything not core to the engine runs in Java. This splits the engine up nicely between engine core and gamelogic.

One of the reasons we ended up using Java for game logic is based on an article in the July 1999 issue of Game Developer Magazine where John Carmack made a good case for using Java as a scripting engine on top of a native C++ engine.

 

Modules and pipes

We also split the engine into modules to minimize coupling. There are modules for physics, rendering, sound, network, gamelogic, gui, input, and the worldbuilder has its own module.

Darkfall module overview

 

This separation is maintained by only allowing communication between modules through messages. The messages are sent through pipes which are setup between the different modules at startup.

The modules have different states for startup, running and shutdown, and we implemented a level-system, where a module could set its run-level. This level was supposed to be used by the modules to enable or disable capabilities based on the hardware it was running on. I don’t think it was used anywhere, except maybe the rendering engine.

The pipes are essentially buffers that would add a message to a list when called the sendMessage() method. Then the receiving module would call getMessage() to get the list of messages sent in, and pull them out in FIFO order.

 

Darkfall pipe example

 

Since Spenefett is not event-driven, each module gets “ticked” in sequence for each engine tick. This makes the sending of messages asynchronous, and lets us batch messages. We will go into the details of this later on.

Lets dig further into the complexity of Spenefett:

 

GUID

Each object in the Spenefett engine has a global unique ID, or Guid. A Guid is a 64 bit long value that uses the first 16bits to indicate the node (server) it is on, the next 28 bits to refer to an ObjectSpace (we will get back to that), and the last 20 bits refer to the object id in the objectspace. This lets us use the Guid as an address when sending messages to objects, and is one of the main elements of the way Spenefett works.

On top of this we defined static Guids for certain objects, like managers and modules. We also defined ranges for the node bits and for the objectspace bits, so that base on a Guid we could find out what kind of node of what kind of objectspace the Guid is referring to.

We had a GUIDServer singleton that would produce GUIDs on request.

 

GUID bit breakdown

Messages

A Message is a base class that contains a to-Guid and a from-Guid. The to-Guid being the object we want to send the message to, and the from-Guid obviously being the sender’s Guid. Subclasses of Message define the payload you want to send. Often the message type itself is enough to trigger something in another object.

Java and C++ communicate through the Java Native Interface (JNI). Back in 2000/2001 when we started implementing the Java/C++ JNI communication layer there were no libraries that did the job so we implemented our own interface.  Between Java and Native we called these messages request and response. Request goes to native from Java, and response goes back to Java from native. We named it this way because mostly Java is requesting stuff to happen in the engine. Stuff like “run this animation”, “rotate this object”, “start this particle effect”, “play this sound” and so on. To make objects go between Java and native through the JNI layer, we had to implement the classes both in Java and in C++, and specified which fields should be converted in a separate xml-file. This ended up being a bit of a pain because we would keep defining things a bit wrong in either Java, C++ or the xml-proxy file, until one of the devs made a tool in python to help with defining new classes for the proxy. If we had started this project now we would have used one of the numerous great libraries that have popped up the last 15 years.

The good thing about using messages for all communication is that we can batch and filter messages. We use a filtering mechanism to deny illegal messages, and delay or stop messages when necessary. We could also in theory record all messages and re-run them later to reproduce bugs. We never implemented recording of messages, but it was something we considered doing and would have if we had the time and resources to do so.

On the other hand, using that many message objects means a lot of extra objects created every tick. We were worried it would become a problem for the Java garbage collector, but it turned out that we didn’t need to worry. The engine handled itself fine, even with thousands of players logged on the server.

 

Objectspace

I have already mentioned Objectspace without telling you what it is. I believe the Objectspace is one of the things that makes the Spenefett engine unique, so let’s get started!

An Objectspace is a container class. It contains objects. The purpose of an Objectspace is to define a collection of objects that have some sort of relation to each other such that they need to be able to communicate synchronously and need to be in the same address-space to function together. Objectspaces are only found on the Java-side of the engine, and are part of the game logic framework. There can be many Objectspaces on a node.

One example of objects that need to be in the same objectspace is a character. A character and all its supporting objects (i.e. backpack, inventory, skills and spells) are pretty tightly coupled and need to be able to communicate in a synchronous fashion, so they are created inside an objectspace, one for each player in the game.

One of the most important benefits of an objectspace is that it enforces a mindset upon the developer, making him very aware of the amount of coupling and type of communication between objects. This reduces complexity where possible, and we believe it helped streamline the development and debugging process for Darkfall.

Any communication between objects in different Objectspaces is forced asynchronous by message sending. Asynchronous messages are not guaranteed to be delivered until the next tick on the same node, and depending on network latency, messages may take longer between nodes. If you need objects to communicate synchronously you have to put them into the same Objectspace.

 

Communication internally and between objectspaces

 

This is important because it also means we could in theory run each of the Objectpaces in separate threads. We never did this in Darkfall (at least when I was still working there) , but it was a possibility if the need did arise. With the number of cores and hyperthreading available to us today, the engine might benefit from enabling threading on the objectspaces.

Objectspaces are controlled by the ObjectSpaceManager class. This class makes sure they are properly initialized, and that messages get sent to the right places, either between objectspaces on the same node or remotely to other nodes.

Each objectspace has its own GUID, so it is possible to send certain messages to an Objectspace.

 

Nodes

Each physical server would run one or more components of Darkfall. One physical server might run 2 gamelogic components/servers. Each of these components would be called a node and have a unique range of GUIDs associated with it. An object on gamelogic server with node Id 5, would have a GUID with 5 as the first 16-bit value. Every client would also have it’s own node Id. This would of course limit the amount of nodes to 65536 nodes, but in practice it was a limit we could live with. 🙂

There are a few different kinds of nodes in Darkfall. Most of them are server-side:  Gamelogic, GamelogicDistributor, AI, Physics, Scenegraph, Persistence, Login and Client.

The GamelogicDistributor (GLD) is the single point of failure in the system. Most of the other components could in theory be taken offline and online as there were multiple nodes of that type, but there was only one GLD. The GLD was responsible for distributing messages between gamelogic nodes and responsible for initiating server-side login and logoff processes. Part of this was loading and saving character data to the persistence server. GLD also handled distribution of chat messages.

 

Wrap up

I have given a description of how the Spenefett engine works and how it runs Darkfall. We have gone deeper into to several subjects, like the Objectspace, messages, modules, GUIDs and nodes. Hopefully it wasn’t too badly explained.

Hopefully some of the concepts are interesting enough to be explored further by others. I definitely think the way we did it was unique for a game, and it worked for us on a pretty resource-intensive mmo with thousands of real time players. I have used some of the concepts in other projects, and I know some of the other programmers from Darkfall have used the concepts in projects they have worked on after Darkfall, with success.

If you find this article interesting or want to explore the concepts further, don’t hesitate to get in touch on twitter (@bag_of_hats) or by email (ricki at sickenger.com).

Thanks for your time! 🙂