DarkStar #0: Scalable Data Storage in Project Darkstar

Scalable Data Storage in Project Darkstar

Abstract
We present a new scheme for building scalable data storage for Project Darkstar, an infrastructure for building online games and virtual worlds. The approach promises to provide data storage with horizontal scaling that is tailored to the special requirements of online environments and that takes advantage of modern multi-core architectures and high throughput networking.
After a brief overview of Project Darkstar, we describe the overall architecture for a caching data store. Then we provide more detail on the individual components used in the solution. Finally, we suggest some of the additional facilities that will be required to bring the full experiment to completion.

P.S. Wiki:Project Darkstar


"The Project Darkstar infrastructure attempts to present a
simple programming model for the developers of the
server-side of such games and virtual worlds. The goal is
to allow these programmers to write their code as if that
code were to be running in a single thread on a single
machine. The code must be structured as a set of tasks
triggered by events sent from the various clients of the
game or world. The infrastructure will then take care of
distributing those tasks over the available machines and
threads, and running those tasks in such a way that
the consistency of the data will not be compromised.

The scaling strategy used by the infrastructure
centers on the properties of these tasks. The tasks are
required to be compiled into JavaTM bytecodes, and
so the code for the tasks can be run on any machine
in the server network without changing the
semantics of that code. All of the tasks are run inside
a transaction, allowing the underlying infrastructure
to detect conflicts in attempts to access and
manipulate shared data. When such a conflict is
detected, one of the competing tasks will be allowed
to continue, while any other task that was in conflict
will be aborted and re-scheduled for running at a
later time. To enable the infrastructure to detect such
data conflicts, all of the data used by a task that
might be visible to another task must be placed in a
data store that is managed by the infrastructure itself.

By having the data store be a resource shared by all
nodes running a game server, we also insure that all
of the data within the store is available on all of
these nodes. This means that we can move tasks
from one machine to another to balance the load
over the network. But this also means that the data
store is a central component of the infrastructure
itself.
"

"The system does not guarantee that all modifications will
be made durable, but it does guarantee that any durable
state will be consistent with the state of the system as
seen at some earlier point in time."