Monday, July 10, 2006

10 MMOGs you shouldn't make

10 MMOGs You Don't Want to Make

(originally published in Develop magazine at the 2005 GDC)

The buzz about massively multiplayer gaming oscillates like a broken TV set. Last year all we heard about were MMOGs being abandoned, even in development; this year has begun with World of Warcraft proving "too" successful for its servers to cope.

The MMOG genre does have some peculiar qualities, which suggest it really could cause a sea-change in the business models of developers. One of the most alluring promises is that studios can create an ongoing revenue stream that requires no marketing(everything's viral), no distribution (everything's downloaded) and where revenues are predictable (a fat cheque being deposited in the studio's bank account each month, without fail).

Who could fail to be excited by that? Better yet, MMOG game design was originally thought to be simple RPG fare (familiar to developers from past projects), allied with the implementation of simple FPS-style 3D engines with humanoid characters running about a pretty LOD-managed landscape. Easy! Except ... it's turned out to be much more complex than that, not least because of several cunningly disguised pit-traps that studios keep falling into (some even jumping down the same trap, several times in a row). Here's a guide to what NOT to make:

1 The MMOG which is also an Engine

The Idea

Make a game and some MMOG middleware at the same time. If one is a commercial disaster, the other will make up for it.

Why it seemed a good idea at the time

Making a successful game is very risky, and making a new genre of game which requires lots of tenhology you have no experience with is even riskier. But selling middleware to other people is, surely, risk free. So balance the risky development wit hthe risk-free one. If the middleware is any good, it can be licensed for millions of dollars a time, just like id Software did with its Quake engines.

But you didn't realise...

The main reason that here's not much good MMOG middleware around is that it's very hard to make. Most middleware sales will probably fall through, yet they are costly even to attempt, making it a high-risk business. Hardly any less so than selling the game.

So what actually happens is...

You thought you'd be working on a game and getting a 'free' middleware product at the same time for almost no extra development cost. In fact, you find you're trying to do two development projects on a budget intended just to cover one of them. Worse, the middleware ususally costs much more to develop than the game, so you've only got the budget to fund the lesser of the two projects. And selling middleware is far from risk-free. It may take six months of hard negotiation to sell just one licence. Anyway, most people won't even buy your middleware because your rushed game "sucked". So you'll need to develop a second, better, game to convince anyone to buy your engine - or else write off a huge waste of money.

2 The Server-less MMOG

The Idea

Networking is fairly simple. We'll get on with the client, and hire someone towards the end of the project to do all that networking stuff.

Why it seemed a good idea at the time

With experience in writing multiplayer games (eight-player RTS's and 16-player- FPS's) you might think networking is simple, so long as you know the pitfalls, how to minimise latency, and so on. The techniques are easy to re-use and almost universally applicable. Also, when it comes to publisher milestones or gaining funding, it's not an invisible server that convinces, it's a flashy graphical demo. It's not worth spending on something you don't need (yet).

But you didn't realise...

There really aren't many people who can write you an MMOG back-end from scratch, even in a couple of years. There are very, very few teams who'd have a chance of coding a simple basic working system in under six months - even without most of the features.

So what actually happens is...

You discover late in the project that your dev servers are utterly useless under real-world loads, and that no-one knows how to make a better one. You hire someone who works in distributed systems to make the server, and they can't believe you need as much done as you tell them you do, so they assume you don't really mean it. When six months later you ask them for the working system, they laugh. Then, slowly, the realisation sinks in and you stare at each other, realising you've either got to ask for another 12 months dev time, or cancel the project (and the client dev was going so well!)

Alternatively, you outsource the server development early, and when they go bankrupt (or just decide they cannot feasibly complete your game-server and choose to bow out), you find yourself with a beautiful tech demo that has no logic (no server system), just a few months before launch. Again, beautiful client, no server - and no-one can possibly get you out of your hole.

3 The MMOG in a Single-Player Retail Box

The Idea

Aim to make most of the sales in the first three weeks, at retail, just like you would with a normal game. All the marketing is arranged for the big, spectacular, launch.

Why it seemed a good idea at the time

You can have your cake and eat it: the multi-million income from the firs few months will pay off most of the development costs, maybe even providing profit - but then youu'll get masses of profit over the coming months from the subsrvier income. Server-hosting costs and support costs (customer support, administration, in-game management) are terriffyingly large, but big launch sales will pay for this. Anyway it's even easier to patch an MMOG post-release than it is to patch CD-only single-player games; as all the players have to be online with decent connections just to play, force-streaming updates will be easy.

But you didn't realise...

The last two MMOGs to launch before yours gained twice as many players in the first few weeks as you'd stress-tested for; there's no way your servers are going to cope. When a server overlaods, but he time it's rebooted, a backlog has built up. One server crash soon brings down all the others, as the backlogs get passed on from server to server.

So what actually happens is...

You only tested your systems for "average" load. On the first day of release, with 30,000 players waiting to log in at once, the 24-hour average may be within what you expected - only they are all going to arrive in the first hour, giving you a peak 20 times what you expected. You spend the rest of eternity fire-fighting, spending all your mopney on trying to fix bugs, with non spare to increase your staff. As a result everyone is always overworked and you never make any progress.

4 An MMOG that wants to be a Single-Player Game

The Idea

Develop an MMOG as if it were a single-player retail title. Most of the efoort will go into the client-side engine, the game design and the client-side testing.

Why it seemed a good idea at the time

You had no idea what you were doing.

But you didn't realise...

You therefore had no idea what stress-testing was. You had no idea why developers complain about "general non-determinism" and "non-deterministic failure modes". You thought that MMOGs could *and should) be written single-threaded, running on a custom soft Real Time scheduler, like most computer games.

So what actually happens is...

Sooner or later, the soft brown sticky stuff hits the rotating cooling-device, and the result is not pretty. The next 12 months will be a nightmare if the project isn't cancelled immediately.

5 The Silent MMOG

The Idea

Community management is hard (and what the hell does it mean, anyway?) and all the players are evangelists. Policing communities is near-impossible, and they'll all chear and use AIM/email/teamspeak. So you let them organise themselves.

Why it seemed a good idea at the time

Analysing MMOGs and trying to be cunning to avoid the hard problems through better design reveals that most of the headaches seem to be about resolving "griefers". Most conclude that static balancing a game with thousands of humans and cheaters is almost impossible.

It's also often noted that players are the best testers. They will work out every aspect of your game: every algorithm, every secret and every loophole. So why not use this as a force for good? Pit players against each other. Those who would normally try to understand and break the system can do so, and be given the power to fix it. A self-organized game-community (a "game without rules, save those you make yourself") sounds like a great USP.

But you didn't realise...

The volume, skill and dedication of people who want to "make the (game) world a better place" are all vastly outweighed by the volume, skill and dedication of those who want to spread chaos. Spreading chaos is just so much more fun, it seems - especially if getting caught depends upon nothing but your own skill.

So what actually happens is...

Local governments come down on you for failing to police laws on hate-speech, rape fantasies and so on. The game-world becomes a wasteland because the player-run society is fundamentally unstable, and the players soon break it beyond all hope of salvation. Customer support and refund costs go through the roof; class-action lawsuits appear on the horizon.

6 Massively-Multiplayer-Simulation

The Idea

Instead of faking and shortcutting, just simulate everything.

Why it seemed a good idea at the time

Even with 60 game-designers producing new content as fast as your 500,000 players can play through it, it's going to be very expensive, if not impossible, to keep the players occupied. Also, the developers are always blamed for all unpopular gameplay decisions and any random events that are "unfair", because the players "know" it has all been pre-ordained.

Instead, by using mathematical models and letting emergent behaviours take over, you will provide a never-ending, always-changing game at little or not cost - and changing yourself from the players' enemy into their ally.

But you didn't realise...

While players love to play with your simulation, by "play" I mean "see how many ways they can break it". Imagine a four year-old left alone with a hand-built model of the Cutty Sark for half an hour, wondering how it's put together.

Simulations never work in the long-run. No-one has yet produced a true, non-trivial simulation that the players can't quickly wreck.

So what actually happens is...

The players collude and destroy your simulation. Just like the Millennium footbridge in London, where every single pedestrian subconsciously started walking in time until huge vibrations built up, your players will all do at once whaever one thing is the most damaging thing to do to your system. It's like everyone on a ship running over to one side to look at the dolphins, and so causing it to capsize. Except that most of your players probably did it deliberately.

7 The Object-Oriented MMOG / DOA MMOG

The Idea

Using the design rules and 20-plus years of OOP design and development experience, you will make a clean, three-tier architecture - all nice and neatly encapsulated, and so easy to manage and maintain.

Why it seemed a good idea at the time

OOP is good, OOP reduces maintenance costs and makes design simpler. An OOP-esque server is very easy for humans to reason about.

But you didn't realise...

A server architecture that is easy for humans to reason about is almost impossible for a cluster of machines to execute efficiently. Fundamentally, the system that is most natural for humans to invent is one with huge amounts of abstraction, which means high-latency.

So what actually happens is...

Even on your local LAN, the latency to get the server to do simple calculations is being coutned in the tens or hundreds of milliseconds. This is before you add-on the 50-250 ms roundtrip-latency imposed by general internet connectivity.

Everyone wanders off, bored.

8 Everquest Evolved: The Best Bits

The Idea

Take Everqyest, copy it and fix the ten most "obvious" mistakes they made in the design.

Why it seemed a good idea at the time

Everquest (EQ) was the benchmark by which all other MMOG successes were measured, and it continues to be spectacularly popular and commercially profitable, despite its age and simplicity. But it had lots of small problems that turned out to have a huge negative impact on players. With the benefit of hindsight, these can ALL be fixed without too much effort.

But you didn't realise...

In the main, the hundreds of thousands of people who continue to subscribe to Everquest are not doing so because they like the game, but because they are already there. They have invested thousands of hours of play in building up their characters, and have large social networks that exist mostly in that game.

So what actually happens is...

Your game takes many months to develop, and by the time it finally comes to market, it's panned by critics for being ugly, simplistic and utterly indistinguishable from the rest of the crowd, not to mention EQ. Your game makes few sales; and the EQ players carry on playing EQ and bitching louder than ever. This despite the fact that your game fixes ALL the problems they are bitching about.

9 The Players MMOG

The Idea

Make the perfect MMOG, determining throught lots of player interviews and from discussion boards what players actually want.

Why it seemed a good idea at the time

Everyone is trying to make a mass-market MMOG. Sony pinned its hopes on the huge appeal of the Start Wars licence, and it failed. What's the secret? Well, every game has thousands of fans bitching constantly on the boards, in public view, often with very similar complaints - and for every one who complains, there are probably 100 more who don't bother. You plan to listen to them.

But you didn't realise...

As Richard Bartle has recently pointed out, the things players complain about bear little actual relationship to the things that need to be fixed to make the game better.

Moaning doesn't qualify players as game-designers. Why is it that after years of confidently telling players that their "brilliant" new game ideas have already been thought of hundreds of times by professional developers, we suddenly cave in and act as though vociferous jobless students who spend all their life playing one single online game are somehow automatically better at seeing what makes a good game?

Besides, the best community-building force is shared suffering. Without suffering, it's harder to rouse people's emotions and harder to get a real community going.

So what actually happens is...

Your game gets a lukewarm reception on launch and none of the viral marketing gets you many players. Your die-hard supporters are few indeed, and most people condemn you with faint praise: they simply never got hooked.

10 The Outsourced MMOG

The Idea

Outsource all the stuff that is unique to MMOGs: clustered servers, in-game custoemr support, monthly billing, even network layers. Concentrate on the stuff you know best: gameplay and making fast, beautiful, clients.

Why it seemed a good idea at the time

Analysis of previous MMOG failures shows up some common patterns. The most common is that while the 3D graphical client is perfect and the game greate, the non-traditional stuff (launch, customer support, patching, server management, bandwidth provisioning, and so on) fails for simpel reasons.

MMOG's don't fail because of incompetence by the developers, but rather ignorance in certain areas. So you will avoid those areas entirely.

But you didn't realise...

It's out of the frying pan and into the fire: you're turning your studio into an outsourced programming contractor. You might (finally) have escaped the strictures of being beholden to a publisher, only to put your masterpiece into the hands of a third party just as soon as initial development is complete.

And once they've got you by the short and curlies, they ain't gonna let go.

So what actually happens is...

The middleware providers suddenly get very unhelpful and start blocking each and every request for improvements to the system, because if they do nothing they make lots of money with little expenditure - and so it's just not worth the hasssle to them to change things. You lose creative control over your game: someone else is doing the in-game administration, and all the promotion and customer-support that seemed so much hassle suddenly - very obviously - becomes the major thing shaping how the game can develop.

Worst of all, you've got no contact with your customers; someone else is billing them. It's their logo on the monthly statement, their contact numbers, their branding on your success. They have an exclusive, free, audience of paying customers (of hundreds of thousands) to whom they can pump large amounts of advertising and cross-selling.

Weep as you watch them displace you entirely in the money-making side of the game.

Sunday, April 09, 2006

Java and Databases - the Great Disaster

There is a great tragedy in the world of Java that rarely gets discussed. All around us, every day, the horror continues, yet we pretend it's not there, even as we stand in the shadow of the towering monsters it begets. The horror mocks and humiliates java programmers across the world, constantly pulling the rug out from java and it's Great Promise: to rid the world of the evil's of C. We won some battles - no more header files! no more malloc()! - but for some of us it seems we may have lost the war. I'm talking about JDBC.

What is C?

C was not some rosy nirvana of a programming language, devised over the course of 20 years by the brightest and best theoreticians and experts around the globe. It was a small language put together quickly to solve a certain domain of problems. In short, in many many ways, it sucked. It sucked even more so when people started using it as a Universal Hammer ("when all you have is a hammer, all the problems in the world start looking like nails...") and it was exposed to vast swathes of usage that it was never designed with in mind.

C++, and later Java and C#, in correcting a vast number of those problems (and, let us not forget, borrowing/stealing a lot of good ideas from other people's attempts to fix C - Modula for instance), sparked a legion of religious wars, from "Object Oriented Programming is too slow for the real world!" to "Garbage Collection is a waste of time!"... you've seen these arguments around (they don't die easily). C++ initially spearheaded the OOP revolution (now cut and dried), and Java/C# started the managed-memory/VM revolution (not entirely finished, but now comfortably in a position of world domination). Let us not forget C had neither.

C didn't even have the concept of namespaces. For non-C coders, imagine what life would be like if you couldn't put your classes into packages. If every time you wanted to make a new class, you had to think of a name for it that *no one else in the world had ever used for their class*. Nightmare? You bet.

What is JDBC?

Well, in a nutshell, it's " java".

What is ODBC?

Open DataBase Connectivity - this is nothing more or less than a strictly defined interface between applications and databases, that enables you to switch your database without having to rewrite your application. It came about in 1992, at the end of the eighties, where a lot of people had lost an awful lot of money from being tied-in to a single DB vendor and yet being unable to switch to a cheaper/better one (because they couldn't afford to rewrite all their application code from scratch to work with the different DB). Bear in mind this is also back in the days before OOP's world-domination, and long before the term "refactoring" had even been coined, so applications were generally much harder to switch to new systems.

Hmm. Hold up a minute there ... ODBC predates the OOP revolution. ODBC ... isn't OO. OOP has revolutionised development, *especially* when it comes to systems interfacing and data/code interactions and handling. i.e. ODBC doesn't use the most important invention ever made in it's own domain. In short, as far as 21st century application and systems development goes, ODBC sucks.

Java, the Giant-Killer

Java's core architects and early developers (Sun, IBM, BEA, etc) didn't shy away from turning conventional wisdom about VM's, GC, etc on its head. C# probably wouldn't have happened for many years to come had Java not got out there first and spent half a decade proving that, if you did it right, these concepts worked really really well in practical, real-world, programming.

Speak to developers who've spent a decade using Java professionally, and you'll generally find that improvements such as this are what have kept them with the language all this time - the oft-quoted (and oft-disparaged) claims that Java can reduce project total development time by typically 20%-30% compared to writing it in C++ are generally accepted and believed by these people, who've seen the same kind of time savings again and again. And it's fundamental improvements like this that provide those savings.

For instance, have a look at the Java 1.0/1.1 networking libraries. Compare them to C++ (don't even bother looking at C, it'll just depress you). Java was the first mainstream language to have "the internet" built-in to the core language. The standard libraries were full of things like the ImageObserver class, specifically designed to let you asynchronously download images over the net (back in the mid nineties when most people had very slow dialup access). There was an entire core library devoted to using the internet, and *it made the TCP and IP packets for you*. It actually arrived speaking HTTP out-of-the-box. (Of course, this was a major pain for all the people e.g. writing computer games, who actually *wanted* to be manually assembling their IP packets).


So, when it came to doing a library for possibly the most important thing in the computer world - accessing databases - what did Sun do? Did they rail against the status quo, and start with new technologies that would drag people into the new century, a fresh start that fixed all the problems of the last decade, and made DB development "easy", "quick", and "reliable"? Hell no! They ran away from the problem, and hid. They just pushed people towards a port of ODBC. A good port, yes, with lots of goodness of its own. But, fundamentally, ODBC again - with all the attendant problems of completely ignoring what OOP actually means, for instance.

Yes, JDBC is an API that uses classes, objects, etc. But ... have you actually looked at those objects? Some of them are barely even usable (which idiot thought it was a smart idea to make a new class that would represent a java Date as another class, *also called Date*, and would simply wrap the java Data class? Suddenly, all your code has to be full of fully-qualified class names because of a stupid, short-sighted decision).

Fundamentally, OOP programmers aren't interested in working in byte arrays: they want Objects. They don't want to write convoluted relationships in SQL: they want to use the nice, terse, clear language *they are actually writing in* to describe relationships.

Is there Hope?

Yes. It's called JDO - Java Data Objects. And Sun's been trying to kill it off, because the only thing Sun understands is J2EE, which means they CANNOT allow something as useful as JDO because their J2EE stack already duplicates and/or architects differently a lot of the fundamental concepts of "data", "code", and "development". This is all "in a good way", just so long as you are a J2EE developer. J2EE is great for the things its designed for. Sadly, for J2SE developers, J2EE can only handle a VERY small fraction of the programming tasks that java itself is excellent at.

So, Sun's not at all keen on "these JDO thingies" that seem to them to have no purpose. But they are quite possibly the ray of light that will be the salvation for Java programmers everywhere, the future of DB development. The two questions now are: will JDO survive the cull, and ... how good is it anyway - does it fulfil its promise?

More to follow...

Wednesday, March 22, 2006

GDC 2006 Day 2

Alcohol units: 10

Parties attended: 4

Parties crashed: 2

So, firstly a thank-you to CDV for their quiet-but-nice Pirates party, which just happened to be right next to the Fierce Develop wireless (mobile, for all Europeans) party, and on the way from there to the rave-tastic Autodesk party. Lots of gorgeous 21"+ monitors showing the pirates game - looks great, but really dudes, you need to put the game's name + developer on the walls, not just your name and your previous published games. We blagged our way in as pseudo-press, and I loved the game, but I *still* don't know what it's called!

Secondly, FierceDeveloper was useful, I made some really good contacts with people in the mobile gaming space in USA - sadly, relating to internal MindCandy projects that are far too confidential for me to describe here, but the party itself was good. Thanks are due to the MaryMargaret folks for sponsoring the party and bringing us together - excellent networking opportunity indeed, and the traditional early 20th century band was ... novel, as far as GDC parties go.

PS: the barmaids were by turns incredibly gorgeous and rather scary. Or both at once. Um. I'm pretty sure one was flirting outrageously with me (6 free drinks?) and so a big shout out to Lori - you're lovely :).

First and foremost, of course, we have the IGDA party. Jason, you are a cruel, cruel man. You place half a dozen dishes of food in front of a starving audience - who, I must point out, were told to "arrive early in order to be sure of getting food" - and then proceed to force-feed them half an hour of talks before the waiters are allowed in to open up the food :).

Anyway, the IGDA is going from strength to strength - a very impressive graph was shown demonstrating both exponential membership growth year-on-year (network effect taking hold?) and that we were about to hit the 10,000 member boundary. So, everyone - join now, join often; the IGDA *needs you!* Seriously, it does some great work, and the more members the greater its clout in improving things for all of us (like a union, only a lot less formal and IMHO much better ;)).

Finally...the Autodesk party. Wow. As one of the girls put it:

No, you don't know how my clothes work. Actually, the box on my back contains a colony of fireflys. I change colour depending on their mood (its not optical fibres, despite what you think), and the blue ones are really rare - they were imported from Madagascar.

Tuesday, March 21, 2006

Developing a new game

Part 1 - vision, planning, schedule

Sitting in a GDC session this week, mildly phased out from a combination of jet lag, intense concentration, and sitting in large dim rroms packed with people for hours on end with the faint buzz of lighting and big speaker rigs, I had the inspiration to put together a new game. Since I’m at a conference, which is all about sharing, I thought it might be interesting to some people for me to publish the documenting and development as I go along.

So, here goes...


Mandarin Invaders - a space-invaders game that forces you to learn to speak Mandarin simply by playing the game and trying to get the highest score possible.

There is NO teaching in the game itself - no dialogues, no lessons, no objectives - it's like playing a game aimed at a Mandarin speaker, that a non-speaker struggles with, and gradually learns-by-experimentation the vocabulary and meaning of the language.

Space invaders is the simplest form of collision-based game I can program in the shortest time possible - I'm a busy guy, I've got half a dozen exciting and cool projects on the go, and if this is to happen at all, I need to get it done in under 50 hours start to finish.


This is where my personal background and experience comes in. I've managed small games development for half a dozen real games, some commercial, some free, and tried out various techniques along the way.

The first issue is to assemble enough info for a Game Design Document. Gamasutra has an excellent article on GDD's - - which we used as the basis for everyone who came to us at Grex pitching ideas for new MMOG's, and wanting to use our technology when it was in alpha (a big commitment for us). I reviewed approximately 50 game designs this way, and found that a good design by a designer who knew what they were doing typically filled 1.5-2.5 pages.

Here's a dump of my mindmap notes for this (NB: download Freemind - free mind mapping software with excellent keyboard nav defaults that make it trivial to write out extensive notes very very rapidly. I've got courtesy copies of Microsoft's latest and greatest note-taking software, and it sucks donkey; it's really awful, and misses the trick completely when it comes to UI simplification and ease-of-note-taking. MS needs to hire some more good HCI people. Again.)

Mandarin Invaders

  • standard up-scrolling schmup
  • some ships friends, others foes, only differentiated by characters
  • bonus points for collecting items in numerical order, e.g. yi then ar then san then ze
  • labelling objects with individual characters, and gaining/losing powers based on the objects you run into
  • bosses have obvious visual weaknesses that require modification of the ship via choice of powerups
  • when a ship is hit, or a powerup collected, it is "triggered"
  • triggering a thing causes its effect to happen AND visually reveals the meaning of the word
  • e.g. for powerups they simply take effect obviously, but accompanied by the powerup itself briefly animating iconicall
  • e.g. when collecting fire powerup, there is a brief animation of flames that underlays the player's ship
  • triggering ALSO causes a sound-sample of that item to be played out loud, as pronounced correctly in mandarin

For a small game like this, where there will be very little innovation in the *technology*, and the problem space is very well known, I've found that semantically described milestones are the best starting point. Over the years, its become easier and easier to invent the milestones for a new game from scratch - the first few times, I found it painfully difficult to come up with anything meaningful.


  1. blitting of a ship sprite onto a texmap BG
  2. ship moving on screen under user control with gameloop
  3. invaders pathing on screen on bezier and straight line segments
  4. invader paths loaded from XML files
  5. collision detection between ship and invaders
  6. player death on collide and limited lives
  7. OO-events programmatically-walkable list for entire play of level
  8. invader death and score changes (collision response)
  9. chinese characters as graphic for ships
  10. powerups using chinese characters as graphic
  11. voice recordings of mandarin for all characters used in game
  12. (optional) simultaneous pin-yin display of characters when voice recording plays

Additional notes

M7: this is used for a load of cool features, several of which are going to be fundamentally necessary for game core features, in that they make it easy to implement said features. Essentially it's a tree like the DOM (Document Object Model) that records everything that's happened in the game, chronologically, enabling game-logic scripts to have simple logic such as:

if( last_enemy_killed.type == all_enemies_killed[ 1 ] ) then bonus_multiplier += 1

...which causes you to get increasing numbers of points for killing the same type of enemy

  1. end-of-level stats
  2. OOP-navigable list
  3. aliased multiple refs to identical items
  4. e.g. "last event" and "last invader death"


This one makes extensive use of the OO list of events, as described above

M9, M10, M11, M12

Fortunately, my girlfriend is a native Mandarin speaker, so I'm hoping these won't prove too tricky to get translations and voiceovers for.

Artwork I'm hoping to do myself primarily by exporting from unicode fonts and converting into 3D objects (simple extrude).


Not necessary to make the game "work", but pin-yin is extremely powerful in helping non-natives distinguish between very similar-sounding elements, such as s/z/c etc.

I saw a quote on Sun's javagaming forums, and felt it fundamentally pointed out a problem a lot of people have with understanding why we would ever want structs in java. It underlies a fear that structs are dragging java away from it's "managed" approach to memory. The truth is:

Structs do NOT undermine java's automatic memory management, they make it even better

Fundamentally, those of us who want structs LOVE java's automatic memory management, and we hate having to write C/C++ code any more because it's so damn painful to go back to the mad, bad world of malloc(). But we've found a place where java only does HALF the normal amount of memory management, and we want to fix it. It's not possible to just use the mechanisms that JVM's use internally, and apply them directly to this (or else Sun would have done that in the first place, and not left this only half-done!). Something has to be added. Structs is that thing: it makes it possible for us to impose JVM memory-managing goodness on all the dirty non-JVM memory in a modern computer.

The Misunderstanding...

Quote: "handling memory is something the jvm should do, I have nothing however to giving the jvm hints . if I wanted to pricisely handle memory I would have chosen an other language."

In the kindest possible way, there is something fundamentally wrong with that declaration :). It's not, perhaps, massively important to the speaker or the rest of the world *right now*, but its a statement they are almost guaranteed to retract in the future, when thet find the opposite turns out to be the case.

More importantly, it keeps coming up when people-who-dont-know-why we want structs, let alone *need* them, try to talk to those who do. For that reason alone, I'm going to try and explain.

Programs, OS Imperfection, Location, and The World

From it's earliest days, java has had a problem with self-isolation: Sun brought it into the world and deliberately or accidentally presented it as a very self-contained, theoretical language which you could write programs in theoretically, but which didn't exist as a concrete system. For instance, back in the days of Java 1.0.x when I first started, 95% of the tutorials I found gave you 5+ pages of guiding through writing a java program, but:

...they didn't tell you how to run them.

This may not seem like much, but they would completely dodge the question of how you could get a machine to run a java app. As a C coder, I was accustomed to a process that went like this:

  1. Write app
  2. Compile
  3. Link
  4. Type name of program at dos prompt / double-click on the .exe file

In java, I clearly had a compiler - javac - but it produced a .class file. Further, the tutorials said the compile process was different with Java, that java did automatic linking at runtime:

  1. Write app
  2. Compile
  3. Run

But compiling java source files has never (and hopefully never will!) produced .exe files, for several reasons, the biggest of which is that .exe files are not cross-platform and probably never will be. If you're reading this as a linux user or OS X user, for instance, you'll be wondering why I was after a .exe - you'd be able to do about as much with that as I could do with a .class file back then.

Why? A: Location, Location, Loc...OS

Java was the first major commercial (i.e. non-academic, non-niche) language to use a true Virtual Machine. Personally, like many others, I'd used two types of languages: Interpreted, and Compiled+Located/Linked. The latter were typically known as Compiled for short, and that's the same usage as today, although I think there's a reasonable argument that perhaps the value of C# and Java would be much better understood by people who don't use them if C were called Compiled+Located/Linked, and they were called Compiled.

It is not possible to link a java application without compiling it to a non-cross-platform EXE.

NB: this is a gross simplification...

There are many good reasons for using a VM, but one of the biggest and the one relevant here is that the hardware CPU which is executing the compiled java code is identical (which is the benefit people usually talk about when mentioning this feature(1)), and:

it is running a carefully-designed virtual Operating System which is identical no matter what OS you are actually running on your desktop/server


When you declare a variable and use it, the compiler has to turn that into assembly instructions. The massive, frustrating problem is that CPU's can't talk in terms of named-variables at random places in memory: they only understand absolute addresses. You have to say exactly which of the 1,024^3 sets of 8 electrical connections in your RAM chip is the one which is to be changed. So:

A = 1 + 2

has to be rewritten (before the CPU can execute it) as:

RAM-chip-1.cell-643,234 = 3

where the bit after the dot specifies the exact location. Obviously, there are two problems here:

  1. Not everyone has the exact same amount of memory, so there are some cell addresses that exist on some computers but not on others
  2. If every program were hard-coded to use the same set of memory locations every time it ran, you'd get programs that couldn't run simultaneously, because they were both hard-coded to use the same sets of cells on the RAM chips

There have been several different "solutions" to this problem put forward by CPU manufacturers and OS developers over the years, and nowadays you have several to choose from. Each has disadvantages, and hybrid solutions are common. For the most part, application developers don't get involved, although some of the technologies which "solve" the problem are ones they use on a daily basis for other reasons (e.g. per-process virtual memory - one of the side-effects of giving each process "virtual" memory is that each app can be compiled to assume it is the only app on the computer, and that it has an unlimited amount of RAM (independent of how much RAM is actually in the computer)).

The one that Java offered was simple: The CPU, OS, and RAM you are compiling for are identical on every computer, even though the real computer in each case has a different CPU, a different OS, and a different amount of RAM. It's very similar, in effect, to Virtual Memory, but it's one step further: it's a Virtual Machine (Virtual-Memory + -CPU + -OS).

All of a sudden, many important big problems in the world of software development vanish, in theory. This is because most of the really big problems in the world of software development today are, at their source, incompatibility issues and interfacing issues. And that is something that is very well known, and has been for years, and has been often cited by people like IBM as why they jumped on java when it came along. C# is just as good at this in every regard, it's just that Java was first ... and wasn't owned by Microsoft.

So, everyone loves the VM. Lots of problems are solved - or, if not automatically solved in Java 1.0, everyone can see how those problems WILL be solved fairly easily (only one company - Sun - has to solve them, and the solution will work for everyone) in future updates. No-one needs to worry about Hardware anymore. No-one has to worry about OS's, POSIX compliance (or, in fact, the big problem of POSIX: few OS's being as POSIX compliant as they ought to be / claim to be), or endianness (are you a BigEndian or a LittleEndian?), or any of the annoying details of accessing RAM hardware that keeps changing fundamentally every 3-4 years (SIMM, DIMM, DDR, Rambus, etc).

...and we've come full-circle: java developers got used to the idea that the only machine that mattered was the Virtual Machine. Java evangelists preached the cult of the VM, and often didn't bother to explain things in terms of the actual machines, because "you don't need them any more; cast off your chains, and start working with a VM".

But the story, I'm afraid, doesn't end there...

Mid-air collision: VM, jumping from a high height, meets Real World. And it's pretty solid.

One of the first things I noticed about academic teaching of programming (on Computer Science courses) is that you are largely not expected to interact with the real world. That includes not outputting to the screen, nor even taking keyboard input! What matters is that the computer has been programmed to generate the answer *internally*. In C.S. teaching, the answer to the user's question doesn't matter, except as proof that the real question - getting the algorithm and source code right - has actually been answered. (NB: which is fine, it's fit for teaching. It's just that it's not fit for getting any work done in the real world)

Try getting a functional-programming language to output a progress bar whilst it works on an hour-long calculation.

On a theoretical level, coding to a VM is much the same: you have declared that you will *not* interact with the outside world, and, ultimately, that means no keyboard, no networking, no monitor. Java's VM has, just about, managed to pull keyboards, networks, and monitors into the VM. But... the problem is that it hasn't pulled *everything* into it. And, practically speaking, it never will. To date, Sun has added the most important handful of things into the private universe of the JVM, but they've got better things to do with their time than bring everything in (if we were talking about Microsoft, with the 2,000-programmers-working-on-.Net, it might be a different story. But we're not).

The importance of Memory

Memory is the interface between processes, applications, systems. If you can't bring something into the VM, the minimum technology support you require from the OS/VM is the ability to speak to some shared memory space, allowing you to send and receive messages from that other thing. Sun chose to add deep, high-performance support for memory interaction with Java 1.4 (the NIO packages) because they knew that improvements to the memory interaction unleash a lot of potential for developers to create their own bridges to other applications and systems. The most exciting use of NIO so far has been the addition of hardware-accelerated 3D graphics to Java with practically no overhead (there is overhead, the interfaces aren't yet perfect, but the end performance penalty for a 3D game is very small).


How did Sun add Hardware-accelerated OpenGL to the JVM, cross-platform?

That's an awful lot of Device Drivers they had to write. Even Nvidia's and ATi's drivers for windows are still under massive active development, let alone other OS versions; how did they manage it?

They didn't; they simply made a bridge from the JVM to native memory in the OS, and provided a simple means for a developer knowledgeable in the ways of OS memory to hook-up the native-memory that in a modern OS happens to correspond to the 3D graphics card. Other developers then came along and did the hooking-up. All OpenGL implementations in Java use the bridge (NIO) as the way that you pass data in to the OpenGL API, and get it out again - you are writing directly to/from the graphics card, because behind the scenes the library you're using is doing the memory-handling that the JVM cannot do, unless the JVM had device drivers for all this stuff.

The VM's (virtual) memory is handled by the VM. Period.

The VM's (non-virtual) memory is handled by the OS. Period.

And so, ...

  • Until the VM incorporates every single application on the planet as a piece of sub-code (i.e. whilst there are still applications that don't run in the VM),
  • ...until the VM incorporates ALL hardware ever made (i.e. whilst there is still hardware that does NOT have a "Java Device Driver" as well as a windows/linux/OS X Device Driver)
  • ...until every single computer on the Internet is running a Java OS, and all the JVM's are linked together

most memory will be outside the VM, and hence not physically possible for the JVM to handle.

So...what are we still missing?

The memory interfacing in java, as provided by NIO, is still lacking in a few areas. It's a hell of a lot better than what went before - it's easy to use, the API's are well-thought-out, and (depending on JVM vendor; free ones aren't anything special at the moment, but are improving) they're able to provide extremely high performance. The "java structs" campaign is about fixing / adding the most important of the missing / broken pieces of the puzzle. This post isn't really about what they're trying to achieve, it's about explaining the importance of memory, memory handling, and why it's so important to have access to memory that is NOT handled by the JVM. However, I've got some notes on one of the problems, which I'll copy/paste here. I'll probably remove this and write it up into a followup post.


TCP/IP / UDP/IP has become the easiest way to connect applications together that run on different OS's. It is, like it or not, the Universal Language for computers. These days you can guarantee that every computer talks TCP/IP, and all the structures, the grammar, and the vocabularly are 100% fully documented, explained, and discussed all over the net and in thousands of books.

At it's heart, you are writing to and reading from memory. It just so happens that when you write to memory, what you wrote is put into an IP packet and sent across the network, and when you read, what you read has had to come out of a packet. This is a huge amount of work, especially with TCP, and is (thankfully!) fully handled by the JVM.

What the JVM doesn't interfere with is the data you're actually accessing. The JVM has no way of knowing whether you're talking to a 20 Tb database on a 5,000-machine cluster on the internet, or just sending tiny messages back and forth with your friend using an instant messenger.

Until it does, you as the java programmer have to do all the handling of that memory. Because IP is inherently a serial network - you send bytes one at a time, one after the other - pretty much every specification of every network protocol used by every application everywhere is written in terms such as:

  • First 20 bytes: (player's first name)
  • Second 20 bytes: (player's location/city)
  • Next 2 bytes: (colour of player's shirt in-game)
  • Next byte: (whether this player wants other players to be able to see their name when they play together)
  • Final 50 bytes: (message that the player is sending to other players, and is the reason this network message is being sent)
  • Total: 93 bytes

...i.e. a fully serial specification of the data, ordered deterministically. It's common for the data to be of fixed length ("Total: 93 bytes") independent of what data the application is sending, for several reasons:

  1. You never run out of memory: you know exactly how much you need
  2. You don't need to spend any memory keeping track of variable lengths
  3. Each packet can be handled statelessly: there's nothing *at the network / memory-handling level* that differs between messages
  4. Computers can skip messages very very very efficiently if they are fixed length - you can leap to the 5,674,074th byte of a message of fixed-size messages on any computer almost instantly. If you had to read every message along the way to find out how big it was, most computers would take millions or billions of instructions to do so.
  5. In network-processing, you often have to do skipping.
    1. Networks tend to lose data. You need to skip "the bad bits".
    2. Networks are unpredictable. You need to grab all the network-data as it comes in, and later on cherry-pick which bits to deal with
    3. Networks have *no shared physical memory* so often have to send huge amounts of data. To deal with massive amounts of data efficiently, you need to know how much of it you have (so you can e.g. decide if its too big to stick in RAM, and instead allocate room on the hard disk for it)