back to (virtual) reality

Got a chance to talk with Ben Goertzel (http://www.goertzel.org/blog/blog.htm). He's got some fresh ideas about integrating virtual agents into worlds like Second Life. The problem he's running up against is what I would call the "hand of God" paradox.

If you create a simulated, Matrix-like world, and you want it to be internally consistent, can you interact with that world (as opposed to just observe it), and not screw it up in some subtle way?

Descartes wrestled with this problem long ago; Wikipedia claims it's known as "the problem of interactionism". Well, we have a concrete example of that problem right in front of us, in the form of a software conundrum.

In a perfectly simulated world, our simulated entities would act according to their own internally perfect physical properties. Simulated energy would be used to contract simulated muscles or turn simulated motors, which in turn would run simulated limbs, wheels, and so on. But what controls the actions of these agents, simulated vehicles, humans, etc? In today's video game worlds (including social worlds like Second Life, which use essentially the same sort of technology as WoW or Grand Theft Auto), a user manipulates an input device (mouse, keyboard, joystick), giving commands to his/her character, such as run, jump, shoot, pick things up, etc. Presently, these commands are typically translated into animations, so the character will run through a predetermined sequence of moves that relate to the command requested. There are some clever tricks to segue between animations, but basically it's like each command starts a flipbook movie of what the character should do.

But this throws off the whole idea of a consistent, simulated virtual world. These pre-determined motions don't pay any mind to things like gravity, inertia, or the fact that solid bodies don't move through each other. If you watch closely, most video games occasionally give out on the illusion of reality in subtle ways, such as trolls and warriors walking through each other like ghosts in World of Warcraft. Other games, especially FPS (First-Person Shooters), typically control what you can do so that you don't have the freedom to perform actions that would tear the fabric of the illusion.

But all this becomes a big problem if you want to create realistic, artificial beings that will live in this world and act consistently with each other, as well as interact meaningfully with the godlike avatars that we control from outside. The paradigm has to shift significantly, in ways that ripple through the architectural choices that are made when you put together a software engine that can run this sort of simulation.

An environment like this should be designed more like the kind of simulations that are used in serious robotics work, such as the Gazebo component of Player/Stage (ok, Microsoft has something like that as well). These simulators don't let you just go ahead and do whatever an animator can think of. If you want to walk across a room, well you better have some control software that is capable of keeping you from falling down. If you want to pick up an object, you need grippers, with proper friction coefficients at their 'fingertips'; you need to control them in such a way that they grasp the object, pick it up with a force greater than gravity, and manipulate it in the way that you envision. And, your simulator needs to be capable of faithfully and efficiently simulating all this interaction without running into the common problems of stability and consistency you see with these sorts of algorithms.

At the top level, where a human user might be manipulating a joystick, you still want things to be relatively easy. But what's going on under the hood is now at least an order of magnitude more complex than in today's video game engines. Commands to move your avatar, instead of just branching into some cool-looking animated move, have to control your avatar as if it were a robot in the virtual landscape. There are things we can do to cheat a bit, such as having virtual balancing forces, but the basic paradigm has to be that the physics -- the rules underlying the virtual Universe -- are inviolate.

This is a whole new way of thinking about (virtual) reality.