virtual world rant# 29

Some thoughts about virtual worlds, avatars, prims, mesh, and maybe a little politics.

Lately I've been spending some time working on openviewer (www.openviewer.org), a project underway to build a BSD licensed virtual world client. Initially, the target server platform is SL/opensim. My goal for the project was just to have a sandbox to learn about the client side of things, without 'tainting' myself with the GPL code from Linden (their viewer is licensed under GPL, which is incompatible with opensim's less restrictive BSD license).

Anyway, what this has allowed me to do is get deeper into the technical underpinnings of physics, animation, and client/server relationship in this architecture. That in turn has led me to refine and focus my ideas about 'existents' and such, and the general issue of making virtual worlds safe for AI-controlled v-bots.

The upshot is that modern game engine front-ends (the graphics library part; we're using Ogre, but I suspect this applies generally) -- have a set of features that has evolved over the last 20+ years to include all the tools necessary to visually represent a world that looks something like the 'real' world, or at least looks pretty damn cool, and appeals to human perceptual sensibilities. Starting with the basic atomic 3D building block, the triangle, these engines have built up a vast set of capabilities, such as texture mapping, shading, deformable meshes, 'skeleton' animation, and other gizmos, which collectively form the palette that designers can access.

The kinds of things you can simulate with this bunch of tools runs the gamut of things we experience in real life. Assuming we're outdoors, we start with terrain, sky, and sea. Then we add fixed forms, both natural (plants, vegetation), and artificial (buildings, roads, etc). On top of that, we have inanimate objects that move about -- vehicles mostly, but they could be anything, like a ball, a door, or a (mechanical-looking) robot.

Finally, we move on to living things -- animals and humans (not to quibble; obviously humans are animals, but they get pretty special treatment because of their status within most applications). Humans (and human-like creatures, ie trolls, monsters, aliens etc) have been the recepients of a massive amount of design and implementation refinement. 'Skin' (deformable mesh) is stretched over 'bones' (linked sets of rigid sticks), with complex mathematical relationships so that animation (generally derived from motion capture) will look natural. Then you have clothes (algorithmic mesh), hair (various approaches), and finally additional inanimate accessories (attachments, in SL parlance) such as jetpacks, earpieces, glasses, etc and so on.

The point of going over all this is to focus on the fact that these graphics engines are not really general-purpose reality simulators. They have some pretty specific features that are for the most part tweaked to perform functions that relate to specific elements that we might want to visualize in the context of virtual reality. Actually, the game engines themselves are usually quite flexible; you don't have to have a humanoid skeleton -- you can put the bones together in any strange way you choose. You can utilize the various tools in ways not envisioned by the designers of the engine, as long as you have the resources to code and test it.

On top of these grahics engines, we have (in the case I'm analyzing) one more layer -- the client/server foundational architecture, starting with Linden's Second Life protocol, and presently being extended by folks like RealXtend. Obviously there are other platforms (multiverse, wonderland, Qwaq...) but this is the one I'm familiar with, and frankly it looks like the one free and open source platform that is getting enough traction to have a good shot at becoming some sort of defacto standard. Anyway that's a different discussion; for now, it's the case I'm familiar with, and I'm willing to bet the others have a similar set of features and limitations.

What Linden did with Second Life is to make some pretty deep assumptions about what kinds of entities can exist in their virtual worldscape. At the top of the heirarchy is the 'avatar' -- a nominally human form, customized to varying degrees by users. These entities have a special status, and (shades of the old nobility) sometimes their status includes restrictions on what they can do (kind of a karmic quid-pro-quo maybe).

On the other side of the tracks, we have what Second Life calls 'prims' -- primitive shapes with varying adornments; potentially linked together in complex ways, and possibly scripted with computer code.

It might be helpful to spell out in list form what the various capabilities and limitations of these two entity types are:

Avatars:
controlled by: human users or 'bots' (pseudo-users running a program that acts like a client)
Motion and behavior: animations (.bvh), simple physics (capsule-based)
basic form: skeleton + deformable mesh, human-based
visual enhancements: clothes, hair,
special status: name, groups, friends, ownership (inventory)

Prims:
controlled by: server-side scripting (could be plug-in for opensim)
motion and behavior: scripting and more realistic physics (all parts can be physical)
basic form: sets of geometric shapes with algorithmic modifiers (cuts, shear, rotation, taper etc)
visual enhancements: custom textures, particle systems, special lighting (?)
special status: owned by regions &/or avatars

In general terms, when you see something that looks vaguely human, and it talks to you, and buys stuff, it's almost definitely an avatar. If it looks like a vehicle, or a machine, it's almost definitely prim-based.

The problem I have with this state of affairs is this. My interest in virtual worlds revolves around the idea that, eventually, we might be able to make major advances in AI, by leveraging the co-existence of human and computer-controlled entities in the same world. However, platforms like Second Life have been developed with the animate/artificial distinction so deeply baked in, that this goal becomes somewhat difficult, if not impossible.

Getting concrete, what I want in a virtual world system is as follows:

* Whether you are controlled by a human or a computer should not have a major impact on your capabilities (modulo issues of privacy and transparency -- social issues are real, but the technology should not dictate them).

* Computer-controlled entities should be able to look like humans, animals, aliens, robots, or anything else the graphic engine could conceivably generate

* Human-controlled entities should be able to look non-human, be fully physical, and use low-level control within the parameters of a realistic simulation (as opposed to relying solely on pre-cooked animation sequences)

* There shouldn't be a-priori distinctions between clients and servers. Issues of trust and access should be generalized, and dealt with in a rigorous and flexible manner. Any process that can possibly be distributed should be able to be distributed among an arbitrary number of machines with arbitrary network topologies

* Animation and physical simulation need to be seamlessly integrated. There are techniques for doing this at the bleeding edge of game design and movie-making. The hooks for these capabilities need to be in the protocol.

* All of this has to be done in such a way that today's model of entertainment-based games and worlds is not negatively impacted (otherwise no one will care, and my ideas will go nowhere)

Sometimes I feel as if I'm pissing into the wind here. Who really cares about this stuff? Very few people, from what I can tell. There are a few more who vaguely understand what I'm getting at, but have their own work and agenda, so can't really be bothered to help beyond just saying "yeah, that sounds nice". I have so far failed to really make the case that this stuff matters, and so it's falling on deaf ears. Perhaps a tree needs to fall down and get itself back up multiple times, before anyone in the forest happens to notice.

-dan

Collaboration Is Social

this needed saying. Credit where due: the phrase came up at an Intel-sponsored meeting about the state of the Metaverse. I think I was the originator, though it's often hard to know for sure at these sorts of events. But it's a good meme nonetheless.

The point is, working on something together is a social phenomenon. This points up the genius of Second Life. Why did it succeed (if only for the requisite 15 minutes) where so many other virtual worlds failed to catch on, either with a dedicated user base or the imagination of the public? It took me a while to figure this out, because it's not immediately obvious (at least it wasn't to me). The key is the fact that the development tool is the world itself. It's not just that it was free (there are free tools like Blender for typical 3D work); it's not that it was particularly good (it wasn't). It's not even just the fact that users were encouraged to create content, and the learning curve was not so steep. I think it's the fact that you actually build things right there *in* the actual world, and other people can see you do it. You can collaborate right there, in real time -- by moving an object, changing its properties, etc. Or just by commenting. It's the difference between playing a recording and lip-syncing, versus a live performance (whether or not you are a direct participant).

This is I think what makes this technology so potentially powerful, and why big companies are showing an interest. It's something of an evolution from the 'mods' of gamers -- but the right-there aspect of tweaking your avatar, wearing different clothes, building a house or terraforming -- this is a new element.


(ps yes, I know there was a financial collapse and an historic election. Life goes on)

virtual worlds and robots

I've become involved in an interesting discussion group, http://groups.google.com/group/roboworld

It was started by Ben Goertzel, of Novamente. Ben is a pretty serious dude when it comes to AI (see his classic startup story: http://www.goertzel.org/benzine/WakingUpFromTheEconomyOfDreams.htm

Anyway, here is a recent post I made there, which outlines the problem we're trying to solve (at least I think it does):

----------------------------------------------------------------
[subject: "square two"]

I think we need to outline our goals for this project, so we can make reasonable implementation decisions wrt rex, opensim, and SecondLife.

Here are my personal goals -- the possibilities that motivate me to be involved.

I want to build a world where autonomous agents can be developed, either through human design, and/or through a process of artificial evolution (a subject I have been deeply involved in for a while). These agents should be able to interact, pursue goals, observe other agents, and in turn build structures and agents of their own. In short, they should have available to them every operational capability that human agents have in a SecondLife sort of environment.

However, they need something more. As humans, we have the benefit of physical embodiment in a complex, rich physical world. This embodiment endows us with a critical advantage, which is the ability to experiment, play, and learn from the bottom up. What I mean by this is that as babies, we spend a significant amount of time playing with our physical environment, learning the ropes so to speak. We build on these experiences to develop layers of abstraction, working our way up the chain of complexity to the point where we can do things like walk, manipulate objects, and eventually communicate with other humans and interact with other entities such as animals and machines.

The autonomous agents I want to (meta)create will need the same sort of ladder of complexity to climb, otherwise they will not be able to get to the heights I have in mind for them. Virtual worlds such as Second Life (as presently implemented) basically forgo the lower levels of complexity, and implement a world at a somewhat abstract level -- a level of objects, qualities, and complex transactions. This makes sense if your goal is to create an efficient simulation of the world at the level of abstraction that an adult human typically works in. Since the customer base for virtual worlds is adult (or near adult) human beings, that goal is reasonable.

We are aiming here for something quite a bit more ambitious. Not only do we want to simulate a world at the level appropriate for research into AI and artificial evolution; in addition, we want to allow humans to interface with that world as well. The problem is that our artificial entities are embodied in the simulation, while we are embodied in the real world.

There are really two possible solutions to this problem, and they are not mutually exclusive. Humans can fully embody themselves into a rich simulation (Virtual Reality), and/or the agents can embody themselves in the 'real' world (Robotics). Let's examine each solution in detail.

Virtual Reality

The holy grail of VR research (for most anyway) is to create a world so realistic that the human immersed in it can hardly distinguish it from real life. This involves not only the obvious simulation of the main senses (vision and hearing), but also tactile, vestibular, and prioperception. To really achieve this, we will need either a direct neural connection, or an extremely sophisticated (and expensive) set of technologies involving stereo vision, haptics, and some sort of gymbaled bodysuit or exoskeleton. For this to actually work in any useful sense, the lag time between the simulation and human senses and actions will need to be measured in single milliseconds. Suffice to say this is probably a long way off, and will not be available to the average consumer for at least 20 to 30 years (IMSHO). It is unlikely to ever be cheaper than an expensive consumer purchase, so it will have to provide some real value, rather than just being an
entertainment device (though I suppose folks do pay dearly for some entertainment). Anyway, it's not around the corner, technically speaking.

On the other hand, it is quite possible that a level of immersion that goes beyond today's joystick interface could arrive sooner. Technology is afoot that could allow you to virtually mimic your upper body and facial motions in an avatar. I've mentioned before that even this level of VR immersion will require some major retooling of the virtual world interface, to support direct avatar control as opposed to canned animations.

Robotics

For two years now, I've worked full time on bipedal legged locomotion and manual manipulation with human-sized robots. It's really, really hard. I have become much more impressed with biology and evolution than I ever was before. The fact is, building capable robots is really difficult. Most of the work I do is in fact in simulation, because at least there I can depend on the robot working consistently from one day to the next, and I can run hundreds and thousands of simultaneous simulations in the time it takes to do a single experiment on the real robot. I can say with confidence that we are still at least a decade away from truly human-capable machines, even simply on the mechanical and power train front, much less control and autonomy.

Yet, there are interesting products such as Ugobe's Pleo that redefine what is possible today at an economical scale. Kits such as Lego and the RoboOne class of mini-bipeds are bringing physical embodiment of smaller robots with limited but interesting capabilities into homes and schools.

The other interesting development in robotics is the increasing use of simulation. Every serious robotics project I am aware of in the last 10 years (and perhaps longer) has used some level of simulation as part of the design process. Microsoft has a popular product, and there is a large community using open-source tools such as Player/Stage/Gazebo. In my case, my experience writing a robotic simulator using ODE allowed me to contribute to the OpenSim codebase, which relies on ODE to perform its physics calculations.

Putting It All Together

The point of intersection between these two disciplines is the use of physics simulation technology to mimic the behavior of objects and materials found in the real world. Right now, there's a bit of an 'impedance mismatch', in that the use of physics in popular virtual worlds such as Second Life and RealXtend is much more limited in its granularity of realism than what is typically required to do actual robotic-level simulation. More than just an issue of efficiency and scale, there are deep architectural differences between how the two disciplines have evolved. Making them play nice together is going to take some work.

I've got some further thoughts about how to get there from here, which I will put in another post.

-danx0r

back to (virtual) reality

Got a chance to talk with Ben Goertzel (http://www.goertzel.org/blog/blog.htm). He's got some fresh ideas about integrating virtual agents into worlds like Second Life. The problem he's running up against is what I would call the "hand of God" paradox.

If you create a simulated, Matrix-like world, and you want it to be internally consistent, can you interact with that world (as opposed to just observe it), and not screw it up in some subtle way?

Descartes wrestled with this problem long ago; Wikipedia claims it's known as "the problem of interactionism". Well, we have a concrete example of that problem right in front of us, in the form of a software conundrum.

In a perfectly simulated world, our simulated entities would act according to their own internally perfect physical properties. Simulated energy would be used to contract simulated muscles or turn simulated motors, which in turn would run simulated limbs, wheels, and so on. But what controls the actions of these agents, simulated vehicles, humans, etc? In today's video game worlds (including social worlds like Second Life, which use essentially the same sort of technology as WoW or Grand Theft Auto), a user manipulates an input device (mouse, keyboard, joystick), giving commands to his/her character, such as run, jump, shoot, pick things up, etc. Presently, these commands are typically translated into animations, so the character will run through a predetermined sequence of moves that relate to the command requested. There are some clever tricks to segue between animations, but basically it's like each command starts a flipbook movie of what the character should do.

But this throws off the whole idea of a consistent, simulated virtual world. These pre-determined motions don't pay any mind to things like gravity, inertia, or the fact that solid bodies don't move through each other. If you watch closely, most video games occasionally give out on the illusion of reality in subtle ways, such as trolls and warriors walking through each other like ghosts in World of Warcraft. Other games, especially FPS (First-Person Shooters), typically control what you can do so that you don't have the freedom to perform actions that would tear the fabric of the illusion.

But all this becomes a big problem if you want to create realistic, artificial beings that will live in this world and act consistently with each other, as well as interact meaningfully with the godlike avatars that we control from outside. The paradigm has to shift significantly, in ways that ripple through the architectural choices that are made when you put together a software engine that can run this sort of simulation.

An environment like this should be designed more like the kind of simulations that are used in serious robotics work, such as the Gazebo component of Player/Stage (ok, Microsoft has something like that as well). These simulators don't let you just go ahead and do whatever an animator can think of. If you want to walk across a room, well you better have some control software that is capable of keeping you from falling down. If you want to pick up an object, you need grippers, with proper friction coefficients at their 'fingertips'; you need to control them in such a way that they grasp the object, pick it up with a force greater than gravity, and manipulate it in the way that you envision. And, your simulator needs to be capable of faithfully and efficiently simulating all this interaction without running into the common problems of stability and consistency you see with these sorts of algorithms.

At the top level, where a human user might be manipulating a joystick, you still want things to be relatively easy. But what's going on under the hood is now at least an order of magnitude more complex than in today's video game engines. Commands to move your avatar, instead of just branching into some cool-looking animated move, have to control your avatar as if it were a robot in the virtual landscape. There are things we can do to cheat a bit, such as having virtual balancing forces, but the basic paradigm has to be that the physics -- the rules underlying the virtual Universe -- are inviolate.

This is a whole new way of thinking about (virtual) reality.

another blogger

talking about digitized physics:

http://alexlamb.livejournal.com/4562.html

two great links

www.ebonmusings.org/atheism/carrot&stick


and:


http://kugell.com/onesmallanswer.html

distributed physics

These are some quick notes on my crazy ideas on how to create a scalable, distributed, shared artificial reality:

* Everything in the system is an object, with a bunch of attributes, some static (not normally changed) and some dynamic. Static ones include size, shape and mass; dynamic attributes include position, orientation, linear and angular velocity.

* Every object has an authoritative server who manages that object

* An avatar is a type of object that happens to be controlled by a user

* Region servers keep track of the objects in their volume of influence (overlapping?)

* Objects compute their own dynamics, based on their attributes, internal state, and information they acquire about their environment. The result is a trajectory -- the object's motion plan based on present information (maybe .5 or 1 second into the future -- enough to cover maximum network lag).

* Collisions are handled as follows: an object queries the objects near it (how does it know who is near? ask the region server). The other object returns information about its planned trajectory (as well as size & shape if we haven't seen that object before, or it's been modified). If the trajectories of the two objects indicate a collision, our object modifies its trajectory accordingly. In most cases, the other object will do the same (query our trajectory, compute whether it needs to deflect).

* A viewer is a piece of code that queries objects and visualizes them for a user. In practice, there will probably be some sort of registration system: a viewer registers with an object (through its server, or proxied through the region server) to get updates. The object will then send a stream of time-stamped messages with its present trajectory (velocity, rotation over some time period -- possibly buffered like an audio stream? smoothness vs. latency, ala VOIP). The trajectory is good for its specified time period, or until a new trajectory message supersedes it. The viewer can choose how to handle missing data (trajectory completes, no new message); it could extrapolate (ala Second Life viewer), just freeze the object, or even run client-side physics to try to guess what the object might do.

A system like this should allow a large world with many objects to be distributed over a number of servers in a scalable fashion. The region server's job is minimal -- it just keeps track of who is 'in' the region (subtlety involves large objects whose center of mass may be in an adjacent region; the region size must be a few times larger than maximum object size). The dynamics and collision detection responsibilities are spread among the active objects, which can be on any number of servers. The region server may also proxy the objects messaging, so viewers don't need point-to-point connections with each object's server -- though perhaps a multi-connection setup would work, and the region server just manages resolving the connection. I'm not enough of a network expert to think this through completely.

The advantage of this sort of architecture, aside from the obvious scalability, is that there is minimal coupling between the parts of the system. The viewer doesn't have to know anything about the physical properties of the objects, or the underlying dynamics used, collision etc; it just presents its stream of object updates as best it can. Different objects can use different physics engines or collision logic; in the end, each object is authoritative for itself, so there can be no long-term discrepancies. The trade-off is probably increased net traffic compared to an SL-style architecture; but I suspect this will be more than offset by the advantages of a distributed topology, and by the ability of viewer clients to make choices about how much data they can handle. You could have 10,000 avatars in a stadium, and each client may choose to fully visualize only the closest 50 people. The rest might be rendered at low resolution, with low-frequency updates; using crowd simulation; or not at all. A viewer with a big pipe might render more of the crowd; a client on a limited bit budget could choose to get less frequent updates and do its own extrapolation.

that's all for now, more to come...

Artificial Reality

When I started this blog, I called it Artiphys, because at the time I was working with Ed Fredkin on discrete physics (* footnote). I had this idea that one could turn Ed's theory, that the Universe is a digital computer, upside down: create a digital universe of our own, and see what happens in it. Maybe that's why ours exists in the first place (as good a cosmology as any perhaps?)

Lately, I've become involved in a project that may have the potential to realize some of these ideas. OpenSim (www.opensimulator.org) is an open-source implementation of the Second Life protocol, meaning you can use it to run your own Second Life-style 'grid'. In case you didn't know, Second Life uses multiplayer videogame technology to create a 3D world where you can interact with the environment and other characters (called Avatars). It's on the Internets.

My involvement with this project has centered mostly around the physics engine, though my interest in the technology and its implications is much wider than that. But the physics is a great place to start, because it brings up a bunch of interesting intellectual and philosophical questions relating to this whole area, which I like to call "Artifical Reality".

Artificial Reality is different than Virtual Reality in the following way: it doesn't just aim to duplicate our reality in virtual form; it aims to be something 'real' in its own right, with its own rules, entities, interactions, and outcomes. Where VR has always had the connotation that you transform into a digital version of yourself and interact with a digital environment, AR implies a world that exists without you or any 'real' entities, independently. It is a persistent world, that does not stop existing because you take off your VR headset and remove the glove.

Second Life is presently at a stopping point between VR and AR. I think of it as Web 3.0 (see previous post) -- adding presence to information and identity. It also has persistence (it exists when you're not there), but that is mostly due to other flesh-and-blood people who are logged on when you're not. Artificial Reality (I will not submit to the temptation to bump the web version number at this point) adds autonomy and independence to the picture. So we have a heirarchy of information structures that leads from the existence of bits and bytes up to an independent reality that takes its own form and structure:

level 1: information
level 2: identity
level 3: presence
level 4: persistence
level 5: independence

To get to level 5, we need more than just pretty avatars, cool graphic bling, and a slapped-on overlay of videogame-style physics. We need the logic of object interactions to be precisely defined, and that logic has to have some very specific properties if we want the resulting world to be interesting, coherent, and stable. My ultimate dream is to create an alternate reality so rich that it can exhibit some of the features of real reality that I find fascinating, such as the emergence of order out of chaos. This has to do with information theory, entropy, discrete vs. continuous, reversibility, A-life, and a host of other subjects. There is a distinct chance that all of these elements can come together to form something completely new and unexpected. Wouldn't that be fun?


footnotes

* Ed's website, www.digitalphilosophy.org, appears to be down; see http://en.wikipedia.org/wiki/Digital_philosophy for background. The paper we wrote together at Carnegie Mellon can be found here: http://www.andrew.cmu.edu/user/danbmil/salt/

Ben Goertzel has been talking about similar stuff, to a much wider audience (see my link to his blog). To my knowledge, he has not proposed a concrete model that addresses the low-level discrete physics issue.

There are a number of sci-fi works that touch on this subject:

- The Matrix -- obviously (here's hoping our emergent AGI's aren't as malevolent as Agent Smith). Note that the paradigm of the Matrix still clutches tightly to the idea that a 'real' body must exist somewhere (the people are all kept alive in vats). What's that about?

- Permutation City, a novel by Greg Egan

- Non Serviam a short story by Stanislaw Lem (reprinted in Hofstaedter's The Mind's I

Web 3.0

web 1.0 is about information

web 2.0 is about identity

web 3.0 is about presence

why we code

"I have been reading, with great satisfaction, the many articles in magazines about Linux free software. People in the business world are not able to comprehend why the computer people give their work away. I am unable to think about this without becoming emotional. It is no mystery to me why they download their intellectual ideas into the vast, evolving and continually improving computer operating system. It is because their thoughts will live forever as part of the "genetic code" of the computer program. They are putting themselves into the program and their “intellectual DNA" will live forever in cyber-space. As the program evolves and changes, the code they wrote will probably remain hidden deep within it. It is almost like a living thing that is continually evolving and improving. For both me and for the programmers that contribute to Linux, we do it because it makes our lives more meaningful."

-Temple Grandin