|
|
Chapter in Emotions in Humans and Artifacts, eds. R. Trappl, P. Petta, and S.Payr, MIT Press, April 2003. Originated from a symposium at OFAI in Vienna, August 1999.
Andrew Stern
Abstract
We have developed a series of lifelike computer characters called Virtual Petz. These socially intelligent agents live on your PC computer desktop. The Petz are autonomous characters with real-time layered 3D animation and sound. Using a mouse the user moves a hand-shaped cursor to directly touch, pet, and pick up the characters, as well as use toys and objects. Virtual Petz grow up over time and strive to be the user's friends and companions. They have evolving social relationships with the user and each other. To implement these agents we have invented hybrid techniques that draw from improvisational drama, cartoons, classical AI and video games. As of June 1997 over a million copies of virtual Petz have been sold around the world.
1: Introduction
During the workshop on “Emotions in Humans and Artifacts” participants presented research addressing such questions as, how do emotions work in the human mind? What are emotions? How would you build a computer program with emotions? Can we detect the emotional state of computer users? Yet as the discussions wore on, instead of providing some answers to these questions, they persistently raised new ones. When can we say a computer “has” emotions? What does it mean for a computer program to be “believable”? Do computers need emotions to be “intelligent”, or not?
These scientific, engineering and philosophical questions are fascinating and important research directions; however, as an artist with a background in computer science and filmmaking, I found my own perspective on emotions in humans and artifacts generally left out of the discussion. I find the burning question to be, what will humans actually do with artifacts that (at least seem to) have emotions? As a designer and engineer of some of the first fully-realized, “believable” interactive virtual characters to reach a worldwide mass audience, Virtual Petz and Babyz (PF Magic 1995, Mindscape 1999), I know this is a question no longer the domain of science fiction. Today millions of people are already encountering and having to assimilate into their lives new interactive emotional “artifacts.”
Emotions have been a salient feature of certain manmade artifacts, namely stories and art, long before the scientific study of emotion began. Looking to the past we see a tradition of humans creating objects in one form or another that display and communicate emotional content. From figurative painting and sculpture to hand puppets and animated characters in Disney films such as Snow White, manmade artifacts have induced emotional reactions in people as powerful and meaningful as those generated between people themselves.
Now we are at point in the history of story and art where humans can create interactive artifacts, using the computer as a new medium. We can now write software programs that can “listen” to a human, process what it just “heard”, and “speak” back with synthetically generated imagery and sound, on machines that are already in the households of millions of families around the world. The new urgent challenge for artists and storytellers is to create interactive artifacts that can induce the same emotional reactions and communicate emotional content as traditional non-interactive stories and art have done. In fact because the computer theoretically can be programmed to tailor the experience to an individual, it could become the most powerful medium of all for creating affective stories and art.
The question becomes, how do you do that? How do emotionally powerful stories and art “work” anyway? Alas, creating an artifact that produces a meaningful emotional reaction in a human is considered an “art” itself. Although techniques and advice on the artistic process have been published – works such as The Art of Dramatic Writing by playwright Lajos Egri, The Illusion of Life by Disney animators Thomas and Johnston, Letters to a Young Poet by poet Rainer Rilke – the act of creating emotionally powerful artifacts is by and large considered elusive, mysterious and unquantifiable. Even in art school the typical approach is to teach students to imitate (“master”) traditional styles and techniques, after which it is hoped the student will be ready to “find their own style,” which sometimes never happens.
Naturally it is difficult to discuss the art of creating emotionally powerful artifacts in the context of a scientific workshop, in the way one would approach a computer science or engineering problem – which helps explain the general reluctance to research the topic, and the not-so uncommon attitude among artificial intelligence researchers that the topic is “mushy”, ill-formed, or worst of all, unimportant. Art and entertainment is considered to be fun, not a serious pursuit. This view is short-sighted. On the contrary stories and art are among the most serious and meaningful pursuits we have. We communicate ideas and experiences to each other in this way. The fact that the problem is, to a degree, mushy and unquantifiable, makes it all the more challenging and difficult to undertake.
This paper puts forth virtual characters as an emerging form of manmade artifact with emotional content. Virtual characters go by a few other names in the AI research community, such as believable agents, synthetic actors and synthetic personalities. These are embodied autonomous agents that a user can interact with in some fashion, animated as real-time graphics or built as physical robots, which appear to have personality, emotion and motivation, designed to be used in art or entertainment. Over the past decade several media labs have been exploring the issues involved in building virtual characters (such as Bates et al 1992, Blumberg 1995, Perlin 1995 and Goldberg 1997, Hayes-Roth 1996, Elliot et al 1997). Some groups have designed architectures and implemented prototypes that have been demonstrated at academic conferences. But it should be made clear that in the business of creating emotional artifacts, in the final analysis, prototypes and demos are not enough. The point of creating emotionally powerful experiences, whether interactive or not, is to induce a reaction in an audience, in “users.” These creations must be experienced by the general public to serve the purpose for which they were created in the first place. Until this happens the work created in closed-door media labs is ultimately incomplete.
The public has been consuming interactive entertainment for two decades now, in the form of software products from the videogame and computer game industry. Unfortunately the experiences offered in these games are mostly juvenile, primarily focused on fighting, shooting, racing and puzzle-solving, and the virtual characters offered in them are most often shallow, one-dimensional cardboard-cutouts. Such games can be emotionally powerful experiences for those who play them, but they do not appeal to the majority of the population. Very few successful pieces of interactive entertainment or art have been made with emotional content for a mass audience – that is, the kind of “personal relationship” stories that books, theater, television and movies offer, or the kind of “high art” exhibited at museums and art shows (Stern 1999a, Mateas 1999).
This remainder of this paper suggests new ways to employ virtual characters to create emotionally powerful interactive experiences, using our Virtual Petz and Babyz projects as case studies. The techniques used to create these projects will be presented and discussed, with emphasis on the importance of design. Finally we will attempt to address the question of what it could mean for a human to have an emotional relationship with a virtual character.
2: Do you feel it? The case for emotional relationships
Animation and artificial intelligence technologies for creating real-time interactive virtual characters are currently being researched and developed in academic labs and industry companies. We are told that soon we will have virtual humans that look photorealistic, with behavior driven by some degree of AI. Many are working with the intention that these characters will become functional agents, at our command to perform a variety of complicated and menial tasks. They will learn our likes and dislikes, able to autonomously communicate and negotiate with others. And they will become teachers in the virtual classroom, always ready and willing to answer our questions.
At first glance it seems natural that adding emotions to these virtual characters should greatly enhance them. After all, real people have emotions, so virtual human characters should have them too. Some researchers in neuroscience and psychology point to emotion as an important factor in problem solving capabilities and intelligence in general (Damasio 1994). As Marvin Minsky put it, “the question is not whether intelligent machines can have emotions, but whether machines can be intelligent without any emotions” (Minsky 1985). Virtual characters may very well need emotions to have the intelligence to be useful.
But when thinking in terms of a virtual character actually interacting with a user, is emotional behavior really appropriate for these types of applications? In real life it is arguable that interactions with “functional agents” (e.g., waiters, butlers, secretaries, travel agents, librarians, salespeople) are often best when emotions are not involved. Emotional reactions can often be irrational, illogical and time-consuming, which work against the efficient performance of tasks. Of course in any transaction, politeness and courtesy are always appreciated, but they hardly qualify as emotional. We expect teachers to be a bit more personable and enthusiastic about their material than a travel agent, but do we want them to get angry or depressed at us?
While emotions may be required for intelligence, I would argue that the most compelling interactions with virtual characters will not be in the area of functional agents. If a user encounters a virtual character that seems to be truly alive and have emotions, the user may instead want to befriend the character, not control them. Users and interactive virtual characters have the potential to form emotional relationships with each other – relationships that are more than a reader’s or moviegoer’s affinity for a fictional character in a traditional story, and perhaps as meaningful as a friendship between real people. By an emotional relationship we mean a set of long-term interactions where the two parties pay attention to the emotional state of the other, communicate their feelings, share a trust, feel empathetic, and establish a connection, a bond.
2.1: Virtual friends
The recent success of several “virtual pet” products, popular among both kids and adults, offers some support for this idea. The most sophisticated of these characters are animated on the computer screen, such as Dogz and Catz (PF.Magic 1995 - 1998), and Creatures (Grand et al 1997), but some are displayed on portable LCD keychain toys or even embodied as simple physical robots, such as Tamagotchi (Bandai 1996), Furby (Tiger Electronics 1998), and Aibo (Sony 1999). Users “nurture” and “play” with these pets, feeding them virtual food, petting them with a virtual hand, and generally giving them attention and care lest they runaway or die. While there can be some blurring into the domain of videogames, in their purest form virtual pets are not a game, because they are non-goal-oriented; it is the process of having a relationship with a virtual pet that is enjoyable to the user, with no end goal of winning to aim for.
As of this writing, there have been no completed formal studies of virtual pets; a study is currently underway by (Turkle 1999). In our experience with the Dogz and Catz products, our anecdotal evidence suggests that depending upon the sophistication of the virtual character, the emotional relationship that a user can form with it ranges anywhere from the attachment one has to a favorite plant to the bond between a master and their dog. Children are the most willing to suspend their disbelief and can become very attached to their virtual pets, playing with and feeding them everyday. It is precisely those irrational, illogical and time-consuming emotional interactions that may hamper a functional agent that are so engaging and entertaining here. (Please refer to the Appendix of this paper to read real customer letters we have received about Petz.) Only a few Petz users, mostly technology-oriented adult men, have requested that their Petz be able to perform functional tasks such as fetching e-mail.
What is most interesting about the phenomenon of virtual pets are not the toys and software themselves – some of which have minimal interactivity and little or no artificial intelligence driving them – but the fact that some people seem to want to form emotional relationships with them. Some appear quite eager to forget that these characters are artificial and are ready and willing to engage in emotional relationships, even when some of the virtual pets offer little or no reward or “warmth” in return. This offers some promise for the public’s acceptance of the concept of a more advanced virtual friend.
As commercially successful as these virtual pets are, it seems likely that emotional relationships at the level of favorite plants or pets will be far easier to accomplish than the level of friendship between two adults. An owner-to-pet relationship dynamic is much simpler than a person-to-person one, and much less communication is required between the two parties. Most importantly, the relationship is inherently unequal. An owner of a real pet chooses (even purchases!) their real-life cat or dog. Therefore the act of buying a virtual pet as a toy or piece of software does not violate the hierarchy of a real-world owner-to-pet relationship.
As people we do not get to choose which other people will be friends with us. Friends, by definition, choose to be friends with one another. Therefore, even if we create an interactive virtual character that can perform all the behaviors required for an emotional relationship between human adults, as a manmade artifact that can be bought and sold, could a “true” friendship could be formed at all? This is an open question that invites exploration.
2.2: Interactive stories
Stories have long been our primary way to observe and understand our emotional relationships. If stories could be made interactive – where users could immerse themselves in virtual worlds with characters they could talk to, form relationships with, touch and be touched by, and together alter the course of events, literally creating a new story in real-time – then we would have a new form of interactive entertainment that eclipses videogames. Like traditional stories from books, theater, television and movies, an interactive story would be affecting and meaningful, but made all the more personal because the user helped shape it and create it (Stern 1998).
Virtual characters programmed to simulate the dynamics of emotional relationships could be used as starting points for creating interactive stories. In her recent book, Hamlet on the Holodeck: The Future Of Narrative in Cyberspace, Janet Murray suggests that interactive virtual characters “may mark the beginning of a new narrative format” (Murray 1997). As a first step in this direction, instead of relying heavily on planning to generate story plots as some previous story researchers have done (such as Meehan 1976, Pemberton 1989, Turner 1994), a developing and ongoing emotional relationship itself could serve as a narrative. For example a user and a virtual character could meet and get to know each other, begin to develop trust for one another, and perhaps (accidentally or not) violate that trust in some way, causing the relationship to take a downturn. The relationship could progress from there in many ways, perhaps recovering, ending, cycling between highs and lows – much like real-life relationships. There are several traditional stories that follow this pattern, such as “boy meets girl, boy and girl fall in love, boy loses girl”, and so on.
2.3: Interactive art
Autonomous interactive virtual characters are only just beginning to make their way into installation and performance art. The goal of Simon Penny’s 1995 “Petit Mal” was, according to the artist, “to produce a robotic artwork which is truly autonomous; which was nimble and had ‘charm’”, to give “the impression of being sentient” (Penny 1997). Petit Mal was a tall, thin robot on bicycle wheels which could quietly and gently move about in a room, able to sense when it was approaching walls or people. Celebrated video artists Lynn Hershman and Bill Viola have begun experimenting with combining video imagery of people and some simple interactivity. Mark Boehlen and Michael Mateas recently exhibited “Office Plant #1”, a robot plant that will bloom, wither and make sounds in response the mood of your office environment (Boehlen and Mateas 1998). Other efforts include the large, destructive autonomous robots from the industrial performance art of Survival Research Labs, the RoboWoggles (Wurst and McCartney 1996), and the robot installations of Alan Rath. The potential for exploration and discovery in this area seems untapped and wide open.
3: I need your love: Virtual dogs, cats and babies
Recognizing a dearth of consumer software with characters that displayed emotions or personality, the startup company PF.Magic was formed in 1992 with the mission to “bring life to entertainment.” We wanted to break out of the mold of traditional videogames (e.g., flight simulators, sports games, running-jumping-climbing games, shooters, puzzle games) to create playful new interactive experiences with emotional, personality-rich characters.
3.1: Dogz and Catz
By 1995 the personal computer was powerful enough to support the real-time animation we felt was required for a convincing virtual character. The first Dogz program, originally conceived by company co-founder Rob Fulop and created by Adam Frank and Ben Resner, was a simple idea: an animated virtual dog that you could feed and play with. As a product it was very risky; an emotional relationship as the sole basis of an interactive experience had never been done before. It was unknown at the time if anyone would pay money to interact with a virtual character in this way.
The program quickly generated interest from a wide range of customers male and female, kids and adults, which is typically unheard of in entertainment software. We followed up Dogz with a companion product, Catz, establishing the explicit design goal to create the strongest interactive illusion of life we could on a PC. To imbue the Petz characters with personality and emotion we began cherry-picking techniques from computer animation and artificial intelligence, to construct what eventually became a powerful real-time animation engine tightly integrated with a goal-based behavior architecture.
The Virtual Petz characters are socially intelligent autonomous agents with real-time 3-D animation and sound. By using a mouse the user moves a hand-shaped cursor to directly touch, pet, and pick up the characters, as well as use toys and objects. Petz grow up over time on the user’s computer desktop and strive to be the user’s friends and companions. The interaction experience is non-goal oriented; users are allowed to explore the characters and their toys in any order they like within an unstructured yet active play environment. This freedom allows users to socialize with the Petz in their own way and at their own pace. This also encourages users to come up with their own interpretation of their pet's feelings and thoughts. To date the Virtual Petz products have sold over two million copies worldwide.
The goal of the Petz characters is to build an emotional relationship with user. Their behaviors are centered around receiving attention and affection. They feed off of this interaction. Without it they become lethargic, depressed, and if ignored for long enough, they will run away.
The most direct way the user can show affection to the Petz is through petting. By holding down the left mouse button users can pet, scratch and stroke with a hand cursor; the Petz immediately react in a variety of ways depending on what spot on their body is being petted, how fast, and how they feel at the time. Users can also pick up the characters with the right mouse button and carry them around the environment. We found that being able to (virtually) touch and hold the characters to be a very effective way of building emotional relationships and creating the illusion of life.
The Petz have equal footing in their relationship with the user. The toys and objects in their environment have direct object-like interaction for both the user and the characters. Petz have full access to the toy shelf, and if they really want something, they have the freedom to get it themselves. This helps express the unpredictability and autonomous nature of the Petz. It also requires users to share control of the environment with them. For example, by picking up and using a toy the user can initiate play. Throwing a ball may initiate a game of fetch, or holding a tugtoy in front of a pet may begin a game of tug-of-war. Similarly, a pet can get its own toy and bring it to the user to initiate play. The act of sharing control of the environment and cooperative decision-making helps further strengthen the relationship.
We have created a variety of personalities - playful terriers, grumpy bulldogs, hyper Chihuahuas, lazy Persian cats, aggressive hunter cats, timid scaredy cats and so on. Each individual character has its own likes and dislikes, spots and body coloration, and personality quirks. Users get to play with individual Petz to see if they like them before deciding to adopt. Once adopted, the user gives them a name. This individual variation allows the user to develop a unique relationship with a particular character. Each owner-pet relationship has the potential to be different.
3.2: Babyz
Our newest virtual characters, Babyz, released in October 1999, have the same non-goal-oriented play and direct interaction interface as the Petz. The user adopts one or more cute, playful babies that live in a virtual house on the computer. The Babyz have a similar cartoony style as the original Petz, but have more sophisticated, emotive facial expressions, and some simple natural language capability. Babyz vary in size, shape and personality, and in some ways appear to be smarter and cleverer than real one-year-old babies would be. They want to be fed, clothed, held and nurtured, but are also quite playful and mischievous. Users can think of themselves as their parent or babysitter, which ever they feel most comfortable with.
The user can nurture a Babyz character by holding and rocking it, tickling it, feeding it milk and baby food, putting on a fresh diaper, giving it a bubble bath, soothing it if it gets upset, giving it medicine if it gets sick, laying it down to sleep in a crib, and so on. Play activities include playing with a ball, blocks, baby toys, music, dancing and singing, and dress-up. Through voice recognition Babyz can understand and respond to some basic spoken words (such as “mommy”, “baby”, “yes”, “no”, “stop that”), and can be read simple picture books.
The Babyz characters develop over time, appearing to learn how to use toys and objects, learning to walk and to speak a baby talk language. In this version of the product, they will always be babies – never progressing beyond a stumbly walk and simple baby talk. (All behaviors are pre-authored, with the user’s interaction unlocking them over time, to create the illusion that the Babyz are learning.) If the user has more than one baby adopted, they can interact and form relationships with one another. Babyz can be friends and play nicely together or engage in long-term sibling rivalries. The program becomes an especially entertaining and chaotic experience with three active Babyz all getting into mischief at the same time (Stern 1999b).
3.3: Behaviors to support emotional relationships
To allow for the formation of emotional relationships between the user and the Petz and Babyz characters, we built a broad base of interactive behaviors. These behaviors offer the characters and the user the means of communicating emotion (or the appearance of emotion) to each other. This section will detail these interactions and behaviors, specifying in each the emotions we intended to be perceived by the user. Our hope is that by having the characters express emotion in a convincing and life-like way, the user will instinctively feel empathetic; and at the same time, if given the means to express their own emotions in return, users will feel like they are connecting to the characters on an emotional level.
Affection. Users can express affection to the characters by touching them and holding them with their mouse-controlled hand cursor. For Petz the touching is petting; for Babyz it is tickling. Petz express affection to the user by sweetly barking, meowing or purring, licking and nuzzling the hand cursor, and bringing the user a toy. Babyz will smile, giggle and laugh, coo, act cute, and say “mama” or “dada” in a loving voice. When perceiving a lack of affection, Petz will howl and yowl, sounding lonely; Babyz will cry and say “mama” or “dada” in a sad tone of voice. The intent is for the user to perceive the feelings of love, warmth, happiness and loneliness in the characters.
Nurturing. Users can feed, clothe and give medicine the characters. Petz express the need to be nurtured by acting excited when food is brought out, begging, acting satisfied and grateful after eating, or disgusted when they don’t like the food. Babyz may hold out their arms and ask for food, whine and cry if hungry or need a diaper change, and may throw and spit up food they don’t like. Users are meant to perceive feelings of craving, satisfaction, pleasure, gratefulness, dislike and disgust.
Play. By picking up a toy users can initiate play with one or more of the characters, such as a game of fetch or building blocks. Petz or Babyz may join the user’s invitation to play, or get a toy of their own and begin playing by themselves or each other, waiting for the user to join them. A character may react if the user is ignoring them and instead playing with another character. Emotions intended to be perceived by the user include excitement, boredom, aggressiveness, timidity, laziness, and jealousy.
Training. Users can give positive and negative reinforcement in the form of food treats, water squirts (for Petz), and verbal praise or discipline to teach the characters to do certain behaviors more or less often. Petz or Babyz are programmed to occasionally act naughty, to encourage users to train them. During these behaviors users are meant to perceive the emotions of feeling rewarded, punished, pride, shame, guilt, and anger.
3.4: Effective expression of emotion
None of the aforementioned behaviors would seem believable to the user unless the characters effectively expressed convincing emotions. We found all of the following techniques to be critical for successful real-time emotion expression in virtual characters.
Emotion expression in parallel with action. During any body action (such as walking, sitting, using objects, etc.) Petz and Babyz characters can display any facial expression or emotive body posture, purr or cry, make any vocalization or say any word in any of several emotional tones. This allows a baby character to look sad and say it wants while it crawls towards the user. Catz can lick their chops and narrow their eyes as they stalk a mouse. Characters can immediately sound joyful when the user tickles their toes. We found if a virtual character cannot immediately show an emotional reaction, it will not seem believable. Timing is very important.
Emotion expression at regular intervals. Programming the characters to regularly pause during the execution of a behavior to express their current mood was a very effective technique. For example, while upset and crawling for a toy that it wants, Babyz may stop in place and throw a short tantrum. Or just before running after a ball in a game of fetch, Dogz may leap in the air with joy, barking ecstatically. These serve no functional purpose (in fact they slow down the execution of a behavior), but they contribute enormously to the communication of the emotional state of the character. Additionally, related to Phoebe Sengers’ concept of behavior transitions (Sengers 1998), when Petz or Babyz finish one behavior and are about to begin a new one, they pause for a moment, appearing to “stop and think” about it, look around, and express their current mood with a happy bark or timid cower.
Emotion expression through customization of behavior. Some behaviors have alternate ways to execute, depending on the emotional state of the character. Mood may influence a character’s style of locomotion, such as trotting proudly, galloping madly in fear, or stalking menacingly. A hungry character may choose to beg for food if lazy, cry and whine for food if upset, whimper if afraid, explore and search for food if confident, or attack anything it sees if angry. The greater the number of alternate ways a character has to perform a particular behavior, the stronger and deeper the perceived illusion of life.
Prioritization of emotion expression, and avoidance of thrashing. It is possible for a character to have multiple competing emotions, such as extreme fear of danger simultaneous with extreme craving for food. We found it to be most believable if “fear” has the highest priority of all emotion expression, followed by “craving” for food and then extreme “fatigue”. All other emotions such as “happiness” or “sadness” are secondary to these three extreme emotional states. It is also important that characters do not flop back and forth between conflicting emotions, else their behaviors appear incoherent.
Theatrical techniques. Our characters are programmed to obey several important theatrical techniques, such as facing outwards as much as possible, looking directly outwards into the eyes of the user, and carefully positioning themselves relative to each other (“stage blocking”). If the user places a character offscreen, behind an object or at an odd angle, the characters quickly get to a visible position and turn to face the user. If two characters plan to interact with one another, such as licking each other’s noses or giving each other an object, they try to do this from a side view, so the user can see as much of the action and emotion expression as possible in both characters. Similar techniques were identified in the context of virtual characters in (Goldberg 1997).
3.5: Animation and behavior architecture
Animation and behavior are tightly integrated in the Petz and Babyz architecture. An attempt was made during software development to construct a clean modular code structure; however both time constraints and practical considerations forced us to at times adopt a more ad hoc, hackish approach to implementation.
The lowest level in the architecture is an animation script layer where frames of real-time rendered 3-D animation are sequenced and marked with timing for sound effects and action cues, such as when an object can be grabbed during a grasping motion. Above this is a finite state machine that sequences and plays the animation scripts to perform generic but complicated low-level behaviors such as locomotion, picking up objects, expressing emotions, and so on. The next level up is a goal-and-plan layer that controls the finite state machine, containing high level behaviors such as “eat”, “hide” or “play with object”. Goals are typically spawned as reactions to user interaction, to other events in the environment, or to the character's own internal metabolism. Goals can also be spawned deliberately as a need to regularly express the character's particular personality or current mood. At any decision point, each goal’s filter function is queried to compute how important it is for that goal to execute under the current circumstances. Filter functions are custom code in which the programmer can specify when a goal should execute. Part of the craft of authoring behaviors is balancing the output of these filter functions; it is easy to accidentally code a behavior to happen far too often or too seldom for believability.
Alongside instantiated goals are instantiated emotion code objects such as “happy”, “sad” and “angry”. Emotions have filter functions much like goals, but can also be spawned by the custom logic in states or goals. These emotion code objects themselves can in turn spawn new goals, or set values in the character’s metabolism. For example, the filter function of a pet’s “observe” goal may be activated in reaction to the user petting another pet. The “observe” goal is written to spawn a “jealousy” emotion if the other pet is a rival and not a friend. The “jealousy” emotion may in turn spawn a “wrestle” goal; any fighting that ensues could then spawn additional emotions, which may spawn additional goals, and so on. Goals are constantly monitoring what emotion objects are currently in existence to help decide which plans to choose and how they should be performed; states monitor emotions to determine which animations, facial expressions, and types of sound to use at any given moment.
Note that by no means did we implement a “complete” model of emotion. Instead, we coded only what was needed for these particular characters. For example, the Babyz have no “fear” emotion, because acting scared was not necessary (or considered entertaining) for the baby characters we were making. Fear was necessary however for the Petz personalities such as the scaredy cat. The emotion lists for Petz and Babyz varied slightly; it would have been inefficient to try to have both to use the same exact model.
At the highest level in the architecture is the “free will” and narrative intelligence layer. This is custom logic that can spontaneously (using constrained randomness) spawn new goals and emotions to convey the illusion that the character has intent of its own. This code is also keeping track of what goals have occurred over time, making sure that entertaining behaviors are happening regularly. It keeps track of long-term narratives such as learning to walk, sibling rivalries and mating courtship.
4: Feeling holistic: The importance of design
In creating an interactive emotional artifact, even the best animation and artificial intelligence technology will be lost and ineffective without a solid design. In this section we discuss the importance of the overall design of an interactive experience to ensure that a virtual character's emotions are effective and powerful to the user.
Concept and context. The type of characters you choose and the context you present them in will have a great impact on how engaging and emotionally powerful the interactive experience is. Judging by the confusing and poorly thought-out concepts in many pieces of interactive entertainment today, we feel this is a design principle too often ignored. In our products were careful to choose characters that people immediately recognize – dogs, cats, and babies – which allow users to come to the experience already knowing what to do. They immediately understand that they need to nurture and play with the characters.
Even though Petz and Babyz are presented in a cartoony style, we were careful to keep their behavior in a careful balance between cartooniness and realism. This was important to maintain believability and the illusion of life; if the Petz stood up on their hind legs and began speaking English to one another, users would not have been able to project their own feelings and experiences with real pets onto the characters. One of our maxims was “if Lassie could do it, our Petz can do it.” That is, the Petz can do a bit more than a real dog or cat would normally do, but nothing that seems physically or mentally impossible.
A very important design principle in Petz and Babyz for supporting emotional relationships is that users play themselves. Users have no embodied avatar that is supposed to represent them to the characters; the hand cursor is meant to be an extension of their real hand. The characters seem to “know” they are in the computer, and they look out at the user as if they actually see them. There is no additional level of abstraction here; you are you, and the characters are the characters. This is akin to a first-person versus third-person perspective. If the user had an avatar that they viewed from a third-person perspective, the other characters would be required to look at that avatar, not at the user directly, thereby weakening the impact of their emotional expression.
Direct, simple user interface. Petz and Babyz are almost completely devoid of the typical user interface trappings of most interactive entertainment products. To interact with the characters, users operate the hand cursor in a “natural”, direct way to touch and pick up characters and objects. No keyboard commands are required. All of the objects in the virtual world are designed to be intuitively easy to use; you can throw a ball, press keys on a toy piano, open cabinets and so on.
Of course this simplicity limits the amount of expressivity offered to the user. We cannot make objects and behaviors that require more complicated operation, such as a holding a baby and a milk bottle at the same time. While we could program some obscure arbitrary keyboard command sequence to accomplish this, we have chosen not to in order to keep the interface as pure and simple as possible. To allow the user more expressivity we would be required to add more intuitive interface channels, such as a dataglove or voice recognition. In fact instead of typing words to your Petz and Babyz (which you would never do in real-life of course), the latest versions of the products allow you to speak some basic words to the characters.
In general we feel that user interface, not animation or artificial intelligence technology, is the largest impediment for creating more advanced virtual characters. With only a mouse and keyboard users are very constrained in their ability to naturally their express emotions to virtual characters. When interacting with characters that speak outloud, users should be able to speak back with their own voice, not with typing. Unfortunately voice recognition is still a bleeding-edge technology. In the future we look forward to new interface devices such as video cameras on computer monitors that will allow for facial and gesture recognition (Picard 1997).
Natural expression. When trying to achieve believability we found it effective for characters to express themselves in a natural way, through action and behavior, rather than through traditional computer interface methods such as sliders, number values, bar graphs or text. In Petz and Babyz the only way the user can understand what the characters seem to be feeling is to interpret their actions and physical cues, in the same way an audience interprets an actor's performance. We do not display bar graphs or text messages describing the characters’ internal variables, biorhythms or emotional state. By forcing a natural interpretation of their behavior, we don't break the illusion of a relationship with something alive.
Favor interactivity and generativity over a high resolution image. In Petz and Babyz we made a tradeoff to allow our characters to be immediately responsive, reactive and able to generate a variety of expressions, at the expense of a higher resolution image. Surprisingly most game developers don’t make this tradeoff! From a product marketing perspective, a beautiful still-frame is typically considered more important than the depth and quality of the interactive experience. Of course there is a minimum level of visual quality any professional project needs, but we feel most developers place far too much emphasis on flashy effects such as lighting, shading and visual detail (i.e., spectacle), and not enough emphasis on interactivity and generativity.
Purity versus “faking it”: take advantage of the Eliza effect. The “Eliza effect” – the tendency for people to treat programs that respond to them as if they had more intelligence than they really do (Weizenbaum 1966) – is one of the most powerful tools available to the creators of virtual characters. As much as it may aggravate the hard-core computer scientists, we should not be afraid to take advantage of this. “Truly alive” versus “the illusion of life” may ultimately be a meaningless distinction to the audience. 99% of users probably won’t care how virtual characters are cognitively modeled – they just want to be engaged by the experience, to be enriched and entertained.
5: Conclusion
This paper has put forth virtual characters as a new form of emotional artifact, and the arrival of emotional relationships between humans and virtual characters as a new social phenomenon and direction for story and art. The design and implementation techniques we found useful to support such emotional relationships in the Virtual Petz and Babyz projects have been presented. We will conclude with some final thoughts on what it could mean for a person to have an emotional relationship with a virtual character.
Are relationships between people and virtual characters somehow wrong, perverse, even dangerous – or just silly? Again we can look to the past to help us answer this. Audiences that read about or see emotional characters in traditional media – painting, sculpture, books, theater, television and movies – have been known to become very “attached” to the characters. Even though the characters are not real, they can feel real to the audience. People will often cry when the characters suffer and feel joy when they triumph. When the written novel first appeared it was considered dangerous by some; today we find that this is not the case. However television, a more seductive medium than the novel, has certainly captured the free time in the lives of many people. Some consider the effect of television and videogames on children’s development to be a serious problem; the media has even reported a few outrageous stories of people going to death-defying lengths to take care of their virtual pet Tamagotchis. Designers should be aware that manmade characters have the potential to have a powerful effect on people.
Why create artificial pets and humans? Isn’t it enough to interact with real animals and people? From our perspective on making Virtual Petz, this was not the point. Our intent was not to replace people’s relationships with real living things, but to create characters in the tradition of stuffed animals and cartoons. And while some people are forming emotional relationships with today’s virtual characters, by and large they are still thought of as sophisticated software toys that try to get you to suspend your disbelief and pretend they are alive. However as we move towards virtual human characters such as Babyz, the stakes get higher. As of this writing we have not yet gotten feedback from the general public on their feelings and concerns about Babyz.
Also, the characters made so far have been “wholesome” ones, such as dogs, cats and babies, but one could easily imagine someone using these techniques to create characters that could support other types of emotional relationships, from the manipulative to the pornographic. Inevitably this will happen. Of course the promise and danger of artificial characters has long been an area of exploration in literature and science fiction, ranging from friendly, sympathetic characters such as Pinocchio and R2D2 to more threatening ones such as Frankenstein and HAL9000.
As virtual characters continue to get more life-like, we hope users keep in mind that someone (human) created these virtual characters. Just as an audience can feel a connection with the writer, director or actor behind a compelling character on the written page or the movie screen, a user could potentially feel an even stronger connection to the designer, animator and programmer of an interactive virtual character. For the artist, the act of creating a virtual character requires a deep understanding of the processes at work in the character’s mind and body. This has always been true in traditional art forms, from painting and sculpting realistic people to novels to photography and cinema, but it is taken to a new level with interactive virtual characters. As the artist you are not just creating an “instantiation” of a character – a particular moment or story in the character’s life – you are creating the algorithms to generate potentially endless moments and stories in that character’s life.
People need emotional artifacts. When the public gets excited about buzzwords like “artificial intelligence” or “artificial life”, what they are really asking for are experiences where they can interact with something that seems alive, that has feelings, that they can connect with. Virtual characters are a promising and powerful new form of emotional artifact that we are only just beginning to discover.
Acknowledgements
Virtual Petz and Babyz were made possible by a passionate team of designers, programmers, animators, artists, producers and testers at PF.Magic / Mindscape that include Adam Frank, Rob Fulop, Ben Resner, John Scull, Andre Burgoyne, Alan Harrington, Peter Kemmer, Jeremy Cantor, Jonathan Shambroom, Brooke Boynton, David Feldman, Richard Lachman, Jared Sorenson, John Rines, Andrew Webster, Jan Sleeper, Mike Filippoff, Neeraj Murarka, Bruce Sherrod, Darren Atherton and many more. Thanks to Robert Trappl and Paola Petta for organizing such a fascinating and informative workshop.
Appendix: Real customer letters
I had a dog that was a chawawa and his name was Ramboo. Well he got old and was very sick and suffering so my parents put him to sleep. Ever since then I have begged my parents for a new dog. I have wanted one soo bad. So I heard about this dogz on the computer. I bought it and LOVE it!!! I have adopted 9 dogs. Sounds a bit to much to you ehhh? Well I have alot of free time on my hands. So far everyday I take each dog out one by one by them selves and play with them, feed them, and brush them, and spray them with the flee stuff. I love them all. They are all so differnant with differant personalitys. After I take them out indaviually then I take 2 out at a time and let them play with me with each other. Two of the dogs my great Dane and chawawa dont like to play with any of the other dogs but each other. This is a incrediable program. I had my parents thinking I was crazy the other night. I was sitting here playing with my scottie Ren and mutt stimpy and they where playing so well together I dont know why but I said good dog out loud to my computer. I think my parents wondered a little bit and then asked me what the heck I was doing. But thankz PF.Magic. Even though I cant have a real dog it is really nice to have some on my screen to play with. The only problem now is no one can get me away from this computer, and I think my on-line friendz are getting a little mad cause im not chatting just playing fetch and have a great time with my new dogz. Thanks again PF. magic. I love this program and will recomend it to everyone I know!!!!!!!
I am a teacher and use the catz program on my classroom PC to teach children both computer skills and caring for an animal. One of the more disturbed children in my class repeatedly squirted the catz and she ran away. Now the other children are angry at this child. I promised to try and get the catz back. It has been a wonderful lesson for the children. (And no live animal was involved.) But if there is any way to get poor Lucky to come homze to our clazz, we would very much appreciate knowing how to do it. Thanks for your help, Ms. Shinnick's 4th grade, Boston, MA
Dear PF. Magic, I am an incredible fan of your latest release,Petz 3,I have both programs and in Janurary 1999,my cherised Dogz Tupaw was born. He is the most wonderful dogz and I thank you from the bottom of my heart, because in Janurary through to the end of April I had Anorexia and i was very sick. I ate and recoverd because i cared so much about Tupaw and i wanted to see him grow up. I would have starved without you bringing Petz 3 out. Please Reply to this,it would mean alot to me. Oh,and please visit my webpage,the url is http://www.homestead.com/wtk/pets.html. Thankyou for releasing petz 3,Give your boss my best wishes, Sincerily, Your Number One Fan, Faynine
I just reciently aquired all your Petz programs and I think they are great! I really love the way the animals react. I raised show dogs and have had numerous pets of all kinds in my life and making something like this is great. I am a school bus driver and have introduced unfortunate kids to your program. Children who not only can they not afford a computer but they can't afford to keep a pet either. This has taught them a tremendous amount of responsibilty. I am trying to get the school to incorporate your programs so as to give all children a chance to see what it is like to take care of a pet. It might help to put a little more compassion in the world. Please keep me updated on your newest releases. Thanks for being such a great company. Nancy M. Gingrich
Dear PF.Magic, Hello! My name is Caitlin, and I'm 10 years old. I have Dogz 1 and Catz 1, as well as Oddballz, and I enjoy them all very much. Just this morning was I playing with my Jester breed Catz, Lilly. But I know how much better Petz II is. For a while, I thought I had a solution to my Petz II problem. I thought that if only I could get Soft Windows 95 for $200, that would work. Well, I took $100 out of my bank account (by the way, that's about half my bank account) and made the rest. I cat-sit, I sold my bike, and I got some money from my parents. Anyway, I really, really love animals (I'm a member of the ASPCA, Dog Lovers of America, and Cat Lovers of America) but I can't have one! That's why I love Petz so much! It's like having a dog or cat (or alien for that matter) only not. It's wonderful! I have a Scrappy named Scrappy (Dogz), Chip named Chip (Dogz), Bootz named Boots (Dogz), Cocker Spaniel named Oreo (Dogz), Jester named Lilly (Catz), and Jester named Callie (Catz). And then every single Oddballz breed made. =) I don't mean to bore you as I'm sure this letter is getting very boring. I would love SO MUCH to have Petz II. I really would. (At this point in the letter I'm really crying) I adopted 5 Catz II catz at my friend's house, but I go over to her house so little I'm sure they'll run away. I'd hate for them to run away. Is there anything I can do? I love my petz, and I'm sure they'd love Petz II. Thank you for reading this. Please reply soon. ~*~ Caitlin and her many petz ~*~
My husband went downtown (to Manchester) and found Catz for sale, and having heard so much about it he bought it on the spot. He put it on his very small laptop and came back from one of his business trips saying, "How many Dutchmen can watch Catz at once on a little laptop on a Dutch train?" The answer was TEN. I asked if any of them said, "Awww," the way we all did, but he said they all walked off saying it was silly. I bet they ran out to buy it anyway, though! Yours, Mrs. H. Meyer
Dear Sirs, Just wanted to thank-you for the pleasure my petz have brought me. I am paralyzed from the neck down yet your program has allowed me too again enjoy the pleasure of raising my own dogz. I have adopted 5 so far. I love them equally as if they were real. Thanks again
References
Bandai. 1996. Tamagotchi keychain toy. Website URL: http://www.bandai.com
Bates, J. 1992. The Nature of Characters in Interactive Worlds and The Oz Project. Tech Report CMU-CS-92-200, School of Computer Science, Carnegie Mellon University.
Bates, J. 1994. The Role of Emotion in Believable Agents. Technical Report CMU-CS-94-136, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
Blumberg, B. 1997. Multi-level Control for Animated Autonomous Agents: Do the Right Thing… Oh, Not That… In Creating Personalities for Synthetic Actors, R. Trappl and P. Petta (Eds.), Springer-Verlag, New York.
Boehlen, M. and Mateas, M. 1998. Office Plant #1: Intimate space and contemplative entertainment. Leonardo, Volume 31 Number 5: 345-348.
Damasio, A. 1994. Descartes' Error: Emotion, Reason and the Human Brain. Putnam Publishing Group, New York.
Dautenhahn, K. 1997. Ants Don’t Have Friends – Thoughts on Socially Intelligent Agents. In Proceedings of the 1997 AAAI Fall Symposium, Socially Intelligent Agents, Menlo Park: AAAI Press.
Elliot, C.; Brzezinski, J.; Sheth, S.; and Salvatoriello, R. 1997. Story-morphing in the Affective Reasoning paradigm: Generating stories automatically for use with ``emotionally intelligent'' multimedia agents. Institute for Applied Artificial Intelligence, School of Computer Science, Telecommunications, and Information Systems, DePaul University.
Egri, L. 1946. The Art of Dramatic Writing. Simon & Schuster, New York.
Frank, A.; Stern, A.; and Resner, B. 1997. Socially Intelligent Virtual Petz. In Proceedings of the 1997 AAAI Fall Symposium, Socially Intelligent Agents, FS-97-02, pp. 43-45. Menlo Park: AAAI Press.
Frank, A.; and Stern, A. 1998. Multiple Character Interaction Between Believable Characters. In Proceedings of the 1998 Computer Game Developers Conference, pp. 215-224. Miller Freeman, San Francisco.
Goldberg, A. 1997. IMPROV: A System for Real-Time Animation of Behavior-Based Interactive Synthetic Actors. In Creating Personalities for Synthetic Actors, R. Trappl and P. Petta (Eds.), Springer-Verlag, New York.
Grand, S.; Cliff, D.; and A. Malhotra. 1997. Creatures: Artificial Life Autonomous Software Agents for Home Entertainment. In Proceedings of the First Intl. Conference on Autonomous Agents, Minneapolis, ACM Press, pp. 22-9.
Hayes-Roth, B.; van Gent, R.; and Huber, D. 1997. Acting in Character. In Creating Personalities for Synthetic Actors, R. Trappl and P. Petta (Eds.), Springer-Verlag, New York.
Kline, C; and Blumberg, B. 1999. The Art and Science of Synthetic Character Design. 1999 Convention of the Society for the Study of Artificial Intelligence and the Simulation of Behavior (AISB), Edinburgh, Scotland.
Loyall, A. 1997. Believable Agents: Building Interactive Personalities, Ph.D. Dissertation, Carnegie Mellon University, Pittsburgh. CMU-CS-97-123.
Mateas, M. 1999. Not Your Grandmother's Game: AI-Based Art and Entertainment. In Proceedings of the 1999 AAAI Spring Symposium, Artificial Intelligence and Computer Games, SS-99-02, pp. 64-68. Menlo Park: AAAI Press.
Meehan, J. 1976. The metanovel : writing stories by computer. Ph.D. Dissertation, Dept. of Computer Science, Yale University.
Minsky, M. 1985. The Society of Mind. Simon & Schuster, New York.
Murray, J. 1997. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. The Free Press, New York.
Pemberton, L. 1989. A Modular Approach to Story Generation. In 4th Conference of the European Chapter of the Association for Computational Linguistics, pp. 217-224.
Penny, S. 1997. Embodied Cultural Agents: At the Intersection of Robotics, Cognitive Science and Interactive Art. In Proceedings of the 1997 AAAI Fall Symposium, Socially Intelligent Agents, FS-97-02, pp. 43-45. Menlo Park: AAAI Press.
Perlin, K. 1995. Real-time responsive animation with personality. IEEE Transactions on Visualization and Computer Graphics.
PF Magic / Mindscape. 1995 - 1999. Dogz and Catz: your Virtual Petz, Babyz. Website URLs: http://www.petz.com and http://www.babyz.net
Picard, R. 1997. Affective Computing. MIT Press.
Reilly, W. 1996. Believable Social and Emotional Agents. Ph.D. Thesis. Technical Report CMU-CS-96-138, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
Rilke, R. 1934. Letters to a Young Poet. W. W. Norton and Company, New York.
Sengers, P. 1996. Socially Situated AI: What It Means and Why it Matters. In Proceedings of the 1996 AAAI Symposium, Entertainment and AI/A-Life. Menlo Park: AAAI Press.
Sengers, P. 1998. Do the Thing Right: An Architecture for Action-Expression. In Proceedings of the Second Intl. Conference on Autonomous Agents, pp. 24-31. ACM Press, New York.
Sony Electronics. 1999. Aibo robot dog. Website URL: http://www.world.sony.com/robot/get/meet.html
Stern, A.; Frank, A.; and Resner, B. 1998. Virtual Petz: A Hybrid Approach to Creating Autonomous, Lifelike Dogz and Catz. In Proceedings of the Second Intl. Conference on Autonomous Agents, pp. 334-5. ACM Press, New York.
Stern, A. 1998. Interactive Fiction: The Story is Just Beginning. IEEE Intelligent Systems, Volume 13, No. 5, pp. 16-18. Los Alamitos: IEEE Computer Society.
Stern, A. 1999a. AI Beyond Computer Games. In Proceedings of the 1999 AAAI Spring Symposium, Artificial Intelligence and Computer Games, SS-99-02, pp. 77-80. Menlo Park: AAAI Press.
Stern, A. 1999b. Virtual Babyz: Believable Agents with Narrative Intelligence. In Proceedings of the 1999 AAAI Fall Symposium, Narrative Intelligence. Menlo Park: AAAI Press.
Thomas, F; and Johnston, O. 1981. Disney Animation: The Illusion of Life. Abbeville Press, New York.
Tiger Electronics. 1998. Furby toy. Website URL: http://www.furby.com
Turner, S. 1994. The Creative Process: A Computer Model of Storytelling and Creativity. Lawrence Erlbaum Associates.
Turkle, S. 1999. Ongoing virtual pet study. Website URL: http://web.mit.edu/sturkle/www/vpet.html
Weizenbaum, J. 1966. Eliza. Communications of the ACM, 9:36-45.
Wurst, K. and McCartney, R. 1996. Autonomous Robots as Performing Agents. In Proceedings of the 1996 AAAI Symposium, Entertainment and AI/A-Life. Menlo Park: AAAI Press.
PF.Magic, 501 2nd St, Suite 400, San Francisco, CA 94107 | tel: +1 415 495 0400
|
