Consciousness Reframed
University of Applied Arts, Vienna,
‘Virtual Emotions, No Feelings’ paper publication, Springer Verlag
Vienna, 3rd July – 5th July 2008
Dr. Barbara Rauch
www.sciria.org.uk
www.icfar.co.uk
Abstract
I would like to suggest a presentation of my AHRC-funded project, ‘Mapping Virtual Emotions: 3D-surface capturing of animated facial expressions in animals and humans’, which I am undertaking in association with the Sensory Computer Interface Research & Innovation for the Arts Unit (SCIRIA) at the University of the Arts London. I use a 3D high-resolution laser scanner to capture animal and human faces. I use the data from these faces, animate and then combine them with human emotional facial expressions. In doing so I hope to visualize through critical experimentation what evolution has selected and accommodated. While it is often through the use of new technologies that we aim to expand our current understanding of the world, I would question whether it is possible to imagine beyond this in terms of human perception and the way in which we analyze and rationalize, taking into account the emotional responses we usually house as human beings.
The focus of this interdisciplinary practice-based research is a theory suggested by Charles Darwin (1872) over 125 years ago. It is the idea that our human facial expressions, contrary to what we often like to believe, are not unique to human beings. Darwin’s metatheory of the continuity of species explains that neither our facial expressions nor the musculature in the face are unique to humans. Both are the product of evolution and internal physiology (Ekman 1998, p.xxv – xxvii).
My presentation includes a visual documentation of the practical models and animations produced over the last two years. An old theory is being re-examined with the use of new technologies and new theories of Virtual Reality.
The project ‘Mapping Virtual Emotions: 3D-surface capturing of animated facial expressions in animals and humans’ adopts an interdisciplinary and practical approach to explore the original overall research question and the study of human emotions, and in particular how we project those onto animals. The data capturing of the animal and human faces was done at two places in different stages. I started with the portable laser scanner from the SCIRIA research unit, University of the Arts London to get data from stuffed animals, and then additional photogrammatic scanning was undertaken at the Molecular Medicine Unit, UCL, to capture the human face and its expressions. As suggested in my original proposal, I have studied theories of emotions through the work of Damasio, Darwin, Ekman and LeDoux. Paul Ekman’s work was particularly relevant to the study of human facial expressions, while I returned to Darwin (and Ekman’s commentary on Darwin’s book ‘The Expression of the Emotions in Man and Animals’, from 1872) for details on animal facial expressions.
Building on my research into consciousness studies and emotions, I have developed a new artwork to reveal aspects of characteristic human emotions (i.e. laughing, crying, frowning, sneering, etc.), which uses new technology, in particular digital scanning devices and special effects animation software. The proposal is to use a 3D high-resolution laser scanner to capture animal faces and, using the data of these faces, animate and then combine them with human emotional facial expressions. The morphing of the human and animal facial data are not merely layers of the different scans; an algorithmic programme has been applied which merges crucial landmarks in the animal face to match with the human. The results are morphings of the physical characteristics of animals with the emotional characteristics of the human face in 3D.
I had the opportunity to exhibit my new work in the ‘Digital and Physical Surfaces’ exhibition in the Triangle Project Space at Chelsea College of Art & Design, University of the Arts London in February 2007. The work was an installation entitled ‘Virtual Emotions’. I presented a monitor piece on a trestle table which showed an animation of a human face morphing in and out of emotional expressions. The intention was to encourage the visitor to feel several emotions themselves while watching the person on screen. There were seven archive boxes for the visitors to file their own handwritten story, according to Ekman’s seven universal human emotions, i.e. anger, fear, disgust, sadness, happiness, etc. To the right of the table a 3D monitor displayed a fox’s head changing randomly to express different emotions, clearly showing signs of anger, disgust, etc. The scene was programmed so that the fox’s face appeared to be projected at some distance from the physical screen. This encouraged visitors to walk around the sculpture and attempt to look at the fox from different angles.
As human beings we are equipped to read any human facial expression. Despite cultural differences emotion research over the last four decades shows evidence that for most emotions a cross-cultural understanding is preserved. Expressions for “basic” emotions such as fear, anger, disgust, sadness, or enjoyment are not culture-specific. Ekman (1998, 1999) explains in more detail that these “basic” emotions are expressed universally by all humans, regardless of culture, race, sex, or ethniticity. It is a psychological fact that loss brings about sadness and threat triggers fear. Ekman employs the term “basic” to differentiate one emotion from another so as to contrast a position that considers “emotions as fundamentally the same, differing only in terms of intensity or pleasantness” (Ekman 1999). A second meaning of “basic” points to the evolutionary aspect that Ekman is interested in. An additional third meaning of the adjective “basic” indicates that emotions have evolved to handle “fundamental life tasks” and Ekman lists here achievement, loss and frustration.
Charles Darwin’s focus in ‘The Expression of the Emotions in Man and Animals’ was on emotions in other primates. His study included close observation of animals and humans.
That emotions are observable in other primates is a defining characteristic of emotion, and yet it is possible that there are some emotions that are unique to humans, though there is no convincing evidence that that is the case. Naturally our language capacity to express and describe emotions in words changes many aspects of emotional experience.
Emotions, as mentioned above, are considered as having evolved to deal with fundamental life tasks, including life threatening situations. For that reason it seems emotions ought to begin quickly; often they happen before we become aware that they have begun. This is important so that we can respond quickly to them in the case of an emergency. Ekman (1999) emphasises that emotions “happen” to us, they are “unbidden” and usually they are “not chosen by us”.
Furthermore Ekman (1999) emphasises that emotions regulate the way we think and this again is evident in memories, imagery and expectations. Emotions are personal and subjective, and how each emotion feels is at the centre of what an emotion is. Ekman (1999) stresses that the use of questionnaires is a well known problem, because people are when filling the questionaire not experiencing the emotion at that moment, but merely try to remember what it felt like.
As human beings one has to deal with fundamental life tasks and Ekman (1999) explains that this influences how we respond to an event which marks the emotions. Often we involuntarily signal the emotion to others with a facial expression or other body language. Animals and humans alike express emotions not only in their faces – though the face is considered a marker for emotions – but they also use the rest of the body, for example general posture, hand position, sequences of reactions or the voice all play crucial parts in the expression of a situation. Some expressions are a series of movements such as head down, back, forward, to the side, and hands can be added: the hand might cover part of the sad face to express shame, or it might cover the expression of enjoyment in the face to indicate coyness (Ekman 1993).
Ekman introduces an understanding of emotions as families of emotions so as to differentiate, for example, the many shades of anger-related emotions. The English language can reflect these subtle differences and scales in that we differentiate between emotions such as irritation, agitation, annoyance, grouchyness, unease, worry, shock, fright and horror. With the descriptive use of language and self-reflection inherent to the human being, emotions in human beings appear to be a more complex experience than that experienced by non-human primates. David Matsumoto (2007, p.43) describes this assumption through the example of moral, what he terms an “interpersonal version of disgust”. While in the animal world a nasty object would trigger vomiting, a human being can be disgusted by others as people and react with an outbreak of extreme feelings.
In addition Matsumoto adds that humans can also feign emotions: we can lie and express something that we do not feel. Ekman (2003, p.225) suggests that one take a test in reading faces. A catalogue of photos of faces and clear instructions on how to read them is appended to his book ‘Emotions Revealed, Understanding Faces and Feelings’. Subtle differences in muscle contractions around the eye, for example, can tell a true smile from a false one (the famous Duchenne Smile (Darwin 1998 p.200)).
Image 2
Reading and understanding the expression on a human face is usually straightforward. One puts oneself in the position of the other and feels what the person making a particular face feels (Hansen 2004, p.158). Making a face might even generate the experience of the particular emotion expressed. This is more difficult with human-animal interaction and even more complicated for human-computer interaction. It is however not impossible, as Derrick de Kerckhove explains, if we consider the computer an expanded biofeedback system that can instruct and teach us how we can adapt ourselves to new perceptions. This idea refers to Marshall McLuhan’s notion of the “extension of man”, and de Kerckhove also draws on Bergson’s distinction “between perception as a virtual action of the body on things and affection as a real action of the body on itself” (Hansen 2004, p.195 and footnote 84. p.311). Furthermore de Kerckhove discusses touch, a tactile modality, as we “[see] with the entire body” (Hansen 2004, p.232). I would like to add that we see and “feel” with the entire body.
Hansen elaborates on the shift from the visual to the affective and haptic. By exploring de Kerckhove’s argument of the disembodiment of visual experience in Virtual Reality, Hansen (2004) engages the facialization of the entire body as imagization of affection. In Hansen’s terms Virtual Reality is not simply the product of advances in technology and developments in computer graphics, but rather he insists that the experience of VR is grounded in the biological potential of human beings. It is to be understood as a body-brain achievement. In that sense VR is not technologically- but biologically-grounded. This new digital Virtual Reality is an adaptation to newly acquired technological extensions provided by New Media. (In the same vein this is further elaborated in my PhD thesis (Rauch 2005); Referring to Revonsuo (2006), I argued for the dreaming brain to be understood as a natural virtual reality model).
My work series ‘Virtual Emotions’ attempts to visualise an evolution of emotions on a scale that ranges from the abstract via animal emotion to the hybrid human body. The virtual digital face seems to suggest an image that does not refer to the Real, in Lev Manovich’s understanding of the term; the new media image has changed our understanding of what an image is: we zoom, we click, we are the active users of the digital image. Furthermore Manovich describes the new image as process, because the image can no longer be restricted to the level of surface appearance (Hansen 2004, p.10). The image must be extended to encompass the entire process by which information is made perceivable through embodied experience. Hansen explains the digitization of a facial image as interfacing with the digital. Hansen uses the digital face to explain affect as interface.
Gabriele Buzzi (2007), in ‘Expression and Dévisage: the face’s signification from art to reality’, describes the face as the most analogical part of the body. She explains how difficult it is to recreate it digitally. This is probably the same challenge that artists have felt for centuries when trying to depict expressions in the human face. Franz Xaver Messerschmidt and Charles Le Brun are two notable artists in this respect. Messerschmidt’s ‘Grimacing head No. 13 “Der Speyer” (lost)’and Le Brun’s hybrid heads depict animal expressions in humans. Le Brun’s drawings return me to the evolutionary account of emotional expressions. The drawings date from the 17th century and yet they are not unlike my recently-generated computer graphics.
Without doubt emotions are evolving as they are influenced by culture, context and behaviour. Matsumoto (2007) elucidates these three influences of human emotion. Western and Eastern societies have changed with the use of new technologies. Will our ability to read facial expressions slowly change with the new communication systems? Might people soon not be able to read facial expressions anymore? With the loss of ability to read an emotion might come too the loss of the experience itself?
Image 4
Captions for:
Image 1 = fox capture, animation stills
Image 2 = partners, animation stills
Image 3 = animation axes
Image 4 = ‘3D Virtual Emotion’, animation still
References:
Cacioppo, J. T, Tassinary, L. G., & Berntson, G. G., 2007. Handbook of Psychophysiology (3rd ed.). New York: Cambridge.
DAMASIO, A.R., 1994. Descartes’ Error: emotion, reason and the human brain. London: Papermac, Macmillan.
DAMASIO, A.R., 1999. The Feeling of What Happens. London: Heinemann.
DARWIN, C., 1872, 1998. The Expression of the Emotions in Man and Animals, introduction and afterword by editor Paul Ekman. London: Harper Collins Publishers.
DAWKINS, R., 1976. The Selfish Gene. Oxford: Oxford University Press.
EKMAN, P., 1993. Facial Expression and Emotion. American Psychologist, 48 (4), p.384-392.
EKMAN, P. ed., 1998. The Expression of the Emotions in Man and Animals. London: Harper Collins Publishers.
EKMAN, P., 1999. Basic Emotions. In T. Dalgleish and M. Power eds., Handbook of Cognition and Emotion. Sussex, U.K.: John Wiley & Sons, Ltd. Ch.3.
EKMAN, P., 2003. Emotions Revealed: Understanding Faces and Feelings. NY: Times Books Henry Holt and Company, LLC.
HANSEN, M., 2004. New Philosophy for New Media. Cambridge, MA: MIT Press.
Manovich, L., 2001. The Language of New Media. Cambridge, MA: MIT Press.
Matsumoto, D., 2007. Playing catch with emotions. In Journal of Intercultural Communication, 10, p.39-49.
MCLUHAN, M., 1964. Understanding Media: the extensions of man. New York: Mentor.
RAUCH, B., 2005. Ph.D. Natural and Digital Virtual Realities: a practice-based exploration of dreaming and online virtual environments. London: University of the Arts London.
RAUCH, B. & HARRISON, D., 2006. A Merging of Mindsets Through Collision and Collusion. In Technoetic Arts: a journal of Speculative Research. 5 (1), p.55–65.
RAUCH, B., 2007. Digital and Physical Surfaces: presentation of practice based research. Catalogue. London: University of the Arts London.
REVONSUO, A., 2006. Inner Presence: Consciousness as a Biological Phenomenon. Cambridge, MA: MIT Press.
Websites
Gabriele Buzzi, 2007. Available at http://www.vjtheory.net/web_texts/text_buzzi.htm [accessed on 24th June 2008]
Charles Le Brun, Paris 1619-1690. Available at http://www.charleslebrun.com [accessed on 24th June 2008]
Franz Xaver Messerschmidt, 1736–1783. Available at http://www.limmat.ch/schmid/fxm/ [accessed on 24th June 2008]
Current Position and Statement
Dr Barbara Rauch is an artist practitioner and academic with a research position at the University of the Arts London, Chelsea College of Art & Design. She is the holder of a 2-year AHRC research grant, ‘The Personalised Surface within Fine Art Digital Printmaking’ (together with Prof P. Coldwell, FADE). As co-applicant and co-investigator she conducts several case studies with an emphasis on 3-dimensional prints and screen-based works. As acting director at SCIRIA, Rauch leads on the ‘Virtual eMotions’ research group. The group investigates emotions and in particular human facial expressions. The project is a continuation of an AHRC funded research project, ‘Mapping Virtual Emotions: 3D-surface capturing of animated facial expressions in animals and humans’, that was completed in June 2007.
Rauch’s PhD thesis (2005) is entitled ‘Natural and Digital Virtual Realities – a practice-based exploration of dreaming and online virtual environments’ and is available through the British Library in London and as reference only at Camberwell College of Arts Library.
Rauch is a visiting senior fellow at the McLuhan Program in Culture & Technology, University of Toronto.
Keywords: 3D-surface capturing, animated facial expressions, evolution of emotions and feelings, technologically transformed realities.
