The Neuroscience of Gestures

96 
  
    
  
 Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 JA 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:Calibri;
	mso-ascii-font-family:Calibri;
	mso-ascii-theme-font:minor-latin;
	mso-hansi-font-family:Calibri;
	mso-hansi-theme-font:minor-latin;
	mso-fareast-language:EN-US;}
 
  “How can you tell what these people are talking about?”

“How can you tell what these people are talking about?”

I’d like to persuade you that gestures are a fundamental building block of human language and thought. This begins a series of blog posts on gestures and how physical movement in VR & AR affects cognition.

Part one of this series will deal with why gestures provide a shortcut to human thought. 

But first, on the tech front:
Devices to capture small hand gestures are already available (like Microsoft Hololens) and more are underway.  Project Soli at Google can use radar to track micro-motions and twitches. The radar from the device senses how the user moves his hands and can interpret the intent. Link to the full Project Soli video here.

Why are gestures powerful shortcuts to cognition?

I’m reposting an article from Scientific American here that answers “Why is talking with gestures so much easier than trying to talk without gesturing?”  Psychology professor Michael P. Kaschak responds:

A person in a fit of rage may have trouble verbalizing thoughts and feelings, but his or her tightly clenched fists will get the message across just fine.

Gesturing is a ubiquitous accompaniment to speech. It conveys information that may be difficult to articulate otherwise. Speaking without gesturing is less intuitive and requires more thought. Without the ability to gesture, information that a simple movement could have easily conveyed needs to be translated into a more complex string of words. For instance, pointing to keys on the table and saying, ‘The keys are there,’ is much faster and simpler than uttering, ‘Your keys are right behind you on the countertop, next to the book.’

The link between speech and gesture appears to have a neurological basis. In 2007 Jeremy Skipper, a developmental psychobiologist at Cornell University, used fMRI to show that when comprehending speech, Broca’s area (the part of the cortex associated with both speech production and language and gesture comprehension) appears to ‘talk’ to other brain regions less when the speech is accompanied by gesture. When gesture is present, Broca’s area has an easier time processing the content of speech and therefore may not need to draw on other brain regions to understand what is being expressed. Such observations illustrate the close link between speech and gesture.

Takeaways for VR/AR Designers:

  • People process information more deeply when they are gesturing
  • Verbal areas of the brain are more active when speech accompanies gestures 
  • The tech exists for picking up human micro-gestures

How to Use Gestures to Learn Faster

Gestures make it easier to learn.  When people are speaking and gesturing at the same time, they process information better.  From New York Magazine

"University of Chicago psychologist Susan Goldin-Meadow and her colleagues have found that when toddlers point at objects, they’re more likely to learn the names for things; that for adults, gesturing as you try to memorize a string of numbers prompts better recall; and that when grade-schoolers gesture, they’re better at generalizing math principles.

The authors found that the students in both gesture conditions were more likely to succeed on follow-up generalization problems, which required understanding the underlying principle beneath the first problem and applying it in novel situations. It’s a case study in how gesture 'allows you a space for abstraction,' Goldin-Meadow says. 'You’re not as tied to the particulars of an item, of a problem, a word, or an experience.' You’re not just talking with your hands, in other words; you think with them, too.

Researchers haven’t yet pinned down exactly how this connection works, but Goldin-Meadow believes part of it is that gestures reduce what psychologists call 'cognitive load,' or the amount of mental energy you’re expending to keep things in your working memory."


Gestures are a good illustration of how humans think with more than just our brains.  The brain can process more information with gestures than without them, which makes them pretty fundamental to human capabilities. 


Takeaways:

  • Users moving their hands inside of a digital experience has cognitive consequences
  • Giving users alternative, embodied ways to learn information will help them retain concepts
  • Gestures are effective because they allow working memory to offload effort

 

Hey Michael Abrash, This Is the Actual Future of VR

Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 X-NONE 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:10.0pt;
	font-family:ArialMT;}
 
    "Hey da Vinci, where’s my 4K resolution?"    

"Hey da Vinci, where’s my 4K resolution?"  

I just watched the OC3 Future of VR keynote from Michael Abrash, Chief Scientist of Oculus and was pretty underwhelmed by his predictions.  Short recap of what Abrash expects by 2021:

  • 4K resolution headsets
  • Increased pixel density
  • Variable depth of focus
  • Wider field of view
  • Perfect eye tracking
  • Augmented VR

Those are the wrong priorities.  Abrash doesn’t mention the actual future of VR, which is using how the mind works to make VR more immersive. The best experiences in the future of VR will harness the way that human cognition is influenced by the environment, social interactions, and body movements in order to influence the experience. 

Hyper focusing on 4K resolution, depth of focus, etc., Abrash is discounting the mind’s ability to fill things in.  The mind can compensate when details are missing.  Think about Minecraft - it’s a pretty low fidelity video game but it’s still fun and engaging. 

Let’s get nerdy and talk about generative models.  A generative model is a complete set of concepts, terms and activities that allows people to create expectations about their experience.  Grammar is an easy example of this.  Grammar is a generative model of language because it allows people to systematically construct and communicate.  In VR, what is the equivalent of grammar?  

A generative model is a body of knowledge that enables you to systematically construct for yourself the kinds of events and states of affairs that the domain tends to present to you.
— Andy Clark

The future of VR belongs to grammarians.  Well, not exactly grammarians, but the designers who create the system that trains people to fill in the gaps of their VR experiences. 

 
Takeaways for VR designers:

  • Understanding how the mind works and building human-centered experiences will unlock the full potential of VR.
     
  • Human cognition is influenced by the environment, social interactions, and body movements in order to influence the experience.
     
  • The future of VR is the creation of a generative model that helps users fill in the gaps of any experience.  It will change people’s expectations about what is occurring inside of the experience and subsequently influence their experience. 

Practical Tips on Time Travel: How to Transport Using Gestures in VR.

Derrick Rose probably wishes that he could travel to a time where he doesn't play for the Knicks.

Derrick Rose probably wishes that he could travel to a time where he doesn't play for the Knicks.

Why do we need time travel, when we already travel through space so far and fast? For history. For mystery. For nostalgia. For hope. To examine our potential and explore our memories. To counter regret for the life we lived, the only life, one dimension, beginning to end.
— James Gleick, Time Travel

Time travel is a powerful tool for storytellers, but here I argue that all VR/AR designers can use associations with time travel to communicate help users grasp concepts more quickly.  

For VR/AR designers, time travel presents an interesting case study of:

  • Using physical gestures to represent abstract concepts
  • Activating known associations with each of those abstract concepts

Make Physical Gestures Represent Time in Space

Conceptually, human move through time. That means that how we think about space affects how we think about time.  As new technologies for tracking gestures improve, that means that VR/AR designers can build existing brain/body connections into their operating systems.

In today’s world of VR devices, imagine that you wanted to activate people’s senses of the future or the past using HTC Vive controllers.  

HTC Vive controller

HTC Vive controller

First start by thinking about what represents the future.  Generally, people believe that the future is in front of them and they will even lean forward when thinking about it.  The opposite is true for the past where people can be observed leaning backwards when reminiscing.  

In VR, it’s a 360-degree environment so it’s no exactly clear what’s the front and back of any experience so you should anchor associations of the future and past to the user’s body using hand gestures. Point your index finger forward to advance time and extend your thumb being you to reverse time.  Or, if the user has a set of controllers, have the forefinger buttons map to future movement and thumb buttons move you into the past.  Plus, give users the sense of moving forward or backward if you want them to feel transported to the future or the past. 

But does it matter if I’m not literally building a VR game about time travel?

Yes! These physical associations with the future and the past can be used by all VR/AR designers. By now, I hope that I’ve convinced you that people are wired to perceive the future in front and the past behind. Now you can use associations your target user has with the future or the past to map back to physical gestures.

The future generally associated with the following:

  • It offers more influence over external events.  Meaning that it is easier to control outcomes in the future compared to similar situations than the past
  • People tend to expect more positive things and fewer negative things will happen to them in the future. 
  • This leads them to anticipate all sorts of self-improvements (being healthier, skinnier, wiser, etc.) due to them exercising, dieting, reading, etc. more.

Now if people have such a large sense of optimism and expect so many positive things in the future, this also means that their rumination on the future is:

  • More imaginative than realistic (no reality checks)
  • Focused on the fulfillment of their greatest desires
  • Filled with more stereotypes and less variety

Now that you know about each of the associations with the future and you can imagine the converse association for the past.  There is much more negativity, uncontrollable outcomes, variety and details in the past.  

Using physical cues for the future [front] and the past [behind] can bring to mind the associations that Americans already have with the future and the past.  Using physical gestures to represent concepts as abstract as the future, but could also bring to mind things associated with the future such as a greater sense of self-control.  

Here’s an example of how to use the gestures and associations with the future in an indirect way.  Let’s say that you wanted to give people a heightened sense of autonomy and control inside of your experience.  Prime them to act with more autonomy by having them lean into your experience.

Another instance where you could utilize these associations would be around storytelling.  If you wanted people to consume a narrative that you have created, let them sink back to watch it.  Indicate that the events have already occurred and that they cannot change.  

Takeaways

  • People come into any experience with pre-existing associations and it’s up to the designers to utilizing existing ones or train new ones.  
  • Abstract concepts such as the future or the past can have physical associations (leaning forwarding, etc).
  • It is possible to use physical gestures to represent abstract concepts.  Pointing with the index finger, or using the trigger button on a controller can map onto the future.  
  • Start by thinking about the feelings and associations that you want your users to have.  What the the associations and references that people have with those things and work backward from there.  

References

Kane, J., McGraw, A. P., & Van Boven, L. (2008). Temporally asymmetric constraints on mental simulation: Retrospection is more constrained than prospection. The handbook of imagination and mental simulation, 131-149.

 

 

Can you trust your own experience?

Does that mountain look steep to you? It depends on how heavy your backpack feels.

Does that mountain look steep to you? It depends on how heavy your backpack feels.

Still convinced that you can trust your own experience?  The behaviors of people using VR or AR are affected by characteristics and timing factors that you might not think would affect them. VR and AR designers have tremendous opportunities to influence people's experiences and subsequent beliefs about those experiences.  The focus of the Extended Mind blog is persuading people that their emotions and decisions are influenced by cues from the environment and the ways that they move their body.  

Here's a short (5 minute) video that explains how humans rely on cues that come in from the environment that aren't sight, hearing, taste, touch, or smell.  [Click here if the embedded link is not working.]

It’s always a combination of many senses that is producing your experience of consciousness at any time.  

Here's a transcript of the most important part of the video for VR/AR designers.  From Barry C. Smith:

We are not as good as learning about our experiences as we think that we are…Many of these senses are occurring at the same time and talking to one another.  Take sight for example, you think that you couldn’t confuse seeing with another sense and yet it’s easily manipulated by one of your other senses.  Next time you sit on an airplane…look along the cabin as you sit on the ground and you’ll see everything in relation to you.  Now, look again when you’re in the climb and you’ve taken off.  It will now look as if the front of the cabin is higher than you.  Now how can it look that way because you’re in exactly the same optical/visual relationship to everything in the cabin.  So what we know is that because of your sense of balance and your ear canals telling you that you’re tilting backwards and also perhaps the rush in the chest feeling the engines that’s actually influencing the visual scene and it’s changing the visual scene.

Once you have a multisensory view of perception, you have a better chance of explaining how we are in touch with ourselves and how we know about ourselves that we ordinarily do.
— Barry C. Smith

Additional evidence shows that human perceptions are influenced by sensory information in unexpected ways.  In one psychology study, researchers put backpacks on students and then asked them to guess the steepness of a hill.  They manipulated how heavy or light the backpacks were.  The people who wore the heavy backpack judged the same hill as more steep.  In the same study, the other people to judge the hill as more steep were those who were fatigued, out of shape, or elderly.  That means that putting a heavy backpack on people results in the same steepness evaluations as someone in declining health!  That's a surprising result if you would anticipate that the steepness of the hill would be a constant across many different types of people.  

Takeaways for VR/AR designers are:

  • People tend to rely on their own experiences, but they do not have full conscious access to all of their experiences and their judgments.
  • People have more than five senses to provide them sensory information.
  • People can experience the same digital environment in many different ways. Test your experience with a diverse group of people to gather a range of perspectives.  

Further Reading

Bhalla, M., & Proffitt, D. R. (1999). Visual–motor recalibration in geographical slant perception. Journal of Experimental Psychology: Human Perception and Performance, 25(4), 1076.

Barry C. Smith. Aristotle was wrong and so are we: there are far more than five senses. Aeon Video October 31, 2016

 

 

Implications of Texture & Pressure in VR

Microsoft unveiled its Normal Touch and Texture Touch controllers.  They are specialized controllers that give people feedback to make them believe they are touching real objects in VR. Longer write up here and video is embedded below.  

The controllers aren't perfect and you are limited to getting feedback on the tip of your finger. However, the technology itself is an important gateway to more immersive experiences. People have powerful associations with texture and pressure that designers can use to change people's feelings inside of the experience.  

In a previous post, Metaphors are Jet Fuel, I discussed how feeling differing textures affected subsequent decision-making without people's conscious awareness of the influence.  The example I used was the metaphor of rough or smooth social interactions. Consider these common metaphors in American English:  

  • Having a rough day
  • Using coarse language
  • Being rough around the edges
  • Acting as a smooth operator

These new Microsoft controllers could be key to experiences that depend on creating feelings that are commonly associated with texture and pressure.  When people feel those sensations on the controllers, it will activate their associations with softness, smoothness, pressure, etc.

Why is novelty important in VR/AR?

When you are in a novel situation, your brain processes information differently than when you are in your humdrum routine.  For example, can you remember your commute home last night?  Or what you ate for lunch last Wednesday?  In a normal routine, attention and memory don't encode as well compared to when you are doing something new and novel.  This is why some neuroscientists insist on driving home via a new route every night.  

Being able to change attention and memory processing has important consequences when you are making a VR/AR experience.  What things do you want your user to pay attention to?  What do you want them to remember after the experience?  Here's a model of cognitive processing that you can use to consider how new information is going to be presented to users called "Top-Down Bottom-Up."

Excerpted from: Mind: Journey to the Heart of Being Human’ by Daniel J Siegel:

In the view we will be using here, top-down refers to ways we have experienced things in the past and created generalised summaries or mental models, also known as schema, of those events. For example, if you’ve seen many dogs, you’ll have a general mental model or image of a generic dog. The next time you see a furry canine strolling by, your top-down processing might use that mental model to filter incoming visual input, and you won’t really see the uniqueness of this dog in front of you. You have overlaid your generalised image of ‘dog’ on top of the here-and-now perceptual stream of energy that creates the neural representation of ‘dog’. What you actually have in awareness is that amalgam of the top-down filtering of your experience.

So here, ‘top’ means that prior experience is activated, making it difficult to notice the unique and vibrant details of what is happening here and now. The top-down generalised notion of dog will shade and limit your perception of the actual animal in front of you. The benefit of top-down is that it makes your life more efficient. That’s a dog, I know what it is, I don’t need to expend any more energy than needed on insignificant, non-threatening things, so I’ll take my limited resources and apply them elsewhere. It saves time and energy, and therefore is cognitively efficient. That’s top-down processing.

On the other hand, if you’ve never seen a spiny anteater before, the first time you come across one on the trail, it will capture all of your attention, engaging your bottom-up processing so that you are seeing with beginner’s eyes. These are eyes leading to circuitry in the brain, not shaping and altering ongoing perception through the top-down filters of prior experience. You’ll be taking in as much pure sensation from eyesight as possible, without the top-down filter altering and limiting what you see now based on what you’ve seen before.

These screenshots are from UpLoadVR's playthrough of the game "Accounting" by Squanchtendo.  The game uses different environments, such as a windowless accounting office (image left) to contrast with fantasy worlds where there are talking trees (image right) and other bizarre characters.  When you are in the fantasy world of "Accounting," you are likely using "bottom up" cognitive processing.  People feel excited when they are trying something new.  They take information from the exterior world, move it up a level, add language, move it up, analyze, and the user experience shifts.  Novel experiences tend to be more pleasurable because they rely on sensation and not cognition. 

The most vivid part of the mind bubbles up through sensation and new experience when unencumbered by analytical thought.
— Daniel J. Siegel

More on sensation from Siegel:

Sensation might be as bottom-up as we get. Since we live in a body, our within-mind experience is shaped by the physical apparatus that lets us take in energy flow from the outside world. We have our first five senses of sight, hearing, smell, taste and touch; we have our proprioceptive sense of motion; and we have our ‘interoceptive’ sense of the signals from the interior of the body. These perceptual capacities to sense the outer world and internal bodily world are built upon the physical neural machinery that enables energy to flow. Information is created with these energy patterns, generated as ions flow in and out of membranes and chemicals are released in the pathways of neural activity. As energy flows into the brain from our external sense organs, such as our eyes and ears, or from internal receptors of our body’s muscles, bones and internal organs, we move from sensation to perception – with pure sensation as close as we get to being fully present in the world.

Key Takeaways for VR/AR experience designers

  • Novelty increases awareness, which influences attention and memory
  • Top-down is the analytical system, which relies on prior experience
  • Bottom-up is the sensation-driven system, which looks with the eyes of a beginner 
  • Take a new route home on your commute tonight to experience novelty yourself.  Then look for ways to integrate that type of novelty into your experience.  

 

Further reading: 

Siegel, Daniel J. The Open Mind.  Aeon Magazine.  October 25, 2016

UploadVR full play through of "Accounting"

Three Ways to Measure Presence in VR

bike sunset.jpeg

In writing my post about how taking physical action increases presence in VR, I got curious - how is presence being measured?  That turned out to be such a big question that it needed a separate post.  Here’s a round up of definitions and ways to measure presence.  I’ve tried to organize the info in a way that helps people who are actively doing user experience research in VR.  

In short, presence is the notion of “being there,” inside of a digital experience.  Here’s a longer definition: 

1. High resolution information displayed to the participant, in a manner that does not indicate the existence of the display devices. 

2. Consistency of the displayed environment across all sensory modalities;

3. The possibility of the individual being able to navigate through and interact with objects in the environment, including interaction with other actors which may spontaneously react to the individual;

4. The individual’s virtual body, their self-representation within the environment, should be similar in appearance or functionality to the individual’s own body, and respond appropriately to the movements of their head, eyes, and limbs;

5. The connection between individual’s actions and effects of those actions should be simple enough for the individual to quickly learn.

Each of these is naturally maximized in the context of a person acting in everyday life (Usoh, Catena, Arman, and Slater 2000).

The concepts of presence, immersion, and performance have been studied in VR for more than 20 years so there are many different sources to draw on. I’m going to discuss how physical, psychological, and behavioral data on presence is collected. 

Physical Evidence of Presence

Physical signs that can be objectively measured (rather than reported by the participant) are attractive to researchers. Heart rate is one such metric. A lesser know measure is the Galvanic skin response (GSR). When a human experiences psychological arousal, a broad term meaning they are alert, awake, and attentive, then their skin becomes temporarily better at conducting electricity. The implication for VR is that if people are going through an experience and their skin conductance increases, then you would assume that participants have a stronger sense of “being there” inside of an experience.  More details on GSR from the MIT Media Lab:

The skin conductance response is measured from the eccrine glands, which cover most of the body and are especially dense in the palms and soles of the feet. (These are different from the apocrine sweat glands found primarily in the armpits and genital areas.) The primary function of eccrine glands is thermoregulation -- evaporative cooling of the body -- which tends to increase in aerobic activity, so yes, activity can affect conductance. However, the eccrine glands located on the palms and soles have been found to be highly sensitive to emotional and other significant stimuli, with a measurable response that precedes the appearance of sweat…Arousal has been found to be a strong predictor of attention and memory.

In the case of the heart rate or the GSR, they are useful for signaling to designers that something is happening inside of the experience. However, more steps are necessary to determine exactly what is the user’s experience. Measuring heart rate or arousal is different from measuring what people are feeling. To learn that, you will have to ask participants or analyze their behavior.  

Assessing Psychological Presence

There are loads of existing survey questions that you could co-opt for your own user research.  The Slater, Usoh and Stead (1995) and Witmer and Singer (1998) questionnaires are both very highly cited.  Consider what the aims of your study are when deciding what type of survey questions to use with your participants.  

Click  here  to download a commercial from the MIT Media Lab on their skin conductance measurement device, the Galvactivator

Click here to download a commercial from the MIT Media Lab on their skin conductance measurement device, the Galvactivator

Witmer and Singer (1998) has 32-questions and covers the domains of Control, Sensory Input, Distraction, and Realism.  Example questions:

A. How much were you able to control events? 

B. How compelling was your sense of objects moving through space? 

C. How aware were you of your display and control devices? 

Questions from Slater, Usoh, and Stead (1995) are below: 

1. Please rate your sense of being in the virtual environment, on a scale of 1 to 7, where 7 represents your normal experience of being in a place. 

2. To what extent were there times during the experience when the virtual environment was the reality for you? 

3. When you think back to the experience, do you think of the virtual environment more as images that you saw or more as somewhere that you visited? 

4. During the time of the experience, which was the strongest on the whole, your sense of being in the virtual environment or of being elsewhere? 

5. Consider your memory of being in the virtual environment. How similar in terms of the structure of the memory is this to the structure of the memory of other places you have been today? By ‘structure of the memory’ consider things like the extent to which you have a visual memory of the virtual environment, whether that memory is in color, the extent to which the memory seems vivid or realistic, its size, location in your imagination, the extent to which it is panoramic in your imagination, and other such structural elements. 

6. During the time of your experience, did you often think to yourself that you were actually in the virtual environment? 

Behavioral analysis

When conducting user research, it's ideal to film the participant so you have the opportunity to view the footage afterward.  You might not know exactly what behavior you are looking for until you’re already on the last participant of the day.  It most cases it will work to take an inductive approach and learn what behaviors are important by observing people going through your experience.  

Examples of behaviors to track would be how people are interacting with menus.  Do they have to make multiple attempts to accomplish a task? Are they taking time to read your instructions or do they just dive right in?

Key Takeaways

  • Don't try to reinvent the wheel when it comes to measuring presence - there's plenty of valid and reliable research to draw on.  
  • Combine various sources of data in order to build the most robust results.
  • Physical data typically only tells you that something is happening (increased heart rate or skin conductance), but not what the user's actual experience is.  
  • Surveys are used to gather psychological measures.  An advantage of them is that data collected can easily be compared over time and across experiences / platforms / etc.
  • Behavioral data can be tricky to collect because you may not know what you’re looking for, but it’s essential to telling the complete story of the user experience. 

 

Further reading

Galvactivator FAQ product page at the MIT Media Lab. 

Slater, M., Usoh, M., & Steed, A. (1995). Taking steps: the influence of a walking technique on presence in virtual reality. ACM Transactions on Computer-Human Interaction (TOCHI), 2(3), 201-219.

Slater, M., McCarthy, J., & Maringelli, F. (1998). The influence of body movement on subjective presence in virtual environments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 40(3), 469-477.

Usoh, M., Catena, E., Arman, S., & Slater, M. (2000). Using presence questionnaires in reality. Presence: Teleoperators and Virtual Environments, 9(5), 497-503.

 

Reality Is What You Do (Not What You See)

“Being there” means the capability to act there. 

“Being there” means the capability to act there. 

Your perception of reality is based on what you can do.  When you are inside of a VR environment, the more functionality that you have, the more the experience resembles your every day life.  You believe an object is real when you can interact with it, not just when you see it.  

Presence is defined as a sense of “being there,” or the extent to which virtual environments are perceived as places visited rather than images seen.  If you accept that presence is a design ideal for VR environments, there are systematic ways to increase users’ feelings of it.  Here I review two scientific papers on using body movement to heighten presence.  

Locomotion: Walking in Place vs. Using a Mouse

The degree of presence depends on the match between proprioceptive and sensory data.  Researchers at the University of London asked people to walk in place while they were inside of a virtual experience.  The gaze of the participant in the HMD determined what direction people felt they were walking in.  They compared the walking-in-place experience against the use of a computer mouse for locomotion.  The researchers believed that walking in place offered an advantage because is that it doesn’t require people to use their hands for navigation.  

The hand may be entirely reserved for the purposes for which it is used in everyday reality, that is, the manipulation of objects and activation of controls.
— Slater, Usoh & Steed (1995)
Valve's The Lab uses a teleportation function built into the hand controllers.  The user points a light beam where they want to go and release the trigger to teleport.  In this case, the user will land in the green circle in front of them.

Valve's The Lab uses a teleportation function built into the hand controllers.  The user points a light beam where they want to go and release the trigger to teleport.  In this case, the user will land in the green circle in front of them.

From the Slater, Usoh, M. and Steed, A. (1995) article: 

A fundamental requirement for an effective virtual reality is, therefore, that there is a consistency between proprioceptive information and sensory feedback, and in particular, between the mental body model and the virtual body…Proprioception is “the continuous, but unconscious sensory flow from the movable parts of our body (muscles, tendons,joints) by which their position and tone and motion [are] continually monitored and  and adjusted, but in a way that ishidden from us because it is automatic and unconscious.” (Sacks 1985).  Proprioception allows us to form a mental model that describes the dynamic spatial and relational disposition of our body and its parts. We know where our left foot is (without having to look) by tapping into this body model. We can clap our two hands together (with closed eyes) similarly by relying on this unconscious mental model formed from the proprioceptive data flow.

The control groups (the “pointers”) navigated the environment using a 3D mouse, initiating movement by pressing a button, with direction of movement controlled by pointing. The experimental groups (the “walkers”) used the walking technique. In each case the mouse was also used for grasping objects. The task was to pick up an object, take it into a room, and place it on a particular chair. The chair was placed in such a way that the subjects had to cross a chasm over another room about 20 feet below in order to reach it…With respect to the ease of navigating the environment, subjects in both experiments marginally preferred to use the pointing technique. This result was not surprising: as Brooks et al. [ 1992] noted, with the real treadmill more energy certainly is required to use the whole body in a walking activity, compared to pressing a mouse button or making a hand gesture (or driving a car, with respect to the similar comparison in everyday reality).

This is quite interesting…People found that the mouse was EASIER, but walking was more natural.  More evidence that the best experiences might not be the easiest ones.

Other results showed that “for the “walkers” the greater their association with the virtual body the higher the presence score, whereas for the “pointers” there was no correlation between virtual body association and the presence score. In other words, participants who identified strongly with the virtual body had a greater degree of reported presence if they were in the “walking” group than if they were in the “pointing” group. Association with the virtual body is important.…We argue that the walking technique [helps people match their proprioception to their sensory information,] compared to the pointing technique,and therefore other things being equal should result in a greater sense of presence. However, we found that this is modified by the degree of association of the individual with the virtual body…The virtual body association is significantly positively correlated with a subjective presence for the walkers but not for the pointers, which is certainly consistent with the proposed model.

This is important because it means that Samsung Gear VR experiences where a person is represented as a black hole and has no virtual body, presence is going to be very difficult to create.

Beyond Walking: The Influence of Bending, Standing, and Task Complexity on Presence

The researchers of the walkers vs. pointers study conducted a follow up experiment where they asked people to “walk” in place through a forest.  They varied the height of the trees for some participants, meaning that when there was high (vs. low) variability, people inside of the experiment had to bend down and look up more.  

The results showed a significant positive association between reported presence and the amount of body movement, in particular head yaw, and the extent to which subjects bent down and stood up…The practical importance of the results of this experiment is that since there does seem to be a relationship between body movement and presence, it is a reasonable goal to design interactive paradigms that are based on semantically appropriate whole body gestures. These will not only seem more ‘natural’, but may also increase presence. We further believe that the increase in presence in itself will engender more body movement, which in turn will generate higher presence, and so on.

Interestingly, adding a layer of cognitive effort did not increase user feelings presence.  They manipulated task complexity by asking some participants to count the number of trees that they saw and remember the distribution of diseased trees.  However, there was no increase in presence by having to exert mental effort.  

Head Yaw is Good for Presence

Head Yaw is Good for Presence

Using Walking in Place to Make Stairs and Ladders

If you are doing a Harry Potter wizarding experience, then flying or teleporting might be the best locomotion.  However, if you are trying to do an education simulation, such as training fire fighters, consider integrating more humdrum actions.  

The same idea [of walking-in-place] can be applied to the problem of navigating steps and ladders. One alternative is to use the familiar pointing technique and to “fly.” While in some applications there maybe a place for such magical activity, the very fact that mundane objects such as steps and ladders are in the environment would indicate that a more-mundane method of locomotion be employed. The walking-in-place technique carries over in a straightforward manner to this problem. 

When the collision detection process in the virtual reality system detects a collision with the bottom step of a staircase, continued walking will move the participant up the steps. Walking down the steps is achieved by turning around and continuing to walk. If at any moment the participant’s virtual legs move off the steps (should this be possible in the application), then they would “fall” to the ground immediately below. Since walking backward down steps is something usually avoided, we do not provide any special means for doing this. However, it would be easy to support backward walking and walking backward down steps by taking into account the position of the hand in relation to body line: a hand behind the body would result in backward walking.

Ladders are slightly different; once the person has ascended part of the ladder, they might decide to descend at any moment. In the case of steps, the participant would naturally turn around to descend. Obviously this does not make sense for ladders. Also, when climbing ladders it is usual for the hands to be used. Therefore, in order to indicate ascent or descent of the ladder, hand position is taken into account. While carrying out the walking-in-place behavior on a ladder, if the hand is above the head then the participant will ascend the ladder and descend when below the head. Once again it is a whole-body gesture, rather than simply the use of the hand, that is required in order to achieve the required result in an intuitive manner. If at any time the virtual legs come off the rungs of the ladder, then the climber will “fall” to the ground below.

Key Takeaways to Maximize Presence in VR

  • Presence is defined as the user’s sense of “being there” inside of a simulated environment.  
  • The way that you believe you can interact with your environment is just as important as what you see in VR.
  • Walking-in-place has been proven to be a metaphor for locomotion and navigation that increases presence.
  • There is evidence that using body movements such as walking, bending down, and moving your head also heightens a sense of presence.
  • Cognitive complexity does not increase a sense of presence.  

Further reading

Sacks, Oliver. (1985). The Man Who Mistook His Wife for a Hat. Picador, London.

Slater, M., McCarthy, J., & Maringelli, F. (1998). The influence of body movement on subjective presence in virtual environments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 40(3), 469-477.

Slater, M., Usoh, M., & Steed, A. (1995). Taking steps: the influence of a walking technique on presence in virtual reality. ACM Transactions on Computer-Human Interaction (TOCHI), 2(3), 201-219.

 

Metaphors are Jet Fuel

96 
  
    
  
 Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 X-NONE 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:Calibri;
	mso-ascii-font-family:Calibri;
	mso-ascii-theme-font:minor-latin;
	mso-hansi-font-family:Calibri;
	mso-hansi-theme-font:minor-latin;}
 
   C’est ne pas une brush

C’est ne pas une brush

VR/AR experiences will live or die by how quickly a user can learn how the experience works.  Thankfully, designers have an arsenal of tools in the form of metaphor to help speed the acquisition of knowledge.  Metaphors are cognitive shortcuts.  If I tell you, “I had a rough morning,” you know exactly what I mean. 

In Google Tilt Brush, you can choose the type of brush that you want from the palette.  However, you’re not choosing an actual brush.  You are choosing the type of line/pattern/texture that you want to use. Metaphors, or words applied to objects, actions, or concepts to which they are not literally applicable, are extremely efficient means of communicating complex ideas.  

“For instance, if I say “This is a Trash Bin,” you may not know a computer’s file management system or directory structures, but you’ve got a pretty good idea of how trash bins work, so you can deduce that the unwanted files go in the trash bin, and you’ll be able to retrieve them until the bin is emptied. Metaphors are assistive devices for understanding.”
— Frank Chimero

VR/AR offer tremendous possibility in creating entirely new experiences and environments.  The primary obstacle that designers and developers will encounter is that they can only design for the speed of the user’s understanding.

When virtual objects and actions in an app are metaphors for familiar experiences - whether these experiences are rooted in the real world or the digital world - users quickly grasp how to use the app.” - iOS Human Interface Guidelines, 2015

Learning on Jet Fuel

Metaphors that "embody," or perfectly represent, an idea will communicate those ideas faster.  Same for qualities, feelings, etc.  Furthermore, exposure to metaphor can be incidental!  It doesn’t have to be something that users are conscious of in order to influence them. Researchers from MIT, Harvard and Yale tested how mere exposure to smoothness or coarseness affected people’s subsequent decision-making.

One half of the participants completed a jigsaw puzzle with a smooth surface.  The other half completed the same jigsaw puzzle except that was covered in sandpaper.  After participants finished the jigsaw puzzle they were given a second task to read and evaluate a story.  They were told that this was an unrelated task.  However, the researchers were really studying how the haptics of touching a smooth or a rough puzzle would change people’s judgments of the story.  In American culture, roughness is associated with coarseness, difficulty and harshness.

Here’s how the researchers described the rest of their study:

“After the puzzle task, participants read a scenario describing an interaction between two people and formed impressions about the nature of this interaction. This passage described both positive components (e.g., kidding around) and negative components (e.g., exchange of sharp words) of a social interaction and thus was ambiguous as to the overall tenor of the interaction…After reading, participants rated whether the social interaction was: adversarial/friendly, competitive/cooperative, a discussion/argument, and whether the target people were on the same side/on opposite sides using 1-9 scales.

Results indicated that participants who completed the rough puzzle rated the interaction as less coordinated (more difficult and harsh) than did participants who completed the smooth puzzle, F(1, 62) = 5.15, P = 0.027. Thus, roughness specifically changed evaluations of social coordination, consistent with a 'rough' metaphor.”

96 
  
    
  
 Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 X-NONE 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:Calibri;
	mso-ascii-font-family:Calibri;
	mso-ascii-theme-font:minor-latin;
	mso-hansi-font-family:Calibri;
	mso-hansi-theme-font:minor-latin;}
 
   Experience of texture influences unrelated task of judging social interaction.  A rough texture led to more competition and less coordination.       Half of the participants touched the rough textured puzzle and the other half touched the smooth textured puzzle.  Next they read a story about an ambiguous social interaction.  The people who touched the rough texture believed that the interaction was more difficult and harsh.  

Experience of texture influences unrelated task of judging social interaction.  A rough texture led to more competition and less coordination.   Half of the participants touched the rough textured puzzle and the other half touched the smooth textured puzzle.  Next they read a story about an ambiguous social interaction.  The people who touched the rough texture believed that the interaction was more difficult and harsh.  

So why would touching a smooth or a rough puzzle piece change an American’s subsequent evaluations of a social experience?  Likely because of how it essential it is to the following metaphors:

  • Having a rough day
  • Using coarse language
  • Being rough around the edges
  • Acting as a smooth operator

This study is an example of how sensory input (touching rough or smooth puzzle pieces) affected people’s subsequent decision-making. The puzzle pieces acted as embodiments (e.g., tangible representations) of social dynamics. The lesson for people making VR/AR experiences is that people are making automatic associations with everything that they see, touch, and hear. Users are in a constant state of monitoring their environment and taking in new information.  Metaphors are a powerful cognitive shortcut to help users learn what your world is and how to navigate it. 

Metaphors can help you communicate abstract information quickly.  Consider which metaphors capture the experience that you want to create and how your want your user to feel.  Then work backward from there to how to represent it.  For example, if I wanted to create a narrative about love, the metaphor “Love is a Journey” would be extremely useful.  Consider the following examples used to describe love (and its challenges):

  • Look how far we’ve come.
  • We have a long way to go.
  • It’s been a long, bumpy road.
  • We can’t turn back now.
  • We’re at a crossroads.
  • We may have to go our separate ways.
  • This relationship isn’t going anywhere.
  • We’re spinning our wheels.
  • Our relationship is on the rocks.
  • This relationship has hit a dead-end street.

This list of metaphors likely triggered ideas for how to represent love inside of a digital experience.  Hopefully, it also provided an example of taking an abstract concept and representing it in a way that users can quickly grasp.  For additional advice on metaphor, I recommend Maggie Appleton's article, Why Metaphors Matter for App Designers. It's not specific to VR/AR, but it contains some nuggets.  

Further reading

Ackerman, J. M., Nocera, C. C., & Bargh, J. A. (2010). Incidental haptic sensations influence social judgments and decisions. Science,328(5986), 1712-1715.

Chimero, Frank.  What Screens Want.  Build conference Belfast.  Nov 14, 2013.  

 iOS Human Interface Guidelines, 2015

Inuit "Snow" is the Future of VR

In 1978, Roz Chast published her first cartoon in The New Yorker, Little Things.  It featured imaginary widgets with their nonsense labels.  Chast’s debut is a satire of the arbitrary, nonsensical nature of the words we assign to novelty.

In the world of VR, people are creating radically new experiences and have the opportunity to name and label, which have consequences on cognition.  What starts out as unfamiliar will become familiar.  Newly coined terms have become essential tools for thriving in new contexts. Consider the use of “mouse”, “dongle”, “spam”, “ping”, “meme”, “hashtag”, “lulz” online.  

Smart labeling is one of the most important (and underestimated!) aspects of designing a successful VR/AR experience.  I would argue that if you’re designing VR/AR experiences, failure to effectively label could imperil your whole project.

Access to language and labels changes cognitive processing. In VR there is the opportunity to create whole worlds with novel objects and new labels.  Designers should be mindful of the cognitive effort that this creates for people inside of their experience.  By giving people access to language and labels, they can go through an experience more quickly and easily.  Consider two examples from the world of VR / AR.

The Microsoft Hololens teaches people the "Bloom" hand gesture. Bloom is a special system gesture that is used to go back to the Start Menu.  It's a common word that most English speakers know and it makes it easy to remember as a navigation tool.  In contrast, Aperture Robot Repair (an experience made by Valve for the HTC Vive) gives very generic instructions for people to place their controllers in a certain area to charge them, but it can take some users (like me!) a long time to actually figure out what I'm actually supposed to do to get to the next part of the experience.  

 

MS Hololens: To do the bloom gesture, hold out your hand, palm up, with your fingertips together. Then open your hand.

MS Hololens: To do the bloom gesture, hold out your hand, palm up, with your fingertips together. Then open your hand.

Aperture Robot Repair gives you vague instructions: "Charge your multi-tools at the charging station."

Aperture Robot Repair gives you vague instructions: "Charge your multi-tools at the charging station."

It's not that Hololens is good or Aperture Robot Repair is bad.  It's just a different experience for people.  When designing for new mediums, consider how much information and guidance users should get inside of their experience.  Perhaps developing novel objects such as the “Chent” or the “Spak” is the right choice for your VR experience, but it will slow down your users and cause more effort, especially if you only give them one path to learning it (vision) instead of multiple pathways in the brain (language and vision). Let’s consider what is the right amount of information to help people learn and track information inside of a digital experience.  

ACCESS TO LABELS SPEEDS PROCESSING

Language speeds cognitive processing and reaction times.  That means that if you want to introduce new objects, make access to language and labels easy. I’m saying “access” to labels because designers don’t have to specifically label a red, round fruit with the word “apple.”  However, they can use objects that are easy for people to label with their own mental resources.  The following excerpts are from Drunk Tank Pink:

"The notion of that labels change how we see the world predates the blue-matching experiment by almost eighty years.  In the 1930s, Benjamin Whorf argued that words shape how we see objects, people, and places.  According to one apocryphal tale, the Inuit people of the Arctic discern dozens of types of snow because they have a different words for each type.  In contrast, the rest of the world has perhaps several words - like snow, slush, sleet, and ice.  The story isn’t true (the Inuit describe snow with roughly the same number of words as [non-Inuit] do), but it paints a compelling picture: it’s much harder to convey what’s in front of you if you don’t have words to describe it.  Young children illustrate this difficulty vividly as they acquire vocabulary - once they learn to call one four-legged creature with a tail a “dog,” every four-legged creative with a tail is a dog.  Until they learn otherwise, cats and ponies share the same features, so they seem just as doggish as real dogs.”

There was a clever experiment that tested this phenomenon.  Due to linguistic differences between English and Russian, cognitive scientists were able to parse how the ability to label a color with specificity affected people’s reaction time. 

“Colors and their labels are inextricably linked.  Without labels, we’re unable to categorize colors - to distinguish between ivory, beige, wheat, and eggshell and to recognize that broccoli heads and stalks are both green despite differing in tone. To show the importance of color labels, in the mid-2000s, a team of psychologists capitalized on a difference between color terms in the English and Russian languages.  In English, we use the word blue to describe both dark and light blues, encompassing shades from pale sky blue to deep navy blue.  In contrast, Russians use two different words goluboy (lighter blue) and siniy (darker blue).  

The researchers asked English-speaking and Russian-speaking students to decide which of the two blue squares matched a third blue target square on a computer screen.  The students performed the same task many times.  Sometimes both the squares were light blue and sometimes both were dark blue, and sometimes one of them was light blue and the other was dark blue.  When both fell on the same side of the blue spectrum - either light or dark blue - the English and Russian students were equally quick to determine which of the squares matched the colors of the third target square.  But the request was quite difference when one of the colored was lighter blue (or goluboy according to the Russian students) and the other was siniy (darker blue).  On those trials, the Russian students were much quicker to decide which square matches the color of the target square."

While the English students probably looked at the target blue square and decided that it was “sort of lightish blue” or “sort of darkish blue” their labels were never more precise than that.  They were forced to decide which of the other blue squares matched that vague description.  The Russian students were at a distinct advantage, they looked at the square and decided that it was either goluboy or siniy.  Then all they had to do was look at the other squares and decide which one shared the label.  Imagine how much easier the task would have been for the English students if they had been looking at one blue square and one green square; as soon as they determined whether the target square was blue or green, the task was trivially easy.  In fact, an experiment published one year later showed that Russian students perceive dark blue to be just as different from light blue as the color green is from the color blue to English students.  When Russian student located a dark blue square wishing an array of light blue squares, part of the visual field within their brain light up to a signal that they had perceived the odd square.

The same brain areas were much less active when English students look at the same array of squares - except when the odd square was green within an array of blue squares. When the colors had different labels for the English students, their brain responded like the brains of the Russian students. 

 

In comparison with hard-to-name colors, perceptual discrimination of easy-to-name colors elicited unique activation in the posterior portion of the left superior temporal gyrus, left inferior parietal lobule, left precuneus, and left postcentral gyrus were statistically stronger for easy-to-name colors. No regions showed stronger activity for the discrimination of the hard-to-name colors.

In comparison with hard-to-name colors, perceptual discrimination of easy-to-name colors elicited unique activation in the posterior portion of the left superior temporal gyrus, left inferior parietal lobule, left precuneus, and left postcentral gyrus were statistically stronger for easy-to-name colors. No regions showed stronger activity for the discrimination of the hard-to-name colors.

We also know that the Russian students relied on these category names, because their advantage of the English students disappeared altogether when they were asked to remember a string of numbers while they were performing the color discrimination task.  Since their resources for processing language were already occupied with the task of repeating the number string, they weren’t able to rehearse the names of the colors. Without the aid of linguistic labels, they were forced to process the colors just like the English-speaking students. This elegant experiment shows that color labels show how people see the world of color. The Russian and English students and the same mental architecture - the same ability to perceive and process the colors in front of them - but the Russians had the distinct advantage of two labels where the English students had just one.  This example is striking because it shows that even our perception of basic properties of the world, like color, is malleable in the hands of labels.  

Interestingly, the researchers didn’t have to actually label the squares with words in order for people to activate the language centers of their brain.  And when they put people under cognitive load by asking them to remember a string of numbers, the Russian-speakers cold not access the linguistic labels and their performance decreased to the same baseline of the English speakers.   

Failure to use language and labels in an effective way can sabotage an experience in VR/AR. Try working backwards from the experience that you want your user to have and consider what their level of knowledge will be when they arrive to your experience. 

People use language as part of perception.  Language affects patterns of brain activation.  In my next post, I'm going to discuss language metaphors because they are one of the most important tools of knowledge acquisition that humans possess! All VR/AR experience designers should command metaphors to immerse people in an experience.  

 

Further Reading

Roz Chast has published over 1200 cartoons in the New Yorker since 1978.

Alter, A. (2013). Drunk tank pink: And other unexpected forces that shape how we think, feel, and behave. Penguin. Pages 27-29.  

Winawer, J., Witthoft, N., Frank, M. C., Wu, L., Wade, A. R., & Boroditsky, L. (2007). Russian blues reveal effects of language on color discrimination. Proceedings of the National Academy of Sciences, 104(19), 7780-7785.

Tan, L. H., Chan, A. H., Kay, P., Khong, P. L., Yip, L. K., & Luke, K. K. (2008). Language affects patterns of brain activation associated with perceptual decision. Proceedings of the National Academy of Sciences, 105(10), 4004-4009.

Unlocking the Potential of the Extended Mind

96 
  
    
  
 Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 X-NONE 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:Calibri;
	mso-ascii-font-family:Calibri;
	mso-ascii-theme-font:minor-latin;
	mso-hansi-font-family:Calibri;
	mso-hansi-theme-font:minor-latin;}
 
       In "Limitless," Bradley Cooper plays a struggling writer who takes a pill to expand his intelligence.  What if VR experiences can make people smarter just by giving new physical forms to thought?

In "Limitless," Bradley Cooper plays a struggling writer who takes a pill to expand his intelligence.  What if VR experiences can make people smarter just by giving new physical forms to thought?

This blog is called the Extended Mind because it’s about how people use their bodies or objects in the environment as part of cognition.  Now, suppose that rather than just studying the existing ways that people are using their bodies, phones, etc to think you could imagine new ways to interact with information and how those interactions would unlock new thoughts.

That’s the topic of Bret Victor’s talk “The Humane Representation of Thought” that he delivered at the closing keynote at the 2014 Splash and UIST conferences.  (H/T to Leonard Lin for telling me about this talk.)  Victor is a UI designer who formerly worked on tech for the iPhone, MacBook, and iWatch. 

Victor claims that his talk is not about Virtual Reality, but his insights are applicable to anyone doing experience design and grappling with new ways to represent information. Below are the excerpts of Victor’s talk that I found most relevant to VR / AR designers.  I would recommend watching the entire talk, but here are the highlights from the Extended Mind perspective. 

Victor begins with some history and ways that human cognition expanded once new ways of representing data were invented, such as graphs. 

“Representations, by which I mean the ways we externalize thought, have been responsible over the last 2,000 years in large part for the intellectual progress of humanity, enabling us to think thoughts that we couldn’t think before. But while these media that we’ve invented have empowered us in certain ways, they cripple us in other ways and we now have an opportunity to design a new medium of thought that undoes some of that damage in a way that’s both humane and empowering."

Victor doesn’t talk much about the mechanics of cognition.  However, it’s worth calling out that the working memories of humans are very limited. By reducing the amount of stress on a person’s working memory by giving them new ways to externalize thought, you will expand people’s capacity for thought. 

Victor later quotes from James Gleick’s book Genius: The Life and Science of Richard Feynman to support the notion that abstract thought is aided by having a physical intuition.  

“[Physicists] found they needed imagery…a style of thinking based on a kind of seeing and feeling.  That was what physical intuition meant.

Feynman said to Dyson, and Dyson agreed, that Einstein’s great work had sprung from physical intuition and that when Einstein stopped creating it was because ‘he stopped thinking in concrete physical images and became a manipulator of equations.’ Intuition was not just visual but also auditory and kinesthetic.  Those who watched Feynman in moments of intense concentration came away with a strong, even disturbing sense of the physicality of the process, as though his brain did not stop with the grey matter, but extended through every muscle in his body.”


Then Victor goes on to review the various models of representing thought from various academic disciples.

“The point of all of this is that the space of how we understand things, the space of our cognitive capabilities is so vast and so diverse that even people who have devoted their entire careers to coming up with a neat little theory come up with different answers.  There’s so much there.  It’s such a rich space.

Every circle up there is a superpower.  Every circle is a capability that we’ve been honing for hundreds of thousands of years that we use in innumerable ways and can combine with the other ones in really powerful ways. And what happened with the invention of the printing press and tiny rectangle based knowledge work is this”


 

Formulas...yuck!  Yet, this is what intellectual/analytical work looks like in academic circles.

Formulas...yuck!  Yet, this is what intellectual/analytical work looks like in academic circles.

Then Victor gets into the real substance of his talk.  I would recommend watching from minute 20-30 to hear the gist of his talk on importance of expanding experience design to include all of human senses and capabilities.  Here is a transcript of the key ideas: 

“And all this stuff drops out and we’re working with visual symbols.  We’re reading with visual symbols, we’re manipulating visual symbols.  That’s what it means to do intellectual work nowadays.  And you might think ‘Well, maybe that was true with paper and books, but with computers, it’s getting better, right?’  And no, with computers it is getting worse.  With a book, a book is at least a physical object which exists in the world with some amount of tactile response.  You can hold it and move it around.  With a book you can make a shelf you can make a shelf, which is a spatial representation of knowledge.  It’s is not very good but it is something you can understand spatially.  When you’re writing with ink on paper, you can move freely from drawing imagery and writing in language because the paper doesn’t really care what type of marks you make on it and you can move between those two modes.  But then we invent these things [imac / ipad], these flat, glassy screens that have no tactile response so [tactile] drops out and they are little tiny screens that take a portion of your view of view and special stuff drops out.  And they have these keyboards.  The only convenient thing to do is punch in symbols especially for us anything from writing an email to writing a computer program, anything other than typing in letters is incredibly cumbersome.  Whatever you are trying to express, you express it in symbols because that’s what the interface encourages you to do.  So [iconic] drops out.  And so this is the cage that we have trapped ourselves in.  This is the way that we have constrained our range of experience, which we have created a tiny subset of our intellectual abilities and have forbidden ourselves to use our full intellect.

There’s two things wrong with this.  One is that it’s Inhumane…using media that restrict our thinking to this tiny subset is inhumane because do all our thinking things.  We can’t think in all the ways that humans can think so there’s a kind of moral argument there.  If you don’t buy that, there’s also a practical argument, which is that it’s just wasteful.  One way to think about it is as programmers to imagine that you have an eight core processor and you’re writing some code.  It’s kind of a sequential program.  Can’t parallelize it so you’re maxing out of the cores and the other seven cores are just idling and that hurts.  As a programmer you get a feeling in your stomach, 'God this is so inefficient…if only I could parallelize this algorithm, there’s so much latent power I can draw on' and that’s exactly the emotion that I get when I’m looking at this tiny, rectangle based knowledge work that we do.  There’s all these cores.  We have all these capabilities and they are just sitting there idle.  If only we could parallelize our representations across all of these capabilities, who knows what we would be capable of thinking."


In the last section of the talk, Victor offers an optimistic take on how designers can break out of the design paradigm of the rectangular screen.  

"So the good news is that we now have an opportunity to do something about this.  So we are now, I believe, at a unique moment in history where we are inventing the next medium of thought after the printing press.  We are inventing the dynamic medium.  So in the same way that there are certain thoughts that can be conveyed in print or in theater, I believe there’s a wide range of thoughts that can be conveyed in programs once we understand how to do that."

In an ideal world, designers will make experiences and represent information in an dynamic medium.  This dynamic medium will be responsive to the user’s capabilities and connect multiple modes of understanding at once. 

Main takeaways for experience designers:

1.     Cognition is dependent on representations of data.  At the most basic level, if your working memory is clogged up with too much data, your internal processing power decreases. 
2.     You have multiple sensory superpowers to help you think.  However, the current dominance of rectangular screens primarily relies on vision and shuts out the others. 
3.     VR & AR designers have an open canvas for representing thought and they should actively look for avenues to enhance connections among human modes of understandings. 
 

The link to the full talk is here:


 

Eschew Easy. Effort Makes it Better.

effort.jpeg

Designing for VR is unique compared to other digital experiences because you have the opportunity to demand how people move through space. The rigging of VR devices provide hundreds of opportunities to ask users to stand or move in a particular way.  It's important to consider exactly which actions that designers should require from their users and what's the right level of difficulty.  If the movements are too hard, people may abandon the experience. If your VR experience is too easy, people will get bored. But, if it requires just the right amount of effort, it will be more immersive and the experience will be processed at a much deeper cognitive level, which has surprising implications for what people remember and value about it.

This is my second post on gauging the right level of effort for users going through a digital experience. In my first post on why it can be good to require high effort, I reviewed a scientific study where people wore a gel mask and then took an empathy test. People who wore the gel mask on their face outperformed those who just wore it on their arm. This is because part of how people can understand the feelings of others is by mimicking them. The face-gel participants had to work harder to mimic the expressions of others, and the result was higher empathy. 

Physical or mental effort changes our cognitive processes and subsequent experiences.  Digital designers must be mindful of what demands they are putting on their users and how those demands affect emotions and decision-making.  Effort should be considered as part of every use case. Once upon a time, I did UX for an e-commerce site and minimizing effort was a priority.  The entire UX team wanted to eliminate friction and make it as easy as possible for people to find a product they wanted and then input credit card info. Sometimes it felt like my entire job was getting digital experiences in front of users and then reporting back to the design team on what was easy, what was hard, and how to make the e-commerce flow easier.  

However, what's desirable for e-commerce isn't the same for every experience. In the context of games, effort is a key element. If it’s not challenging, people will get bored.  Jesse Schell, author of The Art of Game Design: A Book of Lenses, discusses effort in the context of game design and here are his key questions on presenting challenges to your users:

1. Are your challenges too easy, too hard, or just right?

2. Can your challenges accommodate a wide variety of skill levels?

3. How does the level of challenge increase as the player succeeds?

4. Is there enough variety in the challenges? 

Jesse’s work is great because he makes the guideposts of effort so explicit. Also, it underscores the importance of testing and validating experiences before launching them.  Since the game designers have insider knowledge and have multiple rounds of practice, they lose touch with what is effortful and what’s not. 

Niko Christenfeld, a psychologist at UC San Diego, analyzed sports like baseball, basketball, and American football and suggests that these sports survived to the present day because of natural selection.  They offered fans and players the right combination of skill and chance in predicting the outcome.  “Contests with too much chance are pointless as measures of relative ability.  Those with too little chance in the mix provide no suspense.  The superior team should probably, but not certainly, win.”  For a sport like baseball, it’s unclear that it was ever a calculation that the baseball season must be 162 games long in order for the best teams to emerge into the playoffs. Rather, the length of the baseball season is driven by giving all teams the right balance between demonstrating their skills and luck. 

Designing for the right level of effort is a complicated thing. While some designers, such as game makers, have either a conscious or unconscious frame of mind about calibrating the right level of effort, many other designers eschew challenges. For the latter group of designers, I want to highlight a handful interesting studies on how more effort translated into a better experience. 

 1. Difficult cognitive experience: People who read fonts in cursive (vs. print) had better reading comprehension. 

2. Easy cognitive experience: As people become more expert, they value their own expertise less.  This means that if a task is easy, people interpret the task being easy as the task being less valuable.

3.  Difficult physical experience #1: People who were only allowed to write with one hand turned in higher quality essays.  By slowing people down from their normal typing speed, the study participants constructed better sentences and used to time to choose better words. This resulted in higher quality essays compared to a control group that typed with two hands.  Quoting one of the study researchers, Srdan Medimorec, “It seems that what we write is a product of the interactions between our thoughts and the tools we use to express them.”

4. Difficult physical experience #2: Students who assembled their own IKEA furniture liked it more.  In the words of the researchers, “Labor Leads to Love.”

These academic studies show the various ways that there’s more to evaluations and decision-making than just what your actual experience is.  Perhaps this company BlueSpec has taken its cue from the research on effort.  See their advertisement here (and h/t Bret Victor for originally posting this):

blue spec.jpg

The brilliance of this marketing is that they have equated the value of solving your challenges with the value of your learning something difficult.  Of course, fixing your problems is difficult so why would you expect the solution be easy?

Here are the main takeaways on how effort changes experience.   

1. More effort often means that the experience is internalized and processed at a deeper level. 

2. If something is really easy, it can be valued as less important

3. Consider setting people’s expectations that what they are about to experience may be challenging, and that is a good thing. 

Test and iterate to find the sweet spot of challenging users and giving them a free pass through your experience.  I hope this post has persuaded you that minimizing all effort inside of the experience is the wrong goal.  Aim to right-size the effort instead.  

 

Further reading

Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General136(4), 569.

Christenfeld, N. (1996). What makes a good sport.Nature383(6602), 662-662.

Norton, M. I., Mochon, D., & Ariely, D. (2011). The'IKEA effect': When labor leads to love.Harvard Business School Marketing Unit Working Paper, (11-091).

People Who Write Well Do This One Simple Thing, Psych Study Finds

Schell, J. (2014). The Art of Game Design: A book of lenses. CRC Press. 

Design to Score an "A" for Effort

How much effort should you require from users when they engage in your experience?  A lot. In some surprising ways, challenge is good for user engagement.  The effort required from users doesn’t just affect their experience - it changes their cognitive processing.  

Usability experts strive to make experiences easier. However, I want to share an example of how making things harder may improve outcomes. 

Exerting more physical effort improved performance on cognitive tasks

Earlier I wrote about how people with Botox have a difficult time identifying the emotions of the people around them.  This is because they cannot mimic the expressions of others since Botox paralyzes their facial muscles - their brains lack feedback that comes from mimicry.  The consequence is that the Botox-ed participants were less empathetic compared to control groups. The same researchers who did the Botox study conducted a follow up where half the participants applied a gel mask on their face and half wore it on their arm.  Then, they all went through the same “Reading the Mind in the Eyes Test (RMET - more details here), which measures a person’s grasp of the emotional states of others.  

The gel used in this study was an over-the-counter facial mask. 

The gel used in this study was an over-the-counter facial mask. 

The participants who had the gel mask on their face outperformed the ones with it on their arms by a statistically significant amount on the RMET empathy task.  The face mask people had a 77.2% accuracy compared to 72.5% for the arm mask participants serving as a control. The researchers inferred that by forcing people to exert more effort to mimic others, this exaggeration led to afferent nerves (the same one that Botox had paralyzed in the first study) being amplified, and firing stronger signals to the Central Nervous System.  

This study provides further evidence that people are thinking with their bodies and not just their brains.  The “amplified emotion” face mask group performed better than the arm-mask control group at identifying emotion because they had to work harder to move their facial muscles to mimic the faces they viewed in RMET. 

Higher effort can deepen engagement

The lesson is that when people exert more effort to perform a task, they tend to process the information on a deeper level, which increases their engagement. There is additional research on the benefits of requiring more effort from users to improve outcomes. Different researchers discovered that people who read in a difficult, cursive-y font do better on subsequent memory and analytical reasoning tasks compared to those who read in a normal print font. 

For anyone making a VR experience, consider how you are calibrating the level of effort required from your users.  Are there physical tasks you can build in that would help amplify emotions and deepen engagement?  

 

Further Reading

Neal, D. T., & Chartrand, T. L. (2011). Embodied emotion perception amplifying and dampening facial feedback modulates emotion perception accuracy. Social Psychological and Personality Science2(6), 673-678.

Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136(4), 569.

 

Stop Forcing Users to Text and Drive

Here's a good read with practical tips for VR designers. Adrienne Hunter of Tomorrow Today Labs just posted "Reducing Cognitive Load in VR: 6 Ways to Improve your VR UX."  

Cognitive load has to do with how much information you are trying to hold inside of your working memory at any given time.  Information is filtered in working memory as a transient step before being encoded in short-term or long-term memory (or forgotten).  

We're really just talking about attention.  People have a finite capacity to pay attention, which is why it's difficult to manage two complicated tasks at once, such as texting and driving.  The more attention that you are paying, the more that your working memory is being used.  That makes it harder to learn or to solve puzzles.  But, it also has substantial effects on decision-making.  Here's a few ways that people made decisions differently when they were under high cognitive load and had to divide their attention:

Conclusion

Only ask your users to do one thing at a time. Setting them up to do a sequence a tasks is preferred to having them do anything simultaneously.  

Why Botox Makes VR Experiences Worse

Previously I showed an example of action-perception couplings. In another post about power posing, I talked about how there’s a powerful connection between your brain and body, which means that your body language shapes thought.  Now I’m going to give an example of what happens when the feedback loop between body and brain is disrupted.  

Facial Feedback Helps People Perceive Emotion

When you are talking to someone and they make a facial expression, you tend to mimic that. And there are social reasons for that—you want to show empathy, understanding —but it turns out that there are perceptual reasons too. You are better at judging their emotions if you are mimicking them. If you prevent this mimicry (via Botox) you are objectively worse at judging emotions that other people are making.

A person’s beyond-the-brain body plays an important role in their cognitive processing and knowledge acquisition. The sub-field of psychology called “Embodied Cognition” provides a framework for collecting “You think of with your body and your brain” research findings together. One of the cleverest studies on embodied cognition measured how the social intelligence of people decreased after they had Botox injections. This research has important implications for people who are building experiences or creating narratives in VR because it shows the limitations of cognitive processing alone. Designers should be considering how to harness the power of the body when they are creating experiences, particularly for head mounted displays (HMD) that could affect how people experience their facial muscles. 

 How Facial Feedback Changes Cognition

There are basically three steps to understanding how the process of embodied cognition, specifically in how facial feedback improves perception of emotions in others. First, mimicry. You are automatically perceiving and mimicking facial expressions in those around you. Next, this copying means that your muscles are contracting, which means that there is muscle feedback being conducted from face to brain. Lastly, this feedback helps you experience and understand the emotional meaning behind the expressions that you see. The Botox study here is unique because it’s one of the first ones to demonstrate that people can use feedback from their own muscles in order to more accurately understand the emotions of the people around them. 

This research is unique because it isolated the exact role of facial feedback in study participants’ effort to interpret the expressions of others.  The way that the researchers measured accuracy in emotions is via a social intelligence task called Reading the Mind in the Eyes Test or RMET.  Adults are shown a photograph of a pair of eyes and then answer a multiple choice test on which emotion the person in the photograph is feeling. Here are two examples:

An example photograph from the RMET.  What emotion is this male displaying?  Choose one of the following: Serious (correct), Ashamed, Alarmed, Bewildered.

An example photograph from the RMET.  What emotion is this male displaying?  Choose one of the following: Serious (correct), Ashamed, Alarmed, Bewildered.

Another example from the RMET.  What emotion is this female displaying?  Choose one of the following: Reflective (correct), Aghast, Irritated, Impatient.

Another example from the RMET.  What emotion is this female displaying?  Choose one of the following: Reflective (correct), Aghast, Irritated, Impatient.

When a participant takes the RMET, they view 36 pairs of eyes like the ones above and have access to a glossary to read any definitions of emotions that they wanted.  Adults recruited from the general population accurately reported the emotion 72.7% of the time.  The researchers compared them to a Cambridge student population who performed better, correct 77.8% of the time, but there was no statistically significant difference between the two groups. 

The RMET was developed by psychologists at Cambridge and their 2001 paper only documents how well adults do on this sort of social intelligence test.  The Botox study utilized RMET in order to study the body-brain feedback loop, and specifically what happens to performance when that feedback is distorted.  David Neal of USC and Tanya Chartrand of Duke recruited one group of participants who got Botox and another group who got Restylane (another cosmetic facial filler) within the previous 1-2 week window. While the participants were not randomly assigned to receive Botox or Restylane, they were well matched; they did not differ in age, ethnicity, or social economic status.  The big difference between the two cosmetic fillers is that Botox freezes the facial muscles in place and Restylane does not.  Botox paralyzes expressive muscles by blocking the release of acetylcholine, a neurotransmitter, at the neuromuscular junction.  This results in a paralysis in the facial muscles, meaning that there is less feedback conducted from the Botox-ed muscles to the brain.  In contrast, Restylane is simply a dermal filler and has no impact on muscle or neurotransmitter function.  It is interesting to study these two groups because if facial feedback from a person’s body to their brain actually helps them mimic and subsequently read the emotions of the people around them, then the Botox group would perform worse. 

That’s exactly what happened. The Botox participants performed poorly compared to the Restylane group.  People with Botox accurately stated the emotion of the people in the photographs 69.9% of the time.  The Restylane group were correct with 76.9% of the faces, which was a statistically significantly difference from the Botox group.  When compared to the groups of adults from the original development of the RMET, the Restalyne group performed in the same range as those controls.  

The reduction of muscular feedback due to Botox meant that the participants had a worsened ability to perceive the emotions of others, wheras the Restalyne group was unaffected.  The implications for VR designers is that participants have different emotional capabilities for reasons outside of your control.  That could lead to less immersive storytelling.  This is something to consider for all VR designers - not just experiences made for Botox injectors.  People with any type of disruption of the conduit between facial muscles and brain may come away from your experience with a different feeling than those who don’t have that disruption.  It’s also an open question - does the mere presence of HMD affect the facial feedback loop? 

In my next post, I’m going to talk about a different study conducted by the Botox researchers.  In it, they put a gel on people’s faces that made it difficult for people to move their muscles and gave them the same social intelligence task.

 

Further Reading

Baron‐Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the Mind in the Eyes” test revised version: A study with normal adults, and adults with Asperger syndrome or highfunctioning autism. Journal of child psychology and psychiatry, 42(2), 241-251.

Neal, D. T., & Chartrand, T. L. (2011). Embodied emotion perception amplifying and dampening facial feedback modulates emotion perception accuracy. Social Psychological and Personality Science, 2(6), 673-678.

Paul, Pamela.  “With Botox, Looking Good, Feeling Less.” The New York Times.  June 17, 2011.  

The Science Behind the Most Watched TED Talk

Why do people standing like Wonder Woman feel more powerful than people sitting with their legs crossed and arms folded?  Why do changes in the physical body systematically influence feelings and actions?

Amy Cuddy, a social psychologist who was part of the team that conducted the “Wonder Woman” study, gave a TED Talk entitled “You Body Language Shapes Who You Are.”  It has nearly 38 million views, making it the most highly viewed TED talk since 2012.  It showcases how cognitive processes are deeply rooted in the body’s movements.  But why does it matter to VR designers?

If a user makes any action, even as minor as a head nod, it can activate greater feelings of agreeableness compared to shaking the head. VR designers are creating entire environments that engage the body. While I doubt that any adult walking around in the world today is surprised that nodding leads to agreeableness, the point is that VR designers should be conscious of these and other body-brain links in order to build the easiest and most immersive experiences. 

Why are body-brain interactions frequently overlooked? 

I suspect these body-brain interactions are frequently overlooked because they are happening fluidly and non-consciously.  These “action-perception couplings” usually happen automatically. 

The relationship between action and perception is in continuous flux from the world, through the sensory system, into your body and back out into the world again in the form of actions.  This is the way that humans work and it’s so obvious that it’s easily overlook. 

Here’s another example of pulling vs. pushing an object:

People who pull something towards them (think of activating your bicep) consider the object with more positivity, excitement, and acceptance.  They value it more and express a higher willingness to pay for the object. 

People who push something away from them (think of extending your tricep) consider the exact same object with more negativity, disinterest, and rejection.  They value it less and express a lower willingness to pay for the object. 

Professor Amy Cuddy demonstrating an expansive posture during a talk.

Professor Amy Cuddy demonstrating an expansive posture during a talk.