VR Designers, Don’t F*^k This Up

Our human desire for meaning is the future of VR.

Coraline caught between worlds.

Coraline caught between worlds.

Ever wonder what a historian would say about VR?  Yuval Noah Harari, a historian at Hebrew University of Jerusalem and the author of Sapiens: A Brief History of Humankind and Homo Deus: A Brief History of Tomorrow, speculates on VR:

"If you don't have a job anymore and the government provides you with universal basic income, the big problem is how do you find meaning in life. What do you do all day? And here the best answer so far we've got is drugs and computer games. People will regulate their moods with all kinds of bio-chemicals and they will engage with 3-dimensional virtual realities that they’ll be absorbed in [sic], and these things will provide emotional engagement and interest more than anything in the outside world. This idea of humans finding meaning in virtual reality games is actually not a new idea. It's a very old idea. We have been finding meaning in virtual reality games for thousands of years, we've just called it religion until now."
 

"This idea of humans finding meaning in virtual reality games is actually not a new idea. It's a very old idea."


"You can think about religion simply as a virtual reality game. You invent rules that don't really exist.  It's just in your mind.  But you believe these rules and for your entire life, you try to follow the rules.  If you're Christian then if you do this, you get points.  If you sin, you lose points.  If, by the time you finish the game (when you are dead), you gain enough points you get up to the next level.  You go to heaven.  People have been playing this virtual reality game for thousands of years.  It made them relatively content, happy with their lives.  In the 21st century, we'll just have the technology to create far more persuasive virtual reality games than the ones we've been playing for the last thousands of years.  We'll have the technology to actually create heavens and hells, not in our minds, but using bits and using direct brain-computer interfaces."  


Whatever your religious beliefs, Harari’s perspective that VR will become a primary source of meaning in our lives is dystopian. Considering that VR will offer higher emotional engagement and interest than outside of the headset, it’s clear that people can easily fall into narratives that they don’t have control over or even choice in. That is what Harari is calling the ‘heavens and hells’ of our own creation. 


Digital Addiction

If people have a behavioral pattern to find meaning inside of VR, will they spend more or less time in VR?  Obviously, more time.

Adam Alter, a psychologist at NYU, published a new book about digital addictions called Irresistible.  You can buy it here.

Alter defines addiction as “something you enjoy doing in the short term, that undermines your well-being in the long term — but that you do compulsively anyway.”  He goes on to state that “You never have to remember anything because everything is right in front of you. You don’t have to develop the ability to memorize or to come up with new ideas.”

Now consider the combined implications of Harari and Alter’s research:

  • VR has the potential to replace religion—defining religion as the stories we tell ourselves to find meaning in life.
  • VR experiences will be addictive. 
  • It’s unlikely that people will come up with new ideas and stories for themselves.

Takeaways for VR designers

  • People will become addicted to the heavens and hells that you create for them. 
  • Don't fuck this up. 


Additional reading

H/t to Austin Ramsland for directing me to the Vox interview with Yuval Noah Harari.  Link to the original podcast is here.

Interview with Adam Alter in the NY Times.

Adam Alter.  Irresistible: The Rise of Addictive Technology and the Business of Keeping Us Hooked.

Why You Should Science the Sh*t out of Your VR Experience

I was reading University of Illinois professor Steve LaValle’s book Virtual Reality and he makes a forceful argument for doing research in any VR experience. LaValle starts out talking about research by talking about the difference between having a first-person vs. third-person perspective.

"When a scientist designs an experiment for an organism, as shown in Figure 1.2, then the separation is clear: The laboratory subject (organism) has a first-person experience, while the scientist is a third-person observer. The scientist carefully designs the VR system as part of an experiment that will help to resolve a scientific hypothesis. For example, how does turning off a few neurons in a rat’s brain affect its navigation ability?"

"On the other hand, when engineers or developers construct a VR system or experience, they are usually targeting themselves and people like them. They feel perfectly comfortable moving back and forth between being the 'scientist' and the 'lab subject' while evaluating and refining their work…this is a bad idea!"

There are concrete benefits to testing an experience outside of your immediate team.  LaValle continues:

  • "The creators of the experience are heavily biased by their desire for it to succeed without having to redo their work. 
  • They also know what the experience is supposed to mean or accomplish, which provides a strong bias in comparison to a fresh subject. 
  • VR disrupts the ordinary perceptual processes of its users…It is hard to predict how others will react to your own writing. Also, it is usually harder to proofread your own writing in comparison to that of others. In the case of VR, these effects are much stronger and yet elusive to the point that you must force yourself to pay attention to them.

It should be clear from this section that proposed VR systems and experiences need to be evaluated on users to understand whether they are yielding the desired effect while also avoiding unwanted side effects. This amounts to applying the scientific method to make observations, formulate hypotheses, and design experiments that determine their validity.”

Takeaways for designers:

  • You have incentives to believe that first-time users will understand your experience, but testing is the only way to really find out. 
  • Develop a test plan before you bring in people to try your experience. For example, let’s say you are updating version A to Version B. Get very specific about what you predict the improvements of Version B will offer, such as:
    ⁃          reduction in adverse symptoms
    ⁃          improved comfort
    ⁃          greater efficiency at solving tasks
    ⁃          higher presence, or sense of being there
    ⁃          greater enjoyment of the activity
     

LaValle’s book is a great resource for all things VR. It’s downloadable here.

Total Recoil: The Uncanny Valley Is an Uncanny Cliff

This is part two of a series on what uncanny and the uncanny valley mean and how to accurately use them to describe experiences (in VR or otherwise).

TL;DR

  • The Uncanny valley is specifically about the relationship between being human-like and likability. 
  • Every act of perception involves an act of categorization

  • Science shows that toy non-humanoid robots (think WALL-E) are preferred to human-like ones (and actual humans!)

Background

1919 – Sigmund Freud published an essay called The Uncanny” (translated from the German unheimlich), defining it to mean weird/eerie/unfamiliar.

1970 – Valley of Eeriness (translated from the Japanese bukimi no tani) was coined by robotics professor Masahiro Mori to model affinity for androids as they become more humanlike.

1978 – Uncanny Valley first appeared in English inside a book by Jasia Reichardt called "Robots: Fact, Fiction, and Prediction.”

2007 – Uncanny Cliff was introduced to more accurately reflect the shape of the curve.   


What Is the Uncanny Valley?

The uncanny valley is a very precise feeling of weirdness or fear that nearly-humans (but not quite humans) evoke. The uncanny valley is elicited when a robot or AI has some human likeness, but isn’t human. Long time art director and VR/AR insider Spencer Lindsay uses “creepy corpse” in his definition of the uncanny valley.

You can see the valley in the graph below:

Image credit here

Image credit here

It’s assumed that liking of a robot increases the more human like that it is, until it crosses into the valley (the grey area). Lindsay’s creepy corpse would appear to be at the nadir of the valley – with zombies! 

It is likely that the uncanny valley response is evolutionarily adaptive. Studies done with rhesus monkeys show a similar pattern of likability. Monkeys will look longer at the unrealistic (left) and real (right) faces below, rather than the realistic one in the center. 

Monkey visual behavior falls into the uncanny valley

Monkey visual behavior falls into the uncanny valley

The Uncanny Valley Confuses Categories.

To categorize is a fundamental psychological process that happens automatically. The very act of perception involves an act of categorization. You haven’t completed the process of perception until you have categorized it and matched it up to other things that you know about. That is, “this object is the same as other things I know about and different from these other things.” We judge new things based on their similarity to previous things. And a human-like puppet easily confuses our judgment process. 

“It’s the process that grinds away constantly and generates much of our understanding and response to the world,” says Berkeley psychologist Dacher Keltner.  “First of all, it’s how do you categorize things? And that’s everything. Do I sleep with him or not? Is that a boy or a girl? Is that predator or prey? If you solve how this process works, then you solve how you know things. It’s how knowledge about the world is organized. It’s like the thread that is woven through everything in the mind.”

"Do I sleep with him or not? Is that a boy or a girl? Is that predator or prey? If you solve how this process works, then you solve how you know things."

Encountering a creepy android from the trough of the uncanny valley leads to conflicting perceptual cues. Is it human or not? It interferes with the automatic, System 1 processing that we rely on to get through our day.

The Uncanny Valley Is an Uncanny Cliff.

Lastly, there have been empirical studies that map the uncanny valley since Mori originally theorized it. Researchers show 11 different images as an object morphs from a thing to a human, and there is a decrease in liking midway through. 

It’s actually more like a cliff, because likability doesn’t fully recover:

Image source here

Image source here

"The uncanny valley appears to be more of a cliff than a valley since even pictures of humans do not reach the level [of likeability] of pictures of toy robots. It has to be acknowledged that there is a small upwards trend again towards highly human-like entities, which results in a small valley. However, the most dominant feature in the graph is not the valley, but the cliff preceding it."  – Bartneck et al.

Takeaways for designers:

  • Avoid anything related to the uncanny valley by populating your experience with non-humanoid avatars. People prefer non-humanoids to humanoids
  • The uncanny valley doesn’t actually exist. It’s more like a cliff.  Once you get too humanoid, liking decreases dramatically and never fully returns. 

 

Additional info

 If you are interested in the uncanny valley, you’ll like Kimberley Voll’s interview on the fidelity contract. Listen to her full interview with @VoicesofVRKentBye here.

Bartneck, C., Kanda, T., Ishiguro, H., & Hagita, N. (2007). Is the Uncanny Valley an Uncanny Cliff? Proceedings of the 16 th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN 2007, Jeju, Korea, pp. 368-373.

H/T to @spencerlindsay for letting me quote him here. 

Sigmund Freud: The Original VR Designer

Loss of one’s eyes is the source of uncanny feelings - Freud

Loss of one’s eyes is the source of uncanny feelings - Freud

TL;DR

  • Freud wrote an essay in 1919 arguing that the origin of the uncanny is the loss of one’s eyes. If you are covering a person’s eyes with an HMD, no wonder it brings up feelings of strangeness.
     
  •  Uncanny is an adjective best used to describe something weird or unexpected.  It is defined in relation to the familiar or expected. 

I’m over the uncanny valley.  When I hear people describe a VR experience by saying, “It’s super uncanny valley,” my first response is usually:

People are overgeneralizing and applying this very precise term to any VR experience that they don’t like. It’s like using “epic” for anything that is even mildly interesting, or “interesting” to describe, well, anything. To elevate discussions of VR, I am going to dissect the meaning and use of “uncanny” in the context of current experiences, so people can confidently describe their subjective experiences of VR & AR with a richer vocabulary.

Let’s Get Freudian

 "The Uncanny” is an essay from 1919, written by Sigmund Freud. While most of Freud’s work has been discredited, this essay remains relevant to VR creators who are making new experiences. The first third of the essay describes what uncanny means in terms of aesthetics and psychology. He states that the uncanny is the aesthetic of anxiety and fear, rather than the aesthetic of beauty.

Freud (being who he is) argues that the origin of the feeling of uncanny is a revival of repressed infantile memories or the return of primitive beliefs of the human species. He also spends a long time giving a psychoanalytic perspective on a story from 1817 called “The Sandman” wherein a man who was traumatized in childhood by almost having his eyes removed falls in love with a woman who turns out to be a doll with unusual eyes. At some point he sees the doll’s eyes on the floor, loses it, and is committed for insanity. Freud believes that the origin of uncanny is the loss of one’s eyes. What does this mean for someone creating for an HMD?

The uncanny has to do with our feelings of what is familiar. Digging into the German words of heimlich (canny/homey) and unheimlich (uncanny/unhomey), you find distinct meanings:

heimlich

1. belonging to the house; friendly; familiar; tame (as in animals); intimate, comfortable; i.e: secure, domestic(ated), hospitable.

2. concealed, secret, withheld from sight and from others; secretive, deceitful = private.

>>> 

unheimlich

1. unhomey, unfamiliar, untame, uncomfortable = eerie, weird, etc.

2. unconcealed, unsecret; what is made known; what is supposed to be kept secret but is inadvertently revealed. (BTW - This would be related to a Freudian slip, saying something accidentally that reveals a hidden truth).


For good measure, I’ll include the Merriam-Webster dictionary definitions of uncanny:

1.  seeming to have a supernatural character or origin :  eerie, mysterious

2.  being beyond what is normal or expected :  suggesting superhuman or supernatural power

Given these definitions of uncanny, I believe that most of the time that people use the phrase “uncanny valley,” they likely mean uncanny. They want an adjective to describe something unfamiliar, eerie, or unexpected. In my next post, I’ll write about the meaning and use of the term Uncanny Valley, coined by a robotics professor in Japan in 1970. Until then, feel free to use “inconceivable” as your general VR adjective. 


Takeaways for designers

  • Uncanny is an adjective best used to describe something weird or unexpected. It’s about the relationship to something familiar or expected. 
     
  • If you don’t want your VR experience to feel quite so uncanny, amp up the heimlich elements of your experience. What can you do to make it more familiar, tame, or hospitable? 
     
  • With current devices, you lose sight of your own surroundings, plus you cannot see the eyes of others, or what they are looking at. If Freud was right and the loss of eyes is the source of the uncanny, we might have to resign ourselves to VR being perceived as uncanny for the near future.

 
Further Reading:

Freud, S. (1919). The uncanny.

Lecture Notes on Freud’s The uncanny.  University of Washington. 

H/T to German scholar Elizabeth Bridges who pointed me to Freud’s essay and patiently answered my questions. You can find her writing on the Uncanny Valley here. 

The Dark Side of Empathy

A viewer watches the process of factory farming as part of the 360 VR documentary series iAnimal.  Photo Credit: Laika Magazine

A viewer watches the process of factory farming as part of the 360 VR documentary series iAnimal.  Photo Credit: Laika Magazine

Empathy is a real buzzword in the world of VR. VR filmmaker Chris Milk has said he aspires to build the ultimate empathy machine. The purpose of the iAnimal series seems to be to raise people's empathy for animals in order to activate their moral reasoning that animals deserve better treatment.

However, while empathy has a role in our lives, I don’t think it should be the end goal of any VR experience, because empathy does not necessarily lead to the fair treatment of others. One study on empathy for blind people woke me up to this.  Adam Waytz summarizes in HBR:

Participants were asked how capable they thought blind people were of working and living independently. But before answering the question, some were asked to complete difficult physical tasks while wearing a blindfold. Those who had done the blindness simulation judged blind people to be much less capable. That’s because the exercise led them to ask "what would it be like if I were blind?" (the answer: very difficult!) rather than "what is it like for a blind person to be blind?" 

People are so egocentric that even empathy tasks get reframed to be first person perspective. Humans are not terribly good at predicting how others feel. Yale psychology professor Paul Bloom wrote an entire book called Against Empathy that advocates using reason rather than empathy.

Consider learning about a ten-year-old named Sheri Summers who had a fatal disease and was waiting in line for treatment that would relieve her pain. Research participants were told that they could move her to the front of the line. When simply asked what to do, they acknowledged that she had to wait because other needy children were ahead of her.  But if they were asked to imagine what Sheri felt, they tended to choose to move her up, putting her ahead of children who were presumably more deserving.  Here, empathy was more powerful than fairness. – Against Empathy, p. 25

While there is certainly a role for empathy and compassion in life, it can sometimes narrowly focus us on the wrong details. And empathy does not appear to be a reliable way to activate people’s moral reasoning. 

Another example of the underbelly of empathy is its ability to give people a reason or motive to be hurtful toward others.  In a study where people were primed to empathize with someone (Person A), the research participants assigned Person A’s competitor (Person B) to eat more hot sauce as a punishment. It appears that creating an intervention that increases empathy toward Person A increased aggression toward Person B.

Rather than relying on empathy to guide fairness, ask people to make judgments based on logic and reasoning.  Gather data by actually speaking with people about their experience, rather than just imagining how they feel. 

Lastly, here’s a link to Paul Bloom speaking on how empathy blinds us to the long-term consequences of our decisions.

 
Additional reading:

Does Empathy Guide or Hinder Moral Action?  The New York Times.  December 29, 2016.  http://www.nytimes.com/roomfordebate/2016/12/29/does-empathy-guide-or-hinder-moral-action.

Waytz, Adam.  The Limits of Empathy.  Jan-Feb 2016.  https://hbr.org/2016/01/the-limits-of-empathy.

Bloom, P. (2017). Against Empathy. Bodley Head Limited.

Shamoon, Evan. Virtual Immersion into the Reality of Farm Animals. Feb 24, 2016.  http://www.laikamagazine.com/reality-animal-suffering/.

 

Invitation to talk at Upload VR January 11th

UploadVR-Logo1.jpg

 

Body Language in VR: How Movement Changes Our Decisions

NBA players who touch each other during games (fist bump, high-five, head grab) cooperate more and outperform their prickly counterparts on other teams. Consider the effect of mimicking those high-fives, hugs, and team huddles in your VR experience – people would feel a high sense of trust and liking for others.

Come learn the behavioral science of how body movement influences thoughts, language, and decisions. Psychology and neuroscience research has established that human behaviors are affected by characteristics that you might not think would affect them – just pulling an item toward yourself leads to greater desire and willingness-to-pay for it.

This talk will address how simple physical movements affect people’s decision processes and how to optimize them in VR interaction design. VR designers can create a broad range of emotional responses (empathy, courage, competitiveness, etc.) just by designing the right movements and gestures for users to do inside of their experiences. No haptics required.

Ticket includes beer, wine, snacks and VR demos on HTV Vive, Oculus, and Gear VR.  Buy yours here.

Details: 

Wed, January 11, 2017

7:00 PM – 10:00 PM PST

Upload Collective

1535 Mission Street

San Francisco, CA 9410 

 

How Social Placebos Boost Performance in VR

Stephen Curry, Andre Iguodala, and Kevin Durant doing a trust exercise – Ezra Shaw/Getty Images

Stephen Curry, Andre Iguodala, and Kevin Durant doing a trust exercise – Ezra Shaw/Getty Images

Are you interested in boosting player performance? Giving people the right social environment will help them achieve more.

NBA players who touch each other a lot during games (fist bump, high-five, head grab) cooperate more and outperform their prickly counterparts on other teams. Consider the effect of mimicking those high-fives, hugs, and team huddles in your VR experience – people will feel a high sense of trust and liking for others.

I’m interested in how small things influence actions and decisions. Why would a small social gesture like a high-five help a professional athlete perform better? The stakes are extremely high for them so you might imagine they are already maxed out on motivation to win. 

Like most things in life, there’s an evolutionary explanation. People who belonged to a strong tribe knew that they could take more risks. In the event of a negative outcome, there were people who could care for you. These small things like fist bumps signal strong social ties.

“I’m not the guy who’s afraid of failure. I like to take risks, take the big shot and all that.” 
    – Steph Curry

Consider how social placebos would change a VR game like Surgeon Simulator: Meet the Medic by Bossa Labs. You are a surgeon in the game and have to perform tasks like heart transplants to save the patient. This is a gaming experience where having people around could boost a player’s performance. Having another person watching you would make you move faster.*

Screen Shot 2016-12-30 at 12.32.29 PM.png

What are the limitations of the social placebo?

 Having an audience when doing a complicated task for the first time could sabotage performance. But, if it’s a straightforward action that doesn’t require any particular skill, having supporters would likely help. And it can be a complex activity, as long as the user has already rehearsed. 

 Also, the encouragement should probably come from the person’s in-group.

Kobe Bryant congratulating Steph Curry on a trifecta

Kobe Bryant congratulating Steph Curry on a trifecta

Does the social placebo work when you are surrounded by AI avatars instead of human avatars? 

 Most likely. It might not last as long or be as effective compared to being around humans you know well and like, but a high five from an AI is likely better than no high five at all.

 How many viewers are optimal?

 It really depends on your goal. One person might be enough. Building a stadium of AI spectators might be overkill, but athletes do get a buzz from those national anthems, pre-game rituals, and cheering fans. 

Takeaway for VR designers:

  • Usage will increase if you build in social placebos. People will be perform at a higher level and have more fun.

*Human runners go faster when they are under observation than when they are solo.  Same effect in cockroaches.  Those pests fun faster when other cockroaches are watching them.

 

How VR Can Lead to False Memories

John Hamm in Black Mirror.  Credit: House of Tomorrow

John Hamm in Black Mirror.  Credit: House of Tomorrow

The human mind functions more like a coping mechanism than a well-oiled machine. We humans are easily fooled into erroneously believing that we have encountered people, objects, and events before when in fact we have not, especially when they are similar to ones that we have genuinely experienced. So what does this mean in VR?

Presence has a downside in VR

Somehow it has become ideal in VR to achieve a high feeling of presence (“being there”). However, the more real a VR experience feels to people, the more likely they are to confuse it with something similar that they have encountered in real life—something to keep in mind when designing darker VR experiences.

But how does this work? It has to do with the malleability of memory. Any content online or offline can change your memory. Even telling a story at a dinner party about your childhood changes your memory of it. The process of recalling a memory itself strengthens it in comparison to other memories.

As Jeremy Dean puts it, “Many memories which have the scent of authenticity may turn out to be misremembered, if not totally fictitious events, if only we could check. Without some other source with which to corroborate, it is hard verify the facts, especially for events that took place long ago…”

In VR, the high presence and photorealism of Google Earth VR could easily lead to people making mistakes in their memory. For example, it would be easy to fool a native of Canton, OH by putting him in Google Earth VR, telling him he’s looking at Canton (when it’s actually Dayton, OH) and asking him to tell you a story about growing up in this neighborhood. I would guess that almost all people would tell you a memory of the place that did not actually occur there, and by doing so would be confusing their own memory of the event (where it actually happened). This is what can happen when photorealism intersects with the nature of memory—the more similar things are to authentic memories, the more likely one is to be tripped up.

Canton or Dayton?  You decide!

Canton or Dayton?  You decide!

Our Memories and Fantasies Intermingle

Here’s a recap of a study where participants were shown images and came to believe they had seen the real art in a museum, but in fact had only ever seen photos. Jeremy Dean summarizes:

"In the experiment participants took part in a self-guided museum tour where they were told to stop at particular exhibits along the way (St. Jacques & Schacter, 2013). These stops on the museum tour are the experimental stand-ins for the events you’ve experienced across your life.

Participants were asked to look at a series of photos showing the stops they visited during their tour, a sort of ‘event-movie’, thus retrieving or reactivating memory. Following retrieval, they were shown a new photo taken at the museum that showed stops that were not part of their tour.

...In the study participants returned for a third session and were shown photos and asked whether they’d stopped at the exhibit or not. Once again, some of the time they were shown pairs of exhibits they had or hadn’t looked at and other times real and false memories were mixed up together.

Across the three sessions, then, the researchers had simulated the recall of the jumble of real and false memories that are likely to be returned to consciousness when we try to recall past events. Real aspects of a memory get mixed up with false aspects and the whole confection gets stirred up each time we recall it.

In the study they found that participants’ memories were both enhanced and distorted by the process of recall. People found it easier to remember those exhibits which they were subsequently shown photographs of. This shows that merely recalling a memory is enough to strengthen it.

This is one illustration of how memory is an active, reconstructive process; recalling something is not a neutral act, but rather it strengthens the recalled memory in comparison to others.

What this is showing is how false memories can grow in the mind. Of course, in real life things don’t happen as cleanly as they do in the psych lab. Our memories and fantasies are intertwined, crossing over and interfering with one another. Thinking about the past continues this process of interweaving…

Choosing to recall certain events rather than others is a way of choosing how we live now and what decisions we make in the future.”
 

Takeaways for Designers:

  •  Human memory is easily influenced by fantasy.
     
  • Giving people experiences in VR can subsequently alter their real life memories. 
     
  • Presence is not always desirable, especially if you are making dark, realistic content.  

Additional reading:

 Dean, Jeremy. “Reconstructing the Past: How Recalling Memories Alters Them” <http://www.spring.org.uk/2013/02/reconstructing-the-past-how-recalling-memories-alters-them.php>

How Google Earth VR Ruins Childhood Memories

On a recent “Voices of VR” podcast, I heard host Kent Bye talking about using Google Earth VR to reminisce and share stories with friends. Now, I had a very happy childhood, but my first reaction was, “I would never reminisce or share stories about my past in Google Earth VR. That’s just going to corrupt those memories.”

Memory is one the pillars of human cognition, in addition to attention, problem-solving, evaluation, and decision-making.  It is also extremely fragile and easy to influence.  If our experiences and memories are what make each of us unique, then it is sobering to realize how unreliable human memory can be. 

Your brain is not a computer. 

Human memory does not behave like a computer.  Memories are not files on a hard drive just waiting to be loaded. They are not fossils to be re-visited in a perfectly maintained museum.  Just by reminiscing about an event in your life, your memory of that event is impacted. 

Recalling memories changes them.

The act of revisiting our memories alters them. Here’s a good recap from Jeremy Dean at PsyBlog: 

"How can recalling a memory change it? Well, just by recalling a memory, it becomes stronger in comparison to other memories. Let’s run this through an example. Say you think back to one particular birthday from childhood and you recall getting a Lego spaceship. Each time you recall that fact, the other things you got for your birthday that day become weaker in comparison.

The process of recall, then, is actually actively constructing the past, or at least the parts of your past that you can remember.

This is only the beginning though. False memories can potentially be created by this process of falsely recalling the past. Indeed, psychologists have experimentally implanted false memories.  This raises the fascinating idea that effectively we create ourselves by choosing which memories to recall."


And Dean writes of another illustration of how easy it is to manipulate human memory:

"A neat experiment by Goff and Roediger (1998) demonstrates how easily our memory can transform fantasy into reality. Participants were asked either to imagine performing an action or actually asked to perform it, e.g. breaking a toothpick. Sometime later they went through the same process again. Then, later still they were asked whether they had performed that action or just imagined it. Those who imagined the actions more frequently the second time were more likely to think they’d actually performed the actions the first time."


You might think - I can remember details about my childhood much better than if a researcher in a white coat asked me to break a toothpick. Possibly, though we tend to have the best memory for the most recent events. Think about the implications of this false memory study for VR. People could easily believe that what they experienced virtually happened in real life.  It makes me wonder if creating realistic presence in VR should actually be a goal, given how it could affect people’s perceptions of reality.

If you are interested in memory, it has been studied extensively by researchers in criminal justice. Cognitive psychologist Elizabeth Loftus has done tremendous work in false memory and the inaccuracies of eyewitness testimony. In short, eyewitnesses are not reliable due to poor viewing conditions, brief exposure, being under stress, having pre-conceived expectations, biases, stereotypes, and more.  
 

Takeaways for VR Designers:

  • The human brain is not a computer.
  • The act of recalling memories changes them. 
  • Maintain skepticism of the memories that people recount to you (and the memories that you personally retrieve).  We are actively constructing our own experiences everyday.

 

 

 

The Impact of Gestures on Social Intelligence

VR and AR rely on interactions from the participant. Being inside of VR/AR experiences requires the user to take action. Users in these open-loop systems have partial control over the experience by moving their eyes, head, hands, or legs. In the PSVR game Headmaster, the player moves his head in order to bounce incoming soccer balls into the net and score points. It’s an easy example of gameplay relying on player movement.  

In Headmaster in PSVR, you act like a soccer player and use your head to spike the ball into the net and score points

In Headmaster in PSVR, you act like a soccer player and use your head to spike the ball into the net and score points

There are important implications from relying on human movement for gameplay. Here I’ve collected some research on how gestures specifically affect people’s cognitive capabilities. From Science of People:

•   "You’re born to speak with your hands. Researchers have found that infants who use more hand gestures at 18-months old have greater language abilities later on. Hand gestures speak to great intelligence.
•    Hand gestures make people listen to you. Spencer Kelly, associate professor of Psychology and co-director of the Center for Language and Brain at Colgate University found that gestures make people pay attention to the acoustics of speech. Kelly said, 'Gestures are not merely add-ons to language – they may actually be a fundamental part of it.'
•    We can’t help it. Hand gestures come to us naturally. Spencer even found that blind people use hand gestures when speaking with other blind people.  After studying native English and Turkish speakers as well as blind and sighted people, the researchers established that people learn gestures from language and grammar NOT from watching others
•    Gesturing helps you access memories. Using hand gestures while you speak not only helps others remember what you say, it also helps you speak more quickly and effectively!
•    Nonverbal explanations help you understand more. One study found that forcing children to gesture while they explained how to solve math problems actually helped them learn new problem-solving strategies." 

And from Psyblog:

    "Gesture for persuasion - The way people’s hands cut through the air while they talk is…more than just a by-product of communication. Maricchiolo et al. (2008) found that hand-gestures help increase the power of a persuasive message when compared to no use of gesture. Most effective are gestures which make what you are saying more understandable. For example, when referring to the past, point behind you."

There are two additional studies on the impact of gestures on thought:

  • fMRI brain imaging studies suggest that people tend to look for meaning in gestures when observing others. This means that VR/AR designers should be thoughtful when creating characters that gesture and limit gestures to things that are relevant and understandable to the user.
     
  • Gestures activate the Mirror Neuron System in the brain, which is associated with mimicry and empathy in social relationships.  Mirror neurons are activated when the user makes gestures or when observing others do the same
     

Takeaways for designers:

  • Gestures activate cognitive processing at a deeper level than just words alone
     
  • The right gestures can make VR/AR feel more immersive, memorable, or persuasive
     
  • Don’t add gestures in just for fun. People look for meaning inside of gestures so limit their use in a way that helps people grasp information more quickly

The Neuroscience of Gestures

“How can you tell what these people are talking about?”

“How can you tell what these people are talking about?”

I’d like to persuade you that gestures are a fundamental building block of human language and thought. This begins a series of blog posts on gestures and how physical movement in VR & AR affects cognition.

Part one of this series will deal with why gestures provide a shortcut to human thought. 

But first, on the tech front:
Devices to capture small hand gestures are already available (like Microsoft Hololens) and more are underway.  Project Soli at Google can use radar to track micro-motions and twitches. The radar from the device senses how the user moves his hands and can interpret the intent. Link to the full Project Soli video here.

Why are gestures powerful shortcuts to cognition?

I’m reposting an article from Scientific American here that answers “Why is talking with gestures so much easier than trying to talk without gesturing?”  Psychology professor Michael P. Kaschak responds:

A person in a fit of rage may have trouble verbalizing thoughts and feelings, but his or her tightly clenched fists will get the message across just fine.

Gesturing is a ubiquitous accompaniment to speech. It conveys information that may be difficult to articulate otherwise. Speaking without gesturing is less intuitive and requires more thought. Without the ability to gesture, information that a simple movement could have easily conveyed needs to be translated into a more complex string of words. For instance, pointing to keys on the table and saying, ‘The keys are there,’ is much faster and simpler than uttering, ‘Your keys are right behind you on the countertop, next to the book.’

The link between speech and gesture appears to have a neurological basis. In 2007 Jeremy Skipper, a developmental psychobiologist at Cornell University, used fMRI to show that when comprehending speech, Broca’s area (the part of the cortex associated with both speech production and language and gesture comprehension) appears to ‘talk’ to other brain regions less when the speech is accompanied by gesture. When gesture is present, Broca’s area has an easier time processing the content of speech and therefore may not need to draw on other brain regions to understand what is being expressed. Such observations illustrate the close link between speech and gesture.

Takeaways for VR/AR Designers:

  • People process information more deeply when they are gesturing
  • Verbal areas of the brain are more active when speech accompanies gestures 
  • The tech exists for picking up human micro-gestures

How to Use Gestures to Learn Faster

Gestures make it easier to learn.  When people are speaking and gesturing at the same time, they process information better.  From New York Magazine

"University of Chicago psychologist Susan Goldin-Meadow and her colleagues have found that when toddlers point at objects, they’re more likely to learn the names for things; that for adults, gesturing as you try to memorize a string of numbers prompts better recall; and that when grade-schoolers gesture, they’re better at generalizing math principles.

The authors found that the students in both gesture conditions were more likely to succeed on follow-up generalization problems, which required understanding the underlying principle beneath the first problem and applying it in novel situations. It’s a case study in how gesture 'allows you a space for abstraction,' Goldin-Meadow says. 'You’re not as tied to the particulars of an item, of a problem, a word, or an experience.' You’re not just talking with your hands, in other words; you think with them, too.

Researchers haven’t yet pinned down exactly how this connection works, but Goldin-Meadow believes part of it is that gestures reduce what psychologists call 'cognitive load,' or the amount of mental energy you’re expending to keep things in your working memory."


Gestures are a good illustration of how humans think with more than just our brains.  The brain can process more information with gestures than without them, which makes them pretty fundamental to human capabilities. 


Takeaways:

  • Users moving their hands inside of a digital experience has cognitive consequences
  • Giving users alternative, embodied ways to learn information will help them retain concepts
  • Gestures are effective because they allow working memory to offload effort