cognition

How Google Earth VR Ruins Childhood Memories

On a recent “Voices of VR” podcast, I heard host Kent Bye talking about using Google Earth VR to reminisce and share stories with friends. Now, I had a very happy childhood, but my first reaction was, “I would never reminisce or share stories about my past in Google Earth VR. That’s just going to corrupt those memories.”

Memory is one the pillars of human cognition, in addition to attention, problem-solving, evaluation, and decision-making.  It is also extremely fragile and easy to influence.  If our experiences and memories are what make each of us unique, then it is sobering to realize how unreliable human memory can be. 

Your brain is not a computer. 

Human memory does not behave like a computer.  Memories are not files on a hard drive just waiting to be loaded. They are not fossils to be re-visited in a perfectly maintained museum.  Just by reminiscing about an event in your life, your memory of that event is impacted. 

Recalling memories changes them.

The act of revisiting our memories alters them. Here’s a good recap from Jeremy Dean at PsyBlog: 

"How can recalling a memory change it? Well, just by recalling a memory, it becomes stronger in comparison to other memories. Let’s run this through an example. Say you think back to one particular birthday from childhood and you recall getting a Lego spaceship. Each time you recall that fact, the other things you got for your birthday that day become weaker in comparison.

The process of recall, then, is actually actively constructing the past, or at least the parts of your past that you can remember.

This is only the beginning though. False memories can potentially be created by this process of falsely recalling the past. Indeed, psychologists have experimentally implanted false memories.  This raises the fascinating idea that effectively we create ourselves by choosing which memories to recall."


And Dean writes of another illustration of how easy it is to manipulate human memory:

"A neat experiment by Goff and Roediger (1998) demonstrates how easily our memory can transform fantasy into reality. Participants were asked either to imagine performing an action or actually asked to perform it, e.g. breaking a toothpick. Sometime later they went through the same process again. Then, later still they were asked whether they had performed that action or just imagined it. Those who imagined the actions more frequently the second time were more likely to think they’d actually performed the actions the first time."


You might think - I can remember details about my childhood much better than if a researcher in a white coat asked me to break a toothpick. Possibly, though we tend to have the best memory for the most recent events. Think about the implications of this false memory study for VR. People could easily believe that what they experienced virtually happened in real life.  It makes me wonder if creating realistic presence in VR should actually be a goal, given how it could affect people’s perceptions of reality.

If you are interested in memory, it has been studied extensively by researchers in criminal justice. Cognitive psychologist Elizabeth Loftus has done tremendous work in false memory and the inaccuracies of eyewitness testimony. In short, eyewitnesses are not reliable due to poor viewing conditions, brief exposure, being under stress, having pre-conceived expectations, biases, stereotypes, and more.  
 

Takeaways for VR Designers:

  • The human brain is not a computer.
  • The act of recalling memories changes them. 
  • Maintain skepticism of the memories that people recount to you (and the memories that you personally retrieve).  We are actively constructing our own experiences everyday.

 

 

 

The Impact of Gestures on Social Intelligence

VR and AR rely on interactions from the participant. Being inside of VR/AR experiences requires the user to take action. Users in these open-loop systems have partial control over the experience by moving their eyes, head, hands, or legs. In the PSVR game Headmaster, the player moves his head in order to bounce incoming soccer balls into the net and score points. It’s an easy example of gameplay relying on player movement.  

Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 JA 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:10.0pt;
	font-family:"ArialMT","sans-serif";
	mso-fareast-language:EN-US;}
 
    In Headmaster in PSVR, you act like a soccer player and use your head to spike the ball into the net and score points

In Headmaster in PSVR, you act like a soccer player and use your head to spike the ball into the net and score points

There are important implications from relying on human movement for gameplay. Here I’ve collected some research on how gestures specifically affect people’s cognitive capabilities. From Science of People:

•   "You’re born to speak with your hands. Researchers have found that infants who use more hand gestures at 18-months old have greater language abilities later on. Hand gestures speak to great intelligence.
•    Hand gestures make people listen to you. Spencer Kelly, associate professor of Psychology and co-director of the Center for Language and Brain at Colgate University found that gestures make people pay attention to the acoustics of speech. Kelly said, 'Gestures are not merely add-ons to language – they may actually be a fundamental part of it.'
•    We can’t help it. Hand gestures come to us naturally. Spencer even found that blind people use hand gestures when speaking with other blind people.  After studying native English and Turkish speakers as well as blind and sighted people, the researchers established that people learn gestures from language and grammar NOT from watching others
•    Gesturing helps you access memories. Using hand gestures while you speak not only helps others remember what you say, it also helps you speak more quickly and effectively!
•    Nonverbal explanations help you understand more. One study found that forcing children to gesture while they explained how to solve math problems actually helped them learn new problem-solving strategies." 

And from Psyblog:

    "Gesture for persuasion - The way people’s hands cut through the air while they talk is…more than just a by-product of communication. Maricchiolo et al. (2008) found that hand-gestures help increase the power of a persuasive message when compared to no use of gesture. Most effective are gestures which make what you are saying more understandable. For example, when referring to the past, point behind you."

There are two additional studies on the impact of gestures on thought:

  • fMRI brain imaging studies suggest that people tend to look for meaning in gestures when observing others. This means that VR/AR designers should be thoughtful when creating characters that gesture and limit gestures to things that are relevant and understandable to the user.
     
  • Gestures activate the Mirror Neuron System in the brain, which is associated with mimicry and empathy in social relationships.  Mirror neurons are activated when the user makes gestures or when observing others do the same
     

Takeaways for designers:

  • Gestures activate cognitive processing at a deeper level than just words alone
     
  • The right gestures can make VR/AR feel more immersive, memorable, or persuasive
     
  • Don’t add gestures in just for fun. People look for meaning inside of gestures so limit their use in a way that helps people grasp information more quickly

The Neuroscience of Gestures

96 
  
    
  
 Normal 
 0 
 
 
 
 
 false 
 false 
 false 
 
 EN-US 
 JA 
 X-NONE 
 
  
  
  
  
  
  
  
  
  
  
 
 
  
  
  
  
  
  
  
  
  
  
  
  
    
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  
   
 
 /* Style Definitions */
table.MsoNormalTable
	{mso-style-name:"Table Normal";
	mso-tstyle-rowband-size:0;
	mso-tstyle-colband-size:0;
	mso-style-noshow:yes;
	mso-style-priority:99;
	mso-style-parent:"";
	mso-padding-alt:0in 5.4pt 0in 5.4pt;
	mso-para-margin:0in;
	mso-para-margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:12.0pt;
	font-family:Calibri;
	mso-ascii-font-family:Calibri;
	mso-ascii-theme-font:minor-latin;
	mso-hansi-font-family:Calibri;
	mso-hansi-theme-font:minor-latin;
	mso-fareast-language:EN-US;}
 
  “How can you tell what these people are talking about?”

“How can you tell what these people are talking about?”

I’d like to persuade you that gestures are a fundamental building block of human language and thought. This begins a series of blog posts on gestures and how physical movement in VR & AR affects cognition.

Part one of this series will deal with why gestures provide a shortcut to human thought. 

But first, on the tech front:
Devices to capture small hand gestures are already available (like Microsoft Hololens) and more are underway.  Project Soli at Google can use radar to track micro-motions and twitches. The radar from the device senses how the user moves his hands and can interpret the intent. Link to the full Project Soli video here.

Why are gestures powerful shortcuts to cognition?

I’m reposting an article from Scientific American here that answers “Why is talking with gestures so much easier than trying to talk without gesturing?”  Psychology professor Michael P. Kaschak responds:

A person in a fit of rage may have trouble verbalizing thoughts and feelings, but his or her tightly clenched fists will get the message across just fine.

Gesturing is a ubiquitous accompaniment to speech. It conveys information that may be difficult to articulate otherwise. Speaking without gesturing is less intuitive and requires more thought. Without the ability to gesture, information that a simple movement could have easily conveyed needs to be translated into a more complex string of words. For instance, pointing to keys on the table and saying, ‘The keys are there,’ is much faster and simpler than uttering, ‘Your keys are right behind you on the countertop, next to the book.’

The link between speech and gesture appears to have a neurological basis. In 2007 Jeremy Skipper, a developmental psychobiologist at Cornell University, used fMRI to show that when comprehending speech, Broca’s area (the part of the cortex associated with both speech production and language and gesture comprehension) appears to ‘talk’ to other brain regions less when the speech is accompanied by gesture. When gesture is present, Broca’s area has an easier time processing the content of speech and therefore may not need to draw on other brain regions to understand what is being expressed. Such observations illustrate the close link between speech and gesture.

Takeaways for VR/AR Designers:

  • People process information more deeply when they are gesturing
  • Verbal areas of the brain are more active when speech accompanies gestures 
  • The tech exists for picking up human micro-gestures