cognitive processing

Why is novelty important in VR/AR?

When you are in a novel situation, your brain processes information differently than when you are in your humdrum routine.  For example, can you remember your commute home last night?  Or what you ate for lunch last Wednesday?  In a normal routine, attention and memory don't encode as well compared to when you are doing something new and novel.  This is why some neuroscientists insist on driving home via a new route every night.  

Being able to change attention and memory processing has important consequences when you are making a VR/AR experience.  What things do you want your user to pay attention to?  What do you want them to remember after the experience?  Here's a model of cognitive processing that you can use to consider how new information is going to be presented to users called "Top-Down Bottom-Up."

Excerpted from: Mind: Journey to the Heart of Being Human’ by Daniel J Siegel:

In the view we will be using here, top-down refers to ways we have experienced things in the past and created generalised summaries or mental models, also known as schema, of those events. For example, if you’ve seen many dogs, you’ll have a general mental model or image of a generic dog. The next time you see a furry canine strolling by, your top-down processing might use that mental model to filter incoming visual input, and you won’t really see the uniqueness of this dog in front of you. You have overlaid your generalised image of ‘dog’ on top of the here-and-now perceptual stream of energy that creates the neural representation of ‘dog’. What you actually have in awareness is that amalgam of the top-down filtering of your experience.

So here, ‘top’ means that prior experience is activated, making it difficult to notice the unique and vibrant details of what is happening here and now. The top-down generalised notion of dog will shade and limit your perception of the actual animal in front of you. The benefit of top-down is that it makes your life more efficient. That’s a dog, I know what it is, I don’t need to expend any more energy than needed on insignificant, non-threatening things, so I’ll take my limited resources and apply them elsewhere. It saves time and energy, and therefore is cognitively efficient. That’s top-down processing.

On the other hand, if you’ve never seen a spiny anteater before, the first time you come across one on the trail, it will capture all of your attention, engaging your bottom-up processing so that you are seeing with beginner’s eyes. These are eyes leading to circuitry in the brain, not shaping and altering ongoing perception through the top-down filters of prior experience. You’ll be taking in as much pure sensation from eyesight as possible, without the top-down filter altering and limiting what you see now based on what you’ve seen before.

These screenshots are from UpLoadVR's playthrough of the game "Accounting" by Squanchtendo.  The game uses different environments, such as a windowless accounting office (image left) to contrast with fantasy worlds where there are talking trees (image right) and other bizarre characters.  When you are in the fantasy world of "Accounting," you are likely using "bottom up" cognitive processing.  People feel excited when they are trying something new.  They take information from the exterior world, move it up a level, add language, move it up, analyze, and the user experience shifts.  Novel experiences tend to be more pleasurable because they rely on sensation and not cognition. 

The most vivid part of the mind bubbles up through sensation and new experience when unencumbered by analytical thought.
— Daniel J. Siegel

More on sensation from Siegel:

Sensation might be as bottom-up as we get. Since we live in a body, our within-mind experience is shaped by the physical apparatus that lets us take in energy flow from the outside world. We have our first five senses of sight, hearing, smell, taste and touch; we have our proprioceptive sense of motion; and we have our ‘interoceptive’ sense of the signals from the interior of the body. These perceptual capacities to sense the outer world and internal bodily world are built upon the physical neural machinery that enables energy to flow. Information is created with these energy patterns, generated as ions flow in and out of membranes and chemicals are released in the pathways of neural activity. As energy flows into the brain from our external sense organs, such as our eyes and ears, or from internal receptors of our body’s muscles, bones and internal organs, we move from sensation to perception – with pure sensation as close as we get to being fully present in the world.

Key Takeaways for VR/AR experience designers

  • Novelty increases awareness, which influences attention and memory
  • Top-down is the analytical system, which relies on prior experience
  • Bottom-up is the sensation-driven system, which looks with the eyes of a beginner 
  • Take a new route home on your commute tonight to experience novelty yourself.  Then look for ways to integrate that type of novelty into your experience.  


Further reading: 

Siegel, Daniel J. The Open Mind.  Aeon Magazine.  October 25, 2016

UploadVR full play through of "Accounting"

Design to Score an "A" for Effort

How much effort should you require from users when they engage in your experience?  A lot. In some surprising ways, challenge is good for user engagement.  The effort required from users doesn’t just affect their experience - it changes their cognitive processing.  

Usability experts strive to make experiences easier. However, I want to share an example of how making things harder may improve outcomes. 

Exerting more physical effort improved performance on cognitive tasks

Earlier I wrote about how people with Botox have a difficult time identifying the emotions of the people around them.  This is because they cannot mimic the expressions of others since Botox paralyzes their facial muscles - their brains lack feedback that comes from mimicry.  The consequence is that the Botox-ed participants were less empathetic compared to control groups. The same researchers who did the Botox study conducted a follow up where half the participants applied a gel mask on their face and half wore it on their arm.  Then, they all went through the same “Reading the Mind in the Eyes Test (RMET - more details here), which measures a person’s grasp of the emotional states of others.  

The gel used in this study was an over-the-counter facial mask. 

The gel used in this study was an over-the-counter facial mask. 

The participants who had the gel mask on their face outperformed the ones with it on their arms by a statistically significant amount on the RMET empathy task.  The face mask people had a 77.2% accuracy compared to 72.5% for the arm mask participants serving as a control. The researchers inferred that by forcing people to exert more effort to mimic others, this exaggeration led to afferent nerves (the same one that Botox had paralyzed in the first study) being amplified, and firing stronger signals to the Central Nervous System.  

This study provides further evidence that people are thinking with their bodies and not just their brains.  The “amplified emotion” face mask group performed better than the arm-mask control group at identifying emotion because they had to work harder to move their facial muscles to mimic the faces they viewed in RMET. 

Higher effort can deepen engagement

The lesson is that when people exert more effort to perform a task, they tend to process the information on a deeper level, which increases their engagement. There is additional research on the benefits of requiring more effort from users to improve outcomes. Different researchers discovered that people who read in a difficult, cursive-y font do better on subsequent memory and analytical reasoning tasks compared to those who read in a normal print font. 

For anyone making a VR experience, consider how you are calibrating the level of effort required from your users.  Are there physical tasks you can build in that would help amplify emotions and deepen engagement?  


Further Reading

Neal, D. T., & Chartrand, T. L. (2011). Embodied emotion perception amplifying and dampening facial feedback modulates emotion perception accuracy. Social Psychological and Personality Science2(6), 673-678.

Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136(4), 569.