Virtual Reality for Theater: Engaging narrative in VR

A VR based pre-show experience for an upcoming theater musical - The Next Fairytale


Overview

Problem

To create a narrative and theater based experience in Virtual Reality to help people empathize with the characters through role play

Solution

Created a VR narrative based game that guides the player through an emotional journey and helps them empathize with the characters in the play - The Next FairyTale. This VR experience explores a traumatic event in the antagonist’s backstory, allowing the audience to interact with characters from the musical.

Project Duration

This was a 6 month project from January 2018 to June 2018.

My Role

The team comprised of 8 people with designers, developers, producers, and artists. We were mentored by the Apples and Oranges studio, Tim Kashani and Zach Anderson, and by our professor, Dr. Tess Tanenbaum. We collaborated with script writers and voice artists. We then ran a study on the system. My role was as a concept designer, developer, and lead researcher.

Academic paper

Our paper is in review for CHI 2020

Design Process

Why Theater based VR?

Theater plays a fundamental role in showing diverse stories, entertaining audiences, providing culturally rich experiences, and connecting audiences and actors emotionally. We believe that combining theater with emerging technology can help individuals connect with theatrical plays in stand alone experiences and can also help the theater industry by providing them a platform for pre-show experiences for broadcasting their upcoming plays.

Team Process

We started by analyzing products that presented theater in VR and understood the limitations. After receiving the play's script, we had multiple brainstorming sessions. We evaluated our ideas with the help of our mentors and iterated through brainstorming and evaluation sessions until we reached our final product.


Existing Products

After studying current solutions such as cirque de soleil and the circle of life, we found that within this niche area, most solutions target a 360 view of the play. On further discussions, we felt that these solutions did not capture the core essence of theater - connecting the audience emotionally with the actors and the story. Hence, we decided to create a solutions that could individually engage one person in VR at an empathetic level with the narrative.


Challenges

The first challenge that we encountered was stepping into a filed that had not been well explored. We also thought that VR may subvert the lively vibe of the troupe and venue. This led us to think how we could engage the VR player and enable them to become a part of the play, giving them an interactive theatrical experience. We then studied the different levels of interaction that the audience and the viewers could achieve.

Initial Design

We first explored how the audience could view the play from different character perspectives, giving them the ability to warp in and out of any actor's frame of view. We realized that this concept did not encourage and empathetic connection between the audience and the actor. We then moved on to other different storyboards


Storyboards based on Interactivity

After numerous iterations, we came up with four key ideas. In all these ideas we saw a common theme emerge - the concept of a character's space that the player would enter. In concept 1, the player would enter a character's 'the mirror's' house and see the play along with the mirror, where the house would react to the play. In concept 2, players would be asked to choose a costume and dress up as a character to replace a missing actor in the play. They would then be introduces to the back stories of characters before going through the play. Concept 3 was about performing situated tasks along with the characters and concept 4 used theories of back-leading and audience engagement. Using these ideas, we came up with out final concept.


Final concept

Goal
For our final concept, we went back to our goal - To help the player develop an empathetic connection with the character.

Story
We decided to present the back story of the antagonist, 'Minerva', to help the audience empathize with her reasoning and decisions. Minerva is the queen of all fairy god mothers. In the pre-show VR experience, players take the role of Minerva's sister 'Calliope', as she tries to convince Minerva to marry the love of her life. Minerva is duty bound and stubborn on sacrificing her love for the realm of the kingdom.

Setting
This interaction takes place in a forest glade giving a magical effect, to keep the theme of the play




Transform
To help readers take the role of Calliope, they are positioned in front of the magical mirror. Here Calliope transforms from her commoner disguise into her natural queen from. This transformation aims to aid players into taking Calliope's identity, as they can see their actions being reflected in the mirror.




Emotions
The main interaction of the players is to guide the emotional tone of the conversation by using emotional spells - happy, calm, sad, and angry. These spells do not change the underlying content of the dialog, rather they only present the dialog through varying emotional settings. As Calliope, players tap different emotional spells to change the atmosphere of the conversation. Minerva's appearance also changes in response to the spells, as shown in the images below. Through this comversation, players understand Minerva's views on love and life.




Theory
We used concepts of back leading and 'mask' effect in theater and non verbal digital communication



Art
A voxel artist joined our team to develop the art work



Development
We worked on a sprint based development cycle to make the application in 9 weeks.

Play Experience


Study

We recruited 12 participants through convenience sampling to test our system and answer our research questions. Sessions were conducted in our lab and lasted for a duration of one hour each. The participants were first given story context and a verbal overview of the basic interactions, followed by a short practice session / demo in VR. Once the participants signaled that they were ready, they were taken to the main VR experience, which lasted 7-10 minutes. We asked the participants to think aloud and took observation notes. The experience was followed by a semi-structured interview where we asked them about their understanding of the system, how they felt about role play, and their views on Minerva. We analyzed the interview transcripts and the think aloud notes taken during the session, through open and axial coding. We then visualized the data on a whiteboard to make connections between the various codes and themes. Based on what we learned in our findings, we conducted brainstorming sessions and redesign exercises to inform our design recommendations.



Research Questions

RQ1: How can we encourage people to take on role playing in narrative based VR experiences?
RQ2: How do people act in such a VR experience and what does role playing enable them to do?

Findings

We observed how people acted when they took Calliope's role, design element that encouraged and hindered role playing, and learning within this space. Here are a few findings from the study:

Connecting with the characters
Three participants got into the role of Calliope through the personal connection they felt with Minerva.

P07: "I felt like Calliope! Like I felt like I was actually talking to a sister and trying to persuade her.”


Many participants described a sense of coming to better understand Minerva through play. They also felt a sense of responsibility towards her. They had developed and emotional attachment to the character and wanted to see how the story would end. They were sensitive towards Minerva and they chose to have a meaningful conversation with her.
Participants chose positive emotions such as happy and calm to show they were understanding and encouraging. Participants chose negative emotions such as angry and sad to make Minerva reflect on her decisions and bring out what she truly felt.

P11: "I felt like when I picked more happy options, she would still do what she needed to do but she would be more optimistic about that decision. I would pick the sad one to know what are her true emotions because that what I thought would lead me to what her actual reaction.”


Sad spells were effective in evoking a sense of sympathy. One participant even wanted to reach out to comfort Minerva. He expressed his urge to put his hand on her shoulder and give a sense of being there for her.

Factors that enhanced role taking
Participants took Calliope’s role when they saw their reflection in the mirror. They got context of who they were and what Minerva’s troubles were by conversing with the mirror. The mirror also helped them visually transform into Calliope and leave their own identity behind.

P11: "Instead of being a random person or me inserting myself into the story, it (mirror) visually tells you are not you, you are someone else and that helps to put you in the mindset, ok, I am playing someone, I am not playing me.”


Nearly all participants reported feeling like they had stepped into a different world and forgetting about the real world within the immersive space of VR. Two participants reported that the action of selecting a spell and casting it with their wand made them feel like the fairy queen Calliope. We observed that almost all participants followed basic real-life social norms such as maintaining personal space and having a mutual gaze with Minerva while talking to her.


Factors that hindered role taking
Some of the most useful lessons from our study came from hearing the participants critique the experience. Five of our participants found the initial entry into VR disorienting. The new environment, the unfamiliar interface, and the immediate litany of instructions from the Mirror was overwhelming.

P04: "There was a point of getting used to the environment and this overlapping narrative. That wasn’t an easy transition for me as I had to pay attention to two things."


We also observed that some people were engaged in the experience merely due to VR’s novelty effect and not the storytelling experience. Six people pointed they wanted to spend time exploring the world rather than just hearing the narrative.

P03: "I think if there could be a clear guidance in it, it would be better. Because when I first in the environment, I am so excited and pay a lot attention to explore the surroundings and ignore the things I should do."


Three participants wanted more interactive elements (such as more characters and objects) to engage with. Five people said that they did not pay attention to the mirror due to the reasons above, which was crucial in telling people who they were and giving context of the story.

P05: "If I could explore a little bit, there might be some other cues for example a book about her family history or something that I can explore a little bit before having a conversation with her."


People who did not feel they were Calliope could not place themselves in the scene and did not feel as if they were a part of the story. Most of their spell selections were exploratory rather than conscious decisions.

Design Recommendations

Based on our findings, we derived design lessons for roleplay, social interactions, and learning for VR environments. Here are a few from the paper:

Setting player expectations with explicit roles
Recognizing that entering VR can be disorienting, we argue that designers should saturate their experience with cues and information about the role the player will inhabit and the expectations of what they are going to do.
We recommend that showing how people look in VR can help change their identity visually. We felt that more backleading cues such as the mirror asking them to do more bodily movement may have helped some participants realize that the reflection belonged to them. Once a player knew who they were in the experience, they needed to know what to do next.
In TNFT, the mirror recites information about Calliope and Minerva, but it goes by quickly and is not repeated. It would help to spread out such important information throughout the experience rather than giving it all at once in the beginning. This context of being Calliope could be repeated at a visual level through the mirror, and at an audio level by the mirror calling out and reminding them what to do.
The fact that players were responsible for Minerva and they used affective spells to impact her emotional tone helped players act out Calliope’s role.

Leveraging the “mask and the mirror” phenomenon
Our second design recommendation is to give players a clear opportunity to put on their “mask” and then reflect it back at them as often as possible. Actions such as using the spells and tapping them with a wand helped a few participants feel like the fairy godmother character.
We can envision easing people into the character of Calliope at the beginning by making the act of transforming into the character more directly under the player’s control. Rather than appearing fully equipped, the player could be asked to collect their wand and cloak on the way to the mirror.
Other interactions with more characters that bowed to them and called them Queen Calliope could have helped them stay in Calliope’s role as well.