Overview
This project emerged from an existing partnership between Rochester Institute of Technology (RIT) and Genesee Country Village & Museum (GCV&M). This perhaps unlikely pairing between a research university with more than 19,000 students and the largest living history museum in New York provides opportunities for faculty, staff, and students from virtually every college within RIT to foster collaborations from a variety of disciplines. Many projects and research areas have multi-disciplinary or cross-disciplinary foci. This project, “Blending Disciplines for a Blended Reality: Virtual Guides for a Living History Museum,” is one such example where an interdisciplinary, research-inspired question forges connection among multiple constituencies within the university with the museum as the site for developing tangible skills and undertaking projects that have scholarly reach and long-term, mutual benefit. Because of this partnership, and the trust and history of association between the two organizations, we, as faculty researchers, have the freedom and flexibility to foster interdisciplinary collaboration in a meaningful way and to engage our students in developing skills in storytelling, digital composition, and multimodal literacy. The museum contributes to, and benefits from, the research and output of this collaboration, thereby serving as a site where our research can thrive.
Process
While this concept at present involves faculty and students from several disciplines with the production geared toward realizing work around one historical person that we have developed and provided with historically accurate contextual narrative, our project began with a much broader framing that our students helped to refine. The collaboration began in 2018 between computer science and museum studies faculty who wanted to set a research problem at the museum, employ technology as a possible solution, and engage our students in the research and scholarship around this project. Inspired by ongoing research with intelligent virtual agents (IVA) (Norouzi et al. 2018), we pondered as to what role an IVA might play in the context of a living museum. We posed the question: “Could a stylized avatar, serving as a historical guide, be used to augment visitors’ physical experiences at Genesee Country Village & Museum?” Over two semesters, the faculty and students from museum studies and computer science, with the help of faculty and students from theatre, developed 38 historical narratives which were recorded via audio only or audio and motion capture. This earliest phase of exploration was evaluated by team members, Decker and Geigel, in November 2018 and in April 2019, which in turn enabled them to pivot the project in five ways over the past several months. The project team expanded to include collaborators from among art and design faculty. In turn, we began to focus on researching and developing one character initially; to create historically accurate clothing, props, and environment for the character; and to refine our workflow—all with the end goal of stacking historical narratives into a six-minute story, delivered in monologue form as a vignette to engage with the visitor. This article outlines the project over this entire span of 18 months, with primary focus on the past several months, which is the period of robust project development and testing by students and faculty.
Exploration
The earliest iteration of this project was exploratory in order to see the viability of our launching a long-term project. Led by two of our four-person faculty team (Decker and Geigel) from spring 2018 through spring 2019, we tested the technology and research/script-writing as well as recording. In terms of technology, we chose a Microsoft HoloLens as a delivery platform as it provides an intuitive, hands-free interface as well as built-in voice recognition. Furthermore, the HoloLens has been shown to be an effective platform in other museum contexts (Hammady et al. 2019). Development was done using Unity, a 3D software platform for rapid prototyping of VR and AR applications. We developed three short sprints using prescribed, pre-loaded character types in Unity, writing an application in Unity to allow for placement of the virtual storyteller at specific, appropriate spots on our campus (in lieu of the museum). As the app was running on a HoloLens, users had the opportunity to interact with the application by asking pointed questions of the avatar, to which the storyteller would respond, making users feel like they were having a conversation with a real person.
Research
Initially, neither the text nor the visuals used in this exploratory phase were keyed to our historical site. However, simultaneous to the technology testing, we asked museum studies students to research the buildings situated at the museum and to develop “character types” who might be viable suggestions for developing an AR character for this project. Over two semesters of increasingly focused, exploratory research, the students created 38 one-to-three-minute monologues situated at 12 of the 68 historic structures at the museum. Each of these monologues was historically based and researched using resources from the museum as well as contextual sources (Bolger 1985). While only a small portion of this overall work was, in turn, used as part of our refined prototype (explored fully in this article), the initial research phase informed our workflow, as well as the decision to develop one character more fully to focus our team’s concept development and execution.
Over the summer of 2019, in consultation with the Genesee Country Village & Museum staff,1 the team selected Dr. Frederick F. Backus (1794–1858) as the inspiration for our first fully developed character. First and foremost, Backus had myriad interests and connections to Rochester history, making his story rich with intersections that could, in turn, be amplified through research-informed narrative writing. Second, we chose this individual in order to tether our virtual character with his actual home, one of the first grand mansions in Rochester which Backus purchased in 1838. Third, and perhaps most interestingly in terms of creative output, no images exist of his appearance, thereby making him an opportunity to blend historical reality and interpretation.
After deciding upon a character, primary and secondary research guided the script-writing, with the immediate need to develop one narrative for this phase of testing. Immediately, the decision was made to situate the character a bit later in his life, so as to draw upon a wealth of experiences documented by Backus in letters. As the museum interprets the home to the year 1850, writing a script that would be situated at around the same time of the museum’s interpretation bolstered our ability to render a seamless integration between the AR experience and the museum environment.
Historically accurate assets were gathered as part of the research. These include professional and domestic contexts, including newspaper accounts from the years that Backus served in the New York State Senate and background information on Backus’s neighborhood gleaned from property records and maps of Rochester’s wealthy Third Ward. This portfolio of research was passed on to the museum studies students in the spring of 2020 to guide their development of academically and historically rigorous narratives for five characters (Backus and four additional characters).
The current student cohort (spring 2020) developed 15 monologues focused on individuals who lived in the region over the years that the museum interprets (Pioneer Settlement era of 1780s through the 1920s), with particular attention to the 1820s–1860s. These individuals included the aforementioned Frederick Fanning Backus (1794–1858); Candace Beach (1790–1850), a teacher at a one-room schoolhouse who lived through the historic “year without summer” that occurred in 1816, over a three-year period of climate change and uncertainty as a result of the eruption of Indonesia’s Mt. Tambora in the spring of 1815; John Carlin (1813–1891), a poet and painter who graduated in 1825 from Pennsylvania Institute for the Deaf and Dumb before traveling to England and France for a Grand Tour and returning to New York and picking up clients across the state; Austin Steward (1793–1869), who was born to enslaved parents in Virginia before moving to New York and becoming engaged in antislavery and temperance as well as the black convention movement, all the while being engaged as a merchant, publisher, and orator years before Frederick Douglass settled in this region; and Lavinia Fanning Watson (1818–1900), a Philadelphia socialite, with ties to the region, who was the first woman to commission a naval ship, the USS Germantown (1846). The monologues were sited at three of the buildings on the museum campus.2
Design: character, model, and rigging
Throughout the research phase, the faculty team had discussed how to proceed with the digital design phase. The decision process for creating the first 3D character (described below) would also inform projects and workflow for continued production, including the development of additional characters in spring 2020 and beyond.
The design process began with the choice to build a stylized avatar, rather than a realistic 3D animation, so as to avoid the “uncanny valley”: a feeling of unease and disconnect experienced when humans encounter robotic or audio/visual simulations that are too realistic. This key decision was informed by the work of Masahiro Mori who presented the theory of the uncanny valley five decades ago (Mori 2012). Mori posited that an individual’s feeling about a human-like robot would go from empathy to revulsion the closer the representation grew to reality, because the representation would naturally not achieve true realism. Mori’s premise has been applied to the development of digital characters as well, as the uncanny valley is often referenced vis-à-vis the film Polar Express (Noe 2012) and CGI characters that fail to achieve true realism and therefore alienate the viewer (Weschler 2011). For Weschler in particular, the “vacant” quality of the eyes and unrealistic movement are cited as features that foster the eeriness associated with the uncanny valley.
While some scholars are now exploring the ability of digital artists to create avatars realistic enough to foster trust and empathy, such production is at a level of digital artistry that requires mastery and extensive experience. Students would not have the expertise to overcome this valley and therefore we chose to pursue a stylized character. The choice meant the final agent would be distinctly unrealistic in an authentic historical environment. We accepted this anachronism as a way of attuning to the museum’s approach to onsite interpretation. GCV&M does not presume visitors are transported to 1850; it sets out to interpret and demonstrate the era authentically while acknowledging that the museum staff, chiefly the costumed interpreters and the guests, are inhabitants of the present. Additionally, we intended to utilize modern technology (HoloLens) to immerse the viewer in the experience of interacting with the character, further removing them from the idea of being transported to the past. Our digital agent, viewed through the HoloLens, would clearly be an AR animation and not an actual human interpreter, so the decision to opt for a stylized avatar meant students could design all aspects of the character with the burden of bridging the uncanny valley relieved.
The avatar needed to be approachable and warm in order to appeal to older adults and children alike. To avoid a sense of unease, certain attributes are exaggerated in digital human representations—most often the size of the head, hands, and feet. For continuity of design, the style developed would be carried through into additional avatars, to be executed by 3D digital design students.
As a character, Backus presented the unusual but fortunate position of having an actual historical figure for whom there is no visual record, only written references. With no extant images, the team was left to interpret his appearance through the use of his father’s portrait from Hamilton College and his own writings of his life experiences. The character design incorporates the physical input of Azel Backus as the subject’s father, with historic, social, and economic aspects of the 1851 time period. As we move forward with students to develop further avatars and agents for the museum, classes will follow the same pattern of character analysis in the design, regardless of any visual references we may have of subjects. Thorough research of the fashion of the period was balanced with the knowledge that Rochester, NY in 1851 was both rural and remote and therefore not on trend with the latest styles. It was also clear from the writings of our historic subject that he had traveled the area and experienced the hardships of practicing medicine in such a time and place.
It was also important to have knowledge of the character’s setting in the actual house and take color scheme into account. The AR device through which the character will be viewed will superimpose the image on the surroundings, so it was important to make sure the agent would stand out from the environment. The buildings at the Genesee Country Village & Museum are from several different decades and span a wide range of architectural styles. Everything from the number of windows in a building to the color trends and financial status of its residents will impact how well the avatar is seen in the setting. For Backus, this meant opting for cool, darker colors so he would be better distinguished amongst the tans, browns, and reds of the well-lit entryway.
We began with a rough sketch to outline the physical properties of the character before moving into 3D development. The 3D digital design program utilizes software from a variety of companies in order for students to experience the full range of programs in use throughout the professional industry. For this project we chose to use software from well-established and reliable companies with the idea that we will be able to upgrade and improve the designs as the software advances. In order to achieve the stylized character we had determined would best suit our needs, we utilized Character Creator by Reallusion, a 3D software that would allow us to morph realistic human proportions. This software utilizes an interface and key strokes that are common in several 3D programs, making it approachable and intuitive for students of 3D art. Facial and body features were exaggerated; the nasal, cheek, and chin areas were expanded to match historic drawings of Azel Backus, along with digitally sculpted hair and sideburns matching historic styles. The head and eyes were enlarged, as seen in many animated characters, to make them less realistic and more childlike. The avatar’s physique and appearance were also altered to better reflect that of an older gentleman of 1851.
We then used Marvelous Designer 8, a digital patterning and simulation software to build period-appropriate clothing for Backus. This software in particular is not only widely embraced by the 3D industry, but is advancing rapidly in its effectiveness and efficiency. As we develop further historic digital avatars for the museum, students will be utilizing this software to create historically accurate garments that are uncommon in the 3D world.
In selecting garments for Backus, as well as any future characters for the museum, it was important to keep in mind that clothing production was not yet industrialized, meaning it was not mass produced nor readily available (Holkeboer 1993; Gorsline 1994; Armstrong 1995; Tortora and Marcketti 2015). Most, if not all, of the garments worn by Backus would have been home or locally produced. Men’s shirts in particular were traditionally made by a wife or mother, but a man’s tailored waistcoat and frock coat would have been made by a skilled, male tailor. Additionally, the materials used would have been relatively expensive, so tailored menswear tended to be an investment that was worn for several years. Considering the remoteness of Rochester to any major metropolitan hub of 1851, it’s likely his garments could have been 5–10 years old at the time. To that end, we opted to dress Backus in a slightly dated frock coat with the soft, sloping shoulders and high back collar of the mid-1840s, and a waistcoat with a wide lapel and only slightly rounded hem of the 1840s. Here, we opted for a deep navy blue melton wool that would be a strong contrast to the wood staircase and tan wallpaper of the home’s entry. His trousers also bear the marks of the 1840s, with the relatively new center front fly closure, as opposed to the earlier fall front. Men’s trousers of this early Victorian era were tapered and narrow at the hem and tended towards large-scale patterns, especially plaids. We opted for a somewhat subdued gray wool plaid flannel as Backus was more an elder statesman than fashionable dandy. Students’ detailed character analysis informs these design decisions, and the design choices inform how the patterning software is used. These decisions, coupled with the research prepared by museum studies students in their development of monologues, inform the 3D students’ design choice, right down to the type of fabric used in a waistcoat and whether or not a collar is top stitched.
Capture
Transitioning from research and design to capture and render meant involving actors from performing arts faculty who, based on their vocal style, could offer a viable presentation of Dr. Frederick F. Backus. In order to preserve the legibility of the narrative in performance, the actor’s voice-over track was recorded in advance, which enabled the actors to adjust inflection and vocal emphasis of segments of the script in a sound-isolated recording booth. The performers then recreated the character’s movements in front of a motion capture system, using the audio playback as reference. Using Character Creator, the motions were then transferred to the avatar and any jitter was removed. Additionally, a digital face rig was created and lip-synched to the narrated audio track recorded earlier, and additional gestures were added as required.
Delivery
The rigged and animated model was exported as an FBX file from Character Creator into the Unity game engine for use in the HoloLens, which would simultaneously display the character in the museum space and create the user experience of interaction with the virtual guide. (A user interacting with Backus in the museum setting and the view seen through the HoloLens are shown in Figures 8 and 9.) Using the speech-recognition capabilities of the device, the application can recognize key spoken phrases to which the character will respond with a predefined and prerecorded monologue.
In this way, the narrative is designed as contextually rich, narrative-driven storytelling delivered by a historical character, set in the home that the historical Backus did, at one time, inhabit. The AR character, house, and site are woven together in a storytelling construct that engenders historical information, situates a conversation between an agent and a visitor at a site that the character may have once visited, and presents an opportunity for incidental learning—the learning along the edges that Falk and Dierking proclaim as critical to the museum visitor experience (Falk and Dierking 2012). In short, the monologue as written, performed in audio and motion capture, and associated with the digital asset, which includes the character creation as well as historical treatment, are tethered and presented through the HoloLens. These facets come together to create an experience that provides an opportunity for visitors to engage with a person from the past that is only possible through this medium.
Informal feedback
Our ultimate goal is to deploy at GCV&M with visitors, wearing HoloLens, who have entered the threshold of the doorway of the Livingston-Backus house at the museum. While we have not yet deployed at the museum, as of November 2019, our team has reached a significant milestone of creating the Backus who can deliver a monologue and respond to voice commands from the user. As of this writing (March 2020), our research and design has continued with four new characters that will be captured in the coming months.
Our progress thus far has been informed by preliminary informal feedback in two phases in 2019, both working toward the goal of user testing this system and content onsite at Genesee Country Village & Museum.3 These two iterations of informal feedback, while very different in design and nature, have offered us the opportunity to see an increase in ease of use and interest, as well as fulfillment. These facets will be measured again as we move into our third iteration of informal feedback, involving students from across the collaboration team, as well as museum staff. We plan to develop and conduct formalized user testing at the museum in the summer of 2020. Each of these feedback scenarios has enabled us to reflect on our work as faculty, and, in coordination with our students, to assess our pedagogical goals and structure our next advancement.
Authenticity and living history museums
The creation of a virtual museum guide may seem at odds with the history and context of our museum partner and our intended location for delivering the XR experience, Genesee Country Village & Museum, which belongs to the classification of living history museums. This genre grew out of world’s fairs and international displays in the 19th century that offered exhibitions arranged in village-like settings to provide viewers with an engaging sense of culture and history simultaneously (Alexander, Alexander, and Decker, 118). Founded in 1966 and open to the public a decade later, GCV&M has sought, from its earliest days, “an endeavor to visualize and interpret this bygone era…[and] has assembled authentic examples—functional buildings and artifacts of the period—from a score of area towns. It has not endeavored to recreate any specific village but to recapture and portray the character and atmosphere of the village era” (McKelvey in Bolger 1985, 2).
Walking through the gates of the toll house and entering into the historic village, visitors are treated to a vision of the past before their eyes. Such a treatment of living history museums affirms folklorist Jay Anderson’s (1985) definition of living history as “the simulation of life in another time.” Museum interpretation at living history museums is often mediated through the costumed interpreters who may take on a particular role, often with the premise that they are conveying what it was like to live in the past, and the modern visitor has encountered them in their daily life (Reid 2001; Roth 2005; Thierer 2010). Because they are the primary communicators with museum visitors, costumed interpreters are essential to the interpretation function of living history museums, which are entirely re-contextualized environments. Interpreters serve as the key factor of on-site engagement for visitors. They communicate with visitors through demonstration and conversation. As theorists Handler and Saxton (1988) argue, living history practitioners are keenly concerned with authenticity and that the role of the interpreter is to bridge past and present.
This connection between past and present while simultaneously seeking authenticity is key to our project which utilizes extended reality as a medium for the dissemination of a first-person narrative keyed to the identity of a known, historical person. These choices were made by the project team so as to distance the digital work and its outlay from the onsite, face-to-face, interpreter-to-visitor experience. In addition, we wanted to push the limits of this medium to see the extent to which our virtual tour guide can convey authenticity even while avoiding the aforementioned uncanny valley.
Whereas traditionally, onsite at the museum, visitors come into contact with costumed interpreters who staff approximately a dozen buildings and engage in third-person dialogue, meaning that they are dressed in historically accurate costume yet use contemporary language and are fully aware of the present, our virtual tour guide offers the opportunity to hear from a character speaking in first-person, performing a role for visitors, and speaking in paraphrases or direct quotes from diaries, notes, and primary sources. Both methods of interpretation—the third-person, interpreter-based and first-person, avatar-based—seek to serve as relevant, authentic, and historically accurate bridges between past and present for visitors.4
Pedagogy
Our project design has been informed by pedagogy, as this project was conceived from the outset as a collaboration among faculty and student researchers across several disciplines. Over the eighteen months of this project, students and faculty have been involved at every phase of our project (see figure 1, Roles and Tasks). Some aspects have been developed within the framework of a course assignment for museum studies students, including research, monologue development, participation in audio and motion capture, and collection of feedback. The early iteration of the virtual museum guide was developed by computer science students enrolled in Applications in Virtual Reality, a course focusing on the use of VR/AR technologies in creating unique mixed reality experiences. And, enhancements of the application have been taken on by several master’s students in computer science as part of their capstone projects. Other facets took place outside of the classroom assignment or context; students self-selected to become involved in that phase of the work. For instance, theatre students were involved as actors for motion and audio capture, a 3D design student facilitated the motion capture as part of advanced study toward her thesis project, and museum studies students facilitated informal feedback in November 2019.
The application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom experience shares attributes of constructivist learning of educational systems in general (Dewey 1998). Specifically, our project—involving students and faculty from three colleges at a research university, along with a museum partner—encourages discourse during knowledge construction. For instance, the collaboration necessary for success of this project provides a unique learning opportunity for computer science students. Though the focus of the work of the computer science students may be technical in nature, the design, implementation, and approach of the application development are shaped by the continual interaction with the creative team. Back and forth communication regarding the assets, both visual and aural, guides the development activities of the application, and, at the same time, directs the work of the design team creating the assets as they must assure proper formatting, timing, and synchronization of the models and animations to work on the HoloLens device.
Faculty have served as mentors to one another and students, but also have let go of prescriptive outcomes for classroom assignments or milestones of our project in an effort to foster experimentation within the context of our collaboration. We have embraced key facets of Pomerantz’s “spirit of experimentation” (2019) which contends that success can be measured by virtue of experimentation rather than meeting criteria on a traditional rubric. As Pomerantz notes, “Sometimes experimentation is the point.” As faculty, too, our learning experiences as collaborators and facilitators guiding our students’ work throughout this project have embraced this facet of experimentation.
Our blending of disciplines to create a blended-reality experience realizes constructivist pedagogy and further mirrors attributes of visitor experiences at museums, where knowledge is actively produced by the learner. For instance, throughout this project, students engage in incidental learning, which may be defined as “unplanned or tacit learning, stemming from the learner’s actions” which is “an often hidden aspect of higher education” (Crawford and Machemer 2008, 106, 109). These attributes are hallmarks of a “learner-centered environment” (104) and are key to understanding the pedagogical outcomes of our project.
Our conception of a virtual museum guide to be developed among a cohort of interdisciplinary researchers and their students intended, from the outset, for incidental learning to occur as a means of individual student work (as an assignment or other framework for involvement in this project). In fact, we found, in review of Crawford and Machemer’s characterization of 19 incidental learning skills associated with project-based learning, that students across the project were developing (and continue to develop) each of these skills at various points throughout the project.5 In addition to particular facets skills gained by particular cohorts of students involved in our project, all students and faculty gained “teamwork skills,” “time management skills,” and “potential to apply what is learned here to other situations” (variables 2, 4, and 19 of Crawford and Machemer). Each of these attributes described above enabled the students to develop skills that were not part of the initial project requirements, indeed, but they also fostered a sense of real-world experience. That is, the workflow and processes defined above—with collaborators having domain knowledge and expertise entering into a project for a particular purpose and then exiting until called upon again—mirrors the work world of industry where various aspects of a large-scale project are completed independently in contribution of a larger whole. Importantly, the undergraduate students across all disciplines expressed an interest in continuing to be updated on the project’s progress, long after their semester or other engagement had come to an end, thereby affirming the pedagogical impact of this project.
While much of our decision-making was informed by pedagogical aims and aspirations for cross-disciplinary learning, we were collectively interested in how XR can inform storytelling practices. Our conception of storytelling based at a living history museum was informed by Bedford’s proclamation of storytelling as a key attribute of museum work (2001) and Lowe’s articulation of who defines stories as the “interpretive tales we craft” and narrative as “the way that we consciously and unconsciously shape those stories” (Lowe 2015, 45). Such a framing of the past impacts the process of meaning making. As David Allison notes, “The way that museums present the history and the prejudices and biases they bring to the design process [of living history interpretation] will affect the meaning that individuals construct for themselves” (2016, 29). Allison thus affirms Lowe’s assertion that particular institutions do a “much better job explaining the complexity of history making—the craft, the methods, and the narrative construction” and sees such places as sites of innovation where leveraging “the old, bad history” (Lowe 2015, 47, 52) can—through storytelling—foster multivocality and inclusive interpretations of the past. Such museum-focused outcomes cross over to our pedagogical aims of storytelling and our project’s framing by affirming the value, relevance, and importance of storytelling as a form of historical communication, bridging between past and present as well as opportunities for authenticity, empathy, and inclusion.
