Fostering Engagement and Learning Through Multimodal Pedagogy

By Francisco Medina

Introduction

Since March 2020 many of us have had to rediscover, abandon, and transform our pedagogy to adjust to teaching during an unprecedented pandemic. From technical issues on Zoom, inconsistent internet connection to not being able to see students on screen and navigating the health, emotional, psychological, sociopolitical, and economic impacts of Covid19, remote teaching and learning proved challenging. For many instructors, what worked in the face-to-face classroom no longer seemed to stimulate discussions or participation online. While the break-out room feature on Zoom substituted in-person group discussions, it was still difficult to navigate classroom discussions that involved the whole class. Similarly, many instructors had to re-design assignments that required students to work in groups, make observations, visit a museum, interview a person, or even change the format of exams. For this reason, many of us had to figure out new ways of fostering engagement and participation in the online classroom through transforming our pedagogy and assignments.

For many educators, engaging students in the online classroom during a pandemic, whether through written assignments or Zoom discussions, was ineffective and furthered the alienation of students. For example, the first few weeks in my introduction to psychology course at one of CUNY’s community colleges, it was difficult to center my pedagogy around my students’ needs. With over thirty students, many of whom attended class while parenting or taking care of younger siblings— participation, engagement, and learning suffered significantly. Students rarely spoke on the microphone, and those who shared on the chat chose to message me privately rather than the entire class. Lessons and in-class assignments that normally stimulated discussion in the face-to-face classroom, were met with silence on Zoom. Some students attended class while at work, some logged in from their cellphones, many had technological problems, and some did not feel comfortable with this new form of learning. As the days went by, the lack of engagement and participation took a toll on the course, as it often felt like I was speaking to myself since all the students had their cameras off.

For these reasons, I had to find new ways to foster engagement, participation, and accessibility in the context of the synchronous online classroom. Pre-pandemic, my teaching involved group discussions, classroom community-building activities such as scavenger hunts and trivia games based on course materials, and watching documentaries, among others. In this sense, my pedagogy has always involved multiple forms of engagement, however, none of these were working on Zoom. As I began to talk to colleagues and read online blogs about what other educators were doing in their classroom, I came across multimodal pedagogy. This approach focuses on using a combination of multiple modalities, such as visual, auditory, and spatial representations to teach, learn, and design assignments. It enabled me to encourage creativity beyond essay writing and/or exams as well as to engage students who had their cameras off on Zoom

In what follows, I use the final creative project in my course as a departing point to brainstorm what multimodality might look like in the classroom and to reflect on how to counter its challenges. I explore how multimodal assignments enable accessibility and multiple forms of engagement that increased student participation and discuss the challenges and limitations that might arise when making one’s pedagogy multimodal. Multimodal assignments can help faculty and students develop, engage, and reflect on academic work more deeply, diversify the production of knowledge, and make learning more personally meaningfully.

What is multimodal pedagogy?

Broadly speaking, pedagogy is conceptualized as a face-to-face and text-based activity in the social sciences. Research papers, oral presentations, multiple choice exams, written reflections, textbooks, academic articles, and PowerPoints are typical in most classrooms. Multimodal pedagogy seeks to disrupt this by crossing the boundaries between linguistic, written, and visual content both during teaching and when designing assignments. It offers the opportunity to experiment and play with kinesthetic, visual, audio, and other creative forms (e.g., fiction) as a way to move beyond traditional forms of pedagogy and to open up a space for students to tap into their existing and/or developing skills and talents (Stein, 2008; Archer, 2017). Students create their own meanings of what they have learned in class by mixing and assembling modes that are not normally valued in the classroom, such as using multiple languages in their writing, or by communicating through gestures, music, dance, images, fashion, and even jewelry-making (Archer & Newfield, 2014). Multimodal pedagogy emphasizes the combination of modes to give meaning to learning rather than simply “substituting” text with visual art or kinesthetics. Multimodality therefore expands the terrain of teaching and learning outside of the constraints of academic writing by enabling educators and students to not only recognize and bring their talents and interests into the classroom, but by basing learning “on what could be useful, effective and relevant to students’ needs and interests” (Ladeem & Mallahi, 2019, P. 33).

Since the manner in which “something is represented shapes both what is to be learned, that is, the curriculum content, and how it is to be learned” (Jewitt, 2008, P.241), multimodality is not simply about “adding” art or music to one’s classroom. Rather, multimodality encompasses modal affordances— that is, each mode transforms “what it is possible to express and represent” thus generating specific logics and constructing meaning in particular ways (ibid, P. 247). For example, music not only engages other senses, but expresses meaning in a way that is not possible through writing or painting. Multimodality transforms learning and how it is expressed.

Many educators use multimodal pedagogy in their classrooms without realizing it. For instance, I often incorporate videos, podcasts, images, and trivia games in my face-to-face lectures. However, I am now more intentional about it. Whereas before I unknowingly was using multimodality in my lectures, during the pandemic I began to design assignments and classroom activities that supported various modes and encouraged student participation.

A case study of multimodality in the classroom

Introductory psychology courses (psychology 101) at CUNY’s community colleges are notoriously challenging for both faculty and students. For one, many majors such as nursing, business, and liberal arts require students to take introduction to psychology, and it also functions as corner stone for psychology majors. This course usually consists of a large number of students which can range from campus to campus, though in my experience, anywhere from 30 to 40 students at the community colleges I’ve worked at. In addition, introduction to psychology courses cover theoretical topics such as the mind and behavior, biology, history, ethical concerns, research methods, and mental health—which means that students are required to learn a large amount of information in a short period of time. For these reasons, introduction to psychology is considered a “high risk” course where the majority of students earn a C grade. Teaching this course online during the pandemic amplified the pre-existing anxieties and challenges that come with teaching psychology 101. Moreover, the broader issues of teaching online that faculty from all disciplines experienced, such as Zoom fatigue, technical issues, and students not turning on their cameras, made the creation of multimodal participation and assignments necessary.

Fostering engagement with multimodal participation

Multimodality helped redefine participation in my classroom into one that was more appropriate for online teaching. Initially, many students were shy to speak on Zoom, turn on their cameras, and even hesitated to share their thoughts on the chat publicly. I tried to involve students by posing open-ended questions, asking students to share their opinions, or even asking if anyone remembered the definition of a term we had discussed in the previous lecture. Nonetheless, it felt like no matter what I did, it was always the same two students who typed in the chat. By this point, we were three weeks into the semester and only one student had spoken on the microphone. In the hopes of stimulating any sort of engagement, I started to use Zoom’s built-in survey to practice exam questions, survey students’ prior knowledge about a given subject (e.g. asking True or False questions at the beginning of lecture), and to make democratic decisions about extending deadlines. I noticed that nearly all the students engaged with the surveys, however, when I tried to get them to expand on their answers either through chat or the microphone, I was once again met with silence. I realized that many of them seem to appreciate the anonymity of surveys over the chat function.

Shortly, I began to use Mentimeter – a live-online platform I had used before at conferences to survey the audience. I used the word cloud and speech bubble feature on Mentimeter to pose open ended questions to students, ranging from general thoughts about the readings, what was missing or being taken for granted by the author, and any question they might have. The anonymity of the application had a significant effect, with many students submitting their thoughts in contrast to the 2-3 students who usually shared in the chat or via microphone. Quickly, students began to not only share their own opinions, but to respond to their classmates’ bubbles and engage in meaningful conversations. I then used these conversations to guide my lecture, and some students even began to use Mentimeter to ask me to repeat or clarify a point. Over time, students started to speak on the microphone and use the chat feature on Zoom to follow up on what they or someone else had written on Mentimeter. Using Mentimeter as a mode of participation seemed to ease students’ comfort with Zoom and each other. This was something that became evident toward the end of the semester when more students began to use the microphone in addition to the increased participation on the chat.

In addition to Mentimeter, Padlet can help encourage students’ participation in the online/hybrid classroom. It enables students to post images, text, audio, and videos on a live board that updates immediately and continuously. Students can comment on each other’s posts and the professor can share it on screen, upload materials, post new questions, etc. during class time. While I did not use Padlet in my course, several of the graduate courses I took during the pandemic did. As a student, I appreciated the opportunity to engage in courses through posting a photograph of my neighborhood, a voice recording, or a meme on Padlet instead of the typical classroom discussion. Toward the end of the semester, like many students and faculty, I experienced Zoom fatigue, however, I was still able to contribute to course discussions and dialogue with my classmates through Padlet. In contrast, I (and many of my classmates) struggled to participate in courses that only focused on Zoom discussions during the end of semester. At the very least, Padlet offered an alternative form of engagement and a “break’ from Zoom.

Creating a multimodal final assignment

Typically, introduction to psychology courses assigned a research paper or a final exam at the end of the semester. As I continued to learn about multimodal pedagogy, I began to consider not only how many of my students struggled with writing, but how writing might be limiting what students were able to express. For example, when I asked students to write reflections, they often regurgitated what was in the textbook or what we’d covered during lecture, with few students stating their opinions or making connections to their personal life. However, when I spoke to students during office hours or class discussions, it became clear that they were making meaningful connections between their personal lives and the material. I started to wonder if text-based assignments were perhaps limiting, and after seeing a friend’s zine-making final project, I was inspired to create a multimodal, final creative assignment.

For my class, I designed a final creative project where I posed questions to students with regard to topics covered in class (see figure 1). I encouraged them to answer the questions in a way that was personally meaningful while also using their talents, skills, and interests to create the project. For example, I invited students to move beyond an essay and to write poetry, fictional stories, draw comic books, make TikTok videos, Instagram infographics, posters, zines, interpretive dance, fake newspaper articles, workout routines, and any other mode that they were interested in. In order to facilitate this, I asked students to write a proposal and to meet with me one-on-one to discuss what form their project would take and brainstorm ideas.

Notably, some students expressed anxiety and discomfort with doing something artistic. Some explained that they felt they did not have a talent for drawing or painting and therefore they wanted to stick to an essay. Others said that the idea made them nervous because they were unsure of how to do a creative project. I think part of this discomfort had to do with students not being used to creating multimodal or artistic projects in their courses, or even perhaps writing being seen as “more professional.” I also think that some students were concerned about their grades if they were to make something artistic.

Although many students chose to write an essay, several students in the class employed multimodality for their final assignment. For example, a student submitted a poetry book that highlighted the lack of discussions about the emotional experience of dealing with mental health in the textbook paired with artworks. Another student created a comic book about operant condition and the Id, Ego, and Superego where these were represented as curses that haunted a person and a one-page essay explaining how he’d come up with the idea. Many students opted to make zines and posters where they analyzed mental health representation in the media or sought to inform their communities about the stigmatization of mental health through a mixture of images and text. A student created a script and a video in TikTok format with captions and word clouds that appeared on screen as he spoke and pointed. All students also gave a short oral presentation about their project.

The benefits of implementing multimodal pedagogy

Students who participated in the multimodal approach seemed to generally make sense of the course material in a way I had not seen in previous face-to-face courses. It was inspiring to see a student represent psychological concepts as entities that cursed people in a comic book. Students embodied their learning in videos through gesturing, facial expressions, and the tone of their voices. The course material traveled differently through different modes. As an instructor, I was not only surprised by this, but I realized the impact that the course was having on students and how it transformed their views about mental health, behavior, and emotions. Often, essays tend to be serious in tone and rely heavily on citations (e.g., what experts have said about a given topic), whereas the multimodal projects enabled students to be more playful, creative, and make connections to their experiences. While some students who wrote essays made connections to their personal lives or broader issues, they stuck to the source material for the most part. A lot of them summarized or did a literature review of a given topic while their points of views, learning, and reflectivity were underdeveloped in their writing. Those who made a multimodal project, not only made connections to society and their personal lives, but their understanding of the material itself appeared to be deeper, and more insightful. This was evident in how they approached the topics, such as discussing what the textbook didn’t cover about mental health (e.g., how does it feel to live with a mental illness?), offered creative outreach to communities (e.g., how can films be used to educate the public?), and making explicit connections between their majors, goals, and psychology (e.g., how can a nurse help patients with mental health? How to educate parents about the psychology of children?).

Research studies have also found that multimodality benefits students’ academic writing (Archer, 2017), improves English language teaching (Ladeem & Mallahi, 2019), facilitates social interactions in the classroom (Taylor & Leung, 2019), and nurtures students’ communication skills, cultural learning, translation, video editing, and multiliteracies more generally (Yeh, 2018). In addition, multimodality challenges students to engage more deeply with materials, reflect on their learning, and make meaning. Multimodal assignments also invite students to use their talents and skills while affording them different forms of learning that are not necessarily possible with an essay or research paper.

Moreover, the multimodal projects challenged my own assumptions and understanding as an instructor. Through the works of students, I reflected on my own teaching and how I could be more creative with my pedagogy, how to use metaphors to explain complex theories, or use films and music in the future to talk about psychological concepts. It was exciting to feel and see the course material being reflected upon and explained through other mediums that I personally would have never considered before. To this extent, multimodality was not only beneficial to students, but also to me as the instructor.

Challenges of using multimodality and how to overcome them

One of the challenges of multimodality is grading equity. How do we grade a video in comparison to a comic book? How do we grade these in contrast to an essay? How much time might someone need to revise a video vs an essay? Because each modality is unique and offers different affordances, it is crucial to have a plan in place and scaffold the project throughout the semester (e.g., submitting a proposal, script, etc). In my class, I developed a grading rubric that focused on having a thesis, drawing on course materials and at least 2 scholarly sources, demonstrating reflexivity, and providing accurate information. I also developed equivalents across modes, so students would have a similar amount of work. For example, a 4-page essay was equated to a two-minute video, a 6-page zine, and a 12-panel comic book (see figure 2). We discussed this as a class to make sure everyone felt these were reasonable expectations and fair for all modes.

Other ethical considerations included:

• Being mindful of the added labor some modalities might require, such as hours of video editing or redrawing an entire scene. In addition, instructors might have to develop rubrics for different modes, figure out how to support multiple modes, and provide resources for students which adds more time and labor to design the course.
• Ensuring that whatever the medium, students meet the criteria for the assignment (e.g., having to cite two scholarly sources, a thesis, etc.)
• Reflecting on how asking students to develop multimodal assignments might highlight inequality. For example, some students might have access to video editing software and others do not. How can you ensure that all students have access to materials? On the other hand, students can choose to work with the materials they have access to.
• Prioritizing and providing free resources for students to carry out their multimodal projects. In my class, I provided students with a list of free online programs and software, such as Canva to make posters and infographics, free photo editing websites, and encouraged them to use social media they might already be using (e.g., TikTok) to make and edit videos.
• It is also important to ensure that these free resources can be accessed by mobile and not just computers since there might be students who don’t have access to a desktop. Moreover, these resources should also be accessible, such as being text-to-speech friendly or providing a warning if there are flashing lights on a website that might trigger epilepsy.

Depending on the course content, it might be necessary to discuss the ethical components of student projects. In my case, I taught an introductory psychology course that discussed issues of mental health and many students chose to base their project on mental disorders. For this reason, not only was it important that the information they utilized was accurate, but that they did not share their project on social media. Specifically, we discussed that none of us were mental health providers or licensed therapists, and I therefore strongly discouraged students from posting their videos publicly on TikTok or sharing their infographics/posters on Instagram because someone could misinterpret their work as mental health advice. This issue might not apply to all disciplines and some instructors might even want to encourage students to share their work publicly, but those of us who are in psychology, medicine, social work, and other applied fields have to ensure that students understand the ethical responsibilities and possible repercussions if they are to share their projects publicly.

Conclusion

Multimodality has gained traction in academia and education in the last several decades. As our world becomes increasingly digital and researchers are encouraged to engage with the public, multimodality prepares students to communicate and disseminate complex information to a wider audience in a way that is accessible. Some academic journals, such as the Journal of Multimodal Rhetoric, have even begun to accept multimodal submissions to acknowledge both the need to expand our conceptions of academic writing and to experiment with different modes (see Jones & Woglom (2013) for an example of a comic book based on research findings; see Andreotti (2016) for an example of a multimodal theoretical paper).

In the classroom, multimodality offers the possibility of experimenting and implementing various modes that can foster engagement, participation, and creativity. Videos, paintings, poetry, posters, and music, amongst others, can expand and transform students’ meaning making, including constructing knowledge in ways that differ from what is possible through essay writing. It invites students to tap into their existing skills, talents, and abilities as well as to reflect on what they have learned by “transforming” it into other modes.

Given the challenges that come with implementing multimodal pedagogy, including issues of grading equity and accessibility to resources, instructors interested in using a multimodal approach should reflect on how multimodal pedagogy might impact their classroom. How might multimodal pedagogy and assignments work or not work in your discipline? How might it change your current teaching and syllabus? And as we return to face-to-face, how does multimodality translate from online to in-person?

Multimodal pedagogy can add discomfort, additional labor, and other unforeseeable challenges, yet it is also a rewarding experience for both instructors and students. It allows for experimentation, playfulness, collaboration, and creativity beyond the constraints of text-based assignments and more traditional approaches to pedagogy. Multimodal pedagogy can be an opportunity to make the classroom more accessible, to encourage students to develop their own voices and re-interpret what they have learned via other mediums, and for instructors to expand our teaching repertoire.

References

Andreotti, V. (2016). (Re) imagining education as an un-coercive re-arrangement of desires. Other Education, 5(1), 79-88.

Archer, A., & Newfield, D. (2014). Challenges and opportunities of multimodal approaches to education in South Africa. In Multimodal Approaches to Research and Pedagogy (pp. 19-34). Routledge.

Archer, A. (2017). Using multimodal pedagogies in writing centers to improve student writing. Stellenbosch Papers in Linguistics Plus, 53, 1-12.

Jewitt, C. (2008). Multimodality and literacy in school classrooms. Review of research in education, 32(1), 241-267.

Jones, S., & Woglom, J. F. (2013). Teaching bodies in place. Teachers College Record, 115(8), 1-29.

Laadem, M., & Mallahi, H. (2019). Multimodal Pedagogies in Teaching English for Specific Purposes in Higher Education: Perceptions, Challenges and Strategies. International Journal on Studies in Education, 1(1), 33-38.

Stein, P. (2008). Multimodal pedagogies in diverse classrooms: Representation, rights and resources. London and New York: Routledge.

Taylor, S. V., & Leung, C. B. (2020). Multimodal literacy and social interaction: Young children’s literacy learning. Early Childhood Education Journal, 48(1), 1-10.

Yeh, H. C. (2018). Exploring the perceived benefits of the process of multimodal video making in developing multiliteracies. Language Learning & Technology, 22(2), 28-37.