Introduction
The foundation of modern psychology is the scientific study of behavior and mind. As such, the ability to utilize statistics to interpret data and evaluate claims is a fundamental skill for undergraduate psychology students to master (Lutsky 2006). The American Psychological Association (APA) Guidelines for the Undergraduate Psychology Major emphasize statistical literacy as a necessary component of psychology coursework at all levels (APA 2023). To meet the guidelines, instructors need to create lessons that help students understand and communicate research findings, calculate basic statistics, manipulate and visualize datasets, and evaluate information presented in varied formats, including graphs and tables. Creating opportunities for students to engage in quantitative reasoning may help students distinguish evidence-based claims from subjective opinions and pseudoscience (Johannssen et al. 2021; Lilienfeld et al. 2009). This is of considerable importance given the prevalence of misleading and deliberately false information shared via the internet and other media (Ridgway et al. 2017). Quantitative reasoning is also valuable in shaping students’ career paths as well. According to the National Association for Colleges and Employers (NACE 2021), critical thinking abilities, such as solving problems and accurately summarizing and interpreting data, are essential skills for college students entering the job market. Also listed among the NACE competencies is the ability to use technology (e.g. software, computers) effectively to complete specific tasks and improve efficiency and productivity.
Research on teaching science, technology, engineering, and mathematics (STEM) subjects, including psychology, indicates learning outcomes improve when students are engaged in “hands-on” active learning as opposed to traditional lecture-based instruction (Freeman et al. 2014). Active learning provides students with opportunities to engage in authentic and meaningful problem solving and discovery; it is most effective when learning is scaffolded through demonstrations, hands-on step-by-step tutorials, and feedback (Alfieri et al. 2011). In prior work, Screencast video tutorials, which are screen recording technologies, were used in upper-level psychology statistics courses to teach students how to analyze data using Statistical Product and Service Solutions (SPSS) software (Breneiser et al. 2018; Lloyd and Robertson 2012). Results indicated that SPSS recorded tutorials were effective in supporting students’ grasp of statistics across both in-person and online course formats. Despite these positive findings, SPSS is an expensive software and may not be as accessible to students compared to other software tools, such as Excel. Further, this prior work focused on upper-level courses, whereas efforts should focus on teaching statistical literacy in introductory courses in order to enhance access to practical skills early on in the students’ academic career. Therefore, we sought to expand this approach by implementing similar activities in an introductory-level psychology course, while also increasing access to relevant technology by using Excel. Introductory Psychology (also known as PSY100) is a general education course taken by approximately 1.6 million students annually in the United States (Gurung et al. 2016), including many who do not pursue additional coursework in psychology. This course is thus an ideal setting for developing quantitative reasoning skills not just for psychology majors, but for students pursuing a broad range of majors and future careers.
Brooks et al. (2024) provided an initial report on our department-level efforts to promote quantitative reasoning, interpretation of research findings, data analysis, and visualization skills in PSY100. Students enrolled at a non-selective minority-serving public college were introduced to statistical literacy concepts through a series of homework assignments created with Qualtrics survey software. In a subset of the assignments, students were shown how to engage with psychology-relevant data using Microsoft Excel software. The instructional team used Excel rather than SPSS or R because students have institutional access to this application and Excel promotes a broader application of skills beyond the classroom. Software used to introduce statistical literacy should be both affordable and user-friendly (Adams et al. 2013). Robust statistics software applications like SPSS are user-friendly but also prohibitively expensive, and while R is a free open-source alternative, it requires a steep learning curve for developing coding skills, rendering it inappropriate for PSY100. Excel strikes a balance, as it is both fairly inexpensive and relatively easy to learn (Ozgur et al. 2015). In the present implementation, Microsoft Office 365 (including Excel) was made available to all students at no cost through a university subscription. We introduced Excel as a tool for data manipulation, analysis, and visualization through a series of instructor-guided screencast videos, also known as content-acquisition podcasts (CAPs) (Kennedy et al. 2016), that demonstrated use of Excel with psychology-related datasets. Students were also given instructions on how to download Microsoft 365 at no charge by accessing the university subscription.
In Brooks et al. (2024), PSY100 students’ engagement with statistical literacy lessons was positively associated with their statistics knowledge, which was assessed using items adapted from publicly available AP Statistics practice tests and the New York Times “What's Going On in This Graph?” column (New York Times in partnership with the American Statistical Association, The Learning Network 2021), statistics anxiety, which was assessed through items adapted from the statistics anxiety scale (STARS) (Teman 2013), and self-efficacy for online learning (measured at the start of the semester). While statistics knowledge predicted completion of the Excel worksheets, statistics anxiety and self-efficacy predicted homework completion in general. That is, students with greater statistics anxiety and higher self-efficacy for online learning submitted more of the Qualtrics assignments than their classmates who were less anxious and/or had lower self-efficacy. These relevant factors have been noted elsewhere in the literature (Carpenter and Kirk 2017; Foley et al. 2017). Faculty in the U.S. often express concerns about the mathematical readiness of first-year college students and the need for remediation (Er 2018). Such concerns have been amplified since the COVID-19 pandemic, with widespread reports of learning losses in mathematics as well as reading (Toness and Lurye 2022). Students enter college with varied levels of statistical knowledge. Correct understandings coexist with misconceptions and gaps in knowledge (Cook and Fukawa-Connelly 2016). We assert that such variation in statistical knowledge may play a role in their willingness to attempt statistics-related assignments and their performance. Zeidner (1991) defines statistics anxiety as a type of performance anxiety characterized by physiological arousal and negative emotions (e.g. worry) when presented with statistical content. Statistics anxiety, and math anxiety more generally, may exert negative influences on students’ academic achievement and motivation (Chang and Beilock 2016; Pekrun et al. 2017), though not all studies have found this link (Brooks et al. 2024; Trassi et al. 2022). Similarly, several studies have found positive impacts of self-efficacy on course outcomes and student learning in general (Alqurashi 2016; Gurung and Stone 2023; Tsai et al. 2011), though not all studies have found this relation (Gravelle et al. 2024). Given the mixed findings, the present study examined links between prior knowledge of statistical concepts, statistics anxiety, self-efficacy, and learning outcomes.
Besides internal factors such as statistical knowledge, statistics anxiety, and self-efficacy, other barriers to student learning include technological difficulties and limited access to computers for schoolwork (El Mansour and Mupinga 2007; Gravelle et al. 2024). In prior research (Powers et al. 2016), PSY100 students reported challenges in accessing and using instructional software in a hybrid course with an online mode of instruction, despite instructors dedicating considerable class time to helping students set up accounts and troubleshoot difficulties. Such findings suggest that college students may lack familiarity or savvy with software installation and licensing, and may become frustrated when difficulties arise. In another study conducted at our institution (Gravelle et al. 2024), approximately 9% of PSY100 students reported using hand-held devices (i.e., smartphones or tablets) to complete online coursework rather than desktop or laptop computers. In that study, use of hand-held devices for schoolwork was associated with lower rates of passing the course, fewer assignments submitted, and lower test scores. Similarly, Brooks et al. (2024) found that students’ self-reported use of hand-held devices to complete assignments was associated with a lower rate of submitting the Excel worksheets.
Motivating students to engage with demanding coursework is an ongoing concern in higher education, especially at non-selective colleges serving diverse populations. While such institutions provide broad access to higher education, they are subject to high rates of course attrition, chronic absenteeism, low graduation rates, and declining enrollments (National Student Clearinghouse Research Center 2022, 2024), likely related to multiple social and economic factors (Porchea et al. 2010; Zengilowski et al 2023). Many high school graduates question the value of a college education given its high cost (Belkin 2021; Tough 2023) and they may continue to feel ambivalence even if they choose to enroll. While these issues predate the COVID-19 pandemic, the trends appear to have worsened in recent years (Bulman and Fairlie 2022). To address the high attrition and declining enrollments at community colleges and other non-selective institutions, academic faculty and departments need to work together to develop flexible curricula that support learning in ways that are relevant to students’ career goals and broader interests. Students benefit from academic supports that promote engagement through skills development, feedback, and use of technology (Collins et al. 2022). More generally, instructors can support students by helping them develop transferable skills such as statistical literacy in PSY100 to support critical thinking and technological skills that extend beyond the field of psychology and are useful for the workplace and everyday life. Instructors can also support students by adopting free Open Educational Resources (OERs) and institutionally accessed technology to reduce financial barriers that affect many students enrolled at non-selective colleges, open-admissions institutions, etc. (Colvard et al. 2018).
The Present Study
The present study focused on supporting statistical literacy of PSY100 students at a non-selective minority-serving public college, with the aim addressing a noted limitation of our prior research. A major limitation of Brooks et al. (2024) was that it focused exclusively on work submitted without any objective measures of students’ understanding of the psychological and statistical concepts embedded in the lessons. Hence, the present study examined students’ comprehension of course materials via intermittent quizzes to gain further insight into students’ grasp of course content as they progressed through the curriculum. One set of quizzes focused on content taken from Noba Introductory Psychology modules, an open-access textbook available free of charge to students (Diener Education Foundation n.d.). A second set of quizzes provided a formative assessment of students’ comprehension of statistical and general psychology vocabulary terms embedded in the online homework assignments. Taken together, the two sets of quizzes allowed us to examine students’ comprehension of course material and ascertain whether the curriculum and instructional materials were appropriate for general education students. In an effort to identify potential barriers to student success and learning outcomes, we examined students’ prior statistical knowledge, statistics anxiety, self-efficacy for online learning, software installation and access, and devices used as predictors of learning outcomes. Based on prior literature, we hypothesized that each of these factors would be related to students’ quiz scores and rates of submission of assignments.
Method
Course Section and Student Characteristics
Course outcomes assessment data were collected from undergraduates enrolled in 25 course sections of PSY100 taught in Fall 2023 and Spring 2024 at a non-selective (open admissions) minority-serving public college in the Northeastern United States. Course sections were taught by different instructors following a uniform syllabus with instructional materials posted to a learning management system. The research protocol was classified as exempt by the university’s Institutional Review Board. Following best practices for open science, the following instructional materials are publicly available in an Open Science Framework (OSF) repository (Zapparrata et al. 2025): course syllabus, Noba quizzes, online Qualtrics assignments, and Excel worksheet activities. Additionally, the OSF repository includes a de-identified datafile and the R analysis script.
The course sections comprised 20 regular sections with < 50 students and 5 large sections with > 135 students. Eighteen sections met in person (15 regular, 3 large enrollment) and 7 sections met synchronously online via Zoom (5 regular, 2 large enrollment). The analytic sample (N = 1061) consisted of students who submitted the first homework assignment (HW1) and at least one other assignment, thus demonstrating a minimum level of engagement with the asynchronous online components of the course; see Table 1 for a breakdown of demographic characteristics as self-reported by students in HW1. The analytic sample comprised 69.7% of the enrolled students (N = 1523), i.e., roughly 30% of students did not turn in HW1 and at least one other assignment. Of the 1061 students in the analytic sample, 866 (81.6%) completed the course with a passing grade (C or higher), and 195 (18.4%) received D/F grades or withdrew from the course (DFW). Of the 462 students who were excluded from the analytic sample, 440 (95.2%) did not turn in HW1 and 22 (4.8%) turned in HW1 but no other homework assignment. Most of these students (77.3%, n = 357) received DFW grades, while 105 students (22.7%) passed the course. No further information about these students is available.
All courses followed a uniform syllabus and used the same instructional materials. In addition to the asynchronous online assignments that are the focus of this report, students completed three in-class projects as group work, either in person or on Zoom for the online synchronous course sections. The project assignments comprised oral presentations on psychological disorders (Schwartz et al. 2017), a role-play ethics activity (Rose et al. 2022), and poster presentations about hidden figures in psychology (Hurley 2021). These project assignments will not be discussed further in this report.
| Student Characteristics | Descriptive Statistics |
|---|---|
| Mean (SD), Range | |
| Age | 19.6 years (3.7), 16–45 |
| Frequency (%) | |
| Gender | |
| Female | 627 (59.1%) |
| Male | 413 (38.9%) |
| Another Gender Identity/Prefer to Self-describe | 9 (0.8%) |
| Prefer not to Respond | 12 (1.1%) |
| Race/ethnicity (not mutually exclusive) | |
| White | 408 (38.5%) |
| Latinx, Chicanx, Hispanic, or Spanish origin | 310 (29.2%) |
| Black/African American | 189 (17.8%) |
| Asian/Asian American | 119 (11.2%) |
| Middle Eastern/North African | 95 (9.0%) |
| American Indian/Alaska Native | 13 (1.2%) |
| Native Hawaiian/Other Pacific Islander | 5 (0.5%) |
| Some other race | 23 (2.2%) |
| Prefer not to say/Unknown | 50 (4.7%) |
| Either parent attended college (Yes = 1) | 553 (52.1%) |
Online Asynchronous Assignments
Noba textbook quizzes
Students were assigned 25 low-stakes quizzes, each comprising four multiple-choice questions taken from one of the learning modules included in the Noba Introductory Psychology collection (Diener Education Foundation n.d.). Students were allowed to use the Noba textbook (open-book) and could take each quiz up to three times, with the highest score out of the three attempts retained. In creating the test bank, we selected twelve questions per learning module, with four quiz questions randomly selected from the test bank on each attempt. We calculated a single quiz score for each student, indicating the percentage correct out of 100 questions (M = 85.8%, SD = 20.7%, Range = 0–100). Note here that just ten students (0.9% of the analytic sample) failed to complete any of the textbook quizzes and received scores of 0.
Qualtrics homework assignments
Online homework assignments were created using Qualtrics survey software by a team of course instructors, with links to each assignment posted to the learning management system. The homework assignments introduced students to scientific abstracts, database searches for psychology-relevant empirical literature through the college library website, how to interpret research questions and results from research in psychology, TED talks by prominent psychologists in the field (e.g. Carol Dweck, Dan Gilbert), and quantitative reasoning problems developed by the APA Presidential Task Force on Introductory Psychology Statistical Literacy (Neufeld et al. 2022). Students completed 83.3% of the nine Qualtrics assignments (M = 7.5, SD = 1.9, Range = 2–9).
Excel worksheet activities and content acquisition podcasts
Six of the nine HWs included screen-recorded demonstrations, also known as content acquisition podcasts (CAPs) (Kennedy et al. 2016), on how to use Excel to visualize and analyze psychology-relevant data. The CAPs were developed by the instructional team using Zoom for screen recording. Each Excel CAP provided step-by-step instruction on how to compute relevant summary statistics, such as means and standard deviations, create data visualizations, such as scatter plots and bar charts, and construct contingency tables to compare binary count data. The CAPs included subtitles that aligned with the verbal instructions to meet Universal Design standards (CAST, n.d.). For each of the six Excel data analysis and visualization activities, worksheets were prepared using datasets assembled by the instructional team or a publicly available dataset (Emerging Adulthood Measured at Multiple Institutions 2 [EAMMI-2]; Grahe et al. 2018). The datasets featured student responses to psychology-related questionnaires (e.g. Big 5 personality traits, compulsive internet use, mate preferences, sleep habits).
Students were required to upload their completed worksheets to the learning management system. All CAPs were also made available via the learning management system so that students could complete the Excel worksheet activities at a separate time, i.e., before or after completing the associated Qualtrics homework assignment. Table 2 provides a description for each of the six Excel worksheet activities, along with a screenshot of a completed assignment and links to the CAPs. Ideally, students’ final uploaded spreadsheets would match the examples provided in Table 2, with partial credit awarded for incomplete work. On average, students completed 61.7% of the six Excel worksheets (M = 3.7, SD = 2.3, Range = 0–6). In total, 188 students (17.8% of the analytic sample) failed to complete any Excel work.
| Statistical concepts | Screenshot of completed Excel work |
|---|---|
| 1 . Calculating Pearson correlations between scale scores for Big 5 personality traits and compulsive internet use; visualizing correlational data using a scatterplot with linear trend line.
Part 1: Visualize correlational data. Part 2: Calculate correlations between variables. | |
| 2. Calculating category counts for mate preference traits (Buss 1990); visualizing the frequencies of preferred traits reported by men and women in a bar graph.
Part 1: Calculate sum count Part 2: Calculate percentages using fill down Part 3: Visualize data in a bar chart | |
| 3. Calculating Odds Ratios to investigate whether there is a higher likelihood of crime occurring in community lots with no “greening” efforts (Heinze et al. 2018).
Part 1: Recoding binary variables Part 2: Tabulating contingencies using pivot tables Part 3: Calculating Odds Ratios using formulas | |
| 4. Reverse scoring data from a Likert scale on growth mindset (Dweck 2006) and calculating averages for each respondent in the dataset.
Reverse score a scale and calculate averages | |
| 5. Using pivot tables to organize data from students’ self-reported sleep habits (measure adapted from the Pittsburgh Sleep Quality Index, Buysse et al. 1989); visualizing a normal distribution of average hours of sleep
Part 1: Calculating frequencies using pivot tables Part 2: Visualizing properties of the normal distribution | |
| 6. Using independent samples t-tests to compare groups on their attitudes towards risk avoidance behaviors as a marker of emerging adulthood (Grahe et al. 2018).
Part 1: Comparing male and female respondents using a t-test Part 2: Comparing conservative and liberal respondents using a t-test |
Embedded vocabulary quizzes
Five multiple-choice questions were added to each Qualtrics assignment as a formative assessment of content knowledge. There were forty-five questions in total—twenty-eight questions assessing knowledge of general psychology terminology and concepts and seventeen questions assessing knowledge of statistical terminology and concepts. Examples of general psychology items are: What stage of life is characterized by a feeling in-betweenness, where people do not feel like children but don’t necessarily feel like adults either? And Istvan is well liked by his friends because he is trusting, considerate, compassionate, loyal, and easy to get along with. Which of the Big 5 factors of personality would Istvan likely score high in? Examples of statistics items are: A statistical measure that indicates the extent to which two or more variables are related is called __________. Another example is: What does it mean when the p-value of a t-test comparing two groups is LESS than .05?
Students attempted an average of 37.3 (SD = 9.7, Range = 10–45) vocabulary questions (general psychology items: M = 23.8, SD = 5.5, Range = 7–28; statistics items: M = 13.5, SD = 4.5, Range = 1–17). Performance (accuracy) was scored by taking the proportion of correct answers out of the total number of questions attempted. We excluded skipped quizzes so that our measure of comprehension accuracy would be independent of the number of Qualtrics homework assignments completed. Accuracy was similar across item types (general psychology items: M = 85.7%, SD = 16.8%, Range = 21.5%–100%; statistics items: M = 86.8%, SD = 19.3%, Range = 0%–100%).1 Accuracy across general psychology and the statistics vocabulary items was strongly positively correlated, r(1059) = .77, p < .001. Therefore, we combined both item types to create composite scores for the embedded vocabulary quizzes (M = 86.0%, SD = 16.7%, Range = 22.1%–100%).
Additional Measures
Demographic questionnaire
HW1 contained a demographic questionnaire and other measures used as predictor variables in the analyses. These included measures of statistical knowledge, statistics anxiety, self-efficacy for online learning, as well as questions about Microsoft 365 software installation and the device used to complete the assignment.
Statistical knowledge test
To create a formative measure of statistics knowledge, the instructional team adapted twelve items from publicly available AP Statistics practice tests and the New York Times “What's Going On in This Graph?” column, developed by the New York Times in partnership with the American Statistical Association (The Learning Network 2021). The multiple-choice problems used data visualization (i.e., graphs, tables) and/or word prompts to query knowledge of target statistical concepts. Following each problem, students selected an answer from four response options. This test exhibited low internal consistency (Cronbach’s 𝛼 = .57); however, given the wide range of statistical concepts covered, this was not unexpected (i.e., a knowledge test covering diverse topics is not expected to perform like a scale). Table 3 provides item-level statistics organized by difficulty; note that the target concepts are largely mapped onto the content of the Excel worksheet activities. Students exhibited a wide range of statistical knowledge on the test, with an overall average of 50.7% correct (SD = 19.4%, Range = 0.0%–100.0%).
| Target Concept | Percent Correct | 𝛼 if item deleted |
|---|---|---|
| Calculating a median | 77.3% | .55 |
| Means and standard deviations | 77.3% | .53 |
| Calculating a mean | 76.7% | .52 |
| Scatterplots and different associations between variables | 56.8% | .53 |
| Scatterplots and correlation coefficients | 50.5% | .54 |
| Histograms | 49.7% | .54 |
| Uses of bar-graphs | 49.2% | .55 |
| Normal distribution | 45.3% | .56 |
| Comparing different distributions | 43.6% | .54 |
| Independent sample t-tests and alpha levels | 39.4% | .58 |
| Types of t-tests | 22.6% | .56 |
| Features of a normal distribution | 20.5% | .57 |
Statistics anxiety scale
The six-item statistics anxiety scale (STARS) (adapted from Teman 2013) asked students to rate their anxiety towards a statistics-related activity (three items) or their agreement with a statement about doing statistics (three items); see Table 4 for the list of items. Each set of items used a 5-point Likert scale (for anxiety ratings: 1 = Not at all Anxious, 5 = Extremely Anxious; for agreement statements: 1 = Strongly Disagree, 5 = Strongly Agree). Students on average were slightly anxious about statistics (M = 2.27, SD = 0.96, Range = 1–5) and had moderate agreement with statements about their statistics skills (M = 2.81, SD = 1.09, Range = 1–5). We created a composite measure of the two subscales (M = 2.54, SD = 0.81, 𝛼 = .78).
| Item | Mean | SD | 𝛼 if item deleted | |
|---|---|---|---|---|
| 1 | Reading a journal article that includes some statistical analyses | 2.14 | 1.04 | .73 |
| 2 | Trying to understand the statistical analyses described in the abstract of a journal article | 2.41 | 1.06 | .73 |
| 3 | Interpreting the meaning of a table in a journal article | 2.26 | 1.05 | .73 |
| 4 | I have not had math for a long time. I know I will have problems getting through statistics | 3.06 | 1.36 | .76 |
| 5 | I cannot even understand seventh- and eighth-grade math; how can I possibly do statistics? | 2.37 | 1.26 | .76 |
| 6 | Since I have never enjoyed math I do not see how I can enjoy statistics | 3.01 | 1.33 | .77 |
Self-efficacy for online learning
We administered an adapted version of Shen et al.’s (2013) self-efficacy scale for online learning. The adapted scale encompassed minor changes in wording to ensure that the items were appropriate for both in-person and online course sections. The scale consisted of fifteen items assessing students’ confidence in their ability to complete coursework (e.g. Complete the course with a good grade), use online technology (e.g. Download course materials from Blackboard), and interact with their instructor and peers (e.g. Seek help from my classmates when needed). Students responded using a 5-point Likert scale (1 = Not confident at all, 5 = Completely confident). Composite scores were created by averaging across the fifteen items (M = 3.88, SD = 0.68, 𝛼 = .93); see Table 5 for item-level statistics.
| Item | Mean | SD | 𝛼 if item deleted | |
|---|---|---|---|---|
| How confident are you that you could do the following tasks? | ||||
| 1 | Complete the course with a good grade | 3.83 | 0.89 | .92 |
| 2 | Understand complex concepts | 3.44 | 0.97 | .92 |
| 3 | Overcome challenges that could negatively affect my learning | 3.56 | 0.94 | .92 |
| 4 | Successfully complete all of the required activities | 3.94 | 0.90 | .92 |
| 5 | Keep up with the course schedule | 3.83 | 0.94 | .92 |
| 6 | Create a plan to complete a given assignment | 3.80 | 0.93 | .92 |
| 7 | Willingly adapt my study habits to meet course expectations | 3.72 | 0.93 | .92 |
| How confident are you that you could use online tools to complete the following tasks? | ||||
| 1 | Download course materials from Blackboard | 4.13 | 0.96 | .92 |
| 2 | Install software on my computer | 3.95 | 1.06 | .92 |
| 3 | Submit assignments through Blackboard | 4.22 | 0.91 | .92 |
| 4 | Send emails to my instructor | 4.29 | 0.87 | .92 |
| How confident are you that you could interact with your instructor and classmates in the following ways? | ||||
| 1 | Ask my instructor questions | 4.02 | 0.95 | .92 |
| 2 | Inform my instructor when unexpected situations arise | 4.10 | 0.91 | .92 |
| 3 | Express my opinions in class respectfully | 3.85 | 1.08 | .92 |
| 4 | Seek help from my classmates when needed | 3.45 | 1.20 | .93 |
Device used
Students were asked to report the device they used to submit their coursework from the following options: desktop computer, laptop, smartphone, or tablet. Most students (n = 871, 82.1%) reported that they used a desktop computer or laptop while the remainder (n = 190, 17.9%) indicated that they used a smartphone or tablet for Qualtrics HW1.
Software installation and access
Students were asked to self-report their success in installing Microsoft Office 365. The majority (n = 939, 88.5%) reported that they were able to install Microsoft Office 365 on their device or already had access to Microsoft Office while the remaining students (n = 122, 11.5%) reported that they were unable to install the software at the start of the semester.
Results
Preliminary Analysis
We used linear mixed-effects models to examine predictors of Noba textbook quiz scores, embedded vocabulary quiz scores, Qualtrics homework completion, and Excel worksheet completion. All analyses were done in R version 4.4.0 (R Core Team 2024) using the lme4 package version 1.1–35.3 (Bates et al. 2015). To account for students nested within sections, course section was included in all models as a random effect. Preliminary analyses indicated that course section characteristics (in-person vs. online formats, regular vs. large enrollment) were unrelated to students’ inclusion in the analytic sample. That is, the probability of a student turning in Qualtrics HW1 and at least one other assignment did not vary by course section characteristics (in-person vs. online format: 𝛽 = –0.05, p = .89; regular vs. large course enrollment: 𝛽 = –0.15, p = .71).
Additionally, preliminary analyses indicated that course section characteristics (in-person vs. online formats, regular vs. large enrollment) were unrelated to the dependent variables (Noba textbook quiz scores, embedded vocabulary quiz scores, Qualtrics homework completion, and Excel worksheet completion). Course section characteristics were dropped from the final models. Further preliminary analyses indicated that using a computer versus an iPhone/tablet and students’ self-reported success in installing the Microsoft 365 suite at the start of the semester were unrelated to the dependent variables. These variables were also dropped from the final models. The final models are reported below, organized by dependent variable.
Noba Textbook Quiz Scores
We used the following variables to predict Noba textbook quiz scores (M accuracy = 85.8%, SD = 20.7%): statistical knowledge test scores, statistics anxiety scale scores, self-efficacy for online learning, and embedded vocabulary quiz scores. Table 6 presents the final model, which explained 9.7% of the variance in Noba quiz scores. Students with higher scores on the statistical knowledge test at the beginning of the semester did better on the quizzes (𝛽 = 14.32, p < .001), as did students with higher self-efficacy for online learning (𝛽 = 2.12, p = .02). Higher accuracy on the vocabulary quizzes embedded in the Qualtrics homework assignments was associated with higher scores on the Noba textbook quizzes (𝛽 = 18.77, p < .001).
| Variable | 𝛽 | SE | t | p |
|---|---|---|---|---|
| Intercept | 53.913 | 6.038 | 8.929 | < .001 |
| Statistical Knowledge Test | 14.320 | 3.323 | 4.309 | < .001 |
| Statistics Anxiety Scale | 0.069 | 0.814 | 0.084 | .933 |
| Self-efficacy for Online Learning | 2.122 | 0.940 | 2.258 | .024 |
| Embedded Vocabulary Quiz Score | 18.772 | 3.705 | 5.066 | <.001 |
Embedded Vocabulary Quizzes
We used the following variables to predict accuracy on the embedded vocabulary quizzes (M = 86.0%, SD = 16.7%): statistical knowledge test scores, statistics anxiety scale scores, self-efficacy for online learning, and Noba textbook quiz scores. Table 7 presents the final model, which explains 5.4% of the variance in embedded quiz scores. Students with higher scores on the statistical knowledge test at the beginning of the semester exhibited higher vocabulary comprehension (𝛽 = 0.11, p < .001). Mirroring the previous model, higher scores on Noba quiz scores were associated with higher scores on the embedded vocabulary quizzes (𝛽 = .001, p < .001).
| Variable | 𝛽 | SE | t | p |
|---|---|---|---|---|
| Intercept | 0.771 | 0.045 | 17.108 | < .001 |
| Statistical Knowledge Test | 0.106 | 0.027 | 3.878 | < .001 |
| Statistics Anxiety Scale | –0.011 | 0.007 | –1.610 | .108 |
| Self-efficacy for Online Learning | –0.011 | 0.008 | –1.367 | .172 |
| Noba Textbook Quiz Score | 0.001 | 0.000 | 4.867 | < .001 |
Qualtrics Homework Assignments
On average, students completed 7.5 out of 9 Qualtrics homework assignments (SD = 1.9, Range = 2–9). Table 8 presents the final model which explained 11.8% of the variance in homework completion. Students with higher scores on the statistical knowledge test at the start of the semester (𝛽 = 0.13, p < .001), and those with higher scores on the embedded vocabulary quiz scores (𝛽 = 0.18, p < .001) completed more of the Qualtrics assignments.
| Variable | 𝛽 | SE | t | p |
|---|---|---|---|---|
| Intercept | 0.547 | 0.063 | 8.621 | < .001 |
| Statistical Knowledge Test | 0.127 | 0.035 | 3.677 | < .001 |
| Statistics Anxiety Scale | 0.005 | 0.008 | 0.607 | .544 |
| Self-efficacy for Online Learning | 0.013 | 0.010 | 1.326 | .185 |
| Embedded Vocabulary Quiz Score | 0.179 | 0.039 | 4.637 | < .001 |
Excel Worksheets
On average, students completed 3.7 out of 6 Excel worksheets (SD = 2.3, Range = 0–6). Table 9 presents the final model explaining 10.3% of the variance in Excel worksheet completion. Students with higher scores on the statistical knowledge test completed more Excel worksheets (𝛽 = 1.87, p < .001), as did students with higher self-efficacy for online learning (𝛽 = 0.37, p < .001). Students with higher accuracy on the embedded vocabulary quizzes (𝛽 = 1.70, p < .001) also completed more Excel work.
| Variable | 𝛽 | SE | t | p |
|---|---|---|---|---|
| Intercept | –0.101 | 0.670 | –0.151 | .880 |
| Statistical Knowledge Test | 1.867 | 0.369 | 5.057 | < .001 |
| Statistics Anxiety Scale | –0.011 | 0.090 | –0.118 | .906 |
| Self-efficacy for Online Learning | 0.372 | 0.104 | 3.562 | < .001 |
| Embedded Vocabulary Quiz Score | 1.704 | 0.412 | 4.139 | < .001 |
Discussion
The present study reports on a departmental effort to teach quantitative reasoning in Introductory Psychology (PSY100), in alignment with the APA Guidelines for the Undergraduate Major (APA 2023) and the APA Presidential Task Force on Introductory Psychology Statistical Literacy (Neufeld et al. 2022). To promote psychology as a data-driven science, PSY100 students were assigned a series of online homeworks via Qualtrics survey software and accompanying Excel worksheet activities using CAPs to guide students in analyzing and visualizing psychology-relevant datasets. Our use of CAPs served to scaffold active learning by providing step-by-step screencast tutorials for each assignment. We used embedded quizzes as formative assessments of both statistics and general psychology vocabulary comprehension to gauge students’ grasp of lesson content throughout the semester. We also used low-stakes quizzes to assess student comprehension of each of the Noba textbook modules and track their continued engagement with course material. Student performance on both the embedded vocabulary quizzes (M = 86.0%, SD = 16.7%, Range = 22.1%–100%) and the Noba textbook quizzes (M = 85.8%, SD = 20.7%, Range = 0–100) indicated that the curriculum was developmentally appropriate. Moreover, students with higher scores on the Noba textbook quizzes also tended to have higher scores on the embedded vocabulary quizzes, which in turn was associated with their completing more of the Qualtrics homework assignments and Excel assignments over the course of the semester. These results are in keeping with the value of formative assessments not only for gauging and monitoring students’ comprehension of lessons, but also for providing critical feedback to students that may encourage academic persistence (Clark 2012; Grose-Fifer et al. 2019). Students’ high scores on the Noba quizzes underscore that it is possible to engage students in reading their textbooks, which goes against the countervailing opinion that college students do not read anymore (McMurtrie 2024).
Both the Noba textbook quiz scores and the embedded vocabulary quiz scores were associated with prior statistical knowledge. That is, higher scores on the statistical knowledge test at the start of the semester were associated with better comprehension of the textbook modules and statistical and general psychology terminology from the lessons. Higher scores on the statistical knowledge test were also associated with completing more Qualtrics homework assignments and Excel worksheets. Students entered the course with a wide variability in their statistical knowledge, with performance on our statistical knowledge test ranged from 0–100%. Although the statistical knowledge test utilized problems taken from The New York Times and other sources designed for high school students, our college student sample averaged only 50.7% correct. This appears to be in keeping with Cook and Fukawa-Connelly (2015), who reported idiosyncratic patterns of statistical knowledge among first-year college students, with some correct understandings existing alongside various misconceptions and/or a general lack of knowledge. Our findings reflect broader concerns about declining mathematical knowledge of U.S. students at college entry (Carpenter and Kirk 2017; Er 2018), which may have worsened further since the COVID-19 pandemic (Toness and Luyre 2022). Students with insufficient baseline knowledge in mathematics may be less prepared to complete assignments that involve quantitative reasoning and college-level coursework in general.
In contrast to statistics knowledge, statistics anxiety was unrelated to learning outcomes. In other words, despite wide variation in statistics anxiety scores, statistics anxiety did not impede students’ success and engagement with the online homeworks over the course of the semester. Although this result appears to be inconsistent with prior research linking higher math anxiety with either lower (Pekrun 2006; Pekrun et al. 2017) or higher academic achievement (Brooks et al. 2024), it matches other null findings (Trassi et al. 2022). In the present study, the lack of a relation with statistics anxiety could be attributed to the nature of the homework assignments, which were intentionally low-stakes and graded solely on the basis of completion. Assigning low-stakes homework, where students have perceived control over their learning, may mitigate effects of high statistics anxiety on course outcomes (Pekrun et al. 2017). Chang and Beilock (2016) emphasize the need for interventions to reduce the linkage between math anxiety and academic achievement. Using low-stakes assignments to increase self-efficacy and perceived control of learning could be one example of an effective intervention. That said, the varied findings in the literature suggest that the relation between statistics anxiety and academic achievement may be complex and mediated by other factors (Finney et al. 2003; Trassi et al. 2022). It would be fruitful for future research to examine whether statistics anxiety has a stronger association with learning outcomes if assignments are graded based on correctness rather than completion.
As part of our department-wide effort to revise the PSY100 curriculum, we packaged course materials as self-paced, asynchronous online assignments that would be suitable for students enrolled in either in-person or fully online course sections. In contrast with prior findings indicating worse outcomes for students who rely on mobile devices for completing online assignments (Brooks et al. 2024; Gravelle et al. 2024), our preliminary analyses found no significant differences in learning outcomes for students who completed HW1 using a hand-held device as compared to a laptop/desktop computer. Though 18% of students reported completing the assignment on a mobile device, repeated emphasis on the importance of using a laptop/desktop may have encouraged students to use computers for future assignments. Unfortunately, for the present study, we did not track students’ uses of specific devices beyond HW1.
Notably, our preliminary analyses also found no effects of course modality or enrollment (regular vs. large) on outcome measures. These null results indicate that the curriculum was equally accessible across modalities and scalable to the large-format “lecture-based” PSY100 courses taught at many institutions (Long and Coldren 2006). Whereas course characteristics were unrelated to learning outcomes, students’ self-efficacy for online learning was associated with their obtaining higher scores on the Noba textbook quizzes and with submitting more of the Excel worksheets. Although not all studies have found a link between self-efficacy and course outcomes (Gravelle et al. 2024), the current results align with several studies that have found this positive association (Alqurashi 2016; Gurung and Stone 2023; Tsai et al. 2011). Self-efficacy may just be one piece to the puzzle in getting students to engage in online asynchronous learning assignments and establishing learning presence in students, alongside a collection of other student beliefs and behaviors found to impact students’ engagement with online coursework (Shea and Bidjerano 2010).
In addition to self-efficacy, Shea and Bidjerano (2010) emphasize that attendant effort is an integral piece of the learning presence needed to thrive in an online learning environment. Despite the positive learning outcomes for students who engaged with the asynchronous online course materials, we faced a large number of non-responding students who failed to submit the online asynchronous assignments. That is, of 1523 students enrolled in PSY100, 28.9% did not complete even the first assignment, indicating a decision not to attend or engage with the curriculum from the outset of the semester. Most of these students received a grade of D, F, or withdrawal (DFW), indicating that failure to engage with coursework early on is prognostic of a negative course outcome. The observed non-response rate aligns with reports of chronic absenteeism in college courses, which increased dramatically in the aftermath of the COVID-19 pandemic, especially at non-selective minority-serving institutions (Bulman and Fairlie 2022; National Student Clearinghouse Research Center 2022, 2024). Though beyond the scope of the present study, we acknowledge the myriad social and economic factors contributing to college students’ relationships to higher education institutions and affecting their academic trajectories. These may include limited access to computers and high speed internet and other technology barriers (Jaggars et al. 2021), family and work obligations (Perna 2010; Porchea et al. 2010), food and housing insecurity (Ilieva et al. 2018), and systemic racism and social injustice (Williams et al., 2022). These factors create a cumulative burden on students’ self-efficacy for academic work and their sense of belonging in college (DeFreitas and Rin 2013), and should be examined in future research. For example, our self-efficacy scale was limited in its focus on online learning, and did not allow us to consider how students’ identities and lived experiences may have influenced their perceptions as college students. Future studies should include self-report measures of students’ college experiences, including qualitative measures of their sense of community and connection to instructors and peers (Garrison and Arbaugh 2007), and personal and school-related challenges (Gravelle et al. 2024).
Finding ways to engage first-year college students effectively, in the hope that they will find college coursework to be relevant and worth pursuing, remains a challenge (Belkin 2021; Tough 2023). Many students today approach college with ambivalence. Understanding why students fail to engage with college coursework may require a shift away from individual level-factors like self-efficacy to social and societal factors affecting academic persistence (Roberts et al. 2025; Zengilowski et al. 2023). Our findings indicate that PSY100 students often “quit” the course without completing a single assignment. This suggests that strong connections need to be fostered at the start of the semester to have any impact on student retention. Efforts to promote social engagement with peers may help students establish a sense of belonging in higher education and aid in retention (Ahn and Davis 2023). Along these lines, our institution is exploring whether employing successful former students as peer leaders in PSY100 might improve learning outcomes (McCloskey et al. 2024). Promoting peer-to-peer interactions may help to build a sense of community and belonging in college courses, with benefits for both peer leaders and the students enrolled in the course (Stigmar 2016). That withstanding, co-curricular efforts to increase social supports for first-year college students may have limited effects on retention if the wider societal and systemic factors affecting students remain unaddressed (Zengilowski et al. 2023).
In conclusion, the present study demonstrates the feasibility of a department-wide effort to embed quantitative reasoning in a general education PSY100 course. Our curriculum provides a framework for instructors to rethink how they go about teaching general education courses to incorporate broad-based scientific and analytic skills valued in the workplace (NACE 2021) in course design. Key components of the curriculum included (1) self-paced quizzes to encourage close reading of the textbook, (2) homework assignments featuring library resources (e.g. Google Scholar), scientific abstracts, TED talks from prominent researchers, and embedded quizzes as formative assessments, and (3) hands-on Excel worksheet activities to introduce data analysis and visualization techniques. Further research is needed to address issues related to student apathy, absenteeism, and broader problems associated with declining enrollments in higher education. As a starting point, instructors should strive to take necessary steps to engage college students from the start of the semester in developing the analytic and scientific reasoning skills they need for college and for their future careers.
