Research showing that virtual learning is less effective than classroom teaching – right?

Cellini, R. S. (2021) How does virtual learning impact students in higher education? Brown Center Chalkboard, Brookings, August 13

Summary of results

This blog post is based on several studies of post-secondary student performance, comparing online learning with in-person teaching. Cellini summarises the results as follows:

The results are generally consistent with past research: Online coursework generally yields worse student performance than in-person coursework. The negative effects of online course-taking are particularly pronounced for less-academically prepared students and for students pursuing bachelor’s degrees. New evidence from 2020 also suggests that the switch to online course-taking in the pandemic led to declines in course completion. However, a few new studies point to some positive effects of online learning, too.

Although you will see that I question the results of some of the studies, and certainly Cellini’s own conclusions, the following will be useful for anyone asked to write a paper comparing online versus in-person learning, or to answer questions on this topic from faculty or administrators. Because of the number of research papers covered, this is a long post, but make sure you read my analysis at the end, which questions many of these findings.

What were the studies?

Cellini draws on the following studies to come to her conclusions:

1. Figlio, D., M. Rush and Yin, L. (2013) Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning Journal of Labor Economics, Volume 31, Number 4

Students in a large introductory microeconomics course at a major research university were randomly assigned to live lectures versus watching these same lectures in an Internet setting where all other factors (e.g., instruction, supplemental materials) were the same. We find modest evidence that live-only instruction dominates Internet instruction. These results are particularly strong for Hispanic students, male students, and lower-achieving students. We also provide suggestions for future experimentation in other settings.

The important point about this study is that no attempt was made to re-design courses for an online learning environment.

2. Alpert, W., Couch, K. and Harmon, O. (2016). A Randomized Assessment of Online Learning. American Economic Review, Vol. 106, No. 5.

A microeconomics principles course employing random assignment across three sections with different teaching models is used to explore learning outcomes as measured by a cumulative final exam for students who participate in traditional face-to-face classroom instruction, blended face-to-face and online instruction with reduced instructor contact time, and a purely online instructional format. Evidence indicates learning outcomes were reduced for students in the purely online section relative to those in the face-to-face format by 5 to 10 points on a cumulative final exam. No statistically significant differences in outcomes are observed for students in the blended relative to the face-to-face section.

3. Joyce, T. et al. (2015) Does classroom time matter? Economics of Education Review, Vol. 46, June

We randomized 725 college students into traditional twice-per-week and compressed once-per-week lecture formats in introductory microeconomics. [Results]:

• Students in the compressed format scored 3.2 out of 100 points lower on midterm
• Students in compressed format scored 1.6 points lower on final (NOT statistically significant).
• No difference in tests scores by format for top third of students.
• No difference by format in withdrawal rates or the percent of classes attended.
• No difference by format in hours spent online doing assignments.
• Compressed format offers savings but at some cost to student performance.

Note that the overall differences were smail.

4. Bettinger, E. et al. (2017) Virtual Classrooms: How Online College Courses Affect Student Success. American Economic Review, Vol. 107, No. 9 

We study students at one large for-profit university with an undergraduate enrollment of more than 100,000 students, 80 percent of whom are seeking a bachelor’s degree. The university’s average student takes two-thirds of her courses online. The remaining one-third of courses meet in conventional in-person classes held at one of the university’s 102 physical campuses. The data for this paper cover more than four years of the university’s operations, including over 230,000 students enrolled in 168,000 sections of more than 750 different courses…..the university’s online classes attempt to replicate its traditional in-person classes, except that student-student and student-professor interactions are virtual and asynchronous….we find that taking a course online, instead of in-person, reduces student success and progress in college. Grades are lower both for the course taken online and in future courses. Students are less likely to remain enrolled at the university. These estimates are local average treatment effects for students with access to both online and in-person options; for other students, online classes may be the only option for accessing college-level courses.

I think this study needs to be treated with some caution. I’m guessing that the University of Phoenix was the institution in question, before being investigated for financial issues under the Obama administration and subsequently closing most of its campuses. Thus although the results may be relevant for this particular institution, it is not a good exemplar for comparison with online learning at other institutions. 

5. Xu, D. and Jaggars, S. (2014) Performance Gaps between Online and Face-to-Face Courses: Differences across Types of Students and Academic Subject Areas, The Journal of Higher Education, Vol. 85, No.5

Using a dataset containing nearly 500,000 courses taken by over 40,000 community and technical college students in Washington State, this study examines the performance gap between online and face-to-face courses and how the size of that gap differs across student subgroups and academic subject areas. While all types of students in the study suffered decrements in performance in online courses, those with the strongest declines were males, younger students, Black students, and students with lower grade point averages. Online performance gaps were also wider in some academic subject areas than others. After controlling for individual and peer effects, the social sciences and the applied professions (e.g., business, law, and nursing) showed the strongest online performance gaps.

For me, this us by far the best paper on this topic, although it is specific to community college students in one state of the USA (Washington). The authors make the important point that some types of students will need more support or ‘scaffolding’ than others if they are to succeed in online learning. This is required reading in any discussion of comparisons between online and in-person teaching, and especially for those who will have disadvantaged students in their online courses. For a more detailed review of this article see Comparing apples with oranges: online vs face-to-face-learning in community-colleges.

6. Bowen, W. et al. (2013) Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial Journal of Policy Analysis and Management, Vol. 33, No. 1

We measure the effect on learning outcomes of a prototypical interactive learning online statistics course by randomly assigning students on six public university campuses to take the course in a hybrid format (with machine-guided instruction accompanied by one hour of face-to-face instruction each week) or a traditional format (as it is usually offered by their campus, typically with about three hours of face-to-face instruction each week). We find that learning outcomes are essentially the same—that students in the hybrid format are not harmed by this mode of instruction in terms of pass rates, final exam scores, and performance on a standardized assessment of statistical literacy. We also conduct speculative cost simulations and find that adopting hybrid models of instruction in large introductory courses has the potential to significantly reduce instructor compensation costs in the long run.

7. Koefed, M. et al. (2021) Zooming to Class?: Experimental Evidence on College Students’ Online Learning during COVID-19 IZA-Institute of Labour Economics IZA DP No. 14356, May 

This was  a study during the Covid-19 pandemic of students at the U.S. Military Academy at West Point.

In the fall of 2020, we randomized 551 West Point students in a required Introductory Economics course across twelve instructors to either an online or in-person class. West Point limited classroom capacity to incorporate social distancing measures. However, unlike other institutions, West Point still needed to carry out its unique mission to provide physical and military training; thus students returned to campus for the Fall 2020 semester. For this study, each instructor agreed to teach half of their four section teaching load online and half in-person. Each in-person and online class allowed for twelve and eighteen students respectively. 

Final grades for online students dropped by 0.215 standard deviations; a result apparent in both assignments and exams and largest for academically at-risk students. A post-course survey finds that online students struggled to concentrate in class and felt less connected to their instructors and peers. We find that the shift to online education had negative results for learning.

Although not explicitly stated in the paper, it appears that the same lectures delivered in class were also delivered online. It should also be noted that although in both modes class sizes were small, the online classes had 50% more students than the in-person classes.

8. Cacault. M. et al. (2021) Distance Learning in Higher Education: Evidence from a Randomized Experiment Journal of the European Economic Association, Volume 19, Issue 4 

Using a randomized experiment in a public Swiss university, we study the impact of online live streaming of lectures on student achievement and attendance. We find that (i) attending lectures via live streaming lowers achievement for low-ability students and increases achievement for high-ability ones; (ii) students use the live streaming technology only occasionally, apparently when random events make attending in class too costly, and (iii) offering live streaming reduces in-class attendance only mildly. 

Our experiment provides an ideal opportunity to identify the causal effect of what is arguably the most salient feature distinguishing online from offline education, namely, the fact that the material is presented to the students via the screen of an internet connected device located at any distance from the teacher. In our experiment, this is the only difference between the treatment and the control conditions. Our analysis shows that students use live streaming only occasionally, apparently when attending in class is too costly, and that attending lectures online has a negative impact on achievement for low-ability students and a positive one for high-ability ones.

Again, note that the only difference between the online and in-person teaching is that the in-person lessons are also streamed.

9. Hart, C. et al. (2018) Online Course-taking and Student Outcomes in California Community Colleges Education Finance and Policy, Vol. 13, No. 1

[This study] estimates differences in student performance under online versus face-to-face course delivery formats in the California Community College system. On average, students have poorer outcomes in online courses in terms of the likelihood of course completion, course completion with a passing grade, and receiving an A or B. These estimates are robust across estimation techniques, different groups of students, and different types of classes. Accounting for differences in instructor characteristics …dampens but does not fully explain the estimated relationships. Online course-taking also has implications for downstream outcomes, although these effects are smaller. Students are more likely to repeat courses taken online, but are less likely to take new courses in the same subject following courses taken online. Online course-taking is associated with particularly pronounced negative outcomes in Math and Humanities….it is important to realize that our results may miss an array of benefits that online courses may offer students. For instance, students may be able to retain jobs that demand flexible schedules or may save on child care costs if they can complete coursework on a nonstandard schedule.

The study sampled first-time entrants to the community college system in the 2008–09 academic year. The study looked at the performance of 217,194 students out of the 440,405 first-time entrants in the California Community College System, due to a number of restrictions in the sampling. For instance, the sample was limited to courses in which both in-person and online options were offered at the same college in the same term. The sample was also restricted to students aged between 18-40. Importantly, both asynchronous and synchronous online courses were included in the sample. Unfortunately, though, the analysis did not examine whether this made any difference to online performance.

Overall, this is a very sophisticated study, which controls for a number of different variables. However, what was not controlled for was different types of online teaching – synchronous versus asynchronous (or for that matter, different types of in-person teaching).

10. Krieg, J. and Henson, S. (2016) The Educational Impact of Online Learning: How Do University Students Perform in Subsequent Courses? Education Finance and Policy, Vol. 11, No. 4

Using records from nearly 40,000 students from a medium-sized regional comprehensive university, we compare grades earned in subsequent follow-up courses between students who completed the prerequisite in an online format versus a traditional one. We find that students’ grades in follow-up courses can be expected to be nearly one twelfth of a grade point lower if the prerequisite course was taken online.

11. Huntingdon-Klein, N., et al. (2016) Selection into Online Community College Courses and Their Effects on Persistence National Center for Analysis of Longitudinal Data in Education Research Calder Working Paper 131

Student data are derived from the Washington State Board of Community & Technical Colleges (SBCTC) as provided by the Washington State Education Research & Data Center (ERDC). We used student transcript, background, and degree completion data on all students who took a course at a public Washington State community college from fall 2008 to summer 2013, with data from 35 different community colleges.

For the average person, taking an online course has a negative effect on the probability of taking another course in the same field and on the probability of earning a degree. The negative effect on graduation for students who choose to take an online course is stronger than the negative effect for the average student. The penalty for being Black, Hispanic, a veteran, or having limited English proficiency is smaller having taken an online course than a face-to-face course. On the other hand, women, the full-time employed, older students, and more academically able students compare more favorably to their peers having taken a face-to-face course compared to an online course.

Note that this is a study that also uses data from Community Colleges in Washington State (see No. 5 above). There seems to be a discrepancy between the two studies on the impact of online learning on Black students, although I may have been confused by the double negative in the above conclusions.

12. Cellini, S. and Grueso, H. (2021) Student Learning in Online College Programs AERA Open, May 18

We draw on administrative data from the country of Colombia to assess differences in student learning in online and traditional on-campus college programs. The Colombian context is uniquely suited to study this topic, as students take a compulsory exit examination at the end of their studies. We can therefore directly compare the performance on the exit exam for students in online and on-campus programs both across and within institutions, degrees, and majors.

Our results suggest that bachelor’s degree students in online programs perform worse on nearly all test score measures (including math, reading, writing, and English) relative to their counterparts in on-campus programs. Results for shorter technical certificates are more mixed. While online students perform significantly worse than on-campus students on exit exams in private institutions, they perform better in SENA—the main public vocational institution in the country.

The quality of instruction may be higher for online than on-campus courses in SENA. Online programs and classes benefit from the best instructors in SENA, as instructors can teach virtually from the main campus. In contrast, in-person instruction at branch campuses in rural areas may vary in quality, drawing on local instructors that may or may not be as qualified or experienced as those leading online classes. Students in online classes engage directly with instructors in live virtual instruction for about 2 hours per week, which might be one key to the success of these programs. However, administrators also note that students who have trouble with internet connectivity or face other challenges can also access recorded lectures to learn asynchronously, if needed.

This study is one of the few listed that looks at factors that influence the actual quality of online learning.

13. Fischer, C. et al. (2021) Increasing success in higher education: The relationships of online course taking with college completion and time-to-degree  Annenberg Institute at Brown University (EdWorkingPaper: 21-427)

In this study we analyzed six years of institutional data for three cohorts of students in thirteen large majors (N=10,572) at a public research university to examine …effects such as time-to-degree and graduation rates of students’ online course participation. Using online course offering as an instrumental variable for online course taking, we find that online course taking of major-required courses leads to higher likelihood of successful four-year graduation and slightly accelerated time-to-degree. These results suggest that offering online course-taking opportunities may help students to more efficiently graduate college…Online course taking is associated with more efficient college graduation; students who are given the opportunity to take classes online graduate more quickly than those who are not. We also found that online course taking is associated with a higher likelihood of successfully graduating college within four years. 

This study is important as it looks at the longer-term success rate of online learning, rather than short-term outcomes such as course completion or course grades. The study also points out that studies on the effects of online course taking are focused on community colleges or for-profit universities. As the authors state, The student bodies and institutional contexts of community colleges and for-profit colleges are different from four-year institutions such that findings from these schools might not generalize to the four-year context.

14. Bird, K. et al. (2021) Negative Impacts From the Shift to Online Learning During the COVID-19 Crisis: Evidence from a Statewide Community College System Annenberg Institute at Brown University (EdWorkingPaper: 20-299)

We estimate the impact of the shift from in-person to virtual instruction in Spring 2020 on the academic performance of Virginia’s community college students. We find modest negative impacts (four to eight percent) on course completion. Our results suggest that faculty experience teaching a given course online does not mitigate the negative effects of students abruptly switching to online instruction.

How should we interpret these results?

Are you still with me? A lot of reading and these were just the abstracts.

First, I am very grateful to Stephanie Cellini for pulling together into one place so many studies comparing online versus face-to-face instruction. However, I have gone into considerable detail in reviewing this article, because the Brookings Institute is a highly respected and influential think tank on economic, social and educational issues. Its findings then need particularly detailed scrutiny.

Second, as always, I suggest that you read the original papers, but a warning: many use complicated statistical analysis to control multiple variables, and others look like they were written by computers, not humans. Nevertheless, I feel confident in making the following comments:

1. The studies clearly suggest that if you just move traditional classroom teaching online, many students, especially the most disadvantaged, will do less well than if they were in class. The results overall are not disastrous, though. Performance or retention is about 5%-10% poorer in most cases, although somewhat higher for some disadvantaged students.
2. These papers all seek to compare online teaching with face-to-face instruction using a quasi-experimental approach. This approach to comparing modes of delivery (online versus face-to-face; distance education versus face-to-face) or different media (classroom lectures versus video lectures) has a long and troubled history (see, for instance, Saba, 2000). The problem is that there are so many variables that can affect learner performance. The result is that differences within a variable (such as the quality of teaching) may be greater than the differences between variables (the modes of teaching). This is a particular risk when very large samples are used (as in most of these studies). Small differences with a large sample are more likely to be statistically significant than large differences with small samples. Most of the differences in these studies were in the 5-10% range; quite small but statistically significant because the sample sizes were so large. What we don’t know though is what caused these differences.
3. As long ago as 1977, Wilbur Schramm asked: What kinds of learning can different media best facilitate, and under what conditions? Indeed, it is the variables or conditions for success that we should be examining, not just the technological delivery. These papers looked at many different variables, such as type of student, differences between subject areas, even how far from the campus students lived, but none of them looked at the most important variable: the method of teaching. How were the online courses designed? What teaching method was used in the classroom? Are we comparing online video lectures with the same on-campus lectures or are we comparing asynchronous online courses with synchronous classroom lectures? 
4. This raises the question of affordances. To what extent were the online courses adapted to meet the requirements of students studying in isolation, at home? We know that successful online courses need to meet certain criteria, such as clearly communicated learning outcomes, regular pacing, instructor support, a sense of community with other students. This has been the big problem with emergency remote learning: there was not time to introduce best practices into emergency remote learning. It is not surprising then that students during the pandemic in general did less well and were less satisfied compared with being on campus.
5. This raises another point. For many students it is not a choice between online or face-to-face teaching; online learning is the ONLY way many online students can study. Many are working part-time or have child care responsibilities. It is not surprising that there is some loss of performance as these students often face greater practical difficulties in studying than full-time students on campus.
6. Similarly, the choice was not between emergency remote learning or face-to-face teaching, but between emergency remote learning and NO teaching or learning at all. It is interesting that the last study showed that at least in a four-year public university, the availability of online courses increased the probability of students completing a bachelor’s degree within four years. It kept some students from dropping out.
7. This is a very selective list of studies comparing online learning/distance education with face-to-face teaching, probably chosen deliberately to reinforce the main point the author (Cellini) wanted to make. Perhaps the most notable omission was the Department of Education meta-study conducted by Barbara Means et al. (2010): A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning…The meta-analysis found that, on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes…was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face…these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. In other words, in a blended mode, students spent more time on task – an important variable – leading to success. These results were by and large replicated in another meta-analysis by Bernard et al. (2004) A meta-analysis of the comparative distance education (DE) literature between 1985 and 2002 was conducted. In total, 232 studies containing 688 independent achievement, attitude, and retention outcomes were analyzed. Overall results indicated effect sizes of essentially zero on all three measures and wide variability. This suggests that many applications of DE outperform their classroom counterparts and that many perform more poorly. Dividing achievement outcomes into synchronous and asynchronous forms of DE produced a somewhat different impression. In general, mean achievement effect sizes for synchronous applications favored classroom instruction, while effect sizes for asynchronous applications favored DE. However, significant heterogeneity remained in each subset.

In brief, we need both online learning and face-to-face teaching. The issue is not which is better, but how do we make the teaching, in whatever form, effective for the students who are in those courses. If you want to know how to do this, read Teaching in a Digital Age.

(Other) References

Bernard et al. (2004) How Does Distance Education Compare With Classroom Instruction? A Meta-Analysis of the Empirical Literature Review of Educational Research, Vol. 74., No.3

Means, B. et al. (2009) Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies Washington, DC: US Department of Education

Saba, F. (2000) Research in distance education: a status report  The International Review of Research in Open and Distributed Learning, Vol. 1, No. 1

Schramm, W. (1977) Big Media, Little Media Beverly Hills CA/London: Sage