A review of the HEQCO report on productivity and quality in online learning in higher education

The view from HEQCO, Toronto

Carey, T., & Trick, D. (2013). How Online Learning Affects Productivity, Cost and Quality in Higher Education: An Environmental Scan and Review of the Literature. Toronto: Higher Education Quality Council of Ontario

Why this paper is important

In July, the Higher Education Quality Council of Ontario published the above report. This is a very important development for online learning in post-secondary education as it takes a very hard look at quality, cost and productivity and comes forward with recommendations to government. This is a paper that is likely to be read (and should be read) by legislators, state and government policy makers, university and college boards and senior university and college administrators.

I am also exploring through a series of blogs the issue of productivity and online learning, partly because of dissatisfaction with the current state of thinking about this issue, which became apparent working with this project.

For this reason, I am setting aside my hat as an Advisory Board member who commented on the penultimate draft, and and am here providing a full analytic review of the paper. To do this, I have had to reproduce key parts of the document, but I strongly recommend that the HEQCO document is read in full. Quotes from the actual paper are in italics, although I have edited and abbreviated in part.

The paper focuses on the following questions:

• What are the cost implications of a shift to online learning? Specifically, does a greater use of online instruction save institutions or systems money and, if so, under what circumstances?
• What do we know about the relationship between online learning and important variables that are often considered when discussing the “quality” of an institution or of a system?

Main findings

• The evidence reviewed suggests that, for a range of students and learning outcomes, fully online instruction produces learning that is on par with face-to-face instruction.
• the students most likely to benefit are those who are academically well prepared and highly motivated to learn independently. Students who are not well prepared to learn at the postsecondary level or do not devote the necessary time to learning are less likely to benefit from online learning and may in fact do better in a face-to-face setting.
• the provincial government… should have an interest in making sure [well-prepared and motivated students] have online learning opportunities available to them. These opportunities should serve students’ learning needs, and – if carried out at large scale – should produce cost efficiencies for higher education institutions, the student or both.
• there is no evidence that all of the learning outcomes expected of postsecondary students in Ontario can be achieved solely by online learning. 

Main recommendations to the Ontario provincial government and Ontario universities and colleges

• set a target that, within three years, a specified list of high-demand university and college programs that are primarily or entirely online will be available to Ontario students.
• set a target that, within three years, a specified list of high-demand courses will be available online and will be accepted for credit at all Ontario universities and colleges that offer a program in that discipline.
• a set of high-quality degree programs that qualify the student for admission to any Ontario graduate school, and a set of high-quality courses that are accepted for credit by every Ontario institution, will be preferable to a multiplicity of courses and programs that operate on a small scale.
• By working with other institutions in Ontario and elsewhere, Ontario colleges and universities can leverage and help shape emerging developments in online learning.
• Coordination will be required to ensure that economies of scale are achieved in an environment of rapid technological change. 
• Ontario colleges and universities should be encouraged to work with peer institutions to ensure that engagement with advances in online learning fully supports the province’s strategic goals for quality and access in a time of constrained funding. 
• An effective government strategy will begin by adapting existing regulatory infrastructure to remove unnecessary barriers to high-quality online education. 
• Hybrid courses that blend online learning with face-to-face instruction should also be encouraged where they improve learning outcomes. Hybrid courses fit well within the government’s existing regulatory structure and so present fewer policy challenges. 

Methodology

We have looked especially for meta-analyses which compare traditional versus online education at a system, course or activity level. We have made only secondary use of studies and reports from individual instances or instructors where institutionalization and sustained use have not been addressed.

Costs:

There is remarkably little empirical literature that documents the costs of online education relative to face-to-face education. So very little evidence on costs is available in this report

The authors though do provide an extensive list of barriers to cost reduction.

The authors conclude this section as follows:

To the extent that online education reduces costs, there is no consensus about who should or would benefit from the reduction. Students seek lower tuition fees; governments seek reduced subsidies for higher education; university employees seek better compensation. This situation presents a principal-agent problem: it is difficult to motivate change when those affected by change will not receive the contemplated financial benefit.

Emerging developments

The following emerging developments are discussed:

• Affordable and open textbooks
• Adaptive interactions with learning resources
• Optimizing student-instructor interaction time
• Targeting instructional effort based on student program data
• Minimizing marginal costs via Massive Open Online Courses

The authors also identify several common themes across the individual developments:

• Aligning Support to the Student’s Individual Learning Needs
• “Thinking Globally, Acting Locally” to Achieve Benefits at Scale
• Transparency and Knowledge Intensity in Instructional Design
• Reputational Capital From and For Online Learning
• The Challenge of Investment at/for Scale

Observations (for recommendations, see above)

Fully online education presents opportunities for major economies of scale. By definition, these economies can only be achieved if a large scale is reached.

Fully online education has the potential to provide a high-quality education – for some students, in some fields of study – at significantly lower unit costs than traditional forms of instruction. The cost savings have the potential to help fund the cost of improving traditional learning, including the costs of introducing hybrid models that lead to better learning outcomes. The challenge is to make it happen.

What is striking to us about these viewpoints is the agreement that what is least likely to be done effectively at scale and with technological mediation is precisely what matters most in higher education.

My critique

Overall, this is an excellent report that will be valuable to policy makers, if they read it in full. The danger is that they will jump to the recommendations, which are not really the strength of this report. Its value lies in exploring assumptions and beliefs about online learning and productivity and providing data and evidence that sometimes supports such beliefs, and other times challenges them. The section on emerging developments is particularly strong, especially the analysis of common themes across the individual developments.

Comparative quality of online learning

Although the authors focused their literature review on ‘meta-analyses of rigorous experimental studies’, the result is a master lesson on why such studies are usually a waste of time, particularly with regard to ‘quality’ defined in this report as to whether online learning achieves equal learning outcomes to face-to-face teaching. Such studies on using different media and technologies to deliver education date back until the early 1970s, and results are consistent: mode of delivery is less important than method of teaching and multiple other factors. In statistical terms, variance within experimental groups is larger than variations between experimental groups. In plain language, the pedagogy matters, a point recognized by the authors later in the document when they acknowledge the importance of instructional design.

This is one reason why I am cautious about the research on ‘non-traditional’ students that suggests that online learning works less well for them. While I do not disagree with this in general, it can work well for some in this group when designed to meet their specific needs. The problem is that the Jaggars and Di Xu research quoted to support the conclusion in the HEQCO report is based on data from U.S. community colleges, many of which have a very poor record of using instructional design and best practices in online learning. You have to look at the quality of the teaching (in both modes), not just the delivery method.The HEQCO authors also correctly note that while many of Jaggars/Di Xu findings point to performance differences between online and face-to-face learning that are statistically significant, the differences are fairly small.

Costs

This is by far the most disappointing part of the study. The report draws on only two actual studies of the costs of online learning (both from the USA), neither of which are very helpful.

For reasons of time pressure and consistency, the authors decided to limit their research review to studies published in the last five years. As a result, studies such as my own on the cost of the University of British Columbia’s fully online Master in Educational Technology (which was originally published in 2003) are not included, even though the study provides a comprehensive analysis of the costs and more importantly the cost structures of a program that is still running on much the same cost basis as in 2003. This program has been remarkably successful with the following features:

• fully cost-recoverable (including overheads and planning) from tuition fees alone
• tuition fees the same as for on-campus graduate programs (fee level regulated by government)
• over 300 students in the program each year with over 900 course enrolments
• courses can be taken and paid for individually
• 70-80 admissions a year, and 70-80 graduates a year, thus with a degree completion rate (for those enrolling in the full degree program) of over 90%.

This program alone has more than doubled the number of graduate students in the whole of the Faculty of Education and UBC has adopted this cost model for a number of its other professionally based masters programs, such as rehab science and creative writing. Not to include this because the study was done 10 years ago is almost perverse, because it shows that for certain kinds of courses, and certain kinds of students, online learning can be far more productive than face-to-face teaching. It is perverse, because real productivity gains only become apparent over time – a five year window is often too small to see the full benefits.

Emerging developments

For me, this was by far the strongest part of the paper, particularly the analysis of common themes across the developments. The paper is worth reading for this section alone.

Recommendations

Although I would support all the recommendations, they are very cautious.Partly because of the weakness or lack of research into online learning, costs and productivity, the recommendations necessarily have to be cautious.

However, since HEQCO itself is a government-funded policy research organization, perhaps an obvious recommendation would have been for more research on the costs of online learning, given the paucity of studies. Another area for research would be on institutional barriers and government policies that prevent greater scalability or adoption of online learning in Ontario universities.

It is still shocking to me that Ontario has such a poor system of credit transfer even between universities that make it almost impossible to set up consortium programs or enable student students to select combinations of courses/programs from different universities, given that a main advantage of online learning is that students could take courses from any university in Ontario. Maybe government regulation is necessary in this area, since the universities and colleges were given $65 million I believe over a year ago to solve this problem and haven’t done so yet.

None of the recommendations really addresses the issue of scale. I’m not sure I agree with the statement on p. 43:  What is striking to us about these viewpoints is the agreement that what is least likely to be done effectively at scale and with technological mediation is precisely what matters most in higher education, i.e. modelling, coaching, enabling students to construct knowledge, etc. Certainly, most MOOCs don’t do this, but it doesn’t necessarily mean that with a focused effort on instructional design, we could not design more cost-effective, high quality learning experiences through online programs on a larger scale than at present but not necessarily at massive level. This would combine lower cost per student with higher quality learning: the Nirvana of educational productivity

Thus I would like to have seen a recommendation to government and the institutions to put in the same level of investment as for MOOCs, but to develop a model that combines best practices in online learning combined with new technologies such as social media, to build partly self-supporting student learning communities on a larger scale than current campus-based programs, with high quality learning outcomes and completion rates. I think it could be done, but it needs substantial investment beyond the risk level of most individual universities, which is why government should be a partner.

Conclusion

Despite my criticisms this is an excellent report on a difficult topic and completed within a tight timeframe. It provides grist for productive discussions on costs and quality and really advances our understanding of the challenges of increasing productivity without losing quality in higher education.

Reference

Bates, A. and Sangra, A. (2013) Managing Technology in Higher Education San Francisco: Jossey-Bass (especially Chapter 7: Resources, Money and Decision-Making).