Improving productivity in online learning: can we scale ‘the learning that matters most’?

The story so far

This is a continuation of the discussion on whether online learning can increase educational ‘productivity.’ Previous posts in this series include:

• Technology, teaching and productivity: the need for theory
• Is there a link between flexible access and ‘productivity’ in higher education? 
• Book review by Sir John Daniel: Higher Education in the Digital Age
• A review of the HEQCO report on productivity and quality in online learning in higher education
• Tom Carey’s reflections on the HEQCO report on online learning and productivity: 1-Catching a teachable moment
• Tom Carey’s reflections on the HEQCO report on online learning and productivity: 2 – What we left out – and why.
• Towards a theory or model of productivity for online learning: outcomes, scale and design
• How online content development and delivery could improve the productivity of post-secondary education
• Alternative ways to improve productivity through online learner-content interaction

There is a CIDER webinar presentation on the HEQCO report available from here

In the last post, I concluded:

• there are major economies of scale in using computer-based feedback for facilitating comprehension and technical mastery outcomes
• computer-based feedback, when well designed, can also be useful in providing student feedback for more complex forms of learning, such as alternative strategies, critical thinking and evaluation
• however computer-based analyses to date are inadequate for formal assessment of these higher order learning skills, where deep expertise and qualitative assessment is required, and where learners may provide new insights or alternative explanations
• redesign of courses with a greater focus on student discovery (finding, analyzing and applying content) within a learning design offers more modest but still significant potential for increases in productivity, mainly through better learning outcomes (development of 21st century skills) and through more effective use of senior research professors’ time.

Learner-instructor interaction and economies of scale

In this current post, I examine particularly the learner-instructor interaction, and discuss whether online learning can provide economies of scale in this area. This is particularly important, because research on credit-based online learning has shown that course delivery (which includes both learner support and student assessment) accounted for the largest overall cost of an online program (37%), almost three times more than course development, over the life of an online program (Bates and Sangra, 2011).

Can we scale ‘the learning that matters most’?

This important question has been raised in the HEQCO report by Tom Carey and David Trick. It is this issue I wish to address here, since scaling up the delivery of content, and learner-content interaction, through online learning is relatively easy, although both depend on good course design for effective learning.

What is more challenging is whether we can also scale the kind of ‘learning that matters most’, namely helping students when they struggle with new concepts or ideas, helping students to gain deep understanding of a topic or subject, helping students to evaluate a range of different ideas or practices, providing students with professional formation or development, understanding the limits of knowledge, and above all enabling students to find, evaluate and apply knowledge appropriately in new or ill-defined contexts.

Before looking at whether or not such activities can be scaled, it is important to challenge the view, such as Sanjay Sharma’s at MIT, that such forms of learning can only be achieved on campus. There is also more than a hint of this assumption in the HEQCO report, at least with respect to undergraduate education. Those of us who have taught online will know that it is possible to develop these kinds of learning outcomes online, especially but not exclusively at graduate level. Strategies such as scaffolding or supporting knowledge construction through online discussion and dialogue, student reflection through e-portfolios, and above all personal online interventions and communication between students and instructor, have all been found to lead to learning outcomes at least as equivalent to those of students studying the same subjects on-campus (see references below).

There will remain a relatively few learning activities that matter most that are best done on campus, such as the development of hands-on skills, but there will be others, such as knowledge management, that may well be best done online. More importantly, there will be some students who really need the environment provided by a campus, and others that will prefer an online environment.

The issue is not can the learning that matters most be done online, but can it be scaled up through online learning? Certainly, I would argue that the main criticism of xMOOCs is that they spectacularly fail to address this form of learning. However, cMOOCs, when they operate at the level of communities of practice with relatively shared levels of understanding and knowledge among the participants, do have at least the potential for such economies of scale while maintaining or even improving quality of learning outcomes. The challenge though is how one accounts for the hidden costs of the participation of experts in such professional sharing, which rely heavily on volunteering or ‘moonlighting’ from a paid job by those with the expertise. I suspect though that even if these costs were calculated, they would still prove more ‘productive’ than conventional campus-based classes for this type of learner. However, the cost-effectiveness research has yet to be done.

The challenge though is scaling up the kinds of interaction between students and instructors that enable diagnosis of a student’s learning difficulties, that facilitate deep understanding of a subject, that encourage creative and original thinking, especially within undergraduate education. Adaptive learning and learning analytics may help to some extent, but in my view cannot yet come close to matching the skill of an experienced and skilled instructor. If instructors are to have enough time to engage in these kinds of dialogue and communication with students, there is clearly a limit on the number of students they can handle. Thus there is a possibility of small increases in productivity, aided by developments such as adaptive learning and learning analytics, but not major ones, in this aspect of teaching and learning.

Scaling the assessment of ‘learning that matters most.’

When ‘the magic of the campus’ is raised, one of the implicit assumptions is that student assessment is more valid because of the personal knowledge that faculty develop of a student in their entirety, and not just in their formal academic work: how they conduct themselves in class discussion (not just what they say, but how they say it), their interests and knowledge outside the formal curriculum (e.g. do they read widely or participate in valued extra-curricula activities), and the impression students make in social activities with faculty. This ‘tacit’ knowledge of a particular student that faculty acquire on campus can heavily influence the final assessment of a student, beyond that of the final exam. As they say at Oxford University, ‘Is he one of us?’

I was fortunate to have done my undergraduate degree in a department where every ‘honours’ student was well known by every faculty member. We were told that in the final exam, we could not get a worse grade than was already determined, but we could improve on it by a really good performance. In other words, the final exam was more of a rite of passage – the assessment was already more or less in place. This was only possible because of the ‘deep’ knowledge that faculty had already gained of the students. The fear that many faculty have of of online learning is that this kind of knowledge of a student is impossible ‘at a distance.’

Again, however, at least some elements of this ‘getting to know students’ can be achieved online, through continuous assessment, the use of e-portfolios and participation in online discussions. Again, the similarities between online learning and campus teaching are often greater than the differences. The problem is scaling up this kind of in-depth academic relationship between student and instructor, both for classroom and online teaching. Although the actual ratio may be difficult to specify, it is clear that this kind of relationship cannot be built up if the instructor:student ratio is in the thousands.

The fact is though that undergraduate students in most public universities are not in the fortunate position that I was. Even in their final year, many find themselves are in classes of over 100 students. They will probably be better off in an online class of 30 students, and even in an online class of 100, they may have more personal interaction with the instructor than in a lecture theatre, if the course is well designed. However, scaling up much beyond this ratio is not going to enable the more personal intellectual relationship to develop that allows for the more informal ‘I know what this student is capable of’ relationship, either online or on campus.

In short, for assessment based on deep knowledge of a student’s progress and capabilities, the scope for economies of scale are limited. In this sense, teacher:student ratios do matter, so economies of scale through online learning will be difficult to achieve for these higher order learning skills.

Conclusions

This has been a particularly difficult blog to write which suggests I may still not be thinking clearly about this topic, so please help me out! However, here is where I stand on this issue so far:

1. The ‘learning that matters most’ mainly addresses university teaching, but I suspect also increasingly technical, vocational and corporate training; the aim is to develop the knowledge and skills needed in a knowledge-based society.

2. Online learning can handle the ‘learning that matters most’ as well, in most cases, as on-campus teaching, although there will always be some exceptions.

3. However, there are major difficulties in scaling up the learner support and assessment activities that are needed for the learning that matters most, both online or on campus. The danger in scaling up is the loss of quality in terms of learning outcomes.

4. Adaptive learning software that helps individualize learning, and learning analytics, may help to a small degree in enabling instructors to handle slightly more students without loss of quality, but cannot as yet replace a skilled instructor, and probably never will.

5. New online course designs built around the use of new technologies have greater potential for increases in productivity – through producing better learning outcomes – for the learning that matters most, than through scaling up, i.e. by increasing teacher:student ratios.

6. We need more empirical research on the relationship between teaching methods, mode of delivery, costs, and the type of learning outcomes that constitute the ‘learning that matters most’ (not to mention better definitions).

Your input

First I’d really welcome responses to this post. In particular:

• Is ‘the learning that matters most’ a useful concept for university teaching? Do you agree with my descriptions of it?
• Have I missed something obvious in the possibility for scaling these learner support and assessment activities?
• Can adaptive learning software and learning analytics take some or all of the load off instructors in developing such learning outcomes?
• What would new online course designs that increase productivity look like? Do you have actual examples that have been implemented?

Next

In my next post on this topic, I will discuss an area where I think there is huge potential for increasing productivity through online learning, and that is through savings in physical overheads.

References

Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. Journal of Asynchronous Learning Networks, Vol. 5, No.2.

Baker, C. (2010) The Impact of Instructor Immediacy and Presence for Online Student Affective Learning, Cognition, and Motivation The Journal of Educators Online Vol. 7, No. 1

Bates, A. and Sangrà, A. (2011) Managing Technology in Higher Education: Strategies for Transforming Teaching and Learning San Francisco: Jossey-Bass/John Wiley and Son

Garrison, D. R. & Cleveland-Innes, M. (2005). Facilitating cognitive presence in online learning: Interaction is not enough. American Journal of Distance Education, Vol. 19, No. 3

Harasim, L. (2012) Learning Theory and Online Technologies New York/London: Routledge

Jonassen, D., Davidson, M., Collins, M., Campbell, J. and Haag, B. (1995) ‘Constructivism and Computer-mediated Communication in Distance Education’, American Journal of Distance Education, Vol. 9, No. 2, pp 7-26.

Paloff, R. and Pratt, K. (2007) Building Online Learning Communities San Francisco: John Wiley and Co.

Richardson, J. C., & Swan, K. (2003). Examining social presence in online courses in relation to students’ perceived learning and satisfaction. Journal of Asynchronous Learning Networks, 7 (1), 68-8 8.

Salmon, G. (2000) E-moderating London/New York: Routledge

Sheridan, K. and Kelly, M.  (2010) The Indicators of Instructor Presence that are Important to Students in Online Courses MERLOT Journal of Online Learning and Teaching, Vol. 6, No. 4