Engle, W. (2104) UBC MOOC Pilot: Design and Delivery Vancouver BC: University of British Columbia

The University of British Columbia, a premier public research university in Canada, successfully delivered five MOOCs in the spring and summer of 2013, using the Coursera platform. This report is an evaluation of the experience.

The report is particularly valuable because it provides details of course development and delivery, including media used and costs. Also UBC has been developing online courses for credit for almost 20 years, so it is interesting to see how this has impacted on the design of their MOOCs.

The MOOCs

1. Game Theory I: K. Leyton Brown (UBC); M. Jackson and Y.Shoham (Stanford University)

2. Game Theory II: K. Leyton Brown (UBC); M. Jackson and Y.Shoham (Stanford University)

3. Useful Genetics: R. Redfield, UBC

4. Climate Literacy: S. Harris and S. Burch, UBC

5. Introduction to Systematic Program Design: G. Kizcales, UBC

In terms of comparability I’m going to treat Game Theory I and II as one MOOC, as combined they were about the same length as the other MOOCs (between 8-12 weeks)

Basic statistics

330,150 signed up (82,500 on average per course)

164,935 logged in at least once (41,000 per course)

12,031 took final exam (3,000 per course)

8,174 earned course certificate (2,000 per course)

60-70% already had a post-secondary degree

30-40% were North American, with participants from nearly every country in the world.

Course development

None of the instructors had taught an online course before, but were supported by instructional designers, media development staff, and academic assistants (graduate and undergraduate students).

One major difference between UBC MOOCs and its online credit courses (which are primarily LMS-based) was the extensive use of video, the main component of the MOOC pilot courses.

Video production

305 videos constituting a total of 65 hours were produced. Each MOOC used a different method of production:

• Intensive studio (Climate Literacy)
• Hybrid studio plus instructor desktop (Systematic Program Design)
• Light studio production (Game Theory I and II)
• Instructor desktop (Useful Genetics)

Web pages

All the MOOCs except Games Theory also included weekly modules as HTML-based web pages, which is a variation of the Coursera design default model. Altogether 98 HTML module pages were developed. The weekly modules were used to provide guidance to students on learning goals, amount of work expected, an overview of activities, and additional quiz or assignment help. (All standard practice in UBC’s LMS-based credit courses.)

Assessment

1,049 quiz questions were developed, of which just over half were graded.

There were four peer assessments in total across all the MOOCs.

Course delivery

As well as the faculty member responsible for each MOOC, graduate and undergraduate academic assistants were a crucial component of all courses, with the following responsibilities:

• directly assisting learners
• troubleshooting technical problems
• conducting quality assurance activities

There was very little one-on-one interaction between the main instructor and learners, but academic assistants monitored and moderated the online forum discussions.

Costs

As always, costing is a difficult exercise. Appendix B of the report gives a pilot total of $217,657, but this excludes academic assistance or, perhaps the most significant cost, instructor time.

Working from the video production costs ($95,350) and the proportion of costs (44%) devoted to video production in Figure 1 in the report, I estimate the direct cost at $216,700, or approximately $54,000 per MOOC, excluding faculty time and co-ordination support, but including academic assistance.

However, the range of cost is almost as important. The video production costs for Climate Literacy, which used intensive studio production, were more than six times the video production costs of Systematic Program Design (hybrid studio + desktop).

MOOCs as OERs

• the UBC instructors are using their MOOC materials in their own on-campus, for-credit classes in a flipped classroom model
• courses are left open and active on Coursera for self-paced learning
• porting of video materials as open access YouTube videos
• two courses (Climate Literacy and Useful Genetics) added Creative Commons licenses for re-use

Challenges

• copyright clearance (Coursera owns the copyright so third party copyright needs to be cleared)
• higher than expected time demands on all involved
• iterative upgrades to the Coursera platform
• partner relationship management (UBC + Coursera + Stanford University) was time-consuming.
• training and managing academic assistants, especially time management
• the Coursera platform limited instructors’ ability to develop desired course activities
• Coursera’s peer assessment functionality in particular was limiting

Lessons

• UBC’s prior experience in credit-based online learning led to better-designed, more interactive and more engaging MOOCs
• learners always responded positively to instructor ‘presence’ in forums or course announcements
• MOOC students were motivated by grades
• MOOC students were willing to critically engage in critiquing instructors’ expertise and teaching
• open publishing via MOOCs is a strong motivator for instructors
• MOOCs require significant investment.

Conclusion

All the MOOCs received positive feedback and comments from students. UBC was able to gain direct experience in and knowledge of MOOCs and look at how this might inform both their for-credit on-campus and online teaching. UBC was also able to bring its experience in for-credit online learning to strengthening the design of MOOCs. Lastly it was able to make much more widely known the quality of UBC instructors and course materials.

Comment

First, congratulations to UBC for

• experimenting with MOOCs
• conducting the evaluation
• making the report publicly available.

It is clear from the comments of participants in the appendices that at least some of the participants (we don’t know how many) were very pleased with the courses. As usual though with evaluation reports on MOOCs, there is no assessment of learning other than the end of course quiz-based tests. I don’t care too much about completion rates, but some measurement of student satisfaction would have been helpful.

It is also significant that UBC has now decided to move from Coursera to edX as its platform for MOOCs. edX, which is open source and allows partners to modify and adapt the platform, provides the flexibility that Coursera lacked, despite its many iterative ‘improvements’.

This also demonstrates the hubris of MOOC platform developers in ignoring best design principles in online learning when they designed their platforms. It is clear that UBC designers were able to improve the design of their MOOCs by drawing on prior for-credit online experience, but also that the MOOC platforms are still very limited in enabling the kind of learning activities that lead to student engagement and success.

The UBC report also highlighted the importance (and cost) of providing some form of learner support in course delivery. The use of academic assistants in particular clearly made the MOOCs more interactive and engaging, as well as limited but effective interventions from the instructors themselves, once again supported by (and confirming) prior research on the importance of instructor presence for successful for-credit online learning.

I very much appreciate the cost data provided by UBC, and the breakdown of production and delivery costs is extremely valuable, but I have to challenge the idea of providing any costs that exclude the time of the instructors. This is by far the largest and most important cost in MOOCs and the notion that MOOCs are free of instructor cost is to fly in the face of any respectable form of economics.

It is clear that MOOCs are more expensive to date per hour of study time than LMS-based for-credit online courses. We still do not have enough data to give a precise figure, and in any case, as the UBC study shows, cost is very much a factor of design. However, even without instructors costs, the UBC MOOCs at $54,000 each for between 8-12 weeks are already more than the average cost of a 13 week for-credit LMS-based online course, including instructor time.

This is partly due to the increased instructor time in preparation/production, but also to the higher cost of video production.  I am not against the use of video in principle, but it must add value. Using it for content transmission when this can be done so much more cheaply textually and/or by audio is a waste of the medium’s potential (although perhaps more motivating for the instructor).

More importantly, every institution contemplating MOOCs needs to do a cost-benefit exercise. Is it better to invest in MOOCs or credit-based online learning or both? If MOOCs are more expensive, what are the added benefits they provide and does this more than make up for not only the extra cost, but the lost opportunity of investing in (more) credit-based online learning or other forms of campus-based learning? I know what my answer would be.