Screen shots from UBC’s Soil Sciences TopAGraphy mobile app

I am in the process of finalising a second edition of Teaching in a Digital Age. (For more on this, see Working on the second edition of Teaching in a Digital Age: some lessons learned about open publishing.) I will post over the next few weeks the relatively few new sections in the book for comments and feedback.

For the former Chapter 7 (now Chapter 8), on Pedagogical differences between media, I am adding a new section in three parts on emerging media, in particular serious games, virtual/augmented/mixed reality, and artificial intelligence.

This is my draft of the section on virtual/augmented/mixed reality. I am indebted to Saeed Dyanatkar, Executive Producer, UBC Emerging Media Lab, and many others for their help in providing material for this chapter. However, further suggestions, criticisms and constructive feedback will be truly appreciated.

8.7.b Virtual and augmented reality 

As with serious games, virtual and augmented reality are technologies that have been around for some time while making a relatively small impact on education in their earlier development. However more recent technological developments that have moved virtual worlds from two-dimensions (such as Second Life) into three-dimensional, deeply immersive environments have brought more attention to their potential in education (for a good overview of the history and potential of augmented and virtual reality in education, see Elmqadden, 2019).

8.7b.1 What are virtual/augmented/mixed reality?

A simple definition of these technologies is ‘human immersion in a synthetic world‘ (Seidel and Chatelier, 1997). The Franklin Institute provides the following more detailed definitions that attempt to distinguish between the different types of ‘synthetic’ worlds:

Augmented reality (AR) adds digital elements to a live view often by using the camera on a smartphone. Examples of augmented reality experiences include Snapchat lenses and the game Pokémon Go. 

Virtual reality (VR) implies a complete immersion experience that shuts out the physical world. Using VR devices such as HTC Vive, Oculus Rift or Google Cardboard, users can be transported into a number of real-world and imagined environments such as the middle of a squawking penguin colony or even the back of a dragon.

In a mixed reality (MR) experience, which combines elements of both AR and VR, real-world and digital objects interact. Mixed reality technology is just now starting to take off, with Microsoft’s HoloLens one of the most notable early mixed reality apparatuses.

I will use the term ‘immersive technologies’ for all these technologies. However, verbal descriptions will always be somewhat inadequate in describing what are essentially multi-sensory experiences, combining vision, hearing and movement. These technologies are something that need to be experienced rather than explained if they are to be better understood. 

8.7b.2 Why use immersive technologies?

There are several reasons why these technologies are beginning to be used more in education:

• the recent development of relatively low cost and easily wearable end-user technology (headsets in particular);
• deep immersion into three-dimensional, highly realistic learning environments that are strongly compelling/motivating for the end user;
• the ability for end users to manipulate objects within the three dimensional environment;
• more powerful cloud computing technology that allows for the development of more complex and more realistic learning environments, combined with more advanced developments in mobile technologies and high-speed wireless networks;
• the potential for developing a range of skills and knowledge that would be difficult, impossible or dangerous in real-world environments.

8.7b.3 Designing immersive educational environments

This technology is so recent that there are few or no accepted best practices developed yet for educational use. Most educational applications to date have been deliberately exploratory in nature. However, there are several stages of development required that will apply to all educational applications of these technologies:

• identify start-up costs and possible sources of funding: this is not likely to be a cheap exercise, at least initially; for this reason, several universities, such as the University of British Columbia, and Drexel University, have set up their own emerging technologies research labs to experiment with educational applications;
• define learning outcomes/objectives: what is the learner expected to learn? In the early stages of development this may be both a brainstorming exercise (preferably including students/end-users) and an iterative process, because the full potential of the technology is not always clear in first applications. In particular, the instructor needs to have a clear vision of what might be possible using an immersive technology. Thus some familiarity with the technology is essential before starting design;
• determine where the use of this technology fits within the overall design of a course/program: in other words, what knowledge and skills will be developed within the immersive environment, and how does this integrate with what is being taught in the rest of the course/program?
• decide between using an existing immersive design/learning environment that can be applied or adapted relatively easily for ‘local’ use; or designing a new immersive environment from scratch. The latter is obviously more expensive and time-consuming and will require a high level of expertise; as a result the pay-off from design from scratch (improved learning outcomes/return on investment) needs to be worth the effort;
• choice of appropriate/affordable technology. Headsets or mobile apps are the least expensive part of the use of immersive technologies. The main cost will be in developing or adapting the ‘augmented’ or ‘virtual’ world. However, as with serious games, there can be an intermediary step, where an existing ‘world’ can be licensed and adapted for local use (see for instance, Lightwave). In some cases, open access immersive worlds are available for use or adaptation, although they are not common (see OpenSimulator, Art of Illusion, or MayaVerse, for examples.). Often students can be used to help with programming and design of the environment, as part of their studies, but they will need direction as well as the opportunity to offer creative ideas. Truly interactive virtual worlds where learners/users make decisions and the consequences are ‘programmed’ into the learning environment may require large amounts of computing capacity, such as cloud computing;
• to be effective, the VR environment has to be as authentic or realistic as possible. This means paying as much attention to creating the specific learning context. It will be necessary to decide what parts of the learning will best be done outside the VR/AR experience, and which inside. For instance, the procedures for monitoring the state of a nuclear reactor, for identifying critical incidents, for deciding whether or not or when to shut down the reactor, and for actually shutting down the reactor must also be built in to the learning process. Most of this may be taught outside the VR context, but VR can be used to test or develop the skills of applying this knowledge in a realistic, challenging context. In other words, the VR experience needs to be embedded within a broader learning context or environment;
• testing and adaptation: design, at least initially, needs to be an iterative process, where ideas are developed and tried, and feedback received and incorporated into the design;
• assessment: this can be a particular challenge, particularly if new learning outcomes result from the experience. How can assessment best capture what students have learned? Will assessment take place within the ‘virtual’ world, in the real world, or in some other way (and if so, how authentic will such an assessment be)?
• in what ways could the new immersive environment be scaled up to enable costs to be recovered?
• evaluation: what is the best way to evaluate the success or limitations of the design and application of the immersive world? How best to disseminate the knowledge and experience gained?

These may appear formidable challenges, but the potential benefits could be considerable.

8.7b.4 Examples of immersive environments in education

Looking at the challenges above, it may be wondered why anyone would bother with immersive technologies in education. However, the potential benefits have barely been explored. I provide examples here that demonstrate both the potential benefits and how some immersive environments can be developed relatively easily.

8.7b.4.1 Virtual reality

In the Department of Chemistry at the University of Bristol in England, Dr. David Glowacki and his team in their VR laboratory created an interactive molecular dynamics modelling tool in the form of Nano Simbox VR, which allowed anyone to visit and play within the invisible molecular world (O’Connor et al., 2018). The main aim of this particular project was to provide an intuitive feeling of the way molecules operate in multiple dimensions to enable researchers and students to have a better understanding of how nano worlds operate, leading to better hypotheses for testing within this particular domain.

As the authors state in the article:

From a modeling perspective, the nanoscale represents an interesting domain, because the objects of study (for example, molecules) are invisible to the naked eye, and their behavior is governed by physical forces and interactions significantly different from those forces and interactions that we encounter during our day-to-day phenomenological experience. In domains like this, which are imperceptible to the naked eye, effective models are vital to provide the insight required to make research progress….molecular systems typically have thousands of degrees of freedom. As a result, their motion is characterized by a complicated, highly correlated, and elegant many-body dynamical choreography, which is nonintuitive compared to the more familiar mechanics of objects that we encounter in the everyday physical world. Their combined complexity, unfamiliarity, and importance make molecules particularly interesting candidates for investigating the potential of new digital modeling paradigms.

Glowacki and his team in Science Advances (O’Connor et al., 2018) describe how the VR app enabled researchers to:

• easily “grab” individual C₆₀ atoms and manipulate their real-time dynamics to pass the C₆₀ back and forth between each other.
• take hold of a fully solvated benzylpenicillin ligand and interactively guide it to dock it within the active site of the TEM-1 β-lactamase enzyme (with both molecules fully flexible and dynamic) and generate the correct binding mode (33), a process that is important to understanding antimicrobial resistance
• guide a methane molecule (CH₄) through a carbon nanotube, changing the screw sense of an organic helicene molecule,
• tie a knot in a small polypeptide [17-alanine (17-ALA)

Building dynamic models that operate not only in real time but also in three dimensions can require not only specialized virtual reality equipment, but more importantly massive amounts of computing power to handle the visual representation and modelling of highly complex, interactive dynamic molecular processes. However, through the use of cloud computing and faster networks, building such models has now become a reality, enabling not only such models to be represented but allowing some degree of real-time manipulation by researchers in different locations but within the same time-frame. The main advantage of the use of a cloud platform is to allow the scaling up of modelling from simple to much more complex dynamic nano interactions and the synchronous sharing of the virtual reality experience with multiple users. 

Not all applications of VR though need massive computing power. Other exploratory uses of virtual reality are

• for students to find their way round a complex campus
• in architecture/space planning, allowing clients to understand in three dimensions the final ‘look’ of a building design by virtually walking through it (Brandaõ et al., 2018), Google Blocks, a free software program for developing 3D models, is one tool that can support this kind of application.
• in music: at the University of British Columbia, Dr. Jonathon Girard is exploring the use of VR for learning how to conduct an orchestra (the virtual orchestra ‘responds’ to the hand gestures of the conductor)
• in medicine and health: researchers at UBC are exploring the use of VR for pain management

8.7b.4.2 Augmented reality

Augmented reality is a simpler immersive technology than virtual reality, often based on apps for mobile phones. For instance, students in the University of British Columbia’s APBI 200 Introduction to Soil Science learn about the effects of topography on the formation of different soil types. The department has developed the Soil TopARgraphy app, which allows viewing and manipulating a terrain model in the Kamloops region of British Columbia. Students learn how topography impacts the distribution of soil orders through its effects on microclimate (i.e. temperature and water). Students are able to view the terrain model with a color-coded elevation map or a satellite image on their mobile phones. Furthermore, students can tap on flags to read about different soil orders, view images, and take a self-study quiz to reinforce their understanding.

For this project, UBC’s Emerging Media Lab built two mobile apps, an AR viewer for students (Android and iOS) and an editor for the instructor (Android). The AR viewer is the app described above to view a predefined terrain. The instructor can customize contents with the supplementary editor app. They can update soil location on terrain, description, image, and quizzes

Figure 8.7.c.3 Screen images from Soil TopARgraphy

Other examples of AR applications from UBC:

• Dr. Patrick Walls is developing a mobile phone-based app that helps students visualise multivariable functions, in order to learn the underlying concepts at a deeper level much more quickly.
• in GEOG 498: Geographies in the Middle East, students learn about the history of the Syrian Civil War and its ongoing developments. The instructor, Dr. Siobhán McPhee, has developed a mobile app that follows the stories of five Syrian refugees who eventually reached Vancouver. Students are forced to make choices (or given a lack of choice), wait, and run/walk with the app to be able to progress the narrative of the experience. The purpose of this project is to evoke empathy and help students understand the emotional consequences of the Syrian Civil War. This app also applies some gamification principles as well.

8.7b.5 The unique characteristics of immersive technologies

The development of fully immersive technologies is so recent that it is premature to try to identify all the educational affordances that are unique to this medium. New applications are being explored all the time. Most of the evidence is qualitative, based on people’s personal experience of using the technology. Empirical evidence that validates specific educational affordances of VR/AR in terms of improved learning outcomes is currently lacking. However, the potential of VR/AR in terms of assisting learning can be identified.

First of all, many of the affordances or educational characteristics of other media, and in particular video, will apply to VR and AR, but often more intensely, because of the immersive experience.

Virtual and augmented reality applications can provide students with a deep, intuitive understanding of phenomena that are otherwise difficult if not impossible to achieve in other ways. This enables students who often struggle with the abstract nature of an academic subject to understand in more concrete terms what the abstractions mean or represent. This intuitive understanding is critical not only for deeper understanding but also for breakthroughs in research and applications of science.

Educational applications where the cost of alternative or traditional ways of learning are too expensive or too dangerous, will be particularly suitable for virtual reality applications. Examples might be emergency management, such as shutting down an out-of-control nuclear reactor, or defusing a bomb, or managing a fire on an oil tanker, or exploring inside the physical structure of a human brain. In particular, VR would be appropriate for learning in contexts where real environments are not easily accessible, or where learners need to cope with strong emotions when making decisions or operating under pressure in real time.

AR, which is often easier to design and implement, enables learners to practice applications of knowledge in semi-realistic contexts.

However, at the time of writing we are just beginning to understand the potential of this medium. Over time, the educational affordances of this medium will become much clearer.

8.7b.6 Strengths and weaknesses

VR is not just a fad that will disappear. There are already a large number of commercial applications, mainly in entertainment and public relations, but also increasingly for specific areas of education and training. There is already a lot of excellent, off-the-shelf software for creating VR environments, and the cost of hardware is dropping rapidly (although good quality headsets and other equipment are still probably too expensive for required use by large numbers of students).

The fields of application of this technology are unlimited: training in the use of complex equipment, simulation of surgical procedures, architectural design testing, the reconstruction of sites in archeology, virtual museum visits, treatment of pain and phobias, and many other possibilities.

To enable the more emotional aspects of decision making to be handled, the immersive experience needs to be realistic. This will probably require high quality media production. Thus VR may often need to be combined with simulation design, quality media production and powerful computing to be educationally effective, again pushing up the cost. For these reasons, medicine is a particularly likely area for development, where traditional training costs are really high or where training is difficult to provide with real patients.

Once again, though, applications will tend to be very specific to the needs of a particular subject area. This means designers must include subject specialists with a deep understanding of the field who can combine the power of the technology with the needs of learners in a particular learning context. VR in particular requires instructors with imagination and creativity, working with other professionals such as media producers, learners themselves, as well as specialists in VR design.

What has inhibited widespread educational use of earlier two-dimensional VR developments such as Second Life has been the high cost and difficulty of creating the graphics and contexts for learning. Thus even if the hardware and software costs for VR are low enough for individual student use, the high production costs of creating realistic educational contexts and scenarios are likely to inhibit its general use.

Some caution is also needed in assuming that people will behave the same in real life as they do in VR environments. Gallup et al. (2019) found a major difference in the influence of social factors within real-world and virtual environments: social cues in actual reality appear to dominate and supersede those in VR. One of the authors, Alan Kingstone, concluded:

“Using VR to examine how people think and behave in real life may very well lead to conclusions that are fundamentally wrong. This has profound implications for people who hope to use VR to make accurate projections regarding future behaviours. For example, predicting how pedestrians will behave when walking amongst driverless cars, or the decisions that pilots will make in an emergency situation. Experiences in VR may be a poor proxy for real life.”

Rolfsen, 2019

This means we need more experimentation. This is still a relatively new technology, and there may be very simple ways to use it in education that are not costly and meet needs that cannot be easily met in traditional teaching or with other existing technology. For this to happen, though, educators, software developers, and media producers need to come together to play, experiment, test and evaluate.

Nevertheless, VR and AR are exciting technologies with the potential to change radically conventional learning processes.

[to come: an activity with feedback in the form of a short podcast]

Up next

The section on Artificial Intelligence in education.

Over to you

Any suggestions or feedback to improve this section on VR/AR will be very much appreciated.

References

Brandaõ, G. et al. (2018) Virtual Reality as a Tool for Teaching Architecture in Design, User Experience, and Usability: Designing Interactions Las Vegas NV: Proceedings of 7th International Conference, DUXU 2018, held as Part of HCI International 2018,

Connolly, B. (2018) How virtual reality is transforming learning at the University of Newcastle, CIO, 8 March

Elmqadden, N. (2019) Augmented Reality and Virtual Reality in Education: Myth or Reality? International Journal of Emerging Technologies in Learning, Vol. 14, No. 3

Gallup, A. et al. (2019) Contagious yawning in virtual reality is affected by actual, but not simulated, social presence Nature: Scientific Reports, 22 January

O’Connor, M. et al. (2018) Sampling molecular conformations and dynamics in a multiuser virtual reality framework, Science Advances, Vol. 4, No.6, 29 June

Rolfson, E. (2019) People think and behave differently in virtual reality than they do in real life UBC News, 24 January

Seidel, R. and Chatelier, P. (1997) Virtual Reality, Training’s Future?: Perspectives on Virtual Reality and Related Emerging Technologies Berlin: Springer Science & Business Media