Universitat Oberta de Catalunya (2016) An Adaptive Trust-based e-assessment system for learning (@TeSLA) Barcelona: UOC
This paper describes a large, collaborative European Commission project headed by the Open University of Catalonia, called TeSLA, (no, not to develop a European electric car, but) a state-of-the-art online assessment system that will be accepted as equal to if not better than traditional face-to-face assessment in higher education.
The challenge
The project argues that at the moment there is no (European?) online assessment system that:
- has the same level of trust as face-to-face assessment systems
- that is universally accepted by educational institutions, accreditation agencies and employers
- incorporates pedagogical as well as technical features
- integrates with other aspects of teaching and learning
- provides true and secure ‘authentication’ of authorship.
I added the ‘European’, as I think this claim might come as a surprise to Western Governors’ University, which has been successfully using online proctoring for some time. It is also why I used the term ‘next generation’ in the heading, as the TeSLA project is aiming at something much more technologically advanced than the current WGU system, which consists mainly of a set of web cameras observing learners taking an assessment (click here for a demonstration).
Also, the TeSLA proposal makes a good point when it says any comprehensive online assessment system must also be able to handle formative as well as summative assessment, and that this can be a challenge as formative assessment is often embedded in the day-to-day teaching and learning activities.
But the main reason for this project is that online learning assessment currently lacks the credibility of face-to-face assessment.
The solution
A non-invasive system that is able to provide a quality continuous assessment model, using proportionate and necessary controls that will ensure student identity and authorship [in a way that offers] accrediting agencies and society unambiguous proof of academic progression….
Any solution must work fully online and take into account ‘academic requirements’ for assessment, including enriched feedback, adaptive learning, formative assessment and personalized learning.
This will require the use of technologies that provide reliable and accurate user authentication and identification of authorship, face and voice recognition, and keystroke dynamics recognition (see here for video examples of the proposed techniques).
The solution must result in
a system based on demonstrable trust between the institution and its students. Student trust is continuously updated according to their interaction with the institution, such as analysis of their exercises, peer feedback in cooperative activities or teacher confidence information. Evidence is continuously collected and contrasted in order to provide such unambiguous proof.
The players
The participants in this project include
- eight universities,
- four research centres,
- three educational quality assurance agencies,
- three technology companies,
- from twelve different countries.
In total the project will have a team of about 80 professionals and will use large-scale pilots involving over 14,000 European students.
Comment
I think this is a very interesting project and is likely to grab a lot of attention. At the end of the day, there could well be some significant improvements to online assessment that will actually transfer to multiple online courses and programs.
However, I spent many years working on large European Commission projects and I am certainly glad I don’t have to do that any more. Quite apart from the truly mindless bureaucracy that always accompanies such projects (the form-filling is vast and endless), there are real challenges in getting together participants who can truly contribute to such a project. Participants are determined more by political considerations, such as regional representation, rather than technical competence. Such projects in the end are largely driven by two or three key players; the remaining participants are more likely to slow down or inhibit the project, and they certainly divert essential funding away from the those most able to make the project succeed. However, these projects are as much about raising the level of all European countries in terms of learning technologies as becoming a world leader in this field.
These criticisms apply to any of the many European Commission projects, but there are some issues that are particular to this project:
- I am not convinced that there is a real problem here, or at least a problem that requires better technology as a solution. Assessment for online learning has been successfully implemented now for more than 20 years, and while it mostly depends on some form of face-to-face invigilation, this has not proved a major acceptability problem or a barrier to online enrolments. There will always be those who do not accept the equivalence of online learning, and the claimed shortcomings of online assessment are just another excuse for non-acceptance of online learning in general.
- Many of the problems of authenticity and authorship are the same for face-to-face assessment. Cheating is not exclusive to online learning, nor is there any evidence that it is more prevalent in online learning where it is provided by properly accredited higher education institutions. Such a study is just as likely to reduce rather than increase trust in online learning by focusing attention on an issue that has not been a big problem to date.
- Even if this project does result in more ‘trustworthy’ online assessment, there are huge issues of privacy and security of data involved, not to mention the likely cost to institutions. Perhaps the most useful outcome from this project will be a better understanding of these risks, and development of protocols for protecting student privacy and the security of the data collected for this purpose. I wish though that a privacy commissioner was among the eighteen different participants in this project. I fail to see how such a project could be anything but invasive for students, most of whom will be assessed from home.
For all these reasons, this project is well worth tracking. It has the potential to radically change the way we not only assess online learners, but also how we teach them, because assessment always drives learner behaviour. Whether such changes will be on balance beneficial though remains to be seen.
Dr. Tony Bates is the author of eleven books in the field of online learning and distance education. He has provided consulting services specializing in training in the planning and management of online learning and distance education, working with over 40 organizations in 25 countries. Tony is a Research Associate with Contact North | Contact Nord, Ontario’s Distance Education & Training Network.
Thanks for sharing your views. I feel that much of the still lack of proper recognition and credit assignment in formal distance programs is due to a lack of trust in online assessment on the part of regulating bodies and even part of faculty, at least in Argentina. Would you say that more research is still needed on the social representation of online assessment than on technology development in order to promote change in the regulations that would allow the recognition and crediting of on-line programs to the same standard as face-to-face ones? How do you think this mistrust on online assessment compares to the trust and widespread use on-line banking or e-commerce, for instance?
Hola, Nora – good questions.
I think the lack of trust is broader than just online assessment. Usually criticisms of online assessment are just part of a fear or concern about online learning in general, which by and large are unjustified.
In Canada, many of the highly ranked universities now offer the same course both on campus and online. Students take the same assessment, with a final exam usually under supervision. These students do just as well online, and the university makes no distinction on the degree transcript about the mode of delivery of the courses for which students have credit. Years of research has shown there is no significant difference between online and campus-based teaching. What matters more is the effectiveness of teaching within each mode.
The one legitimate area of concern is whether students can get the same kind of experiential learning that they would get in a face-to-face context such as a laboratory or an art studio, but increasingly it has been shown that even this kind of experiential learning can be delivered online, through simulations, virtual reality, etc. So, yes, you do have to look carefully at each specific teaching context, but instead of starting from the assumption that face-to-face teaching is bound to be superior, we should use the law of equal substitution, that whatever you do face-to-face can be done just as well online, or vice versa, then look for the exceptions to the law. Usually it comes down to cost – in other words it could be done online but simulations would be too expensive. Better to use the labs that have already been built. The opposite also applies: in many cases on campus is too expensive and it would be better to deliver online.
So while I am not against research and development into better ways of online assessment, I think we need to focus the research more on appropriate teaching methods within particular contexts and specific learning outcomes. The key thing to win trust is to show clearly that online learning is delivering the learning outcomes that are being demanded by accreditation agencies and employers.
Very interesting review, Tony. Just to add a comment or two on this. We have been piloting online proctored examinations for a little while and though I do think it is good that they are working on sophisticated identification I wonder if some simpler issues are being missed. Our lecturers are happy enough with the human identification of the candidate but are concerned that the proctor cannot look around the room without asking the candidate to move their camera or laptop and that it is difficult for the candidate to sketch diagrams and write mathematical equations. Sometimes I wonder if we have technogeeks who like doing the complicated stuff rather than solving simple unchallenging problems. Recently the company Wiris showed me how they can allow candidates sketch a mathematical equation into a box and convert it into a mathematical formula that can be automatically interpreted. When I asked them if they could add the feature of just saving the image so that it could be manually graded later, they seemed uninterested. (I get the impression that Wiris is a collection of very gifted mathematicians)
Brian