I have just come back from two days at one of my favourite communities of practice, the British Columbian Educational Technology Users’ Group (ETUG) annual workshop, this year at Simon Fraser University, Burnaby, BC. (More on this workshop in another post).
I was there to report on the early response to my online open textbook, Teaching in a Digital Age. I thought it might be of interest to share some of my presentation as a blog post, because it raises some questions about how to measure the success of an open textbook.
Student savings
This is the obvious and most important measure of success: how much does an open textbook save by reducing one of the major costs of education? Here in British Columbia, the average annual cost of textbooks for BC post-secondary institutions is $1,200 per student, if they bought all the required textbooks new (which of course, many don’t).
BCcampus, which has an extensive open textbook project, with over 60 open textbooks currently available, has been tracking their adoption by post-secondary institutions in BC. They have found that to date (actually, April 15, 2015):
- 146 known adoptions of Open Textbooks at 14 of the 25 public post-secondary institutions across British Columbia
- A student cost savings ranging from $475 K- $700 K (see their post on how they calculate student savings to see why they report the savings as a range).
More importantly, other studies have shown that when open textbooks are available, students make greater use of them, and students tend to perform better, as they have them from the first day of the course.
However, my open textbook is aimed at instructors and faculty, and my main aim is not to save them money, but to make the book more accessible than if it had been commercially published. So I need to use other criteria of success. I also have to compare the response to the book to the effort and cost of doing the book.
Quantitative responses
There were several deliberate marketing initiatives:
- March 2014 – April 2015: the use of my blog for draft chapters. However, my blog readership, which consists of mainly professionals in the field of online learning, is a secondary market, although a very important one, as they can bring the book to the attention of the main market I am trying to reach, instructors and faculty;
- April 7, 2015: I posted the final version of the book to the BCcampus Open textbook web site on April 6, 2014, and wrote a blog post the next day announcing that the book was finished and now fully available in various formats;
- April 22, 2015: the Western Inter-State Co-operative for Educational Technology (WCET), is a national (U.S.), member-driven, non-profit which brings together colleges and universities, higher education organizations and companies to collectively improve the quality and reach of e-learning programs. They had invited me to write a blog post for their newsletter, promoting the book, and they published this on April 22;
- April 29, 2015: Contact North in Ontario organised a world wide media release about the book
- May 5, 2015: Academica.ca picked up on the Contact North media release (as did some other specialist media) and published a short announcement about the book.
The BCcampus open textbook web site tracks the number of visits to the book. The following shows the site traffic between April 27 and May
Between April 7 (my blog post) and April 29 (Contact North’s media blitz) the site traffic had been averaging about 1,000 visits a day, but with considerable daily variations. You can see that after a peak around April 30, the number of visits started to decline. Figure 3 illustrates why:
Up until April 7, most people had been accessing/reading the html version of the book direct from the BCcampus web site. From April 7, though, visitors started downloading the book and reading it off-line or via a mobile version. Figure 4 shows the total number of downloads
Thus there were 8,205 downloads of the whole book between April 7 and May 24, i.e. over a seven week period. Interestingly, although the book is available in several mobile or tablet formats, 80% of the downloads were as pdfs.
Although the quantitative data is interesting, I’m not quite sure what to make of it. I struggled to exceed 10,000 copies of my best selling commercial book over a period of 10 years or so and 8,000+ downloads suggest a lot of readers, but it is easy to download something that is online and free then never read it or just a few pages.
In any case there are over half a million people in North America alone in the market I’m trying to reach. Although marketing is important, it also drives the numbers without indicating the level of engagement. So I am looking increasingly to qualitative feedback to measure the book’s success.
Qualitative responses
These can be classified in the following ways:
1. Individual faculty and instructors
This is probably my key market, but also one that is the hardest to reach, because these instructors are discipline and subject-based, and are unlikely to stumble across this book except by accident. To date, I have had a small number (less than 10) of individual instructors contact me to say that they are reading the book and finding it useful, but it is early days yet. I suspect that, like one of my other books, on effective teaching with technology in higher education, it will take time for the word to get out to this market. The earlier book in fact was selling more copies five years after publication than in its first year.
2. Faculty development
This is where someone has read the book, and wants to use it for the professional development of other instructors, usually within the same department. I have so far heard from the following who want to do this:
- Neuroscience, CalTech
- Professional development for digital education, Loma Linda University
- College of Engineering, Drexel University
- Bachelor of Technology, McMaster University
- Faculty of Agriculture, Dalhousie University
Usually the person contacting me has been a dean or head of department, which is particularly rewarding for me.
3. Adoption as a text book for a course
The book has been quickly adopted by at least one instructor as part of either teacher training or graduate programs in education on courses at the following universities:
- UBC (Master in Educational Technology)
- Simon Fraser University (Professional Development certificate)
- Royal Roads University (Master of Arts in Learning and Technology)
- Vancouver Island University (Online Learning & Teaching Graduate Diploma (OLTD) Program)
- Vancouver Community College (Provincial Instructor Diploma Program)
- Seattle Pacific University (MEd in Digital Education).
Again, although these all lead to some form of qualification, they are also providing professional development opportunities for instructors and faculty. And in this case it is saving students money, although the students are mainly professionals.
4. Accreditation agencies
This was even more unexpected. I have received notification from two accrediting agencies that they are either using or recommending the book for continuing professional development:
- Accrediting Bureau of Health Education Schools, USA
- Texas Higher Education Coordinating Board
5. Graduate students use in theses/dissertations
I have heard from a few graduate students who are using the book to help with their dissertation or thesis. I’m hoping this will increase over time.
6. External reviews
One graduate student wrote to me to say she wanted to quote me in her thesis, but her supervisor warned her not to do this, as the book had not been formally peer reviewed. I have in fact commissioned three independent reviews, which will be published at the end of this month (June) alongside the book. It will be nearly a year before any formal reviews will be published, at least in traditional journals, but these will be important, so I am wondering which journals to send it to (as a pdf) – any suggestions?
Conclusions
It is still early days to be evaluating the success of my open textbook, but there are some surprises here for me.
- I didn’t expect so many people to download the whole book, and I certainly didn’t expect those that did download to use the pdf version. I actually designed the book for online use on a laptop or desktop computer, as the book has embedded multimedia and is meant to be interactive, and a resource that is dipped into rather than read from cover to cover. In another post, I discussed the problem of formatting graphics in the different versions, and I am hoping to have a slightly redesigned version for tablets soon, but maybe one should design for a pdf version from scratch.
- It may or may not be significant that I haven’t heard anything yet from units that are professionally responsible for faculty development, such as teaching and learning centres. I will be more than happy if I can reach directly individual faculty and instructors and their deans and heads of department. Teaching and learning centres are probably at their busiest now and already had their workshops designed before the book came out. However, I am hoping that it will eventually be seen as an essential resource by such centres, but maybe I need to network more with organisations such as the Society for Teaching and Learning in Higher Education to make them also more aware of the book.
- Authors need to make sure they are getting regular reports on the utilisation of their open textbooks. The software is there to do this, but it is not always accessible by the authors themselves without running a special report, so there is a bit of work to be done on the interface of Pressbooks to make this information more comprehensive and readily available to authors directly. However, interpreting such data is also tricky, even when it is available, and needs to be balanced with qualitative assessments as well.
I will write about the cost of doing an open textbook in another post, then end with a final post which will discuss whether I felt the whole exercise was worth it. In the meantime, I am wondering if any readers have suggestions for better ways to evaluate the success of an open textbook.
Dr. Tony Bates is the author of eleven books in the field of online learning and distance education. He has provided consulting services specializing in training in the planning and management of online learning and distance education, working with over 40 organizations in 25 countries. Tony is a Research Associate with Contact North | Contact Nord, Ontario’s Distance Education & Training Network.
It does sound like a good idea to connect with the STLHE, especially the Educational Developers Caucus. They’re particularly busy these days, what with the AGM in Vancouver, but any connection with the organisation is likely to be beneficial. If not for book downloads, at least for the communities of practice being established in Canadian Higher Education.
As is often the case with open-ended qualitative research, much of the impact can be difficult to track. It’s quite possible that your book will have a deep impact outside of your target market, if for no other reason that you’ve been very transparent about the whole process. For instance, the fact that Audrey Watters has mentioned your recent post on costs can do a lot to make your book into a “case study”, as part of a broader conversation. But some of the most interesting effects will be much harder to track. If a market research journalist in Mali hears about Scenario G (on getting old) from an instructor who read your book and goes on to write a piece about the fallacy of “generations”, you’re not very likely to hear about it directly unless she goes back to the source and credits you.
Which brings up an interesting point. Because of the book’s open availability, you might be more likely to be credited. And if you are in fact credited, you’re much more likely to notice it (given trackbacks and such). Which is a part of the blog logic which still applies to Pressbooks (though it’s not the reason Hugh used WordPress as a basis of Pressbooks).
In turn, we can think about the whole “remix” dimension of Open Textbooks and Open Educational Resources. Given your Creative Commons license, you are supposed to get attribution if some part of the book is reused in some way. A really neat impact of your book could be that parts of it are repurposed, giving way to interesting material. While it’d be possible to assess such an effect, it’s a bit difficult to integrate those “derivative works” as measures of success. You’ve demonstrated remarkable flexibility in evaluating success (not caught up in a rigid structure). But reuse of your book probably extends outside of the experiment.
Speaking of your book’s impact… Quebec’s Conseil supérieur de l’éducation published a brief about distance education, yesterday. You’re cited about a dozen times and most citations are to your open textbook. They also cite a factsheet you posted in 2011 and quote you from a secondary source.
(The brief itself is only available in French (PDF), but a summary in English (PDF) provides a bit of context.)
What this implies is that your book is being used in a broad conversation about the future of higher education in a Canadian province which wasn’t mentioned in it.