Obtaining independent reviews for an open textbook: what criteria to use?

What is the issue?

One of the questions I had to ask myself as a self-publishing author of Teaching in a Digital Age was whether I needed my book to be independently reviewed before publication. If so, would the same criteria need to be used as if I was publishing commercially?

What is the usual process in academic publishing?

Usually, before publishing an academic book or a textbook, commercial publishers will seek independent reviews at two stages of the process: when an author submits a proposal for a book, and then when the first complete draft is sent to the publisher. Of course, as well as external reviewers, the publishing company will have an in-house specialist editor who will be the main person in the decision-making process, and but even then an editor will usually take the final proposal to an internal committee or even a board meeting for final approval. Each of these stages can take up to three months, sometimes longer for the second stage, much longer if the author is required to make substantial changes before publication. Lastly, after the book is published, it may be reviewed, again independently, in academic journals specializing in the field.

Although this lengthy approval and review process can be very frustrating for an author, the process does ensure that the author gets a lot of feedback, and above all it is part of the quality control process, which is one reason why books count so much in the academic tenure and promotion process. However, the main purpose of external reviews in the publishing process is to ensure that there is a market for the book that is large enough to at least cover costs and hopefully generate a profit for the publisher, to provide quotes or endorsements that will help sell the book, and also to some extent to protect the publisher’s ‘brand’ or reliability.

There are also disadvantages of course with this process. It limits the acceptance of any publication that is outside the accepted norms within a discipline, thus perhaps inhibiting or delaying progress in a field, and, as happened to me once (with my first book), if a ‘rival’ academic with very different views is asked to be a reviewer, a perfectly good book can be unfairly trashed (although to be fair to the publisher, they still went ahead and published, on the basis of the other two reviews they received.)

In general, I have to say that with my 12 commercially published book, as an author I found the external review process, and above all, in two or three cases the feedback from the publisher’s editor, to be extremely valuable and helpful, and this review process resulted in far better books.

Open publishing

Self-published books need not follow any of this process, although open textbooks, such as those from OpenStax or the BCcampus open textbook project, are nearly always independently reviewed by faculty in the jurisdiction where these books may be adopted.

However, my book is somewhat different. It was written from scratch for a different market, faculty and instructors, rather than students, and it is not part of the BC government’s open textbook project that BCcampus manages. Although BCcampus offered me essential technical services, they were not responsible for editing or reviewing the book. (I was fortunate to be well enough known to BCcampus for them to put a lot of trust in me.)

What did I do?

Because the book was to be an open textbook, and I have a blog which is read within the community of practice in which I work, I was able to test early drafts of chapters and get some feedback on an ad hoc and voluntary basis. I also hired an instructional designer/editor to proof read and assess each draft chapter. I also sent drafts to other specialists in the field where I described in detail their work, asking for feedback and comments. I then published each chapter when I thought it was ready, and the Centre for Digital Education at Ryerson University also offered to provide systematic feedback as I published.

As a result I got a lot of useful feedback and comments that influenced the final version of the book, but nevertheless I was a bit shaken when I received an e-mail from a student who wanted to quote me in her graduate thesis, but was advised not to by her supervisor because the examiners might not accept references to a book that had not been independently reviewed.

As a result, after the book was published, and with no guarantee that it would be picked up and reviewed in an academic journal, I decided to obtain three independent reviews, and, as with the BCcampus textbooks, I would publish these reviews as received alongside the book.

Note though that I have obtained the external reviews after, not before, publication, because I felt it was more important to publish and be damned and thus get out the book as soon as possible, and because if there are major changes needed, that can still be done.

Criteria for choosing reviewers

I had three main criteria in mind: independence, qualification, and availability/willingness.

Independence was the most difficult. I had to invite someone who could be as objective as possible. This meant looking for reviewers who were professionals in the digital learning, instructional design, online learning or open education area, but who had not been closely associated with me during my 40 years working in the field. These reviewers should be people within the field who would be seen as being objective and sufficiently ‘distant’ from me and my career.

In terms of qualification, I needed reviewers who were also experts in the field of digital teaching and learning. This was the easiest of the three criteria to meet, but this had to be combined with independence, and this is where it started to get tricky.

Also, because the book is also targeted at faculty and instructors, I wanted a reviewer who is a mainline faculty member interested in teaching and learning but who did not know my previous work, and who would judge it strictly from a faculty or instructor perspective.

The third criterion, availability and willingness, was also important. The amount of work involved in reviewing a 500 page textbook is quite significant. Usually publishers pay a small fee for external reviewers, which no way compensates for the work involved, but at least it helps sweeten the pot. However, if I paid the reviewers as an author, that may be seen as unduly influencing the independence of the reviewer. In any case, I’m not getting revenues from the book, so any payment would have to come out of my own resources. As it turned out, none of the reviewers I approached requested or even mentioned a fee. Nevertheless I realised I was asking a lot of the reviewers with very little to offer them in return (other than a free read).

My choice of reviewers

The mainstream faculty member turned out to be the easiest of my choices. I published each chapter when it was ready, and after publishing the first chapter I received a string of comments from Dr. James Mitchell, Professor and Director of the Architectural & Environmental Engineering Program at Drexel University, Pennsylvania. I had never been in touch with him before and had never visited Drexel, but it was clear he was interested in changing the teaching in his department, and had some good points and questions to raise, so he was my first choice for reviewer.

Secondly, staff at the Centre for Digital Learning at Ryerson University, Toronto, had been tracking the development of the book and also providing feedback. This is a fairly new Centre and I did not know any of the staff there and had not done any consultancy work at Ryerson, so they seemed an obvious second choice to write a review of the whole book once it was finished.

Lastly, I wrote to a distinguished scholar of open learning at a British university, asking this person to be a reviewer, but they did not reply, so I was wondering who else to approach when I received an e-mail from Sir John Daniel, former Vice-Chancellor of the U.K. Open University, former President of the Commonwealth of Learning, and former President and Provost of several Canadian universities. He is also a scholar of open education with several (properly!) published books to his credit. He informed me that he was writing a review of the book for an academic journal and was looking forward to reading the book, and I therefore asked him if I may use his book review also as an external review of the book, which he agreed to. John Daniel is of course someone I have known and respected for many years, but we have never worked directly together.

So I now have my three reviewers, and I am extremely grateful for their willingness to do this.

Criteria for the review

BCcampus sends out a set of criteria to reviewers when they are reviewing books for the BCcampus open textbook project. In order to ensure consistency between the three reviews, I took the BCcampus guidelines and amended it to the slightly different context of my own book, and sent the guidelines to each reviewer. I will publish this as a separate blog post, as this post is already too long. 

Next steps

I now have two of the three reviews and the third is expected shortly. These will be published ‘as is’ with some context as an appendix to the book. If I also come across other reviews of the book from academic journals, I will add these (good or bad) to the book appendices.

Questions for my blog readers

The need for independent reviews for an open textbook has raised a lot of questions for me. If it had not been for the e-mail from the graduate student, frankly I don’t think I would have bothered. Since an open textbook is free and easily accessible, I was more than happy to let readers make their own judgements about the value of the book. It’s not as if you are asking someone to pay a large amount of money for something which they are then disappointed to read after they have paid their money. I also felt awkward about asking someone to read a 500 page book then write a critical review without being able to offer anything in return.

On the other hand, I want the book to be recognized and used by graduate students, and their committees and examiners. I want faculty in particular who read it to be assured that it has been properly peer reviewed. There is no reason why an open textbook cannot be as good if not better than any other book published more traditionally. But, as with distance education, online learning and open education, you have to be twice as good as the alternatives to be recognized. So if it takes external reviews to be accepted, so be it.

But I would really like to get your views on this. In particular:

1. Do you think it is important for open textbooks to be externally reviewed, before or after publication?

2. Is the process I have followed appropriate, or is it flawed? What would you have done or preferred?

3. Would you be happy to use an open textbook in your course if it had not been externally reviewed?

Next

I will share the guidelines I sent to the reviewers, and I will also ask for your feedback on these, so that the guidelines for review can be used by other authors of open textbooks.

I will also post on this blog each of the three external reviews when they are all in.