Maclean’s 2013 rankings of Canadian Universities: a critique

Dehaas, J. (2102) The 2013 Maclean’s University Rankings, Macleans.ca, November 1

I thought the clocks went back tomorrow – are we already in 2013? Well, here are the results from Macleans:

Medical Doctoral (out of 15)

1. McGill University, Montréal, Québec
2. University of British Columbia, Vancouver, BC
3. University of Toronto, Ontario

Comprehensive (out of 15)

1. Simon Fraser University, Vancouver, BC
2. University of Victoria, BC
3. University of Waterloo, Ontario

Primarily undergraduate (out of 19)

1. Mount Alison University, Sackville, New Brunswick
2. University of Northern British Columbia, Prince George, BC
3. University of Lethbridge, Alberta.

The most ‘improved’

• l’Université de Moncton (up five places from last year)

The magazine does not rank schools with fewer than 1,000 full-time students, those that are restrictive due to a religious or specialized mission, newly designated universities or those that are not members of the Association of Universities and Colleges of Canada (AUCC).

Methodology

Love them or hate them (and I hate them), rankings are fascinating. This is the 23rd year Macleans magazine has done this, and this year they noted:

it’s the rise of the west. Every university from Saskatchewan to the Pacific Ocean maintains or improves its standing. All four of British Columbia’s ranked universities placed in the top two in their categories. (Yes, I do love this part of the rankings!)

What does it mean? We have to look at the methodology, and kudos for Macleans for being transparent about this.

Categories

Macleans distinguishes between medical doctoral, comprehensive and mainly undergraduate universities.

This in itself is somewhat arbitrary – it’s as much about size as about some qualitative difference in mandates. For instance Memorial University in Newfoundland has a medical program but is listed as a comprehensive, and Laurentian and Lakehead have a joint medical program, but are listed as primarily undergraduate. Laurentian in fact has an extensive graduate program, but mainly at the applied masters level, such as the online MBA in partnership with the Chartered General Accountants of Canada.

What I suspect is the main criterion for the categories is the extent of research funding. It might be better though to have three categories based on size alone, especially as research funding is also a criterion within categories.

Criteria

Here are the criteria and the weighting they give to each (I have condensed and simplified this – see Measuring Excellence for more details):

• students (10%): based on the quality in terms of awards, grade point averages in high school
• classes (10%): student:faculty ratio
• faculty (20%): proportion of faculty who have won national teaching awards + number and amount of national research grants
• resources (12%): operating money/students + (again) additional research/development money weighted by number of faculty
• student support (13%): proportion of budget on students services (6.5%) and bursaries/scholarships for students (6.5%)
• library (15%): breadth and currency of collection (including digital services)
• reputation (20%): based on surveys to university officials, business CEOs and high school counsellors – response rate: less than 10%

Comparison with world rankings

There are some big discrepancies.

In the QS 2012 world rankings McGill is 18th, Toronto 19th and UBC 45th – a big difference in the case of UBC.

The Times Higher Education Supplement rankings place Toronto 19th, UBC 22nd, and McGill 28th. The next ranked Canadian university is MacMaster, at 65th, but in Macleans’ rankings, it comes after Queens University and the University of Alberta.

Comment

It all depends on the criteria and the weightings. UBC does particularly well in Macleans because of its scholarships and bursaries. It can do this partly because it is half university, half real estate company, having leased large portions of the University Endowment Lands for housing development in North America’s hottest real estate market. The money generated from its real estate business is then ploughed back into bursaries (which is a good thing to do with it). But not all Canadian universities have this advantage. Of course, Macleans magazine is aimed primarily at helping students choose a good university, and focusing on such criteria is reasonable for this purpose. But it’s not necessarily a measure of the quality of a university.

A second problem with ratings is that we need more diversification between institutions, but rankings tend to group all universities together. Macleans at least recognizes this to some extent, but we need to move to a system by which institutions are formally measured against their stated mandates. For instance, a university that focuses on serving disadvantaged or underserved learners shouldn’t be judged on the same criteria as an elite, highly selective and high cost institution.

But most critically, the Macleans rankings (and the other world rankings) are based primarily on input: quality of recruited student; resources; faculty reputation; research money. They don’t measure output: the quality of the graduates; time to completion of a degree; the extent to which students have improved on their learning; impact on innovation and development; cost-effectiveness.

In particular, they don’t measure the extent to which universities are using learning technologies, online learning or are teaching in ways that reflect the needs of the 21st century.

When they start doing that, I’ll begin to take them more seriously.