International university rankings: science or quackery?

“Times Higher Education (THE) is confident that the 2010-2011 world university rankings represent the most accurate picture of global higher education THE has ever produced.” Why was the British magazine so confident about the accuracy of its rankings? The reason was the use of bibliometric measures of performance provided by the multinational firm Thomson Reuters:, “The Times Higher Education World University Rankings 2010-11 were developed in concert with our new rankings data provider  (…) with input from more than 50 leading figures in the sector from 15 countries across every continent, and through 10 months of extensive consultation. We believe we have created the gold standard in international university performance comparisons.” Apparently, the manufacturing of academic rankings was entering a scientific era. It was no longer the era of unjustified criticism or complaint. “Like them or not, rankings are here to stay”, was the lapidary comment.

The era of university rankings had its start in 2003, when the Shanghai Ranking (ARWU) first appeared. In spite of the use of disparate indicators aggregated through arbitrary weights, the rankings had great resonance, especially in those countries where university rankings were inferior to expectations. For instance, Cecile Hoareau observes that the first Shanghai Ranking “generated considerable embarrassment among the French intelligentsia, academia and government: the first French higher education institution in the ranking came only in 65th position, mostly behind American universities and a few British ones.” In 2004, the first THE-QS ranking was published, followed in 2007 by those of Taiwan (HEEACT) and Leiden (CWTS).

In 2010, Times Higher Education and QS (Quacquarelli-Symonds) terminated their collaboration and THE released rankings based on a brand new methodology. The tables used 13 separate indicators, brought together into five categories: Teaching (worth 30%), Research (30%), Citations (worth 32.5%), Industry income (worth just 2.5%), International mix — staff and students (worth 5%). Notably, the largest weight, almost one third of the total score, went to citations. “A university’s research influence — as measured by the number of times its published work is cited by academics […] This weighting reflects the relatively high level of confidence the global academic community has in the indicator as a proxy for research quality.” 

While the supremacy of Harvard, Caltech and MIT was somehow expected, there was also a surprise coming from an African outsider. Ann Mroz, Editor of the Times Higher Education commented: “We would like to congratulate Alexandria University for its performance in this year’s rigorous rankings. Being ranked 147 in the world top 200 is an impressive achievement. The top 200 universities in the world represent only a tiny fraction of world higher education and any institution that makes it into this table is truly world class.”  

Even more surprisingly, in the citations category, Alexandria ranked fourth in the world, defeating giants such as Stanford, Rice and Harvard. It took just a few days for commenters to raise suspicions about the robustness of the adopted citations indicator as a measure of research impact. At first, THE suggested that there was some substance to Alexandria University’s extraordinary score, citing Ahmed Zuweil, a Nobel prize winner who had graduated from Alexandria with a master’s degree some four decades ago, and mentioning some frequently cited papers by a single author in one journal. The latter explanation was indeed the right one: the exploit was due to a unique bibliometric champion, Mohamed El Naschie, a researcher in mathematical physics.

In November 2010, less than one month after the release of the THE rankings, an article entitled Questionable science behind academic rankings appeared in the New York Times (NYT). It explained the reasons behind the success the Egyptian university and also gave details on the bibliometric achievements of El Naschie, who had published no less than 320 scientific articles in a journal, Chaos, Solitons and Fractals (CSF) edited by Elsevier, of which he was the Editor-in-Chief. A great part of the citations were either self-citations or came from another journal, The International Journal of Nonlinear Sciences and Numerical Simulation (IJNSNS), of which he was Co-Editor.

As explained by the NYT, in 2010 El Naschie was no longer the Editor-in-Chief of CSF, as he had resigned in 2009. However, the story of the Self-publishing editor set to retire had even deserved the attention of Nature News in November 2008, about two years before the 2010 rankings. In December 2009, Douglas N. Arnold, a well-known mathematician, had also raised an alarm: thanks to the clever use of self- and cross-citations, the journals CSF and IJNSNS had climbed the journal rankings based on the so-called Impact Factor, a citation-based index. It seems that THE was so confident in its bibliometric methodology to dismiss not only the hints but even the evidence that would have suggested a more cautious attitude towards Alexandria’s performance.

Apparently, THE was not the only one to be fooled by numerical indicators. In 2008, Thomson Reuters, who manages the bibliometric database Web of Knowledge and publishes the Impact Factors of scientific journals, had listed Ji-Huan He, the founder and Editor-in-Chief of IJNSNS, as one of the rising stars of computer science.

The most timely and insightful analyses of the THE 2010-2011 world university rankings were due to Richard Holmes, whose blog University Ranking Watch is an indispensable resource for understanding mechanisms, flaws and errors of higher education rankings. Among other things, Holmes explains how the anomalous scores of a single researcher could substantially affect the outcomes of an entire university. THE was measuring average citation impact, so that small numbers of papers that are cited very frequently could boost the university score. El Naschie’s list of affiliations included three additional universities, but their rankings were not boosted by his citations because, differently from Alexandria, they were producing so many papers that El Naschie’s made little or no difference.

Some years have passed but Holmes’ analyses show that the problems have not been fixed. There are sudden rises or falls and unexplainable achievements. Moreover, when different rankings are compared their results are inconsistent – a demonstration of the arbitrarity of the underlying design choices. In a scientific paper, Jean-Charles Billaut, Denis Bouyssou and Philippe Vincke posed the direct question, “Should you believe in the Shanghai Ranking?” The response was: “The Shanghai Ranking is a poorly conceived quick and dirty exercise […] any of our […] students that would have proposed such a methodology in his or her master’s thesis would have surely failed according to our own standards.”

Again, like them or not, rankings are here to stay. But you should not take them too seriously.