Tag Archives: rankings

Deflation of academic publications

Leave a reply

The top journals publish a similar number of articles as decades ago, but there is a much larger number of researchers competing to get their work into a top journal. Correspondingly, it is more difficult over time to get a paper into a given journal. If articles are analogous to currency in the academic world, then this would be deflation: the value of the currency rises over time. If articles are like goods and services, but research effort is the currency that buys them, then there is inflation, because the amount of currency required to buy a given good rises.

The correct comparison between publications in different decades would take into account the increasing difficulty of publishing in a given journal. Instead of comparing papers in the top n journals, a better metric is papers in the top x percent of journals (accounting for the possibly expanding size of each journal). Similarly, being the number one researcher among a thousand in 1901 is less impressive than being the best among a million in 2001. Again the right comparison is by percentile rank, not by “top n” status.

The norms and metrics in academia are largely made by senior, established researchers. If people do not completely account for the deflation, then the top academics benefit from the increasing difficulty of publishing in the top n journals combined with the metric that counts the top n, not the top x percent. The research of old academics that was published in the top n long ago looks the more impressive the more difficult it is nowadays to get a paper into the top n. Comparison by percentile rank would correct for this artificial advantage, so the established members of the profession would not seem as high-achieving relative to new entrants.

A similar change in difficulty has occurred in getting accepted as a student in the top n universities, or getting hired as faculty in these. The right comparison to the students or faculty decades ago would compare the top x percent of universities, with the appropriate correction if the universities have expanded their enrollment or number of jobs.

Coordination game in choosing a university

Leave a reply

The ranking of universities is very stable over time (http://m.chronicle.com/article/Rank-Delusions/189919) regardless of the difference in resources, scandals and other factors affecting popularity and quality. There are several positive feedback mechanisms keeping the rankings constant. These come from the multiple coordination games when choosing a university.
1) Smart and hardworking students want to be together with other smart and hard workers. If for some reason the best are in one location, then in the future all the best people have a motive to go to the same place. So the best arrive at that location and help attract other best people in the future. Similarly, party animals want to go to a university famous for its parties, and if many party animals come to a university, then it becomes famous for its parties.
Why would smart people want to be together with other intelligent folks? Just for interesting conversation, for useful contacts, collaboration. For these reasons, even the stupid may want to be together with the smart. Then an assortative matching results, as Gary Becker predicted for the marriage market (http://public.econ.duke.edu/~vjh3/e195S/readings/Becker_Assort_Mating.pdf).
2) Students want to go to a school with the best teaching staff, and the best professors want to teach the best students. I have yet to hear anyone wish for stupider students or teachers for themselves. Again the preference is the same among smarter and stupider students and professors, so assortative matching is a reasonable prediction.
3) The best professors want to be with other similar ones. Where there are such people, more will arrive.
4) Smarter graduates are likely to earn more and can donate more to the university. Then the university can hire better teaching staff, which in turn attracts better students, who donate more… The more talented also accumulate more power after graduating, in government institutions for example, which they can use (legally or not) to benefit their alma mater. Predicting this, again many people want to go there, and in the stiff competition the best get in.
5) If the employers believe that from some place, more intelligent people come than from elsewhere, then they are ready to make better offers to those coming from there. This makes the place attractive to all future employee candidates. Due to competition, the best get in, which justifies the belief of the employers.
6) Smarter people can be taught faster, at a pace that the stupider cannot keep. This mechanism gives everyone a motive to go to a school corresponding to their level.
7) Faster teaching means more knowledge during a standard length higher education, which the employers should value. The graduates of a school giving more knowledge are favoured, which makes the place attractive to everyone and leads to only the best getting in. The ability of the average student remains high, which enables faster teaching.

Of rankings

1 Reply

Many universities advertise themselves as being among the top n in the world (or region, country etc). Many more than n in fact, for any n small enough (1000, 500, 100 etc). How can this be? There are many different rankings of universities, each university picks the ranking in which it is the highest and advertises that. So if there are 10 rankings, each with a different university as number one, then there are at least 10 universities claiming to be number one.
There are many reasonable ways to rank a scientist, a journal, a university or a department. For a scientist, one can count all research articles published, or only those in the last 10 or 5 years, or only those in the top 100 journals in the field, or any of the previous weighted by some measure of quality, etc. Quality can be the number of citations, or citations in the last 5 years or from papers in the top 50 journals or quality-weighted citations (for some measure of quality)…
What are the characteristics of a good ranking? Partly depends on what one cares about. If a fresh PhD student is looking for an advisor, it is good to have an influential person who can pull strings to get the student a job. Influence is positively related to total citations or quality-weighted publications, and older publications may be better known than newer. If a department is looking to hire a professor, they would like someone who is active in research, not resting on past glory. So the department looks at recent publications, not total lifetime ones. Or at least divides the number of publications by the number of years the author has been a researcher.
Partly the characteristics of a good ranking are objective. It is the quality-weighted publications that matter, not just total publications. Similarly for citations. Coauthored publications should matter less than solo-authored. The ranking should not be manipulable by splitting one’s paper into two or more, or merging several papers into one. It should not increase if two authors with solo papers agree to add each other as the author, or if two coauthors having two papers together agree to make both papers single-authored, one under each of their names. Therefore credit for coauthored papers should be split between authors so that the shares sum to one.
How to measure the quality of a publication? One never knows the true impact that a discovery will have over the infinite future. Only noisy signals about this can be observed. There currently is no better measure than the opinion of other scientists, but how to transform vague opinions into numerical rankings?  The process seems to start with peer review.
Peer review is not a zero-one thing that a journal either has or not. There are a continuum of quality levels of it, from the top journals with very stringent requirements to middle ones where the referees only partly read or understand the paper, to fake journals that claim to have peer review but really don’t. There have been plenty of experiments where someone has submitted a clearly wrong or joke article to (ostensibly peer-reviewed) journals and got it accepted. Even top journals are not perfect, as evidenced by corrigenda published by authors and critical comments on published articles by other researchers. Even fake journals are unlikely to accept a paper where every word is “blah” – it would make their lack of review obvious and reduce revenue from other authors.
The rankings (of journals, researchers, universities) I have seen distinguish peer-reviewed journals from other publications in a zero-one way, not acknowledging the shades of grey between lack of review and competent review.
How to measure the quality of peer-review in a journal? One could look at the ranking of the researchers who are the reviewers and editors, but then how to rank the researchers? One could look at the quality-weighted citations per year to papers in the journal, but then what is the q    uality of a citation?
Explicit measurement of the quality of peer-review is possible: each author submitting a paper is asked to introduce a hard-to-notice mistake into the paper deliberately, report that mistake to an independent database and the referees are asked to report all mistakes they find to the same database. The author can dispute claimed mistakes and some editor has to have final judgement. It is then easy to compare the quality of review across journals and reviewers by the fraction of introduced mistakes they find. This is the who-watches-the-watchmen situation studied in Rahman (2010) “But who will monitor the monitor?” (http://www.aeaweb.org/articles.php?doi=10.1257/aer.102.6.2767).
One could disregard the journal altogether and focus on quality-weighted citations of the papers, but there is useful information contained in the reputation of a journal. The question is measuring that reputation explicitly.
If there is no objective measure of paper quality (does the chemical process described in it work, the algorithm give a correct solution, the material have the claimed properties etc), then a ranking of papers must be based on people’s opinions. This is like voting. Each alternative to be voted on is a ranking of papers, or there may simply be voting for the best paper. Arrow’s impossibility theorem applies – it is not possible to establish an overall ranking of papers (that is Pareto efficient, independent of irrelevant alternatives, non-dictatorial) using people’s individual rankings.
Theorists have weakened independence of irrelevant alternatives (ranking of A and B does not depend on preferences about other options). If preferences are cardinal (utility values have meaning beyond their ordering), then some reformulations of Arrow’s criteria can be simultaneously satisfied and a cardinal ordering of papers may be derivable from individual orderings.
If the weight of a person’s vote on the ranking of papers or researchers depends on the rank this person or their papers get, then the preference aggregation becomes a fixed point problem even with truthful (nonstrategic) voting. (This is the website relevance ranking problem, addressed by Google’s PageRank and similar algorithms.) There may be multiple fixed points, i.e. multiple different rankings that weight the votes of individuals by their rank and derive their rank from everyone’s votes.
For example, A, B and C are voting on their ranking. Whoever is top-ranked by voting gets to be the dictator and determines everyone’s ranking. A, B, C would most prefer the ranking of themselves to be ABC, BCA and CAB respectively. Each of these rankings is a fixed point, because each person votes themselves as the dictator if they should become the dictator, and the dictator’s vote determines who becomes the dictator.
A fixed point may not exist: with voters A and B, if A thinks B should be the dictator and B thinks A should, and the dictator’s vote determines who becomes the dictator, then a contradiction results no matter whether A or B is the dictator.
If voting is strategic and more votes for you gives a greater weight to your vote, then the situation is the one studied in Acemoglu, Egorov, Sonin (2012) “Dynamics and stability of constitutions, coalitions, and clubs.” (http://www.aeaweb.org/articles.php?f=s&doi=10.1257/aer.102.4.1446). Again, multiple fixed points are possible, depending on the starting state.
Suppose now that the weights of votes are fixed in advance (they are not a fixed point of the voting process). An objective signal that ranks some alternatives, but not all, does not change the problem coming from Arrow’s impossibility theorem. An objective signal that gives some noisy information about the rank or quality of each alternative can be used to prevent strategic manipulation in voting, but does not change the outcome under truthful voting much (outcome is probably continuous in the amount of noise).