An Unbiased Way to Rank Schools: Some Changes in the Lineup

An interesting research paper on an unconventional method of ranking
colleges from authors (Christopher Avery, Mark Glickman, Caroline
Hoxby, Andrew Metrick) from the John F. Kennedy School of Government at
Harvard University; the Department of Health Services at the Boston
University School of Public Health, the Department of Economics at
Harvard University, and the Department of Finance of The Wharton School
at the University of Pennsylvania.

In particular, Caroline Hoxby from Harvard, has done many notable
studies in this area.

Their methodology is based on student preference in where they choose
to finally enroll among acceptances, in effect a series of head to head
competitions.

As one of the authors (Metrick) states from the Wharton site discussion
of this research paper
(http://knowledge.wharton.upenn.edu/index.cfm?fa=viewArticle&id=1104)
(The complete summary article is listed at the bottom): "When a student
decides to enroll at one college among those that have admitted him, he
effectively decides which college won in head-to-head competition. This
model efficiently combines the information contained in thousands of
these wins and losses, and produces a ranking that would be very
difficult for a college to manipulate."

But can colleges be judged based on who "wins" the competition for
students? Metrick and his co-authors contend that they can. "First,
students believe and act as though their peers matter," the researchers
say in the study. "This may be because peer quality affects the level
of teaching that is offered. Alternatively, students may learn directly
from their peers. It is reasonable for students to care about whether
they are surrounded by peers with high college aptitude ... Second,
students - especially the high achieving students on whom we focus -
are not ignorant about college quality. They gather information from
publications, older siblings, friends who are attending college,
college counselors and their own visits to colleges."

The research article ranks the schools as follows (pages 26-28 of the
PDF) (http://knowledge.wharton.upenn.edu/papers/1287.pdf)

1 Harvard 2800
2 Yale 2738
3 Stanford 2694
4 Cal Tech 2632
5 MIT 2624
6 Princeton 2608
7 Brown 2433
8 Columbia 2392
9 Amherst 2363
10 Dartmouth 2357
11 Wellesley 2346
12 U Penn 2325
13 U Notre Dame 2279
14 Swarthmore 2270
15 Cornell 2236
16 Georgetown 2218
17 Rice 2214
18 Williams 2213
19 Duke 2209
20 U Virginia 2197
21 Northwestern 2136
22 Pomona 2132
23 Berkeley 2115
24 Georgia Tech 2115
25 Middlebury 2114
26 Wesleyan 2111
27 U Chicago 2104
28 Johns Hopkins 2096
29 USC 2072
30 Furman 2061
31 UNC 2045
32 Barnard 2034
33 Oberlin 2027
34 Carleton 2022
35 Vanderbilt 2016
36 UCLA 2012
37 Davidson 2010
38 U Texas 2008
39 NYU 1992
40 Tufts 1986
41 Washington & Lee 1983
42 U Michigan 1978
43 Vassar 1978
44 Grinnell 1977
45 U Illinois 1974
46 Carnegie Mellon 1957
47 U Maryland 1956
48 William & Mary 1954
49 Bowdoin 1953
50 Wake Forest 1940
51 Claremont 1936
52 Macalester 1926
53 Colgate 1925
54 Smith 1921
55 U Miami 1914
56 Haverford 1910
57 Mt Holyoke 1909
58 Connecticut College 1906
59 Bates 1903
60 Kenyon 1903
61 Emory 1888
62 Washington U 1887
63 Occidental 1883
64 Bryn Mawr 1871
65 SMU 1860
66 Lehigh 1858
67 Holy Cross 1839
68 Reed College 1837
69 RPI 1835
70 Florida State 1834
71 Colby 1820
72 UCSB 1818
73 GWU 1798
74 Fordham 1796
75 Sarah Lawrence 1788
76 Bucknell 1784
77 Catholic U 1784
78 U Colorado 1784
79 U Wisconsin 1780
80 Arizona State 1774
81 Wheaton (Il) 1750
82 Rose Hulman 1745
83 UCSC 1736
84 Boston U 1736
85 UCSD 1732
86 Tulane 1727
87 U Richmond 1714
88 CWRU 1704
89 Trinity College 1703
90 Colorado College 1698
91 Indiana U 1689
92 Penn State 1686
93 American U 1681
94 Hamilton 1674
95 U Washington 1629
96 U Rochester 1619
97 Lewis & Clark 1593
98 Wheaton (MA) 1564
99 Clark 1551
100 Skidmore 1548
101 Purdue 1525
102 Colorado State 1513
103 Syracuse 1506
104 Scripps 1479
105 Loyola U 1221

The list appears to be pretty intuitive. Some notable 'drops' from US
news rankings are Duke, University of Chicago, Johns Hopkins,
Washington University in St. Louis and University of Michigan.


An interesting finding is that regional preferences don't differ much
from the overall national one (p. 36 of the PDF). The exception appears
to be Brigham Young which is #6 in the region that includes Utah).

The disussion of this paper from Wharton website is reproduced below
(http://knowledge.wharton.upenn.edu/index.cfm?fa=viewArticle&id=1104):


An Unbiased Way to Rank Schools: Some Changes in the Lineup


Ah, December, a time of holiday celebrations, shopping mall traffic
jams, festive family gatherings - and for many high school seniors and
their parents, some nail-biting days waiting to get a particularly
important piece of news.


It's early decision time.


Early decision (ED), an increasingly popular and controversial program
offered by most private and many public colleges and universities in
the U.S., allows applicants to apply to a first-choice college and get
an early decision, often by mid-December, if they promise to attend
(provided they are accepted). But in recent years, ED and other
strategic admissions programs have come under fire, with critics
contending that they are an easy way for schools to manipulate their
admission and matriculation rates: The more students admitted under an
early decision program, the higher a school's matriculation rate.


What's behind all the stage-managing? It comes down to the rankings,
experts say. Colleges and universities, desperate to increase their
standing in the closely-watched annual college ranking articles and
guides, have quietly created programs and systems to boost the
appearance of selectivity and desirability. Critics of college ranking
guides point to the practice of encouraging applications from students
who have little chance of being accepted, a custom that lowers a
school's admissions percentage, thus making the school appear more
selective. Or nudging up matriculation rates by rejecting applicants
they think are applying to their college as a "safety" school and are
not likely to attend.


In a recent study, titled "A Revealed Preference Ranking of U.S.
Colleges and Universities," Wharton finance professor Andrew Metrick
and his co-authors create a new, market-driven college ranking system
they say would help end this maneuvering. Metrick and Harvard's
Christopher Avery and Caroline Hoxby, as well as Mark Glickman from
Boston University, built a statistical model they compare to ones used
to rank professional chess players. It is a system that rates more than
100 colleges and universities based entirely on where America's best
and brightest students actually decide to go.


"A lot of admissions people will tell you off the record that in an
effort to move themselves up a couple of notches in the rankings, they
resort to a number of things to improve their admissions' rate and
matriculation rates that are really anti-competitive," Metrick says.
"Some schools are admitting more than 40% of their freshman classes via
early decision, which began as a wonderful program for students and
schools but has become something entirely different today because of
the pressure colleges feel to improve their standings in the rankings.
I think that's a little out of hand. One way to relieve this pressure
is to have some kind of measure that would allow you to talk about how
selective and preferred a school is without inducing this
counterproductive strategic behavior."


The study, submitted in October to the National Bureau of Economic
Research, creates a model that relies solely on the real-world
decisions of admitted students. "Our system extends models used for
ranking players in tournaments, such as chess or tennis," Metrick says.
"When a student decides to enroll at one college among those that have
admitted him, he effectively decides which college won in head-to-head
competition. This model efficiently combines the information contained
in thousands of these wins and losses, and produces a ranking that
would be very difficult for a college to manipulate."


But can colleges be judged based on who "wins" the competition for
students? Metrick and his co-authors contend that they can. "First,
students believe and act as though their peers matter," the researchers
say in the study. "This may be because peer quality affects the level
of teaching that is offered. Alternatively, students may learn directly
from their peers. It is reasonable for students to care about whether
they are surrounded by peers with high college aptitude ... Second,
students - especially the high achieving students on whom we focus -
are not ignorant about college quality. They gather information from
publications, older siblings, friends who are attending college,
college counselors and their own visits to colleges."


A ranking based on student preference, Metrick and his authors say, is
an efficient way to amass observations about quality from thousands of
students. They cite parallels in the food and hospitality industries,
where consumers judge restaurant and hotel quality based partly on
their own experiences, but also seek out others' opinions. "This is why
there is a demand for guides like Zagat's, which aggregate people's
observations about hotels and restaurants," they write.

In finding student candidates for the study, the researchers worked
with guidance counselors from 510 high schools across the United
States, producing a response rate of 65%, or 3,240 students. Top
ranking seniors from those schools were surveyed and tracked,
completing two questionnaires over the course of the academic year.
Students were asked about their background and college applications,
with each student listing up to 10 colleges where he or she had
applied, as well as questions about his or her test scores, race,
ultimate admission outcomes, financial aid and scholarship offers, and
final matriculation decisions. The sample contained students from 43
states and the District of Columbia.

What schools came out on top? Not surprisingly, Harvard and Yale came
out number one and two, with Stanford, Cal Tech, MIT and Princeton
following. And while the top schools look much the same as the top
liberal arts colleges and universities in the U.S. News & World Report
rankings, their order changes, sometimes significantly. Duke, ranked
fifth in U.S. News, drops to 19th, while Princeton, which U.S. News
tied with Harvard in the number one spot, dipped to 6th. Other schools
seemed to benefit. Brown University, which offers students flexibility
in their curriculum, as well as Georgetown and Notre Dame, which have
strong Catholic followings, fared better than on the U.S. News list.

Metrick has no interest in selling the student preference rating system
he and his colleagues created or joining in the rankings frenzy by
producing an annual list. His hope for the model, he says, is that the
publishers of rankings guides will begin to consider using such
unbiased, scientific measures when creating their guides. "If it were
completely up to us, we would have the College Board or U.S News do a
survey every year to gather this data and report it instead of the
admission rate or the matriculation rate as a measure of desirability -
because people are interested and they want to know," says Metrick.
"This is just a better measure."

Further, the cost of gathering such data would be "a trivial share" of
the revenues associated with college guides, and at least some of the
data are already compiled by organizations like The College Board and
the ACT (an SAT alternative) "so that gathering a highly representative
sample should be very feasible," the authors write. If a student
preference ranking based on "our procedure were used in place of
manipulable indicators like crude admissions rate and crude
matriculation rate, much of the pressure on colleges to manipulate
admissions would be relieved."

"We aren't naïve enough to think that U.S News is going to see this
and say, "We should replace our system with this one,'" says Metrick,
acknowledging that college guide publishers were somewhat blasé about
the study. "But we are academics and we want to get useful ideas out
there. We think that this is important information and that students
have a real interest in knowing what colleges and universities are most
attractive to their peers."

The top 25 schools in Metrick's student preference ranking are:
Harvard, Yale, Stanford, Cal Tech, MIT, Princeton, Brown, Columbia,
Amherst, Dartmouth, Wellesley, University of Pennsylvania, University
of Notre Dame, Swarthmore College, Cornell, Georgetown, Rice
University, Williams College, Duke, University of Virginia,
Northwestern, Pomona, Berkeley, Georgia Tech, Middlebury College,
Wesleyan, University of Chicago, Johns Hopkins, University of Southern
California, Furman College, University of North Carolina, Barnard
College, Oberlin College, Carlton College, Vanderbilt, UCLA, Davidson
College, University of Texas, New York University, Tufts, Washington &
Lee, University of Michigan, and Vassar College.

An interesting approach, not without its pitfalls.

I applaud the proposition that munging about with admission practices is
defective. This paper seems to put it a bit more precisely than some others
who have made that statement, or come close to it.

However...this seems to be another way to enshrine the conventional wisdom.
In particular, I don't think it's appropriate to call this unbiased. The
selection of high schools from which the participants were drawn introduces
the potential for some issues. The paper does not give the state-by-state
breakdown of schools, let alone the specific names involved. And the
selection of students for inclusion is, at bottom, subject to the whims of
the guidance counsellors who selected the students from their school to
participate. So I question how representative the sample is. (Given that
45 percent came from private schools, with a median family income of
$119,929...draw your own conclusions. And I have to question how
representative 3240 students from 396 high schools are, but the researchers
deserve some leeway in this area.)

I think the suggestion, put forth by the authors, that this methodology be
expanded to include all college-bound HS seniors is excellent, and this
would put to rest the bias inherent in the system. [There's a Monty Python
line in there somewhere...]

Another issue which I think introduces bias is in the set of states excluded
from the analysis. Where, for example, might Grinnell rank if HS seniors
from Iowa were not excluded?

And the selection of states for groupings seems a little arbitrary, at least
in the case of region 6 (Alabama, Kentucky, Tennessee). I assume that
Mississippi and West Virginia would be grouped here as well, were they also
not excluded from the survey. I'm not certain what Kentucky has in common
with Alabama, other than they are both in the SEC. But these results seem
to exclude Berea, which is a highly ranked, desirable college for a certain
strata of high-potential Appalachian students...well below the median family
income in this survey.

So...many interesting points, some flaws inherent in the sampling approach
and size of the study, a useful start but more work is needed here IMHO.
Despite the foregoing complaints, the authors are to be commended.

Hank Murphy
speaking only for myself