Kurt Johnson

Maybe the best quote I've seen on the folly of US News (and other) rankings for competitive schools:

Robert L. Woodbury, former chancellor of the University of Maine system, noted the folly of the current institutional U.S. News and World Report rankings: When Consumer Reports rates and compares cars, it measures them on the basis of categories such as performance, safety, reliability, and value. It tries to measure “outputs”—in short, what the car does. U.S. News mostly looks at “inputs” (money spent, class size, test scores of students, degrees held by faculty), rather than assessing what the college or university actually accomplishes for students over the lives of their enrollment. If Consumer Reports functioned like U.S. News, it would rank cars on the amount of steel and plastic used in their construction, the opinions of competing car dealers, the driving skills of customers, the percentage of managers and sales people with MBAs, and the sticker price on the vehicle (the higher, the better).

For a better ranking system, here's a link to an excellent article (at Wharton, no less ;-). It outlines how to build a market based ranking system and illustrates more of the flaws in the current paradigm of college rankings. Like a lot of economics it's generally intuitive but the detailed data opens some questions at the margins.

This graph from the paper showing the dip in Princeton admissions rates for students with SAT scores between the 93rd and 98th percentiles is a great example the distortion that rankings regimes can cause:

 

 

 

 

 

 

 

 

 

 

 

 

 

Ultimately I think understanding a consumer decision-based ranking could really help the schools shape their classes a lot more effectively than the crude rankings do today. For example, BYU has cultivated a strong preference in a subset of applicants (p. 38) that disproportionately prefer BYU to other normally more desirable schools. Understanding what segments of applicants strongly favor or disfavor their institutions should allow admissions committees to more effectively market to these segments in order to shape their class – and ultimately develop their university and brand – on the demand side (applications) instead of the supply side (admissions).I can't decide whether or not taking up this approach (vs the current rankings) would change the way that students apply to schools. Certainly Princeton's example is a little disheartening for the 96th percentile student; if applicants were armed with more detailed admissions figures the 96th percentile students might apply in lower numbers to Princeton (ceterus paribus) which would harm the school's applicant count -> selectivity % -> ranking under the current regime. Thus Princeton could be expected to fight more disclosure in this case.

Another question I haven't resolved is whether a revealed preference/market ranking would do more harm than good. It's certainly more empirically valid in terms of a preference ranking, but I think it's also clear that some schools coast on a reputation earned a long time ago; the brand of a school may lag what that school is actually producing/providing by a generation. In that sense a revealed preference ranking is a trailing indicator that isn't really useful for anything except a prestige index; for a forward looking student or a university trying to pull itself up by its bootstraps a ranking system has to measure more immediate indicators of output quality. I'll draw an analogy to my view on standardized testing for students; I don't think teaching to the test is bad per se, it just means you have to be very careful to structure the test so that teaching to it effects your ultimate teaching goals. Likewise, rankings parameters should be structured so that managing to rankings metrics serves a positive good. Class size, outgoing placement rates/salary, and ROI are good things to manage to while the current metric for 'selectivity' or acceptance percentages is destructive.

Name

Mail (will not be published)

Website