May/June 2011 Issue • Volume 39 • Issue 5

download pdfDownload full issue pdf

From the Executive Officer

ASA Council Weighs in on the NRC Doctorate Program Rankings Debate

sally hillsman

Sally T. Hillsman,
ASA Executive
Officer

At its February 2011 meeting, the Council of the American Sociological Association accepted the report of its ad-hoc committee on the 2010 National Research Council’s (NRC) Assessment of Doctorate Programs. Last fall, ASA President Randall Collins had charged this committee chaired by Council-member-at-large Mario Small to review the NRC report and its methodology and to prepare a set of recommendations to Council on how the ASA should respond, especially given the many concerns being raised by ASA members and sociology departments since the report’s release in September 2010. The ASA President and Council viewed it as important for the ASA to weigh in on behalf of the discipline, if consensus among current members could be reached about this matter of concern to sociology.

Background

The 2010 NRC rankings were preceded by the influential 1995 NRC rankings, which had been used for comparative purposes by students considering graduate programs, research funders, and administrators as well as by universities seeking to evaluate and improve the quality of their doctoral programs. While the 2010 NRC ranking were to replace the outdated 1995 rankings, there were significant funding issues and intellectual controversy within the NRC panel so that the report was released three years behind schedule, and the extensive information collected was already dated. Controversy has not subsided.

The 1995 NRC rankings were based on a reputational survey carried out in 1993. The National Survey of Graduate Faculty asked faculty at U.S. universities their views about their doctoral departments’ overall scholarly reputations, educational effectiveness, and change in quality over the prior decade. (www.nap.edu/readingroom.php?book=researchdoc&page=summary.html). The study, however, did not collect other information about the doctoral programs to substantiate the reputational judgments, such as the scholarly publication record of faculty.

Not surprisingly, the 1995 NRC rankings were criticized for their reliance on reputational data. Critics said that such an approach failed to effectively recognize strong departments in niche fields or smaller programs. They were also criticized for being unable to distinguish scholarly productivity from educational effectiveness, for undermining the value of programs that, while not producing a large number of books or articles, were nonetheless particularly effective in training future scholars. Finally, the 1995 rankings provided no measure of uncertainty regarding the rankings. Readers could not assess whether a difference of a few positions in either direction reflected an actual difference or one due to chance.

The 2010 NRC rankings

The 2010 NRC rankings (http://www.nap.edu/rdp/)were expected by many to avoid the problems of the 1995 rankings while adding the transparency missing from some other existing rankings (i.e., the Gourman Report). Perhaps not surprisingly, the new NRC report on rankings has received even more criticism than the 1995 report. Those criticsms again focus on a range of structural and methodological issues. Many sociology departments that appeared as "winners," as well as those less favorably ranked, voiced concern that the new methodologies used failed in significant ways to evaluate sociology effectively as a scholarly discipline. This was a concern for the ASA Council.1

Ad-hoc Committee Recommendations

The ad-hoc committee recommended that the ASA Council issue a resolution "criticizing the 2010 NRC rankings for containing both operationalization and implementation problems; discouraging faculty, students, and university administrators from using any of the five 2010 NRC rankings to evaluate sociology programs; encouraging faculty, students, and university administrators to be suspicious of the spreadsheet data produced by the report; and indicating that reputational surveys have their own sets of biases." Instead of issuing a separate statement, Council accepted the whole ad-hoc committee report and asked it to be circulated on behalf of the Association (see www.asanet.org/about/Council_Reports.cfm).

This was in response to the report’s  recommendation that "ASA should encourage prospective students, faculty, university administrators or others evaluating a given program to avoid blind reliance on rankings that claim explicitly or implicitly to list departments from best to worst." The diversity of the discipline suggests that department evaluators should first determine what characteristics they value in a program and then employ available sources of information to assess the program’s performance.

The ASA Council requested that the Executive Office make the report public and available on the ASA homepage; provide copies to appropriate officials at the National Academies, the National Research Council, and the National Science Foundation as well as to Sociology Department Chairs, Directors of Graduate Education, and Graduate Deans; and circulate a summary of the report as appropriate.

Last month ASA staff sent the report to 14 high-ranking officials, including Ralph Cicerone, President of the National Academy of Sciences, and Cora Marrett, Senior Advisor at the National Science Foundation. The report was also sent to the Chronicle of Higher Education and to Inside Higher Ed and placed as a news item on the ASA homepage.

Council’s Role

The implications of the NRC graduate department rankings for sociology departments are viewed seriously by Council, and when the ranking’s flaws first became apparent Council responded. Council works deliberately and diligently to best represent the needs of the discipline by also being selective in which issues to address. It felt the potential importance of use and misuse of the NRC rankings in sociology made a statement from the discipline’s national association important. While there may be concerns about the rankings not addressed by the Council report, the key critiques provide a framework for understanding the rankings’ limitations and maximizing useful interpretations of the data by individuals, departments, and universities. Council is also sponsoring an Open Forum on the NRC rankings report at the Annual Meeting in Las Vegas on Sunday, August 21, from 8:30 am to 10:10 am.  

ASA is a membership association and governance sets the tone—internally and externally—for how the Association operates and is perceived. The willingness of Council as ASA’s elected leadership to address publicly discipline-wide issues, including flaws in the NRC rankings, demonstrates a focused empowerment for our discipline, departments and scholars, and a context for discussion.

For more information, ASA members should check the "governance" section of the ASA website under "About ASA" www.asanet.org/about/governance.cfm. There is a wealth of information there, beyond the usual governing documents such as the ASA Constitution and Bylaws. Governance includes the Council Minutes; Council’s Issue-focused Statements; Council-accepted Reports; Amicus Briefs; as well as the Association’s History and Archives. And, of course, the ASA financial audits. 

References

1 These concerns also led some sociologists, such as Jonathan Cole in a Chronicle of Higher Education article, who resigned from the NRC committee rather than write a dissent, to argue that the $4 million five-year study is a failure.

Sally T. Hillsman is the Executive Officer of ASA.
She can be reached by email at executive.office@asanet.org.

Back to Top of Page

Print this article discuss this article

Special Announcement:

twitter ad

Back to Front Page of Footnotes | Table of Contents