Task Force on the Implications of the Evaluation of Faculty Productivity and Teaching Effectiveness

Executive Summary

The Task Force on the Implications of the Evaluation of Faculty Productivity and Teaching Effectiveness was charged in 1999 by ASA Council to determine whether faculty productively measures (including outcomes assessment measures) used by institutions of higher education and various external agencies “threaten the freedom of faculty teaching and research.” The Task Force was asked further to examine the ways in which productivity-reporting requirements affect faculty, to report on “best practices,” and to make any recommendations for appropriate ASA action to the Council.

The report produced by the Task Force highlights the contextual factors that help to explain the increased emphasis in the United States on the evaluation of faculty productivity and the assessment of student learning outcomes including a series of social movements to adapt corporate models to the university and to consider teaching as scholarship. In addition, the Task Force stresses that discussions of faculty productivity are greatly complicated by definitional issues. The Task Force draws a critical distinction between the productivity of individual faculty members and the productivity of larger entities such as departments or colleges. They also distinguish between faculty productivity and student outcomes. Studies of aggregate productivity are complicated by numerous quantitative measures of the “production of students.” While the gathering of data on individuals in the areas of scholarship, teaching, and service has a long history, disputes continue about how to measure these aspects of productivity. Faculty annual activity reports are the most common way to measure these types of productivity. These measures are then aggregated and provided to administrators. When an aggregate is the unit of analysis, faculty productivity can be measured by asking faculty members to complete activity reports that indicate how work time has been used over a given period of time (usually an academic year).

The Task Force found that faculty activity reports raise few red flags for faculty, who see them as a bureaucratic annoyance. However, how such data are aggregated and used by deans and other decision-makers becomes important: departmental “productivity” can guide resource allocation decisions within institutions, and some states have tied appropriations to performance indicators. While some faculty and chairs complain about a lack of connection between productivity data and actual resource allocation decisions, the Task Force cautions that it would be naïve to ignore the potential consequences of productivity measurement for faculty, departments, and institutions. To the extent that these measurements of faculty productivity become part of decision-making systems, they have implications for the substantive nature of the discipline (e.g., creating pressures to focus on training students in areas where students get jobs, encouraging research in areas where grants are plentiful, discouraging teaching in small, writing-intensive, critical courses).

Recognizing that discussions of faculty productivity have led to systems of post-tenure review on many campuses and that discussions of the evaluation of productivity and use of productivity data continue in administrative and legislative circles, the Task Force provides a series of recommendations—abbreviated here—to minimize the misuse of productivity data.

1. The same standard of productivity should not be applied to all disciplines.

2. Caution should be exercised in comparing institutions with differing missions, histories, funding bases, and student bodies.

3. Faculty should be made aware of the types of data that are being collected on them and of the ways in which the data are being reported and used.

4. General discussions of data collection, data quality, and data use should not be divorced from institutions' existing systems of faculty governance and control and faculty need to be ready to organize collectively in opposition to pressures which undermine their ability to control and organize curricula, pedagogy, and research as they see fit given their disciplinary expertise.

5. Faculty—with their considerable methodological expertise across many disciplines—also need to be involved in the technical discussions of how data are to be measured, whether at the departmental, institutional, or system levels.

6. Multiple measures of productivity are inherently preferable to single measures.

7. Non-numeric data need to be incorporated into discussions of productivity so that both context and the quality of efforts are given full weight.

8. The time demands on faculty for data collection need to be kept to a minimum.

9. Junior faculty, in particular, need to be protected from bearing the undue weight of changing pressures for faculty productivity.

The Task Force recognizes that faculty are likely to be familiar with efforts to assess student learning outcomes since outcomes assessment is mandated by all of the regional accrediting bodies in the United States and by some state boards as well. At present, however, there is considerable unevenness in the application of outcomes assessment practices. Further, while a majority of departments in the ASA's 2000 survey of Baccalaureate and Graduate Departments found assessment methods “useful,” there are many individual faculty who, at best, tolerate assessment and who are troubled by the burden of work involved. The Task Force discusses the ways in which outcomes assessment is distinct from the measurement of faculty productivity, yet notes that new linkages that are being forged between these two processes.

Given that assessment is becoming more widespread, the Task Force suggests some general principles that can be used to guide departments and to distinguish between good and bad assessment practices. In sum:

1. Assessment activities need to be separate from personnel decisions.

2. Punitive resource allocation decisions need to be separate from assessment.

3. Departments and faculty must be in charge of the assessment process—from conceptualization through to interpretation.

4. While it is reasonable to ask departments to report on what they are doing and the changes they have made, assessment data must be owned by departments.

5. The assessment process should be decentralized, with departments having the discretion about the framing of learning goals and the collection and interpretation of assessment data.

6. The faculty governance system in place on a particular campus should control the assessment process.

7. A number of faculty must be involved in assessment activities within departments. The most useful component of assessment is the “conversation among faculty” that it encourages.

8. Resources must be available to do the work of assessment and to support faculty development so that departments are not expected to use their existing operating budgets to subsidize assessment activities.

9. The reward structure in place on campuses must give credit to faculty involvement in assessment activities.

10. Assessment is often most successful when it is integrated into existing activities, rather than being seen by faculty or students as an “add on.”

11. The time horizon for completing assessment should be long-term rather than short-term—especially if resources are at stake.

12. Attention must be given to the quality of measurement. Consequently, discussions must focus on issues of reliability and validity, and the use of multiple measures is essential.

13. Understanding the context in which data are gathered is important. Qualitative data can be especially useful to highlight context and may provide insights about student learning which are difficult to garner with quantitative data.

14. Efforts must be made not to compare disparate departments or colleges.

The Task Force believes that sociologists have key research and conceptual skills that qualify them to be active contributors to discussions about both faculty productivity measurement and outcomes assessment. To encourage more active involvement by sociologists in both on-campus and national discussions, the Task Force makes the following recommendations—presented here in condensed form—to the ASA Council.

1. This report in its entirety should be distributed to members of the ASA Council, to ASA staff, to department affiliates, and to the individuals who agreed to be interviewed by task force members. A link to the report should be available on the ASA's web site, the report should be summarized in Footnotes, and a session on this report should be presented at the 2004 annual meetings.

2. While efforts to measure faculty productivity and to engage in outcomes assessment have not led to the dire consequences some have predicted, the ASA should maintain vigilance in these areas by using Footnotes and the Chairs Workshop at the annual ASA meetings to solicit feedback on a biannual basis about whether these kinds of data gathering activities are creating academic freedom and other problems for faculty and departments.

3. ASA staff is encouraged to meet periodically with their counterparts in other learned societies to monitor the ways in which these data gathering activities are impacting faculty and their professions and should share resources and host joint workshops when time and place allow.

4. The ASA should continue to provide resources to departments to assist especially with assessment efforts (e.g., ASA's “teaching and academic resources,” academic workplace workshops and regular sessions at meetings, DRG experts).

5. The ASA should encourage the regional and state associations to hold workshops and sessions on assessment issues.

6. While recognizing the need for editors to maintain control of their publications, the ASA should encourage Teaching Sociology, Sociology of Education, and VUES (the newsletter of the section on Teaching and Learning in Sociology) to give attention to assessment issues.