TF on Faculty Productivity: Outcomes Assessment | American Sociological Association

Outcomes Assessment

Faculty are more likely to be familiar with efforts to assess student learning outcomes, since many academics have been required to undertake such assessments for their programs. Outcomes assessment is being mandated by all of the regional accrediting agencies across the United States as well as by many of the specialized accreditation bodies. In addition, some state boards require that campuses engage in outcomes assessment using either locally created assessment instruments or instruments required by campuses throughout the state. So, it is likely that more faculty will become familiar with it in future. 71 percent of the Chairs surveyed by the Task Force reported that they had been asked to engage in the assessment of student learning.

Numerous commentators have defined assessment. Most definitions suggest that assessment involves three components—the establishment of learning goals or outcomes for students, the determination of the extent to which students have achieved those goals or outcomes, and efforts to make improvements in pedagogy or curricula if a gap exists between expectations for students and actual performance. The very nature of assessment requires that we sociologists agree within our departments on what is important for students to learn and that we develop ways of measuring what we believe is important.

While faculty have always assessed students by assigning grades to their work and credit to their course completion, what is new in contemporary discussions of assessment is the focus on student learning rather than faculty teaching, the emphasis on improvement rather than simply accountability, and the focus on the program or curriculum rather than the individual (isolated) course as the unit of analysis. In fact, assessment challenges the idea that courses belong to individual faculty members and that teaching and learning are synonymous. Assessment also encourages discussions of the advantages of collectively developing an integrated curriculum. Further, some outcomes assessment efforts focus on the process of student learning (e.g., their engagement in academic activities) rather than on simply their knowledge, skills, and attitudes when they graduate.

At present, there is considerable unevenness in the application of outcomes assessment practices. Some institutions have enthusiastically embraced it and made it part of their institutional culture, others are only beginning to talk about it. There are enormous variations in the techniques being used to do assessment: one encounters, among many others, the use of standardized tests, locally produced tests and interviews, portfolios, exit interviews, alumni surveys, questionnaires with employers, and capstone experiences.

Wagenaar (2002) examined the use of outcomes assessment in sociology departments. He found that alumni and senior surveys and capstone products are used most often, while commercial exams, external review of student work, and oral exams are used least often. The American Sociological Association Survey of Baccalaureate and Graduate Programs in Sociology (2000) largely confirmed this finding; departments surveyed reported that Student Surveys, Senior Theses and/or Projects, and Exit Interviews were the most commonly used types of student assessment. The Task Forces less extensive survey of chairs of Sociology departments found a similar pattern. While these general patterns held across different types of institutions, some differences do appear to exist for school type and size. The ASA Survey indicates that departments in Ph.D-granting institutions were less likely than others to use Senior Theses or Projects while departments in Liberal Arts Colleges were more likely than others to use departmental exams. Similarly, departments in Baccalaureate II colleges were more likely than others to use External Exams.

IMPLICATIONS OF MEASUREMENT FOR FACULTY AND CHAIRS

While outcomes assessment is becoming widespread, its impact on the curriculum and teaching is modest at best, with faculty members at private and smaller schools reporting greater impact (Wagenaar 2002). We are now beginning to learn more about faculty attitudes towards assessment. The ASA's survey of Baccalaureate and Graduate Departments (2000) found that the majority of departments in all kinds of institutions found most assessment methods to be “useful” (although there was noticeably less enthusiasm about external and departmental exams, especially in Ph.D.-granting institutions). The same study, however, found that only senior theses were regarded as “very useful” by a majority of departments. These findings suggest that, at the aggregate level, outcomes assessment is accepted but with something less than unbounded enthusiasm.

Attitudes to assessment clearly also vary by department and individual. There are individuals and campuses that believe in the value of assessment. They see it as a tool they can use to determine whether their teaching is effective, whether their curricula make sense to students, and to develop ways to improve and enrich the way in which they teach. In addition, some faculty argue that discussions about assessment lead to more cohesive departments that are more enjoyable places for faculty to work. Chairs surveyed by the Task Force identified curricular revision and departmental self-assessment as the primary advantages of outcomes assessment. Womack, Nichols, and Nichols (1999) argue that the department is the locus of both instruction and the educational experiences of students and should, therefore, also be the locus of most assessment activities. They observe that departmental level assessment enhances departmental commitment to schools' missions.

But there are also many who, at best, tolerate it, seeing it as an exercise that they are obliged to go through. Some see it as an externally imposed requirement whose value to them or their students is unclear. Some also complain of the burden of work involved, particularly where reporting requirements are unclear or where institutions demand the same information in multiple forms. Sociology Chairs mentioned the “time burden” it imposes as assessment's primary disadvantage. Further, the attention to assessment is episodic on some campuses, largely a function of the interests of the chief academic officers or the immediate demands occasioned by accreditation visits.

Faculty also clearly articulates their need for assistance with the assessment process. Sociologists are concerned with issues regarding the measurement of student outcomes. They want examples of how to do assessment well and examples of “best practices.”

RECOMMENDATIONS TO INCREASE USEFULNESS OF OUTCOMES ASSESSMENT

What is abundantly clear is that assessment is becoming more widespread and that it can be done both badly and well. Some general principles are beginning to emerge that can be used to guide departments and to distinguish between good and bad assessment practices. Among them, we can mention:

1. Assessment activities need to be separate from personnel decisions so that faculty will be willing to ask difficult questions that can lead to improved curricula, pedagogy, and student learning without fearing the consequences for their career success.
2. Punitive resource allocation decisions need to be separate from assessment. Departments must have the option of choosing when to report assessment findings when decisions about resource allocation are at issue. Even using assessment data to make positive resource allocations raises critical issues: departments that are not rewarded may feel that they have been “punished” and there are real questions about the fairness of such decisions generated by the uncertainty of measurement and the limited time period for which assessment data are available.

3. Departments and faculty must be in charge of the assessment process—from conceptualization through to interpretation. Assessment is misdirected when Deans provide the interpretation of data or define the learning goals. Assessment must be useful to departments. It must involve asking questions that faculty want to address. No one assessment model works across departments and institutions; while faculty should be encouraged to borrow from one another it is unlikely that a single model can be adopted in toto.

4. While it is reasonable to ask departments to report on what they are doing and the changes they have made, assessment data must be owned by departments. Departments should have discretion about when and whether raw findings are made public.

5. The assessment process should be decentralized, with departments having the discretion about the framing of learning goals and the collection and interpretation of assessment data. The diversity of approaches to assessment that results from this decentralization is, in fact, an advantage. Departments should resist efforts to make comparisons with aggregated, national data.

6. The faculty governance system in place on a particular campus should control the assessment process. Clear policies should be developed and endorsed at the front end, rather than as problems arise.

7. A number of faculties must be involved in assessment activities within departments. The most useful component of assessment is the “conversation among faculty” that it encourages. Consequently, assessment cannot be the responsibility of just one individual. Efforts should be directed to ensuring that junior faculty is not expected to complete a disproportionate amount of a department's assessment activity.

8. Resources must be available to do the work of assessment so that departments are not expected to use their existing operating budgets to subsidize assessment activities. Similarly, faculty development opportunities should be made available to faculty so that they can learn from one another and from the past experiences of other institutions. Assessment can be made more economically efficient by encouraging cooperation among departments with similar learning goals and data collection strategies.

9. The reward structure in place on campuses must give credit to faculty involvement in assessment activities.

10. Assessment is often most successful when it is integrated into existing activities, rather than being seen by faculty or students as an “add on.” On some campuses, assessment will be seen and credited as an extension of teaching rather than as a new faculty responsibility. The data collection associated with assessment can often be made a part of the credit-bearing activities associated with a course or program; this is preferable to expecting students or faculty to complete additional out-of-class activities simply for the purpose of assessment.

11. The time horizon should be long-term rather than short-term—especially if resources are at stake. Faculty has only so much time to devote to assessment in a given year. Further, developing a “culture of evidence” within a department or institution is an on-going effort rather than a one-shot activity designed to meet an administrative calendar. Care should be taken to ensure that assessment is not so time-consuming that other important faculty and department goals suffer because of the energies directed to assessment.

12. Attention must be given to the quality of measurement. Consequently, discussions must focus on issues of reliability and validity, and the use of multiple measures is essential.

13. Understanding the context in which data are gathered is important. Qualitative data can be especially useful to highlight context and may provide insights about student learning which are difficult to garner with quantitative data.

14. As we have seen regarding studies of faculty productivity, efforts must be made not to compare disparate departments or disparate colleges.

A final concern regarding outcome assessment is the possibility that national standards or measures of student learning may emerge. In many states, K-12 teachers find themselves confronted by the need to enable their students to pass state-mandated proficiency tests in various subjects. These tests are controlled not by local schools but by statewide bodies. This, in turn, creates concern that local schools will lose control over their curricula and that teachers are being encouraged to “teach to the test” in order to improve their school's “report card.” Thus far, no analogous movement has developed in higher education. Still, regional accrediting bodies do exercise an at least mild standardizing effect on assessment (since all of the schools evaluated are scrutinized according to the same criteria). Moreover, the temptation to use standardized tests (such as the GRE subject tests) to evaluate student learning encourages faculty to teach to the test. Even the recently developed National Study of Student Engagement (NSSE), which examines how academically engaged students are, could become a standardizing force if it becomes an alternative to the popular US News & World Report measures of institutional quality. If nothing else, faculty need to be aware of the possibility that outcomes assessment, which can be a valuable pedagogical tool when controlled locally, could become something quite different if it becomes a standardized, one-size-fits-all exercise.

As with the measurement of faculty productivity, sociologists have distinctive research and conceptual skills that qualify them to be active contributors to discussions of outcomes assessment and how to do it. The fact that there are both good and bad practices in use, and good and bad ways of reporting and using assessment data, makes it all the more important that sociologists become actively involved in discussions of assessment. Further, sociologists can contribute much to the discussions about the ways in which higher education is changing from pressures both external and internal to it. In this light, it is important to recognize the ways in which assessment can be used against departments, programs, and individual faculty members. With this in mind, the discipline and its members are encouraged to take precautions against data misuse and to be vigilant in the creation of institutional policies and in the analysis of institutional trends.

LINKAGES BETWEEN PRODUCTIVITY ISSUES AND ASSESSMENT

Interest in measuring faculty productivity (particularly at the aggregate level) has different roots than the outcomes assessment movement. The former derives from the growing economic pressures on universities faced with declining state support; the latter is rooted in the concern for student performance highlighted by documents such as A Nation at Risk. There are both positive and negative aspects to both of these practices, as we have noted. However, it is also the case that, at times, these two processes are being linked on some campuses:

Assessment data are being asked for in productivity studies (e.g., alumni survey data, graduation rate data)

Campus reviews of programs for resource allocation purposes (e.g., program review) ask for assessment data (data on student outputs) in addition to the traditional data on “inputs” or faculty “outputs” (e.g., productivity indices).

Furthermore, there are tensions between the growing demand that faculty assess student learning and the concomitant pressures to be productive:

The time-consuming nature of assessment and the consequent focus on teaching may lead to decreased scholarly productivity among faculty.

Assessment itself tends to encourage more attention to teaching while faculty productivity reports have traditionally given more attention to scholarship.

Generally speaking, other important institutional goals—e.g., diversity, general education—may be undervalued when so much attention is placed on either assessment or faculty productivity.

Faculty governance mechanisms need to pay more attention to the contradictory demands being made of faculty. They also need to work to keep separate the activities of assessing productivity and assessing student outcomes so that the latter, in particular, can be done in a way that is useful to both faculty and students.

Those who grumble about assessment appear to be most concerned about the amount of work it requires them to do. There appears to be less discussion of potential threats to academic freedom or faculty rights, although there is always friction when departments make collective decisions about how to “improve” individuals' courses. Some faculty simply do not like the idea of someone else telling them what they should be doing, especially when the “someone else” is external to the academy or the faculty governance system. The time-consuming nature of assessment also leads some faculty to worry that their teaching effectiveness will actually decline as energies are directed to assessing student learning rather than promoting it. In addition, some faculty feel that they must develop assignments and examinations that can easily be used for departmental assessment purposes or feel that with assessment they must “teach to the test.” Interest in measuring faculty productivity (particularly at the aggregate level) has different roots than the outcomes assessment movement. The former derives from the growing economic pressures on universities faced with declining state support; the latter is rooted in the concern for student performance highlighted by documents such as A Nation at Risk. There are both positive and negative aspects to both of these practices, as we have noted. However, it is also the case that, at times, these two processes are being linked on some campuses.