NSF Report Tackles Standards of Evaluation for Qualitative Research
by Michèle Lamont, Harvard University, and Patricia White, National Science Foundation
Academics from the disciplines of anthropology, law and social science, political science, and sociology gathered for a two-day workshop in May 2005, on "Interdisciplinary Standards for Systematic Qualitative Research" at the National Science Foundation (NSF) (see www.asanet.org/footnotes/mar05/fn6.html). The results of this workshop were recently released in an NSF report.
It is well recognized that the four disciplines have different research design and evaluation cultures as well as considerable variability in the emphasis on interpretation and explanation, commitment to constructivist and positivist epistemologies, and the degree of perceived consensus about the value and prominence of qualitative research methods. Within this multidisciplinary and multimethods context, 24 scholars from the four disciplines sought to (1) articulate the standards used in their particular field to ensure rigor across qualitative methodological approaches; (2) identify common criteria across the four disciplines for designing and evaluating research proposals and fostering multidisciplinary collaborations; and (3) develop an agenda for strengthening the tools, training, data, research design, and infrastructure for research using qualitative approaches. The explicit purpose was not to define standards for all qualitative researchers but for proposals submitted for funding to the NSF.
Prior to the workshop, each participant prepared a short paper addressing the three topics (qualitative research standards, evaluation criteria, and agenda of future opportunities and needs). At breakout sessions, each disciplinary group was asked to discuss and list the field’s major qualitative research standards. From these lists the full workshop identified two categories of qualitative research standards: "Shared" or key standards relevant to all four disciplines and standards judged essential by two or three disciplines. Participants also provided recommendations for improving qualitative research, training, and infrastructures and identified promising areas of research in the four disciplines that would benefit from the use of qualitative research approaches.
The workshop was cofunded by a grant from four NSF Programs—Cultural Anthropology, Law and Social Science, Political Science, and Sociology—to Michèle Lamont, Harvard University. Lamont organized the workshop with representatives from each discipline who coordinated group reports. The Cultural Anthropology group was co-chaired by Ted Bestor (Harvard) and Gery Ryan (RAND); Law and Social Science by John Bowen, (Washington University-St. Louis); Political Science by Andrew Bennett (Georgetown University); and Sociology by Kathleen Blee (University of Pittsburgh).
The recently released workshop report can be accessed at www.nsf.gov/sbe/ses/soc/ISSQR_workshop_rpt.pdf. It is organized into four sections: Qualitative Research Design and Methods; Standards for Qualitative Research across Disciplines; Recommendations for Producing Top Notch Qualitative Research; and Promising New Research Areas and Topics. Below is a brief summary of the report and the results from the workshop.
Qualitative Design and Methods
Workshop participants discussed the strengths of qualitative methods, standards qualitative research shares with quantitative research, and standards that are unique to qualitative research. The major strength of qualitative research was judged to be the rich range of methodological tools available to study meaning, social processes, and group variations. The sample sizes and strategies of qualitative and quantitative approaches to research were contrasted, including the recursive character of theory generation in qualitative research. Qualitative research stresses in-depth contextualization, usually with small sample size. Qualitative research sampling techniques, while nonrandom, are usually attentive to demographic and theoretical dimensions. The small and in-depth samples chosen for theoretical relevancy allow qualitative research findings a degree of significance beyond individuals or single cases and provide opportunities to demonstrate rigor in all phases of a qualitative research project.
"Shared" Criteria Across Disciplines
Workshop participants agreed that the four disciplines shared several standards for designing and evaluating high-quality qualitative research. All value projects that:
- Situate the research in appropriate literature; that is, the study should build upon existing knowledge;
- Articulate clearly the connection between theory and data;
- Describe and explain case selection—why particular sites, participants, events, or cases are chosen;
- Pay attention to alternative explanations and negative cases;
- Operationalize constructs and describe expected findings;
- Provide clear and detailed descriptions of both data collection and anticipated data analysis techniques: specify what counts as data, how the researcher will go about obtaining and analyzing data;
- Describe the intellectual, social, and political significance of the research;
- Discuss generalizability or significance beyond the specific cases selected;
- Specify the limitations of the research and anticipate potential reviewer objections;
- Discuss the researcher preparation for the proposed project in terms of: Cultural fluency; Language skill; Appropriate methodological/technical training; Knowledge of particular research context.
In order to produce top-notch results, participants established that qualitative research could be enhanced by increased investments in education, training, and infrastructure. For example, NSF could enhance qualitative research through increased funding for students and scholars as well as publicizing its commitment to supporting high-quality qualitative proposals. Support for pre-dissertation funding, small pilot study grants for faculty, and student training opportunities through professional associations would likely be most effective.
Promising New Research Areas and Topics
Each disciplinary working group articulated new and exciting research areas that would benefit immensely from qualitative research or are taking advantage of qualitative methods, and in the process they are opening up new avenues of understanding. Social and cultural anthropological studies of responses to climate change, natural resources management, genetically modified food, food scarcity, and the global food trade would be greatly enhanced by being subject to systematic qualitative analysis. Law and social science scholars are conducting interdisciplinary research on conservation, intellectual property, and medicine by paring with social studies of science and technology researchers who are using a range of qualitative approaches. In political science the rich contextualization of qualitative research could be used to more fully understand the rise of religious movements, relations between racial/ethnic and class identities and political actions, and policy making processes, especially across institutions. Finally, most topics that sociologists study are amenable to both qualitative and quantitative analysis, but qualitative methods are particularly useful for studying timely topics such as group identities and boundaries; globalization at the micro level; race, class, gender, and age and health outcomes; and social and cultural meanings of food and obesity.