American Sociological Association

Comment: Evidence, Plausibility, and Model Selection

In his article, Michael Schultz examines the practice of model selection in sociological research. Model selection is often carried out by means of classical hypothesis tests. A fundamental problem with this practice is that these tests do not give a measure of evidence. For example, if we test the null hypothesis β = 0 against the alternative hypothesis β ≠ 0, what is the largest p value that can be regarded as strong evidence against the null hypothesis? What is the largest p value that can be regarded as any kind of evidence against the null hypothesis? No clear answer to these questions is possible in the classical tradition: all that can be said is that .05 is the conventional standard for “rejecting” the null hypothesis. Moreover, there is no theoretical basis for saying that .05 is better or worse than any other possible convention. Bayesian hypothesis tests have the advantage of giving a definite measure of evidence: the odds in favor of one model against another. However, to carry out a Bayesian test, it is necessary to begin with a prior distribution that represents the alternative hypothesis: rather than a simple statement that β ≠ 0, there must be a probability attached to each nonzero value. Although a theory might provide guidance about the probable values for parameters of interest, most empirical research considers a large number of other parameters about which there are no clear expectations. Thus, although model selection through Bayesian hypothesis tests is appealing in principle, it usually cannot be implemented in practice.1


David L. Weakliem





Starting Page


Ending Page