Judging Quality: Epistemic Communities,
Evaluative Criteria, and Interpretive Empirical Research
I feel nervous when researchers claim that their accounts of reality are far too
rich and complex to be expressed as measurement.
APSA Political Methodology Section email list participant (February 2003)
With methods, as with people, if you focus only on their limitations you will
always be disappointed.
(Ian Shapiro 2002, 612.)
The ideal of “problem-driven” research—that is, that the substantive problem and
specific research question should drive the choice of method—is routinely raised in social
science methods debates. Although this common-sense ideal is applauded by most,
1
in an
age of increasing specialization, it often founders on the fact that it is epistemic
communities that are the arbiters of research, and these communities are defined by their
research gestalt—a bundle of epistemological and ontological presuppositions,
theoretical commitments, research goals, and methodological and reading practices.
Moreover, despite the explicit methodological training in many disciplines, much
learning is still craft-based; that is, novices learn by reading exemplars and by
collaborating with senior researchers. Thus, a simple listing of interpretive evaluative
criteria, though useful in displaying a vocabulary, does little to communicate the
interpretive research gestalt. But once one is steeped in that gestalt, once it is second
nature, then the relevant evaluative criteria are readily brought to mind for assessing the
1
Shapiro (2003) is one proponent. Atkinson et al. are skeptics, arguing that “in the world
of real research, social scientists do not dream up ‘problems’ to investigate out of thin air,
divorced from concerns of theory and methodology, and only then search for precisely
the right method” (2003, 99).
Convention