James Evans

How should we incorporate algorithms into humanistic scholarship? The typical approach is to clone what humans have done but faster, extrapolating expert insights to landfills of source material. But creative scholars do not clone tradition; instead, they produce readings that challenge closely held understandings. We theorize and then illustrate how to construct bad robots trained to surprise and provoke. These robots aren’t the most human but rather the most alien—not tame but dangerous. We explore the relationship between the reproduction of tradition and the generation of surprise and then show how formalizing a particular humanistic theory as a Bayesian prior allows us to identify readings that disrupt it through a process of algorithmic abduction. Among the exploding universe of surprising (and mostly ridiculous) possible readings, algorithmic efficiency allows us to select readings that nonetheless garner substantial support from the total archive and so merit interpretation and engagement. It is in such interaction between alien and human readers that meaning is made and understanding disturbed; it is by extending alien readings to novel texts that the scholarly community tests the value of those readings.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm – and with far greater efficiency.

Introducing this issue’s triptych on algorithms and culture, this article argues that prevailing modes of analysis that focus on the prospects for algorithms “taking over” are no longer useful. It advocates the need for a new conceptual vocabulary, which recognizes that algorithmic and cultural reasoning processes are already enmeshed with each other. The introduction suggests a need for an enterprise of algorithmic epistemology attuned to the fine structure of the ways in which culture and code have interacted in the past and continue to interact today.

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.