Jens Beckmann

Researchers rely on psychometric principles when trying to gain understanding of unobservable psychological phenomena disconfounded from the methods used. Psychometric models provide us with tools to support this endeavour, but they are agnostic to the meaning researchers intend to attribute to the data. We define method effects as resulting from actions which weaken the psychometric structure of measurement, and argue that solution to this confounding will ultimately rest on testing whether data collected fit a psychometric model based on a substantive theory, rather than a search for a model that best fits the data. We highlight the importance of taking the notions of fundamental measurement seriously by reviewing distinctions between the Rasch measurement model and more generalised 2PL and 3PL IRT models. We then present two lines of research that highlight considerations of making method effects explicit in experimental designs. First, we contrast the use of experimental manipulations to study measurement reactivity during the assessment of metacognitive processes with factor-analytic research of the same. The former suggests differential performance-facilitating and -inhibiting reactivity as a function of other individual differences, whereas factor-analytic research suggests a ubiquitous monotonically predictive confidence factor. Second, we evaluate differential effects of context and source on within-individual variability indices of personality derived from multiple observations, highlighting again the importance of a structured and theoretically grounded observational framework. We conclude by arguing that substantive variables can act as method effects and should be considered at the time of design rather than after the fact, and without compromising measurement ideals.

Many situations require people to acquire knowledge about, and learn how to control, complex dynamic systems of inter-connected variables. Numerous studies have found that most problem solvers are unable to acquire complete knowledge of the underlying structure of a system through an unguided exploration of the system variables; additional instruction or guidance is required. This paper examines whether providing structural information following an unguided exploration also improves control performance, and the extent to which any improvements are moderated by problem solvers’ fluid intelligence. A sample of 98 individuals attempted to discover the underlying structure of a computer-simulated complex dynamic system. After initially controlling the system with their limited knowledge, half of the sample received guidance concerning the actual underlying structure of the system. All participants then controlled the system again. In contrast to the results of previous studies, guidance did result in immediate improvements in control performance. Fluid intelligence moderated the extent to which problem solvers benefited from structural information. These results indicate that guidance in the form of structural information is critical in promoting knowledge and control over complex and dynamic systems.