Jana Uher

This article explores in-depth the metatheoretical and methodological foundations on which rating scales—by their very conception, design and application—are built and traces their historical origins. It brings together independent lines of critique from different scholars and disciplines to map out the problem landscape, which centres on the failed distinction between psychology’s study phenomena (e.g., experiences, everyday constructs) and the means of their exploration (e.g., terms, data, scientific constructs)—psychologists’ cardinal error. Rigorous analyses reveal a dense network of 12 complexes of problematic concepts, misconceived assumptions and fallacies that support each other, making it difficult to be identified and recognised by those (unwittingly) relying on them (e.g., various forms of reductionism, logical errors of operationalism, constructification, naïve use of language, quantificationism, statisticism, result-based data generation, misconceived nomotheticism). Through the popularity of rating scales for efficient quantitative data generation, uncritically interpreted as psychological measurement, these problems have become institutionalised in a wide range of research practices and perpetuate psychology’s crises (e.g., replication, confidence, validation, generalizability). The article provides an in-depth understanding that is needed to get to the root of these problems, which preclude not just measurement but also the scientific exploration of psychology’s study phenomena and thus its development as a science. From each of the 12 problem complexes; specific theoretical concepts, methodologies and methods are derived as well as key directions of development. The analyses—based on three central axioms for transdisciplinary research on individuals, (1) complexity, (2) complementarity and (3) anthropogenicity—highlight that psychologists must (further) develop an explicit metatheory and unambiguous terminology as well as concepts and theories that conceive individuals as living beings, open self-organising systems with complementary phenomena and dynamic interrelations across their multi-layered systemic contexts—thus, theories not simply of elemental properties and structures but of processes, relations, dynamicity, subjectivity, emergence, catalysis and transformation. Philosophical and theoretical foundations of approaches suited for exploring these phenomena must be developed together with methods of data generation and methods of data analysis that are appropriately adapted to the peculiarities of psychologists’ study phenomena (e.g., intra-individual variation, momentariness, contextuality). Psychology can profit greatly from its unique position at the intersection of many other disciplines and can learn from their advancements to develop research practices that are suited to tackle its crises holistically.

Measurement creates trustworthy quantifications. But unified frameworks applicable to all sciences are still lacking and discipline-specific terms, concepts and practices hamper mutual understanding and identification of commonalities and differences. Transdisciplinary and philosophy-of-science analyses are used to compare metrologists’ structural framework of physical measurement with psychologists’ and social scientists’ fiat measurement of constructs. The analyses explore the functions that measuring instruments and measurement-executing persons in themselves fulfil in data generation processes, and identify two basic methodological principles critical for measurement. (1) Data generation traceability requires that numerical assignments depend on the properties to be quantified in the study objects (object-dependence). Therefore, scientists must establish unbroken documented connection chains that directly link (via different steps) the quantitative entity to be measured in the study property with the numerical value assigned to it, thereby making the assignment process fully transparent, traceable and thus reproducible. (2) Numerical traceability requires that scientists also directly link the assigned numerical value to known standards in documented and transparent ways, thereby establishing the results’ public interpretability (subject-independence). The article demonstrates how these principles can be meaningfully applied to psychical and social phenomena, considering their peculiarities and inherent limitations, revealing that not constructs in themselves but only their indicators (proxies) can be measured. These foundational concepts allow to distinguish measurement-based quantifications from other (subjective) quantifications that may be useful for pragmatic purposes but lack epistemic authority, which is particularly important for applied (e.g., legal, clinical) contexts. They also highlight new avenues for establishing transparency and replicability in empirical sciences.

Quantitative data are generated differently. To justify inferences about real-world phenomena and establish secured knowledge bases, however, quantitative data generation must follow transparent principles applied consistently across sciences. Metrological frameworks of physical measurement build on two methodological principles that establish transparent, traceable—thus reproducible processes for assigning numerical values to measurands. Data generation traceability requires implementation of unbroken, documented measurand-result connections to justify attributing results to research objects. Numerical traceability requires documented connections of the assigned values to known quantitative standards to establish the results' public interpretability. This article focuses on numerical traceability. It explores how physical measurement units and scales are defined to establish an internationally shared understanding of physical quantities. The underlying principles are applied to scrutinise psychological and social-science practices of quantification. Analyses highlight heterogeneous notions of ‘units’ and ‘scales’ and identify four methodological functions; they serve as (1) ‘instruments’ enabling empirical interactions with study phenomena and properties; (2) structural data format; (3) conceptual data format; and (4) conventionally agreed reference quantities. These distinct functions, employed in different research stages, entail different (if any) rationales for assigning numerical values and for establishing their quantitative meaning. The common numerical recoding of scale categories in tests and questionnaires creates scores devoid of quantitative information. Quantitative meaning is created through numeral-number conflation and differential analyses, producing numerical values that lack systematic relations to known quantity standards regarding the study phenomena and properties. The findings highlight new directions for the conceptualisation and generation of quantitative data in psychology and social sciences.2519-2548.

Various lines of critique of quantitative psychology, well-established and new, are used to trace along the field's typical steps of research a complex network of misconceptions and fallacies codified in psychological jargon. The article explores what constructs actually are, why they are needed in psychology, fallacies and challenges in construct research, and the crucial role of language. It shows how common misconceptions of language and concepts mislead psychologists to conflate phenomena, qualities, quantities and constructs with one another and with their semiotic encodings in terms, variables and scores. The article clarifies the conceptual relations between nomological networks, representation theorems and psychometric modelling. It reveals conflations of disparate notions of causality and unobservability, and erroneous equations of nomological networks with semantic networks, description with explanation, and measurement theories with explanatory theories. Instead of establishing causal measurand-result relations, common practices match data generation to the results rather than the phenomena and properties studied. Mathematical meaning for scores is often created from differences between individuals and between different phenomena and properties, which constitute mere conceptual entities and cannot reflect magnitudes attributable to individuals. This entails biased inferences on the actual study phenomena and shows that replicability problems may be even larger than assumed.

Data generation methods differ across the empirical sciences. Today’s physicists and engineers primarily generate data with automated technologies. Behavioural, psychological and social scientists explore phenomena that are not technically accessible (e.g., attitudes, social beliefs) or only in limited ways (e.g., behaviours) and therefore generate data primarily with persons. But human abilities are involved in any data generation, even when technologies are used and developed. This article explores concepts and methods of data generation of different sciences from transdisciplinary and philosophy-of-science perspectives. It highlights that empirical data can reveal information about the phenomena under study only if relevant properties of these phenomena have been encoded systematically in the data. Metatheoretical concepts and methodological principles are elaborated that open up new perspectives on methods of data generation across the empirical sciences, highlighting commonalities and differences in two pivotal points: (1) in the accessibility that various kinds of phenomena have for the persons generating the data and for the researchers, and (2), as a consequence thereof, in the processes involved in the encoding of information from these phenomena in the signs (symbols) used as data. These concepts and principles cut across establish method categorisations (e.g., human-generated versus instrument-generated data; quantitative versus qualitative methods), highlighting fundamental issues equally important in all sciences as well as essential differences. They also provide novel lines of argumentation that substantiate psychologists’ and social scientists’ increasing criticism of their own disciplines’ focus on standardised assessment methods and establish connections to concepts of data generation developed in metrology.

This article provides a comprehensive critique of psychology's overreliance on statistical modelling at the expense of epistemologically grounded measurement processes. It highlights that statistics deals with structural relations in data regardless of what these data represent, whereas measurement establishes traceable empirical relations between the phenomena studied and the data representing information about them. These crucial epistemic differences are elaborated using Rosen's general model of measurement, involving the coherent modelling of the (1) objects of research, (2) data generation (encoding), (3) formal manipulation (e.g., statistical analysis) and (4) result interpretation regarding the objects studied (decoding). This system of interrelated modelling relations is shown to underlie metrologists' approaches for tackling the problem of epistemic circularity in physical measurement, illustrated in the special cases of measurement coordination and calibration. The article then explicates psychology's challenges for establishing genuine analogues of measurement, which arise from the peculiarities of its study phenomena (e.g., higher-order complexity, non-ergodicity) and language-based methods (e.g., inbuilt semantics). It demonstrates that psychometrics cannot establish coordinated and calibrated modelling relations, thus generating only pragmatic quantifications with predictive power but precluding epistemically justified inferences on the phenomena studied. This epistemic gap is often overlooked, however, because many psychologists mistake their methods' inbuilt semantics—thus, descriptions of their study phenomena (e.g., in rating scales, item variables, statistical models)—for the phenomena described. This blurs the epistemically necessary distinction between the phenomena studied and those used as means of investigation, thereby confusing ontological with epistemological concepts—psychologists' cardinal error. Therefore, many mistake judgements of verbal statements for measurements of the phenomena described and overlook that statistics can neither establish nor analyze a model's relations to the phenomena explored. The article elaborates epistemological and methodological fundamentals to establish coherent modelling relations between real and formal study system and to distinguish the epistemic components involved, considering psychology's peculiarities. It shows that epistemically justified inferences necessitate methods for analysing individuals' unrestricted verbal responses, now advanced through artificial intelligence systems modelling natural language (e.g., NLP algorithms, LLMs). Their increasing use to generate standardised descriptions of study phenomena for rating scales and constructs, by contrast, will only perpetuate psychologists' cardinal error—and thus, psychology's crisis.

Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article’s basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects’ peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors (‘personality’). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters’ interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena’s properties and the measurement-executing persons’ abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Behaviour is central to many fields, but metatheoretical definitions specifying the most basic assumptions about what is considered behaviour and what is not are largely lacking. This transdisciplinary research explores the challenges in defining behaviour, highlighting anthropocentric biases and a frequent lack of differentiation from physiological and psychical phenomena. To meet these challenges, the article elaborates a metatheoretical definition of behaviour that is applicable across disciplines and that allows behaviours to be differentiated from other kinds of phenomena. This definition is used to explore the phenomena of language and to scrutinise whether and under what conditions language can be considered behaviour and why. The metatheoretical concept of two different levels of meaning conveyed in human language is introduced, highlighting that language inherently relies on behaviours and that the content of what-is-being-said, in and of itself, can constitute behaviour under particular conditions. The analyses reveal the ways in which language meaningfully extends human's behavioural possibilities, pushing them far beyond anything enabled by non-language behaviours. These novel metatheoretical concepts can complement and expand on existing theories about behaviour and language and contribute a novel piece of theoretical explanation regarding the crucial role that language has played in human evolution.

Psychometrics has always been confronted with fundamental criticism, highlighting serious insufficiencies and fallacies. Many fallacies persist, however, because each critic explores only some fallacies while still building on others. This article scrutinizes the epistemological, metatheoretical, and methodological foundations of psychometrics, revealing a complex network of numerous conceptual fallacies underlying its framework of theory and practice. At its core lies a key challenge for psychology: the necessity to distinguish the phenomena under study from the means used to explore them (e.g., concepts, methods, data). This distinction is intricate because concepts constitute psychical phenomena in themselves and many psychical phenomena are accessible only through language-based methods. The analyses show how insufficient consideration of this important distinction and common misconceptions about concepts and language (e.g., signifier–referent conflation, reification of constructs) led to conflations of disparate notions of key terms in psychological measurement (e.g., “variables”, “attributes”, “causality”) and numerous interrelated fallacies (e.g., construct–referent conflation, phenomenon–quality–quantity conflation, numeral–number conflation). These fallacies are maintained and masked by repeated conceptual back-and-forth switching between two incompatible epistemological frameworks, (a) an operationist framework of data modeling implemented through methodical and statistical operations and (b) a realist framework of measurement sporadically invoked in theoretical considerations but neither theoretically elaborated nor empirically implemented. The analyses demonstrate that psychometrics constitutes only data modeling but not data generation or even measurement as often assumed and that analogies to (indirect or fundamental) physical measurement are mistaken. They provide theoretical support for the increasing criticism of psychometrics and its use in research and applied contexts.

The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals builds on established concepts, approaches, and methods from various disciplines that are systematically integrated into coherent philosophical, metatheoretical, and methodological frameworks and that are further developed and complemented by novel ones. The TPS-Paradigm is aimed at supporting scientists exploring individuals to tackle the particular challenges of their field and to make explicit and critically reflect on their philosophical presuppositions, metatheories, and methodologies. The metatheoretical definition of ‘personality’ as individual-specificity is elaborated, highlighting that ‘personality’ is conceived and studied differently because different paradigms focus on individual-specificity in different kinds of phenomena. Critical analyses of the field’s strong reliance on assessments methods reveal fallacies, circular explanations in ‘trait’ concepts, and the fact that taxonomic models of most of the phenomena in which individual-specificity is conceived as ‘personality’ still need to be developed. The TPS-Paradigm is therefore aimed at reviving and broadening the portfolio of methodologies that can and should be complementarily applied to comprehensively explore individuals and their ‘personality.’ Such broad paradigms are needed to enable transdisciplinary collaboration and research that embraces the complex reality of individuals and their ‘personality.’

Psychology's crises (e.g., replicability, generalisability) are currently believed to derive from Questionable Research Practices (QRPs), thus scientific misconduct. Just improving the same practices, however, cannot tackle the root causes of psychology's problems—the Questionable Research Fundamentals (QRFs) of many of its theories, concepts, approaches and methods (e.g., psychometrics), which are grounded in their insufficiently elaborated underlying philosophies of science. Key problems of psychological measurement are critically explored from independent perspectives involving various fields of expertise and lines of research that are well established but still hardly known in mainstream psychology. This comprehensive multi-perspectival review presents diverse philosophies of science that are used in quantitative psychology and pinpoints four major areas of development. (1) Psychology must advance its general philosophy of science (esp. ontology, epistemology, methodology) and elaborate coherent paradigms. (2) Quantitative psychologists must elaborate the philosophy-of-science fundamentals of specific theories, approaches and methods that are appropriate for enabling quantitative research and for implementing genuine analogues of measurement in psychology, considering its study phenomena's peculiarities (e.g., higher-order complexity, non-ergodicity). (3) Psychologists must heed the epistemic necessity to logically distinguish between the study phenomena (e.g., participants' beliefs) and the means used for their exploration (e.g., descriptions of beliefs in items) to avoid confusing ontological with epistemological concepts—psychologists' cardinal error. This requires an increased awareness of the complexities of human language (e.g., inbuilt semantics) and of the intricacies that these entail for scientific inquiry. (4) Epistemically justified strategies for generalising findings across unique individuals must be established using case-by-case based (not sample-based) nomothetic approaches, implemented through individual-/person-oriented (not variable-oriented) analyses. This is crucial to avoid the mathematical-statistical errors that are inherent to quantitative psychologists' common sample-to-individual inferences (e.g., ergodic fallacy) as well as to enable causal analyses of possibly underlying structures and processes. Concluding, just minimising scientific misconduct, as currently believed, and exploiting language-based algorithms (NLP, LLMs) without considering the intricacies of human language will only perpetuate psychology's crises. Rethinking psychology as a science and advancing its philosophy-of-science theories as necessary fundamentals to integrate its fragmented empirical database and lines of research requires open, honest and self-critical debates that prioritise scientific integrity over expediency.

Constructs are central to psychology. And yet, relevance and role of constructs for psychological theories, findings and practices are still debated and even questioned. What actually are constructs? Why are they so central to psychology? What is their explanatory value? (How) can we ‘measure’ constructs? And why are there so many misunderstandings and confusions about them? These are obvious questions for which many psychologists are still seeking answers—such as De Boeck and colleagues in their target article (this issue).