Most of us are familiar with reports of lab animals that, when presented with a lever that provides opioids or cocaine, will press the lever over and over at the expense of food and water (Deneau, Yanagita, and Seevers 1969; Weeks 1962; Deroche-Gamonet, Belin, and Piazza 2004). This paradigm has not only been used by researchers to study drug effects but also entered the culture as an anti-drug parable. In the most famous of these types of studies (Olds and Milner 1954) the lever wasn't deliveri…
Read moreMost of us are familiar with reports of lab animals that, when presented with a lever that provides opioids or cocaine, will press the lever over and over at the expense of food and water (Deneau, Yanagita, and Seevers 1969; Weeks 1962; Deroche-Gamonet, Belin, and Piazza 2004). This paradigm has not only been used by researchers to study drug effects but also entered the culture as an anti-drug parable. In the most famous of these types of studies (Olds and Milner 1954) the lever wasn't delivering drugs, but direct electrical stimulation to the brain’s reward circuits. This research inspired a genre of 'wireheading' (Niven 1969); speculative fiction in which humans build technologies that allow them to directly manipulate reward systems. The general theme of these stories is that self-stimulation leads to loss of agency and social decay. The framework of social distaste does not do justice to the biological, technical, and philosophical implications of an intelligent agent finding ways to subvert its own control system. When we consider the goals and relationships of genomes, nervous systems, social systems, and artificial intelligence, we find that the subversive behavior of hacking your own reward system is not an aberration. Rather, reward hacking it is the default behavior of both evolved and engineered intelligent agents and is overcome only by constant innovation and effort on the part of control systems. The more capabilities available to an intelligent agent, the more likely it is to find ways to hack its own reward system. At the extreme, an organism smart enough to gain complete control over its own reward system, i.e. build itself a golden lever to press, should be expected to completely cease functioning. The seemingly inevitable non-functional endpoint to increasing the capabilities of an intelligent agent is the self-control paradox. By asking whose goals are being subverted when reward systems are hacked, we can better understand our relationship to our genome, to each other, and to artificial intelligence.
