Nick Bostrom

Technological revolutions are among the most important things that happen to humanity. Ethical assessment in the incipient stages of a potential technological revolution faces several difficulties, including the unpredictability of their long‐term impacts, the problematic role of human agency in bringing them about, and the fact that technological revolutions rewrite not only the material conditions of our existence but also reshape culture and even – perhaps – human nature. This essay explores some of these difficulties and the challenges they pose for a rational assessment of the ethical and policy issues associated with anticipated technological revolutions.

Suppose that we develop a medically safe and affordable means of enhancing human intelligence. For concreteness, we shall assume that the technology is genetic engineering (either somatic or germ line), although the argument we will present does not depend on the technological implementation. For simplicity, we shall speak of enhancing “intelligence” or “cognitive capacity,” but we do not presuppose that intelligence is best conceived of as a unitary attribute. Our considerations could be applied to specific cognitive abilities such as verbal fluency, memory, abstract reasoning, social intelligence, spatial cognition, numerical ability, or musical talent. It will emerge that the form of argument that we use can be applied much more generally to help assess other kinds of enhancement technologies as well as other kinds of reform. However, to give a detailed illustration of how the argument form works, we will focus on the prospect of cognitive enhancement.

The Sleeping Beauty problem is test stone for theories about self-locating belief, i.e. theories about how we should reasons when data or theories contain indexical information. Opinion on this problem is split between two camps, those who defend the "1/2 view" and those who advocate the "1/3 view". I argue that both these positions are mistaken. Instead, I propose a new "hybrid" model, which avoids the faults of the standard views while retaining their attractive properties. This model _appears_ to violate Bayesian conditionalization, but I argue that this is not the case. By paying close attention to the details of conditionalization in contexts where indexical information is relevant, we discover that the hybrid model is in fact consistent with Bayesian kinematics. If the proposed model is correct, there are important lessons for the study of self-location, observation selection theory, and anthropic reasoning.

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.

The purpose of this paper, boldly stated, is to propose a new type of philosophy, a philosophy whose aim is prediction. The pace of technological progress is increasing very rapidly: it looks as if we are witnessing an exponential growth, the growth-rate being proportional to the size already obtained, with scientific knowledge doubling every 10 to 20 years since the second world war, and with computer processor speed doubling every 18 months or so. It is argued that this technological development makes urgent many empirical questions which a philosopher could be well-suited to help answering. I try to cover a broad range of interesting problems and approaches, which means that I won't go at all deeply into any of them; I only try to say enough to show what some of the problems are, how one can begin to work with them, and why philosophy is relevant. My hope is that this will whet your appetite to deal with these questions, or at least increase general awareness that they worthy tasks for first-class intellects, including ones which might belong to philosophers.

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence. This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain. Yet the writing is so lucid that it somehow makes it all seem easy. After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life, we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time.

In some dark alley... Mugger: Hey, give me your wallet. Pascal: Why on Earth would I want to do that? Mugger: Otherwise I’ll shoot you. Pascal: But you don’t have a gun. Mugger: Oops! I knew I had forgotten something. Pascal: No wallet for you then. Have a nice evening. Mugger: Wait! Pascal: Sigh. Mugger: I’ve got a business proposition for you.... How about you give me your wallet now? In return, I promise to come to your house tomorrow and give you double the value of what’s in the wallet. Not bad, eh? A 200% return on investment in 24 hours.

Positions on the ethics of human enhancement technologies can be (crudely) characterized as ranging from transhumanism to bioconservatism. Transhumanists believe that human enhancement technologies should be made widely available, that individuals should have broad discretion over which of these technologies to apply to themselves, and that parents should normally have the right to choose enhancements for their children-to-be. Bioconservatives (whose ranks include such diverse writers as Leon Kass, Francis Fukuyama, George Annas, Wesley Smith, Jeremy Rifkin, and Bill McKibben) are generally opposed to the use of technology to modify human nature. A central idea in bioconservativism is that human enhancement technologies will undermine our human dignity. To forestall a slide down the slippery slope towards an ultimately debased ‘posthuman’ state, bioconservatives often argue for broad bans on otherwise promising human enhancements. This paper distinguishes two common fears about the posthuman and argues for the importance of a concept of dignity that is inclusive enough to also apply to many possible posthuman beings. Recognizing the possibility of posthuman dignity undercuts an important objection against human enhancement and removes a distortive double standard from our field of moral vision.

In a recent paper in this journal, Ken Olum attempts to refute the doomsday argument by appealing to the self-indication assumption (SIA) that your very existence gives you reason to think that there are many observers. Unlike earlier users of this strategy, Olum tries to counter objections that have been made against (SIA). We argue that his defence of (SIA) is unsuccessful. This does not, however, mean that one has to accept the doomsday argument (or the other counter-intuitive results that flow from related thought-experiments). A developed theory of observation selection effects shows why the doomsday argument is inconclusive, and how one can consistently reject both it and (SIA).

If a brain is duplicated so that there are two brains in identical states, are there then two numerically distinct phenomenal experiences or only one? There are two, I argue, and given computationalism, this has implications for what it is to implement a computation. I then consider what happens when a computation is implemented in a system that either uses unreliable components or possesses varying degrees of parallelism. I show that in some of these cases there can be, in a deep and intriguing sense, a fractional (non-integer) number of qualitatively identical phenomenal experiences. This, in turn, has implications for what lessons one should draw from neural replacement scenarios such as Chalmers.

Gdzieś w ciemnej uliczce... Bandyta: Ej ty, dawaj portfel! Pascal: A niby dlaczego miałbym to zrobić? Bandyta: Bo w przeciwnym razie cię zastrzelę. Pascal: Ale przecież nie masz broni. Bandyta: A niech to! Wiedziałem, że zapomniałem o czymś. Pascal: No to zapomnij też o moim portfelu. Miłego wieczoru. Bandyta: Stój! Pascal: Co znowu? Bandyta: Jest interes do zrobienia... Co ty na to, żebyś jednak oddał mi portfel? W zamian obiecuję przyjść do ciebie jutro i dać ci dwukrotność kwoty, którą w nim masz. Nieźle, co? 200 procent zwrotu z inwestycji w 24 godziny.

One might think that since life here on Earth has survived for nearly 4 Gyr (Gigayears), such catastrophic events must be extremely rare. Unfortunately, such an argument is flawed, giving us a false sense of security. It fails to take into account the observation selection effect [6, 7] that precludes any observer from observing anything other than that their own species has survived up to the point where they make the observation. Even if the frequency of cosmic catastrophes were very high, we should still expect to find ourselves on a planet that had not yet been destroyed. The fact that we are still alive does not even seem to rule out the hypothesis that the average cosmic neighborhood is typically sterilized by vacuum decay, say, every 10000 years, and that our own planet has just been extremely lucky up until now.

Perhaps the two most important world events during my thirty‐six years are the ending of the Cold War and the beginning of the Internet. Of those two, I think the latter is the more significant. The Internet has impacted my thinking in several ways. It has put me in touch with people I would not otherwise have met and whose ideas I would never have encountered. It has served as a platform for disseminating my work, helping me get faster and more extensive feedback. And it is of course a powerful research tools, giving instantaneous access to an immense and up‐to‐date store of knowledge. Rarely do I need to send a research assistant to the library. It saves time and makes it possible to take into account a wider range of research.

John Leslie presents a thought experiment to show that chances are sometimes observer-relative in a paradoxical way. The pivotal assumption in his argument – a version of the weak anthropic principle – is the same as the one used to get the disturbing Doomsday argument off the ground. I show that Leslie's thought experiment trades on the sense/reference ambiguity and is fallacious. I then describe a related case where chances are observer-relative in an interesting way. But not in a paradoxical way. The result can be generalized: At least for a very wide range of cases, the weak anthropic principle does not giverise to paradoxical observer-relative chances. This finding could be taken to give new indirect support to the doomsday argument.

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies.

Human enhancement has emerged in recent years as a blossoming topic in applied ethics. With continuing advances in science and technology, people are beginning to realize that some of the basic parameters of the human condition might be changed in the future. One important way in which the human condition could be changed is through the enhancement of basic human capacities. If this becomes feasible within the lifespan of many people alive today, then it is important now to consider the normative questions raised by such prospects. The answers to these questions might not only help us be better prepared when technology catches up with imagination, but they may be relevant to many decisions we make today, such as decisions about how much funding to give to various kinds of research. Enhancement is typically contraposed to therapy. In broad terms, therapy aims to fix something that has gone wrong, by curing specific diseases or injuries, while enhancement interventions aim to improve the state of an organism beyond its normal healthy state. However, the distinction between therapy and enhancement is problematic, for several reasons. First, we may note that the therapy-enhancement dichotomy does not map onto any corresponding dichotomy between standard-contemporary-medicine and medicineas-it-could-be-practised-in-the-future. Standard contemporary medicine includes many practices that do not aim to cure diseases or injuries. It includes, for example, preventive medicine, palliative care, obstetrics, sports medicine, plastic surgery, contraceptive devices, fertility treatments, cosmetic dental procedures, and much else. At the same time, many enhancement interventions occur outside of the medical framework. Office workers enhance their performance by drinking coffee. Make-up and grooming are used to enhance appearance. Exercise, meditation, fish oil, and St John’s Wort are used to enhance mood. Second, it is unclear how to classify interventions that reduce the probability of disease and death..

Lang, lang geleden werd de planeet getiranniseerd door een gigantische draak. De draak was groter dan de grootste kerk en was bedekt met dikke, zwarte schubben. Zijn rode ogen gloeiden van haat en uit zijn verschrikkelijke bek stroomde onophoudelijk stinkend, geelgroen slijm. Hij eiste van de mens een ijzingwekkend eerbetoon: om zijn enorme honger te stillen, moesten er iedere avond bij het vallen van het duister tienduizend mannen en vrouwen aan de voet van de berg, waar de draak-tiran woonde, afgeleverd worden. Soms slokte de draak deze arme zielen gelijk op; soms sloot hij ze echter op in de berg waar zij maanden of jaren wegkwijnden totdat zij uiteindelijk opgegeten werden.