Nick Bostrom

Suppose that we develop a medically safe and affordable means of enhancing human intelligence. For concreteness, we shall assume that the technology is genetic engineering (either somatic or germ line), although the argument we will present does not depend on the technological implementation. For simplicity, we shall speak of enhancing “intelligence” or “cognitive capacity,” but we do not presuppose that intelligence is best conceived of as a unitary attribute. Our considerations could be applied to specific cognitive abilities such as verbal fluency, memory, abstract reasoning, social intelligence, spatial cognition, numerical ability, or musical talent. It will emerge that the form of argument that we use can be applied much more generally to help assess other kinds of enhancement technologies as well as other kinds of reform. However, to give a detailed illustration of how the argument form works, we will focus on the prospect of cognitive enhancement.

Aggregative consequentialism and several other popular moral theories are threatened with paralysis: when coupled with some plausible assumptions, they seem to imply that it is always ethically indifferent what you do. Modern cosmology teaches that the world might well contain an infinite number of happy and sad people and other candidate value‐bearing locations. Aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. You can affect only a finite amount of good or bad. In standard cardinal arithmetic, an infinite quantity is unchanged by the addition or subtraction of any finite quantity. So it appears you cannot change the value of the world. Modifications of aggregationism aimed at resolving the paralysis are only partially effective and cause severe side effects, including problems of “fanaticism”, “distortion”, and erosion of the intuitions that originally motivated the theory. Is the infinitarian challenge fatal?

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence. This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain. Yet the writing is so lucid that it somehow makes it all seem easy. After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life, we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time.

In some dark alley... Mugger: Hey, give me your wallet. Pascal: Why on Earth would I want to do that? Mugger: Otherwise I’ll shoot you. Pascal: But you don’t have a gun. Mugger: Oops! I knew I had forgotten something. Pascal: No wallet for you then. Have a nice evening. Mugger: Wait! Pascal: Sigh. Mugger: I’ve got a business proposition for you.... How about you give me your wallet now? In return, I promise to come to your house tomorrow and give you double the value of what’s in the wallet. Not bad, eh? A 200% return on investment in 24 hours.

Positions on the ethics of human enhancement technologies can be (crudely) characterized as ranging from transhumanism to bioconservatism. Transhumanists believe that human enhancement technologies should be made widely available, that individuals should have broad discretion over which of these technologies to apply to themselves, and that parents should normally have the right to choose enhancements for their children-to-be. Bioconservatives (whose ranks include such diverse writers as Leon Kass, Francis Fukuyama, George Annas, Wesley Smith, Jeremy Rifkin, and Bill McKibben) are generally opposed to the use of technology to modify human nature. A central idea in bioconservativism is that human enhancement technologies will undermine our human dignity. To forestall a slide down the slippery slope towards an ultimately debased ‘posthuman’ state, bioconservatives often argue for broad bans on otherwise promising human enhancements. This paper distinguishes two common fears about the posthuman and argues for the importance of a concept of dignity that is inclusive enough to also apply to many possible posthuman beings. Recognizing the possibility of posthuman dignity undercuts an important objection against human enhancement and removes a distortive double standard from our field of moral vision.

In a recent paper in this journal, Ken Olum attempts to refute the doomsday argument by appealing to the self-indication assumption (SIA) that your very existence gives you reason to think that there are many observers. Unlike earlier users of this strategy, Olum tries to counter objections that have been made against (SIA). We argue that his defence of (SIA) is unsuccessful. This does not, however, mean that one has to accept the doomsday argument (or the other counter-intuitive results that flow from related thought-experiments). A developed theory of observation selection effects shows why the doomsday argument is inconclusive, and how one can consistently reject both it and (SIA).

If a brain is duplicated so that there are two brains in identical states, are there then two numerically distinct phenomenal experiences or only one? There are two, I argue, and given computationalism, this has implications for what it is to implement a computation. I then consider what happens when a computation is implemented in a system that either uses unreliable components or possesses varying degrees of parallelism. I show that in some of these cases there can be, in a deep and intriguing sense, a fractional (non-integer) number of qualitatively identical phenomenal experiences. This, in turn, has implications for what lessons one should draw from neural replacement scenarios such as Chalmers.

Gdzieś w ciemnej uliczce... Bandyta: Ej ty, dawaj portfel! Pascal: A niby dlaczego miałbym to zrobić? Bandyta: Bo w przeciwnym razie cię zastrzelę. Pascal: Ale przecież nie masz broni. Bandyta: A niech to! Wiedziałem, że zapomniałem o czymś. Pascal: No to zapomnij też o moim portfelu. Miłego wieczoru. Bandyta: Stój! Pascal: Co znowu? Bandyta: Jest interes do zrobienia... Co ty na to, żebyś jednak oddał mi portfel? W zamian obiecuję przyjść do ciebie jutro i dać ci dwukrotność kwoty, którą w nim masz. Nieźle, co? 200 procent zwrotu z inwestycji w 24 godziny.

One might think that since life here on Earth has survived for nearly 4 Gyr (Gigayears), such catastrophic events must be extremely rare. Unfortunately, such an argument is flawed, giving us a false sense of security. It fails to take into account the observation selection effect [6, 7] that precludes any observer from observing anything other than that their own species has survived up to the point where they make the observation. Even if the frequency of cosmic catastrophes were very high, we should still expect to find ourselves on a planet that had not yet been destroyed. The fact that we are still alive does not even seem to rule out the hypothesis that the average cosmic neighborhood is typically sterilized by vacuum decay, say, every 10000 years, and that our own planet has just been extremely lucky up until now.

Perhaps the two most important world events during my thirty‐six years are the ending of the Cold War and the beginning of the Internet. Of those two, I think the latter is the more significant. The Internet has impacted my thinking in several ways. It has put me in touch with people I would not otherwise have met and whose ideas I would never have encountered. It has served as a platform for disseminating my work, helping me get faster and more extensive feedback. And it is of course a powerful research tools, giving instantaneous access to an immense and up‐to‐date store of knowledge. Rarely do I need to send a research assistant to the library. It saves time and makes it possible to take into account a wider range of research.

John Leslie presents a thought experiment to show that chances are sometimes observer-relative in a paradoxical way. The pivotal assumption in his argument – a version of the weak anthropic principle – is the same as the one used to get the disturbing Doomsday argument off the ground. I show that Leslie's thought experiment trades on the sense/reference ambiguity and is fallacious. I then describe a related case where chances are observer-relative in an interesting way. But not in a paradoxical way. The result can be generalized: At least for a very wide range of cases, the weak anthropic principle does not giverise to paradoxical observer-relative chances. This finding could be taken to give new indirect support to the doomsday argument.

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies.

The Sleeping Beauty problem is test stone for theories about self-locating belief, i.e. theories about how we should reasons when data or theories contain indexical information. Opinion on this problem is split between two camps, those who defend the "1/2 view" and those who advocate the "1/3 view". I argue that both these positions are mistaken. Instead, I propose a new "hybrid" model, which avoids the faults of the standard views while retaining their attractive properties. This model _appears_ to violate Bayesian conditionalization, but I argue that this is not the case. By paying close attention to the details of conditionalization in contexts where indexical information is relevant, we discover that the hybrid model is in fact consistent with Bayesian kinematics. If the proposed model is correct, there are important lessons for the study of self-location, observation selection theory, and anthropic reasoning.

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.

The purpose of this paper, boldly stated, is to propose a new type of philosophy, a philosophy whose aim is prediction. The pace of technological progress is increasing very rapidly: it looks as if we are witnessing an exponential growth, the growth-rate being proportional to the size already obtained, with scientific knowledge doubling every 10 to 20 years since the second world war, and with computer processor speed doubling every 18 months or so. It is argued that this technological development makes urgent many empirical questions which a philosopher could be well-suited to help answering. I try to cover a broad range of interesting problems and approaches, which means that I won't go at all deeply into any of them; I only try to say enough to show what some of the problems are, how one can begin to work with them, and why philosophy is relevant. My hope is that this will whet your appetite to deal with these questions, or at least increase general awareness that they worthy tasks for first-class intellects, including ones which might belong to philosophers.

Blackballing the reaper is an old ambition, and considerable progress has been made. For the past 150 years, bestperformance life-expectancy (i.e. life-expectancy in the country where it is highest) has increased at a very steady rate of 3 months per year.[1] Life-expectancy for the ancient Romans was circa 23 years; today the average lifeexpectancy in the world is 64 years.[2] Will this trend continue? What are the consequences if it does? And what ethical and political challenges does the prospect of lifeextension create for us today? This article comments on some views on the ethics, science, and politics of lifeextension from a recent edited volume, The Fountain of Youth.