Nick Bostrom

The Doomsday argument purports to show that the risk of the human species going extinct soon has been systematically underestimated. This argument has something in common with controversial forms of reasoning in other areas, including: game theoretic problems with imperfect recall, the methodology of cosmology, the epistemology of indexical belief, and the debate over so-called fine-tuning arguments for the design hypothesis. The common denominator is a certain premiss: the Self-Sampling Assumption. We present two strands of argument in favor of this assumption. Through a series of thought experiments we then investigate some bizarre prima facie consequences – backward causation, psychic powers, and an apparent conflict with the Principal Principle.

The future of humanity is often viewed as a topic for idle speculation. Yet our beliefs and assumptions on this subject matter shape decisions in both our personal lives and public policy – decisions that have very real and sometimes unfortunate consequences. It is therefore practically important to try to develop a realistic mode of futuristic thought about big picture questions for humanity. This paper sketches an overview of some recent attempts in this direction, and it offers a brief discussion of four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity.

Anthony Brueckner, in a recent article, proffers ‘a new way of thinking about Bostrom's Simulation Argument’. His comments, however, misconstrue the argument; and some words of explanation are in order.The Simulation Argument purports to show, given some plausible assumptions, that at least one of three propositions is true. Roughly stated, these propositions are: almost all civilizations at our current level of development go extinct before reaching technological maturity; there is a strong convergence among technologically mature civilizations such that almost all of them lose interest in creating ancestor-simulations; almost all people with our sorts of experiences live in computer simulations. I also argue that conditional on you should assign a very high credence to the proposition that you live in a computer simulation. However, pace Brueckner, I do not argue that we should believe that we are in simulation. 1 In fact, I believe that we are probably not simulated. The Simulation Argument purports to show only that, as well as, at least one of – is true; but it does not tell us which one.Brueckner also writes: " It is worth noting that one reason why Bostrom thinks that the number of Sims [computer-generated minds with experiences similar to those typical of normal, embodied humans living in a Sim-free early 21 st century world] will vastly outstrip the number of humans is that Sims ‘will run their … "

Gdzieś w ciemnej uliczce... Bandyta: Ej ty, dawaj portfel! Pascal: A niby dlaczego miałbym to zrobić? Bandyta: Bo w przeciwnym razie cię zastrzelę. Pascal: Ale przecież nie masz broni. Bandyta: A niech to! Wiedziałem, że zapomniałem o czymś. Pascal: No to zapomnij też o moim portfelu. Miłego wieczoru. Bandyta: Stój! Pascal: Co znowu? Bandyta: Jest interes do zrobienia... Co ty na to, żebyś jednak oddał mi portfel? W zamian obiecuję przyjść do ciebie jutro i dać ci dwukrotność kwoty, którą w nim masz. Nieźle, co? 200 procent zwrotu z inwestycji w 24 godziny.

One might think that since life here on Earth has survived for nearly 4 Gyr (Gigayears), such catastrophic events must be extremely rare. Unfortunately, such an argument is flawed, giving us a false sense of security. It fails to take into account the observation selection effect [6, 7] that precludes any observer from observing anything other than that their own species has survived up to the point where they make the observation. Even if the frequency of cosmic catastrophes were very high, we should still expect to find ourselves on a planet that had not yet been destroyed. The fact that we are still alive does not even seem to rule out the hypothesis that the average cosmic neighborhood is typically sterilized by vacuum decay, say, every 10000 years, and that our own planet has just been extremely lucky up until now.

Perhaps the two most important world events during my thirty‐six years are the ending of the Cold War and the beginning of the Internet. Of those two, I think the latter is the more significant. The Internet has impacted my thinking in several ways. It has put me in touch with people I would not otherwise have met and whose ideas I would never have encountered. It has served as a platform for disseminating my work, helping me get faster and more extensive feedback. And it is of course a powerful research tools, giving instantaneous access to an immense and up‐to‐date store of knowledge. Rarely do I need to send a research assistant to the library. It saves time and makes it possible to take into account a wider range of research.

John Leslie presents a thought experiment to show that chances are sometimes observer-relative in a paradoxical way. The pivotal assumption in his argument – a version of the weak anthropic principle – is the same as the one used to get the disturbing Doomsday argument off the ground. I show that Leslie's thought experiment trades on the sense/reference ambiguity and is fallacious. I then describe a related case where chances are observer-relative in an interesting way. But not in a paradoxical way. The result can be generalized: At least for a very wide range of cases, the weak anthropic principle does not giverise to paradoxical observer-relative chances. This finding could be taken to give new indirect support to the doomsday argument.

Because of accelerating technological progress, humankind may be rapidly approaching a critical phase in its career. In addition to well-known threats such as nuclear holocaust, the propects of radically transforming technologies like nanotech systems and machine intelligence present us with unprecedented opportunities and risks. Our future, and whether we will have a future at all, may well be determined by how we deal with these challenges. In the case of radically transforming technologies, a better understanding of the transition dynamics from a human to a "posthuman" society is needed. Of particular importance is to know where the pitfalls are: the ways in which things could go terminally wrong. While we have had long exposure to various personal, local, and endurable global hazards, this paper analyzes a recently emerging category: that of existential risks. These are threats that could case our extinction or destroy the potential of Earth - originating intelligent life. Some of these threats are relatively well known while others, including some of the gravest, have gone almost unrecognized. Existential risks have a cluster of features that make ordinary risk management ineffective. A final section of this paper discusses several ethical and policy implications. A clearer understanding of the threat picture will enable us to formulate better strategies.

The Sleeping Beauty problem is test stone for theories about self-locating belief, i.e. theories about how we should reasons when data or theories contain indexical information. Opinion on this problem is split between two camps, those who defend the "1/2 view" and those who advocate the "1/3 view". I argue that both these positions are mistaken. Instead, I propose a new "hybrid" model, which avoids the faults of the standard views while retaining their attractive properties. This model _appears_ to violate Bayesian conditionalization, but I argue that this is not the case. By paying close attention to the details of conditionalization in contexts where indexical information is relevant, we discover that the hybrid model is in fact consistent with Bayesian kinematics. If the proposed model is correct, there are important lessons for the study of self-location, observation selection theory, and anthropic reasoning.

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence.

The purpose of this paper, boldly stated, is to propose a new type of philosophy, a philosophy whose aim is prediction. The pace of technological progress is increasing very rapidly: it looks as if we are witnessing an exponential growth, the growth-rate being proportional to the size already obtained, with scientific knowledge doubling every 10 to 20 years since the second world war, and with computer processor speed doubling every 18 months or so. It is argued that this technological development makes urgent many empirical questions which a philosopher could be well-suited to help answering. I try to cover a broad range of interesting problems and approaches, which means that I won't go at all deeply into any of them; I only try to say enough to show what some of the problems are, how one can begin to work with them, and why philosophy is relevant. My hope is that this will whet your appetite to deal with these questions, or at least increase general awareness that they worthy tasks for first-class intellects, including ones which might belong to philosophers.

The human brain has some capabilities that the brains of other animals lack. It is to these distinctive capabilities that our species owes its dominant position. Other animals have stronger muscles or sharper claws, but we have cleverer brains. If machine brains one day come to surpass human brains in general intelligence, then this new superintelligence could become very powerful. As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species then would come to depend on the actions of the machine superintelligence. But we have one advantage: we get to make the first move. Will it be possible to construct a seed AI or otherwise to engineer initial conditions so as to make an intelligence explosion survivable? How could one achieve a controlled detonation? To get closer to an answer to this question, we must make our way through a fascinating landscape of topics and considerations. Read the book and learn about oracles, genies, singletons; about boxing methods, tripwires, and mind crime; about humanity's cosmic endowment and differential technological development; indirect normativity, instrumental convergence, whole brain emulation and technology couplings; Malthusian economics and dystopian evolution; artificial intelligence, and biological cognitive enhancement, and collective intelligence. This profoundly ambitious and original book picks its way carefully through a vast tract of forbiddingly difficult intellectual terrain. Yet the writing is so lucid that it somehow makes it all seem easy. After an utterly engrossing journey that takes us to the frontiers of thinking about the human condition and the future of intelligent life, we find in Nick Bostrom's work nothing less than a reconceptualization of the essential task of our time.

In some dark alley... Mugger: Hey, give me your wallet. Pascal: Why on Earth would I want to do that? Mugger: Otherwise I’ll shoot you. Pascal: But you don’t have a gun. Mugger: Oops! I knew I had forgotten something. Pascal: No wallet for you then. Have a nice evening. Mugger: Wait! Pascal: Sigh. Mugger: I’ve got a business proposition for you.... How about you give me your wallet now? In return, I promise to come to your house tomorrow and give you double the value of what’s in the wallet. Not bad, eh? A 200% return on investment in 24 hours.

Positions on the ethics of human enhancement technologies can be (crudely) characterized as ranging from transhumanism to bioconservatism. Transhumanists believe that human enhancement technologies should be made widely available, that individuals should have broad discretion over which of these technologies to apply to themselves, and that parents should normally have the right to choose enhancements for their children-to-be. Bioconservatives (whose ranks include such diverse writers as Leon Kass, Francis Fukuyama, George Annas, Wesley Smith, Jeremy Rifkin, and Bill McKibben) are generally opposed to the use of technology to modify human nature. A central idea in bioconservativism is that human enhancement technologies will undermine our human dignity. To forestall a slide down the slippery slope towards an ultimately debased ‘posthuman’ state, bioconservatives often argue for broad bans on otherwise promising human enhancements. This paper distinguishes two common fears about the posthuman and argues for the importance of a concept of dignity that is inclusive enough to also apply to many possible posthuman beings. Recognizing the possibility of posthuman dignity undercuts an important objection against human enhancement and removes a distortive double standard from our field of moral vision.

In a recent paper in this journal, Ken Olum attempts to refute the doomsday argument by appealing to the self-indication assumption (SIA) that your very existence gives you reason to think that there are many observers. Unlike earlier users of this strategy, Olum tries to counter objections that have been made against (SIA). We argue that his defence of (SIA) is unsuccessful. This does not, however, mean that one has to accept the doomsday argument (or the other counter-intuitive results that flow from related thought-experiments). A developed theory of observation selection effects shows why the doomsday argument is inconclusive, and how one can consistently reject both it and (SIA).

If a brain is duplicated so that there are two brains in identical states, are there then two numerically distinct phenomenal experiences or only one? There are two, I argue, and given computationalism, this has implications for what it is to implement a computation. I then consider what happens when a computation is implemented in a system that either uses unreliable components or possesses varying degrees of parallelism. I show that in some of these cases there can be, in a deep and intriguing sense, a fractional (non-integer) number of qualitatively identical phenomenal experiences. This, in turn, has implications for what lessons one should draw from neural replacement scenarios such as Chalmers.

The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems. Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency. To the extent that ethics is a cognitive pursuit, a superintelligence could also easily surpass humans in the quality of its moral thinking. However, it would be up to the designers of the superintelligence to specify its original motivations. Since the superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop, it is crucial that it be provided with human-friendly motivations. This paper surveys some of the unique ethical issues in creating superintelligence, and discusses what motivations we ought to give a superintelligence, and introduces some cost-benefit considerations relating to whether the development of superintelligent machines ought to be accelerated or retarded.

_This paper outlines the case for believing that we will have superhuman artificial intelligence_ _within the first third of the next century. It looks at different estimates of the processing power of_ _the human brain; how long it will take until computer hardware achieve a similar performance;_ _ways of creating the software through bottom-up approaches like the one used by biological_ _brains; how difficult it will be for neuroscience figure out enough about how brains work to_ _make this approach work; and how fast we can expect superintelligence to be developed once_ _there is human-level artificial intelligence._.

We differentiate morally between actual and potential beings: the latter do not exist now and will never exist unless we bring them into existence. The interests of existing persons should guide the creation of new beings. We ought not to create new beings that are expected to harm the interests of existing persons. If a potential being becomes actual, it becomes a member of the moral community and its interests should be taken into account. A being can be actual even if it does not currently exist: there is a moral reason not to set up a mechanism today that will harm a being that comes into existence tomorrow.