Nick Bostrom

In a recent paper in this journal, Ken Olum attempts to refute the doomsday argument by appealing to the self–indication assumption (SIA) that your very existence gives you reason to think that there are many observers. Unlike earlier users of this strategy, Olum tries to counter objections that have been made against (SIA). We argue that his defence of (SIA) is unsuccessful. This does not, however, mean that one has to accept the doomsday argument (or the other counter–intuitive results that flow from related thought–experiments). A developed theory of observation selection effects shows why the doomsday argument is inconclusive, and how one can consistently reject both it and (SIA).

Over the past few years, a new paradigm for thinking about humankind's future has begun to take shape among some leading computer scientists, neuroscientists, nanotechnologists and researchers at the forefront of technological development. The new paradigm rejects a crucial assumption that is implicit in both traditional futurology and practically all of today's political thinking. This is the assumption that the "human condition" is at root a constant. Present-day processes can be fine-tuned; wealth can be increased and redistributed; tools can be developed and refined; culture can change, sometimes drastically; but human nature itself is not up for grabs.

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order.

When water was discovered on Mars, people got very excited. Where there is water, there may be life. Scientists are planning new missions to study the planet up close. NASA’s next Mars rover is scheduled to arrive in 2010. In the decade following, a Mars Sample Return mission might be launched, which would use robotic systems to collect samples of Martian rocks, soils, and atmosphere, and return them to Earth. We could then analyze the sample to see if it contains any traces of life, whether extinct or still active. Such a discovery would be of tremendous scientific significance. What could be more fascinating than discovering life that had evolved entirely independently of life here on Earth? Many people would also find it heartening to learn that we are not entirely alone in this vast cold cosmos.

There are two boxes in front of you and you are asked to choose between taking only box B or taking both box A and box B. Box A contains $ 1,000. Box B will contain either nothing or $ 1,000,000. What B will contain is (or will be) determined by Predictor, who has an excellent track record of predicting your choices. There are two possibilities. Either Predictor has already made his move by predicting your choice and putting a million dollars in B iff he predicted that you will take only B (like in the standard Newcomb problem); or else Predictor has not yet made his move but will wait and observe what box you choose and then put a million dollars in B iff you take only B. In cases like this, Predictor makes his move before the subject roughly half of the time. However, there is a Metapredictor, who has an excellent track record of predicting Predictor’s choices as well as your own. You know all this. Metapredictor informs you of the following truth functional: Either you choose A and B, and Predictor will make his move after you make your choice; or else you choose only B, and Predictor has already made his choice. Now, what do you choose?

This paper discusses the relation between intelligence and motivation in artificial agents, developing and briefly arguing for two theses. The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. In combination, the two theses help us understand the possible range of behavior of superintelligent agents, and they point to some potential dangers in building such an agent.

Artificial intelligence is a possibility that should not be ignored in any serious thinking about the future, and it raises many profound issues for ethics and public policy that philosophers ought to start thinking about. This article outlines the case for thinking that human-level machine intelligence might well appear within the next half century. It then explains four immediate consequences of such a development, and argues that machine intelligence would have a revolutionary impact on a wide range of the social, political, economic, commercial, technological, scientific and environmental issues that humanity will face over the coming decades.

When our measurement instruments sample from only a subspace of the domain that we are seeking to understand, or when they sample with uneven sampling density from the target domain, the resulting data will be affected by a selection effect. If we ignore such selection effects, our conclusions may suffer from selection biases. A classic example of selection bias is the election poll taken by the Literary Digest in 1936. On the basis of a large survey, the Digest predicted that Alf Langdon, the Republican presidential candidate, would win by a large margin. But the actual election resulted in a landslide for the incumbent, Franklin D. Roosevelt. How could such a large sample yield such a wayward prediction? The Digest, it turned out, had harvested the addresses for its survey mainly from telephone books and motor vehicle registries. This introduced a strong selection effect. The poor of the depression era, a group that disproportionally supported Roosevelt, often did not have phones or cars. Observation selection effects are an especially subtle kind of selection effect that is introduced not by limitations in our measurement apparatuses but by the fact that all evidence is preconditioned on the existence of an observer to “have” the evidence and to build the instruments in the first place. Observation selection effects have only quite recently become the subject of systematic study. As well as being of philosophical interest, they are important in many scientific areas, including cosmology, parts of evolution theory, and the foundations of thermodynamics and quantum theory. There are also interesting applications to the search for extraterrestrial life and questions such as whether we might be living in a computer simulation created by an advanced civilization..

This essay explores some of the social, political, economic and technological issues that the world may have to face in the mid-21 st century. A central theme is the need to regulate molecular nanotechnology because of its immense abuse potential. Advanced nanotechnology can be used to build small self-replicating machines that can feed on organic matter - a bit like bacteria but much more versatile, and potentially more destructive than the H-bomb. The necessity to prevent irresponsible groups and individuals from getting access to nanotechnological manufacturing capability is a prime concern in 2050. The essay shows how this quest for containment shapes many aspect of society, most notably via the institution of a global surveillance network. A dialogue format is used for two reasons. First, in order to enable several conflicting views to be discussed. And second, to illustrate how public debates may be conducted at that time.

The Doomsday argument purports to show that the risk of the human species going extinct soon has been systematically underestimated. This argument has something in common with controversial forms of reasoning in other areas, including: game theoretic problems with imperfect recall, the methodology of cosmology, the epistemology of indexical belief, and the debate over so-called fine-tuning arguments for the design hypothesis. The common denominator is a certain premiss: the Self-Sampling Assumption. We present two strands of argument in favor of this assumption. Through a series of thought experiments we then investigate some bizarre prima facie consequences – backward causation, psychic powers, and an apparent conflict with the Principal Principle.

The future of humanity is often viewed as a topic for idle speculation. Yet our beliefs and assumptions on this subject matter shape decisions in both our personal lives and public policy – decisions that have very real and sometimes unfortunate consequences. It is therefore practically important to try to develop a realistic mode of futuristic thought about big picture questions for humanity. This paper sketches an overview of some recent attempts in this direction, and it offers a brief discussion of four families of scenarios for humanity’s future: extinction, recurrent collapse, plateau, and posthumanity.

Anthony Brueckner, in a recent article, proffers ‘a new way of thinking about Bostrom's Simulation Argument’. His comments, however, misconstrue the argument; and some words of explanation are in order.The Simulation Argument purports to show, given some plausible assumptions, that at least one of three propositions is true. Roughly stated, these propositions are: almost all civilizations at our current level of development go extinct before reaching technological maturity; there is a strong convergence among technologically mature civilizations such that almost all of them lose interest in creating ancestor-simulations; almost all people with our sorts of experiences live in computer simulations. I also argue that conditional on you should assign a very high credence to the proposition that you live in a computer simulation. However, pace Brueckner, I do not argue that we should believe that we are in simulation. 1 In fact, I believe that we are probably not simulated. The Simulation Argument purports to show only that, as well as, at least one of – is true; but it does not tell us which one.Brueckner also writes: " It is worth noting that one reason why Bostrom thinks that the number of Sims [computer-generated minds with experiences similar to those typical of normal, embodied humans living in a Sim-free early 21 st century world] will vastly outstrip the number of humans is that Sims ‘will run their … "

More precisely, transhumanists advocate increased funding for research to radically extend healthy lifespan and favor the development of medical and technological means to improve memory, concentration, and other human capacities. Transhumanists propose that everybody should have the option to use such means to enhance various dimensions of their cognitive, emotional, and physical well-being. Not only is this a natural extension of the traditional aims of medicine and technology, but it is also a great humanitarian opportunity to genuinely improve the human condition.

Cognitive enhancement may be defined as the amplification or extension of core capacities of the mind through improvement or augmentation of internal or external information processing systems. Cognition refers to the processes an organism uses to organize information. These include acquiring information (perception), selecting (attention), representing (understanding) and retaining (memory) information, and using it to guide behavior (reasoning and coordination of motor outputs). Interventions to improve cognitive function may be directed at any of these core faculties.

Evolutionary development is sometimes thought of as exhibiting an inexorable trend towards higher, more complex, and normatively worthwhile forms of life. This paper explores some dystopian scenarios where freewheeling evolutionary developments, while continuing to produce complex and intelligent forms of organization, lead to the gradual elimination of all forms of being that we care about. We then consider how such catastrophic outcomes could be avoided and argue that under certain conditions the only possible remedy would be a globally coordinated policy to control human evolution by modifying the fitness function of future intelligent life forms.