Alexey Turchin

There is a paradox — on an everyday level people consider death terrible and the worst thing there is, but philosophers (from the arguments of Epicurus and Lucretius through to Heidegger's being-toward-death) have expended great effort proving the opposite thesis — that death is not to be feared. At the same time, philosophical justifications of other values correlate with people's basic needs: the badness of murder, slavery, suffering. In this article we seek to establish the fundamental badness of death. However, directly proving the badness of death is difficult, since one cannot unambiguously derive prescriptions from facts. Deriving the badness of death by means of utilitarian principles is also difficult, since the elimination of the subject destroys suffering. Instead, we undertake an investigation into how values are formed in general and apply what we find to postulate a new independent value: death is absolutely bad. As death has its proponents who describe it as a useful agent, we can describe it in subjective terms: death is absolute Evil.

Does the potential creation of a time machine present an existential risk to our current timeline? Time travel is theoretically possible under general relativity, and there is steady progress (similar to Moore's Law) in developing ideas about how to create time machines with decreasing effort. While time travel may seem like a remote possibility due to its dependence on space travel to black holes, there is a concept of a quantum time machine (suggested by Deutsch in 1991 and further developed by Lloyd in 2011) that would require much less energy to send a few bits back in time and could still cause paradoxes. An artificial superintelligence (ASI) might soon create such a device, especially in scenarios where multiple ASIs compete for control with each other. However, the catastrophic implications of time machines are largely unexplored in scientific research. Two common, yet contradictory, perspectives exist: either time machines are impossible, rendering risk moot, or Novikov’s self-consistency principle eliminates any chance of timeline alteration. I argue that Novikov’s principle itself can precipitate various catastrophes, for instance, by annihilating civilizations capable of building time machines. This paper classifies the catastrophic risks associated with potential future time machines. The aim is not to declare time machines inherently risky, but to emphasize the necessity of risk assessment given ongoing theoretical advancements in their construction.

We need to understand personal identity to develop radical life extension technologies: mind uploading, cryonics, digital immortality, and quantum (big world) immortality. A tentative solution is needed now, due to the opportunity cost of delaying indirect digital immortality and cryonics. However, solving the problem of personal identity is not easy. Human personal identity is a complex thing, not similar to other types of identity, such as that of Theseus ship. First of all, human identity consists of two intertwined types of identity: informational identity, which predicts sameness, and identity of consciousness, which predicts what I will experience in the next moment of time. The identity of consciousness is called "hard problem of identity", and it is because it supervenes on the nature of consciousness. The second feature of human personal identity is that it is observed and measured internally, by identity subject himself. The third is that it evolves in time. The main dichotomy in the views on personal identity and copies can be presented as: either my copy=original or a soul exists. In other words, some non-informational identity carrier (NIIC) may exist that distinguishes the original from its exact copy. Typically, it is often claimed that NIIC is either continuity of consciousness, or soul, or perspective, or the sameness of atoms, or position in space. We create an exhaustive map of identity theories. We demonstrate here that the idea of continuity of consciousness is very similar to the idea of soul, but also has several problems: - Is it the continuity of qualia, of causal processes, of computations or of underlying substrate? - We also show that continuity doesn't prevent the emergence of two future copies if they split smoothly, so continuity does not preserve identity as uniqueness. - Also, continuity doesn't ensure uniqueness as I can continually become someone else. - Continuity + MWI means that I will eventually become anyone, that is, open individualism. - Continuity also can be paradoxically preserved without preserving body and mind as a separate process like flame. - Humans also seems to survive discontinuity during sleep without problems. Nevertheless, continuity appears to be inherent to qualia, emerging after observation but before action, and in planning. Only if continuity of consciousness is an ontological necessity, it can be a valid NIIC, but this is true only in a specific version of solipsism (qualia dust theory). In that case, a specific set of personal qualia becomes the personality carrier. If copy=original then even imperfect copies suffice for personal immortality, as self-recognition becomes the main criterion of identity. Open and closed individualism deny the existence of identity but must reinvent it as "practical identity" and therefore revert to copy=original. To resolve the main dichotomy, we must recognize that personal identity requires an overarching system: God, qualia world, social agreement, or evolutionary fitness. This means that we cannot solve identity without solving metaphysics (and the nature of time). Unlikely we'll solve this, before creating superintelligent AI. Therefore, a conservative approach to personal identity is preferable: as we don't know the nature of identity, we should preserve as much as possible and avoid Mars Teleporting unless necessary for survival. We can solve the Mars Transporter (even broken) by incorporating mind merging later, which is another conservative approach. We can solve the "I don't care about my copy" problem through indexical uncertainty and by dividing "copy" into "mirror copy", "personality-copy" and "future copy". If both cyclic universe and continuity as identity are true, I will eventually become any my copy. MWI is functionally equivalent to cyclic universe, so I will become any copy in different timesteps. Therefore, we should care about parallel copies only if future copies don't exist (though in MWI future copies always exist, plus the chance to become someone else).

Sideloading is the creation of a digital model of a person during their life via iterative improvements of this model based on the person's feedback. The progress of LLMs with large prompts allows the creation of very large, book-size prompts which describe a personality. We will call mind-models created via sideloading "sideloads"; they often look like chatbots, but they are more than that as they have other output channels, like internal thought streams and descriptions of actions. By arranging the predictive power of the facts about a person, we can write down the most important facts first. This allows us to get a high level of fidelity in a person's model by writing down a finite number of facts and running it as an LLM prompt with instructions to create a person's chatbot. This instruction also includes an extensive general sideload prompt (we will call it a prompt-loader), which explains how the person's sideload should work, but it is universal and works as an operating system for different persons. I created my personal sideload to test all this and it started working surprisingly quickly. We found that there are three main levels of facts about a person: core facts, long-term memory, and historical facts. Core facts should go to the main prompt of the LLM, long-term memory belongs to RAG, and historical facts – which a person often does not remember – should be only used to extract new relevant data. The most important thing in preparing data for sideloading is to get a full list of the core facts, and not to mix it with long-term memory and historical facts. The LLM-sideload project is inherently future-oriented. New state-of-the-art LLMs with large prompt capabilities emerge quarterly. This allows for the preparation of prompts that exceed current LLM capabilities, anticipating smooth execution in the following year. Below we present the key findings of this work in the form of an executive summary: Quality of sideloads is evaluated using three metrics: • Facts: Accuracy of the answers about the person's life • Vibe: How well the sideload captures the person's style and personality, as judged by the person themselves and their acquaintances • Brilliant insights: Ability to contribute unique, valuable ideas beyond the training data, while maintaining the author's style • Coarseness: Levels of details as percent of the total of recordable textual information from memory estimated to be 10000 pages. Similar to the jpeg level of compression. Estimated performance of Alexey Turchin's sideload: • Facts: 70% correct • Vibe: 20% accurate • Brilliant insights: near zero • Coarseness: 10% The project achieved these results in about one month of work. Vibe is harder to evaluate and regulate than facts. This approach focuses on informational personal identity, setting aside issues of observer identity and internal qualia.

Abstract. In this article is presented a model of the change of the probability of the global catastrophic risks in the world with exponentially evolving technologies. Increasingly cheaper technologies become accessible to a larger number of agents. Also, the technologies become more capable to cause a global catastrophe. Examples of such dangerous technologies are artificial viruses constructed by the means of synthetic biology, non-aligned AI and, to less extent, nanotech and nuclear proliferation. The model shows at least double exponential growth of the probability of the global catastrophe which means that the accumulated probability of the catastrophe will grow from negligible to overwhelming at the period of a few doubling times of the technological capabilities. For biotech and AI, such doubling time roughly corresponds to the doubling period of Moore’s law and its analogues in other technologies and is around 2 years. Thus the global catastrophe in the exponential technologies world will most likely happen during a “dangerous decade”. Such a dangerous decade could start as early as in the 2020s. We also found that the double exponential growth in the model makes the model less sensitive to its initial conditions, like the number of dangerous agents or initial probabilities constants. The model also shows that smaller catastrophes could happen earlier than larger ones, and such a smaller catastrophe may be able to stop the technological growth before larger ones become possible, thus global risks will be self-limiting. However, if the growth of the number of dangerous agents will be very quick, the multiple smaller catastrophes could happen simultaneously and will be equal to a global catastrophe.

Does a single mechanism of aging exit? Most scientists have their own pet theories about what is aging, but the lack of generally accepted theory is mind-blowing. Here we suggest an explanation: evolution works against unitary mechanism of aging because it equalizes ‘warranty period’ of different resilience systems. Therefore, we need life-extension methods that go beyond fighting specific aging mechanisms: such as using a combination of geroprotectors or repair-fixing bionanorobots controlled by AI.

Humanity may underestimate the rate of natural global catastrophes because of the survival bias (“anthropic shadow”). But the resulting reduction of the Earth’s future habitability duration is not very large in most plausible cases (1-2 orders of magnitude) and thus it looks like we still have at least millions of years. However, anthropic shadow implies anthropic fragility: we are more likely to live in a world where a sterilizing catastrophe is long overdue and could be triggered by unexpectedly small human actions. In the same way, an over-inflated toy balloon, which will soon burst, is very fragile. Anthropic fragility can manifest itself in the higher chances of runaway global warming. It has often been suggested that the Earth's atmosphere remained life-supporting for billions of years by sheer chance. Therefore, the survival bias can be strong. It is also known that Earth-like water worlds could experience transitions into deadly moisture greenhouse (mean T = 65C). This means that relatively small anthropogenic actions could put the climate above an unpredictable tipping point, which could lead to the moisture greenhouse. Thus, it is necessary to carry out urgent geoengineering studies and prepare to prevent an unexpected climate catastrophe. There are three main counterarguments against the existence of the anthropic shadow: self-indication assumption (SIA), past observers and the Gaia hypothesis; we show that they fail. It was proposed that SIA exactly compensates the anthropic shadow as an observer unlikely to find herself in a world with a strong anthropic shadow; however, there is a baseline level of the anthropic shadow for all habitable planets, similar to the rate of evolutionary transitions like abiogenesis. There are no “past observers” as qualified observers appeared only 50 years ago. Gaia hypothesis assumes existence of self-stabilizing feedback in climate, but new types of events like quick CO2 growth could override its coping ability. We present a list of other catastrophes that may have been underestimated because of the anthropic shadow, including collider catastrophes, nuclear war and even an alien invasion. We also hypothesized that human intelligence is more likely to emerge in an unstable world that is nearing its end and thus we get a new form of Doomsday argument.

In this article, I present a view on the future of nuclear war which takes into account the expected technological progress as well as global political changes. There are three main directions in which technological progress in nuclear weapons may happen: a) Many gigaton scale weapons. b) Cheaper nuclear bombs which are based on the use of the reactor-grade plutonium, laser isotope separation or are hypothetical pure fusion weapons. Also, advanced nanotechnology will provide the ability to quickly build large nuclear arsenals and AI could be used in designing, manufacturing and nuclear strategy planning. c) Specialized nukes like nuclear-powered space lasers, hafnium bombs and nuclear-powered space ships as kinetic weapons. Meanwhile, the nuclear war strategy also has changed as the bipolar world has ended and as new types of weapons are becoming available. The first strategy change is that Doomsday weapons for blackmail will become attractive for weaker countries which can’t use ICBM to penetrate the anti-missile defense of the enemies. Secondly, the cheaper nukes will become available to smaller actors, who may be involved in “worldwide guerilla”. Cheaper nukes and a larger number of actors also encourage nuclear terrorism and anonymous or false-flag warfare. This will result in the normalization of the use of nuclear weapons in war and growing nuclear arsenals, now distributed between many countries. The third change of the strategy is the use of nukes not against primary military targets but on other objects, which could increase their effects: nuclear power plants, supervolcanos, EMP, tsunamis, taiga fires for global cooling, and even more hypothetical things like asteroid deflection to Earth. All these unconventional means and strategies of nuclear war could become available somewhere in the 21st century and may cause or contribute to the civilizational collapse and even human extinction.

In AI safety research, the median timing of AGI arrival is often taken as a reference point, which various polls predict to happen in the middle of 21 century, but for maximum safety, we should determine the earliest possible time of Dangerous AI arrival. Such Dangerous AI could be either AGI, capable of acting completely independently in the real world and of winning in most real-world conflicts with humans, or an AI helping humans to build weapons of mass destruction, or a national state coupled with AI-based government system. In this article, I demonstrate that the earliest timing of Dangerous AI, corresponding to 10 per cent of its arrival probability, is before 2030. Several partly independent sources of information are in accordance: 1. The growth of the hardware available for AI research makes human-brain-equivalents of compute available for AI research in the 2020s. It is fuelled by specialized AI-chips, the use of many chips in one processing unit, and the larger research budgets, among other things. 2. The neural network performance and other characteristics, like the number of parameters, is quickly increasing every year, and extrapolating this tendency suggests that roughly human-level performance in a few years, around 2025. 3. Expert polls show around 10 per cent of the probability of an early appearance of artificial general intelligence (AGI) in the next decade, that is, before 2030. 4. Hyperbolic growth in different big history models converges around 2025-2030 (the technological singularity). 5. Anthropic arguments (similar to the Doomsday argument) suggest that qualified observers are more likely to appear near the end of the AI research epoch, as the number of such observers grew exponentially. This number doubles every 5-10 years, and thus we are likely to find ourselves around a decade before the end of AI research, which will happen consequently around 2030.

With the fast pace of AI development, the problem of preventing its global catastrophic risks arises. However, no satisfactory solution has been found. From several possibilities, the confinement of AI in a box is considered as a low-quality possible solution for AI safety. However, some treacherous AIs can be stopped by effective confinement if it is used as an additional measure. Here, we proposed an idealized model of the best possible confinement by aggregating all known ideas in the field of AI boxing. We model the confinement based on the principles used in the safety engineering of nuclear power plants. We show that AI confinement should be implemented in several levels of defense. These levels include 1) AI design in fail-safe manner 2) limiting its capabilities, preventing self-improving and circuit breakers on treacherous turn 3) isolation from the outside world and, lastly, as the final hope 4) outside measures oriented on stopping AI in the wild. We demonstrate that the substantial number (more than 50 ideas listed in the article) of mutually independent measures could provide a relatively high probability of the containment of a human-level AI but may be not sufficient to preserve runaway of superintelligent AI. Thus, these measures will work only if they are used to prevent superintelligent AI creation, but not for containing superintelligence. We suggest that there could be a safe operation threshold, on which AI is useful, but is not able to hack containment system from the inside, the same way as a safe level of chain reaction inside nuclear power plants is maintained. However, overall, a failure of the confinement is inevitable, so we need to use the full AGI limited number of times (AI-ticks).

Lucid dreaming (LD) is a fun and interesting activity, but most participants have difficulties in attaining lucidity, retaining it during the dream, concentrating on the needed task and remembering the results. This motivates to search for a new way to enhance lucid dreaming via different induction techniques, including chemicals and electric brain stimulation. However, results are still unstable. An alternative approach is to reach the lucid dreaming-like states via altered state of consciousness not related to dreaming. Several methods such as guided visualization, internal dialog, creative writing, hypnosis, hypnagogia, daydreaming, DMT trips, voice dialog, shamanic journey, rebirthing, and “forcing” tulpas can help in attaining such states. One of the most promising of them is Jungian “active imagination” (AIM) technique, which allows unconscious content to build up inside some mental frames. This article explores the hypothesis of replacing lucid dreaming research with active imagination, and the conditions and ways to accomplish it. Method: An open label pilot experiment was performed in 2004-2005 in Moscow, Russia with 100 participants. Results: The results show that there are two groups of people: ones with “visual imagination screen” and others have “mental imagination screen”. AIM works perfectly as a replacement for lucid dreams only for the first group of people. For the second group, it created interesting content, but not visual or emotional intensity equal to enter lucid dreaming like state. No known instruments helped to move the person from one group to another. The first group consisted of young females, while the second mostly contained males with rational and mathematical type of personality. Conclusion: AIM partly works as a replacement for LD, as it works great only for half of people, and it requires a sitter. However, AIM outperforms LD in reliability and availability in any circumstance: it could be performed even by text chat or in a crowd. It is also better than LD in retaining concentration on topic and the easiness of memorizing the results (which could be recorded). Self-performed AIM is less effective. AIM can be improved by intelligent chat bots as sitters and weak brain stimulation that can increase the probability of attaining something like hypnogogic state.

This article explores theoretical conditions necessary for “quantum immortality” as well as its possible practical implications. It is demonstrated that QI is a particular case of “multiverse immortality”, which is based on two main assumptions: the very large size of the universe ; and a copy-friendly theory of personal identity. It is shown that a popular objection about lowering of the world-share of an observer in the case of QI does not succeed, as the world-share decline could be compensated by merging timelines for the simpler minds, and because some types of personal preferences are not dependent on such changes. Despite large uncertainty about the truth of MI, it has appreciable practical consequences for some important outcomes like suicide and aging. The article demonstrates that MI could be used to significantly increase the expected subjective probability of success of risky life extension technologies, such as cryonics, but that it makes euthanasia impractical because of the risk of eternal suffering. Euthanasia should be replaced with cryothanasia, i.e. cryopreservation after voluntary death. Another possible application of MI is as a last chance to survive a global catastrophe. MI could be considered a Plan D for reaching immortality, where Plan A consists of survival until the development of beneficial Artificial Intelligence capable of fighting aging, Plan B employs cryonics, and Plan C is digital immortality.

The long unbearable sufferings in the past and agonies experienced in some future timelines in which a malevolent AI could torture people for some idiosyncratic reasons (s-risks) is a significant moral problem. Such events either already happened or will happen in causally disconnected regions of the multiverse and thus it seems unlikely that we can do anything about it. However, at least one pure theoretic way to cure past sufferings exists. If we assume that there is no stable substrate of personal identity and thus a copy equals original, then by creating many copies of the next observer-moment of a person in pain in which he stops suffer, we could create indexical uncertainty in her future location and thus effectively steal her consciousness from her initial location and immediately relieve her sufferings. However, to accomplish this for people who have already died, we need to perform this operation for all possible people thus requiring enormous amounts of computations. Such computation could be performed by the future benevolent AI of Galactic scale. Many such AIs could cooperate acausally by distributing parts of the work between them via quantum randomness. To ensure their success, they need to outnumber all possible evil AIs by orders of magnitude, and thus they need to convert most of the available matter into computronium in all universes where they exist and cooperate acausally across the whole multiverse. Another option for curing past suffering is the use of wormhole time-travel to send a nanobot in the past which will, after a period of secret replication, collect the data about people and secretly upload them when their suffering becomes unbearable.

Abstract. The presumptuous philosopher (PP) thought experiment lends more credence to the hypothesis which postulates the existence of a larger number of observers than other hypothesis. The PP was suggested as a purely speculative endeavor. However, there is a class of real world observer-selection effects where it could be applied, and one of them is the possibility of interstellar panspermia (IP). There are two types of anthropic reasoning: SIA and SSA. SIA implies that my existence is an argument that larger total number of observers exists; SSA implies that I should find myself in a region with larger number of observers. However, as S. Armstrong showed, SIA can’t distinguish between different ways how larger number of observers appeared, so it can’t favor IP compared with other ways to get many observers. SSA application here is less controversial as it tells only about relation between regions size: e.g. I am more likely to live in a larger country than in a small one, conditioning that the total number of small countries is also small. Anthropic considerations suggest that the universes in the multiverse with interstellar panspermia will have orders of magnitude more civilizations than universes without IP, and thus we are likely to be in such a universe. This is a strong counterargument against a variant of the Rare Earth hypothesis based on the difficulty of abiogenesis: even if abiogenesis is difficult, IP will disseminate life over billions of planets, meaning we are likely to find ourselves in a universe where IP has happened. This implies that there should be many planets with life in our galaxy, and renders the Fermi paradox even sharper. Either the Great Filter is ahead of us and there are high risks of anthropogenic extinction, or there are many alien civilizations nearby and of the same age as ours—which is itself a global catastrophic risk, as in that case the wave of alien colonization could arrive between 25-500 ky from now.

Abstract: After 2017 NY Times publication, the stigma of the scientific discussion of the problem of so-called UAP (Unidentified Aerial Phenomena) was lifted. Now the question arises: how UAP will affect the future of humanity, and especially, the probability of the global catastrophic risks? To answer this question, we assume that the Nimitz case in 2004 was real and we will suggest a classification of the possible explanations of the phenomena. The first level consists of mundane explanations: hardware glitches, malfunction, hoaxes and lies. The second level involves explanations which are not changing completely our model of the world, but only updating it: new military tech, large psyop operation or new physical effect like ball lightning. The third level of explanations requires a complete overhaul of our world model and includes a large set of fantastic hypothesis: alien starships, interdimensional beings, glitches in the matrix, projections from the collective unconsciousness, Boltzmann brain’s experiences etc. The higher is the level of the hypothesis, the less probable it is, but the bigger is its consequences for possible global catastrophic risks. Thus, “integrating” over the field of all possible explanations, we find that UAP increases catastrophic risks. Technological progress increases our chances of direct confrontation with the phenomena. If the phenomenon has intelligence, the change of its behavior could be unexpected after we pass some unknown threshold. But we could lower the risks if we assume some reasonable policy of escaping confrontations.

The problem of surviving the end of the observable universe may seem very remote, but there are several reasons it may be important now: a) we may need to define soon the final goals of runaway space colonization and of superintelligent AI, b) the possibility of the solution will prove the plausibility of indefinite life extension, and с) the understanding of risks of the universe’s end will help us to escape dangers like artificial false vacuum decay. A possible solution depends on the type of the universe’s ending that may be expected: very slow heat death or some abrupt end, like a Big Rip or Big Crunch. We have reviewed the literature and identified several possible ways of survival the end of the universe, and also suggest several new ones. There are seven main approaches to escape the end of the universe: use the energy of the catastrophic process for computations, move to a parallel world, prevent the end, survive the end, manipulate time, avoid the problem entirely or find some meta-level solution.

Abstract: The field of life extension is full of ideas but they are unstructured. Here we suggest a comprehensive strategy for reaching personal immortality based on the idea of multilevel defense, where the next life-preserving plan is implemented if the previous one fails, but all plans need to be prepared simultaneously in advance. The first plan, plan A, is the surviving until advanced AI creation via fighting aging and other causes of death and extending one’s life. Plan B is cryonics, which starts if plan A fails, and assumes cryopreservation of the brain until technical capabilities to return it to life appear. Plan C is digital immortality in the sense of collecting data about the person now so future AI will be able to recreate a model of a person. Plan D is the hope based on some unlikely scenarios of infinite survival, like so-called “quantum immortality”. All these plans have personal and social perspective. The personal aspect means efforts of the increasing chances of personal survival via taking care about one’s own health, signing cryocontract or collecting digital immortality data. The social aspect means the participation in collective work towards creation and increase of the availability of life extension technologies, which includes funding scientific research, promotion of life extension value and direct performing of research and implementation, as well as preventing global catastrophic risk. All plans converge at the end, as their result is the indefinite survival as an uploaded mind inside an ecosystem, created by a superintelligent AI.

Purpose This paper aims to formalize long-term trajectories of human civilization as a scientific and ethical field of study. The long-term trajectory of human civilization can be defined as the path that human civilization takes during the entire future time period in which human civilization could continue to exist. Design/methodology/approach This paper focuses on four types of trajectories: status quo trajectories, in which human civilization persists in a state broadly similar to its current state into the distant future; catastrophe trajectories, in which one or more events cause significant harm to human civilization; technological transformation trajectories, in which radical technological breakthroughs put human civilization on a fundamentally different course; and astronomical trajectories, in which human civilization expands beyond its home planet and into the accessible portions of the cosmos. Findings Status quo trajectories appear unlikely to persist into the distant future, especially in light of long-term astronomical processes. Several catastrophe, technological transformation and astronomical trajectories appear possible. Originality/value Some current actions may be able to affect the long-term trajectory. Whether these actions should be pursued depends on a mix of empirical and ethical factors. For some ethical frameworks, these actions may be especially important to pursue.

One of the possible solutions of the Fermi paradox is that all civilizations go extinct because they hit some Late Great Filter. Such a universal Late Great Filter must be an unpredictable event that all civilizations unexpectedly encounter, even if they try to escape extinction. This is similar to the “Death in Damascus” paradox from decision theory. However, this unpredictable Late Great Filter could be escaped by choosing a random strategy for humanity’s future development. However, if all civilizations act randomly, this could be the actual cause of universal extinction, as no one will choose the most optimal strategy. Using a meta-random strategy, where the decision to start random actions is also made randomly, solves this.

Abstract. Boltzmann brains (BBs) are minds which randomly appear as a result of thermodynamic or quantum fluctuations. In this article, the question of if we are BBs, and the observational consequences if so, is explored. To address this problem, a typology of BBs is created, and the evidence is compared with the Simulation Argument. Based on this comparison, we conclude that while the existence of a “normal” BB is either unlikely or irrelevant, BBs with some ordering may have observable consequences. There are two types of such ordered BBs: Boltzmannian typewriters (including Boltzmannian simulations), and chains of observer moments. Notably, the existence or non-existence of BBs may have practical applications for measuring the size of the universe, achieving immortality, or even manipulating the observed probability of events. Disclaimer and trigger warning: some people have emotional breakdowns when thinking about topics described in the article, especially the “flux universe”. However, everything eventually adds up to normality.

Abstract: In the last decade, an urban legend about “glitches in the matrix” has become popular. As it is typical for urban legends, there is no evidence for most such stories, and the phenomenon could be explained as resulting from hoaxes, creepypasta, coincidence, and different forms of cognitive bias. In addition, the folk understanding of probability does not bear much resemblance to actual probability distributions, resulting in the illusion of improbable events, like the “birthday paradox”. Moreover, many such stories, even if they were true, could not be considered evidence of glitches in a linear-time computer simulation, as the reported “glitches” often assume non-linearity of time and space—like premonitions or changes to the past. Different types of simulations assume different types of glitches; for example, dreams are often very glitchy. Here, we explore the theoretical conditions necessary for such glitches to occur and then create a typology of so-called “GITM” reports. One interesting hypothetical subtype is “viruses in the matrix”, that is, self-replicating units which consume computational resources in a manner similar to transposons in the genome, biological and computer viruses, and memes.

The goal of the article is to explore what is the most probable type of simulation in which humanity lives (if any) and how this affects simulation termination risks. We firstly explore the question of what kind of simulation in which humanity is most likely located based on pure theoretical reasoning. We suggest a new patch to the classical simulation argument, showing that we are likely simulated not by our own descendants, but by alien civilizations. Based on this, we provide classification of different possible simulations and we find that simpler, less expensive and one-person-centered simulations, resurrectional simulations, or simulations of the first artificial general intelligence’s (AGI’s) origin (singularity simulations) should dominate. Also, simulations which simulate the 21st century and global catastrophic risks are probable. We then explore whether the simulation could collapse or be terminated. Most simulations must be terminated after they model the singularity or after they model a global catastrophe before the singularity. Undeniably observed glitches, but not philosophical speculations could result in simulation termination. The simulation could collapse if it is overwhelmed by glitches. The Doomsday Argument in simulations implies termination soon. We conclude that all types of the most probable simulations except resurrectional simulations are prone to termination risks in a relatively short time frame of hundreds of years or less from now.

Abstract: The field of artificial general intelligence (AGI) safety is quickly growing. However, the nature of human values, with which future AGI should be aligned, is underdefined. Different AGI safety researchers have suggested different theories about the nature of human values, but there are contradictions. This article presents an overview of what AGI safety researchers have written about the nature of human values, up to the beginning of 2019. 21 authors were overviewed, and some of them have several theories. A theory classification method is suggested, where the theories are judged according to the level of their complexity and behaviorists-internalists scale, as well as the level of their generality-humanity. We suggest that a multiplicity of well-supported theories means that the nature of human values is difficult to define, and some meta-level theory is needed.

Abstract. The main current approach to the AI safety is AI alignment, that is, the creation of AI whose preferences are aligned with “human values.” Many AI safety researchers agree that the idea of “human values” as a constant, ordered sets of preferences is at least incomplete. However, the idea that “humans have values” underlies a lot of thinking in the field; it appears again and again, sometimes popping up as an uncritically accepted truth. Thus, it deserves a thorough deconstruction, which I will do by listing and analyzing comprehensively the hidden assumptions of the idea that “humans have values.” This deconstruction of human values will be centered around the following ideas: “Human values” are useful descriptions, but not real objects; “human values” are bad predictors of behavior; the idea of a “human value system” has flaws; “human values” are not good by default; and human values cannot be separated from human minds. The method of analysis is listing hidden assumptions on which the idea of “human values” is built. I recommend that either the idea of “human values” should be replaced with something better for the goal of AI safety, or at least be used very cautiously. The approaches to AI safety which don’t use the idea of human values at all may require more attention, like the use of full brain models, boxing, and capability limiting.

Abstract: In the future, it will be possible to create advance simulations of ancestor in computers. Superintelligent AI could make these simulations very similar to the real past by creating a simulation of all of humanity. Such a simulation would use all available data about the past, including internet archives, DNA samples, advanced nanotech-based archeology, human memories, as well as text, photos and videos. This means that currently living people will be recreated in such a simulation, and in some sense, “resurrected”. Such “resurrectional simulation” could be deliberately created just for this goal: to return to life all people who have ever lived. The main technical problem of such simulation will be uncertainty about the past, which increases exponentially for more remote times. Such problem could be partly addressed by “acausal trade” between different branches of the multiverse, which will create slightly different versions of the simulation using a quantum randomness generator. Such trade will result in resurrection of all possible people (including those who existed in other branches). Ethical problems of such a resurrectional simulation include: a) possible resurrection of some people against their will; b) such simulation may create additional suffering; с) such simulation could be used by hostile AI to return people to life and then torture them. In this work, I explore preliminary ideas about how to address these problems.

Abstract: As there are no visible ways to create safe self-improving superintelligence, but it is looming, we probably need temporary ways to prevent its creation. The only way to prevent it, is to create special AI, which is able to control and monitor all places in the world. The idea has been suggested by Goertzel in form of AI Nanny, but his Nanny is still superintelligent and not easy to control, as was shown by Bensinger at al. We explore here the ways to create the safest and simplest form of AI, which may work as AI Nanny. Such AI system will be enough to solve most problems, which we expect the AI will solve, including control of robotics, acceleration of the medical research, but will present less risk, as it will be less different from humans. As AI police, it will work as operation system for most computers, producing world surveillance system, which will be able to envision and stop any potential terrorists and bad actors in advance. As uploading technology is lagging, and neuromorphic AI is intrinsically dangerous, the most plausible way to human-based AI Nanny is either functional model of the human mind or a Narrow-AI empowered group of people.

Many global catastrophic and existential risks (X-risks) threaten the existence of humankind. There are also many ideas for their prevention, but the meta-problem is that these ideas are not structured. This lack of structure means it is not easy to choose the right plan(s) or to implement them in the correct order. I suggest using a “Plan A, Plan B” model, which has shown its effectiveness in planning actions in unpredictable environments. In this approach, Plan B is a backup option, implemented if Plan A fails. In the case of global risks, Plan A is intended to prevent a catastrophe and Plan B to survive it, if it is not avoided. Each plan has similar stages: analysis, planning, funding, low-level realization and high-level realization. Two variables—plans and stages—provide an effective basis for classification of all possible X-risks prevention methods in the form of a two-dimensional map, allowing the choice of optimal paths for human survival. I have created a framework for estimating the utility of various prevention methods based on their probability of success, the chances that they will be realized in time, their opportunity cost, and their risk. I also distinguish between top-down and bottom-up approaches.

Abstract: Global chemical contamination is an underexplored source of global catastrophic risks that is estimated to have low a priori probability. However, events such as pollinating insects’ population decline and lowering of the human male sperm count hint at some toxic exposure accumulation and thus could be a global catastrophic risk event if not prevented by future medical advances. We identified several potentially dangerous sources of the global chemical contamination, which may happen now or could happen in the future: autocatalytic reactions, exposure to multiple subthreshold sources, and long-term unintended consequences, arising from both natural and bioengineered sources. We list several especially dangerous chemicals—dioxin, organiс compounds, and toxic heavy metals. We also discuss the features of such dangerous chemicals—molecules that can stay in the biosphere for a long time and affect it over time. We explore several social processes and scenarios where global chemical contamination becomes possible: large natural catastrophe like meteorite impact, supervolcano eruption, new ways of predicting properties of the chemicals via machine learning and their manufacturing via synthetic biology, uncontrolled “capitalistic” economic development with a corresponding large waste production, quick adoption of many chemicals with unknown long-term properties and unintended side-effects. These are all low probability, so work on other global catastrophic risks should be prioritized, but chemical risks could exacerbate other types of catastrophe contributing to social collapse

Abstract: Four main forms of Doomsday Argument (DA) exist—Gott’s DA, Carter’s DA, Grace’s DA and Universal DA. All four forms use different probabilistic logic to predict that the end of the human civilization will happen unexpectedly soon based on our early location in human history. There are hundreds of publications about the validity of the Doomsday argument. Most of the attempts to disprove the Doomsday Argument have some weak points. As a result, we are uncertain about the validity of DA proofs and rebuttals. In this article, a meta-DA is introduced, which uses the idea of logical uncertainty over the DA’s validity estimated based on a virtual prediction market of the opinions of different scientists. The result is around 0.4 for the validity of some form of DA, and even smaller for “Strong DA”, which predicts the end of the world in the near term. We discuss many examples of the validity of the DA in real life as an instrument to prove it “experimentally”. We also show that DA becomes strongest if it is based on the idea of the “natural reference class” of observers, that is, the observers who know about the DA (i.e. a Self-Referenced DA). Such a DA predicts that there is a high probability of a global catastrophe with human extinction in the 21st century, which aligns with what we already know based on analysis of different technological risks.