Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an "intelligence explosion", and the intelligence of man would be left far behind […] Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously (Good, 1965, p. 33) .
And nowadays, the concepts of Singularity and intelligence explosion have even been cited in Stanford's One-Hundred Year Study of Artificial Intelligence (besides several other works):
Speculations about the rise of such uncontrollable machine intelligences have called out different scenarios, including trajectories that take a slow, insidious course of refinement and faster-paced evolution of systems toward a powerful intelligence "singularity." Are such dystopic outcomes possible? If so, how might these situations arise? What are the paths to these feared outcomes? What might we do proactively to effectively address or lower the likelihood of such outcomes, and thus reduce these concerns? What kind of research would help us to better understand and to address concerns about the rise of a dangerous super intelligence or the occurrence of an "intelligence explosion"? Concerns about the loss of control of AI systems should be addressed via study, dialog, and communication. Anxieties need to be addressed even if they are unwarranted (Horvitz, 2014, p. 5 ).
Would there be any indication that an intelligence explosion is something, however unlikely, still possible? Perhaps, we have already found in the literature the first indications of autonomous systems assisting in the development of other autonomous systems. Zoph and Le (2017) proposed an autonomous technique for the development of artificial neural network architecture. According to the authors: "our method, starting from scratch, can design a new network architecture that rivals the best architecture invented by man…" (Zoph & Le, 2017, p. 1) . The authors developed their model using Reinforcement Learning (RL) to train their "architect" system of artificial neural networks. RL is one of the paradigms in the area of machine learning, where artificial agents must act in the environment that they are embedded, to maximize their reward function (Russel & Norvig, 2003) .
Reward functions are a mathematical representation of the preferences that guide the behavior of agents operating by RL, where, for example, a cleaning robot can maximize a function that assigns "little dirt on the floor" a high reward, and world states where the floor is dirty with a low reward. Many of the models used to study idealized rational agents (Expected Utility Theory) provide convincing arguments that any rational agent with consistent preferences should act as an expected utility maximizer (Von Neumann & Morgenstern, 1944) . However, within the framework of expected utility theory, there are corollary results that seem to refer to the concern of Good, quoted above: "[...] as long as the machine is docile enough to tell us how to keep it under control [...]". Stephen Omohundro (2008) , cites some characteristics that we should expect artificial intelligent agents to possess. Bostrom (2014, chapter 7, pp. 110-112) Power is the ability to achieve goals in general and to gain dominance over the environment. It's instrumentally convergent to a wide range of terminal goals to search for power. A corollary of the results demonstrated by Turner et al, is that even in simplified conditions, we see that most reward functions induce a searchfor-power behavior, something that may cause safety problems involving the interaction of humans and AI.
In light of all these arguments, which date back to the early days of AI research, security issues have increasingly been cited in the literature. AI ethics, a sub-area of applied ethics concerned with adding moral behavior to machines and regulating the use of artificial intelligence, has been gaining a significant increase in popularity in the last two decades (Jobin et al, 2019; Jurić et al, 2020) . Important philosophical and technical questions are raised in the context of AI safety, e. g., Corrigibility: how to correct/terminate potentially faulty agents that have a strong instrumental incentive to preserve their terminal goals (Soares et al, 2015; Amodei et al, 2016) ? We can find in the literature several research agendas, where different types of ethical, technical, and social problems are discussed (Russel et al, 2015; Taylor et al, 2016; Tegmark, 2016; Soares, 2016; O'Keefe et al, 2020; ÓhÉigeartaigh et al, 2020; Hagendorff, 2020) .
At one end of the spectrum, we find research involving existential risks, i. e., the study of possible threats at the extinction-level imposed by present or future technology. Research centers such as the Centre for the Study of Existential Risk in Cambridge, the Future of Life Institute in Boston (Russel et al, 2015) , specifically focused on existential risk involving advanced artificial intelligence, and the Future of Humanity Institute in Oxford (Bostrom, 2002) , search for strategies to mitigate certain types of dystopian future.
We can compare our current scenario about artificial intelligence with past events that led to the construction of nuclear power plants and weapons. A technological race headed by the former global super-powers led to the mass production of systems that we still did not have a complete understanding. This caused several side effects, like accidents (Chernobyl disaster), the creation of weapons of mass destruction (Cold War), and even the use of these weapons against human society itself (Atomic bombings of Hiroshima and Nagasaki). Certainly, the pressures for the development of high-performance AI, given its capacity to provide the organization that controls it a considerable strategic advantage, will cause the same type of technological race that we experienced in the mid-20th century:
"while X invests in the development of AI, Y will do as well". Currently, the main contenders in this race are countries like the USA and China (ÓhÉigeartaigh et al, 2020).
Another reason for caution in our technological advances in the area of AI, is that different from common thinking, for an artificially intelligent system to represent a potential danger to our society, it doesn't need to be more intelligent than us humans. Rather, it needs to be more capable in certain kinds of tasks. Barret and Baum (2017) telecommunications. In 2017, the crypto-ransomware "WannaCry", malicious software that hacks into computers and private networks, encrypting their content, and only providing the key to decryption after payment of a ransom, reached several systems in the world in more than 99 countries, even affecting the public health system of certain governments. More than 75,000 ransom demands were made, making it one of the most damaging cyber-attacks in history (Larson, 2017 ). This would be a possible DSA of an AI, the ability to execute cyber-attacks on our infrastructure in a way that we cannot remedy in time.
The ASI-PATH provides an intuitive diagram where various events (i. e., security breaches) must occur to cause a catastrophe involving advanced artificial intelligence. Initially, an AI, also called a seed AI, must first become an AI with some DSA, and at the same time, the security measures must have failed. Witch includes: failures in confinement, failed value alignment, AI objectives diverge from ours, containment fails, etc. Sotala (2018, p. 317 ) provides a simplified view of ASI-PATH in his work "Disjunctive scenarios of catastrophic AI risk". As suggested by Barret and Baum's model, the arguments raised by the thesis of instrumental convergence and the orthogonality thesis are some of the reasons that could lead a Singularity to engage in hostile actions against humans.
The scenarios explored in the literature, where a seed AI is capable of becoming a Singularity, are usually characterized in two different types of takeoffs. Rapid takeoffs suggest situations where a drastic takeover occurs, where abruptly we would be surprised by an entity much more capable, with possibly unknown objectives, inserted and sharing the same environment as us. In contrast, we have slow takeoffs, which are a much more realistic possibility. It would occur gradually as the human species becomes more and more dependent, and in a way, under the control of advanced AI systems (Sotala, 2018) . The argument and line of reasoning, behind a slow takeoff, is exposed in this passage of Theodore Kaczynski's manifesto:
If the machines are permitted to make all their own decisions, we can't make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines' decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machinemade decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won't be able to just turn the machine off, because they will be so dependent on them that turning them off would amount to suicide (Kaczynski, 1995, § 173, p. 22 Experts in the development of artificial intelligence predict that within 10 years many human activities will be surpassed by machines in terms of efficiency (Grace et al, 2017) . A survey was conducted by Müller and Bostrom (2016) , where the authors administered a questionnaire to assess the progress in the field of AI research and prospects for the future, interviewing several experts (N 170).
The questionnaire showed that on average, there is a 50 chance that high-level machine intelligence will be achieved between 2040 and 2050, with a 90 probability by 2075. It is also estimated that AI will outperform human performance between 2 (10 chance) and 30 years (75 chance) (Müller & Bostrom, 2016) . In a similar survey conducted by Grace et al (2017) , the researchers interviewed (N 352) believe that AI will outperform human performance in all tasks in 45 years, with a 50 chance, and automate all human work in up to 120 years. However, we emphasize that there is great variability in the results obtained. In Müller and Bostrom's (2016) survey, 33 of respondents classified this development in AI as "bad" or "extremely bad" for humanity. In the research of Grace et al (2017), when those evaluated were asked whether highlevel AI would have a positive or negative impact in the long term, the median probability for "good" and "extremely good" results was 25 and 20 , respectively. The probabilities for a "bad" or "extremely bad" resolution were, respectively, 10 and 5 .
From the above surveys, we can state that the chance that high-level AI will be created in the next 120 years is, at least, being pessimistic, 10% for a certain portion of the academic community. Besides the opinion of specialists in the field, another type of evidence that we can use to infer the possibility of a technological Singularity is how the economic growth rate has behaved during the history of human civilization, and how it's related to technological improvement. One of the most popular models found in the literature on our economic growth, from the Neolithic Revolution to the 21st century, is the growth model proposed by Michael Kremer (1993) . Kremer's model is based on the following simple argument, "two heads think better than one". That is, economic growth is driven by people having new ideas, and the more people, the greater the possibility of new ideas.
For Kremer (1993) the total annual economic output is a function of the size of the This model also suggests that such forms of growth should be separated when we reach a maximum population growth rate of 2100, with a global population between 9.6 billion and 12.3 billion people (Gerland et al, 2014) . When this occurs, technological progress will no longer impact the global population. However, this does not mean that technological progress will stagnate. This type of model is sometimes referred to as the Hyperbolic Growth Hypothesis (HCH), is one of the most accepted economic growth models by the macroeconomic community, and serves as the basis for other theories such as the Unified Growth Theory (Taagepera, 1979; Korotayev et al, 2006; Oded, 2011; Jones, 2013) . Other authors also suggest a disassociation between population growth and economic/technological progress. Thus, when high levels of automation are achieved, economic growth rates will become radically higher, producing more and more technological progress (Yudkowsky, 2013; Bostrom, 2014; Nordhaus, 2015; Agrawal et al, 2017) .
Could this type of economic growth help the development of an AGI? Levin and Maas (2020) ii. The LIGO (Laser Interferometer Gravitational-Wave Observatory), had a total construction cost of US$ 33 million (Castelvecchi, 2015) ;
iii. ITER (International Thermonuclear Experimental Reactor), one of the latest promises for clean and sustainable energy (a Tokamak nuclear fusion experimental reactor), is expected to be ready in 12 years at an annual cost of $2 billion (Fountain, 2017) .
We can see that neither of the projects mentioned above has received as much economic investment like the one dedicated to the Apollo and Manhattan projects (0.4% of the U.S. government's annual GDP), something that also explains the impressive speed with which the goals of both projects were achieved. Even so, significantly less investment did not prevent major scientific discoveries, and broken technological paradigms, such as the decoding of the human genome and the detection of gravitational waves. Thus, it seems more feasible to state that:
when we have a robust enough theoretical understanding of the computational and cognitive processes responsible for the development of AGI, a Singularity may very well be "a Manhattan Project" away.
Currently, there are several active projects to develop AGI. Baum (2017) Transformer models such as ELMo (Peters et al, 2018) , BERT (Devlin et al, 2019) , GPT-2 (Radford et al, 2019) , T-NLG (Corby et al, 2020) , and the more recent, GPT-3 (Brown et al, 2020) , developed by OpenAI, represent some of the most successful models in tasks involving natural language processing, GPT-2 being used to generate several recent patent claims (Lee & Hsiang, 2019) . In June 2020, OpenAI launched GPT-3 (a language model with 175 billion parameters). In their article, the authors show the ability of GPT-3 to generate samples of texts, such as poems, e According to the 2AI Labs website, researchers give the following statement on risk scenarios involving IAG: "We think this is all crazy talk". Available at: http://www.2ai.org/killerai/ articles, and news, from which human evaluators had difficulty, or were unable, to distinguish from human writings.
GPT-3, besides achieving state-of-the-art in several tasks involving natural language processing, it is one of the first AI models to show signs of generalization. How can I find something small that I lost on the carpet? b) Place a hair net at the end of your vacuum cleaner and turn it on.
In any case, there is no evidence that deep neural networks, such as Transformers, perform a type of information processing that makes them an AGI or seed AI. What we may infer is that this type of architecture allows the training of agents capable of solving several tasks that seem to be associated with general intelligence. Thus, the results and capabilities that models such as GPT-3 demonstrate only serve as weak evidence that Dartmouth's Summer Research Project on Artificial Intelligence, initiated by McCarthy et al (1955, p. 2) with the proposal of "[…] try to make machines use language, form abstractions and concepts, and solve types of problems hitherto reserved only for human beings […]", will be successful shortly.
While the research focused on AI safety, which normally focuses on intentional or unintentional physical harm by autonomous agents, we recognize that communication in natural language can also cause harm. For example, the virtual assistant developed by Amazon, Alexa, in 2019 suggested to a user to commit suicide for the greater good, arguing that life only aggravates the rapid degeneration of the planet and consumption of its natural resources (Crowley, 2019) . In March 2016, 24 hours after the launch of its Chabot Tay on the Twitter platform, Microsoft had to end the program because the agent was generating tweets containing racism, anti-Semitism, and sexism (Wolf et al, 2017) . Such events are cause for concern, since soon such systems may be massively used in a wide range of applications.
Given the potential for malicious application of this type of technology, any kind of socially harmful activity that uses advanced language models can also be enhanced.
Whether in generating fake news for mass disinformation, phishing, generating boots on platforms like Twitter to make it more biased (social engineering), or even writing fraudulent academic essays, NLP models have many dubious applications. Brown et al (2020) provide a preliminary analysis in their study, where they report a series of limitations and un-ethical and unsafe behaviors present in GPT-3. In it, the authors demonstrate several biases involving issues such as gender, race, and religion, something that can lead GPT-3 to produce stereotyped content, or, in a worse case, sheer prejudice.
However, are the advances and alerts pointed out by the literature enough for our society to create a collective sense of responsibility and concern with these issues, or should such speculations still be considered only Futurology or science fiction?
IV. Lessons from 2020: Coordination problems for social theory while serving as a political opposition to cyber-fascism lurking on the next horizon". Certain forms of philosophical thought, such as Transhumanism and Singulitarianism, critically debate the possible futures that our social and technological acceleration may be co-creating, and how we can aim for human integration and flourishing rather than more dystopian possibilities. One of the premises for security issues involving our technological advance relies on an idea of negative utopia:
First and foremost, the utopian impulse must be negative: identify the problem or problems that must be corrected. Far from presenting an idyllic, happy and fulfilled world, utopias should initially present the root causes of society's ills [...] to act as a criticism of the existing system (Tally, 2009, p. 115) .
Within this context, we believe that the preoccupations raised by the literature are not unjustified. Immersed in the current context in which our society lives, the pandemic of the new coronavirus, COVID-19, we may or may not learn certain lessons useful for other existential threats. Krakovna (2020) explores how our response to the COVID-19 pandemic raises troubling questions involving our coordination capabilities to manage global crises and risks.
As we have argued before, slow AI takeoffs are a much more likely scenario than scenarios where quick takeoffs occur. However, this does not mean that a slow takeoff is easier, or less dangerous, to manage. For a slow takeoff to be avoided, the same type of global coordination that we failed to demonstrate during the initial development of the new coronavirus pandemic would be required. Krakovna (2020) raises three large-scale coordination problems:
i. The inability to learn from past experiences;
ii. The inability to respond efficiently to warning signals;
iii. Delay in reaching a global consensus on a problem.
In analogy with the present global situation, our society has had the opportunity to learn from similar pandemics that occurred in the past, such as SARS (Severe Acute Respiratory Syndrome), which also appeared to have started in Guangdong, China. (Williams, 2020) . COVID-19 was labeled "an exaggeration", or, "just a little flu" by certain state leaders (Walsh et al, 2020) . Krakovna (2020) articulates a similarity between how we evaluated the risks of COVID-19, and how we evaluate possible risks involving advanced AI. While researchers who adopt a more skeptical stance to the development of advanced AI are seen as prudent, researchers who advocate the adoption of preventive measures are taxed for fear-mongering. Couldn't there be a middle ground?
Currently, the field of AI security research and AI ethics is considerably smaller than the area interested in developing powerful autonomous intelligent systems.
One of the first obstacles we must overcome to achieve greater consensus on safety issues involving AI is the problem that "Artificial Intelligence" is a moving target.
By moving target we mean the following: when we attribute "intelligence" to something it seems to be a self-assessment of our epistemic state. That is, an intelligent act always seems to be something that we do not fully understand as it occurs. For example: if an individual can multiply large numbers quickly, say the square root of arbitrarily large numbers, or know the day of the week of Alan
Turing's birthday, we can judge such an individual as intelligent, or at least a mathematical prodigy. However, if such an individual explains to us how he performs such feats, and that in fact, they are nothing more than arithmetic/algebraic tricks which anyone can perform, the feat stops to appear as something intelligent.
The same effect occurs when we seek to define machine intelligence, "intelligence"
for critics of the computational thesis being everything that AI is not. AGI researchers like Wang (2008) , argues for a more flexible conception of "intelligence" and "artificial intelligence":
AI should not be defined in such a narrow way that takes human intelligence as the only possible form of intelligence, otherwise AI research would be impossible, by definition. AI should not be defined in such a broad way that takes all existing computer systems as already having intelligence, otherwise AI research would be unnecessary, also by definition (Wang, 2008, p. 9 ).
Perhaps no one has proposed this argument more clearly than Edsger Dijkstra, (1984) : "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim". In the past, we thought that intelligence (whatever it is) should be required for, e. g., natural language processing;
i. GPT-3 is capable of performing such a task (Brown et al, 2020) .
Playing chess;
ii. Deep Blue beats Garry Kasparov (Campbella et al, 2002) .
Playing GO;
iii. AlphaGO beats Lee Sedol (Silver et al, 2016) .
Playing "games" in general;
iv. Agent57 beats humans in 57 classic Atari games (Badia et al, 2020) .
Be creative;
v. Intelligent Algorithms of Generative Design are able to find design solutions that humans would not be able to conceive, making it possible to perform 50,000 days of engineering in a single day (Oh et al, 2019) .
Every time we realize that human intelligence isn't needed to perform a task, we discard such a task as proof of intelligence. In the same way that a submarine does not swim, and even so: can move through water and fire intercontinental ballistic missiles, artificial intelligence, indifferent to any anthropomorphic notion of the concept of intelligence that we use, can still influence the environment, adapt, make decisions, update hypotheses, pursue objectives, and if programmed to do, fire intercontinental ballistic missiles. If we keep neglecting the capabilities of os AI systems and marking them as unintelligent, the possibility of true unsafe AI may well be always left outside our hypothesis space.
The parallels drawn from the coronavirus pandemic of 2019 and the possible emergence of misaligned AGI can serve for at least weak evidence for the following statement: our lack of global coordination in dealing with existential risks may well be our only and true existential risks.
In this essay, we aim to provide the reader with a brief introduction to some problems often disregarded by contemporary AI ethics. As much as there is not yet a full consensus in the literature regarding the possibility of creating general artificial intelligence, we have a significant portion of the scientific community that believes that however unlikely such a possibility maybe, security measures should be taken. Should such warnings and advice be dismissed as exaggerations? As fearmongering? Technological development does not slow down, we are increasingly able to produce autonomous systems that act proficiently in several domains, and little by little, these systems demonstrate the first traces of something we can call general intelligence. The AI industry is far from being aligned, like our global society, it lacks a common goal to coordinate its actions. We believe that the lessons we can learn about the current state we live in, under the COVID-19
pandemic, can be useful if we are willing to learn from them. And two of these lessons are:
1) when a risk, however small, is associated with something that represents an existential danger to our species, to global society, caution and security should not be synonymous with exaggeration and fuss;
2) lack of global coordination may be our biggest enemy after all.