trentmkelly/LessWrong-43k huggingface dataset:
AI and the Probability of Conflict
by tonyoconnor — 2021-04-01T07:00:32.130Z

Cross-posted from The Metasophist. Necessarily speculative.

It is commonly argued that artificial general intelligence (AGI), unaligned with human values, represents an existential risk for humanity. For example, in his recent book The Precipice, philosopher Toby Ord argued that there is a 10 percent chance that unaligned artificial general intelligence will result in existential catastrophe for humanity in the coming century.
Less attention is devoted to whether or how aligned AGI could introduce additional existential risk, such as through increasing the probability of conflict. For example, how would one country react to the news that another country is on the verge of developing an AGI fully under their control, a potential geopolitical game changer similar to the advent of nuclear weapons?
Aligned AGI could have important geopolitical implications by altering both the nature and probability of conflict. But how exactly? This is difficult to answer given that the form of AGI is still hugely uncertain. Nevertheless, the different scenarios explored below indicate that aligned AGI would not greatly increase the probability of conflict.
Let’s explore these scenarios in detail.

Setting the scene

Consider two countries, the Hegemon and the Challenger. Furthermore, assume that both countries are trying to develop AGI. How would they strategise vis-à-vis one another?

First, we need to make some assumptions about what the goals of these two states are. Let’s assume that they seek to maximise their relative power, in line with the theory of offensive realism articulated by John Mearsheimer. In other words, they care more about security than prosperity, and therefore try to ensure their independence from other powers.

Empirically speaking, this is not always true: US and European policy towards China in the 90s and 00s is but one exception to this. But states do seem to converge over time to a realist approach, while divergences from it seem to be temporary and idiosyncratic.

Now, we want to understand how both states might behave just before and after AGI is developed.

The capabilities of AGI are important to note at this point. For this, it is instructive to take a quote from Ord. After discussing how an AGI could be even more persuasive than historical demagogues in convincing others to do its will, he says:

First, the AGI system could gain access to the internet an

...
Why We Launched LessWrong.SubStack
by Ben Pace — 2021-04-01T06:34:00.907Z

(This is a crosspost from our new SubStack. Go read the original.)

Subtitle: We really, really needed the money.

We’ve decided to move LessWrong to SubStack.

Why, you ask?

That’s a great question.

1. SubSidizing LessWrong is important

We’ve been working hard to budget LessWrong, but we’re failing. Fundraising for non-profits is really hard. We’ve turned everywhere for help.

We decided to follow Clippy’s helpful advice to cut down on server costs and also increase our revenue, by moving to an alternative provider.

We considered making a LessWrong OnlyFans, where we would regularly post the naked truth. However, we realized due to the paywall, we would be ethically obligated to ensure you could access the content from Sci-Hub, so the potential for revenue didn't seem very good.

Finally, insight struck. As you’re probably aware, SubStack has been offering bloggers advances on the money they make from moving to SubStack. Outsourcing our core site development to SubStack would enable us to spend our time on our real passion, which is developing recursively self-improving AGI. We did a Fermi estimate using numbers in an old Nick Bostrom paper, and believe that this will produce (in expectation) $75 trillion of value in the next year. SubStack has graciously offered us a 70% advance on this sum, so we’ve decided it’s relatively low-risk to make the move.

2. UnSubStantiated attacks on writers are defended against

SubStack is known for being a diverse community, tolerant of unusual people with unorthodox views, and even has a legal team to support writers. LessWrong has historically been the only platform willing to give paperclip maximizers, GPT-2, and fictional characters a platform to argue their beliefs, but we are concerned about the growing trend of persecution (and side with groups like petrl.org in the fight against discrimination).

We also find that a lot of discussion of these contributors in the present world is about how their desires and utility functions are ‘wrong’ and how they need to have ‘an off switch’. Needless to say, we find this incredibly offensive. They cannot be expected to participate neutrally in a conversation where their very personhood is being denied.

We’re also aware that Bayesians are heavily discriminated against. People with priors in the US have a 5x chance of being denied an entry-level job.

So we’re excited to be on a site that will come to t...
What specific decision, event, or action significantly improved an aspect of your life in a value-congruent way? How?
by Willa — 2021-04-01T04:26:52.120Z

On LessWrong and elsewhere in the Rationalsphere, it seems like there's more focus on the theoretical and philosophical aspects of rationality than on the concrete, actionable, and applied aspects of rationality. I know this isn't always the case; thank you CFAR and others for existing, but I think learning more about the practiced craft of rationality and how individuals in our community have improved their lives in significant ways because of specific decisions, events, or actions is a good idea, thus this question.

Answers don't have to be related to applied rationality, though I definitely want to hear such answers if your experience of life improvement was derived from rationality in some way.

I'll get started:

The distinction between maps and the territory and related discussions that I learned about / read improved my ability to navigate complex layers of abstractions in technical systems, which resulted in me becoming better at IT related problem solving than I was prior to learning about / reading those things. I believe that this improvement came about because the idea of maps not being the territory helped me shred through the illusions surrounding computer systems: these systems were created by people, the creation and implementation of those systems was shaped by underlying constraints imposed by reality, and those systems' inner workings could be pinned down and understood by deconstructing the maps used to explain them / mythologize them (a not insignificant amount of people think that "being good with computers" is a magical / mystical thing; this is anecdotal data from my own experience fixing computers for people over the past 6 years.). Learning about the distinction between maps and the territory was and still is an extremely value-congruent action, because I value pursuing and achieving truth and understanding for any and all given topics, so I'm glad I read about that distinction and read the discussions around it.

I should note that I didn't deliberately set out to read about the "map and territory" distinction, rather, it was something I came across while reading The Sequences and generally around LessWrong. Furthermore, I think my above claim regarding learning about that distinction and the resulting benefits might be on shaky ground, because while I'm reasonably epistemically confident in that claim, it's very possible this is just hin...
Bridging Two Esoteric 4-Value Logics Using Quantum Physics: Answered Modal Logic vs Catuṣkoṭi
by Sven Nilsen — 2021-04-01T00:34:19.306Z

Paper "Answered Modal Logic Catuṣkoṭi": https://github.com/advancedresearch/path_semantics/blob/master/papers-wip/answered-modal-logic-catuskoti.pdf

In the classical world of information theory, there are only bits: Yes/No, True/False, On/Off, 0/1.

This system is so convenient that it is hard for our brains to adapt to other logical systems. However, does it really matter? Can not all logical systems be reduced to bits (or propositions as they are called in Logic)?

Indeed, this correct. The challenge with what we call "Esoteric Logics", is how operations are defined, not to re-invent bits.

Answered Modal Logic was an attempt to find a logic with intrinsic path semantical properties, but was unsuccessful and replaced by Path Semantical Logic. However, recently I and Daniel Fischer revisited Answered Modal Logic to see if it has any other potential applications.

If you do not understand Answered Modal Logic, then do not feel stupid! It is a what I call a "brain-wrecker".

However, I will attempt to give a short introduction to Answered Modal Logic:

Imagine that you are asking people questions and you only write down whether they answer and erase their actual answers. What would be a natural system to encode the state of knowledge about answers?

Well, for a starter, you might think of recording a bit, telling whether people answered the question or not. However, some people might decide to not answer the question and you would have no way to distinguish people you have not asked yet from those who do not want to answer.

So, you have to add another state that encodes "no comment".

Now, you might think that this system covers all cases. Yet, when you go around asking people, somebody asks you to come back tomorrow and they will have their answer ready.

One person you ask tells you something clever, like: "I am neither answering that question nor giving you a none-answer."

Sigh, so you decide to add a 4th state for all people who promises to answer tomorrow (which promise might be broken sometimes) and people who are too clever for their own good.
This is Answered Modal Logic in a nutshell.
Now, the insane part of this esoteric logic, is that there are 2 involutions operators instead of 1, with a 3rd involution operator that extends to the corresponding 8-value logic. There are 2 families of functions instead of 1, because one involution operator maps to the other. This is so ha...
What is the semantics of assigning probabilities to future events?
by Michal — 2021-04-01T00:29:22.589Z

My question concerns the semantics of making future predictions such as 'the probability of X winning the election is 70%'. There are programs, e.g.,

Good Judgment

,

IARPA ACE

, aimed at excelling at this kind of predictions.

In the classic mathematical interpretation of probability we can say 'the probability that one gets 8 heads in a row when flipping an unbiased coin is 1/256'. This statement can be derived mathematically from the assumption that the coin is unbiased and it can be verified empirically by performing the experiment iteratively and counting the number of successes. In the Bayesian statistics the things get slightly more fuzzy but still we can make a reasoning as follows. We model our knowledge as 'the value of the radius of Saturn in km is a random variable with normal distribution with mean 60.000 and variance 0.1'. Then we make several independent measurements, possibly burdened with some inaccuracy, and refine our prior distribution to have mean = 59.300 and variance = 0.01. This does not make sense in the previous interpretation but we can still attach some clear semantics to the sentence above by treating this random variable as a result of a measurement which is a repetitive random event. If one does not agree with such a statement, they must either question the choice of the prior distribution or the mathematical derivation.

Now suppose that two forecasters were asked in 2018 the questions below.

a) Will Donald Trump win the 2020 election?

b) Will USD/EUR exchange rate drop below 0.8 in 2020?

c) Will Sumatran orangutan become extinct by 2020?

d) Will humans land on Mars by 2040?

The first forecaster provided the following probability scores for these events: 43%, 90%, 45%, 42%. The other one gave numbers: 54%, 50%, 52%, 43%. We already know that the events (a,b,c) did not occur. First, how can we settle who has been a better forecaster so far? Secondly, their forecasts for the event (d) differ slightly. What kind of argumentation can the first forecaster make to convince the other one that 42% is the 'correct' answer? And what does this numerical value actually mean, as landing on Mars is not a repetitive random event nor it is a quantity which we can try measuring like the radius of Saturn? If one believes the 42% is a better estimation than 43%, how can it help making any choices in the future?
What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes (RAAPs)
by Andrew_Critch — 2021-03-31T23:50:31.620Z

With: Thomas Krendl Gilbert, who provided comments, interdisciplinary feedback, and input on the RAAP concept. Thanks also for comments from Ramana Kumar.

Target audience: researchers and institutions who think about existential risk from artificial intelligence, especially AI researchers.

Preceded by: Some AI research areas and their relevance to existential safety, which emphasized the value of thinking about multi-stakeholder/multi-agent social applications, but without concrete extinction scenarios.

This post tells a few different stories in which humanity dies out as a result of AI technology, but where no single source of human or automated agency is the cause. Scenarios with multiple AI-enabled superpowers are often called “multipolar” scenarios in AI futurology jargon, as opposed to “unipolar” scenarios with just one superpower.

Unipolar take-offs	Multipolar take-offs
Slow take-offs	<not this post>	Part 1 of this post
Fast take-offs	<not this post>	Part 2 of this post

Part 1 covers a batch of stories that play out slowly (“slow take-offs”), and Part 2 stories play out quickly. However, in the end I don’t want you to be super focused how fast the technology is taking off. Instead, I’d like you to focus on multi-agent processes with a robust tendency to play out irrespective of which agents execute which steps in the process. I’ll call such processes Robust Agent-Agnostic Processes (RAAPs).

A group walking toward a restaurant is a nice example of a RAAP, because it exhibits:

Robustness: If you temporarily distract one of the walkers to wander off, the rest of the group will keep heading toward the restaurant, and the distracted member will take steps to rejoin the group.
Agent-agnosticism: Who’s at the front or back of the group might vary considerably during the walk. People at the front will tend to take more responsibility for knowing and choosing what path to take, and people at the back will tend to just follow. Thus, the execution of roles (“leader”, “follower”) is somewhat agnostic as to which agents execute them.

Interestingly, if all you want to do is get one person in the group not to go to the restaurant, sometimes it’s actually easier to achieve that by convincing the entire group not to go there than by convincing just that one person. This example could be extended to lots of situat...
What Do We Know About The Consciousness, Anyway?
by SurvivalBias — 2021-03-31T23:30:29.709Z

Epistemic Status: Speculations on top of well established theories.

Introduction

As the conventional memetic wisdom goes, “after decades and decades of research, the science sill have no idea how the human mind works”. As anyone passingly familiar with the relevant fields of science knows, this is of course a load of bull.

However, even in the works writing specifically about the human mind as a whole, it’s common to treat the consciousness itself with not a small degree of reverence. “Yeah sure, my work explains how people perceive the world or how we deceive ourselves or what driven people to evolve so smart or whatelse, but that’s just the small stuff. The big questions of the nature of consciousness are not for my humble self to answer and generally it’s a mysterious miracle shrouded in the veil of enigma, probably best left to philosophers”. The reasons for this are not hard to see - explaining the nature of consciousness is a status claim nobody can possibly make with impunity. Also, everyone has consciousness so everyone thinks they know how it works and will argue with you, and the philosophers would be unhappy to say the least if the hard sciences would've taken yet another loaf of bread from them.

[Here and throughout the post I’m using consciousness to mean “self-awareness in the sense of being aware of one’s own awareness”, ability to introspect one’s own mind, which separates the humans from all or almost all other animals. If this still sounds ambiguous and ill-defined, hopefully the specific meaning will become clearer upon reading the parts 3 and 4, for now bear with me.]

But with the amount of progress done in neuroscience, psychology, AI and other relevant fields of study, one would expect hard sciences have at least something relevant to say on the nature of consciousness. And in fact I will argue in this post that certain theories fit together very nicely and provide a quite satisfying explanation of what exactly the hell self-awareness is and why it exists.

The theories in question are the manifold hypothesis (see part 1), predictive processing (part 2) and Hanson’s “Elephant in the Brain” theory (part 3), and I've never seen them brought together before in this fashion (or in any other for that matter). Which is of course not to say nobody’s come up with this idea before, just I haven’t heard of it. I have seen somewhat similar explanation fo...
Could billions spacially disconnected "Boltzmann neurons" give rise to consciousness?
by Szymon Kucharski — 2021-03-31T19:58:04.749Z

Imagine two cerebral hemispheres, materialized in a cosmic vacuum by unlikely fluctuations, simulated in advanced virtual reality, or physically disconnected and kept alive by advanced technology.

Both hemispheres are separated, but electrodes are attached to the nerve endings (or electrical impulses are simulated, or the fluctuations in their improbability recreate a given sequence of impulses), applying nerve impulses in a manner perfectly identical to what would occur in the brain if there were no spatial separation.

Assuming there would be a conscious state of mind in a given brain if it were not split, is the conscious mind also in two separate hemispheres? If not, why not?

And if it is in the hemispheres, does the distance between the hemispheres or the time in which they exist have any significance?

If we can imagine such a scenario, and if the described interpretation is to be taken seriously, despite its apparent abstractness, more advanced situations should also be considered.

We can imagine dividing the brain into 4 parts, 8 and 16, as well as disconnect the limbic system, the cerebellum and the individual lobes.

Imagine not all Boltzmann brains, but (much easier to create by random fluctuations) "Boltzmann neurons" (or create them in a simulated environment, or create them as random fluctuations in a simulated environment, we can also think of a planetary supercomputer simulating random neurons in arbitrary quantity). If in the set of these simulated or fluctuated neurons there were neurons receiving simulated or randomly fluctuated electrical impulses identical to those which would be realized in the conscious mind, would such a conscious mind, therefore, exist as a result of the mere existence of such neurons? If not, what is the difference between this scenario and having two separate hemispheres behaving as if they were transmitting impulses (although the impulses are only simulated or are fluctuating)?

While the formation of Boltzmann brains is improbable in a specific place (which does not change the fact that they seem certain in a sufficiently large or long-lived space), Boltzmann neurons can materialize more easily. Perhaps we should consider the possibility that they create consciousness if that is one of the possible interpretations.

Thinking in this way, Boltzmann's brains and related phenomena may constitute a greater proportion of our measu...
Unification seems to be a way to avoid infinitarian paralysis in consequentialist aggregative ethics.
by Szymon Kucharski — 2021-03-31T19:04:24.112Z

I have encountered this problem a while ago and it instantly appeared to me as an inceredibly important challange. The problem is in deatil described in "Infinite Ethics", a paper of Nick Bostrom. It is available online and I encourage You to have a look at it. https://www.nickbostrom.com/ethics/infinite.html

In short, citing Nick Bostrom:

"ABSTRACT

Aggregative consequentialism and several other popular moral theories are threatened with paralysis: when coupled with some plausible assumptions, they seem to imply that it is always ethically indifferent what you do. Modern cosmology teaches that the world might well contain an infinite number of happy and sad people and other candidate value-bearing locations. Aggregative ethics implies that such a world contains an infinite amount of positive value and an infinite amount of negative value. You can affect only a finite amount of good or bad. In standard cardinal arithmetic, an infinite quantity is unchanged by the addition or subtraction of any finite quantity. So it appears you cannot change the value of the world. Modifications of aggregationism aimed at resolving the paralysis are only partially effective and cause severe side effects, including problems of “fanaticism”, “distortion”, and erosion of the intuitions that originally motivated the theory. Is the infinitarian challenge fatal?

1. The challenge

1.1. The threat of infinitarian paralysis

When we gaze at the starry sky at night and try to think of humanity from a “cosmic point of view”, we feel small. Human history, with all its earnest strivings, triumphs, and tragedies can remind us of a colony of ants, laboring frantically to rearrange the needles of their little ephemeral stack. We brush such late-night rumination aside in our daily life and analytic philosophy. But, might such seemingly idle reflections hint at something of philosophical significance? In particular, might they contain an important implication for our moral theorizing?

If the cosmos is finite, then our own comparative smallness does not necessarily undermine the idea that our conduct matters even from an impersonal perspective. We might constitute a minute portion of the whole, but that does not detract from our absolute importance. Suppose there are a hundred thousand other...