trentmkelly/LessWrong-43k huggingface dataset:
The SIA population update can be surprisingly small
by Stuart_Armstrong — 2021-07-08T10:45:02.803Z

With many thanks to Damon Binder, and the spirited conversations that lead to this post, and to Anders Sandberg.

People often think that the self-indication assumption (SIA) implies a huge number of alien species, millions of times more than otherwise. Thought experiments like the presumptuous philosopher seem to suggest this.

But here I'll show that, in many cases, updating on SIA doesn't change the expected number of alien species much. It all depends on the prior, and there are many reasonable priors for which the SIA update does nothing more than double the probability of life in the universe[1].

This can be the case even if the prior says that life is very unlikely! We can have a situation where we are astounded, flabbergasted, and disbelieving about our own existence - "how could we exist, how can this beeeeee?!?!?!?" - and still not update much - "well, life is still pretty unlikely elsewhere, I suppose".

In the one situation where we have an empirical distribution, the "Dissolving the Fermi Paradox" paper, the effect of the SIA anthropics update is to multiply the expected civilization per planet by seven. Not seven orders of magnitude - just seven.

The formula

Let ρ∈[0,1] be the probability of advanced space-faring life evolving on a given planet; for the moment, ignore issues of life expanding to other planets from their one point of origin. Let f be the prior distribution of ρ, with mean μ and variance σ2. This means that, if we visit another planet, our probability of finding life is μ.

On this planet, we exist[2]. Then if we update on our existence we get a new distribution f′; this distribution will have mean μ′:

μ′=μ(1+σ2μ2).

To see a proof of this result, look at this footnote[3].

Define Mμ,σ2=1+σ2/μ2 to be this multiplicative factor between μ and μ′; we'll show that there are many reasonable situations where Mμ,σ2 is surprisingly low: think 2 to 100, rather than in the millions or billions.

Beta distributions I

Let's start with the most uninformative prior of all: a uniform prior over [0,1]. The expectation of ρ is ∫10ρdρ=1/2, so, without any other information, we expect a planet to have life with 50% probability. The variance is σ2=1/12.

Thus if we update on our existence on Earth, we get the posterior f′(ρ)=2ρ; the mean of this is 2/3 (either direct calculation or using M1/2,1/12=1+4/12=4/3).

Even though this change in expectation is multiplicativel...
Anthropics in infinite universes
by Stuart_Armstrong — 2021-07-08T06:56:05.666Z

When talking about anthropics, people often say things like "assume the universe is finite; weird things happen in infinite universes". I've myself argued that SSA breaks down when we encounter infinities; SIA breaks down sooner, when we encounter expected infinities.

You can formalise this informally[1] with the thought that:

In an infinite universe, anything can happen, no matter how unlikely: life must exist somewhere. So our existence doesn't tell us anything about life; its probability could be anything at all.

A superficially convincing argument; but not one you'd use for anything else. For instance, consider the following:

In an infinite universe, anything can happen, no matter how unlikely: if gravity didn't exist, somewhere it must seem to exist by shear chance. So our observation of gravity doesn't tell us anything about gravity; its probability could be anything at all.

I've argued before that anthropic questions are pretty normal. Why would we accept the reasoning in question 1, but reject it in question 2?

We shouldn't. We can deal with questions like 2 by talking about limits of probabilities in larger and larger spaces, or by discounting distant observations (similar to sections 2.3 and 3.1 in infinite ethica). So we might define conditional probabilities like P(X∣Y) in an infinite universe in the following way:

Let Prl(X∣Y) be the ratio of observers, within a large hypersphere of radius r centered on location l, that observe X and Y, relative to the proportion that observes Y. If this tends to a limit as r→∞, independently of l, then define that limit to be P(X∣Y).

Note that this definition works just as well for Y= "we observe the force of gravity to be blah" as with Y= "we exist".

Now, that definition might not be ideal (in particular, "radius" is not defined for relativistic space-time). No problem: different definitions of probability are asking different questions, and can lead to different anthropic probabilities, just as in the finite case.

I'll call these class of questions "SIA-limit questions", since they are phrased as ratios of observers, and dependent on how we use limits to define probability in infinite universes. They each lead to various "SIA-limit anthropic probability theories"; in most standard situations, these should reach the same answers as each other.

Yes, it's perfectly possible to formalise informally, and I en

...
A world in which the alignment problem seems lower-stakes
by TurnTrout — 2021-07-08T02:31:03.674Z

The danger from power-seeking is not intrinsic to the alignment problem. This danger also depends on the structure of the agent's environment.

In The Catastrophic Convergence Conjecture, I wrote:

But are there worlds where this isn't true? Consider a world where you supply a utility-maximizing AGI with a utility function.

The AGI is in a "separate part of the universe"; after the initial specification of the utility function, the left half of the universe evolves independently of the right half. Nothing you can do after specification can affect the AGI's half, and vice versa. No communication can take place between the two halves.

The only information you have about the other half is your utility. For simplicity, let's suppose you and the AGI have utility functions over universe-histories which are additive across the halves of the universe. You don't observe any utility information about the other part of the universe until the end of time, and vice versa for the AGI. That is, for history h,

uhuman(h)=uhuman(hleft)+uhuman(hright).

If the AGI uses something like causal decision theory, then it won't try to kill you, or "seek power" over you. The effects of its actions have no causal influence over what happens in your half of the universe. Your universe's evolution adds a constant term to its expected utility.

(Other decision theories might have it precommit to minimizing human utility unless it attains maximal AGI-utility from the left half of the universe-history, or some other shenanigans. This is beside the point I want to make in this post, but it's important to consider.)

However, the setup is still interesting because

Goodhart's law still applies: if you give the AGI an incomplete proxy objective, you'll get suboptimal true performance.
Value is still complex: it's still hard to get the AGI to optimize the right half of the universe for human flourishing.
If the AGI is autonomously trained via stochastic gradient descent in the right half of the universe, then we may still hit inner alignment problems.

Alignment is still hard, and we still want to get the AGI to do good things on its half of the universe. But it isn't instrumentally convergent for the AGI to seek power over you, and so you shouldn't expect an unaligned AGI to try to kill you in this universe. You shouldn't expect the AGI to kill other humans, either, since none exist in the...
Announcing My Free Online Course "Original Seeing With a Focus On Life"
by LoganStrohl — 2021-07-08T01:26:00.985Z

In May and June of 2021, I ran an online nature study course with about two dozen participants. I had three goals:

Help people create or deepen a personal connection with nature.

Learn how to run month-long online courses with over a dozen participants.

Test some hunches about the rationality material I’ve been working on.

One of my main hunches was that if you complete this course, you’ll find you’re able to study your own mind, or just about anything else, in the way a naturalist studies nature, with no further guidance. I obviously need more data, but the preliminary feedback looks promising.

A solo version of the course is now available for free through my website, thanks largely to support from the Long Term Future Fund.

If you want to get better at original seeing, I know of no better resource. I hope that some of you will try it out and tell me how it goes.
You should ask for a third-party audit of your cryopreservation
by oge — 2021-07-08T01:18:32.560Z

Hello (potential) cryonics customer, you should ask for third-party audit of your cryopreservation. This ensures that the cryonics providers aren't performing the procedure badly, say through negligence, ignorance, or something else.

Specifically, you should ask that the following be sent to an imaging specialist of your choice AFTER your brain has been preserved

An x-ray of your preserved brain, and
A needle biopsy of a few brain areas (say, the frontal cortex, parietal cortex, brain stem, basal ganglia, and thalamus)

The x-ray will roughly show if large parts of your brain were not preserved well. Micrographs of the biopsy samples will show in fine detail (tens of nanometers resolution) the quality of your preservation.

This information will give your loved ones peace of mind that you've maxed your chances of revival and it will also give other cryonics customers confidence that your cryoprovider is worth paying money to.

These auditing requirements come from a video by the Brain Preservation Foundation (BPF), Preservation Done Right, and from their article, Vitrifying the Connectomic Self (p. 5, 2nd para).
Disclaimer: I volunteer for the BPF, and am an ex-Alcor member.
How much chess engine progress is about adapting to bigger computers?
by paulfchristiano — 2021-07-07T22:35:29.245Z

(This question comes from a discussion with Carl Shulman.)

In this post I describe an experiment that I'd like to see run. I'm posting a $1,000 - $10,000 prize for a convincing implementation of these experiments. I also post a number of smaller prizes for relevant desk research or important corrections to this request.

Motivation

In order to understand the dynamics of the singularity, I'd like to understand how easy it is to improve algorithms and software.

We can learn something about this from looking at chess engines. It's not the most relevant domain to future AI, but it's one with an unusually long history and unusually clear (and consistent) performance metrics.

In order to quantify the quality of a chess engine, we can fix a level of play and ask "How much compute is needed for the engine to play at that level?"

One complication in evaluating the rate of progress is that it depends on what level of play we use for evaluation. In particular, newer algorithms are generally designed to play at a much higher level than older algorithms. So if we quantify the compute needed to reach modern levels of play, we will capture both absolute improvements and also "adaptation" to the new higher amounts of compute.

So we'd like to attribute progress in chess engines to three factors:

Better software.
Bigger computers.
Software that is better-adapted to new, bigger computers.

Understanding the size of factor #1 is important for extrapolating progress given massive R&D investments in software. While it is easy to separate factors #1 and #2 from publicly available information, it is not easy to evaluate factor #3.

Experiment description

Pick two (or more) software engines from very different times. They should both be roughly state of the art, running on "typical" machines from the era (i.e. the machines for which R&D is mostly targeted).

We then carry out two matches:

Run the old engine on its "native" hardware (the "old hardware"). Then evaluate: how little compute does the new engine need in order to beat the old engine?
Run the new engine on its "native" hardware (the "new hardware"). Then evaluate: how much compute does the old engine need in order to beat the new engine?

With some effort, we can estimate a quantitative ratio of "ops needed" for each of these experiments. For example, we may find that the new engine is able to beat the old engine using only 1% of the "ol...
Agency and the unreliable autonomous car
by Alex Flint — 2021-07-07T14:58:26.510Z

Financial status: This is independent research, now supported by a grant. I welcome financial support.

Epistemic status: I believe ~85% that the technical argument presented in this piece is correct.

Outline

This is my attempt to explain the basic Löb situation with the 5-and-10 problem.

This post considers an autonomous car choosing between a long route and a slow route, with the goal of minimizing the time to its destination.

If the autonomous car makes its decisions using a certain seemingly-reasonable algorithm based on proof search, this post shows that it may nonsensically take the slow route.

This is the 5-and-10 problem phrased in terms of autonomous driving.

Parable of the unreliable autonomous car

A long time ago, in a far away place, there was an autonomous car.

This particular autonomous car was programmed to use the agent model in its reasoning. That is, it was programmed to model the world as consisting of an agent and an environment, the agent being itself, and the environment being a network of roads that it would navigate each day. This is how the autonomous car was programmed to model the world.

Beyond using the agent model, this autonomous car had been given, by its designers, an accurate model of its own behavior. It could tell you how its own decision algorithm would behave under any set of circumstances. It could tell you this without being faced with those actual circumstances because it had a model of itself, and it could answer questions based on that model.

One day, a passenger got into the autonomous car and gave it a destination. There were two ways for the autonomous car to get there: a fast route, which would get it there at 1pm, and a slow route, which would get it there at 2pm. The goal of the autonomous car was to get to its destination as quickly as possible. This parable is about the reasoning followed by the car in deciding which of these two routes to take.

Now in this far-away place, autonomous cars were designed by a strange cadre of alien engineers, and thus they were programmed with strange algorithms. This particular autonomous car was programmed to decide among routes using the following procedure:

Make a list of the possible routes

Make a list of the possible arrival times

Make a list of all possible logical sentences of the form IF route 1 is taken THEN I will arrive at such-and-such a time AND IF rout

...
Intelligence without Consciousness
by Andrew Vlahos — 2021-07-07T05:27:20.359Z

I sleep-talk. In fact, I don't just talk in my sleep, I can carry on a conversation. Advice and insight I have is much lower quality than when I'm awake, but I'm mostly coherent. Once someone recorded a conversation with me, and it was pretty funny. For example: "A lot of people like Batman. We should buy stock in Batman".

I'm not conscious, though. I'm not self-aware. If I have qualia, it's a very tiny amount of qualia. I definitely wouldn't wonder "do I have qualia?" If I didn't keep waking up, there wouldn't be a moral reason for others not to treat me how they find most convenient.

What does this mean? It means that Nonsentient Optimizers and Nonperson Predicates are definitely possible. There is some threshold of intelligence and/or self-awareness needed to change a neural net into a person.

(Lucid dreaming is a separate issue. I'm talking about something that happens even when not lucid dreaming.)

Also, people in the comments keep saying maybe I did have consciousness and just don't remember it. No: the conversation quality is sort of like this. When people talk to me, my response is usually connected to what they said, but I'm definitely not conscious.
A second example of conditional orthogonality in finite factored sets
by DanielFilan — 2021-07-07T01:40:01.504Z

Yesterday, I wrote a post that gave an example of conditional non-orthogonality in finite factored sets. I encourage you to read that post first. However, I'm kind of dissatisfied with it because it doesn't show any interesting cases of conditional orthogonality (despite the title seeming to promise that). So I'd like to show you one today.

First, let's imagine that Alice is a person who has some height. Bob and Charlie both measure her height, and take note of the measurements. However, their measuring instruments have independent sources of error, such that neither gets exactly the right answer. In this world, Bob's measurement is not independent of Charlie's measurement, because they'll both be pretty close - the error isn't that big. However, once you know Alice's height, they will be independent, because given that knowledge, learning Bob's measurement doesn't tell you anything about Charlie's measurement. Below, we'll see how we can formalize that in the language of finite factored sets.

Our finite factored set will be the set of tuples (a,ϵb,ϵc), and the basis factors will be A, which partitions the tuples by their value of a, EB, which partitions them by their value of ϵb, and EC, which partitions them by their value of ϵc. These represent Alice's height, and the error that Bob's and Charlie's machines respectively introduce. Note that you might have imagined we'd have the basic factors as Alice's height, Bob's measurement, and Charlie's measurement, but then these wouldn't be probabilistically or logically independent, and so would violate the assumptions that go into modelling things as finite factored sets. [footnote 1]

Next, we'll define the function b(a,ϵb,ϵc)=a+ϵb, which gives the height that Bob measures, and the partition B which groups tuples with the same value of b together. Similarly, we'll define c(a,ϵb,ϵc)=a+ϵc, which tells us the height that Charlie measures, and the partition C that groups tuples together by their value of c.

What's the history of B? Well, it's the smallest set of factors such that if we know the 'value' of the factors, then we know the 'value' of B, and that's {A,EB}. Similarly, the history of C is {A,EC}. So B's and C's histories have A in common, and therefore aren't orthogonal.

Now, let's consider the set A2={(a,ϵb,ϵc)∣a=2}, which represents the worlds where Alice is 2 metres tall, and check out the conditional histori...
Chess and cheap ways to check day to day variance in cognition
by KPier — 2021-07-07T00:08:46.668Z

Lately I've been playing chess. I'm not very good at it. I started because our four-year-old got into it, and I preferred not to lose to her before she had earned it. I play online on chess.com, which has instantaneous matchups with strangers at approximately your level; I mostly play 5-minute timed games, where each player has a 5-minute clock for all of their moves and loses if they run out of time.

Doing this every day for a couple of months made some interesting patterns obvious. Firstly, I have good days and bad days, rather than just winning or losing mostly at random at my competence level.
Bad days are predicted by some of the things you'd expect, like poor sleep, high stress, or emotional distraction, but sometimes I have a bad-day-as-measured-by-chess when I wouldn't have predicted it, or (more rarely) a strikingly good day when I thought I was likely sleep deprived.
Secondly, playing chess on a good day versus on a bad day allows a lot of access to the feeling of being slightly smarter or slightly stupider than usual. In general, I think my ability to do high-level cognition varies a lot day to day and even within a day, but it's often really hard to tell if I'm blocked on a problem because I'm a little low on ability to think about it; usually, a problem will be different from other problems in a bunch of ways, and it's hard to tell how much of the difficulty is that I'm fatigued/distracted/whatever.
With chess, you're basically solving the same category of problem every time, with instantaneous feedback about whether your solution was any good, so I get unusually good felt-sense of where I would normally have seen something, but this time missed it, or alternatively where options and counteroptions are coming to me unusually quickly and have unusual depth and clarity.
Thirdly, having a good day at chess seems pretty strongly associated with having a good day at high-cognition-requiring tasks in general. In fact, it seems like a better predictor of this than asking myself 'do I expect to be good at cognition today?'.
On days when I do well at chess, I'm also likely to succeed at reading a complicated paper I previously bounced off because it was too hard, or at writing an article that has been blocked on loading all of it into working memory, or at having productive new thoughts if I dwell on an old problem for thirty minu...
Relentlessness
by Alicorn — 2021-07-06T20:27:09.527Z

There’s not a great way to convey the merciless relentlessness of having a child who insists on continuing to exist and want and need regardless of how much sleep you got, how sick you are, how many times you have already read that book, how tired your arms or how aching your feet, how hungry or sweaty or needy-for-cognition you’ve gotten, how much or little support happens to be available that day. It’s a lot.

...

Parents aren’t better at parenting tasks because of magic or even because they responsibly read the entire parenting manual, they’re better at them because they are forced to practice way way way more than anyone would naturally choose to practice any such tasks (and accordingly your skills will be uneven depending on how you divide those tasks).

(from a tumblr post I wrote about having kids)

Why is immersion the best way to learn a language?

I submit that it is because you do not get to stop.

If you were in, say, Java, then you probably would pick up Javanese as long as you did things reasonably aimed at continuing to be immersed in Javanese (that is, not immediately finding the nearest English-speaker and latching onto them, or adopting a convenient orphan and teaching them English, or buying a ticket to Australia). In spite of the fact that this strategy does not necessarily draw on anything we know about deliberate practice, or language education, it would still probably work. It's how everybody learns their first language, and in that case it basically always works, because babies really can't do anything else about it since they don't already speak anything else. To communicate at all, a very basic human need, you have to match the standard of other talking entities around you, and you will not stop having to do this, so eventually you will.

Most things are not like this, or they are like this but not enough for it to be a good idea to try to learn them this way. If you are in the ocean, and you cannot stop being in the ocean, this is actually a terrible way to learn to swim and an even worse way to learn about sharks. If you are a subsistence farmer, and you cannot stop being a subsistence farmer, you might learn a ton about your plot of land, but also you might have bad weather one year and starve. If you are in a typical American math class, and you cannot stop being in a typical American math class, you...