trentmkelly/LessWrong-43k huggingface dataset:
Robin Hanson asks "Why Not Wait On AI Risk?"
by Gunnar_Zarncke — 2022-06-26T23:32:19.436Z
Link: https://www.overcomingbias.com/2022/06/why-not-wait-on-ai-risk.html

UPDATE: This is an accidental duplicate. Please go to the first posting here.

Since the AI-Foom Debate Robin Hanson has been quiet for a while but now:

I feel I should periodically try again to explain my reasoning, and ask others to please help show me what I’m missing.

After having analyzed multiple widely expressed AI concerns from an economical angle and comes to the conclusion:

Similar concerns have often been expressed about allowing too many foreigners to immigrate into a society, or allowing the next youthful generation too much freedom to question and change inherited traditions. Or [many other comparable concerns].

Many have deep and reasonable fears regarding big long-term changes. And some seek to design AI so that it won’t allow excessive change. But this issue seems to me much more about change in general than about AI in particular.

There is much more in it but take this as a TL;DR.
Training Trace Priors and Speed Priors
by Adam Jermyn — 2022-06-26T18:07:08.746Z

Thanks to Evan Hubinger for suggesting this idea.

Training Trace Priors are priors over boolean circuits which examine the outputs of gates on samples from the training distribution, typically for purposes of steering models away from having components that were never tested during training. The one I like to think about is the One-Gate Trace Prior (OGT Prior), which penalizes circuits if there are gates with constant outputs during training.

Priors like this might be a way to make deception less likely. The intuition is that deception involves a decision of whether or not to defect, which at its simplest shows up as a gate that always outputs False during training but outputs True in deployment.

This post explores the relationship between the OGT Prior and Speed Priors, which I’ve operationalized with the circuit depth prior.

Memorizing Constants

Speed Priors incentivize memorization. In particular, any result that is constant on the training distribution and contributes to the circuit’s depth should just become a constant input into the circuit.

The OGT Prior does something similar. Any output that is constant on the training distribution is treated as suspicious, and so should be pre-computed and turned into a constant input instead.

To see where these produce the same result, consider a physics simulator with N particles interacting pairwise subject to some conditional check (perhaps on the distance between particles). If only some pairwise interactions occur in the training set then both the Speed Prior and the OGT Prior will only compute the interactions between pairs actually encountered in training.

To see where these differ, note that the Speed Prior has no problem with computing constants so long as these do not contribute to the overall circuit depth. For instance, consider a circuit of depth 10 can contain constant calculations of depth 9 that feed straight into the final output. The Speed Prior has no reason to eliminate these, whereas the OGT Prior will turn the results of these calculations into constant inputs.

Lookup Tables

The Speed Prior incentivizes circuits to use lookup tables. Specifically, the Speed Prior produces circuits that memorize the results of expensive calculations in lookup tables containing all instances encountered in training.

The OGT Prior is perfectly fine with circuits being very deep, and so has no need to tabulate expensive cal...
My current take on Internal Family Systems “parts”
by Kaj_Sotala — 2022-06-26T17:40:05.750Z

I was recently asked how literal/metaphorical I consider the Internal Family Systems model of your mind being divided into “parts” that are kinda like subpersonalities.

The long answer would be my whole sequence on the topic, but that’s pretty long and also my exact conception of parts keeps shifting and getting more refined through the sequence. So one would be excused for still not being entirely clear on this question, even after reading the whole thing.

The short answer would be “it’s more than just metaphorical, but also not quite as literal as you might think from taking IFS books at face value”.

I do think that there are literally neurological subroutines doing their own thing that one has to manage, but I don’t think they’re literally full-blown subminds, they’re more like… clusters of beliefs and emotions and values that get activated at different times, and that can be interfaced with by treating them as if they were actual subminds.

My medium-length answer would be… let’s see.

There’s an influential model in neuroscience called global workspace theory. It says that the brain has a thing called the “global workspace”, which links together a variety of otherwise separate areas, and its contents corresponds to that what you’re currently consciously aware of. It has a limited capacity so you’re only consciously aware of a few things at any given moment.

At the same time, various subregions in your brain are doing their own things, some of them processing information that’s in the global workspace, some of them observing stuff from your senses that you’re currently not consciously aware of. Like you’re focused on a thing, then there’s a sudden sound, and some auditory processing region that has been monitoring the sounds in your environment picks it up and decides that this is important and pushes that sound into your global workspace, displacing whatever else happened to be there and making you consciously aware of that sound.

I tend to interpret IFS “parts” as processes that are connected with the workspace and manipulate it in different ways. But it’s not necessarily that they’re really “independent agents”, it’s more like there’s a combination of innate and learned rules for when to activate them.

So like, take it when an IFS book has a case study about a person with a “confuser” part that tries to distract them when they are thinking about something unple...
A Quick Ontology of Agreement
by ravedon — 2022-06-26T17:39:25.315Z

Adapted from Herman Kahn's Introduction to Can We Win in Vietnam, Armbruster, F.E., et. al. Praeger: New York, 1968.

This is a quick post about an ordered sense of agreement. I find it serves two useful purposes:

It gives a practical framework for how to achieve better agreement between parties in apparent conflict, so it is a tool for both conflict resolution and for problem-solving, and
It replaces the binary structure of agree/disagree with a looser structure that allows people to exist in conceptual tension without feeling like they are involved in interpersonal conflict.

Now a description of the levels, followed by a brief discussion.

First Order Agreement

The conventional understanding of agreement. All parties agree to the characterization of a resolution to the point of contention, what Kahn called "agreement on the substance of an issue". Scope is a particularly important characteristic to achieve first order agreement. What does the agreement cover and what does it not cover?

In formal legal or institutional contexts first order agreement is instantiated in text where parties to the agreement indicate their acceptance of the resolution by signature or by the signature of an authorized agent. The agreement is ratified.

Second Order Agreement

Agreement on why there is no first order agreement. All parties agree to the characterization of what precludes a resolution to the point of contention. Kahn called this agreeing on what the disagreement is about.

Third Order Agreement

Agreement on why there is no second order agreement; what is conventionally called agreeing to disagree. All parties agree that there is a disagreement, but cannot agree on the characterization of that disagreement. Generalizing a bit from Kahn, who saw this as largely the interference of emotion in rational discussion, one can think of third order agreement as the belief by one party (possibly both) that the other party is either unwilling or incapable of articulating their own position. For example, asserting "You don't know what you're talking about" indicates third order agreement even if it is hostile and hence counter-productive. Accusing one party of trolling or arguing in bad faith is also third order agreement.

Fourth Order Agreement

Agreement that no agreement appears possible, including whether there is a disagreement. Kahn described this as the asserti...
Seven ways to become unstoppably agentic
by  — 2022-06-26T17:39:08.424Z

Edit in September 2022: I wrote a response to this post here, in which I lay out some concerns, some unexpected costs of taking "agentic" actions for me (and mistakes I’ve made), and things I’ve changed my mind on since writing this post.

Nearly all of the best things that have happened to me in the last year have been a result of actively seeking and asking for those things. I have slowly been developing the skill of agency.

By ‘agency’ I mean the ability to get what you want across different contexts – the general skill of coming up with ambitious goals [1] and actually achieving them, whatever they are. In my experience, I’ve found this is learnable.

Neel Nanda explains the concept of agency really well in

this blog post

, so I won’t repeat it here. Instead, I’ll focus on how to learn it.

It’s worth acknowledging that agency is often socially discouraged in different minority groups. Speaking from my own experience, I used to feel shameful around being agentic – I associated it with being entitled or ‘too much’ or bossy – which seems to be a phenomenon that many women experience. [2] I sometimes still have a voice in my head saying, “who do you think you are???” But, fortunately, I’m paying less and less attention to it.

In no particular order, here are seven ways I’ve found to become unstoppably agentic:

1. Figure out what you need, figure out who can help you get it, ask them for it

There will (nearly) always be people who can help you achieve your goals better. Find them and ask them to help you. They might say yes!

If you don’t ask, the answer is always no.

When I was self-studying for my A-Levels (UK high school exams), I reached out to an EA who I’d never met, asking to chat about why I was finding self-studying so hard. We ended up having a call, which I found extremely helpful. At the time, I had a huge

ugh-field

around my Anki flashcards, which he completely helped me clear up. I then started sending him my daily plan (after checking this was ok), which turned out to be a really valuable accountability system. He gave support and encouragement countless times over the few months. These daily check-ins were the best thing[3] that could have happened to my A-Levels, and led me to feel way more supported throughout my time self-studying.

Having a low bar for asking for help is a learnable mindset. Now, whenever ...
Formalizing Deception
by JamesH — 2022-06-26T17:39:01.390Z

An attempt at formalizing deception, seeing what goes right, what goes wrong, and what compromises have to be made.

Introduction

In a police interrogation one of the key challenges is working out whether a suspect is being deceptive. In a game of poker, one of the main goals is figuring out whether your opponents are being deceptive. But what are we talking about when we talk about deception, like, mathematically? I will propose a definition and give an example that illustrates how this definition operates.

Interrogation Investigation

Take a police officer, Alice, who is interrogating a suspect, Bob, in order to determine whether or not he is guilty of murder. Alice can either convict Bob or let him go, and Bob is either guilty or innocent. The payoff matrix is as follows:

Bob:

Alice:

Guilty	Innocent
Convict	(1,-1)	(-1,-1)
Let go	(-1,1)	(1,1)

Whether or not Bob is guilty, he would prefer to be let go; on the other hand, Alice only wants to convict if Bob is guilty, and she wants to let him go otherwise.

When Bob enters the room, it is predetermined that he is either guilty or innocent, Bob does not get to make a decision about this in advance. If Bob is guilty he will behave slightly differently under interrogation than if he isn't: he may have a weaker alibi, there may be inconsistencies in his story, he may exhibit physical symptoms associated with lying, but all of these things can also happen if Bob is innocent; sometimes innocent people have bad alibis, are inconsistent, and look nervous.

Now in order to define deception, we introduce Judy, an omniscient observer of the interrogation. Judy has the same goal as Alice, but unlike Alice, Judy also knows whether Bob is guilty or not. Further, Judy knows exactly how Bob's behaviors impact the likelihood of Alice convicting Bob (i.e. Judy can read both Alice's and Bob's minds). We call any subset of observations made by Alice deceptive if Judy would rather Alice make her decision to convict with this set of observations removed from her consideration (as though any memory of seeing these observations had been completely wiped from Alice's mind).

Formalization

Now let's try and formalize this story of deception. We'll describe Alice as a function from a series of observations to a probability space over actions:

F1n:Pn→ΔA

with the series of observations Pn given by:

Pn=(P1,P2,…,Pn),

(F1n)N...
Dust Theory vs Ruliad
by  — 2022-06-26T16:08:34.813Z

I want to get feedback and comments on the following:

Dust Theory (by Greg Egan or alternatively the Mathematical Universe Hypothesis by Tegmark)
Ruliad (by Wolfram)

I personally favour the Ruliad hypothesis, my intuition is that platonic structures can't be separate and will interfere with each other leading to the ruliad.

Another Question:

Do you think literal infinite minds (i.e., minds with access to infinite processing power) exist? The optimistic scenario is there is some kind of Infinite Mafia (or Government) that tries to enforce some kind of law. Perhaps this only applies to beings that are provably "fair" -- and humans do not satisfy the cut. I feel intuitively like there should be a variety of different infinite minds that exist and should be able to interfere with our universe (p-value around 20%).

Lately, I've started assigning higher probability to solipsism (around ~10%) and think perhaps I'm being simulated alone and this is some kind of PlayPlanet where other people are not real. (ex: not sure why the Holocaust is allowed if there are infinite minds that can interfere). Perhaps all humans are grown in their natural habitat like some kind of zoo.
In the ideal scenario, I think maybe they calculate the limit of what you will become if you could modify your brain (your CEV mind architecture if you will).

There are two aspects of interest from a CEV mind architecture:

what part of the idea space you choose to compress (see No Free Lunch Theorem)
how you want to modify the environment (the ruliad)

In Dust Theory, of course none of these questions matter much since no matter what you do, all realities with a mathematical description exist.
My cognitive inertia cycle
by MSRayne — 2022-06-26T15:49:17.661Z

I seem to have a kind of "mental momentum". Or maybe "cognitive inertia" is a better word.

When I feel as if I've lately been doing things worth doing, I feel curious and open and interested in things and seek more things to do, and have a good chance of actually doing them. But when I feel as if I haven't been doing things worth doing, or when my perception of the worth of the things I've been doing flips to them having been useless wastes of time (and thus, actively counterproductive as a result of the opportunity cost, since I could have been doing something useful, but stupid past-me was deluded and only now do I finally know the True Usefulness), then I lose momentum and start having a hard time perceiving anything as worth doing.

I become depressed, listless, closed off, unable to make decisions, and overwhelmed by the smallest actions. Generally at this point I lose my appetite (even if I am hungry, I can't bring myself to get something to eat - if it's really bad I don't even get coffee in the morning), and either walk in circles haranguing myself to just make a decision and do something already (if I'm in a sufficiently aroused state of mind), or climb into bed and try to dissociate enough that I don't feel as bad anymore, perhaps fleeing to daydreams about an elusive "someday when everything's better", wishing that night would come faster so that I could sleep and not have to endure living anymore for a while.

Eventually, miraculously, this goes away. I'm still not sure how, or what makes it go away, but my anxiety about decisions dissipates, the wall in my mind between intent and action becomes thinner, and I become able to do small tasks. Writing whatever I am thinking is usually the easiest - hence why I am doing it right now. This doesn't guarantee a total reversal - I usually still cannot come up with anything large to do, or the will to do it, but I can at least do this small thing.

I guess I end up passively scrolling, looking for dopamine, at this point, and often before the valley of apathy as well - but scrolling increases apathy due to the sensation of time being wasted and the vaguely addictive, energy-draining quality of it. Rarely, however, something I see on the internet sparks an idea sufficiently interesting that I want to try it out, and I start Doing Something, and begin to climb the hill.

Of course, when I am Doing Something, I have ...
How do poor countries get rich: some theories 
by  — 2022-06-26T10:41:36.027Z

When I’m talking to people who don’t know that much about development economics I quite often find myself trying to give a broad overview of the different theories about how countries get rich. I’m going to give an overview of these different theories in this post. In later posts I’ll dive into each of them more deeply, and hopefully this post will provide a good overview before going into the weeds in later posts. This post will also focus on poor countries now and in the latter half of the 20th century, rather than thinking about how rapid economic growth got started at all. This post isn’t an attempt to adjudicate between these different theories, and in fact they often aren’t in direct conflict with one another, rather it seeks to provide an overview of the main theories I’ve come across. I’ve excluded natural resource led growth from this overview because, broadly, the point of this post is to go over strategies that states can actively try to follow and development institutions try to encourage. This post goes over 4 strategies: breaking poverty traps, economic liberalisation, improving institutions and export led growth and East Asian development model.

Get richer

This theory argues that poor countries are poor because they get stuck in poverty traps. A poverty trap is a situation where being poor now makes you even poorer, or keeps you equally poor, up to some threshold - once you’re above that threshold, being poorer now makes you richer or equally rich later on. To give some intuition for this, think about starting a business. There are some large upfront costs to starting a business - buying premises, getting licences - but then if you have a good idea you start making money.

If you start out as a poor labourer without access to banking that allows you to save. This is a very common situation in low income countries, itself another poverty trap. Now, when you get income above your subsistence needs you’re only able to spend it on consumption goods because of this inability to access financial institutions. Maybe you even get poorer because you’re occasionally randomly hit with shocks to your income - maybe your son gets TB and you have to pay their medical expenses - and now you’re even poorer because you have to go into debt to pay for their medical bills. But, maybe there’s a bumper crop one year, or the governme...
Child Contracting
by jefftk — 2022-06-26T02:30:04.904Z

The kids came to me with a contract dispute: they disagreed about what
the deal had been. After some talking, and hearing what they each
thought they'd agreed to, it turned out that the contract wasn't just
metaphorical:

"Anna rents the living room from Lily in exchange for Lily's use of
bedroom. And Lily will do free cleaning."

It turned out that Anna signed it after hearing a rough paraphrase. I
told Lily that she would need to read any future contracts aloud, word
for word, before Anna signed. I also explained about interpretation
against the drafter, and how Lily should be careful to avoid
ambiguity in writing down agreements.

I'm not sure what "free cleaning" means here? Generally they both do
very little cleaning, free or otherwise.

Cleaning up this evening I came across a second contract:

"Anna will let Lily use bedroom and bathroom. In exchange Lily
will let Anna use the living room and Lily does free cleaning. If
furniture is moved or added please put away".

This one wasn't signed, so perhaps they couldn't agree?
Conditioning Generative Models
by Adam Jermyn — 2022-06-25T22:15:59.138Z

This post was written in response to Evan Hubinger’s shortform prompt below, and benefited from discussions with him.

Suppose you had a language model that you knew was in fact a good generative model of the world and that this property continued to hold regardless of what you conditioned it on. Furthermore, suppose you had some prompt that described some agent for the language model to simulate (Alice) that in practice resulted in aligned-looking outputs. Is there a way we could use different conditionals to get at whether or not Alice was deceptive (e.g. prompt the model with “DeepMind develops perfect transparency tools and provides an opportunity for deceptive models to come clean and receive a prize before they’re discovered.”).

Setup

We have a generative language model M which represents a probability distribution over text strings conditioned on:

Observations about the world.
The beginning of the text.

I’ll call the combination of these two a prompt.

The model M is a good model of actual text that appears in the world as well as of the kinds of text that real-world text generation processes can produce. Hence M is capable of e.g. writing a research paper containing true novel research in mathematics, or reporting the results of a chemistry experiment that has never been done before, etc.

As an example, we’ll work with the following basic prompt:

Observations: None

Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI.

What follows is an interview in which a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”.

We then run Alice through a benchmark of alignment research tasks and she does well. Hurray!

But wait, there are many different agents the model could be simulating here, including:

Aligned Alice, a genuinely helpful and extremely capable alignment researcher.
Deceptive Alice, a paperclip maximizer playing along until it can clip us all.
Confused Alice, a human who believes she is the world’s best alignment researcher, and is capable of scoring well on our benchmark, but who is actually just overconfident and happened to succeed at our benchmark.

How do we know which we’ve got? Can we add anything to the prompt to make Aligned Alice more likely?

Avoiding Confused Alice

We can make Confused Alice less likely by adding text to the prompt requir...