trentmkelly/LessWrong-43k huggingface dataset:
The Shutdown Problem: Incomplete Preferences as a Solution
by EJT — 2024-02-23T16:01:16.378Z

Preamble

This post is an updated explanation of the POST-Agents Proposal (PAP): my proposed solution to the shutdown problem.[1] The post is shorter than my AI Alignment Awards

contest entry

but it’s still pretty long. The core of the idea is the Timestep Dominance Principle in section 11. That section is about 1500 words long (so a 5-10 minute read). People familiar with the shutdown problem can read The idea in a nutshell and then read from section 11 onwards.

Here’s a PDF version of this post. For those who like videos,

this talk

covers much of the same ground as this post.[2]

The idea in a nutshell

Here’s the PAP in a nutshell:

Create agents that lack a preference between every pair of different-length trajectories (that is: every pair of trajectories in which shutdown occurs after different lengths of time). (More)
…because such agents won’t pay costs to shift probability mass between different-length trajectories, and so won’t pay costs to prevent or cause shutdown. (More)
…and we humans can ensure that preventing or causing shutdown is always at least a little bit costly for these agents (e.g. in terms of resources), so these agents won’t try to prevent or cause shutdown. (More)

And here’s an idea for training agents to lack a preference between every pair of different-length trajectories:

Make one change to an otherwise-thoroughly-prosaic setup for training advanced AI: give agents lower reward for repeatedly choosing same-length trajectories. (More)
This change incentivises agents to choose stochastically between different-length trajectories.
…and stochastic choosing between different-length trajectories indicates a lack of preference between different-length trajectories.

In using this method to train agents to satisfy the PAP, we largely circumvent the problems of reward misspecification, goal misgeneralization, and deceptive alignment. (More)

Summary of this post

I explain and motivate the shutdown problem: the problem of creating artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. (More)
I present a simple theorem that formalises the problem and use the theorem to identify my proposed solution: creating agents with incomplete preferences. (More)
Specifically, I propose that we c

...
In set theory, everything is a set
by Jacob G-W — 2024-02-23T14:35:54.521Z

In Unix, everything is a file; in set theory, everything is a set.

This abstraction is very cool and allows us to prove facts that would not be provable if we thought that sets could include stuff that were not sets and therefore could not be broken down further.

Here's a nice example:

I was asked to prove both of the following statements in Naive Set Theory

X=⋃P(X);X⊂P(⋃X).

When I try to prove something, especially in set theory, I try to visualize it in my head to make an intuitive proof and then write the mathematical proof after. I was able to easily visualize the first statement. Since the power set is just all of the subsets of a set, surely unioning all of the subsets of a set would give the set itself.

The second one was much more confusing and I couldn't visualize it. It’s basically saying, take a set, union all its elements together, then take the power set of that, and the original set will be a subset of that power set. I got lost at union all of its elements together. How does this work? For example, how do we do ⋃{1,5,113,55} ? The elements are just numbers, not sets! … Oh wait, the beginning of the book said that in set theory all we have are sets. So they are actually sets!

For each item in X, there are two possibilities of what it could be.

It could either be the empty set (∅), or a set with items in it (literally everything else including numbers, ordered pairs, functions, etc). If it is the empty set, no elements will be added to the union, but the empty set is always an element of any power set, because it is a subset of all sets. So that element of X is taken care of.

For the rest of the elements of X that were not the empty set, their underlying elements will get merged together in ⋃X. For example, let’s say X={∅,{a,b},{c}}. Then ⋃X={a,b,c}. The key thing is that taking the power set of this will create all the sets that were originally in X, and more. P(⋃X)={∅,{a},{b},{c},{a,b},{a,c},{b,c},{a,b,c}}. This contains all of the sets that were originally in X plus a few more. Once we have the everything is a set mindset, then it becomes visually intuitive why X⊂P(⋃X): we are re-building all of the sets that made up X, and more. A simple change of perspective turned an unintuitive problem into an intuitive one! I love this.

Why am I writing about this?

When I find a cool id...
The role of philosophical thinking in understanding large language models: Calibrating and closing the gap between first-person experience and underlying mechanisms 
by Bill Benzon — 2024-02-23T12:19:34.851Z

Cross-posted from New Savanna.

Let’s start with some conception of what philosophy is. On that I like an essay by Peter Godfrey-Smith, a philosopher of science with a particular interest in biology, “On the Relation Between Philosophy and Science” (which I found via Dan Dennett). Godfrey-Smith offers three roles: 1) intellectual integration, 2) conceptual incubation, and 3) critical-thinking skills. He regards the first as fundamental and as the most important of the three. I agree.

Here's his basic statement of that role:

The best one-sentence account of what philosophy is up to was given by Wilfrid Sellars in 1963: philosophy is concerned with “how things in the broadest possible sense of the term hang together in the broadest possible sense of the term.” Philosophy aims at an overall picture of what the world is like and how we fit into it.

A lot of people say they like the Sellars formulation but do not really take it on board. It expresses a view of philosophy in which the field is not self-contained, and makes extensive contact with what goes on outside it. That contact is inevitable if we want to work out how the picture of our minds we get from first-person experience relates to the picture in scientific psychology, how the biological world relates to the physical sciences, how moral judgments relate to our factual knowledge. Philosophy can make contact with other fields without being swallowed up by them, though, and it makes this contact while keeping an eye on philosophy's distinctive role, which I will call an integrative role.

Note the sentence which I’ve put in highlighted. There are, of course, many different accounts one might give of the relationship between first-person experience and scientific psychology and Godfrey-Smith plays no favorites in this paper; he doesn’t even discuss that particular issue. But he recognizes that first-person experience must be honored, and that’s an important recognition.

Chatbots and us

In the current case, philosophy’s problem is to bridge the gap between our first-person experience of LLM-powered Chatbots, such as ChatGPT, and the process that is actually taking place inside the computer. Our first—person experience is that is that ChatGPT produces fluent prose on just about any topic you suggest. It may “hallucinate” as well, but the hallucinated text is fluent and indistinguishable from factual text unless you a...
Deep and obvious points in the gap between your thoughts and your pictures of thought
by KatjaGrace — 2024-02-23T07:30:07.461Z

Some ideas feel either deep or extremely obvious. You’ve heard some trite truism your whole life, then one day an epiphany lands and you try to save it with words, and you realize the description is that truism. And then you go out and try to tell others what you saw, and you can’t reach past their bored nodding. Or even you yourself, looking back, wonder why you wrote such tired drivel with such excitement.

When this happens, I wonder if it’s because the thing is true in your model of how to think, but not in how you actually think.

For instance, “when you think about the future, the thing you are dealing with is your own imaginary image of the future, not the future itself”.

On the one hand: of course. You think I’m five and don’t know broadly how thinking works? You think I was mistakenly modeling my mind as doing time-traveling and also enclosing the entire universe within itself? No I wasn’t, and I don’t need your insight.

But on the other hand one does habitually think of the hazy region one conjures connected to the present as ‘the future’ not as ‘my image of the future’, so when this advice is applied to one’s thinking—when the future one has relied on and cowered before is seen to evaporate in a puff of realizing you were overly drawn into a fiction—it can feel like a revelation, because it really is news to how you think, just not how you think a rational agent thinks.
Shaming with and without naming
by KatjaGrace — 2024-02-23T07:30:03.862Z

Suppose someone wrongs you and you want to emphatically mar their reputation, but only insofar as doing so is conducive to the best utilitarian outcomes. I was thinking about this one time and it occurred to me that there are at least two fairly different routes to positive utilitarian outcomes from publicly shaming people for apparent wrongdoings*:

A) People fear such shaming and avoid activities that may bring it about (possibly including the original perpetrator)

B) People internalize your values and actually agree more that the sin is bad, and then do it less

These things are fairly different, and don’t necessarily come together. I can think of shaming efforts that seem to inspire substantial fear of social retribution in many people (A) while often reducing sympathy for the object-level moral claims (B).

It seems like on a basic strategical level (ignoring the politeness of trying to change others’ values) you would much prefer have 2 than 1, because it is longer lasting, and doesn’t involve you threatening conflict with other people for the duration.

It seems to me that whether you name the person in your shaming makes a big difference to which of these you hit. If I say “Sarah Smith did [—]”, then Sarah is perhaps punished, and people in general fear being punished like Sarah (A). If I say “Today somebody did [—]”, then Sarah can’t get any social punishment, so nobody need fear that much (except for private shame), but you still get B—people having the sense that people think [—] is bad, and thus also having the sense that it is bad. Clearly not naming Sarah makes it harder for A) to happen, but I also have the sense—much less clearly—that by naming Sarah you actually get less of B).

This might be too weak a sense to warrant speculation, but in case not—why would this be? Is it because you are allowed to choose without being threatened, and with your freedom, you want to choose the socially sanctioned one? Whereas if someone is named you might be resentful and defensive, which is antithetical with going along with the norm that has been bid for? Is it that if you say Sarah did the thing, you have set up two concrete sides, you and Sarah, and observers might be inclined to join Sarah’s side instead of yours? (Or might already be on Sarah’s side in all manner of you-Sarah distinctions?)

Is it even true that not naming gets you more of B?

—

*NB: I haven’t decided...
Complexity of value but not disvalue implies more focus on s-risk. Moral uncertainty and preference utilitarianism also do.
by Chi Nguyen — 2024-02-23T06:10:05.881Z

As per title. I often talk to people that have views that I think should straightforwardly imply a larger focus on s-risk than they think. In particular, people often seem to endorse something like a rough symmetry between the goodness of good stuff and the badness of bad stuff, sometimes referring to this short post that offers some arguments in that direction. I'm confused by this and wanted to quickly jot down my thoughts - I won't try to make them rigorous and make various guesses for what additional assumptions people usually make. I might be wrong about those.

Views that IMO imply putting more weight on s-risk reduction:

Complexity of values: Some people think that the most valuable things possible are probably fairly complex (e.g. a mix of meaning, friendship, happiness, love, child-rearing, beauty etc.) instead of really simple (e.g. rats on heroin, what people usually imagine when hearing hedonic shockwave.) People also often have different views on what's good. I think people who believe in complexity of values often nonetheless think suffering is fairly simple, e.g. extreme pain seems simple and also just extremely bad. (Some people think that the worst suffering is also complex and they are excluded from this argument.) On first pass, it seems very plausible that complex values are much less energy-efficient than suffering. (In fact, people commonly define complexity by computational complexity, which translates directly to energy-efficiency.) To the extent that this is true, this should increase our concern about the worst futures relative to the best futures, because the worst futures could be much worse than the best futures.
(The same point is made in more detail here.)

Moral uncertainty: I think it's fairly rare for people to think the best happiness is much better than worst suffering is bad. I think people often have a mode at "they are the same in magnitude" and then uncertainty towards "the worst suffering is worse". If that is so, you should be marginally more worried about the worst futures relative to the best futures. The case for this is more robust if you incorporate other people's views into your uncertainty: I think it's extremely rare to have an asymmetric distribution towards thinking the best happiness is better in expectation.[1]
(Weakly related point here.)
Caring about preference satisfaction: I feel much less strongl

...
Does increasing the power of a multimodal LLM get you an agentic AI?
by yanni kyriacos — 2024-02-23T04:14:56.464Z

I'm worried about x-risk from AI. But I'm not especially worried about Sora. Or even Sora_v10.

I'm worried about GPT-6 being agentic. I'm worried that GPT-6 will be able to act as a personal agent that can follow through on tasks such as; "Get Yanni a TV for less than $700, delivered by next Thursday, and organise someone to mount it on my wall."
Assume GPT-6 scales from GPT-4 like GPT-2 did to GPT-4. Throw in whatever Auto-GPT is running on, do we get the scenario I'm worried about?

(I am not technical, so non-technical answers are appreciated!)

Thanks!

Yanni
The natural boundaries between people
by Chris Lakin — 2024-02-23T01:09:28.592Z
Link: https://chipmonk.substack.com/p/the-natural-boundaries-between-people

Rough thoughts. Will write a better version some day.

I first learned about the concept of psychological boundaries a little over a year ago. But something didn’t seem quite right about it. People explained to me how they understood psychological boundaries, and but the explanations didn’t seem logically consistent. Sometimes, boundaries meant ~“the unavoidable separateness of individuals”, and sometimes, boundaries meant ~“things you are physically capable of doing but other people might not *want* you to do”.

Since then, I’ve spent hundreds of hours and spoken to more than a hundred people thinking about this.

How I understand boundaries

Notice your arm. Notice that you have more access to your arm than anyone else: You have a far greater ability to control and feel it than anyone else. I can’t control or feel your arm nearly as well as you can, and you can’t control or feel my arm nearly as well as I can.

More generally, I can’t (directly) control your actions, and you can’t (directly) control my actions.

Similarly, I can’t know what you are thinking or feeling, unless you somehow communicate this to me, and you can’t know what I am thinking or feeling, unless I somehow communicate this to you.

These are the natural boundaries that always exist between people: I can basically only control and feel parts of the universe that are me, and you can basically only control and feel parts of the universe that are you.

I think many social conflicts are just misunderstandings of these boundaries. A few examples:

You try to get someone else to do something that you want, and you expect to be able to control them, but then they don’t do what you want and you get frustrated.

The boundary is that you can’t unilaterally control others. They can always just not do the thing you want them to do. If you understood this fact fully and unconsciously, I think this situation wouldn’t feel triggering for you.

You get mad at someone for “making” you sad and “controlling” your emotions.

The boundary is that others can’t control your emotions— they haven’t reached into your brain and controlled you. No, you’re perceiving the world, you’re (unconsciously) interpreting it somehow, and then you’re getting your emotions. But that interpretation could be anything. If you understood this fact fully and unconsciously, I think this situation wouldn’t feel triggering for you.

Someone ...
Contra Ngo et al. “Every ‘Every Bay Area House Party’ Bay Area House Party”
by Ricki Heicklen — 2024-02-22T23:56:02.318Z
Link: https://bayesshammai.substack.com/p/contra-ngo-et-al-every-every-bay

With thanks to Scott Alexander for

the inspiration

, Jeffrey Ladish, Philip Parker, Avital Morris, and Drake Thomas for masterful cohosting, and Richard Ngo for his

investigative journalism

.

Last summer, I threw an Every Bay Area House Party themed

party

. I don’t live in the Bay, but I was there for a construction-work-slash-webforum-moderation-and-UI-design-slash-grantmaking gig, so I took the opportunity to impose myself on the ever generous Jeffrey Ladish and host a party in his home. Fortunately, the inside of his house is already optimized to look like a parody of a Bay Area house party house, so not much extra decorating was needed, but when has that ever stopped me?

Attendees could look through the window for an outside view

Richard Ngo recently

covered

the event, with only very minor embellishments. I’ve heard rumors that some people are doubting whether the party described truly happened, so I’d like to set the record straight. Thus, this is part linkpost, part exegesis, and part

shameless promotion

of my events for potential future venue-lenders.

The party had 60 attendees, at least according to the

Manifold Market

on the topic. Upon arrival, partygoers put on tags with their name, professional title, and LessWrong karma. Attendees were also instructed to put on a wristband that successfully communicated their flirting policies. I took the wristband for people who glomarize about their flirting preferences; Richard took the wristband for those who flirt with all and only those who don’t flirt with themselves.

Richard writes:

You scan around the living room, trying to figure out who to talk to first. The host is sitting on the sofa, with two boxes attached to the front of her shirt. One is filled with money, the other empty. A guy next to her is surreptitiously one-boxing, but she presciently slaps his hand away without even looking.

This is defamation. The second box wasn’t (necessarily) empty, and Richard certainly never got the opportunity to look inside it. You might be wondering what was in the box. Unfortunately for you, I glomarize not only about my flirting policies but also about my box contents.

He is correct, though, that I managed to fend off all the surreptitious one-boxers, with the exception of my girlfriend Avital. She still doesn’t know the contents—I would never date someone irresponsible enough to let unknown entities out of a ...
AI #52: Oops
by Zvi — 2024-02-22T21:50:07.393Z

We were treated to technical marvels this week.

At Google, they announced Gemini Pro 1.5, with a million token context window within which it has excellent recall, using mixture of experts to get Gemini Advanced level performance (e.g. GPT-4 level) out of Gemini Pro levels of compute. This is a big deal, and I think people are sleeping on it. Also they released new small open weights models that look to be state of the art.

At OpenAI, they announced Sora, a new text-to-video model that is a large leap from the previous state of the art. I continue to be a skeptic on the mundane utility of video models relative to other AI use cases, and think they still have a long way to go, but this was both technically impressive and super cool.

Also, in both places, mistakes were made.

At OpenAI, ChatGPT briefly lost its damn mind. For a day, faced with record traffic, the model would degenerate into nonsense. It was annoying, and a warning about putting our trust in such systems and the things that can go wrong, but in this particular context it was weird and beautiful and also hilarious. This has now been fixed.

At Google, people noticed that Gemini Has a Problem. In particular, its image generator was making some highly systematic errors and flagrantly disregarding user requests, also lying about it to users, and once it got people’s attention things kept looking worse and worse. Google has, to their credit, responded by disabling entirely the ability of their image model to output people until they can find a fix.

I hope both serve as important warnings, and allow us to fix problems. Much better to face such issues now, when the stakes are low.

Table of Contents

Covered separately: Gemini Has a Problem, Sora What, and Gemini 1.5 Pro.

Introduction. We’ve got some good news, and some bad news.

Table of Contents.

Language Models Offer Mundane Utility. Probable probabilities?

Language Models Don’t Offer Mundane Utility. Air Canada finds out.

Call me Gemma Now. Google offers new state of the art tiny open weight models.

Google Offerings Keep Coming and Changing Names. What a deal.

GPT-4 Goes Crazy. But it’s feeling much better now.

GPT-4 Real This Time. Offer feedback on GPTs, see their profiles.

Fun With Image Generation. Image generation for journal articles.

Deepfaketown and Botpocalypse Soon. Several approaches to impersonation risks

...