trentmkelly/LessWrong-43k huggingface dataset:
General alignment plus human values, or alignment via human values?
by Stuart_Armstrong — 2021-10-22T10:11:38.507Z

Thanks to Rebecca Gorman for discussions that lead to these insights.

How can you get a superintelligent AI aligned with human values? There are two pathways that I often hear discussed. The first sees a general alignment problem - how to get a powerful AI to safely do anything - which, once we've solved, we can point towards human values. The second perspective is that we can only get alignment by targeting human values - these values must be aimed at, from the start of the process.

I'm of the second perspective, but I think it's very important to sort this out. So I'll lay out some of the arguments in its favour, to see what others think of it, and so we can best figure out the approach to prioritise.

More strawberry, less trouble

As an example of the first perspective, I'll take Eliezer's AI task, described here:

"Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level." A 'safely' aligned powerful AI is one that doesn't kill everyone on Earth as a side effect of its operation.

If an AI accomplishes this limited task without going crazy, this shows several things:

It is superpowered; the task described is beyond current human capabilities.

It is aligned (or at least alignable) in that it can accomplish a task in the way intended, without wireheading the definitions of "strawberry" or "cellular".

It is safe, in that it has not heavily dramatically reconfigured the universe to accomplish this one goal.

Then, at that point, we can add human values to the AI, maybe via "consider what these moral human philosophers would conclude if they thought for a thousand years, and do that".

I would agree that, in most cases, an AI that accomplished that limited task safely would be aligned. One might quibble that it's only pretending to be aligned, and preparing a treacherous turn. Or maybe the AI was boxed in some way and accomplished the task with the materials at hand within the box.

So we might call an AI "superpowered and aligned" if it accomplished the strawberry copying task (or a similar one) and if it could dramatically reconfigure the world but chose not to.

Values are needed

I think that an AI could not be "superpowered and aligned" unless it is also aligned with human values.

The reason is that the AI can and has to interact with the world. It has the capability to do so, by assumption - it is not contai...
Epistemic Strategies of Safety-Capabilities Tradeoffs
by adamShimi — 2021-10-22T08:22:51.169Z

Introduction: Epistemic Strategies Redux

This post examines the epistemic strategies of Steve Byrnes’

Safety-capabilities tradeoff dials are inevitable in AGI

.

(If you want to skim this post, just read the Summary subsection that display the epistemic strategy as a design pattern)

I introduced the concept in

a recent post

, but didn’t define them except as the “ways of producing” knowledge that are used in a piece of research. If we consider a post or paper as a computer program outputting (producing) knowledge about alignment, epistemic strategies are the underlying algorithm or, even more abstractly, the

design patterns

.

An example of epistemic strategy, common in natural sciences (and beyond), is

Look at the data
Find a good explanation
Predict new things with that explanation
Get new data for checking your prediction

More than just laying out some abstract recipe, analysis serves to understand how each step is done, whether that makes sense, and how each step (and the whole strategy) might fail. Just like a design pattern or an algorithm, it matters tremendously to know when to apply it and when to avoid it as well as subtleties to be aware of.

Laying this underlying structure bare matters in three ways:

It clarifies the research’s purpose and value for newcomers and researchers from other fields, with minimal assumptions of shared approaches.

Just like a programmer switching to a new domain of problems will get up to speed faster and more reliably if they get access to the patterns/algorithms/tricks used in their new domain.

It focuses feedback and criticism on the most important parts of the idea/proposition/argument.

Issues with an algorithm more often focus on the point of it instead of the details,whereas issues with an implementation of that algorithm can be as much about typos, optimization tricks and bad structure than about the actual core (the algorithm.

It builds a library of such strategies for alignment in particular, a cookbook newcomers and senior researchers alike can browse for inspiration or a take on some new post/paper they don’t grok.

Like the glorious

Game Programming Patterns

who does exactly that for game programming

Thanks to Steve Byrnes for feedback on a draft of this post.

Defining Safety-Capabilities Tradeoffs

What sort of knowledge is Steve attempting to create in his post? He set up explicitly to show that any alignment proposal must dea...
Coordination Motivation: The Pandemic
by Raemon — 2021-10-21T21:57:11.524Z

I first started thinking about the meta-coordination 4 years ago, in the context of rationalists arguing about community norms. It seemed to me that people were getting into fights that involved a lot of wasted motion, and failing to accomplish what seemed like obvious shared goals.

For a few years, the bulk of my thought process was a vague, dissatisfied "surely we can do better than this, right?". Many of the people arguing eventually went off to focus on their individual orgs and didn't interact as much with each other. Maybe that was the right solution, and all this worrying about meta-coordination and norm arguments was just a distraction.

Then a pandemic hit. Coordination became much more practical and important to me, and the concept of coordination pioneering became more directly relevant.

Here were some issues that felt coordination-shaped to me. In this post, I’m speaking largely from my experiences with the Bay Area rationality community, but I think many of the issues generalize.

Negotiating policies and norms within a single household. Do you lock down? If so, how do you go about it? What do you do if people disagree on how dangerous covid is, what practices are effective, or what’s worth trading off for safety?
Community contract tracing. If someone at a party later gets covid, are people entitled to share that information? How do we negotiate with each other about sharing that information? This includes concerns about privacy, public safety, and how to socially navigate trading those off against each other during a crisis.
Maintaining social connection. This might involve negotiation with your housemates over covid policy, or the housemates of your friends. Even if you and a friend each live alone, figuring out what kind of contact to have during a pandemic is at least a two-player game.
Housemate swapping/matchmaking. Housemates hadn't generally been selected for "having similar preferences of how to handle pandemics". There were several reasons people might have wanted to relocate. But people also had reason to not necessarily want to advertise that they were looking for new housemates – they might risk antagonizing their current roommates, or airing drama that was still unfolding. Switching houses is also an effortful, high cost decision that was difficult during an already stressful time.
Allocation of labor (intellectual and otherwise). The

...
An Idea for a More Communal Petrov Day in 2022
by Ben Pace — 2021-10-21T21:51:15.270Z

(This post is a lightly edited memo I wrote for a Lightcone Infrastructure team meeting on the topic of Petrov Day, that Ruby then asked me to publish on the site.)

The main thing I want with Petrov Day is a sense of community, trust, and the respect of the principle of taking responsibility for the ultimate consequences of your actions.

I think the current format for Petrov Day has lots of room to grow. I spent an hour or two thinking about what a better Petrov Day would look like, here is a pointer to something we could do next year.

An Idea for a More Communal Petrov Day Ritual

Next Petrov Day, we host a public, online ceremony that 100s of people attend to watch. It is based around the Ceremony Readings Jim Babcock has put together. It involves lots of people taking turns to read quotes, basically everyone who ahead of time signed up to do a reading, and could assure that they had a sane AV setup. It's open invite to anyone to view.

After the ceremony, we run an online Gather Town party for 100s of people. Perhaps it's for LWers only, or perhaps it's open-access for everyone.

During the day, a large red button is on the site. Several months in advance, we open a sign-up to be trusted with codes for the day, and encouraged people to participate, and most people are given the codes. If the button is pressed, the online ceremony is ended / the party shuts down after 10 minutes.

There is an online record of the day. How many people showed up, their names, and who spoke. This is a web-page designed for purpose, somewhat more in the style of the www.dontdoxscottalexander.com site that Jacob and I made.

Possible further ideas

Possible idea: A member of Petrov’s family is invited to attend and to give a comment at the end of the ritual.
Possible idea: Every year some speaker is invited to give a short talk about what the day means to them. Similar to how there’s an annual moment-of-darkness speech at Solstice.
Possible idea: a bit of singing, using Bucket Brigade.
Possible idea: we encourage lots of local groups to do their own ceremonies.
Possible idea: Somehow a planned false alarm? I would like the red button to have a serious game to it, but I don’t know how to do it every year for multiple decades. Some actual uncertainty every year?
Possible idea: To build up the numbers, if you apply and we give you codes, you can have codes every year (though you can remove yourself f

...
Petrov Day Retrospective: 2021
by Ruby — 2021-10-21T21:50:40.042Z

I apologize for not posting this closer to Petrov Day. It’s been a busy month and there was much to think about.

You can view the

EA Forum’s retrospective here

.

This year was the third

Petrov Day

celebration on LessWrong in which the site was endangered, and the first year we joined together with the EA Forum. In case you missed it, neither site was taken down, despite 200 people being issued codes that would allow them to do so [1][2]. Huzzah!

Although neither site went down (and thus there's no need for a blow-by-blow analysis of whodunit and why), there are some interesting things to review. In particular, there were some substantial criticisms of the Petrov Day ritual this year and last year that I want to address.

Why Petrov Day

The annual Petrov Day post recounts the basic story of Petrov Day, yet given the questions that were asked this year about what Petrov Day should be, I think it’s right to first revisit why we celebrate Petrov Day in the first place. The following is my own personal take, the one from which I’ve acted, but it is Not Official.

We find ourselves at what may be one of the most critical periods in the history of humanity and the universe. This is kind of crazy–though I’ll refer you to the

writings of Holden Karnofsky

for a compelling argument for why believing anything else is equally crazy. In the next few decades, we might go

extinct

(

or worse

), or we might commence an explosion in progress and productivity that propels us to the stars, allowing us to take the seemingly barren universe and fill it with value.

Petrov Day is a celebration of not going extinct. It’s a commemoration of not taking actions that would destroy the world. It’s about how Petrov chose not to follow policy and relay his alarm because, in his personal estimation, it was probably a false alarm. If he had relayed the alarm, there’s a chance his superiors would have chosen to launch nuclear missiles at the US, and history would be very different.

We can identify two virtues worth applauding in the story:

Choosing actions that don’t destroy the world
Even in the face of pressures otherwise, using one’s judgment to not destroy the world

On September 26th, we celebrate these virtues and attempt to enshrine them in our community. We say to ourselves and others I accept the virtue of not destroying the world, even when there’s pressure to do it! We don’t do this for idle spiritu...
Feature idea: Notification when a parent comment is modified
by Viliam — 2021-10-21T18:15:54.160Z

Not sure how many people would consider this feature useful: Imagine that you reply on someone else's comment, and the person edits their comment later. I think it might be useful (perhaps depending on circumstances) to get a notification.

Notification "XY edited a comment you replied to" should appear at the same place as when you get a reply. In perfect case, the tooltip would highlight the difference between the original and the updated comment.

Use cases that I imagine:

Person A makes a comment. Person B makes a reply disagreeing with the original comment. Upon reading this, person A changes their mind and updates the original comment like this: "[EDIT: Actually, B makes a good argument against]". This feature would show this information in person B's inbox, without A having to write a separate reply to their comment.
Person A makes a comment. Person B makes a disagreeing reply. Person A silently updates their original comment to make B's response seem silly.

This feature would apply only to comments that reply to comments, i.e. not to the top-level comments, because I assume that minor modifications of articles are sometimes too frequent (and would flood the inboxes of top-level commenters), and because more people would notice a substantial stealthy article edit.

An argument against this feature is that some people can make frequent insubstantial edits to their comments (e.g. fix typos), which also could flood the inboxes of the repliers. Then this feature would be annoying. Possible solutions:

Multiple unread notifications for the same comment are merged into one.
Some heuristic (e.g. only punctuation or isolated words are modified) to detect insubstantial edits.
Or an opposite solution (covering only the first use case), where a heuristic would identify substantial edits (e.g. where the added text contains a word like "edit", "update", "change", "modify", "remove", "delete").

Your opinions?
Stoicism vs the Methods of Rationality
by Space L Clottey — 2021-10-21T18:07:04.787Z

Crossposted from spacelutt.com

I fell in love with Stoicism for about 5 months in 2021, but am now no longer so infatuated, and this post takes a look at a few of the reasons why.

Determining Control

The only thing that’s really out of your control is things that happen in the past, since time really only flows forward.

The “Challenging the Difficult” sequence in The Sequences is about how often you’ll be wrong at labelling something “impossible” (which is of course synonymous with “outside of your control”, except in “outside your control” being even more retreat-y than impossible as it implies another human can do it, just not you).

Your body will be ultimately destroyed, but this is not seen as bad, since it is out of your control.

~ Marcus Aurelius, Meditations

Even if you can’t control it, it still seems bad.

It seems wrong to be changing your definition of bad based on what’s in your control or not.

You should care, I think, but not fall into turmoil.

—

This is particularly bad when it comes to things you don’t have personal stake it. If you were to say “I don’t have control over my fatigue problem” too soon, they’re you’re gonna still want to think about it like it’s in your control and still go wild with attempts and ideas, because it still effects you, there’s no way of getting out of your sleep problem mattering in your world. You still feel it, whether or not it’s in your control.

But on a larger scale…

For example if some random kid in Africa is starving to death and someone tells you this and helping would be inconvenient, you could very well say “it’s not in my control” and totally forget about it and have it lose its effect on you. But the kid still starves, but the kid is not in your world. Thus you’re not motivated to really, really try before you actually declare something is impossible.

This is even more true when discussing abstract threats to the future.

I have seen many people struggling to excuse themselves from their ethics. Always the modification is toward lenience, never to be more strict. And I am stunned by the speed and the lightness with which they strive to abandon their protections. Hobbes said, “I don’t know what’s worse, the fact that everyone’s got a price, or the fact that their price is so low.” So very low the price, so very eager they are to be bought. They don’t look twice and then a third time for alternatives, before deciding that

...
Noticing the Value of Noticing Confusion
by Space L Clottey — 2021-10-21T18:04:51.749Z

Crossposted from spacelutt.com

Your strength as a rationalist is your ability to be more confused by fiction than by reality. Either your model is wrong or this story is false.

~ Your Strength as a Rationalist, Eliezer Yudkowsky

Your model of the world is how you understand the world to work. If I think ice is frozen water, and ice is frozen water, then my model of the world is right. If I’m six and I think Santa Clause is the one who brought me presents, when really it was my parents, then my model of the world is wrong, whether I know it or not (and the whole point is that you don’t, because if you knew your model was wrong it wouldn’t be you’re true model anymore).

Confusion is not understanding. If you are never confused, it means you understand everything all the time. It means your model of the world is exactly perfectly right.

Ha.

Looking at it this way, it’s extremely obvious why over the past year as I’ve been practicing the skill of noticing my confusion more there’s been more and more opportunities until it’s just a constant, never-ending flow. So many micromysteries.

And it’s not only lying that it prevents you against. I’ve caught out lots of people in semi plausible lies, with the thought pattern being “If x, then almost definitely y, and I see no y so more likely you’re lying” and then I’m right and I feel really good.

One of the key aspects is knowing that your model is supposed to forbid things. People lying to you is a very obvious time when the story you’re being fed is wrong. But also you telling yourself an inaccurate story is the same as a lie for these purposes, as it’s another opportunity to say “hey wait, this doesn’t quite fit” and then say “I’m wrong!”.

Like when you’re eating dinner with someone, and they go to get up and you don’t know why they left. 99% of the time people get up, it’s to go and get water. So you assume they went to get water, and think you have understood what is going on.

But their cup is still sitting on their tray.

EITHER YOUR MODEL IS WRONG OR THIS STORY IS FALSE.

And after you’re good at noticing when you don’t understand, you get to play one of the funnest games available to man: trying to figure out what the hell is actually going on. And this is the bit where your brain is forever warped, this is the bit for why Eliezer says this is your strength as a rationalist. Did you notice your confusion when Eliezer said this...
What's Stopping You?
by Neel Nanda — 2021-10-21T16:20:18.911Z
Link: https://www.neelnanda.io/blog/44-agency

Introduction

This post is about the concept of agency, which I define as ‘doing what is needed to achieve your goals’. As stated, this sounds pretty trivial - who wouldn’t do things to achieve their goals? But true agency is surprisingly hard and rare. Our lives are full of constraints, and defaults that we blindly follow, going past this to find a better way of achieving our goals is hard.

And this is a massive tragedy, because agency is incredibly important.

The world is full of wasted motion

. Most things in both our lives and the world are inefficient and sub-optimal, and it often takes creativity, originality and effort to find better approaches. Just following default strategies can massively hold you back from achieving what you could achieve with better strategies.

This should not be surprising. Thinking past defaults, and your conception of ‘normal’ is hard, and takes meaningful effort. But this leaves a lot of value on the table! Personally, I’ve found cultivating agency to be one of the most useful ways I’ve grown over the last few years. And, crucially, this was deliberate - agency is a hard but trainable skill. In this post, I try to outline my models of what agency is, and my thoughts on how to cultivate it.

Why Care?

I’ve been pretty abstract so far about the idea of agency. A reasonable reaction would be to be pretty skeptical that there’s any value here. That you already try to achieve your goals, that’s what it means to have goals! But I would argue that it’s easy to be missing out on opportunities and better strategies without realising it. An unofficial theme of this blog is that the world is

full of low-hanging fruit

, you just need to look for it. Further, many of the people I most admire, who’ve successfully changed the world for the better, had a lot of agency - changing the world is not the default path!

To make this more concrete, I want to focus on examples of how agency has been personally valuable to me. The times I managed to step off the default path, look past my constraints, and be more ambitious and creative about how I achieved my goals.

By far the most significant was realising that I could take agency with my career and life path. That I could step away from the default of continuing my undergrad to a fourth year and masters and then doing a maths PhD or working in finance. And instead, I’ve spent the past year doing 3 back-to-b

...