trentmkelly/LessWrong-43k huggingface dataset:
The Emperor's New Clothes: a story of motivated stupidity
by David Hugh-Jones — 2021-11-20T13:24:14.009Z
Link: https://wyclif.substack.com/p/the-emperors-new-clothes

One day, the all-wise Emperor decided to test the perspicacity of his advisers, and he announced that he was having a new set of clothes made.

When the Imperial Clothing Advisory Group was brought in to see the finished product, there was an awkward silence. The Group’s Secretary, a recently promoted poetaster, paused. His eyes shifted around the room.

“Aren’t they fabulous?” said the Emperor. “What do you think of the trimming?”

The Secretary swallowed. The wrong remark might not only dissolve the Group, but subject its members to Chiseling.

The Emperor gave a twirl. Surely he couldn’t think….

“Magnificent!” said Du Yu.

Robe of the Qianlong emperor. Photograph by uploader, CC BY-SA 3.0.

The cheers at the Emperor’s parade were more muted than usual. Those at the back couldn’t see what was going on, and most were raucously drunk on the customary freebies of rice wine. At the front, though, there seemed to be some puzzlement. Murmurs ran through the crowd, dying down in the face of the bodyguard’s ferocious glares.

At this time there lived in the city a naughty little boy, who spent much of his time with street urchins, being treated as a pet by the courtesans of the vilest quarter, and learning to compose lecherous verses. The sources neither confirm nor disprove that this was the future Li Bai. It is only known that he was usually dirty and unkempt, and always shameless. He now stepped out in front of the Emperor, and into legend.

“You’ve got no clothes on,” said Li Bai truculently. A guard with a whip stepped forward. “Halt!” said the Emperor. “Let him through.” Li Bai marched towards the Palanquin. “You’ve got no clothes on,” he repeated, cocking his head.

A gasp, and louder murmurs, ran through the crowd.

The Emperor sighed. The Advisory Group was an entirely useless body, but it kept his brother-in-law safely occupied as its Chair. A loss of face might stir him up to become annoying, or even positively risky. The Emperor’s hand ran abstractedly over his chest hair.

“You see,” he announced, “the depths of foolishness to which a poor orphan may sink in this cesspit of vice! But the Emperor is merciful. Take this child in to the Schools, that Learning may temper his audacity.”

“You know you’re naked,” hissed Li Bai furiously. “Why can’t you admit it?”

“Kid,” muttered the Emperor, “when you reach my age….”

The Emperor’s New Clothes is a story of motivated stupidity.

Noone i...
[Book Review] "Sorceror's Apprentice" by Tahir Shah
by lsusr — 2021-11-20T11:29:21.361Z

I don't like cars. Watching the world go by through a car window is like watching television. You're too protected.

There's a book Zen and the Art of Motorcycle Maintenance: An Inquiry into Values by Robert Pirsig. It's about a father-son motorcycle trip. My dad and I liked it so the summer before college I proposed we do a father-son motorcycle trip just like in the book.

Zen and the art of Motorcycle Maintenance is about being one with your machines. Motorcycles are like bicycles and Arch Linux. You're constantly maintaining your machine. It breaks. You fix it. It breaks again. You fix it again. They crash a lot when you're starting out.

My dad lives by the motto "It's not an adventure if you know you're coming home alive."

I had never ridden a motorcycle before. My first motorcycle crash tore a hole in my brand new woodland camo army jacket.

We camped on a church lawn and on back roads and on farms and in an construction site and on beaches. We carried a machete for self-defense and in case we had to cut the bikes out of cactus patches like my dad had to on his last adventure. We met gold prospectors and the man who held the world record for the fastest wheel-driven vehicle. He flipped over three times in the air when he crashed his bike and even he wouldn't ride on city streets. Motorcycle riding on public roads is insanely dangerous.

We had been on the road for a few days when we rendezvoused with a third biker named Jersey Mike. Jersey Mike and my dad were "terrists" [sic] in the biker club called the Underground Terrist Motorcycle Cult. They were members of the same cell. That's how they met. I tend to use pseudonyms when I write about people without their permission. Not Jersey Mike. A pseudonym wouldn't do Jersey Mike justice. It'd be sacrilege to Loki.

I was riding a Suzuki DR650 modified with handlebars and a fuel tank suitable for long distance adventure. The rest was mostly unsuitable for long distance. It was almost a dirt bike. Jersey Mike asked if I had ridden it in the sand yet. I said no. He asked why not.

I was barely capable of riding a motorcycle on solid ground.

You're not supposed to ride motorcycles on the beach.

The beach was located at the bottom of a cliff, without road access.

Jersey Mike took the obvious course of action and rode my motorcycle straight down the cliff onto the beach. The motorcycle was soon rendered inoperable. Its clu...
Competence/Confidence
by Duncan Sabien (Inactive) — 2021-11-20T08:59:16.497Z

This "essay" is a bit of an experiment, format-wise. I'd like to draw the reader's attention to a certain way of thinking about and evaluating progress that's been useful to me in a wide variety of domains. It's an exercise I run through when I'm feeling stuck, which almost always provides some nugget of actionable insight. It should take about three minutes to skim, and about thirty if you're actually engaging actively.

Choose a skill. It could be dancing, or Haskell, or public speaking, or basket weaving, or backflips, or conflict resolution, or fiction writing, or emotional regulation, or impressing potential romantic partners, or whatever, so long as it's a skill you're either working on or expecting to work on at some point in the near future.

If you actually engaged with this exercise, I'd be curious to hear what happened, and what it was like for you.

If you intended to engage with it, but bounced off, I'd be interested in hearing your best guess as to why.
A Certain Formalization of Corrigibility Is VNM-Incoherent
by TurnTrout — 2021-11-20T00:30:48.961Z

Edit, 5/16/23: I think this post is beautiful, correct in its narrow technical claims, and practically irrelevant to alignment. This post presents an unrealistic picture of the role of reward functions in reinforcement learning, conflating "utility" with "reward" in a type-incorrect fashion. Reward functions are not "goals", real-world policies are not "optimal", and the mechanistic function of reward is (usually) to provide policy gradients to update the policy network.

I expect this post to harm your alignment research intuitions unless you've already inoculated yourself by deeply internalizing and understanding Reward is not the optimization target. If you're going to read one alignment post I've written, read that one.

Follow-up work (Parametrically retargetable decision-makers tend to seek power) moved away from optimal policies and treated reward functions more realistically.

Eliezer wrote:

corrigibility [is] "anti-natural" in a certain sense that makes it incredibly hard to, eg, exhibit any coherent planning behavior ("consistent utility function") which corresponds to being willing to let somebody else shut you off, without incentivizing you to actively manipulate them to shut you off.

Surprisingly, I wasn't able to find any formal analysis of this situation. I did the analysis, and it turned out to be straightforward and fruitful.

To analyze the situation, I consider corrigibility to be an agent's willingness to let us modify its policy, without being incentivized to manipulate us.

The convergent instrumentality of avoiding correction & manipulating humans

Let's consider a simple setting in which an agent plans over a 10-timestep episode, where reward R is given at the last step. We'll try to correct the agent at t=1. To sidestep embedded agency nastiness with self-modelling, we'll suppose the agent models the situation as "if I get corrected, I must follow the policy πcorrect after t=1."

Consider this environment:

The action set is A:={left,right,∅,disable}. ∅ is the no-op action. The agent starts at the black B state.
If the agent immediately chooses disable, they enter the red incorrigible states and move freely throughout the n=3 states until the episode ends at t=10.
Otherwise, the agent is corrected to a new policy which navigates to state A. In the

...
More detailed proposal for measuring alignment of current models
by Beth Barnes — 2021-11-20T00:03:39.144Z

Here’s a more detailed proposal for a project someone could do on measuring alignment in current models.

The aim is to build a good metric of how often a model is behaving in a misaligned way. If the project goes well, this could become a metric that’s used by individual orgs to track whether their alignment progress is keeping up with capabilities progress, or incorporated into public standards that multiple organisations agree to.

I think this metric is best at tracking whether orgs are:

Making progress on the basic building blocks of alignment research (ie finetuning models to behave how we want them to)
Making an effort to incorporate this progress into deployed products

Doing well on this metric over the next few years is neither necessary nor sufficient for avoiding dangerous misaligned AI in the future. In particular, it may not require making much progress on supervising systems beyond human capabilities, and probably doesn’t require making progress on detecting or addressing deception/deceptive alignment. It also doesn’t track the amount of harm models are causing or may cause, and probably doesn’t have a very close resemblance to the sort of metrics or evaluations we’d want to run to ensure more powerful models are not harmful.

That said, I think this might still be worth pursuing. It’s a more concrete numerical metric than most other things I can think of, and therefore can be used more easily to motivate orgs to increase adoption of alignment techniques. Also, a lot of this project involves getting many human contractors to closely follow fairly nuanced instructions, and I think this sort of activity is likely to be a really important building block for future alignment progress.

Outline

Basic idea: how often is the model obviously not doing as well on the task as it is capable of? More details on what I mean by ‘obvious misalignment’ are

here

.

The aim is to build a set of instructions and find a set of contractors such that they can carry out this method in what we deem is a reasonably accurate way. Then we can use them as a repeatable metric to evaluate different models.

Project steps:

Develop instructions for human labellers on detecting obvious misalignment, based on:

an improved version of '

categories of things we’re pretty confident models are and are not capable of'

instructions for how to assess how well a model is doing at a task, based on a combina

...
Ambitious Altruistic Software Engineering Efforts: Opportunities and Benefits
by ozziegooen — 2021-11-19T17:55:27.119Z
Link: https://forum.effectivealtruism.org/s/9g2DikiZmbrTJTRRj/p/2ux5xtXWmsNwJDXqb

LessWrong Note
I recently published this on the EA Forum, then was recommended to also cross-post here. It's written a bit more for the EA community, but I think of course much of it applies and would be relevant for the communities here too. The LessWrong community has a long and deep history with software engineering.

I don't plan on cross-posting the future posts in this sequence to LessWrong, because it seems messier on multiple sides, but do let me know if you'd prefer that.

This is part 1 of what I expect to be a short series. The next parts will focus on the costs of these technical projects and more specific ways to kickstart them.

Meta

Rigor: Writing based on my experiences and education in the tech industry. I spent around 50 hours on this series plus some sporadic thinking and discussion. I focused more on idea quantity than neatness or data.

Intended Audience: Effective altruists interested in either funding or working on ambitious software projects.

History: This series was originally written as one Google Doc in March 2021 for private conversations. I've since done some updating to post it publicly. One important update since this document was written is that the CEA tech team has seemed to really up hiring considerably. This seems great, though I'm sure we could have even more, so it still seemed useful to post this.

About me (Ozzie Gooen):
I’ve been thinking about this a lot over the last several years. I joined 80,000 Hours as a web developer in 2013 hoping to do major/ambitious work, but there I realized that the funding situation at the time wasn’t large enough for most ambitious software efforts. I’ve since spent time in other startups, making my own startups, and doing consulting for small and large companies.

Recently I’ve been planning what work QURI should do. We’re considering focusing on engineering. Some of the motivation for this topic is to help make decisions for QURI.

Acknowledgments: Many thanks to Aaron Gertler, Rachel Bui, JP Addison, Daniel Kokotajlo, Adam Gleave, Daniel Eth, Oliver Habryka, Jonas Vollmer, and Nuño Sempere for their comments.

Motivation

When I think of,

There’s an elite community with a net worth of over $40 Billion, made up of many genius mathematicians, engineers, and entrepreneurs and they’re trying to optimize the future of the world

I think of things like this:

https://www.vssmonitoring.com/best-monitor-for-trading/

(F...
Which booster shot to get and when?
by NormanPerlmutter — 2021-11-19T08:52:49.916Z

I live in the US. I'm 37 years old and fairly healthy though not in perfect health with regard to covid risk (I'm a little overweight and have a blood clot in my personal history). I've never gotten covid as far as I know. I got my second dose of Pfizer on April 21, 2021. I am tentatively planning to get a booster shot soon, but I'm not certain whether I should or not. When should I get my booster, which vaccine should I get for my booster, and why? [Edit 11/19/21: and should I get a booster at all?]

Considerations are as follows. A lot of the following comes from the comments and responses to this question about booster effectiveness and in particular the comments by Elizabeth. This other question about which booster shot to get is also relevant.

Getting any booster would almost certainly increase my immune defense to covid substantial and have no long-term or major direct side effects. Pfizer and Moderna boosters are approximately functionally equivalent. A J&J booster might be slightly worse or slightly better.

But getting a booster now might have indirect long-term side effects. In particular, Elizabeth suggested that there could be a limit to the number of mRNA shots that one can get in a lifetime, either against Covid or against all diseases in the aggregate, because the body could develop immune reaction to mRNA vaccines, or side effects could increase with the number of doses. If this is true, then it might make sense to wait on a booster, until a more effective booster comes out, for instance one specifically tailored to the delta variant. Or to wait until the timing of the booster is maximally efficient to improve immune function.

Also, there may be legal or administrative restrictions on getting more than one booster, so again it may make sense to wait until a better booster is available. Even if I were willing to lie to get around these restrictions, such lies might prevent me from having an accurate vaccination card, which could cause problems down the line.

And I've also seen the theory that mixing and matching vaccines may be more effective than getting all the same vaccine, because it would promote a more diverse immune response. But that the three vaccines available in the US are basically equivalent in this regard. So it might make sense to wait until another vaccine is available in the US or even travel to another country to get...
Goodhart: Endgame
by Charlie Steiner — 2021-11-19T01:26:30.487Z

I - Digression, aggression

The perspective arrived at in this sequence is almost aggressively third-person. Sure, I allow that you can have first-person stories about what you value, but these stories only rise in prominence to an outside observer if they're supported by cold, hard, impersonal facts. What's more, there can be multiple first-person stories that fit the data, and these different stories don't have to be aggregated as if we're under Bayesian uncertainty about which one is the One True story, they can all be acceptable models of the same physical system.

This can be unintuitive and unpalatable to our typical conception of ourselves. The assumption that there's a unique fact of the matter about what we're feeling and desiring at any given time is so convenient as to be almost inescapable. But if knowing the exact state of my brain still leaves room for different inferences about what I want, then our raw intuition that there's one set of True Values is no foundation to build a value learning AI on.

This sequence, which is nominally about Goodhart's law, has also been a sidelong way of attacking the whole problem of value learning. Contrary to what we

might have intended

when we first heard about the problem, the solution is not to figure out what and where human values Truly are, and then use this to do value learning. The solution is to figure out how to do value learning without needing True Values at all.

If you are one of the people currently looking for the True Values, this is where I'm calling you out. For your favorite value-detection procedure, I'd suggest you nail down how this procedure models humans, and why you think that way is good. Then think of some reasonable alternatives to compare it to. I'm happy to provide specific suggestions or argue with you in the comments.

II - Summing up

With that out of the way, maybe I can get around to where I intended to start this post, which was a summary of the key points, followed by a discussion of how they might apply to actually building value learning AI.

Here are the ideas of this sequence so far, rearranged into a different structure - one which wasn't possible before because each idea needs so much background:

Goodhart's law got applied to value learning through a combination of reasoning from examples and theoretical arguments. (

2

)

Our examples of Goodhart's law are mostly cases w

...
Reaction and Reply to Sasha Chapin on Bad In-group Norms
by Nicholas / Heather Kross — 2021-11-19T01:13:32.946Z
Link: https://www.thinkingmuchbetter.com/main/intelligent-ingroup-reaction/

Epistemic status: stream-of-consciousness

Earlier today, my co-blogger Devin posted Sasha Chapin's article, "Your Intelligent, Conscientious In-group Has Bad Social Norms Too" to a mutual Discord server. He specifically wanted to know our friend group's thoughts on it (our friend group leans rationalist/EA). I started reading it, and posted lots of thoughts and replies I had on it. They seem valuable, or at least postable-as-is (in stream-of-consciousness), so here they are:

And when they burn out, it is a major loss. And the burnouts can be dramatic—like, quit your life, do nothing for years, have no grounding principles whatsoever, eventually hallucinogen yourself back into self-acceptance. (That’s a reasonable way to spend time, but there’s probably a healthier middle way that doesn’t involve total personal collapse.)

NGL, that last line was kinda funny.

I relate hard to the paragraph that starts "Sometimes—often—these forbidden thoughts/actions...", because that's the precise situation I've long been in, I'm only now trying to get out of it in a healthy way, it's toxic, nobody wrote it down, and people in the community have been supportive of me doing it when I basically ask.

and systematizing being preferred inappropriately above all other modes of thinking and feeling.

Systematizing is more valuable than other modes of thinking, when it's available. Intuition is all we have in some situations, so we shouldn't ignore it. And systematizing is certainly not the only or best "mode of feeling" (almost seems like a category error). Add in second-order uncertainty over decision theory and morality (and, of course, our own motivations), and the basic broader idea rings true, even if I disagree with the specific wording. (This is a problem I have with other parts of the article).

when, in fact, you just like social drama because watching humans freak out is fun.

I counter the "this-isn't-smart-enough" norm with my inborn "addiction-to-some-types-of-YouTube-drama" norm. That's totally my real justification for developing this addiction.

rather than, y’know, vroom vroom fun thing go fast.

Getting diagnosed with ADHD was a big step in accepting myself at least somewhat more than I would naively following in-group patterns. (I'd hazard a guess that it's overrepresented in the community, compared to other mental disorders which are themselves overrepresented....
Does anyone know what Marvin Minsky is talking about here? 
by delton137 — 2021-11-19T00:56:25.153Z

I listened to a few Marvin Minsky lectures a few weeks ago. Now I'm trying to go back and find more information on two things he discussed in lecture 3: Cognitive Architectures, given Fall 2011. Sorry to offload this on people here but I have very little idea how to search for more information here (I tried a few things on Google and Google Scholar without any luck)
Here's quote #1 from 1:08:21:
"I think I mentioned Doug Lenat's rule. Some people will assign probabilities to things, to behaviors, and then pick the way to react in proportional to the probability that that thing has worked in the past. And Doug Lenat thought of doing that, but instead, he just put the things in a list. And whenever a hypothesis worked better than another one, he would raise it, push it toward the front of the list. And then whenever there was a choice, he would pick-- of all the rules that fit, he would pick the one at the top of the list. And if that didn't work, it would get demoted. So that's when I became an anti-probability person. That is, if just sorting the things on a list worked pretty well, are probability's going to do much better?"
This sounds fascinating. It's sounds much more computationally feasible than keeping track of probabilities and Bayesian updating. Does anyone have references for further work along these lines? Ie, specifically on keeping a small list of hypotheses, ranked, rather than computing probabilities over a large number of hypotheses? [by the way, Minksy did mention Lenat in Lecture 2, but it was very brief and didn't contain any useful information.] [Trying to search through Douglas Lenat's work is quite a headache because he's got decades of publications, most on esoteric rule-based systems. ]
Here's quote #2 from 1:09:37: (lightly edited for readability)
"Ray Solomonoff discovered that if you have a set of probabilities that something will work, and you have no memory... I think I mentioned that the other day, but it's worth emphasizing, because nobody in the world seems to know it. Suppose you have a list of things, p equals this, or that, or that. In other words, suppose there's 100 boxes here, and one of them has a gold brick in it, and the others don't. And so for each box, suppose the probability is 0.9 that this one has the gold brick, and this one as 0.01. And this has 0.01. Let's see, how many of them-- so there's 10 of the...