trentmkelly/LessWrong-43k huggingface dataset:
The metaphor you want is "color blindness," not "blind spot."
by Duncan Sabien (Inactive) — 2022-02-14T00:28:55.119Z

It's genuinely possible that the title is all you need; I was tempted to write nothing further. Feel free to take ten seconds and see if it's already clicked. If not, read on.

When something is in your literal blind spot, it's invisible to you, but your brain stitches together everything else around it to make you think that you're seeing a complete picture.

Reality:

Blind spot:

Perception:

We often use "blind spot" as a metaphor to gesture toward things we are unaware of, while also being at least somewhat unaware of our unawareness—we know that something fishy is going on, but we can't quite get our eyes on it.

e.g. "I think I might have a blind spot when it comes to status dynamics."

The thing about status dynamics, though, is that they aren't in one spot. There isn't a whole world that is being fully and accurately perceived, except for one blank space that's being glossed over.

Instead, what's usually going on (at least in my experience) is that the person can see everything, but there's some crucial component of the picture that they are unable to process or comprehend.

What this looks like, in practice, is an inability to distinguish two things which are very, very different, à la red-green color blindness:

"Look," says Alexis. "Look at the beautiful contrast."

Blake hesitates. "...you mean between the trees and the sky?"

Being (metaphorically) color-blind to something can be deeply frustrating. You keep pushing the X button, and very different things keep happening.

e.g. you are learning to play a Formula 1 racing simulator, and it feels like you did exactly the same thing on exactly the same curve both times, but one time you spun out and crashed and the other time you smoothly navigated right through.

e.g. you are repeating back to the native French speaker exactly the sounds she said to you, and sometimes getting nods and smiles and other times getting sympathetic winces.

e.g. you asked your romantic partner what you should have said, instead, to avoid this huge disagreement, and the words they wished you'd said are literally equivalent in meaning, that's the exact same sentiment, what the hell is going on, here?

It's straightforwardly analogous to being literally color-blind, where you might try on one t-shirt, and hear snickers and giggles and see people trying not to laugh, but then you swap it out for a t-shirt ...
Returns on cognition of different board games
by Morpheus — 2022-02-13T20:40:49.163Z

Pure curiosity

I recently realized why sometimes I find board games/puzzles boring after a while:

The returns on "cognition" for these games has a sharp cutoff.
While this seems a bit obvious in retrospect, I haven't seen anyone talk about "returns on cognition" in any other context apart from thinking about AI-Doom scenarios.

One thing that always made me a bit frustrated with different board games is that a lot of them get relatively boring once you've played a few times, and there are a lot of heuristics that are obviously close to optimal play. Other games really encourage me to "actually think", because there are highly non-obvious situations coming up all the time.

One example for a game that is boring, once you have figured out you always need to put the big numbers in the corner, is 2048. But there is a close cousin of this game Dive where you can merge the numbers if they are multiples of each other. For Dive I don't have any good heuristics like this, and it feels more engaging for that reason.

Interesting concept I stumbled upon while googling:

game complexity: the Wikipedia-article goes into a lot of different measures for how hard it is to play a game optimally
Whence the sexes?
by Richard_Ngo — 2022-02-13T20:40:01.753Z

There are many explanations of the evolutionary value of sex in terms of gene exchange (I particularly like this one). But these don’t explain the evolutionary value of having sexes: of the differentiation between males and females. A species of hermaphrodites would get all the genetic benefits of sex, but without the massive cost of half its population being unable to bear offspring. On average, each individual could have twice as many offspring, unless other problems arose. And indeed, most plants are hermaphroditic - but only a few animals. So why aren’t most animals hermaphrodites? A quick search doesn’t turn up any widely accepted answer, so I’ve brainstormed a few possibilities. I may well be missing something obvious; if so, let me know.

Resource cost of being hermaphroditic. If there’s a strong division of labour between males and females, then maybe it’s harder for hermaphrodites to gather enough resources to support offspring. But in many species the males contribute little in terms of resources - e.g. in orang-utans, who are very solitary.
Developmental or metabolic costs of being hermaphroditic. Maybe it’s just a very expensive adaptation. But this seems unlikely to be the main factor - the relevant baseline is the existence of males at all, which is a huge energy cost.
Difficulty of evolving hermaphroditism. Maybe this is just hard for evolution to find? But non-reproductive hermaphroditism seems like it arises via mutations pretty frequently, so I’d be surprised if reproductive hermaphroditism were unachievable by evolution. And in fact there are a few hermaphroditic species - so why haven’t they spread much more widely?
Difficulty of fixating hermaphroditism. A hermaphrodite in a species without many hermaphrodites is likely not as attractive to females as most males are, nor as fertile as most females are. So maybe, even after arising, the trait will be selected against. But on timeframes where sexual desires can themselves evolve, all else equal we should expect stabilising sexual selection towards the best combination of fertility and attractiveness. E.g. it would be undesirable to be impregnated by overly masculine conspecifics, because the resulting offspring would be less fertile themselves. So this adds a bit more difficulty to reaching the hermaphroditic equilibrium, but doesn’t answer the core question of why that’s a less fit equilibrium

...
Question 4: Implementing the control proposals
by Cameron Berg — 2022-02-13T17:12:41.702Z

Whose problem is it, really?

Let’s imagine that AGI safety research has been remarkably successful: researchers have (1) correctly identified the most plausible learning architectures that AGI will exhibit, (2) accurately anticipated the existential risks associated with these architectures, and (3) crafted algorithmic control proposals that provably mitigate these risks. Would it finally be time to pack up, go home, and congratulate ourselves for saving the species from an extinction-level event? Not if there isn’t a plan in place for making sure these control proposals actually end up getting implemented!

Some safety researchers might protest: “but this is not our job! Our goal is to figure out technical solutions to the AGI safety control problem. If we do this successfully, we’re done—let’s leave the implementation details to the AI governance people.” I am definitely sympathetic to this sort of thought, but I still think it isn’t quite right.

Throughout the hierarchical framework put forward in this sequence, I have worked to demonstrate why any end-to-end AGI safety research agenda that focuses solely on the AGI and not at all on the humans who are supervising it, assigning it goals, tweaking its internal dynamics, etc. is almost definitely going to fail. I see this final question in the hierarchical framework—how are we going to get AI engineers to use the correct control mechanisms?—as arguably the most important extension of this principle. Successful control proposals will only mitigate AGI-induced existential risks—the putative goal of AGI safety research—if they are actually used, which means that it is incumbent on AGI safety researchers to devote serious effort and attention to the problem of implementing their best safety proposals.

It is important to get clear about the nature of the implementation problem. Steve Byrnes frames it well: “The AI community consists of many thousands of skilled researchers scattered across the globe. They disagree with each other about practically everything. There is no oversight on what they’re doing. Some of them work at secret military labs.” The implementation problem is hard because it requires (at least) coordinating behavior across all of these actors. It is also conceivable that a ‘lone wolf’—i.e., some independent programmer—could also succeed in building an AGI, and we would definitely want to make s...
Knowing
by LoganStrohl — 2022-02-13T07:53:59.031Z

I’m going to draw some practical distinctions among types of knowledge, as an attempt to tap into your intuitions and avoid having to give some convoluted, ivory-tower definition of the word. I request that you try not to get distracted by where I’ve drawn my distinctions—the precise placement of the borders is not the point of the exercise. The point of the exercise is to shine your attention on the richness, depth, and complexity of your capacity to know—that the word “know” means more than one thing.

Let’s begin with a little formative assessment.

— Do you know what comes out of your kitchen faucet?

— Do you know what glaciers are made of?

— Do you know what those big white fluffy things in the sky are?

If so, then you are familiar with water. When someone talks about it, you’re not completely lost.

— Do you know at what temperature water boils?

— Do you know the atomic composition of an ordinary water molecule?

— Do you know what percentage of your body’s volume is water?

If so, then you know some facts about water. Your concept of water contains (at minimum) a few isolated pieces of accurate information.

— Do you know the way water sounds when it pours into a cup?

— Do you know how water feels when it runs over your skin?

— Do you know the look of a stream's surface as it glitters in the sun?

If so, then you are able to identify water when you encounter it in real life. You have direct, experiential data. You are able to predict how various encounters with water will impact your senses, and you probably recognize water when those sensations occur (at least sometimes).

— Do you know what happens if you leave a beer bottle in the freezer overnight?

— Do you know how a water mill grinds grain into flour?

— Do you know why it rains?

If so, then you probably have at least one model[1] of water.

Whether or not that model is explicit, it includes enough structure that you can predict the behavior of water in various situations, even if those situations are outside of your own direct experience. Your model might be rudimentary, in which case the above questions probably produced hesitation or “sort of?” and you’d maybe only be able to produce a short paragraph in response to each. Or your model might be rich and deep, in which case your “yes” was confident, and you could in principle write multiple essays on the subject.

— Do you know how to swim?

— Do you know wh...
Intro to Naturalism: Orientation
by LoganStrohl — 2022-02-13T07:52:03.503Z

A note on how to approach this sequence:

If you were exactly like me, I would ask you to savor this sequence, not scarf it. I would ask you to approach each of these essays in an expansive, lingering, thoughtful sort of mood. I would ask you to read them a little bit at a time, perhaps from a comfortable chair with a warm drink beside you, and to take breaks to make dinner, sing in the car, talk to your friends, and sleep.

These essays are reflections on the central principles I have gradually excavated from my past ten years of intellectual labor. I am a very slow thinker myself; if you move too quickly, I expect we’ll miss each other completely.

There’s a certain kind of thing that happens when a person moves quickly, and relies a lot on their built-up structures—their familiar, tried-and-true habits of thought and perception. There is a different kind of thing that happens when a person can step back and bring those very structures into view, rather than standing atop them. I'm hoping for the latter.

But since you’re not exactly like me, there might be a better way to approach this sequence, in your particular case, than the exact one I’d suggest to myself. I hope you’ll take a moment to check.

What matters to me is not how fast you read, or how many sittings it takes; what matters is that you create for yourself enough space to explore, to observe the real world beyond all these words, to watch how your own thoughts and experiences unfold in dialog with mine. Any method that allows you to maintain that kind of space as you read is perfect, as far as I’m concerned.

“Naturalism” is a label for a conceptual framework, investigatory discipline, and semi-formalized way of looking at and learning about the world. I’ve been developing and teaching naturalism for the past couple of years, if you start counting on the day I chose the term, or since 2013, if you take a more historical perspective. I’ve made some

relevant

content

available

, but I’ve had trouble writing a straightforward introductory post.

The reason for this, as far as I can tell, is that the naturalist perspective is suspicious of categories, projections, and preconceptions, and seeks to move closer toward (relatively) unfiltered, direct observations. It’s specifically a frame-breaking and frame-escaping discipline, so it’s hard to describe in frame-terms without being importantly misleading.

I arde...
Abstractions as Redundant Information
by johnswentworth — 2022-02-13T04:17:30.013Z

What is it that makes the concept of “pencil” a good abstraction?

One way to frame it: there are lots of “pencils” in the world - lots of little chunks of the world which I could look at which all contain information about pencils-in-general. Many different places from which I could gain information about “pencils”, and many different places where I can apply that information to make predictions. If I don’t know that pencils are usually made of wood with a graphite core, there are many different chunks of the world which I could look at to gain that information. If I do know that pencils are usually made of wood with a graphite core, there are many different chunks of the world where I could apply that information to predict the materials from which something is made.

Alternatively, consider a gear, spinning in a gearbox. What makes the gear's rotational speed a good abstraction? Well, there are lots of little patches of metal comprising the gear. If I don't know the gear's rotation speed, I could precisely estimate it from any of the little patches. If I do know the gear's rotation speed, I can use it to predict the rotation of many different little patches of metal within the gear.

This seems like an intuitively-reasonable notion of what makes abstractions “good” in general: there are many different places from which we can learn the information, and many different places where we can apply the information to make predictions. In other words, a good abstraction is highly redundant: it appears in many different places, and in any of those places we can use the abstract information to make predictions and/or gain more information about the abstraction in general.

In this post I’ll sketch out one way to formalize this idea mathematically, and show that it’s equivalent to the formalization of

abstraction as information-at-a-distance

. In particular, in the gear example: conditional on the highly redundant information (i.e. the overall rotation of the gear), the low-level rattling of far-apart chunks of metal is statistically independent. More generally, redundancy yields the same high-level abstract information as the

Telephone Theorem

, but in a mathematically-cleaner form, and without the need for different summaries between different variables.

Meta

Advice for readers: I try to keep the more-dense math in a few specific sections and the appendices, which can all be...
The innocent self
by Richard_Ngo — 2022-02-13T03:10:01.970Z

I loved Joe Carlsmith’s blog post on "the innocent gene". It’s a post about perspective shifts. One of them occurs when, as Dawkins suggested, we shift from thinking about organisms as agents, and start to think of them as vehicles for their “selfish” genes. From the selfish gene perspective, a lion is armour which the genes use to propagate themselves. In making this shift, we implicitly transfer agency from the lion to the genes: we conclude that the genes “want” the lion to hunt down antelope. That’s predictively useful, but Joe observes that in doing so we often implicitly assign moral agency to the genes - as if we should judge them for the outcomes they give rise to, like we judge humans. The word “selfish”, with its loaded connotations, is a good example of this moral transfer.

So Joe offers a third perspective: that of the innocent gene. From this perspective, genes have the property of building up lions around them - but not because that’s something they “want”, just because that’s how the world interacts with them. In a particularly evocative passage, Joe explains:

I remembered that the genes are passive, innocent patterns, rather than agents — that they had no desire to survive, or knowledge of the snarls and claws that had accumulated around them. The lion was still armor; but armor for something that doesn’t want defending. The sleeping beauties inside it did not ask for a lion. Yet over evolutionary time, a lion has grown up around them regardless — a giant, ferocious eddy, swirling around something that snagged.

That’s made me think about an analogous perspective shift in the way we view ourselves and others. One view - the most standard one - sees people primarily as agents deliberately pursuing their goals and ends. That’s not a bad approximation in many contexts, but when you look more closely at the emotions behind people’s behaviour, it starts to break down. We often have contradictory drives, tugging us in different directions. We can think of our behaviour as being shaped by these needs, in an analogous way to how the lion’s behaviour is shaped by its genes. But then, as with the lion, you can make the second shift - to seeing those needs not as “selfish”, but rather as innocent entities that happen to have built up ways to influence behaviour.

I can’t remember if, when I started writing this blog post last year, I already knew about th...
Idea: build alignment dataset for very capable models
by Quintin Pope — 2022-02-12T19:30:34.626Z

In Training language models to follow instructions with human feedback, OpenAI showed that we can significantly improve the alignment of language models through reinforcement learning targeting a model of human preferences. I.e., they train GPT-3 to generate text that gets a high score from a learned reward model.

To train the reward model, they first use a GPT-3 model to generate many possible completions of different prompts. Then, human labelers generate relative rankings of different completions of the prompts. Finally, they train the reward model to predict the labelers' rankings of the prompts.

This approach relies on the reward model to ensure that the language model learns our preferred behavior. I don't think it will scale indefinitely to vastly superhuman systems. However, something similar may scale to somewhat superhuman systems.

The issue is that any fine-tuning-based approach requires a reward model that's able to provide effective feedback on the AI's proposed course of action. Unfortunately, none of the data currently available to train such a reward model pertain to the case where the AI has superhuman capabilities. They seem more focused on basic morality issues or where the model is assumed to have limited capabilities (e.g., the ETHICS dataset or the dataset OpenAI generated for their reward model).

In contrast, I'm imagining a dataset where the AI is presented with a scenario, prompt or request, then there are descriptions of multiple possible behaviors the AI could produce in response, most of which involve the AI demonstrating some form of superhuman capability. Each scenario comes with rankings of the relative desirability for each described behavior. Features of desirable behaviors for very capable models might include:

The AI asking clarifying questions of the humans about what the humans want.
The AI explaining its current plans to humans and describing the plan's potential consequences.
The AI frequently getting feedback from humans about whether to continue its current course of actions.
The AI being incredibly honest, even when doing so annoys the humans or complicates the AI's attempts to do what the humans asked.
The AI warning us about potential failures of our alignment techniques

The dataset should also include negative behavior examples. E.g.,

The AI lying or giving technically correct but misleading descriptions of

...
Question 3: Control proposals for minimizing bad outcomes
by Cameron Berg — 2022-02-12T19:13:48.075Z

Necessary conditions for successful control proposals

At this point in the framework, let’s stipulate that we have placed our bets for the most plausible learning architecture we would expect to see in an AGI and we have a coherent account of the existential risks most likely to emerge from the learning architecture in question. At last, now comes the time for addressing the AGI safety control problem: how are we actually going to minimize the probability of the existential risks we care about?

As was the case previously, it would be far too ambitious (and inefficient) for us to attempt to enumerate every plausible control proposal for every plausible existential risk for every plausible AGI learning architecture. In place of this, I will focus on a framework for control proposals that I hope is generally risk-independent and architecture-independent—i.e., important control-related questions that should probably be answered regardless of the specific architectures and existential risks that any one particular researcher is most concerned about.

Of course, there are going to be lots of control-related questions that should be answered about the specific risks associated with specific architectures (e.g., “what control proposals would most effectively mitigate inner-misalignment-related existential risks in a human-level AGI using weak online RL?”, etc.) that I am not going to address here. This obviously does not mean I think that these are unimportant questions—indeed, these are probably the most important questions. They are simply too specific—and number too many—to include within a parsimonious theoretical framework.

Specifically, we will build from the following foundation:

In other words, we are looking for whatever prior conditions seem necessary for ending up with ‘comprehensive alignment’—a set-up in which an AGI is pursuing the right goal in the right way for the right reasons (i.e., AGI alignment + human alignment). Again, note that whatever necessary conditions we end up discussing are almost certainly not going to be sufficient—getting to sufficiency will almost certainly require additional risk- and architecture-specific control proposals.

Within this ‘domain-general’ framework, I think two control-related concepts emerge as most important: interpretability and corrigibility. These seem to be (at least) two background conditions that are a...