trentmkelly/LessWrong-43k huggingface dataset:
Refine's First Blog Post Day
by adamShimi — 2022-08-13T10:23:10.332Z

Yesterday was the first blog post day at Refine. It came after a week of readings, discussions and exercises about epistemology (more concretely shapes of mind, productive mistakes, epistemological vigilance, and mosaics and palimpsests).

I'm excited about how the incubator is going, and about each of these posts.

Here is the list of blog posts by Refine participants:

How I think about alignment
the Insulated Goal-Program idea and goal-program bricks
Steelmining via Analogy
I missed the crux of the alignment problem the whole time
All the posts I will never write

Here's the list of blog posts by friends of Refine and members of my Epistemology Team at Conjecture:

Shapes of Mind and Pluralism in Alignment
The Dumbest Possible Gets There First
An extended rocket alignment analogy
The Dumbest Possible Gets There First
by Artaxerxes — 2022-08-13T10:20:26.564Z

This was written as part of the first

Refine

blog post day.

A sneaking suspicion that I’ve found difficult to shake off while following AI risk discussion is that concerns regarding superintelligent AI, while clearly valid and important, have seemed to me to be very plausibly jumping the gun when not altogether unlikely nearer term AI risks are looming and have potential to wipe us out long before we reach that point, especially under the conditions where we do very little to attempt to mitigate them.

One reason I think I have for this intuition comes from extending an occasionally-repeated idea about human intelligence: that humans are nearly the dumbest possible creature capable of developing a technological civilisation. Part of the reasoning behind this take involves something along the lines of invoking how difficult it is to be wading against entropy to create sophisticated and powerful and complex intelligent agents.

That is to say, evolution as an optimisation process was very unlikely to produce intelligence and general capability greatly in excess of what was needed for an instantiation of something like “technological civilisation”, because of these two assumptions taken together:

1. The first species evolution produced that crossed the necessary thresholds of capability would be the ones to instantiate it in the world first, regardless of how barely above those thresholds they might have been, and

2. Greater intelligence/capability is in some sense generally more “difficult” to produce or requires stronger optimisation and luckier dicerolls as compared to lesser intelligence/capability.

In a similar way, it seems very possible to me that we might have no particular need to worry about being wiped out by the consequences of our creating unaligned superintelligent beings far beyond our comprehension, if only because I expect that significantly less sophisticated AI systems we create that nevertheless have just enough capability juice to meet the threshold to do the job, will indeed do so first.

Of course, it might be the case that there are large discontinuities in the relationship between difficulty of creation and capability of AI. For example, if it happens to be relatively easy to fall into gravity wells of classic fast-takeoff style recursive self-improvement in the landscape of possible AI designs, where we very suddenly happen to produce signi...
I missed the crux of the alignment problem the whole time
by zeshen — 2022-08-13T10:11:24.826Z

This post has been written for the first

Refine

blog post day, at the end of the week of readings, discussions, and exercises about epistemology for doing good conceptual research. Thanks to Adam Shimi for helpful discussion and comments.

I first got properly exposed to AI alignment ~1-2 years ago. I read the usual stuff like Superintelligence, The Alignment Problem, Human Compatible, a bunch of posts on LessWrong and Alignment Forum, watched all of Rob Miles’ videos, and participated in the AGI Safety Fundamentals program. I recently joined Refine and had more conversations with people, and realized I didn’t really get the crux of the problem all this while.

I knew that superintelligent AI would be very powerful and would Goodhart whatever goals we give it, but I never really got how this relates to basically ‘killing us all’. It feels basically right that AIs will be misaligned by default and will do stuff that is not what we want it to do while pursuing instrumentally convergent goals all along. But the possible actions that such an AI could take seemed so numerous that ‘killing all of humanity’ seemed like such a small point in the whole actionspace of the AI, that it would require extreme bad luck for us to be in that situation.

First, this seems partially due to my background as a non-software engineer in oil and gas, an industry that takes safety very very seriously. In making a process safe, we quantify the risks of an activity, understand the bounds of the potential failure modes, and then take actions to mitigate against those risks and also implement steps to minimize damage should a failure mode be realized. How I think about safety is from the perspective of specific risk events and the associated probabilities, coupled with the exact failure modes of those risks. This thinking may have hindered my ability to think of the alignment problem in abstract terms, because I focused on looking for specific failure modes that I could picture in my head.

Second, there are a few failure modes that seem more popular in the introductory reading materials that I was exposed to. None of them helped me internalize the crux of the problem.

The first was the typical paperclip maximizer or ‘superintelligent AI will kill all of us’ scenario. It feels like sci-fi that is not grounded in reality, leading to me failing to internalize the point about

u

...
Shapes of Mind and Pluralism in Alignment
by adamShimi — 2022-08-13T10:01:42.102Z

This post is part of the work done at Conjecture.

This post has been written for the first Refine blog post day, at the end of a week of readings, discussions, and exercises about epistemology for doing good conceptual research.

I have recently presented my model behind the Refine incubator that I'm running. Yet in the two weeks since this post was published, multiple discussions helped me make legible an aspect of my intuitions that I didn't discuss in this post: the notion of different "shapes of mind".

There are two points to this intuition:

Different people will have different "shapes of mind" — ways of revealing hidden bits of evidence from the world;
And alignment is the kind of hard problem where the bits of evidence are dispersed, such that there's no one-trick that is enough.

I've given my current best model of the different forms of pluralism and when to use them in another recent post. What I want to explore here is the first point: this notion of shape of mind. For that, let's recall the geometric model of bits of evidence I introduced in Levels of Pluralism.

We have a high-dimensional space with objects in it. The space is the problem and the objects are bits of evidence.
Because we suck at high-dimensional geometry, we use frames/perspectives that reduce the dimensionality and highlight some aspects of the space. These are operationalizations.
There are clusters of bits of evidence in the space (whether they are rich or poor). These clusters are veins of evidence.

Here the shapes of mind are favored operationalizations — that is, the favored low-dimensional compression of the high-dimensional space where the bits of evidence lie. More precisely, a shape of mind is a cluster of "close" such transforms.

What makes someone have a given shape of mind?

(Education) One of the most obvious I've observed on how people tackle problems come from their background. For an alignment example, John tackle problems like a statistical physicist whereas Paul tackles problem like a theoretical computer scientists, leading to very different perspectives: True Names vs Building-Breaker.
(Knowledge) What you know influences your shape of mind, since you can see and link more things. But when I'm talking about knowing here, I mean the kind of deep knowledge that framing exercises are supposed to provide.[1]
(Past Life) This one feels easily missed by most peopl

...
How I think about alignment
by Linda Linsefors — 2022-08-13T10:01:01.096Z

This was written as part of the first

Refine

blog post day. Thanks for comments by Chin Ze Shen, Tamsin Leake, Paul Bricman, Adam Shimi.

Magic agentic fluid/force

Edit: "Magi agentic fluid" = "influence" (more or less). I forgot that this word existed, so I made up my own terminology. Oops! To my defence, before writing this post I did not have a word, just a concept, and a sort of visualisation to go with it in my head. My internal thoughts are majority non-verbal. So when trying to translate my thoughts to words, I landed on the term that best described how this concept looked in my mind.

Somewhere in my brain there is some sort of physical encoding of my values. This encoding could be spread out over the entire brain, it could be implicit somehow. I’m not making any claim of how values are implemented in a brain, just that the information is somehow in there.

Somewhere in the future a super intelligent AI is going to do some action.

If we solve alignment, then there will be some causal link between the values in my head (or some human head) and the action of that AI. In some way, whatever the AI does, it should do it because that is what we want.

This is not purely about information. Technically I have some non-zero causal influence over everything in my future lightcone, but most of this influence is too small to matter. More relevant, but still not the thing we want is deceptive AI. In this case the AI’s action is guided by our values in a non-neglectable way, but not in the way we want.

I have a placeholder concept which I call magic agentic fluid or magic agentic force. (I’m using “magic” in the traditional rationalist way of tagging that I don’t yet have a good model of how this works.)

MAF is a high-level abstraction. I expect it to not be there when you zoom in too much. Same as how solid objects just exist at a macro scale, and if you zoom in too much there are just atoms. There is no

essence

, only shared properties. I think the same way about agents and agency.

MAF is also in some way like energy. In physics energy is a clearly defined concept, but you can’t have just energy. There is no energy without a medium. Same as how you can’t have speed without there being anything that is moving. This is not a perfect analogy.

But I think that this concept does point to something real and/or useful, and that it would be valuab...
Steelmining via Analogy
by Paul Bricman — 2022-08-13T09:59:28.866Z
Link: https://paulbricman.com/hypothesis-subspace/

This post has been written for the first Refine blog post day, at the end of a week of readings, discussions, and exercises about epistemology for doing good conceptual research. Thanks Adam Shimi, Linda Linsefors, Dan Clothiaux for comments.

To steelmine (as per Tamsin): to intentionally look for what productive mistakes a research direction is hinting at.

There are quite a few analogies used regularly across alignment. Some popular ones include "prompts are like programs" or "interpretability is like neuroscience on ML models." While no analogy is perfect, some are useful, as they help us recycle years of intellectual labor if we get the translation key right. This opportunity is particularly relevant if you side with relatively short timelines, because you can use it to quickly scan through entire regions of hypothesis space for (part of) a silver bullet.

This is where I'm coming from in my work at Refine. I want to use the fellowship as an opportunity to investigate a dozen or so themes which connect prosaic alignment to other tangentially related disciplines and see which ones yield productive mistakes. This top-down approach of starting with broad themes and then zooming in on details is explicitly baked into the linked artifact which I'll be using as a sketchpad throughout the program. The left-to-right tiled layout represents branches exploring various assumptions, technical details, and failure modes, while allowing (and welcoming) targeted feedback.

That said, here are a few handpicked excerpts from said artifact to give you a taste:

Memetic Colonies

Based on a new formalism from argumentation theory, we might be able to turn large language models into Petri dishes for cultivating belief systems. In such a memetic pressure cooker, a host of belief systems would be incentivized to generally maintain internal consistency while systematically undermining each other. In this framework, we could develop specialized "lab equipment" to help us (1) extrapolate belief systems, (2) accelerate conceptual research, and (3) synthesize a memetic variant of the GPU-burner. For instance, an Overton probe could be used in tandem with a Gould fork to...

Parametric Ecologies

Life is arguably the primordial world-optimizer, with biotic factors being known to have caused major shifts in the abiotic world to suit their needs: oxygen-rich atmosphere, fertile soils, etc. We find

...
Appendix: Jargon Dictionary
by CFAR!Duncan — 2022-08-13T08:09:16.387Z

80-20

To “80-20” something is to obtain most of the result (80%) with only a small proportion of the work (20%). This expression originates with the Pareto principle which states that for many events, 80% of the effects come from 20% of the causes (e.g. most of a com- pany’s sales come from a small number of its clients).

Adaptive problem

A problem whose solution contains steps or methods that are unknown or uncertain, often requiring experimentation, novel strategies, or entirely new ways of thinking. Contrast with “technical problems,” which may be equally difficult but whose difficulty lies in the execution of known or knowable processes.

Affect

One’s emotional state or disposition, especially as evidenced by one’s body language, facial expression, word choice, and tone of voice.

Affordance

An opportunity or potential-for-action arising from a given context; a door handle creates an affordance for pulling. In particular, it is a genuine or felt opportunity—while there may be no physical difference between picking up a pen on one’s own desk, one’s coworker’s desk, one’s manager’s desk, or the CEO’s desk, each of those contexts provides a different degree of affordance.

Againstness

The quality of resistance to information, often caused by strong emotion or sympathetic nervous system activation, but also resulting from conflicts with one’s identity, inherent biases, and other entangled beliefs.

Agency

The property of both having and exercising a capacity for relevant action; one’s ability to meaningfully affect the world around oneself and effectively move toward achieving one’s goals. In particular, agency implies the ability to move beyond default patterns and cached answers, and to think and act strategically. Because it is not possible to achieve agency in a total sense, CFAR often refers to being more or less “agenty.”

Alief

A deeply-felt belief, sometimes called an “anticipation,” emerging naturally from one’s mental model of the universe. Sometimes contradictory to one’s professions, which are explicitly stated beliefs—for instance, one might profess/believe that a high wooden bridge over a canyon is entirely safe, and yet reveal an anticipation/alief of danger by tensing, moving gingerly, and refusing to look down.

Aumanning

From Aumann’s agreement theorem, which demonstrates that two rational agents with the same back- gro...
Appendix: Hamming Questions
by CFAR!Duncan — 2022-08-13T08:07:09.447Z

Author's note: these questions were usually the prework for an activity called "Hamming Circles," in which four or five people sat down and spent 20-30 minutes with each person's Hamming Question at the center of a slow, intimate discussion. The Hamming Circles activity is not fully described in this sequence, but these are the prompts given to participants before engaging in it.

Richard Hamming was a mathematician who worked at Bell Labs during the 1940s-1970s. He had a habit of sitting down with scientists in other fields and asking them “What are the important problems of your field?” After they explained their field's most important open problem, he would ask them: why aren't you working on that?

Though somewhat annoying, these questions are also extremely helpful for focusing one's attention on problems that are worthy of one's efforts. (See Hamming's talk “You and Your Research” for more.)

It is valuable to pose an analogous question to oneself: What are the most important problems in my life, and what's stopping me from working on them?

Prompt 0: Initial thoughts

Did anything come up already? Is anything staring you in the face right now? Now that you have the idea of a “Hamming question” in mind: is anything obvious as your most important problem?

You might also try: Picturing another person, identical to you, and asking what their main bottleneck is.

Prompt 1: Rate-limiting step

The speed of a chemical reaction is determined by the speed of the slowest step—the rate-limiting step. What is yours? Or: what's your bottleneck? Is there a problem where solving it would be the equivalent of “wishing for more wishes?”

Prompt 2: What are you not allowed to care about?

Or: Is there some good outcome that you generally don't think about because it's too big to picture? Or too impossible?

Conversely, what do you have to care about, even though you'd really rather not/it doesn't feel like where your heart and soul actually want to be?

Prompt 3: Genre-savviness

When you're reading a novel, sometimes it seems like the book is dragging/stuck because there is an obvious thing that the character needs to do next in order to advance the plot (e.g. clearly she needs to go talk to the magician—can't she just do it already?). If your life is a novel: what is that obvious next thing?

Prompt 4: What are you already pursuing, badly, in a convoluted/distorte...
Building a Bugs List prompts
by CFAR!Duncan — 2022-08-13T08:00:39.838Z

Prompt 0:

Think about the way computer programmers talk about “bugs” in the program, or “feature requests” that would make a given app or game much better. Bugs are things-that-are-bad: frustrations, irritations, frictions, problems. Feature requests are things-that-could-be-great: opportunities, possibilities, new systems or abilities.

Write down as many “bugs” and “feature requests” as you can, for your own life.

Prompt 1:

A genie has offered to fix every bug you’ve written down, and to give you every feature you’ve requested, but then it will freeze your personality—you won’t be able to grow or add or improve anything else.

Hearing that, are there other things you’d like to write down, before the genie takes your list and works its magic?

Prompt 2:

Imagine someone you know well, like your father or your best friend or a longtime boss or colleague or mentor. Choose someone specific, and really imagine them sitting right there beside you, looking over your shoulder, and reading your list. You say to them, “Look! That’s everything!” and they’re skeptical. What things do they think you’ve forgotten, and should add?

Prompt 3:

Do a mental walkthrough of your day and/or week, starting from waking up on Monday and going step by step through all of your routines and all of the places you go, things you do, and people you interact with. Look for snags or hiccups, as well as for exciting opportunities. Be as thorough as you can—often, actually taking the day or week step by step will cause you to remember things you hadn’t thought of.

Prompt 4:

Take a nice, long, slow thinking pause for each of the following broad domains, one at a time (at least ten seconds and maybe as many as thirty):

Work/career
Education/learning/curiosity
Family
Money/finances
Exercise
Food/diet
Sleep
Scheduling and time management
Hobbies
Friends
Romance
Communication
Social interaction
Emotions
Boundaries

Prompt 5:

Now read this sentence, nice and slow, letting it percolate (maybe read it two or three times, or pause in the middle if it sparks thoughts, and then come back and read the rest).

For this last prompt, think of ways you want the world itself to be different, opportunities you haven’t seized (and what’s kept you from seizing them), problems you haven’t solved (and what makes them sticky), things you knew would go wrong (and

...
Gradient descent doesn't select for inner search
by Ivan Vendrov — 2022-08-13T04:15:19.373Z

TL;DR: Gradient descent won't select for inner search processes because they're not compute & memory efficient.

Slightly longer TL;DR: A key argument for mesa-optimization is that as we search over programs, we will select for "search processes with simple objectives", because they are simpler or more compact than alternative less dangerous programs. This argument is much weaker when your program search is restricted to programs that use a fixed amount of compute, and you're not optimizing strongly for low description length - e.g. gradient descent in modern deep learning systems. We don't really know what shape of programs gradient descent selects for in realistic environments, but they are much less likely to involve search than commonly believed.

Note on terminology (added in response to comments): By "search" I mean here a process that evaluates a number of candidates before returning the best one; what Abram Demski calls "selection" in Selection vs Control . The more candidates considered, the more "search-like" a process is - with gradient descent and A* being central examples, and a thermostat being a central counter-example.

Recap: compression argument for inner optimizers

Here's the argument from Risks From Learned Optimization: [emphasis mine]

In some tasks, good performance requires a very complex policy. At the same time, base optimizers are generally biased in favor of selecting learned algorithms with lower complexity. Thus, all else being equal, the base optimizer will generally be incentivized to look for a highly compressed policy.

One way to find a compressed policy is to search for one that is able to use general features of the task structure to produce good behavior, rather than simply memorizing the correct output for each input. A mesa-optimizer is an example of such a policy. From the perspective of the base optimizer, a mesa-optimizer is a highly-compressed version of whatever policy it ends up implementing: instead of explicitly encoding the details of that policy in the learned algorithm, the base optimizer simply needs to encode how to search for such a policy. Furthermore, if a mesa-optimizer can determine the important features of its environment at runtime, it does not need to be given as much prior information as to what those important features are, and can thus be much simpler.

and even more forceful phrasing from John Wentwort...