trentmkelly/LessWrong-43k huggingface dataset:
Ritual as the only tool for overwriting values and goals
by mrcbarbier — 2023-01-04T11:11:19.009Z

Updated on Jan 14, 2023 to address comments by kithpendragon, Richard_Kennaway, DonyChristie, Kaj Sotala and others.

Introduction

Here are two reasonable sentences:

1) “The brain makes predictions.”

2) “Rituals teach us values.”

You will probably grant me that both are trivially true in their weakest form: sure, this happens sometimes. Your friend predicted Trump would get elected; your neighbor learned Christian values by attending mass every Sunday.

In both cases, this is missing the utter weirdness and significance of taking these sentences seriously.

Predictive processing is what happens when we take “the brain makes predictions”, and make it strong: let’s think of everything the brain does, a galaxy of disjoint phenomena ranging from optical illusions to reflex actions to psychopathology, and try to make sense of it by assuming that (almost) everything is to be understood as prediction.

My hope (here and in years to come) is to do the same with “rituals teach us values”. That is, try to understand everything about rituals, down to the intricacies of why you would burn a cow in Jerusalem and make a cow dung offering in Varanasi, by assuming a strong bijection between ritual and instilling values: (almost) every detail in a ritual serves the purpose of determining the values of human agents, and there is (almost) no other way to do so.[1]

I want to insist on this last point: If ritual is just an act that means something, why can't you use language to convey that meaning more efficiently? If it exists to evoke emotion or belonging, why don't you simply play music or games? None of these common intuitions about rituals allow us to argue that sacrificing a hundred bulls to Zeus is a reasonable way to achieve any goal at all -- and yet, I don't think that these intuitions are entirely wrong either (contra some theorists who see rituals as purposeless memes optimized for self-perpetuation). Explaining why we are ourselves confused in our explanations is part of what I want to achieve.

First, I want to foreshadow the crux of my argument, so that you have a rough idea of where I am going with this.

To claim that ritual is the only way of shaping the values of a human agent, I must argue that everything else, especially the non-ritual aspects of everyday communication and education, is unable or disallowed from doing so. I must also argue that ritual has a...
Causal representation learning as a technique to prevent goal misgeneralization
by PabloAMC — 2023-01-04T00:07:29.649Z

Summary: This is a submission for the goal misgeneralization contest organized by AI Alignment Awards, as well as the third iteration of some slowly improving AI Safety research agenda that I aim to pursue at some point in the future.

Thanks to Jaime Sevilla for the comments that helped sharpen the specific research proposals. Needless to say, I encourage people to take part in these contests and am happy to receive comments or criticisms.

Main proposal

Causal representation learning (CRL) is a set of techniques that can be used to find “high-level causal variables from low-level observations” (Schölkopf, 2021). By causal representation, we mean a causal model (usually Structural Causal Model, SCM) that disentangles the most important features and independent mechanisms of the environment. Reconstructing causal models has been addressed in the literature via encoders and decoders acting on a latent space in which the causal structure is described (Schölkopf, 2022). My proposal is to use these techniques to build robust representations of human preferences. This is challenging but has certain advantages (Shah, 2022):

More diverse training data: The agent may request human demonstrations in settings specially designed to disentangle strongly correlated features, or test and refine the representations of human preferences.

Maintaining uncertainty: SCMs can be seen as generalizations of Bayes models where the information is not only observational but also interventional and counterfactual (Bottou, 2013). SCMs can thus represent uncertainty via exogenous variables and unobserved confounders.

Understanding and improving inductive biases and generalization: Learned causal structures can be used to transport data learned in an environment to a structurally-related but statistically different one. Thus, this technique would naturally give robustness to distribution/concept shifts, if the causal mechanisms remain unchanged.

CRL techniques can also leverage a combination of unsupervised datasets and small amounts of supervised data (Locatello 2019, Locatello 2020). While perfectly correlated features cannot be disentangled, if in the new environment they are no longer so, small amounts of data might be sufficient to quickly adapt the model (Wang, 2021). Finally, having a causal model may enable interpretability, agent incentive, or multi-agent game theoretic ana...
What makes a probability question "well-defined"? (Part II: Bertrand's Paradox)
by Noah Topper — 2023-01-03T22:39:06.660Z
Link: https://naivebayes.substack.com/p/what-makes-a-probability-question-e0e

See also: What makes a probability question "well-defined"? (Part I), Bertrand's Paradox, More on Bertrand's Paradox, and When (Not) To Use Probabilities

(Follow-up to my last post and cross-posted from my new(ish) blog. Sorry for the very long delay, life was crazy. )

I. Introduction

In my last essay, I argued that when we ask for the probability that some event E holds, the very meaning of the question is to ask for our best guess given the information available to us. As such, we can never claim that we have insufficient information to answer a probability question. That is simply the nature of probability. We must give an answer regardless, at least assuming the event E is meaningful at all.

I further argued that when we call a probability question “undefined”, what we really mean is that it’s not clear how to move from the information we do have to a precise numerical answer to the question. But it would be more proper to call such questions computationally difficult, rather than undefined. Let’s apply this reasoning to some more exotic problems, shall we?

II. Bertrand’s Paradox

Bertrand’s Paradox is a famously “undefined” probability problem. The linked video (which I recommend watching) visualizes the problem quite well, so I’ll just briefly describe it in words. Consider choosing a random chord on a circle. What’s the probability that the length of this chord is greater than the side length of an equilateral triangle inscribed in the circle?

To answer this, we need to figure out what we mean by a “random” chord. We haven’t been given any information about the chord’s distribution, but based on what we learned in the last essay, we can use a uniform distribution. The problem lies in what we consider “uniform”. There are many ways to think about this. Three in particular are called out in this paradox.

In the first, we choose two points on the circle uniformly at random, and form their chord. In the second, we choose the midpoint of the chord uniformly at random from within the circle. The third is a bit trickier to state mathematically, but visually is still intuitive. Imagine a circle with a vertical line to the left of it. Gradually shift the line to the right, and consider all the chords it forms as it intersects the circle. We assume each one of these is equally likely. But this only covers vertical chords, so also assume the line is equally l...
"AI" is an indexical
by TW123 — 2023-01-03T22:00:31.089Z
Link: https://aiwatchtower.substack.com/p/ai-is-an-indexical

I recently started a blog called AI Watchtower. I plan to write essays on the latest developments in AI and AI safety. The intended audience of AI Watchtower is mostly people interested in AI who don't have a technical background in it. If you're interested, you can subscribe here. This post is probably old news for most LessWrong readers, though the "indexical" framing is novel (as far as I know).

Linguists use the concept of an indexical: a word that changes meaning depending on the context of the speaker or listener. For example, “today” means January 2nd as I write this, but by the time you read it, it might mean some other day. Another example is the word “I”: when said by me, it means Thomas Woodside; when said by a text-generating AI system, it means something completely different. “Artificial intelligence” is likewise an indexical. Across time, it has referred not to one set of technologies or even approach to technology, but rather to multiple, qualitatively different fields. For reasons I’ll discuss below, knowing that “AI” is an indexical makes it significantly easier to understand it.

“AI” has always been slippery

In university classes, it’s common for professors to explain to students how notoriously difficult it is to make a good definition of artificial intelligence. Practically speaking, “AI” is a slippery term that constantly changes meaning. In the past, those professors like to say, even spreadsheets were considered “AI.” Throughout its history, the term has typically meant something along the lines of “whatever class of methods currently produces outputs most similar to those that humans do through cognitive processes.” It is not one immutable set of technologies and techniques over time. And as we will see, the field is nearly unrecognizable compared with how it was ten years ago.

I think this can sometimes cause confusion among people who are learning about AI for the first time, or who aren’t regularly exposed to the technical aspects. Even many AI classes do not tend to explicitly discuss just how different modern methods are from their predecessors, as the classes tend to be years out of date. In this post I am going to lay out what I see as the key considerations that policymakers and educators should know about the basics of how AI has changed in recent years. In later posts, I hope to write about particular systems and their risks.

“AI

...
An ML interpretation of Shard Theory
by beren — 2023-01-03T20:30:28.830Z

Crossposted from my personal blog

Epistemic Status: I have spent a fair bit of time reading the core Shard Theory posts and trying to understand it. I also have a background in RL as well as the computational neuroscience of action and decision-making. However, I may be misunderstanding or have missed crucial points. If so, please correct me!

Shard Theory has always seemed slightly esoteric and confusing to me — what are ‘shards’, why might we expect these to form in RL agents? When first reading the Shard Theory posts, there were two main sources of confusion for me. The first: why an agent optimising a reward function should not optimise for reward but instead just implement behaviours that have been rewarded in the past?

This distinction is now obvious to me. The distinction between amortised vs direct inference, and shards as cached behaviours falls directly out of amortized policy gradient algorithms (which Shard Theory uses as the prototypical case of RL [1]). This idea has also been expanded in many other posts.

The second source of my confusion was the idea of shards themselves. Even given amortisation, why should behaviour splinter into specific ‘shards’? and why should the shards compete with one another? What would it even mean for ‘shards’ to compete or for there to be ‘shard coalitions’ in a neural network?

My best guess here is that Shard Theory is making several empirical claims about the formation of representations during training for large-scale (?) RL models. Specifically, from an ML lens, we can think of shards as loosely-coupled relatively-independent subnetworks which implement specific behaviours.

A concrete instantiation of Shard Theory's claim, therefore, appears to be that during training of the network, the tendency is for the optimiser to construct multiple relatively loosely coupled circuits which each implement some specific behaviour which has been rewarded in the past. In a forward pass through the networks, these circuits then get activated according to some degree of similarity between the current state and the states that have led to reward in the past. These circuits then ‘compete’ with one another to be the one to shape behaviour by being passed through some kind of normalising nonlinearity such as softmax. I am not entirely sure how ‘shard coalitions’ can occur on this view, but perhaps some kind of reci...
Talking to God
by abramdemski — 2023-01-03T20:14:20.955Z

(low-effort post)

The following is not supposed to be an account of all religion, only a very specific thing that happens sometimes.

In logic, a "modality" is a qualifier you can put on a statement, which changes the rules for interacting with that statement. For example, "possibly X", "necessarily X", "I believe X", "I know X", and so on.

I'm going to use the term "modality" somewhat loosely, to refer to different modes of interaction with statements.

I think there's a part of rationality that could be described as "the search for the trustworthy modality". For example, "scientifically" is a pretty good modality. It has a set of rules around it which has, historically, been wildly and vividly successful in comparison with other modalities.

However, rationalists have some problems with the "scientifically" modality, especially in contrast to a "bayesian" modality.

This quest of finding useful modalities isn't really restricted to rationalists, of course. Basically all humans are looking for the best ways to have their words understood, and looking to understand others. Qualifiers like "I believe" are important tools which allow opinions to be expressed without introducing raw contradiction into the conversational context.[1]

The "God told me" modality, when you can pull it off, allows statements to be made which invite absolutely no disagreement. This is of course very very useful in the right context.

Looking around at the world, it seems clear that many of our problems are coordination problems. War is the most obvious -- it could simply stop and everything would be better! Of course, in these parts, we call the general problem Moloch.

So, catching a glimpse of how much better it could be if everyone would coordinate, it might seem easy to call it a glimpse of God's will. And you can see that if everyone would simply listen to this Will, things would be much better.

So you set out preaching the benefits of this modality, and try to create a community of practice around it.

It almost doesn't matter exactly what outcome we coordinate on, so long as we coordinate. So you might try to create a very viral form of the God modality. And you might have a lot of arbitrary practices in your community of practice, because you're implicitly more focused on coordinating on something than on making sure you've coordinated on the best possible thing.

Once, at a Les...
My Advice for Incoming SERI MATS Scholars
by Johannes C. Mayer — 2023-01-03T19:25:38.678Z

I have participated in SERI MATS 2.0 in John's stream. Here is some advice based on my experience.

Be Nice

The AI alignment community is pretty small. If you are an ass, everybody will know that you are an ass. The same holds to a lesser extent for being nice. When I was visiting Edinburgh to attend a talk by David Krueger, there were several people there, that I had first met at Lightcone. When I was visiting Trajan House, the same thing happened. You never know when you might be talking to a grantmaker over dinner.

Epistemic status: I did not actually behave like an ass. I expect this to be true, based on how many people I ran into that I've seen before, in different parts of the world.

Use Lunch and Dinner at Lightcone

During MATS 2.0 lunch and dinner were both served at Lightcone every day of the week. There were always many cool people around, and the conversations were unusually insightful. My favorite heuristic is to just join whatever conversation John is in. I am pretty sure that at least 15% of the value of SERI MATS came from eating lunch and dinner at Lightcone. Probably much more than that.

Epistemic status: It feels like this was very useful, but it is hard to quantify.

Take care of yourself

At the beginning of SERI MATS, there were many social events (mostly just general Berkeley EA/Rationalist events). They were all happening pretty late. For some reason, I need to sleep 10:30 to 12:00 hours every day or I will be tired. My team was meeting at 10:00 every day. For the first 3 weeks, I was basically sleep-deprived almost every day. John's workshops are pretty great, and being sleep-deprived during them destroyed probably more than 20% of the value. That being said, at least one of the socials was high-value, and it was probably worth the cost.

The worst thing was that I got used to being sleep-deprived. I sleep-deprived myself, even when there were no socials happening. I made similar mistakes with doing sports and eating healthily. Somehow it's hard to keep up all the good habits when you change your environment.

Epistemic status: It's hard to evaluate the counterfactual where I was not sleep-deprived. I estimate I could have gotten 5-35% more value by not making the mistakes I listed.

Learn to detach yourself from your ideas

Check out this comment.

Be Agentic

If something doesn't fit right, try to fix it.

Do you have a crazy idea about how to i...
Touch reality as soon as possible (when doing machine learning research)
by LawrenceC — 2023-01-03T19:11:58.915Z

TL;DR: I think new machine learning researchers often make one of two kinds of mistakes: not making enough contact with reality, and being too reluctant to form gears-level models of ML phenomena. Stereotypically, LW/AF researchers tend to make the former mistake, while academic and industry researchers tend to make the latter kind. In this post, I discuss what I mean by “touching reality” and why it’s important, speculate a bit on why people don’t do this, and then give concrete suggestions.

Related to:

Making Beliefs Pay Rent

,

The Feeling of Idea Scarcity

,

Micro-Feedback Loops and Learning

,

The Three Stages of Rigor

,

Research as a Stochastic Decision Process

,

Chapter 22 of HPMOR

.[1]

Epistemic status: Written quickly in ~3 hours as opposed to carefully, but I'm pretty sure it's directionally correct. [2]

Acknowledgments: Thanks to Adrià Garriga-Alonso for feedback on a draft of this post and Justis Mills for copyediting help.

Introduction: two common mistakes in ML research

Broadly speaking, I think new researchers in machine learning tend to make two kinds of mistakes:

Not making contact with reality. This is the failure mode where a new researcher reads a few papers that their friends are excited about, forms an ambitious hypothesis about how to solve a big problem in machine learning, and then spends months drafting a detailed plan. Unfortunately, after months of effort, our new researcher realizes that the components they were planning to use do not work nearly as well as expected, and as a result they’ve wasted months of effort on a project that wasn’t going to succeed.
Not being willing to make gears-level models. This is the failure mode where a new researcher decides to become agnostic to why anything happens, and believes empirical results and only empirical results even when said results don’t “make sense” on reflection. The issue here is that they tend to be stuck implementing an inefficient variant of

grad student descent

, only able to make small amounts of incremental progress via approximate blind search, and end up doing whatever is popular at the moment.

That’s not to say that these mistakes are mutually exclusive: embarrassingly, I think I’ve managed to fail in both ways in the past.

That being said, this post is about the first failure mode, which I think ...
Kolb's: an approach to consciously get better at anything
by jacquesthibs — 2023-01-03T18:16:00.018Z

Note: this serves as the first post in a sequence on how to efficiently become a great alignment researcher. I do not claim that I am a great researcher myself nor do I claim to have the best methods to become one, but I do hope this sequence at least marginally improves our ability to solve alignment.

Practice does not always make perfect. How you practice a skill matters, and you should consider reflecting to iteratively improve on whatever thing you are trying to get better at, e.g. being a better alignment researcher, getting better quality sleep, finding a long-term partner, maintaining great relationships, mental health, etc.

There are periods of my life where I wasted so much effort simply because I did not reflect on what I was doing in order to improve as efficiently as possible. And, in fact, by not reflecting and improving on mistakes, I simply reinforce bad habits!

There's a process I've begun to use in order to improve any skill (whether it's research, learning, getting better sleep, etc.): Kolb's Experiential Cycle. Very similar to deliberate practice (here’s a video about what people get wrong about deliberate practice).

Kolb's is something you can use every day to actively figure out what is working and what isn't. It allows you to reflect and improve on your 'mistakes.' It requires effort (an entire cycle could take from 15 minutes to 1 hour), but I believe it is worth it. In fact, I see Kolb's cycles as being a necessary component to becoming a great alignment researcher as efficiently as possible. Without something like it, you will potentially be stuck doing something that is above average in effectiveness, but not ideal.

The purpose is to use it for quickly iterating and getting better at a skill rather than letting things continue naturally with very little improvement. Without proper reflection, it could take you years to get to the same skill-level it should take you months. You could even plateau and never reach a higher skill-level you could have if you did some reflection.

Doing Kolb's cycles is a crucial component of the course I'm taking on learning efficiently and it's expected that every student in the course uses them regularly. The students who end up doing the best in school afterwards are the ones who actually make the effort to do Kolb's cycles rather than sticking to what is natural and comfortable.

Here's what Kolb's Expe...
{M|Im|Am}oral Mazes - any large-scale counterexamples?
by Dagon — 2023-01-03T16:43:37.682Z

Raemon's recent post reminded me that I'm unsure how best to think about bureaucracies and apparent organizational friction, and that I keep meaning to explore why something about my intuition rebels against the "Maze" framing. I've read a bit of economic and public choice theory, and worked in for-profit organizations very large, large and small, and been exposed to (via friends' work) nonprofits medium and small. I kind of think it's a working equilibrium, that's likely to reoccur or become worse if we disrupt it.

Most people are ... not particularly great at coordination or even surface-level alignment. And not all that motivated to spend more energy than needed on abstract concepts like mission or utility. Hierarchical mechanisms for coordination are lossy and annoying, but the best we know how to do, once we're doing things that take an order of magnitude bigger than Dunbar's number, and require imperfect people (all of us) to spend much of our time and energy doing things we don't particularly enjoy, but that improve someone else's life enough for us to get paid. "Get paid", of course, is shorthand for "hard-to-fake signal that someone values our behaviors more than they value that money".

Certainly there's a large variance in just how "mazey" an organization is, even within a size and mission-category group (say, comparing local governments of similar-sized populations, or tech companies of a given size range). But I don't know of any over a few thousand that aren't mazes to some noticeable extent. Do you?