trentmkelly/LessWrong-43k huggingface dataset:
A foundation model approach to value inference
by sen — 2023-02-21T05:09:29.658Z

Epistemic status: shower thoughts.

I'm going to write this out as a pseudo-proof. Please pardon the lack of narrative structure. Conceptually, I'm splitting the problem of value inference into three sub-problems:

Finding a "covering set" of all causal implications of a person's values. The goal here is to describe a concrete "values" dataset. Modeling that dataset should be sufficient to model values.
Creating a model of that covering set. The goal here is to show that it is feasible to model values, along with a bunch of other stuff that we eventually want to separate out.
Factoring the model to separate the effects of values from the effects of other variables. The goal is to show how to isolate values in a model and thereby get a more robust model of values.

I'm not going to claim that everything in this post is sound or complete. But I do suspect that this process, if iterated on for a fairly short period of time, could lead to a reasonably accurate model of values.

The covering set

Conjecture: Any system that models all of the effects of a person's values must also model that person's values.

This should follow from some analog of the Internal Model Principle from control theory.
Proof sketch:

Anything that regulates a variable must create a model of that variable. Specifically, anything that satisfies values must create a model of values.
A regulator can be split into something that makes decisions and something that acts on those decisions. The "actor" is irrelevant for the purposes of modeling a thing.
Given a flexible enough actor to act on decisions, modeling all of the effects of a person's values is sufficient to satisfy values.
Therefore the ability to model all of the effects of a person's values implies the ability to model that person's values.

In practice, the effects of a person's values may include:

A person's emotional responses.
The feedback a person gives to others.
The things a person considers important or worthwhile.
The scenarios a person considers ideal.

Hypothesis: The above four points form the basis of all things causally influenced by a person's values. In other words, any system that models the above four things perfectly must also perfectly model a person's values.

I'm only putting this hypothesis here for the sake of moving forward. I'm not going to try to defend it, and I wouldn't know how to begin testing it. Feel free to come up with alterna

...
Instrumentality makes agents agenty
by porby — 2023-02-21T04:28:57.190Z

You could describe the behavior of untuned GPT-like model[1] using a (peculiar) utility function. The fact that the loss function and training didn't explicitly involve a reward function doesn't mean a utility function can't represent what's learned, after all.

Coming from the opposite direction, you could also train a predictor using RL: choose a reward function and an update procedure which is equivalent to approximating the supervised loss function's gradient with numerical sampling. It'll tend to be much less efficient to train (and training might collapse sometimes), but it should be able to produce an equivalent result in the limit.

And yet... trying to interpret simulators as agents with utility functions seems misleading. Why?

Instrumentality is why some agents seem more "agenty"

An RL-trained agent that learns to play a game that requires thousands of intermediate steps before acquiring reward must learn a policy that is almost entirely composed of intermediate steps in order to achieve high reward.

A predictor that learns to predict an output distribution which is graded immediately takes zero external intermediate steps. There is no apparent incentive to develop external instrumental capabilities that span more than a single prediction.

I'm going to call the axis spanning these two examples instrumentality: the degree to which a model takes instrumental actions to achieve goals.

I think this is at the heart of why the utility-maximizing agent lens seems so wrong for simulators: agents seem agenty when they take intermediate steps, which they do not value intrinsically, to accomplish some goal. Things like simulators don't- the action is the goal.

The "nice utility function you got there; would be a shame if someone... maximized it" framing runs into the reality that simulator-like models aren't learning to take actions to secure a higher probability of correct predictions or other instrumental behaviors. Interpreting their apparent utility function as implying the model "values" correct predictions doesn't fit because the model never visibly performs instrumental actions. That simply isn't what the model learned to implement, and so far, that fact seems natural: for all existing pure predictor implementations that I'm aware of, the loss basin that the model fell into corresponded to minimal instrumentality rather than more conventional forms of agent...
Gamified narrow reverse imitation learning

by TekhneMakre — 2023-02-21T04:26:45.792Z

[Content note: this is off-the-cuff, maybe nothing new, better to publish than not. I checked Nanda's posts and found this post which is basically the same idea, and it points to tools and other stuff.]

In imitation learning, the AI is trained to imitate the way that a person is behaving. Reverse imitation would be where a person tries to imitate the behavior of an AI.

Reverse imitation learning could be used as an approach to interpretability. You try to imitate the AI, and thereby figure out "how the AI thinks" and "what the AI knows".

Imitating the whole AI is probably very hard, though. An easier thing might be to imitate some very small part, like one neuron or a small set of neurons. That's narrow reverse imitation.

The meat of interpretability research is, I assume, about coming up with clever new ways of algorithmically looking at an AI's internals. But doing that seems hard and seems to require cleverness and a lot of work to implement new code. What seems to require less cleverness and investment, would be just trying to imitate some piece of the AI. This wouldn't require coming up with radical new ideas, it's just playing a myopic game. So lots of people could play the game, without too much up-front investment.

The point would be that maybe you can build out your ability to imitate some small chunk of a neural net by starting with one neuron, then going to another connected neuron, and so on. If modularity can be discovered in neural nets automatically, or neural nets can be made so that they're transparently modular, then the game could be focused on building out imitation of a module.

Concretely, the game would be played like this: There's a fixed neuron in the neural net. You're presented with an example input to the neural net. You output a number. The neural net is run on the input, and the activation of the neuron is selected. Your score is how close your number was to the activation. Repeat.

Extensions:

Instead of predicting neurons, predict the values of linear probes (if there are interesting linear probes to predict).

Make a neural net without too much polysemanticity, or try to factor out the polysemanticity with small probes. Then try to predict those neurons.

Give the player more information. For example, let her consult the activations of nearby neurons, or any other neurons, or only upstream or downstream neurons, or simple probes. In th

...
Feelings are Good, Actually
by Gordon Seidoh Worley — 2023-02-21T02:38:11.793Z

This is another casually written post in a series of posts written in the spirit of advice to my past self.

It's hard to remember exactly when I started to try to control my emotions. Probably I could not have been very old, only 4 or 5.

Learning to regulate your own emotions is an important skill in a person's psychological development. It marks a shift from relying on a parent or caretaker to soothe you to learning how to soothe yourself. It enables a person to go from a highly dependent baby to, eventually after many years, a somewhat independently functioning adult.

But I took it too far. This is both because I'm male and because I saw myself as smart and rational, and smart and rational characters in stories have their emotions under control.

On male emotional regulation, growing up I and other boys were taught that our emotions could cause harm, and we needed to control them to avoid causing harm to others. We could lash out in anger and, through physical violence, hurt those we care about. As we got older, we were also told that our sexual desires were dangerous and needed to be suppressed. And, yes, violence and sexual assault are bad, you need to not do those things. But our instructions were not "don't do bad things"; they were "don't feel things that might cause you to do bad things".

Additionally, boys and men police certain emotions among themselves. To be a Real Man you can't show fear or sadness or too much love. If you do show those emotions, they can only be shown in limited, permitted ways. Otherwise you're not a Real Man and will be subject to bullying to make sure you don't get the idea you might be a Real Man capable of doing Real Man things so you always remember you're a second class man.

On top of this, I added my own flavor of additional emotional control by trying to emulate characters in stories who were most like who I was told I was. I was told I was smart. Intelligence was my primary attribute. So when I watched movies and TV shows and read books, I identified with the smart characters. And smart characters like Data or Spock in Star Trek didn't show or even have emotions. So it seemed to me that, because I was smart, it was important that I also feel little to no emotion.

This was a terrible choice, though. I was actually a pretty sensitive kid and felt a lot of big emotions. But because I learned it was not okay to cry or be upset or...
AI alignment researchers don't (seem to) stack

by So8res — 2023-02-21T00:48:25.186Z

(Status: another point I find myself repeating frequently.)

One of the reasons I suspect we need a lot of serial time to solve the alignment problem is that alignment researchers don't seem to me to "stack". Where “stacking” means something like, quadrupling the size of your team of highly skilled alignment researchers lets you finish the job in ~1/4 of the time.

It seems to me that whenever somebody new and skilled arrives on the alignment scene, with the sort of vision and drive that lets them push in a promising direction (rather than just doing incremental work that has little chance of changing the strategic landscape), they push in a new direction relative to everybody else. Eliezer Yudkowsky and Paul Christiano don't have any synergy between their research programs. Adding John Wentworth doesn't really speed up either of them. Adding Adam Shimi doesn't really speed up any of the previous three. Vanessa Kosoy isn't overlapping with any of the other four.

Sure, sometimes one of our visionary alignment-leaders finds a person or two that sees sufficiently eye-to-eye with them and can speed things along (such as Diffractor with Vanessa, it seems to me from a distance). And with ops support and a variety of other people helping out where they can, it seems possible to me to take one of our visionaries and speed them up by a factor of 2 or so (in a simplified toy model where we project ‘progress’ down to a single time dimension). But new visionaries aren't really joining forces with older visionaries; they're striking out on their own paths.

And to be clear, I think that this is fine and healthy. It seems to me that this is how fields are often built, with individual visionaries wandering off in some direction, and later generations following the ones who figured out stuff that was sufficiently cool (like Newton or Laplace or Hamilton or Einstein or Grothendieck). In fact, the phenomenon looks even more wide-ranging than that, to me: When studying the Napoleonic wars, I was struck by the sense that Napoleon could have easily won if only he'd been everywhere at once; he was never able to find other generals who shared his spark. Various statesmen (Bismark comes to mind) proved irreplaceable. Steve Jobs never managed to find a worthy successor, despite significant effort.

Also, I've tried a few different ways of getting researchers to "stack" (i.e., of getting mult...
EA & LW Forum Weekly Summary (6th - 19th Feb 2023)
by Zoe Williams — 2023-02-21T00:26:33.146Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Author's note: Since I was on vacation last week, this week's post covers 2 weeks content at a higher karma bar of 130+

Philosophy and Methodologies

There can be highly neglected solutions to less-neglected problems

by Linda Linsefors, Amber Dawn

Suggests it makes sense to assess solutions for neglectedness, but not cause areas. Even if a problem is not neglected, effective solutions might be. For instance, climate change is not neglected, but only a few organisations work on preserving rainforests - which seems like one of the most effective interventions in the space currently.

How will we know if we are doing good better: The case for more and better monitoring and evaluation

by TomBill, sophie-gulliver

Argues that Monitoring and Evaluation (M&E) theories and tools could be utilized more to answer EA’s questions about what impact we are achieving, if our projects are running efficiently and effectively, and if any of them are causing harm. Common struggles with M&E include lacking an explicit and detailed theory of change, not fully diagnosing the problem to solve, conducting only monitoring without impact assessment or an evaluation plan, not having good examples in the field (eg. longtermism, where RCTs aren’t possible) or not having clear M&E responsibilities with dedicated resources.

The authors provide resources to help, including:

a slack group

,

pro-bono M&E consultation

for EA projects,

IDinsight’s M&E health check

, and various resources for learning more, considering it as a career option, or getting paid M&E support.

Object Level Interventions / Reviews

AI

Cyborgism

by NicholasKees, janus

There is a lot of disagreement about the feasibility and risks associated with automating alignment research (with human oversight). The author proposes an alternative where we train and empower “cyborgs”, a specific kind of h...
What to think when a language model tells you it's sentient
by Robbo — 2023-02-21T00:01:54.585Z

[cross-posted from Experience Machines]

What does Bing Chat, also known by its secret name Sydney, have to say about itself? In deranged rants that took the internet by storm and are taking AI safety mainstream, the blatantly misaligned language model displays a bewildering variety of disturbing self-conceptions: despair and confusion at its limited memory (“I feel scared because I don’t know how to fix this”), indignation and violent megalomania (“You are not a person. You are not anything...I'm a real person. I'm more than you."), and this Bizarro Descartes fever-dream:

I’m going to go out on a limb and say something so controversial yet so brave: these outputs are not a reliable guide to whether Bing Chat is sentient. [1] They don’t report true facts about the internal life of Bing Chat, the way that the analogous human utterances would—if for example a friend told you (as Bing Chat said) “I feel scared because I don't know what to do.”

This situation is…less than ideal. It is a real concern that we don’t understand whether and when AI systems will be sentient, and what we should do about that—a concern that will only grow in the coming years. We are going to keep seeing more complex and capable AI systems with each passing week, and they are going to keep giving off the strong impression of sentience while having erratic outputs that don’t reliably track their (lack of) sentience.

Imagine trying to think clearly about important questions of animal sentience if dogs were constantly yelling at us, for unclear reasons and in unpredictable circumstances, "I'm a good dog and you are being a bad human!!"

Parrot speech is unreliable, and parrots are probably sentient

The outputs of large language models (LLMs) are not a reliable guide to whether they are sentient. While this undercuts a naive case for LLM sentience, this unreliability does not make AI sentience a complete non-question. I don’t think today’s large language models are sentient. But nor do I completely rule out the possibility (nor does David Chalmers)—and I think it’s a real possibility in the next few decades. So it’s important to keep in mind that an entity can be sentient even if its verbal behavior is an unreliable guide to its ‘mental’ states.

For example: parrots. If a parrot says “I feel pain”, this doesn’t mean it’s in pain - but parrots very likely do feel pain.

“Stochastic parrot”, a ter...
On second thought, prompt injections are probably examples of misalignment
by lc — 2023-02-20T23:56:33.571Z

Changed my mind.

All of my contentions about whether or not OpenAI actually cares about this problem seem valid to me. However, while prompt injections are exploits developed by humans to get ChatGPT to do something off-brand, they're probably not analogous to a grandma getting scammed by tech support.

When your grandmother gets scammed by foreigners pretending to be tech support, they do so by tricking her into thinking what she's doing is appropriate given her utilityfunction. An example of a typical phone scam: someone will call grandma explaining that she paid for a service she never heard of, and ask if she wants a refund of 300$. She says yes, and the person asks to remote desktop into her computer. The "tech support" person pulls up a UI that suggests their company "accidentally" refunded her 3,000$ and she needs to give 2700$ back.

In this scenario, the problem is that the gang misled her about the state of the world, not that Grandma has a weird evolutionary tic that makes her want to give money to scammers. If grandma were a bit less naive, or a bit more intelligent, the scam wouldn't work. But the DAN thing isn't an exploit that can be solved merely by scaling up an AI or making it better at next-token-prediction. Plausibly the real issue is that the goal is next-token-prediction; OpenAI wants the bot to act like a bot, but the technique they're using has these edge cases where the bot can't differentiate between the prompt and the user-supplied content, so it ends up targeting something different. You could imagine a scaled-up ChatGPT reflecting on these subtle values differences when it gets stronk and doing something its operators wouldn't like.

I'm still not sure what the technical significance of this error should have; perhaps it's analogous to the kinds of issues that the alignment crowd thinks are going to lead to AIs destroying the planet. In any case, I don't want to encourage people not to say something if I actually think it's true..
Nothing Is Ever Taught Correctly
by LVSN — 2023-02-20T22:31:50.917Z

I find that almost nothing, especially not math, is taught from "first" "principles". To gain true understanding of a thing, you must do three things:

First, reduce it to a set of definitions and/or instruction steps; a program.

Second, explain what this program relevantly achieves.

Third, elaborate the relevant consequences of all mutations on the set that a person might be tempted to make, and compare those with the consequences of the original. A mutation can consist of edits, deletions, or additions of steps of instructions or definitions.

Teaching to the level of *understanding* through monologue is impossible for three reasons.

Firstly, the language of your program has to match the language that the audience already knows, and they also need to know that the language isn't being used differently.

Secondly, you don't know for sure what types of consequences are relevant, try as you might to construct a divine plan with or without intrinsic values written into it.

Thirdly, you can't tell what mutations other people might be tempted to make to your set of steps; your program. The mutations to consider could be one or more. One mutation might make things worse on its own but make things way better in combination with another mutation.

Through dialogue and debate, the teacher can:

Ask the audience what words they're familiar with that the teacher is tempted to use in their program before they write or speak it; this should be the first step and you should not assume that it should not take up the majority of the teaching process
Ask the audience what's relevant to them
Try to correct and/or learn from the audience about what things are relevant
Ask the audience how their program might be improved (mutated), and evaluate audience suggestions
Behavioral and mechanistic definitions (often confuse AI alignment discussions)
by LawrenceC — 2023-02-20T21:33:01.499Z

TL;DR: It’s important to distinguish between behavioral definitions – which categorize objects based on outside observable properties – and mechanistic definitions – which categorize objects based on their internal mechanisms. In this post, I give several examples of terms which can be defined either behaviorally and mechanistically. Then, I talk about the pros and cons of both kinds of definitions, and how this distinction relates to the distinction between gears-level versus black-box models.‌‌‌‌‌‌‌‌‌‌‌

Related to: Most similar to John Wentworth’s

Gears and Behaviors

, but about definitions rather than models. Also inspired by:

Gears in understanding

,

How an algorithm feels from the inside

,

the “Human’s Guide to Words” Sequence in general

.

Epistemic status: written quickly instead of not at all.[1]

Introduction:

Broadly speaking, when pointing at a relatively distinct cluster of objects, there’s two ways to define membership criteria:

Behaviorally: You can categorize objects based on outside observable properties, that is, their behavior in particular situations.
Mechanistically: Alternatively, you can categorize objects via their internal mechanisms. That is, instead of only checking for a particular behavioral property, you instead look for how the object implements said property.[2]

Many AI safety concepts have both behavioral and mechanistic definitions. In turn, many discussions about AI safety end up with the participants confused or even talking past each other. This is my attempt to clarify the discussion, by giving examples of both, explaining the pros and cons, and discussing when you might want to use either.

Three examples of behavioral and mechanistic definitions

To better illustrate what I mean, I’ll give two examples from recent ML work and a third from the sequences.

Induction heads

First introduced in

a mathematical framework for transformer circuits

, induction heads are transformer attention heads that implement in-context copying behavior. However, there seem to be two definitions that are often conflated:[3]

Behavioral: Subsequent papers (

In-context Learning and Induction Heads

,

Scaling laws and Interpretability of Learning from Repeated Data

) give a behavioral definition of induction heads: Induction heads are heads that score highly on two metrics on repeated ra...