trentmkelly/LessWrong-43k huggingface dataset:
Using Information Theory to tackle AI Alignment: A Practical Approach
by  — 2022-12-17T01:37:44.519Z

[Edited]: I have removed the experiments after realizing that I had implement the algorithm incorrectly. I have replaced the experiment with different research papers that use similar objectives.

Introduction

I have been working on an idea in relation to AI alignment and I want feedback from this community on both the validity of the idea and to comment with any future directions that this work should be taken in.

I am going to try and accomplish 3 things in this post.

Introduce and explain a reframing of AI alignment.
Present a mathematical version of the problem and a potential solution.
Suggest research papers that implement versions of this objective.

Quick AI alignment overview

Before we start reframing AI alignment it is important to understand what people mean when they discuss AI alignment. There is a lot of information out there explaining what AI alignment is (

Blog Posts

,

Free Curriculums

, etc.). However, to keep it short, I will stick with the definition that Paul Christiano presents, which is:

A is trying to do what H wants it to do.

In this case A is the AI agent and H is the human.

A common way of tackling this problem is through trying to understand what it is that the human wants and encode that as the goal for the AI agent. This is the core idea behind Value Learning. Value Learning takes information, such as human behaviour, and uses that information to predict what the true goal the human is optimizing for.

This way of solving AI alignment though has some pitfalls including:

Ambiguity of the reward function: With limited data, there may be many reward functions that can fit the data, making it difficult to accurately predict the true goal.
Human irrationality: Humans are not always rational or omniscient, meaning we may make mistakes that the AI model may try to justify under the reward model, leading to the wrong goal being encoded.
Misunderstood goals: If the model does not fully understand the desired goal, it may push variables to extremes to satisfy the goal from the reward model, leading to unintended consequences.

So what can we do to help reduce some of these problems? Well what if we look at the data we use for Value Learning from a different perspective.

Fundamental Assumption

Before we go into reframing the AI alignment problem I am going to make a big assumption and explain my reasoning behind this assumption because it informs the re...
Paper: Constitutional AI: Harmlessness from AI Feedback (Anthropic)
by LawrenceC — 2022-12-16T22:12:54.461Z
Link: https://www.anthropic.com/constitutional.pdf

The authors propose a method for training a harmless AI assistant that can supervise other AIs, using only a list of rules (a "constitution") as human oversight. The method involves two phases: first, the AI improves itself by generating and revising its own outputs; second, the AI learns from preference feedback, using a model that compares different outputs and rewards the better ones. The authors show that this method can produce a non-evasive AI that can explain why it rejects harmful queries, and that can reason in a transparent way, better than standard RLHF:

Paper abstract:

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through selfimprovement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use ‘RL from AI Feedback’ (RLAIF). As a result we are able to train a harmless but nonevasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

See also Anthropic's tweet thread:
https://twitter.com/AnthropicAI/status/1603791161419698181
Vaguely interested in Effective Altruism? Please Take the Official 2022 EA Survey
by Peter Wildeford — 2022-12-16T21:07:36.975Z
Link: https://rethinkpriorities.qualtrics.com/jfe/form/SV_1NfgYhwzvlNGUom?source=lw

The 2022 Effective Altruism Survey is now live at the following link: https://rethinkpriorities.qualtrics.com/jfe/form/SV_1NfgYhwzvlNGUom?source=lw

The survey will close at the end of the year.

What is the Effective Altruism Survey?

The Effective Altruism (EA) Survey is the official survey run by Rethink Priorities and the Center for Effective Altruism to provide valuable information about the demographics of the EA community and other vital data used for EA community building. LessWrong users remain an important part of this community and we want to ensure they are represented.

We appreciate it when EAs share the survey with others. If you would like to do so, please use this link (

https://rethinkpriorities.qualtrics.com/jfe/form/SV_1NfgYhwzvlNGUom?source=shared

) so that we can track where our sample is recruited from.

What's new this year?

The EA Survey is substantially shorter. Our testers completed the survey in 10 minutes or less.
We worked with CEA to make it possible for some of your answers to be pre-filled with your previous responses, to save you even more time. At present, this is only possible if you took the 2020 EA Survey and shared your data with CEA. This is because your responses are identified using your EffectiveAltruism.org log-in. In future years, we may be able to email you a custom link which would allow you to pre-fill, or simply not be shown, certain questions which you have answered before, whether or not you share your data with CEA, and there is an option to opt-in to this in this year’s survey.

Why take the EA Survey?

The EA Survey provides valuable information about the EA community and how it is changing over time. Every year the survey is used to inform the decisions of a number of different EA orgs. And, despite the survey being much shorter this year, this year we have included requests from a wider variety of decision-makers than ever before.

Prize

This year the Centre for Effective Altruism has, again, generously donated a prize of $1000 USD that will be awarded to a randomly selected respondent to the EA Survey, for them to donate to any of the organizations listed on EA Funds. Please note that to be eligible, you need to provide a valid e-mail address so that we can contact you.
Abstract concepts and metalingual definition: Does ChatGPT understand justice and charity? 
by Bill Benzon — 2022-12-16T21:01:05.315Z

Our conceptual worlds are filled with abstract concepts, such as truth, love, beauty, and justice. Such concepts cannot be defined in physical terms. You can’t see them, taste them, hear them, touch them, or smell them. And yet for human society they are as real as sticks and stones, thunder and lightning, a horse, a sparrow, cherry blossoms, or Mt. Fuji.

We can learn about abstractions through stories that exhibit them. Rene Girard’s concepts of mimetic desire and of sacrifice are abstract. They characterize patterns of action that people take in their interactions with one another. Whether he realized it or not – probably not – Steven Spielberg showed us those concepts in Jaws. And I demonstrated that ChatGPT was able to recognize them, to interpret the film as exhibiting them: Conversing with ChatGPT about Jaws, Mimetic Desire, and Sacrifice.

* * * * *

I begin this post by examining ChatGPT’s ability to work with the concept of justice. First I it to define justice, and then say a little about how Plato treated it in The Republic. Then we see how Chat deals with justice in stories, creating stories about justice, but also recognizing justice, or its lack, in stories. I then introduce Chomsky’s well-known, albeit somewhat obscure, distinction between competence and performance. Next, another example, charity, which I use to set-up a discussion of semantic or cognitive networks, a formalism from the now ‘classic’ symbolic AI and computational linguistics from the 1960s, 70s, and 80s. Would that notation be suitable for characterizing some (non-trivial) aspect of ChatGPT’s behavioral competence? I conclude by tempting the reader – that’s you – to consider the implications, suggesting that Neil Stephenson’s The Diamond Age: Or, A Young Lady's Illusttrated Primer might be a useful source of insight.

What is justice?

Let’s start by asking ChatGPT to define justice, which it does. I then asked about Plato’s treatment of justice in The Republic.

What is justice?

Justice is the concept of fairness and equality. In the legal context, it refers to the fair and proper administration of laws. In a broader sense, justice is the ideal state in which individuals are treated equally and without prejudice, and in which the laws and institutions of society are fair and just. It is often considered to be one of the fundamental principles of society, and it is an important concept i

...
Beyond the moment of invention
by jasoncrawford — 2022-12-16T20:18:10.818Z
Link: https://rootsofprogress.org/beyond-the-moment-of-invention

Derek Thompson has a feature in The Atlantic this week on “Why the Age of American Progress Ended”—thoughtful and worth reading. Some comments and reactions.

One of the ideas explored in the article is that what matters for progress is not just the moment of invention, but what happens after that. I generally agree. Some ways I think this is true:

The first iteration of an invention is generally just good enough to be practical, but far below optimal: there are decades of incremental improvements that get it to what we know today. Edison’s light bulb was not as bright or long-lasting as today’s bulbs.
Often there is a whole system that needs to be built up around the invention. It wasn’t enough to invent the light bulb, you also needed the generators and the power grid.
Such a system not only has to be invented, it has to be scaled. Scaling up from a prototype to a large, efficient, reliable system is its own challenge. Again with the power grid, it was a big challenge to figure out how to efficiently serve large regions, do load balancing, etc. With railroads, there was a challenge in figuring out how to manage a schedule with many trains and routes. With the telegraph and later the telephone, a system had to be invented to route messages and calls.
Merely working is not enough for wide distribution: other characteristics really matter, like cost, efficiency, and reliability. People underrate these—especially reliability, which can be a huge barrier to adoption.
An invention that works and is practical, cheap and reliable still doesn’t automatically sell. You have to convince people to change the way they do things, and that is tough. Sometimes you have to help people imagine uses for things: when the telegraph was first invented, they would demonstrate it by playing chess long-distance, and people would come watch this happen, but still not imagine what they personally would use a telegraph for.
Often regulations and legal frameworks have to be updated. E.g., containerization: the ICC regulated rates and they set rates based on the type of cargo; with containers you want to charge based on volume and weight alone, and this clashed with the regulatory framework. This kind of thing happens all the time.
Even after an invention is widely available, it can take further decades for all the implications to be worked out and for it to fundamentally change the way people

...
What's the best time-efficient alternative to the Sequences?
by trevor — 2022-12-16T20:17:27.449Z

The sequences are pretty long, and even the Highlights of the Sequences and the CFAR Handbook are pretty long, even though they're the next best thing. Much too long for me to recommend to the busy people in my life so they can try out rationality for the first time.

Akash made an effort to heavily trim down the Highlights of the Sequences, but too much was lost without the evocative examples.

What I'm wondering is: what is there here on Lesswrong, and elsewhere, that amps up someone's intelligence/rationality as much as possible, but is short enough that nobody looks at it and thinks "I'm too busy/tired for this"? News articles seem optimized for this, they seem to gravitate towards <1000 and there's some pretty strong market optimization pressures there. I've heard some good things about Scott Alexander's blog.

If provably increasing intelligence/rationality is too much to ask, then what affected you the most, what resulted in enduring, net positive change, such that after reading it, you had permanently diverged from the kind of person you were before?
Can we efficiently explain model behaviors?
by paulfchristiano — 2022-12-16T19:40:06.327Z

ARC’s current plan for solving ELK (and maybe also deceptive alignment) involves three major challenges:

Formalizing probabilistic heuristic argument as an operationalization of “explanation”
Finding sufficiently specific explanations for important model behaviors
Checking whether particular instances of a behavior are “because of” a particular explanation

All three of these steps are very difficult, but I have some intuition about why steps #1 and #3 should be possible and I expect we’ll see significant progress over the next six months. Unfortunately, there’s no simple intuitive story for why step #2 should be tractable, so it’s a natural candidate for the main technical risk.

In this post I’ll try to explain why I’m excited about this plan, and why I think that solving steps #1 and #3 would be a big deal, even if step #2 turns out to be extremely challenging.

I’ll argue:

Finding explanations is a relatively unambitious interpretability goal. If it is intractable then that’s an important obstacle to interpretability in general.
If we formally define “explanations,” then finding them is a well-posed search problem and there is a plausible argument for tractability.
If that tractability argument fails then it may indicate a deeper problem for alignment.
This plan can still add significant value even if we aren’t able to solve step #2 for arbitrary models.

I. Finding explanations is closely related to interpretability

Our approach requires finding explanations for key model behaviors like “the model often predicts that a smiling human face will appear on camera.” These explanations need to be sufficiently specific that they distinguish (the model actually thinks that a human face is in front of the camera and is predicting how light reflects off of it) from (the model thinks that someone will tamper with the camera so that it shows a picture of a human face).

Our notion of “explanation” is informal, but I expect that most possible approaches to interpretability would yield the kind of explanation we want (if they succeeded at all). As a result, understanding when finding explanations is intractable may also help us understand when interpretability is intractable.

As a simple caricature, suppose that we identify a neuron representing the model’s beliefs about whether there is a person in front of the camera. We then verify experimentally that (i) when this neuron is on it l...
Proper scoring rules don’t guarantee predicting fixed points
by Johannes Treutlein — 2022-12-16T18:22:23.547Z

Johannes Treutlein and Rubi Hudson worked on this post while participating in SERI MATS, under Evan Hubinger's and Leo Gao's mentorship respectively. We are grateful to Marius Hobbahn, Erik Jenner, and Adam Jermyn for useful discussions and feedback, and to Bastian Stern for pointing us to relevant related work.

Update 30 May 2023: We have now published a paper based on this post. In this paper, we also discuss in detail the relationship to the related literature on performative prediction.

Introduction

One issue with oracle AIs is that they might be able to influence the world with their predictions. For example, an AI predicting stock market prices might be able to influence whether people buy or sell stocks, and thus influence the outcome of its prediction. In such a situation, there is not one fixed ground truth distribution against which the AI's predictions may be evaluated. Instead, the chosen prediction can influence what the model believes about the world. We say that a prediction is a self-fulfilling prophecy or a fixed point if it is equal to the model's beliefs about the world, after the model makes that prediction.

If an AI has a fixed belief about the world, then optimizing a strictly proper scoring rule incentivizes it to output this belief (assuming the AI is inner aligned to this objective). In contrast, if the AI can influence the world with its predictions, this opens up the possibility for it to manipulate the world to receive a higher score. For instance, if the AI optimizes the world to make it more predictable, this would be dangerous, since the most predictable worlds are lower entropy ones in which humans are more likely dead or controlled by a misaligned AI. Optimizing in the other direction and making the world as unpredictable as possible would presumably also not be desirable. If, instead, the AI selects one fixed point (of potentially many) at random, this would still involve some non-aligned optimization to find a fixed point, but it may be overall less dangerous.

In this post, we seek to better understand what optimizing a strictly proper scoring rule incentivizes, given that the AI's prediction can influence the world. Is it possible for such a rule to make the AI indifferent between different fixed points? Can we at least guarantee that the AI reports a fixed point if one exists, or provide some bounds on the distance of a predic...
A learned agent is not the same as a learning agent
by Ben Amitay — 2022-12-16T17:27:28.037Z

[Edit: after reading the comments and thinking more about in-context learning, I don't endorse most of what's written here. I explain why I'm the end of the post.]

I notice a common confusion when people talk about deep learning. Before I try to describe it in general, let’s start with an example.

Like everyone, I lately had many conversations with friends about ChatGPT. A friend of mine said that while ChatGPT is indeed impressive, it highlights how amazing is the ability of human children to learn language from much less data. While I strongly share the sentiment, I think that the comparison is wrong. As Chomsky hypothesized, children’s ability to learn language from experience is so amazing, that we should doubt that this is really what happens. The child’s hearing language is probably merely fine-tuning to the shallow difference between human languages – building on a very strong prior knowledge about language general shape. The prior knowledge is mostly based on evolution’s “experience” with your great-great aunt who misunderstood instructions and ate poisonous mushrooms as a result – and many similar circumstances. Experience that was in turn aggregated to that of many humanoid great-great uncles who died in fights for dominance after mistaking their opponent’s threats for a bluff. And on the birth and death of earlier creatures, with brains that are something between genius controllers and very stupid agents, exploiting simple patterns in their environments[1].

What was my friend’s source of confusion? Probably that he did not properly distinguish the algorithm that generated ChatGPT as its output from the algorithm that is ChatGPT itself. If you want to compare ChatGPT itself to a human, you should compare its training loop to evolution, not to a human. Did training ChatGPT require more data than human evolution? I’m honestly not sure. But if I want to evaluate ChatGPT’s own learning abilities rather than those of the training loop, I should focus on how it uses information from earlier in a long conversation, as “earlier in the conversation” is it’s equivalent to “earlier in life”. Not information from his training data – which is equivalent to the unfortunate death of my greate-greate aunt. Even with this interpretation of ChatGPT “learning”, it probably doesn’t compare that well with humans – but for different reasons that I’ll mention be...
College Selection Advice for Technical Alignment
by TempCollegeAsk — 2022-12-16T17:11:30.828Z

Hi all, I'm a highschool senior trying to make some college-related decisions, and I'd like to ask for some advice.

My current situation is:

I want to work on technical alignment. For exogenous reasons, not going to college (e.g., taking a year off, just being an autodidact/independent researcher) is not an available option, so I'll have to leverage my undergraduate experience as much as possible to upskill on technical alignment.

I'll probably double major somewhere along CS/Math and maybe CompBio.

Accepted to Harvard (non-binding REA). Was planning to apply to Stanford, MIT, Harvey Mudd for RD, but ...
... I truly despise the application writing process, every single second of it, and it has taken a significant toll on my mental health. I'd prefer not to go through that again, although I can if necessary.

My considerations are:

Flexibility - Is it possible to take advanced (under)graduate courses while skipping prerequisites? I've been (and currently am) self-studying a bunch of (under)graduate subjects that I think would be helpful (mainly from the Study Guide) and it'd really suck to have to take them all again just for meeting prereqs for advanced classes.

I don't really care much about getting class credits as long as (1) I don't get kicked out of school for low credit and (2) the low credits or lack of prereqs won't prevent me from taking advanced subjects later on.

Are there any alignment research community/group/event nearby?
No need for financial aid right now.

The impression I got about Harvard (probably not so well-justified, just from anecdotes across reddit/etc) is that they're much less flexible in terms of class choices or prereqs compared to more traditionally "engineering" colleges like eg MIT. I also think the alignment community is mostly centered around the Bay area and that it hasn't really developed much around Harvard yet (I know about HAIST, though!)

Would Harvard be a good option to just go with, or is there enough additional value from Stanford/MIT/Harvey Mudd that it would be worth applying to any one of those colleges? Thanks!

(Apologies in advance if I broke any posting norms.)