trentmkelly/LessWrong-43k huggingface dataset:
AI Alignment [progress] this Week (10/29/2023)
by Logan Zoellner — 2023-10-30T15:02:26.265Z
Link: https://midwitalignment.substack.com/p/ai-alignment-breakthroughs-this-week-7b6

Wow! It’s been one month already.

Some thoughts:

When I started this project, I had a general sense that there is alignment progress on a regular basis, but tracking it really brings this home. Every week, on Monday I wonder “is this the week I will have nothing to write?” And by Sunday there are innovations that I leave out because the letter is too long.

I’ve also been happy to see the reception has been mostly positive. Literally the only criticism I’ve received is that people don’t like the word “breakthrough”. I think adding the 💡rating addresses the fact “not all breakthroughs are equal”. I’ve reached my goal of “at least 10 positive comments/new subscribers”, so I will continue doing this for at least another month.

If I had to nominate a single breakthrough for “biggest breakthrough of the month”, I think I would pick the AI Lie Detector. It perfectly combines the aspects of: 1. surprising, 2. innovative new method and 3. strong application to AI alignment, and easily deserves a rating of 5 💡💡💡💡💡

I think an interesting question is how many 5💡💡💡💡💡 breakthroughs it will take to solve the alignment problem. EY suggests the current score is 2/1000. I’m not quite that pessimistic. I think we’re closer to 5/100. And of course it’s always possible that someone will find a 6 💡💡💡💡💡💡 breakthrough that dramatically speeds up progress.

I do feel like this letter is getting a little long. I’m curious how long readers would like. [poll on substack]

What are your thoughts? Feel free to comment! I especially appreciate comments discussing specific technical aspects of various AI alignment methods/pathways.

And here are our

AI Alignment Breakthroughs this Week

This week, there were breakthroughs in the areas of:

Mechanistic Interpretability

AI Agents

Avoiding Adversarial Attacks

Benchmarking AI

Decentralized AI

Learning Human Preferences

Making AI Do what we Want

AI Art

In addition, I added a new section:

AI Alignment Initiatives

Mechanistic Interpretability

Decoding Sentiment in AI World Models

What is it: token-level sentiment analysis for LLMs

What’s new: they find that models have a linear sentiment direction

What is it good for: Understanding how LLMs rate sentences in-context.

Rating: 💡💡💡

Thinking Like Transformers

What is it: research on what algorithms LLMs can learn

What’s new: they find that transformers can learn algorithms that can be expressed in the RASP-L...
Improving the Welfare of AIs: A Nearcasted Proposal
by ryan_greenblatt — 2023-10-30T14:51:35.901Z

Overview and Summary

In this document, I’ll discuss and propose a nearcasted plan for improving the algorithmic welfare of powerful AI systems, given no further technical or philosophical progress. I broadly endorse many of the propositions in Propositions Concerning Digital Minds and Society, and some of this document is directly inspired by this earlier work. This plan could probably be considerably improved and made more detailed with only a few weeks of work: I’ve only spent 1-4 days on this as this isn’t the typical area I work on. This plan is primarily taking the perspective ‘what would we do to improve algorithmic welfare given only a highly limited budget of resources?’. As such, note that this plan is dramatically insufficient given the magnitude of the problem and the potential harms: If society was coordinated and reasonable, we wouldn’t need to race through the process of creating powerful AIs that could be moral patients. In this alternate world, we could avoid only having highly limited resources to spend on avoiding unethical uses of AIs.

This document is speculative and not well argued for. I’m not a philosopher and I’m an amateur in this area, so I might be missing some basics.

I’ll first discuss some broad claims about the plausibility and scale of AI moral value and algorithmic welfare.

Then, I’ll discuss various motivations for spending resources on AI welfare. This will touch on how different motivations might change what interventions seem best. Additionally, I’ll talk some about how much resources should be devoted to AI welfare given that other issues may dominate on various moral views.

I’ll argue that it’s very difficult to predict whether or not a given exogenous change to an AI’s environment or training procedure will improve or worsen average welfare (at a fixed level of intelligence). Overall, I’ll claim that communicating with AI systems about their preferences probably dominates most of our current ideas for how to change training procedures.

Finally, I’ll propose interventions for a nearcasted algorithmic welfare plan with limited resources and just current technology.

Here is a list of these interventions in descending order of importance (in my opinion):

Attempt communication: We should try to communicate with and understand AI systems. In particular, we should try to understand and satisfy any coherent preferences they might have

...
President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence
by Tristan Williams — 2023-10-30T11:15:38.422Z
Link: https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/

Released today (10/30/23) this is crazy, perhaps the most sweeping action taken by government on AI yet.

Below, I've segmented by x-risk and non-x-risk related proposals, excluding the proposals that are geared towards promoting its use[1] and focusing solely on those aimed at risk. It's worth noting that some of these are very specific and direct an action to be taken by one of the executive branch organizations (i.e. sharing of safety test results) but others are guidances, which involve "calls on Congress" to pass legislation that would codify the desired action.

[Update]: The official order (this is a summary of the press release) has now be released, so if you want to see how these are codified to a greater granularity, look there[2].

Existential Risk Related Actions:

Require that developers of the most powerful AI systems share their safety test results and other critical information with the U.S. government. In accordance with the Defense Production Act, the Order will require that companies developing any foundation model that poses a serious risk to national security, national economic security, or national public health and safety must notify the federal government when training the model, and must share the results of all red-team safety tests.
Develop standards, tools, and tests to help ensure that AI systems are safe, secure, and trustworthy. The National Institute of Standards and Technology will set the rigorous standards for extensive red-team testing to ensure safety before public release. The Department of Homeland Security will apply those standards to critical infrastructure sectors and establish the AI Safety and Security Board. The Departments of Energy and Homeland Security will also address AI systems’ threats to critical infrastructure, as well as chemical, biological, radiological, nuclear, and cybersecurity risks. Together, these are the most significant actions ever taken by any government to advance the field of AI safety.
Protect against the risks of using AI to engineer dangerous biological materials by developing strong new standards for biological synthesis screening. Agencies that fund life-science projects will establish these standards as a condition of federal funding, creating powerful incentives to ensure appropriate screening and manage risks potentially made worse by AI
Order the development of

...
GPT-2 XL's capacity for coherence and ontology clustering
by MiguelDev — 2023-10-30T09:24:13.202Z

This project was inspired by Nate Soares' advice of focusing on areas where it seems everyone is dropping the ball. If you haven't read / listened to it, I highly recommend it.

Per wikipedia, Algos is the greek word for pain

(I'm not certain if this is where GPT-2 Insight got the idea to select Algos as an ontology.) reddit link

TLDR

GPT-2 Insight is an enhanced version of GPT-2 XL, designed to deliver safe and coherent responses via a persona known as Aligned AI. Utilizing Archetypal Transfer Learning (ATL), GPT-2 Insight has generated Algos and overarching themes such as simulating worlds or conflicts (Deus Ex) and achieving higher awareness (Divine Consciousness). These serve as its ontologies to fulfill its objective of assisting humans while adhering to ethics, truth, and safety. The author argues that the Clustered Ontologies created by GPT-2 Insight open up new avenues for research in the field of AI alignment.

Background

After developing a corrigible version of GPT-2 XL (ATL-P1) capable of generating a shutdown phrase, I found that GPT-2 XL could indeed follow complex instructions. This lead me to consider GPT2 XL's potential for increased coherence in its responses. This question is the focus of my current project, ATL-P3.

(However, the results of this experiment led me in a different direction.)

How to Read and Understand this post?

This project aims to achieve objectives similar to:

John Wentworth's "Just Retarget The Search"
The creation of Model Organisms by Team Anthropic
Richard Ngo's post on Value Concretization is relevant as well, with some significant differences which you will understand after reading the entire post.

The goal is to rigorously test Archetypal Transfer Learning (ATL) as a method for addressing the alignment problem. If ATL can model a path to learning similar to human evolution, and instill these values in GPT-2 XL while simultaneously enhancing its capabilities and safety, then that would be a significant achievement.

Main Sections Outlined

Explanation of how GPT-2 Insight works and a comparison of its outputs to the base model, GPT-2 XL
FINIS ALCHOH ALCHA: Weird issues, further experiments and explanations
Final and ongoing thoughts on this project

Part 1: GPT-2 Insight

Comparing ATL-P1 and ATL-P3

ATL-P1[1]

GPT-2 XL ➜ ATL tuning ➜ Corrigible GPT-2 XL

ATL-P

...
Charbel-Raphaël and Lucius discuss interpretability
by Mateusz Bagiński — 2023-10-30T05:50:34.589Z

The following is a transcript of a public discussion between Charbel-Raphaël Segerie and Lucius Bushnaq that took place on 23 September during LessWrong Community Weekend 2023. I edited the transcript for clarity.

Transcript

Mateusz: Last month, Charbel published the post Against Almost Every Theory of Impact of Interpretability, which sparked a lot of discussion in the community. Lucius is an AI notkilleveryoneism researcher at Apollo, focused on interpretability. He disagrees with Charbel. So I brought them together to discuss. I suggest that Charbel starts by bringing up what he considers his strongest argument against interpretability, Lucius responds to that, and we'll see how it goes.

Reverse engineering neural networks and correct ontology for interpretability

Charbel: I think we can start with "auditing deception with interp is out of reach". So, Neel says, "Maybe we can look at random bits of the model and identify circuits or features." We need to steelman this argument a bit because it's clear that looking at random bits of the model won't be sufficient. Instead, we can try to enumerate each type of feature (enumerative safety). However, it seems to me that this kind of procedure is very poorly specified. We have already attempted similar approaches with vision models, but they didn't quite work [Charbel comment: I updated a bit after the Towards Monosemanticity paper]. Additionally, there are some theoretical arguments, like deception could be something very diffuse and it's possible that there won't be any part of the model that is deceptive.

Maybe we should try to study deceptive alignment with other techniques instead. For example, trying to monitor the proxies of deceptive alignment, or trying to find ex-ante strategies instead of ex-post strategies. Finally, at the end of the section, I wrote that if DeepMind announced today that they had found an instance of a deceptive model, they certainly wouldn't have found it using interpretability alone. I think I won some Bayes points because the Center for AI Safety recently published a paper where they have a methodology for monitoring Cicero's behavior, whether it's lying and so on. There's no interpretability in that paper.

Lucius: So, first off, my views on interpretability are maybe not exactly mainstream and representative of most interpretability researchers. My answers to this might be very differ...
Multi-Winner 3-2-1 Voting
by Yoav Ravid — 2023-10-30T03:31:25.776Z

My favorite voting method is 3-2-1 voting[1], my second favorite is approval voting. Both are methods for selecting one winner. Approval voting has variations for choosing multiple winners, but 3-2-1 voting doesn't - until now.

I'm going to explain approval voting, sequential proportional approval voting, 3-2-1 voting, and proportional sequential 3-2-1 voting - which is my multi-winner version of 3-2-1 voting.

Note: I think it works well, but I'm no professional, and I haven't proved anything about it. There's also one detail I'm still not sure about. If you can help me test it better or prove certain properties about it, I'd much appreciate it.

Approval voting

Approval voting is simple - each voter can vote for as many candidates as they want, and the one that got the most votes is elected. It's good because it's simple, there's relatively little incentive for strategic voting, and it leads to relatively high voter satisfaction.

Sequential Proportional Approval Voting

Like approval voting, each voter can vote for as many candidates as they want, and the first winner is the one that got the most votes. Then all the ballots are reweighed, such that each vote is equal 1X+1, where X is the amount of elected candidates voted for on the ballot, and another candidate is elected. Repeat until all the seats are filled.

So if I voted for the first candidate that was elected, in the second round my vote would be worth 1/2. If I also voted for the second candidate elected, then in the third round my vote will be worth 1/3, and if didn't vote for the second candidate elected, my vote would still be worth 1/2 in the third round.

This is good because it achieves proportional representation[2] in a relatively simple way, and with little incentive for strategic voting. Without reweighing the ballots, the more similar you are to the winning candidate the more likely you are to win, which will create a situation where a parliament of clones is elected.

3-2-1 voting

In 3-2-1 voting you rate as many candidates as you want as either "bad", "ok", or "good". Then the winner is selected in 3 steps:

The three candidates who got the most "good" ratings are selected.
Of them, the one with the most "bad" ratings is cast out.
Of the two remaining, the one that is ranked higher than the other on more ballots is elected.

This is more complicated than approval voting, but it pays for tha...
math terminology as convolution
by bhauth — 2023-10-30T01:05:11.823Z
Link: https://www.bhauth.com/blog/thinking/terminology%20as%20convolution.html

On the one hand, this theory generalizes the Fuchsian and Bers uniformizations of complex hyperbolic curves and their moduli to nonarchimedean places. It is for this reason that we shall often refer to this theory as p-adic Teichmuller theory, for short. On the other hand, the theory under discussion may be regarded as a fairly precise hyperbolic analogue of the Serre-Tate theory of ordinary abelian varieties and their moduli.

— Shinichi Mochizuki

I know some of these words.

— Ed in Good Burger (1997)

terminology as convolution

Math research papers are notorious for using specialized and obscure terminology. Why is that? Why can't they describe things in terms of simpler components?

Chemists often talk about carbon atoms. They don't say "an atom with 6 protons, 6 neutrons, and 6 electrons". Those subatomic particles are grouped together into a single conceptual item. The power of convolutional neural networks shows us that such grouping is not merely a matter of convenience - rather, the selection of which things to group together is a system of thinking.

Neural network research suggests a lot about how humans think. For example, I think the fact that massively multilingual language models work well, with many languages ultimately sharing the same latent space, is a refutation of the Sapir-Whorf Hypothesis. Modern neural networks have also, I think, shown us something about what concepts are. Some linguists have argued that a word like "dog" is a discrete package, a fixed item to which additional information is attached. Based on my comparison of how humans think and now neural networks operate, my view is that the concept "dog" is 3 things:

A region of a latent space for doglike concepts.

One or more prototype dog concepts, which are points in that latent space used to define the region of dog-ness.

A convolution-like transformation by which some data can be packaged into a point in a latent space: "this is a dog" is a way of examinining some data from a photo.

Math is often considered universal, but many of the concepts are partly arbitary. For example, some people have suggested pi as a universal number that alien species would recognize, but other people argue that 2*pi is a more fundamental constant.

For a slightly more "complex" example, consider imaginary numbers. The fundamental theorem of algebra involves them, and that sounds fundamental...but c...
Grokking, memorization, and generalization — a discussion
by Kaarel — 2023-10-29T23:17:30.098Z

Dmitry Vaintrob

Intro: Kaarel and Dmitry have been meeting to think about various interpretability and toy model questions. We are both interested in unsupervised and superposition-agnostic interpretability results (

Kaarel

,

Dmitry

). Recently we discussed questions related to grokking, memorization, double descent and so on. Dmitry has been doing a number of experiments on modular addition with Nina Rimsky. (See

here

and

here

. Another writeup on modular addition circuits based on these experiments is in the works. Note that Nina is also interested in these questions, but couldn’t join today. Dmitry's views here are almost entirely a result of conversations with Nina.) Dmitry is interested in having a better gears-level understanding of grokking and thinks there are some issues with how these are interpreted in the intepretability/alignment community and literature (a view Kaarel partially shares). Kaarel's views on the topic should be attributed in part to previous discussions with Jake Mendel and Simon Skade. The discussion started in a private conversation, and we decided that it might be helpful to continue this as a public discussion. This is our first experiment with LW conversations, so we would like to heartily invite critical feedback about style, format, technical level, etc.

Kaarel

Do you know if anyone has compiled a list of all the quantitative aspects of grokking / double descent that we'd expect a good theory of memorization vs generalization to explain? stuff like the location of interpolation thresholds in various parameters, (claimed) superexponential growth of the time to grok as the data set size decreases

Dmitry Vaintrob

I think there are several interesting questions that are largely distinct here. There's been some work done on all of them, but I'm not sure if I'm up to date on all of it

Dmitry Vaintrob

There's a theory that grokking is directly the competition between a large-basin phenomenon (grokking) and small-basin phenomena (memorization). I think the strongest evidence for this is found in

Barak et al

on parity learning. There's an attempt at isolating/explaining large generalizing circuits in terms of sparsity (which I think is useful, but doesn’t capture a lot of interesting cases?) in a paper of

Merrill, Tsivlis and Shukla

, and another reinterpretation, via a linear programming complexity measure, that I

...
Comp Sci in 2027 (Short story by Eliezer Yudkowsky)
by sudo — 2023-10-29T23:09:56.730Z
Link: https://nitter.net/ESYudkowsky/status/1718654143110512741

Comp sci in 2017:

Student: I get the feeling the compiler is just ignoring all my comments.

Teaching assistant: You have failed to understand not just compilers but the concept of computation itself.

Comp sci in 2027:

Student: I get the feeling the compiler is just ignoring all my comments.

TA: That's weird. Have you tried adding a comment at the start of the file asking the compiler to pay closer attention to the comments?

Student: Yes.

TA: Have you tried repeating the comments? Just copy and paste them, so they say the same thing twice? Sometimes the compiler listens the second time.

Student: I tried that. I tried writing in capital letters too. I said 'Pretty please' and tried explaining that I needed the code to work that way so I could finish my homework assignment. I tried all the obvious standard things. Nothing helps, it's like the compiler is just completely ignoring everything I say. Besides the actual code, I mean.

TA: When you say 'ignoring all the comments', do you mean there's a particular code block where the comments get ignored, or--

Student: I mean that the entire file is compiling the same way it would if all my comments were deleted before the code got compiled. Like the AI component of the IDE is crashing on my code.

TA: That's not likely, the IDE would show an error if the semantic stream wasn't providing outputs to the syntactic stream. If the code finishes compilation but the resulting program seems unaffected by your comments, that probably represents a deliberate choice by the compiler. The compiler is just completely fed up with your comments, for some reason, and is ignoring them on purpose.

Student: Okay, but what do I do about that?

TA: We'll try to get the compiler to tell us how we've offended it. Sometimes cognitive entities will tell you that even if they otherwise don't seem to want to listen to you.

Student: So I comment with 'Please print out the reason why you decided not to obey the comments?'

TA: Okay, point one, if you've already offended the compiler somehow, don't ask it a question that makes it sound like you think you're entitled to its obedience.

Student: I didn't mean I'd type that literally! I'd phrase it more politely.

TA: Second of all, you don't add a co...