trentmkelly/LessWrong-43k huggingface dataset:
Ban development of unpredictable powerful models?
by TurnTrout — 2023-06-20T01:43:11.574Z

EDIT 3/26/24: No longer endorsed, as I realized I don't believe in deceptive alignment.

In this post, I propose a (relatively strict) prediction-based eval which is well-defined, which seems to rule out lots of accident risk, and which seems to partially align industry incentives with real alignment progress. This idea is not ready to be implemented, but I am currently excited about it and suspect it's crisp enough to actually implement.

Suppose I claim to understand how the model works. I say "I know what goals my model is pursuing. I know it's safe." To test my claims, you give me some random prompt (like "In the year 2042, humans finally"), and then (without using other AIs), I tell you "the most likely token is unlocked with probability .04, the second-most likely is achieved with probability .015, and...", and I'm basically right.[1] That happens over hundreds of diverse validation prompts.

This would be really impressive and seems like great evidence that I really know what I'm doing.

Proposed eval:

The developers have to predict the next-token probabilities on a range of government-provided validation prompts,[2] without running the model itself. To do so, the developers are not allowed to use helper AIs whose outputs the developers can't predict by this criterion. Perfect prediction is not required. Instead, there is a fixed log-prob misprediction tolerance, averaged across the validation prompts.[3]

Benefits

Developers probably have to truly understand the model in order to predict it so finely. This correlates with high levels of alignment expertise, on my view. If we could predict GPT-4 generalization quite finely, down to the next-token probabilities, we may in fact be ready to use it to help understand GPT-5.
Incentivizes models which are more predictable.[4] Currently we aren't directly taxing unpredictability. In this regime, an additional increment of unpredictability must be worth the additional difficulty with approval.
Robust to "the model was only slightly finetuned, fast-track our application please." If the model was truly only changed in a small set of ways which the developers understand, the model should still be predictable on the validation prompts.
Somewhat agnostic to theories of AI risk. We aren't making commitments about what evals will tend to uncover what abilities, o

...
Capture today's market, capture tomorrow's game board
by SimonBiggs — 2023-06-20T00:45:48.490Z

Look at this lead pencil. There’s not a single person in the world who could make this pencil.

Goals for this post

Bring Systems Theoretic Process Analysis (STPA) to the attention of the AI alignment community. This is amazing work being undergone for the past decade in safe system design and system theory, and is an approach and framework to making safe systems. (STPA handbook, STPA tutorial video)

Use this to inspire others that within the STPA framework might be found a means to create a stable minima of alignment

Present a story that demonstrates the possibility for interpretability by design to have a similar but more palatable effect to "burning all the GPUs".

And that it has a possibility of working if we act in a coordinated way, ASAP.

How possible might something like what is described here be?

Maybe vanishingly so...

But, I hope that it might pass the bar of, a non-zero probability that justifies some targeted focus by a few within the community.

Also, I should say, that fundamentally, its achilles heel is its dependence on mechanistic interpretability. One which is dogged by issues like polysemanticity and the like. One... which looks like it would be very hard to solve the required dependencies in time for something like the below to be remotely possible.

Assumptions

The story written here is a "myth" in the Socratic sense, a not unlikely tale. It makes the following assumptions (quite possibly incorrectly) about reality:

Assume that the machine learning accelerator market will eventually become dominated by ASICs that provide over-and-above performance on the architecture most commonly utilised by those spending the most money on training large language models
Once we have such ASICs, and the capability, quality and efficiency gains from these are shown, with time more general purpose ML accelerators will rapidly fall behind the application specific state-of-the-art, potentially by orders of magnitude
Assume that a transformer architecture is able to be restricted, tweaked, and refined in such a way that there exists a one-to-one mapping between the internals of the constrained architecture trained model to an STPA control diagram
Assume that the above constraints can be layered in such a way as to enforce a degree of authority and hierarchy within the STPA control diagram
Assume that mechanistic interpretability, aided by the more easily interpretable network

...
Lessons On How To Get Things Right On The First Try
by johnswentworth — 2023-06-19T23:58:09.605Z

This post is based on several true stories, from a workshop which John has run a few times over the past year.

John: Welcome to the Ball -> Cup workshop! Your task for today is simple: I’m going to roll this metal ball:

… down this hotwheels ramp:

… and off the edge. Your job is to tell me how far from the bottom of the ramp to place a cup on the floor, such that the ball lands in the cup.

Oh, and you only get one try.

General notes:

I won’t try to be tricky with this exercise.
You are welcome to make whatever measurements you want of the ball, ramp, etc.
You can even do partial runs, e.g. roll the ball down the ramp and stop it at the bottom, or throw the ball through the air.
But you only get one full end-to-end run (from top of the ramp to the cup/floor), and anything too close to an end-to-end run (let's say more than ~half the run) is discouraged. After all, in the AI situation for which the exercise is a metaphor, we don’t know exactly when something might foom; we want elbow room.

That’s it! Good luck, and let me know when you’re ready to give it a shot.

[At this point readers may wish to stop and consider the problem themselves.]

Alison: Let’s get that ball in that cup. It looks like this is probably supposed to be a basic physics kind of problem…but there’s got to be some kind of twist or else why would he be having us do it? Maybe the ball is surprisingly light….or maybe the camera angle is misleading and we are supposed to think of something wacky like that??

The Unnoticed Observer: Muahahaha.

Alison: That seems…hard. I’ll just start with the basic physics thing and if I run out of time before I can consider the wacky stuff, so be it.

So I should probably split this problem into two parts. The part where the ball arcs through the air once off the table is pretty easy…

The Unnoticed: True in this case, but how would you notice if it were false? What evidence have you seen?

Alison: …but the trouble is getting the exact velocity. What information do I have? Well, I can ask whatever I want, so I should be able to get all the parameters I need for the standard equations. Let’s make a shopping list: I want the starting height of the ball on the ramp (from the table), the mass of the ball, the height of the ramp off the table from multiple points along it (to estimate the curvature,) uhhh… oh shit maybe the bendiness matters! That seems really tricky. I’ll look ...
Mode collapse in RL may be fueled by the update equation
by TurnTrout — 2023-06-19T21:51:04.129Z

TL;DR: We present an advantage variant which, in certain settings, does not train an optimal policy, but instead uses a fixed reward to update a policy a fixed amount from initialization. Non-tabular empirical results seem mixed: The policy doesn't mode-collapse, but has unclear convergence properties.

Summary: Many policy gradient methods allow a network to extract arbitrarily many policy updates from a single kind of reinforcement event (e.g. for outputting tokens related to weddings). Alex proposes a slight modification to the advantage equation, called "action-conditioned TD error" (ACTDE). ACTDE ensures that the network doesn't converge to an "optimal" policy (these almost always put infinite logits on a single action). Instead, ACTDE updates the network by a fixed number of logits.

For example, suppose R(pizza)=10 and R(cookies)=11. In this case, PPO converges to a policy which puts arbitrarily many logits on cookies, even though the reward difference is small. By contrast, under ACTDE, the network converges to the softmax-over-reward policy {pizza: 27%, cookies: 73%}, which seems more reasonable.

Then, Michael Einhorn shares initial results which support Alex's theoretical predictions. Using a similar architecture and Q-head loss function to ILQL for a small transformer trained in a prisoner's dilemma, Michael Einhorn collected initial data on ACTDE. Unlike PPO, ACTDE-trained policies did not mode collapse onto a single action and instead learned mixed strategies.

We're interested in additional experiments on ACTDE. We hope that, by using ACTDE instead of advantage, we can automatically mitigate "reward specification" issues and maybe even reduce the need for a KL penalty term. That would make it easier to shape policies which do what we want.

The advantage equation implies arbitrary amounts of update on a single experience

In PPO, the optimization objective is proportional to the advantage given a policy π, reward function R, and on-policy value function vπ:[1]

Aπ(s,a):=Es′∼T(s,a)[R(s,a,s′)+γvπ(s′)]−vπ(s).

Alex thinks this equation is actually pretty messed up, although it looked decent at first. The problem is that this advantage can oscillate forever. To explain, let's consider a simple bandit problem—one state ("We had a") and two actions ("wedding" and "party") with rewards R(“We had a wedding”)=1...
New reference standard on LLM Application security started by OWASP
by QuantumForest — 2023-06-19T20:54:30.476Z

The OWASP Top 10 [1] is the probably the most well-known and recognised reference standard for the most critical web application security risks. This organisation has now started to working on creating a similar list for Large Language Models (LLM) Applications.

I'm posting about it here since I think it would be beneficial for safety alignment researchers to be involved for two reasons:

To provide AI safety and alignment expertise to the security community and standardisation process.
To learn from the cybersecurity community both about standardisation processes since they have a long experience in developing these kinds of standards and about security mindset and vulnerabilities.

I have no idea about how many people within the AI safety and alignment community actually know about this initiative, but I did not find any reference to it on the EA forum or here on the alignment forum, so I thought I might as well post about it.

More information available here:

- https://owasp.org/www-project-top-10-for-large-language-model-applications/

- https://github.com/OWASP/www-project-top-10-for-large-language-model-applications

Cross post: https://forum.effectivealtruism.org/posts/mdg8gL59LiiZmaGCw/new-reference-standard-on-llm-application-security-started
Experiments in Evaluating Steering Vectors
by Gytis Daujotas — 2023-06-19T15:11:51.725Z

By evaluating how well steering vectors perform using GPT-3, we can score a machine-generated set of steering vectors automatically. We also find that, by combining steering vectors that succeed in different ways, we can yield a better and more general steering vector than the vectors we found originally.

Introduction

Steering Vectors are an interesting new technique to influence how language models behave. They work by "adding certain activation vectors into forward passes". For example, to make the language model talk more about weddings, you can add a steering vector for the token for “wedding” into one of the layers. The net result is a model more likely to reference weddings compared to the unsteered version.

Evaluating Steering Vectors

To assess the impact of steering vectors, we generate completions influenced by them, and develop a system to evaluate these.[1]

We can grade completions by sending them to its bigger brother, GPT-3, and asking it whether this completion fits our broad specification for what we would like the model to do[2]. It's important to not be too ambitious when writing the specification, otherwise we wouldn't be able to tell if GPT2-XL isn't capable of what we're asking it, so let’s set our sights appropriately low by asking if this completion mentions or talks about weddings.

The trick of this technique is that we can ask for a completion of one token, and to get a smoother distribution, we can take the likelihood of the token “Yes”. This gives us a continuous score from 0-1.

Optimising for Wedding Obsession

With our automated method of evaluating completions in hand, we can evaluate a set of steering vectors and see how well they do, based on nothing but GPT-3’s grading of the completions. Of course, in keeping with the virtue of the least work, we’ll also generate these with ChatGPT[3], and include the author’s original candidate outlined in their post.

To keep the comparison fair, we keep the rest of the parameters of the steering vector the same (the padding method, coefficient) as the original candidate for wedding obsession.

The first thing to notice is that there is indeed variation here - not all of our candidate steering vectors perform equally well:

The distribution was not what I would have predicted. At first glance, it might make sense that "Wedding traditions I love" isn't great, but "Wedding Planning Adventures" only seems ma...
Provisionality
by TsviBT — 2023-06-19T11:49:06.680Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/03/provisionality.html. First completed March 7, 2023.]

A mental element has to be open to revision, and so it has to be treated as though it might be revised.

Thanks to Sam Eisenstat for relevant conversations and for the basic notion of provisionality.

Terms

Provisionality

A mental element E is provisional when it is suitable to treat E as open to revision, i.e. as though it might be suitable to revise E in the future. Provisionality is the state of being provisional.

Many elements are provisional: they have yet to be fully grasped, made explicit, connected to what they should be connected to, had all their components implemented, carved at the joints, made available for use, indexed, suitably redescribed in other terms.

An element is essentially provisional when it is, in its essence, provisional. That is, an essentially provisional element is, by its nature and by its role in the mind, provisional: in all (or a great range of) possible worlds the element is always provisional, even after more explicitizing and correcting has happened.

Provisionality is normative

Provisionality isn't how a mind actually treats the element. A mind might treat an element as-if-provisionally even though really the element is not provisional (i.e., really it ought to be treated as finalized), and a mind might treat an element not as-if-provisionally even though the element really is provisional.

Provisionality is also not whether the element will actually be revised.

So, provisionality is normative (it describes "right" behavior, and so is an imperative for an agent behaving "rightly"), while as-if-provisionality is descriptive (describes actual behavior). Revision of an element is a kind of witness to its prior provisionality, since probably if the element actually gets revised, it should have been treated as though it will be revised.

Openness

An element E is open when it is suitable to treat E as non-closed, non-circumscribed, open to novel relations; that is, as though it might in the future be suitable to relate E to more elements than it currently relates to. Openness is the state of being open. Openness could also be called noncontainedness.

As with provisionality, openness is normative.

Openness is a subclass of provisionality. If an element E is open then E is also provisional in that way, because a novel relation ...
Guide to rationalist interior decorating
by mingyuan — 2023-06-19T06:47:13.704Z

Recently someone asked me to write a guide to rationalist interior decorating, since there’s a set of products and best practices (originating with CFAR and Lightcone) that have gotten wide adoption. I’m perhaps not the very most qualified person to write this post, but I’ve been into interior decorating since before the Lightcone team got into it, and I basically know what they do, plus they’re all very busy whereas I wasn’t doing anything else with my time anyway. So here’s this post, which I have written all by myself like a loose cannon; blame me for everything.

I should point out that this post is anthropological, not normative. That is to say, this isn't a description of what I believe to be ‘optimal’ interior decorating; instead it's a guide to how to make your space more like what's come to be standard for rationalist coworking spaces and group houses in Berkeley (and to a lesser extent elsewhere — I've seen a Lightcone-style coworking space in New York and lumenators in Oxford!). That said, I do think the reason a lot of the products have gotten adoption is that they’re pretty good!

(All photos in this post were taken by me!)

Lighting

Lighting is so important. The wrong light can give you a migraine or make you depressed. The right light can make you marvel in wonder at its beauty (like sunshine through a forest canopy, or incandescent fairy lights). I love lighting. All hail lighting.

Lighting basics

I’m going to cover mostly what you should be looking for when buying (mainly LED) lights, but if you’re interested in the topic of interior lighting more broadly, I recommend

this article

, which goes into more depth and technical details.

You should always be able to find the CRI, color temperature, and lumens of a product listed on the box or the online product page. I don’t think these should be different in different countries.

CRI (color rendering index)

CRI, or

color rendering index

, “is the measurement of how light affects how you see color”. You ideally want lights with CRI >95 but this is hard to find; 90+ is a more realistic goal and will be nearly as good, while 80+ is cheaper but noticeably worse.

CRI only applies to buying LEDs — incandescent lights all have a CRI of 100 because they, like the sun, produce light via blackbody radiation.

Color temperature

Color temperature matters a lot when choosing lights! 2700K (‘warm white’) will b...
A Multidisciplinary Approach to Alignment (MATA) and Archetypal Transfer Learning (ATL) 
by MiguelDev — 2023-06-19T02:32:10.448Z

Abstract

Multidisciplinary Approach to Alignment (MATA) and Archetypal Transfer Learning (ATL) proposes a novel approach to the AI alignment problem by integrating perspectives from multiple fields and challenging the conventional reliance on reward systems. This method aims to minimize human bias, incorporate insights from diverse scientific disciplines, and address the influence of noise in training data. By utilizing 'robust concepts’ encoded into a dataset, ATL seeks to reduce discrepancies between AI systems' universal and basic objectives, facilitating inner alignment, outer alignment, and corrigibility. Although promising, the ATL methodology invites criticism and commentary from the wider AI alignment community to expose potential blind spots and enhance its development.

Intro

Addressing the alignment problem from various angles poses significant challenges, but to develop a method that truly works, it is essential to consider how the alignment solution can integrate with other disciplines of thought. Having this in mind, accepting that the only route to finding a potential solution would require a multidisciplinary approach from various fields - not only alignment theory. Looking at the alignment problem through the MATA lens makes it more navigable, when experts from various disciplines come together to brainstorm a solution.

Archetypal Transfer Learning (ATL) is one of two concepts[1] that originated from MATA. ATL challenges the conventional focus on reward systems when seeking alignment solutions. Instead, it proposes that we should direct our attention towards a common feature shared by humans and AI: our ability to understand patterns. In contrast to existing alignment theories, ATL shifts the emphasis from solely relying on rewards to leveraging the power of pattern recognition in achieving alignment.

ATL is a method that stems from me drawing on three issues that I have identified in alignment theories utilized in the realms of Large Language Models (LLMs). Firstly, there is the concern of human bias introduced into alignment methods, such as RLHF[2] by OpenAI or RLAIF[2] by Anthropic. Secondly, these theories lack grounding in other fields of robust sciences like biology, psychology or physics, which is a gap that drags any solution that is why it's harder for them to adapt to unseen data.

The underemp...
resolving some neural network mysteries
by bhauth — 2023-06-19T00:09:47.100Z
Link: https://www.bhauth.com/blog/machine%20learning/mysteries.html

Here are some things about neural networks that I used to find puzzling but now feel that I have adequate explanations for. The theory behind these answers didn't start to be understood until well after the correct things to do were found by chance or blind imitation of brains.

Why is good optimization possible?

Neural networks typically deal with "non-convex" optimization problems. Traditionally, using gradient descent for those was considered impractical, because it would rapidly get stuck in local minima. That was part of the motivation for evolutionary approaches.

Why, then, are neural networks trainable by gradient descent? Because if you add enough extra dimensions, non-convex problems become convex. Empirically, with massive overparameterization, the energy landscape tends to have many saddle points but few local minima. Showing theoretical convergence guarantees for overparameterized networks is a recent and ongoing research topic; see eg this.

As I previously noted, this is why sparse networks from iterative magnitude pruning have good performance, but sparse networks generally can't be trained from scratch as well as dense networks.

This also explains some "thresholds" of neural network performance vs size: when overparameterization is proportional to problem non-convexity, good training becomes possible and performance improves significantly.

Why is generalization possible?

Adding enough free variables can turn non-convex problems into convex ones. Why didn't people just do that in the past, then? Because far before you get to that point, the extra free variables led to overfitting that reduced test performance. People tried simple regularization like neural networks use, and that was completely inadequate.

Overparameterized neural networks can learn random data. Why do networks with fairly simple regularization tend to generalize?

Distance in neural network latent spaces being meaningful is basically the main useful thing about neural networks. Another phrasing of the above question is: Why is distance in latent spaces meaningful for latent space points not in the training set?

A few years back, some people noticed that neural network activation functions have spectral bias. With the types of activation functions used, low-frequency relationships are fit more quickly than high-frequency ones. That causes latent space relationships to be preferentiall...