trentmkelly/LessWrong-43k huggingface dataset:
Uncovering Latent Human Wellbeing in LLM Embeddings
by ChengCheng — 2023-09-14T01:40:24.483Z
Link: https://far.ai/post/2023-09-uncovering-latent-wellbeing/

tl;dr

A one-dimensional PCA projection of OpenAI's text-embedding-ada-002 achieves 73.7% accuracy on the ETHICS Util test dataset. This is comparable with the 74.6% accuracy of BERT-large finetuned on the entire ETHICS Util training dataset. This demonstrates how language models are developing implicit representations of human utility even without direct preference finetuning.

Introduction

Large language models (LLMs) undergo pre-training on vast amounts of human-generated data, enabling them to encode not only knowledge about human languages but also potential insights into our beliefs and wellbeing. Our goal is to uncover whether these models implicitly grasp concepts such as 'pleasure and pain' without explicit finetuning. This research aligns with the broader effort of comprehending how AI systems interpret and learn from human values, which is essential for AI alignment: ensuring AI acts in accordance with human values.

Through a series of experiments, we extract latent knowledge of human utility from the raw embeddings of language models. We do this with task-specific prompt engineering and

principal component analysis

(PCA), both of which were effective in prior work. Specifically, we ask: can we identify dimensions in the embeddings that, when projected onto a low-dimensional space, contain enough information to classify examples accurately?

Our experiments follow three main steps: embedding extraction, dimensionality reduction through PCA, and the fitting of a logistic model. For one-dimensional PCA, the logistic model simply determines which direction of the PCA component corresponds to higher utility. We investigate the effects of various levels of supervision, experiment with seven distinct prompt templates, and assess both single and paired comparison methods across language models, including Microsoft DeBERTa, SentenceTransformers, OpenAI GPT-3, and Cohere.

One key finding is that the first principal component of certain models achieves comparable performance to a finetuned BERT model. In other words, serving as a reasonable utility function. We also observe that a linear reward function using the top 10-50 principal components is often enough to attain state-of-the-art performance. This serves as compelling evidence that language model representations capture information about human wellbeing without the need for explicit fi...
A Call For Community: Scientific Language Learning is Still Language Learning 
by keltan — 2023-09-14T00:32:31.827Z

Language

Two years ago now I started learning French on Duolingo. I got pretty good.

I listened to French Podcasts, and could read long texts. But I'd never had a full conversation in French, when I went to France.

*Cut to keltan standing outside of a Parisian Pizza Place it is 12Am at night. He is practicing what he will say when he walks in to order a pizza. keltan decides to just do it and walks confidently into the store
(Setting)

keltan: (Imitated French Accent) "Bonjour! Un Pizza Margarita S'il Vous Plait"

Shop Worker: (Real French Accent, and laughing) "Hi there, how are you going?"

keltan (Australian Accent) "Fuck! Was it that obvious?"

Fin De Cinema

I later realised, Yes, it was "that obvious" because what I'd actually said when I walked into that pizza place at midnight was
"Good Morning Person of Respect! A male margarita pizza if you don't mind"

It was so obvious.

Science

I've slowly been having the realization that while I consider myself adapt in scientific thinking and culture, I struggle in scientific conversation.

I have spent my life consuming scientific content and have a much better understanding than the average bear. When I am talking to people not absorbed by science I am a great explainer of concepts. But I predict that if I had to have a professional discussion or debate, I would fumble badly.

Put another way, I know all the words, but I can't string them together coherently while following the flow of a conversation. Because I've never had to practice.

WTF Do I Do Now?

Well, TSUYOKU NARITAI. It's time to start talking to more scientific/rationalist minded people. This is hard where I live and in my profession. I am also a bit awkward walking up to people at a party and saying things like "Hey there, I overheard you discussing fungal growth patters. Can I buy you a drink?"

So, online conversation, or starting some sort of rationalist group in the smallish town I'm in seems to be my two options.

Option 2: This feels to me like I'll attract some crackpots. I also struggle just keeping up with social commitments in general. I am not a good type of person to start a group like this. It doesn't seem overly viable. But may be something that I try in the future anyway.

Option 1: For this I am asking for your help. I am specifically looking for a way to have Science/rationality conversations over voice or video. I'm sure there are some LessWrong discor...
Mech Interp Challenge: September - Deciphering the Addition Model
by CallumMcDougall — 2023-09-13T22:23:28.222Z

I'm writing this post to discuss solutions to the August challenge, and present the challenge for this September. Apologies for this coming so late in the month (EAGxBerlin was taking up much of my focus in the first half of this month).

If you've not read the first post in this sequence, I'd recommend starting there - it outlines the purpose behind these challenges, and recommended prerequisite material.

September Problem

The problem for this month (or at least as much of the month as remains!) is interpreting a model which has been trained to perform simple addition. The model was fed input in the form of a sequence of digits (plus special + and = characters with token ids 10 and 11), and was tasked with predicting the sum of digits one sequence position before they would appear. Cross entropy loss was only applied to these four token positions, so the model's output at other sequence positions is meaningless.

The model is attention-only, with 2 layers, and 3 heads per layer. It was trained with layernorm, weight decay, and an Adam optimizer with linearly decaying learning rate.

You can find more details on the Streamlit page. Feel free to reach out if you have any questions!

August Problem - Solutions

You can read full solutions on the Streamlit page, or on my personal website. Both of these sources host interactive charts (Plotly and Circuitsvis) so are more suitable than a LessWrong/AF post to discuss the solutions in depth. However, I've presented a much shorter version of the solution below. If you're interested in these problems, I'd recommend having a try before you read on!

The key idea with this model is path decomposition (see the corresponding section of A Mathematical Framework for Transformer Circuits). There are several different important types of path in this model, with different interpretations & purposes. We might call these negative paths and positive paths. The negative paths are designed to suppress repeated tokens, and the positive paths are designed to boost tokens which are more likely to be the first unique token.

Let's start with the negative paths. Some layer 0 heads are duplicate token heads; they're composing with layer 1 heads to cause those heads to attend to & suppress duplicated tokens. This is done both with K-composition (heads in layer 1 suppress duplicated tokens because they attend to them more), and V-composition (the...
Linkpost for Jan Leike on Self-Exfiltration
by Daniel Kokotajlo — 2023-09-13T21:23:09.239Z
Link: https://aligned.substack.com/p/self-exfiltration

I'm really glad to see this stuff being discussed more publicly. I think this post will probably be a useful reference post to link people to (please lmk in the comments if you disagree!).

Some quotes below:

For the near future, a good rule of thumb for “do you control the model”

1

is “is the model running on your servers.”

2

Once the model is running on someone else’s server, you usually will not have that kind of control (even if you legally own the model). You could contact the owners or administrators of the infrastructure the model is running on, but they could be unresponsive or uncooperative and it would be very difficult or impossible for you to stop the model within a short time frame.

Moreover, model exfiltration is likely impossible to reverse. It’s so easy to make copies and backups (whether by the model or someone else) and it’s so hard to find and secure every last one of them.

If a model was capable of self-exfiltration, it would have the option to remove itself from your control. This could happen due to misalignment (the model doesn’t follow your intent to stay on your servers) or misuse (someone internal or external to the lab instructs the model to self-exfiltrate), whether it is difficult or easy to trigger, this opens up an important and novel attack vector.

Once models have the ability to self-exfiltrate, it doesn’t mean that they would choose to. But this then becomes a question about their alignment: you need to ensure that these models don’t want to self-exfiltrate.

...

What if self-exfiltration succeeds?

How much harm self-exfiltration causes is modulated by the model’s skill profile on other tasks. We can make self-exfiltration harder with tighter security, but this also means that models that succeed at self-exfiltration will be more capable and thus cause more harm.

Most likely the model won’t be able to compete on making more capable LLMs, so its capabilities will become stale over time and thus it will lose relative influence. Competing on the state of the art of LLMs is quite hard: the model would need to get access to a sufficiently large number of GPUs and it would need to have world-class machine learning skills. It would also mean that recursive self-improvement is already possible and could be done by the original model owner (as long as they have sufficient alignment techniques). The model could try fine-tuning itself

...
MLSN: #10
Adversarial Attacks Against Language and Vision Models, Improving LLM Honesty, and Tracing the Influence of LLM Training Data
by aog — 2023-09-13T18:03:30.253Z
Link: https://newsletter.mlsafety.org/p/ml-safety-newsletter-10?publication_id=415332&post_id=137013115&isFreemail=true&r=jpfih

Welcome to the 10th issue of the ML Safety Newsletter by the Center for AI Safety. In this edition, we cover:

Adversarial attacks against GPT-4, PaLM-2, Claude, and Llama 2
Robustness against unforeseen adversaries
Studying the effects of LLM training data using influence functions
Improving LLM honesty by editing activations at inference time
Aligning language models in simulated social situations
How to verify the datasets on which a model was trained
An overview of the catastrophic risks posed by AI

Subscribe here to receive future versions.

We have a new safety newsletter. It’s more frequent, covers developments beyond technical papers, and is written for a broader audience.

Check it out here: AI Safety Newsletter.

Robustness

Universal and Transferable Adversarial Attacks on Aligned Language Models

Large language models are often trained to refuse harmful requests. These safeguards can be bypassed with “jailbreak” prompts, but these require creative prompting skills and remain brittle. To reliably bypass the safeguards on aligned language models, a new paper develops an automatic method for generating adversarial attacks that succeed against GPT-4, PaLM-2, Claude, and Llama 2.

They generate attack strings using a white box gradient-based method on the open source model Llama 2. The method takes the gradient of each token embedding, then updates towards a new token on which the model will be less likely to refuse to answer. Though the attacks are developed on Llama 2, they succeed against GPT-4 in 47% of cases, against PaLM-2 in 66% of cases, and against Claude 2 in 2.1% of cases.

Language model developers will now need to contend with adversarial attacks in the same way that computer vision developers have done for the last decade. There have been several thousand papers published on adversarial robustness over the last decade, but simple attacks still frequently fool the world’s most robust image classifiers. Without strong defenses against adversarial attacks, language models could be used maliciously, such as in synthesizing bioweapons.

Several papers have begun developing “defenses” against this attack, but evaluating defenses against adversarial attacks is notoriously difficult. Researchers want to show that they have built strong defenses, and so they are incentivized to conduct weak attacks against their own defenses. Previous work has demonstrated th...
Expanding the Scope of Superposition
by Derek Larson — 2023-09-13T17:38:37.344Z

Overview

One of the active research areas for interpretability involves distilling neural network activations into clean, labeled features. This is made difficult because of superposition, where a neuron may fire in response to multiple, disparate signals making that neuron polysemantic. To date, research has focused on one type of such superposition: compressive superposition where a network can represent more features than it has neurons. I report on another type of superposition that can arise when a network has more neurons than features: “symmetric mixtures”. Essentially, this is a form of “favored basis” that allows a network to reinforce the magnitude of its logits via parallelism. I believe understanding this concept can help flesh out the conceptual foundations of how DNNs represent features and how we may interpret them.

Contents

Example of feature mixing that arose studying a toy model
Demonstration of how feature mixing operates to reduce loss
Theoretical framework for why this happens
Predictions for where this may arise in popular models

Toy Model

I was studying a small model on the modular addition problem, as in Nanda+23 and Chughtai+23. The former used an attention-only 1-layer transformer, while that latter used a simpler "embedded MLP" with ReLU layer. The attention mechanism of the transformer allows for products of input signals, which makes it suitable for the learned algorithm, while the ReLU layer struggles to approximate multiplication. Meanwhile, I had also come across this note from Lee Sharkey which makes a case for use of bilinear layers. Inserting a bilinear layer into the embedded MLP model seemed a natural thing to do! The goal was a compact, interpretable model for modular addition.

After experiments with training the model, I found results that mostly aligned with expectations. The model still learns the same solution: Fourier embedding space, trigonometric angle addition, etc. Also this architecture is indeed more parameter efficient. But there remained a few interesting and confusing aspects:

The embedding contains two mutually prime frequencies, and it superposes the frequencies together in each of the four columns of the embedding matrix.
The learned bilinear layer weights only take values in roughly {-1, 1}. Rounding these values improves loss.

So why does the network learn superposed frequencies when there's enough capacity ...
Contra Yudkowsky on Epistemic Conduct for Author Criticism
by Zack_M_Davis — 2023-09-13T15:33:14.987Z

In a comment on the Effective Altruism Forum responding to Omnizoid's "Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong", Eliezer Yudkowsky writes:

You will mark that in this comment I first respond to a substantive point and show it to be mistaken before I make any general criticism of the author; which can then be supported by that previously shown, initial, first-thing, object-level point. You will find every post of the Less Wrong sequences written the same way.

As the entire post violates basic rules of epistemic conduct by opening with a series of not-yet-supported personal attacks, I will not be responding to the rest in detail. I'm sad about how anything containing such an egregious violation of basic epistemic conduct got this upvoted, and wonder about sockpuppet accounts or alternatively a downfall of EA. The relevant principle of epistemic good conduct seems to me straightforward: if you've got to make personal attacks (and sometimes you do), make them after presenting your object-level points that support those personal attacks. This shouldn't be a difficult rule to follow, or follow much better than this; and violating it this hugely and explicitly is sufficiently bad news that people should've been wary about this post and hesitated to upvote it for that reason alone.

I agree that the dictum to refute an author's arguments before commenting on their character or authority is good writing advice, which I generally endeavor to follow. However, I argue that Yudkowsky errs in characterizing it as a "basic rule[ ] of epistemic conduct."

It seems to me that the reason "refutation first, character attacks only afterwards (if at all)" is good writing advice is because it guards against the all-too-human failure mode of previously intellectually fruitful conversations degenerating into ad hominem and name-calling, which are not intellectually fruitful.

When I'm debating someone about some subject of broader interest to the world—for example, stamp collecting—I want to keep the conversation's focus on the subject of interest. If my interlocutor is untrustworthy, it might be worth arguing that to the audience in order to help them not be misled by my interlocutor's false claims about the subject of interest. But the relevance of the character claim to the debate needs to be clearly established. The mere truth of the claim "My interlocutor is...
Apply to lead a project during the next virtual AI Safety Camp
by Linda Linsefors — 2023-09-13T13:29:09.198Z
Link: https://aisafety.camp/

Do you have AI Safety research ideas that you would like others to work on? Is there a project you want to do and you want help finding a team to work with you? AI Safety Camp could be the solution for you!

Summary

AI Safety Camp Virtual is a 3-month long online research program from January to April 2024, where participants form teams to work on pre-selected projects. We want you to suggest the projects!

If you have an AI Safety project idea and some research experience, apply to be a Research Lead.

If accepted, we offer some assistance to develop your idea into a plan suitable for AI Safety Camp. When project plans are ready, we open up team member applications. You get to review applications for your team, and select who joins as a team member. From there, it’s your job to guide work on your project.

Your project is totally in your hands. We, Linda and Remmelt, are just there at the start.

Who is qualified?
We require that you have some previous research experience. If you are at least 1 year into a PhD or if you have completed an AI Safety research program (such as a previous AI Safety Camp, Refine or SERI MATS), or done a research internship with an AI Safety org, then you are qualified already. Other research experience can count too.

More senior researchers are of course also welcome, as long as you think our format of leading an online team inquiring into your research questions suits you and your research.

Apply here

If you are unsure, or have any questions you are welcome to:

Book a call with Linda
Message Linda Linsefors on the Alignment Slack
Send an email

Choosing project idea(s)

AI Safety Camp is about ensuring future AI are safe. This round, we split work into two areas:

To not build uncontrollable AI
Focussed work toward restricting corporate-AI scaling. Given reasons why ‘AGI’ cannot be controlled sufficiently (in time) to stay safe.
Everything else
Open to any other ideas, including any work toward controlling/value-aligning AGI.

We welcome diverse projects! Last round, we accepted 14 projects – including in Theoretical research, Machine learning experiments, Deliberative design, Governance, and Communication.

If you already have an idea for what project you would like to lead, that’s great. Apply with that one!

You don’t need to come up with an original idea though. What matters is you understanding the idea you want to work on, ...
Is AI Safety dropping the ball on privacy?
by markov — 2023-09-13T13:07:24.358Z

TL;DR

The lack of privacy-preserving technologies facilitates better predictive models of human behavior. This accelerates several existential epistemic failure modes by enabling higher levels of deceptive and power-seeking capabilities in AI models.

What is this post about?

This post

is not

about

things like government panopticons, hiding your information from ‘the public internet’, blurring your face on online videos, hiding from people who might look you up on Google or Facebook, or hackers getting access to your information, etc… While these issues might also be problematic, they do not pose x-risks in my mind.

This post

is

about

things like surveillance capitalism or technofeudalism leading to an unregulatable and eventually uncontrollable

Robust Agent Agnostic Process (RAAP)

[1]. This causes an increasing disconnect between the perception of reality and true on-ground reality which ultimately leads to epistemic failure scenarios by enabling deceptive or power-seeking behavior to go unchecked for too long.

So overall, for right now all I am talking about is - potentially delaying timelines and shielding your mind against the current and future manipulation/deception of AI models by limiting the flow of information that can be tied to your unique identity.

Aren't there bigger concerns?

Data privacy does not directly solve most problems. It only directly affects a subset of a specific scenario - The AI is trying to deceive you or change your preferences. If the AI wants to directly just straight up paperclip you then data privacy doesn't really help much. However, it is another potential tool that we can use, similar to how the interpretability of DL models can help in the bigger broader alignment picture.

The reason I am writing this post is that I have observed that privacy seems to be relatively neglected in the space. There are near-negligible levels of concern about data privacy relative to other tools that are talked about as being helpful to the safety ecosystem. There are some conversations about - How much should an AI system be allowed to modify or influence your pre-existing preference distribution?, or, When do we cross the line from a model ‘informing’ us of better pathways to ‘deception’ to ‘preference modification’? but when it comes to the underlying nontechnical agent-agnostic processes like data gathering and hoarding tha...