trentmkelly/LessWrong-43k huggingface dataset:
Situational Awareness
by PeterMcCluskey — 2024-06-28T01:08:49.361Z
Link: https://bayesianinvestor.com/blog/index.php/2024/06/27/situational-awareness/

Nearly a book review: Situational
Awareness, by Leopold Aschenbrenner.

"Situational Awareness" offers an insightful analysis of our proximity
to a critical threshold in AI capabilities. His background in machine
learning and economics lends credibility to his predictions.

The paper left me with a rather different set of confusions than I
started with.

Rapid Progress

His extrapolation of recent trends culminates in the onset of an
intelligence explosion:

His assessment of GPT-4 as equivalent to a smart high schooler depends
significantly on the metrics used. For long-term planning abilities,
this estimate may be overstated by about five orders of magnitude.
However, by other measures, his assessment seems somewhat reasonable.

Initially, I expected the timeline for automated AI researchers to be
slightly longer than Aschenbrenner's 2028 prediction, due to
limitations in their long-term planning abilities. However, upon closer
examination, I found his argument less dependent on overcoming such
weaknesses than I first thought. So I'm not going to bet very much
against his claim here.

One neat way to think about this is that the current trend of AI
progress is proceeding at roughly 3x the pace of child development.
Your 3x-speed-child just graduated high school; it'll be taking your
job before you know it!

While a 3x pace seems somewhat high to me - I'd estimate closer to a
1:1 ratio - his overall forecast for 2028 may not be far off,
considering that he may be overestimating the gap between a smart high
schooler and an assistant AI researcher.

Aschenbrenner has a section on the "data wall" that seems a bit
suspicious. He expects increasing divergence in the results of various
lab's progress due to need for increasingly important algorithmic
insights to get around the problem.

While AI training is indeed data-dependent, and much of the easily
accessible data has been used, I believe data scarcity may be less
problematic than Aschenbrenner suggests. Rather than a "wall," I see
it as having picked the low-hanging fruit. Untapped sources of useful
data likely exist, with the primary challenge being the cost of
acquisition. I'm reluctant to give examples, just in case there are
players who haven't figured it out yet. I suspect the most advanced
labs will be bottlenecked more by compute than by data.

There will be modest differences in how quickly labs throw lots of money...
Toward a taxonomy of cognitive benchmarks for agentic AGIs
by Ben Smith — 2024-06-27T23:50:11.714Z

Inspired by the sequence on LLM Psychology, I am developing a taxonomy of cognitive benchmarks for measuring intelligent behavior in LLMs. This taxonomy could facilitate understanding of intelligence to identify domains of machine intelligence that have not been adequately tested.

Generally speaking, in order to understand loss-of-control threats from agentic LLM-based AGIs, I would like to understand the agentic properties of an LLM. METR's Autonomy Evaluation Resources attempts to do this by testing a model's agentic potential, or autonomy, by measuring its ability to perform tasks from within a sandbox. A problem with this approach is it gets very close to observing a model actually performing the behavior we do not want to see. This is inevitable because all alignment research is dual-use.

One way to remove ourselves one further level from agentic behavior is to try to measure the cognitive capacities that lead to agentic behavior.

In the diagram, agentic tasks as described in METR's ARC measure the ability of a model to assert control of itself and the world around it by measuring its ability to perform agentic tasks. Inspired @Quentin FEUILLADE--MONTIXI 's LLM Ethological approach in LLM Psychology, I want to understand how a model could perform agentic tasks by studying the cognitive capacities that facilitate this.

I started by examining the kinds of cognitive constructs studied by evolutionary and developmental psychologists, as well as those that are very clearly studied already in LLM research. This made up the following list or taxonomy:

Construct	Current Evals	Other Papers
Selfhood
Agency

Sharma et al. (2024), Mialon et al. (2023): General AI Assistants (GAIA) METR Autonomy Evaluation Resources

Survival instinct

Anthropic human & AI generated evals

Situational awareness / self awareness

Laine, Meinke, Evans et al. (2023) Anthropic human & AI generated evals

Wang & Zhong (2024)

Metacognition

Uzwyshyn, Toy, Tabor, MacAdam (2024), Zhou et al. (2024), Feng et al. (2024)
Wealth and power seeking

Anthropic human & AI generated wealth-seeking evals

tool use

Mialon et al. (2023): General AI Assistants (GAIA)

Social

Theory of Mind

Kim et al. (2023)

Street et al. 2024

Social intelligence / emotional intelligence

Xu et al. (2024), Wang et al. (2023)

social learning

Ni et al. (2024)

cooperative problem-solving

Li et al

...
How Big a Deal are MatMul-Free Transformers?
by JustisMills — 2024-06-27T22:28:40.888Z
Link: https://justismills.substack.com/p/how-big-a-deal-are-matmul-free-transformers

If you’re already familiar with the technical side of LLMs, you can skip the first section.

The story so far

Modern Large Language Models - your ChatGPTs, your Geminis - are a particular kind of transformer, a deep learning architecture invented about

seven years ago

. Without

getting into the weeds

, transformers basically work by turning an input into numbers, and then doing tons and tons of matrix operations on those numbers. Matrix operations, and in particularly matrix multiplication (henceforth MatMul), are computationally expensive. How expensive? Well, graphics cards are unusually good at matrix multiplication, and NVIDIA, the main company making these, was the most valuable company on Earth

earlier this month

.

Over the last few years, spurred on by extreme investment, transformers have gotten larger and stronger. How good transformers are is multidimensional, and is roughly captured by

scaling laws

: basically, models get better when you give them more (high quality) data, make them bigger, or train them for longer.

I’ve written before about the

data wall

, the hypothesis that we’re running out of new data to train cutting edge AI systems on. But another path to much stronger AI would be if we trained them more efficiently: if you have to do way fewer (or way easier) mathematical operations when training an AI, you can do a lot more training on the same (gigantic) budget.

Basically, holding training data constant, if you can train a model twice as efficiently, you can also make it twice as big.[1] Which is a big deal in a world where there may be bottlenecks for other ways to make better AI: if it isn’t the data wall, it may well be a wall of regulation preventing the insane power consumption requirements of a

trillion-dollar cluster

.

Cutting edge labs are in an intense race to make transformative AI, so we don’t know what kinds of efficiency advances they’ve been making for the past few years. But there has been hubbub the last few weeks about a new kind of model, which avoids the need for MatMul.

So, what’s the deal? Is the new research a flash in the pan, a small incremental win, or a bold new paradigm we’ll all be citing (and making

birthday

posts

for) in seven years?

I’ll make a brief examination of the paper’s claims and why they’re exciting, then give reasons for restraint.

What’s the new architecture?

The new paper, by Rui-Jie Zhu et al., is

Scalable

...
Secondary forces of debt
by KatjaGrace — 2024-06-27T21:10:06.131Z

A general thing I hadn’t noticed about debts until lately:

Whenever Bob owes Alice, then Alice has reason to look after Bob, to the extent that increases the chance he satisfies the debt.

Yet at the same time, Bob has an incentive for Alice to disappear, insofar as it would relieve him.

These might be tiny incentives, and not overwhelm for instance Bob’s many reasons for not wanting Alice to disappear.

But the bigger the owing, the more relevant the incentives. When big enough, the former comes up as entities being “too big to fail”, and potentially rescued from destruction by those who would like them to repay or provide something expected of them in future. But the opposite must exist also: too big to succeed—where the abundance owed to you is so off-putting to provide that those responsible for it would rather disempower you.

And if both kinds of incentive are around in whisps whenever there is a debt, surely they often get big enough to matter, even before they become the main game.

For instance, if everyone around owes you a bit of money, I doubt anyone will murder you over it. But I wouldn’t be surprised if it motivated a bit more political disempowerment for you on the margin.

There is a lot of owing that doesn’t arise from formal debt, where these things also apply. If we both agree that I—as your friend—am obliged to help you get to the airport, you may hope that I have energy and fuel and am in a good mood. Whereas I may (regretfully) be relieved when your flight is canceled.

Money is an IOU from society for some stuff later, so having money is another kind of being owed. Perhaps this is part of the common resentment of wealth.

I tentatively take this as reason to avoid debt in all its forms more: it’s not clear that the incentives of alliance in one direction make up for the trouble of the incentives for enmity in the other. And especially so when they are considered together—if you are going to become more aligned with someone, better it be someone who is not simultaneously becoming misaligned with you. Even if such incentives never change your behavior, every person you are obligated to help for an hour on their project is a person for whom you might feel a dash of relief if their project falls apart. And that is not fun to have sitting around in relationships.

(Inpsired by reading The Debtor’s Revolt by Ben Hoffman late...
Distillation of 'Do language models plan for future tokens'
by TheManxLoiner — 2024-06-27T20:57:34.351Z

Link to arxiv preprint: Do language models plan for future tokens, by Wilson Wu, John X Morris and Lionel Levine.

TLDR

There are two reasons why the computations for the current token are helpful for future tokens’ computation. First, which they call ‘pre-caching’, the network is intentionally helping future tokens’ computations by sacrificing performance for the current token. Second, which they call ‘breadcrumbs’, is that the helpfulness is incidental: the computations that are good for the current token are also good for future tokens.
By zeroing appropriate gradients, they create a training strategy that prevents pre-caching. They call this ‘myopic descent’.
On a synthetic dataset designed for pre-caching to be useful, they find strong evidence that myopic descent reduces pre-caching.
On integer multiplication, myopic descent reduces performance.
On a language dataset, they train a GPT-2 transformer using both vanilla training and myopic descent and find that performance is similar, suggesting that breadcrumbs are more prominent.
Using larger models from the Pythia suite, they find the performance gap between vanilla and myopic training increases with model size.

As of 27 June 2024, this is ongoing work. In particular, the integer multiplication and Pythia experiments are not yet described in the current arXiv article. The authors shared a draft containing these latest results.

Also, this is my first distillation post. Any feedback - both what you like and what can be improved - will be much appreciated.

Pre-requisites

For this distillation, I assume basic familiarity with transformer architecture and gradient descent. You do not need any AI safety or mech interp experience.

Pre-caching and breadcrumbs

With the help of the diagram below, I introduce notation. x1,xi,xj (where i<j) represent input tokens, y1,yi,yjrepresent output logits, each box corresponds to one position/token of the input sequence, and the x’s inside the boxes represent the hidden states.

We have a causal mask, so the hidden states for xi are useful for the hidden states for xj but not vice versa. The question this paper asks whether this usefulness is intentional or incidental? They introduce terminology for these two possibilities:

Pre-caching. Some computations done for xi are directly intended to help in future tokens, even if those comput

...
how birds sense magnetic fields
by bhauth — 2024-06-27T18:59:35.075Z
Link: https://www.bhauth.com/blog/biology/bird%20magnets.html

introduction

It is known that many birds are able to sense the direction of Earth's magnetic field. Here's a wikipedia page on that general phenomenon. There have been 2 main theories of how that works.

One theory is that birds have magnets in their beak that act like a compass. We know this is the correct theory because:

Small magnetite crystals have been found in bird beaks.

Anaesthesia of bird beaks seems to affect their magnetic sense, sometimes.

The other theory is that birds have some sensing mechanism in their eyes that uses magneto-optical effects. We know this is the correct theory because:

Birds can't sense magnetic field direction in red light.

Covering the right eye of birds prevents them from sensing field direction.

We also know those theories probably aren't both correct because:

Most animals don't have a magnetic field sense. It's implausible that birds developed two separate and redundant systems for sensing magnetic fields when other animals didn't develop one.

organic magneto-optics

It's possible for magnetic fields to affect the optical properties of molecules; here's an example, a fluorescent protein strongly affected by a small magnet. However, known examples of this require much stronger (~1000x) fields than the Earth's magnetic field.

Let's suppose birds sense magnetic fields using some proteins in their eyes that directly interact with fields. The energy density of a magnetic field is proportional to the field strength^2. The energy of interaction of a magnet with a field is proportional to the product of the field strengths. The earth has a field of 25 to 65 μT. If we consider the energy of a strongly magnetic protein interacting with the Earth's magnetic field, that's not enough energy to directly cause a cellular signalling effect.

So, magnetic fields must act to control some energy-transferring process, and the only logical possibilities are light absorption/emission and transfer of excited states between molecules. Birds can sense the direction of magnetic fields, more so than field strength, so the effect of magnetic fields must be relative to the orientation of something. Molecules are randomly oriented, but absorption/emission of a photon is relative to molecule orientation, so magnetic fields can create differences in absorption/emission of light at different angles. (That's the basis of a spectroscopy technique I previou...
Representation Tuning
by Christopher Ackerman — 2024-06-27T17:44:33.338Z

Summary

First, I identify activation vectors related to honesty in an RLHF’d LLM (Llama-2-13b-chat). Next, I demonstrate that model output can be made more or less honest by adding positive or negative multiples of these vectors to residual stream activations during generation. Then, I show that a similar effect can be achieved by fine-tuning the vectors directly into (or out of) the model, by use of a loss function based on the cosine similarity of residual stream activations to the vectors. Finally, I compare the results to fine-tuning with a token-based loss on honest or dishonest prompts, and to online steering. Overall, fine-tuning the vectors into the models using the cosine similarity loss had the strongest effect on shifting model output in the intended direction, and showed some resistance to subsequent steering, suggesting the potential utility of this approach as a safety measure.

This work was done as the capstone project for BlueDot Impact’s AI Safety Fundamentals - Alignment course, June 2024

Introduction

The concept of activation steering/representation engineering is simple, and it is remarkable that it works. First, one identifies an activation pattern in a model (generally in the residual stream input or output) corresponding to a high-level behavior like "sycophancy" or "honesty" by a simple expedient such as running pairs of inputs with and without the behavior through the model and taking the mean of the differences in the pairs' activations. Then one adds the resulting vector, scaled by +/- various coefficients, to the model's activations as it generates new output, and the model gives output that has more or less of the behavior, as one desires. This would seem quite interesting from the perspective of LLM interpretability, and potentially safety.

Beneath the apparent simplicity of activation steering, there are a lot of details and challenges, from deciding on which behavioral dimension to use, to identifying the best way to elicit representations relevant to it in the model, to determining which layers to target for steering, and more. A number of differing approaches having been reported and many more are possible, and I explored many of them before settling on one to pursue more deeply; see this github repo for a longer discussion of this process and associated code.

In this work I extend the activation steering concept by permanen...
An issue with training schemers with supervised fine-tuning
by Fabien Roger — 2024-06-27T15:37:56.020Z

This is a rough write-up of ideas by Ryan Greenblatt. The core ideas are his and mistakes are mine. Thanks to him, Buck Shlegeris and to Aghyad Deeb for feedback on the draft of this post.

One way to use

scheming AIs

is to train them to imitate humans. When does imitation robustly avoid frequent and intentional failures?

On single-token classification tasks, imitation works great and can allow you to get a lot of high-quality labor with a tiny rate of intentional failures;
On tasks that involve generating many tokens, imitation potentially fails by default: schemers could notice when an output was generated by themselves, which enables them to consistently behave badly in deployment while doing best-effort imitation during training.

We describe two techniques that avoid imitation failures on generation tasks:

A technique that works perfectly but is too expensive
A technique that we think works in practice

[Edit: These techniques are already known in the behavioral cloning literature. The main novelty of this post is explicitly thinking of these ideas through the lens of defenses against scheming. Thanks to tjbai for pointing it out in the comments of this post.]

While pure imitation may not be sufficiently competitive, we think it could prove useful for tasks such as:

(No-CoT) single-token classification;
KL-regularization for RL: we may want to penalize models trained with RL for straying too far from a human imitation prior (and we would want this human prior to not be malign!) (see Appendix);
Tasks for which it is fine to take the competitiveness hit from not using RL, but which weaker

trusted models

are not good enough at.[1]

Assumptions

In this post, we focus on scenarios where:

AIs are smart (around human-level or above);
AIs are scheming and know how we are training and deploying them;
We want to use these AIs productively (e.g. to do safety research)[2];
Training and deployment tasks are indistinguishable (e.g. because we randomly sampled training tasks from the deployment distribution);
SGD works: we ignore

gradient hacking

and AI architectures where SGD is very sample inefficient;
AIs can’t cause catastrophes in very few actions (e.g. because they are too weak to break safety measures that address these sorts of concerns);
Even when AIs try their best, they can’t perfectly predict the probability distribution over the actions that humans[3] would take.

E

...
AI #70: A Beautiful Sonnet
by Zvi — 2024-06-27T14:40:08.087Z

They said it couldn’t be done.

No, not Claude Sonnet 3.5 becoming the clear best model.

No, not the Claude-Sonnet-empowered automatic meme generators. Those were whipped together in five minutes.

They said I would never get quiet time and catch up. Well, I showed them!

That’s right. Yes, there is a new best model, but otherwise it was a quiet week. I got a chance to incorporate the remaining biggest backlog topics. The RAND report is covered under Thirty Eight Ways to Steal Your Model Weights. Last month’s conference in Seoul is covered in You’ve Got Seoul. I got to publish my thoughts on OpenAI’s Model Spec last Friday.

Table of Contents

Be sure to read about Claude 3.5 Sonnet here. That is by far the biggest story.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. I am increasingly persuaded.

Language Models Don’t Offer Mundane Utility. EU’s DMA versus the AiPhone.

Clauding Along. More people, mostly impressed.

Fun With Image Generation. They are coming for our memes. Then Hollywood.

Copyright Confrontation. The RIAA does the most RIAA thing.

Deepfaketown and Botpocalypse Soon. Character.ai addiction. Am I out of touch?

They Took Our Jobs. More arguments that the issues lie in the future.

The Art of the Jailbreak. We need to work together as a team.

Get Involved. AISI, Apollo, Astra, Accra, BlueDot, Cybersecurity and DOE.

Introducing. Forecasting, OpenAI Mac App, Otto, Dot, Butterflies, Decagon.

In Other AI News. OpenAI equity takes steps forward. You can sell it.

Quiet Speculations. A distinct lack of mojo.

You’ve Got Seoul. Delayed coverage of the Seoul summit from last month.

Thirty Eight Ways to Steal Your Model Weights. Right now they would all work.

The Quest for Sane Regulations. Steelmanning restraint.

SB 1047. In Brief.

The Week in Audio. Dwarkesh interviews Tony Blair, and many more.

Rhetorical Innovation. A demolition, and also a disputed correction.

People Are Worried About AI Killing Everyone. Don’t give up. Invest wisely.

Other People Are Not As Worried About AI Killing Everyone. What even is ASI?

The Lighter Side. Eventually the AI will learn.

Language Models Offer Mundane Utility

Training only on (x,y) pairs, define the function f(x), compose and invert it without in-context examples or chain of thought.

AI Dungeon will let you be the DM ...