trentmkelly/LessWrong-43k huggingface dataset:
The Xerox Parc/ARPA version of the intellectual Turing test: Class 1 vs Class 2 disagreement
by hamishtodd1 — 2024-06-30T15:34:53.729Z

I've been reading the excellent book The Dream Machine about ARPA and PARC, the research communities that invented the personal computer (before them computers were just tools for military research projects).

Xerox PARC was managed by Bob Taylor, a great manager who drew on the management style of many great people before him. PARC is an astonishing example of far-out thinking colliding with cutting-edge technology, and it involved many extremely intelligent free-thinkers (read: difficult-to-"manage" people) coming together. There are many PARC-isms; another one is the fact that they had meetings on beanbags instead of chairs, apparently in order to stop people rising to their feet to denounce others.

Taylor distinguished class 1 and class 2 disagreement. In our terminology, a class-2 disagreement is one in which both sides could pass an Ideological Turing Test(ITT) - that is, if they paused and were asked to state their opponent's opinion, they could do so in a way that their opponent would be happy with (perhaps going so far as to successfully convince someone who didn't know better that they actually held their opponent's opinion, hence the name).

Class 1 is apparently "just both sides yelling at each other". This, I suppose, skips over the fairly likely situation that one party could pass an ITT but the opposing party can't. But perhaps that's not a useful distinction; the one-sided situation is an unstable equilibrium (eg rage-inducing for the side that has taken the time to check their ability to pass the ITT).

So it's a high and rarefied standard to meet for something called "Class 2" (one wonders if they had class 3, and that's why no lab has made as significant breakthroughs as PARC since their glory days).

I think in order to get to it, it can sometimes take a very long time; I have an ongoing debate intelligence with two friends of mine that has lasted days of discussion (spread across a years!), and in spite of huge amounts of patience on their part, they don't think they can pass the ITT for my position. But, I endlessly appreciate them for their desire to pass it; plausibly that's a part of what friendship is. I mean that partly in the cute way, but also partly with a sad implication: that we're being unrealistic to hope people including ourselves will try as hard pass it for our enemies.
LLMs Universally Learn a Feature Representing Token Frequency / Rarity
by Sean Osier — 2024-06-30T02:48:01.839Z
Link: https://github.com/sosier/LLM_Token_Frequency_Feature

Summary

LLMs appear to universally learn a feature in their embeddings representing the frequency / rarity of the tokens they were trained on
This feature is observed across model sizes, in base models, instruction tuned models, regular text models, and code models
In models without tied weights, this feature is present in both models' embedding and unembedding matrices
This feature can be used to find "glitch tokens" (e.g.

SolidGoldMagikarp

)
In limited experiments using this feature for activation steering, the model pays relatively little attention to this feature. As such, it's likely not a strong target for activation steering efforts

Definitions / Notation

I use

Olah et al.'s

definition of a feature as a direction in the vector space of a model's weights / activations in a given layer.

Tied weights

are a common LLM architectural choice where in order to reduce the number of parameters in the model and align the embeddings and unembeddings, the same weights matrix is used for both. Of the models discussed here the GPT 2 series, Gemma series, and Phi 3 Small models use tied weights. All others have separate weight matrices for their embeddings and unembeddings. For brevity, in models with tied weights, I will refer to both the embeddings and unembeddings as simply the model's "embeddings" since the two are identical.

Linear probing is just a fancy name for using linear regression (or classification) on the internal representations (weights or activations) of models.

How to Find the Feature

Ideally, get access to the model's training data

When this is not possible (i.e. for basically all well known open source LLMs), I use the

OpenWebText dataset

as a rough proxy

Load the model's tokenizer
Tokenize the dataset and count the frequency of each token in the tokenizer's vocabulary. The end result should be a vector of length V (the size of the vocabulary) with the counts ordered by token ID
(Optional) Because steps 1-3 can be slow, save the resulting counts somewhere for future use

Saved counts for all the models I studied as well as code to do OpenWebText token counting can be found

here

Load the embedding to be analyzed. This should be a matrix of size V×D where D is the dimensionality of the embedding

Note: It may be necessary to transpose the matrix if it's an unembedding and is of shape D×V

Calculate all D

principal components

of

...
My 5-step program for losing weight
by nsokolsky — 2024-06-30T01:05:40.408Z
Link: https://nsokolsky.substack.com/p/my-5-step-program-for-losing-weight

I can’t claim to be an expert on the subject of weight loss but wanted to share the steps I’ve followed to get from ~81 kg (178 pounds) to ~70 kg (154 pounds) over a period of 8 months. Several people have asked me for tips, so I’ve figured I might as well write it all down for future reference.

Weight Gurus stats from Oct 2022 to Oct 2024

Axiom: CICO (Calories In - Calories Out = Weight loss/gain) works. If you’re a ~healthy individual and need to lose weight, it makes sense to just accept CICO as a given and use it as a mental shorthand. Of course there’s always exceptions: my own dad would eat at least 4,000 calories daily and never gained any weight. But if you’re reading this text, you’re probably not an exception.

Step 0: accept that you lack willpower

I’m not a fan of Nike’s Just Do It™ because it overlooks the fact that people’s willpower is finite and if you aren’t “just doing X” already, you probably won’t do it tomorrow without a change in circumstances. Hence the diet steps are all very gradual and try to get you to a plateau of discomfort before requiring the next sacrifice. Feel free to slow the progression down even further if that works better for you.

Step 1: start measuring your weight and food intake (30 days)

Buy these two products:

A Wifi-enabled scale. I recommend this model but any option with phone synchronization works fine. The linked scales have an app which can export its data into Apple Health or Google Fit.
Paid subscription for the Macrofactor app, which is currently the best app for tracking your calories. The app supports importing your weight data from Apple Health or Google Fit, which it can then use to track your daily energy expenditure and calculate your “true” weight by averaging out your weight measurements over a 7-day period.

Every single morning you’ll get up, remove heavy items of clothing, step on the scale and ensure the weight is recorded. I specifically recommend getting a Wifi-enabled scale because it removes the extra friction of having to pull out your phone every morning to write down your weight. Then during the day you’ll want to track everything you eat or drink with as much precision as reasonably possible.

MacroFactor screenshot

Do not try to change your diet just yet! The goal is to build up an intuition for how many calories each food has, as well as how these calories affect your weight. I’ve been eating aroun...
Datasets that change the odds you exist
by dynomight — 2024-06-29T18:45:14.385Z
Link: https://dynomight.net/datasets/

1.

It’s October 1962. The Cuban missile crisis just happened, thankfully without apocalyptic nuclear war. But still:

Apocalyptic nuclear war easily could have happened.
Crises as serious as the Cuban missile crisis clearly aren’t that rare, since one just happened.

You estimate (like President Kennedy) that there was a 25% chance the Cuban missile crisis could have escalated to nuclear war. And you estimate that there’s a 4% chance of an equally severe crisis happening each year (around 4 per century).

Put together, these numbers suggest there’s a 1% chance that each year might bring nuclear war. Small but terrifying.

But then 62 years tick by without nuclear war. If a button has a 1% chance of activating and you press it 62 times, the odds are almost 50/50 that it would activate. So should you revise your estimate to something lower than 1%?

2.

There are two schools of thought. The first school reasons as follows:

Call the yearly chance of nuclear war W.
This W is a “hidden variable”. You can’t observe it but you can make a guess.
But the higher W is, the less likely that you’d survive 62 years without nuclear war.
So after 62 years, higher values of W are less plausible than they were before, and lower values more plausible. So you should lower your best estimate of W.

Meanwhile, the second school reasons like this:

Wait, wait, wait—hold on.
If there had been nuclear war, you wouldn’t be here to calculate these probabilities.
It can’t be right to use data when the data can only ever pull you in one direction.
So you should ignore the data. Or at least give it much less weight.

Who’s right?

3.

Here’s another scenario:

Say there’s a universe. In this universe, there are lots of planets. On each planet there’s some probability that life will evolve and become conscious and notice that it exists. You’re not sure what that probability is, but your best guess is that it’s really small.

But hey, wait a second, you’re a life-form on a planet with conscious life! Given that you exist, should you increase your guess for how likely conscious life is to evolve on a random planet?

Again, you have two schools of thought. One says yes, you have data, increase your guess, while the other says no, don’t increase, if there wasn’t life you wouldn’t be here, anthropic principle—anthropic principle!

4.

After many years of being confused by these questions, I think I now understand what’s happening. Th...
A “Scaling Monosemanticity” Explainer
by latterframe — 2024-06-29T17:50:49.855Z

Coauthored by Fedor Ryzhenkov and Dmitrii Volkov (Palisade Research)

At Palisade, we often discuss latest safety results with policymakers and think tanks who seek to understand the state of current technology. This document condenses and streamlines the various internal notes we wrote when discussing Anthropic's "Scaling Monosemanticity".

Executive Summary

Research on AI interpretability aims to unveil the inner workings of AI models, traditionally seen as “black boxes.” This enhances our understanding, enabling us to make AI safer, more predictable, and more efficient. Anthropic’s Transformer Circuits Thread focuses on mechanistic (bottom-up) interpretability of AI models.

Their latest result, Scaling Monosemanticity, demonstrates how interpretability techniques that worked for small, shallow models can scale to practical 7B (GPT-3.5-class) models. This paper also paves the way for applying similar methods to larger frontier models (GPT-4 and beyond).

Key Findings of Scaling Monosemanticity

Anthropic has demonstrated how to extract high-level features from AI models. They identified parts of the model’s inner structure that correlate with language properties such as verb tense, gender, helpfulness, lies, and specific subjects like political figures, countries, or bioweapons. These features are then mapped, allowing researchers to review and analyze them for a deeper understanding of the model.

Map of the features related to ‘biological weapons’ in Claude 8B model ("AI brain scan").
See

here

for an interactive map.

Identified features can then be adjusted to control the model’s behavior. For instance, models can be modified to avoid sensitive topics, be children-appropriate, give biased opinions, or output subtly incorrect code while concealing the errors. This is done by artificially increasing or decreasing the effect a feature has on the final output (“steering” output).

Left-hand side: behavior of the default model. Right-hand side: “brain sciences” feature amplified to 10x.

Implications

The results from Anthropic’s paper are early but promising. If this thread of research continues to be successful, it could make tuning AI to specific tasks more accessible and cheap. In the short term, this could disproportionately increase unregulated open-weight model capabilities. In the longer term, this might enable frontier AI labs to build saf...
Analysis of key AI analogies
by Kevin Kohler — 2024-06-29T10:55:21.925Z

The following is an analysis of seven prominent AI analogies: aliens, the brain, climate change, electricity, the Industrial Revolution, the neocortex, & nuclear fission. You can find longer versions of these as separate blogposts on my substack.

0. Why?

AI analogies have a real-world impact

For better or worse, analogies play a prominent role in the public debate about the long-term trajectory and impacts of AI.

Analogies play a role in designing international institutions for AI (e.g. CERN, IPCC) and in legal decisions
Analogies as mental heuristics can influence policymakers in critical decisions. Changes in AI analogies can lead to worldview shifts (e.g. Hinton)
Having worked with a diverse set of experts my sense is that their thinking is anchored by wildly different analogies

Analogies can be misleading

Boaz Barak (“Metaphors for AI, and why I don’t like them”) and Matthew Barnett (“Against most, but not all, AI risk analogies”) have already discussed the shortcomings of analogies on this forum

Every individual analogy is imperfect. AI is its own thing, and there is simply no precedent that would closely match the characteristics of AI across 50+ governance-relevant dimensions.

Overly relying on a single analogy without considering differences and other analogies can lead to blind spots, overconfidence, and overfitting reality to a preconceived pattern.

Analogies can be useful

When facing a complex, open-ended challenge, we do not start with a system model. It is not clear which domain logic, questions, scenarios, risks, or opportunities we should pay attention to. Analogies can be a tool to explore such a future with deep uncertainty.

Analogies can be an instrumental tool in advocacy to communicate complex concepts in a digestible and intuitively appealing way.

My analysis is written in the spirit of exploration without prescribing or proscribing any specific analogy. At the same time, as a repository, it may still be of interest to policy advocates.

1. Aliens

(full text)

Basic idea

comparison to first contact with an alien civilization
symbolizing AI's underlying non-human reasoning processes, masked by human-like responses from RLHF

Selected users

Yuval Noah Harari (2023, 2023, 2023, 2023, 2023, 2023, 2024)
Ray Kurzweil (disanalogy - 1997, 1999, 2005, 2006, 2007, 2009, 2012, 2013, 2017, 2018, 2023)

Selected c

...
Georgism Crash Course
by Zero Contradictions — 2024-06-29T06:18:39.073Z
Link: https://zerocontradictions.net/civilization/georgism-crash-course

For my first post on Less Wrong, I was advised to link to an existing blog post of mine. My Georgism Crash Course has been one of my most viewed essays. It aims to explain what Georgism is, why it would boost economic efficiency, and it addresses some of the possible drawbacks of it. All comments, constructive criticism, and feedback are appreciated.

Georgism1 is the position that income tax, sales tax, property tax, and all other taxes should be abolished and replaced with taxes on natural resources (NRT) and land values, which would fund all government services. The idea is that society would greatly benefit from overhauling the taxation system. (see the rest of the post in the link)
Activation Pattern SVD: A proposal for SAE Interpretability
by Daniel Tan — 2024-06-28T22:12:48.789Z

Epistemic status: This is a rough-draft write-up about a thought experiment I did. Reasonably confident about the broad arguments being made here. That said, I haven't spent a lot of time rigorously polishing or reviewing my writing, so minor inaccuracies may be present.

Interpretability Illusions from Max-Activating Examples

When interpreting an SAE feature, one common technique is to look at the max-activating examples, and try to extrapolate a pattern from there. However, this approach has two flaws, namely:

Premise 1: It can be hard to extrapolate the correct pattern. Correctly identifying a pattern relies on accurate understanding of the semantics present in text. Many hypotheses may be likely, given the data, and the actual truth could be non-obvious. It seems easy to make a mistake when doing this.

Premise 2: Max activating examples may be anomalous. The most likely element of a distribution can look very different from the typical set. Conclusions drawn based on one (or a few) highly activating examples may turn out to be incorrect when evaluated against the majority of examples.

In the following discussion, I outline a proposal to interpret SAE features using the singular value decomposition of SAE activation patterns, which I think neatly addresses both of these issues.

Activation Pattern SVD

Suppose we have a set of M SAE features, which we would like to interpret using a dataset of N unique context windows. To do this, we compute the activation A∈RN×M, where Aij describes the activation of feature j on (the last token of) context window i. We then compute the singular value decomposition A=UΣV and take the top k elements. (Assume that we have a good way of choosing k, e.g. by looking for an "elbow point" in a reconstruction loss curve)

Remark 3: The SVD defines activation prototypes. note that U∈RN×k; each column (in RN) describes a "prototypical" activation pattern of a general SAE feature over all context windows in the dataset.

Remark 4: The activation patterns are linear combinations of prototypes. Define the coefficient matrix C=ΣM∈Rk×M. Each column (in Rk) contains coefficients which are used to reconstruct the activation pattern of a given SAE feature as a linear combination of prototypes.

Conjec...
Podcast: Elizabeth & Austin on "What Manifold was allowed to do"
by Austin Chen — 2024-06-28T22:10:41.607Z
Link: https://share.descript.com/view/2jbEVe05hPt

Elizabeth Van Nostrand and Austin Chen discuss unconventional approaches to running startups, nonprofits, and events. We go over:

How publicly sharing Manifold’s salaries and company information helped with recruitment and user engagement
The origin and evolution of Manifund, an independent charity
Balancing creative, "weird" projects with core business goals
Organizing great conferences as a marketing strategy
Finding the right co-founders and bootstrapping vs VC funding

You can also listen to the episode here: https://share.descript.com/view/2jbEVe05hPt

Links

Milan Cvitkovic on Things you’re allowed to do
Patrick Mackenzie on Salary negotiation
Scott Alexander on Should you reverse any advice you hear?
Manifold - prediction markets
Manifund - grants platform
Manifest - conference
Levels.fyi
ACX Grants

Transcript

Generated via Descript and lightly edited.

Austin: [00:00:00] Cool. I think we're live.

Elizabeth: Okay, great.

Context. I am compiling a list of things startups are allowed to do. , which is going for the things that are options people don't know about, basically. And you were one of the first people I asked because you had done a whole bunch of things that Anyone could have done and no one was stopping anyone from doing, but I hadn't seen other people think about. And previously you gave me a list of these and I had a bunch of followup questions. So we decided to talk in person synchronously.

[00:00:33] Publicly sharing salaries

Elizabeth: Okay. So the first, one of the things you gave me was you publicly share salaries and I've worked for a company that internally shared all salaries and all notes, which I thought was mostly useful, but was imposing costs just doing it within the company.

How does that work when it's completely public?

Austin: Yeah. And I'm curious for background, how big was the company that you were at? Which shared notes sorry, salaries publicly.

Elizabeth: It was under 20 people when I joined and was over 50 when I left.

Austin: I see. Interesting.

Elizabeth: And you would definitely seeing the systems we use change as the company got bigger.

Austin: Yeah, I myself would be quite interested in seeing how well this salary transparency thing scales. So we started it when we were, very small. I think we've had it since we were, three or four people at Manifold when just the co founders plus maybe one or two early employees.

And we just have stuck with it. ...
The Incredible Fentanyl-Detecting Machine
by sarahconstantin — 2024-06-28T22:10:01.223Z

An NII machine in Nogales, AZ. (Image source)

There’s bound to be a lot of discussion of the Biden-Trump presidential debates last night, but I want to skip all the political prognostication and talk about the real issue: fentanyl-detecting machines.

Joe Biden says:

And I wanted to make sure we use the machinery that can detect fentanyl, these big machines that roll over everything that comes across the border, and it costs a lot of money. That was part of this deal we put together, this bipartisan deal.

More fentanyl machines, were able to detect drugs, more numbers of agents, more numbers of all the people at the border. And when we had that deal done, he went – he called his Republican colleagues said don’t do it. It’s going to hurt me politically.

He never argued. It’s not a good bill. It’s a really good bill. We need those machines. We need those machines. And we’re coming down very hard in every country in Asia in terms of precursors for fentanyl. And Mexico is working with us to make sure they don’t have the technology to be able to put it together. That’s what we have to do. We need those machines.

Wait, what machines?

You can remotely, non-destructively detect that a bag of powder contains fentanyl rather than some other, legal substance? And you can sense it through the body of a car?

My god. The LEO community must be holding out on us. If that tech existed, we’d have tricorders by now.

What’s actually going on here?

What’s Up With Fentanyl-Detecting Machines?

First of all, Biden didn’t make them up.

This year, the Department of Homeland Security reports that Customs and Border Patrol (CBP) has deployed “Non-Intrusive Inspection” at the US’s southwest border:

“By installing 123 new large-scale scanners at multiple POEs along the southwest border, CBP will increase its inspection capacity of passenger vehicles from two percent to 40 percent, and of cargo vehicles from 17 percent to 70 percent.”

In fact, there’s something of a scandal about how many of these scanners have been sitting in warehouses but not actually deployed. CBP Commissioner Troy Miller complained to NBC News that the scanners are sitting idle because Congress hasn’t allocated the budget for installing them.

These are, indeed, big drive-through machines. They X-ray cars, allowing most traffic to keep flowing without interruption.

Could an X-ray machine really detect fentanyl inside a car?

To answe...