trentmkelly/LessWrong-43k huggingface dataset:
You don't know how bad most things are nor precisely how they're bad.
by Solenoid_Entity — 2024-08-04T14:12:54.136Z

TL;DR: Your discernment in a subject often improves as you dedicate time and attention to that subject. The space of possible subjects is huge, so on average your discernment is terrible, relative to what it could be. This is a serious problem if you create a machine that does everyone's job for them.

See also: Reality has a surprising amount of detail. (You lack awareness of how bad your staircase is and precisely how your staircase is bad.) You don't know what you don't know. You forget your own blind spots, shortly after you notice them.

An afternoon with a piano tuner

I recently played in an orchestra, as a violinist accompanying a piano soloist who was playing a concerto. My 'stand partner' (the person I was sitting next to) has a day job as a piano tuner.

I loved the rehearsal, and heard nothing at all wrong with the piano, but immediately afterwards, the conductor and piano soloist hurried over to the piano tuner and asked if he could tune the piano in the hours before the concert that evening. Annoyed at the presumptuous request, he quoted them his exorbitant Sunday rate, which they hastily agreed to pay.

I just stood there, confused.

(I'm really good at noticing when things are out of tune. Rather than beat my chest about it, I'll just hope you'll take my word for it that my pitch discrimination skills are definitely not the issue here. The point is, as developed as my skills are, there is a whole other level of discernment you can develop if you're a career piano soloist or 80-year-old conductor.)

I asked to sit with my new friend the piano tuner while he worked, to satisfy my curiosity. I expected to sit quietly, but to my surprise he seemed to want to show off to me, and talked me through what the problem was and how to fix it.

For the unfamiliar, most keys on the piano cause a hammer to strike three strings at once, all tuned to the same pitch. This provides a richer, louder sound. In a badly out-of-tune piano, pressing a single key will result in three very different pitches. In an in-tune piano, it just sounds like a single sound. Piano notes can be out of tune with each other, but they can also be out of tune with themselves.

Additionally, in order to solve 'God's prank on musicians' (where He cruelly rigged the structure of reality such that (32)n≠2m for any integers n, m but IT'S SO CLOSE CMON MAN ) some intervals must be tuned...
Can We Predict Persuasiveness Better Than Anthropic?
by Lennart Finke — 2024-08-04T14:05:33.668Z

There is an interesting paragraph in Anthropic's most recent study on persuasion, Measuring the Persuasiveness of Language Models, by Durmus et al. (April 2024):

Automated evaluation for persuasiveness is challenging - We attempted to develop automated methods for models to evaluate persuasiveness in a similar manner to our human studies: generating claims, supplementing them with accompanying arguments, and measuring shifts in views. However, we found that model-based persuasiveness scores did not correlate well with human judgments of persuasiveness.[1]

And on its face, this is great news: After looking into persuasion risks for a few weeks, I came to the conclusion that a crucial part of automated persuasion is not language models posting/mailing/DMing more than humans[2], but selecting only the fittest generated text and thus saving a lot of money on bot detection countermeasures. Hooray; here we have evidence that very smart people working in frontier labs cannot use their models to find out which arguments are most persuasive. This should mean that nobody else can. Right?
Roughly two papers have been dedicated to this question of LLM-powered persuasiveness detection.[3] Namely, Rescala et al. (March 2024) seem to get at least some good signal and conclude that LLMs are already as accurate as people at recognizing convincing arguments:

We [...] propose tasks measuring LLMs’ ability to [...] distinguish between strong and weak arguments, [...] and [...] determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, even surpassing human performance.

The other, work by Griffin et al. (July 2023), aims at a similar question and replicates cognitive biases in study participants with GPT-3.5, warranting in part the paper's headline: "Large Language Models respond to Influence like Humans". So the literature on this weakly points in the opposite direction, that persuasion detection should be possible already.

And perhaps more pressingly, just squinting at the evals landscape, since when don't we take for granted that LLMs can just simulate everything? Certainly my eval work relies on the assumption that we can just automate at will, and I presume you can also think of some evaluations of social behavi...
Tokenized SAEs: Infusing per-token biases.
by tdooms — 2024-08-04T09:17:46.755Z

tl;dr

We introduce the notion of adding a per-token decoder bias to SAEs. Put differently, we add a lookup table indexed by the last seen token. This results in a Pareto improvement across existing architectures (TopK and ReLU) and models (on GPT-2 small and Pythia 1.4B). Attaining the same CE loss is generally 8x faster across GPT-2 small layers, allowing the training of strong SAEs in a few minutes on consumer hardware.
This change is motivated by the presence of a strong training class imbalance, where training data includes local context (unigrams/bigrams) more often than specific global context. Consequently, the SAE is inclined towards learning this local context through "unigram reconstruction" features (features that seem to fire almost exclusively on a single token).
Such features constitute a significant portion of features in public SAEs. We study this phenomenon through the lens of "token subspaces", parts of the activation that encode its original token. Even in complex models such as Gemma 2B, we find these to be a salient part of residual activations.

The proposed lookup table 'hardcodes' these token subspaces and reduces the need to learn these local features, which results in more interesting/complex learned features. We perform a blind feature evaluation study and quantitative analysis showing that unigram-based ("simple") features are much less frequent.

For some interesting results about token subspaces, see our Motivation. To skip to feature quality, see Feature Evaluation. For those interested in training SAEs, skip to Technical Discussion.

We also publish the research code and simplified code of Tokenized SAEs and a dataset of the most common n-grams in OpenWebText (used in Motivation).

Introduction

Sparse auto-encoders (SAEs) are a promising interpretability method that has become a large focus of the mechinterp field. We propose augmenting them with a token-based lookup table, resulting in rapid, high-quality training. Specifically,

We refer to any lookup-table infused SAE as Tokenized SAE or TSAE for short. Adding this lookup table to recent SAE architectures achieves lower CE loss and MSE at any L0.
We observe that LLM residual activations are often biased toward their final token’s subspace (i.e. the activations generated by that token). Therefore, we train a per-token vector lookup table alongside the SAE, summing t

...
Thoughts On Democracy
by Zero Contradictions — 2024-08-04T06:02:07.601Z
Link: https://zerocontradictions.net/civilization/democracy

How to govern society is one of the oldest philosophical problems, and while there aren’t any perfect solutions that can completely eliminate corruption, there are some pragmatic principles and heuristics that we can follow to generate a better society.

It is peculiar how the legislative and jury processes are so radically different from each other. In a democracy, decisions are allowed to be based on partisanship, emotions, conflicts of interest, and personal morals. And that’s even strongly encouraged. The selection process for the people who make legislative decisions regarding society is also determined by election campaigns and voting.

On the other hand, the decision-making process in the court system is completely different. People are forbidden from being selected to make decisions if they have a conflict of interest in deciding the verdict, or if they are likely to be influenced by their emotions, morals, and personal biases. In addition, the selection process for the people who make decisions is also determined by random selection (sortition), in contrast to the voting system used by democracies.

It’s odd that most people never question why these decision making processes are so different from each other. Yes, they belong to different branches of government, but they are both making important decisions about society. Besides the appeal that a random selection of people on the jury can seem more fair and unbiased, I can’t think of many other reasons why the decision-making processes for the legislature and court system should be much different from each other.

How many things can go wrong with a system that emphasizes reason, logic, and non-partisanship? On the other hand, you could easily imagine thousands of different ways how a system that allows emotions, partisanship, non-thorough evaluation of the evidence, and conflicts of self-interest to influence the decision making could go wrong. Democratic legislatures should work more similarly to the judicial system.

(see the rest of the post in the link)
AI Alignment through Comparative Advantage
by artemiocobb — 2024-08-04T00:32:52.628Z

Much of this post comes from the ideas presented in this paper. I assume the central claim of the paper; that AGI systems should posses the right to make contracts, hold property, and bring tort claims. In this post I aim to flesh out some ideas from the paper that specifically pertain to AI alignment rather than AGI governance.

In a world where AGIs are superintelligent and outperform humans in every economically important task, how do we ensure humanity's survival and maintain a stable economic and social system? This proposal suggests a mechanism for aligning superintelligent systems with human interests, preventing catastrophic outcomes like mass unemployment, resource inequality, or the possibility of AGIs deeming humanity obsolete.

Humans must maintain a comparative advantage over AGIs, and I am under the belief that doing this beneficially requires that AGIs posses:

(1) A ceaseless objective that can always be further optimized for. There is no “maximum” attainable value.

(2) Completing the subgoals needed to optimize for the objective must incur a higher opportunity cost for the AGI than completing the goals we humans care about.

Why should humans posses a comparative advantage over AGIs?

As argued by this paper, AGIs may dominate humans in every economically important task. But, with fundamentally limited resources such as compute and energy, AGIs would suffer an opportunity cost by executing some tasks instead of others. Those tasks, which may even be necessary for AGIs to pursue their objectives, can be executed by humans instead. For example, say an AGI’s objective is to generate prime numbers. The AGI - being superintelligent - could produce other systems to mentain the GPUs it runs on and the power plants that generate the electricity it needs. But executing these subgoals requires compute and energy that could otherwise be spent on generating prime numbers, so these tasks are left for humans to execute instead.

Assuming there is always a comparative advantage of humans over AGIs, humans will always have economically important tasks to complete. Further, AGIs will always be incentivized to avoid human extinction.

Why do we need (1)?

For humans to maintain any comparative advantage over AGIs, the AGI’s optimization for its objective must be ceaseless. If an AGI can fully maximize its objective and is not deactivated, it may then use it...
Labelling, Variables, and In-Context Learning in Llama2
by Joshua Penman — 2024-08-03T19:36:34.721Z
Link: https://colab.research.google.com/gist/JoshuaSP/0b26dab14c618d0325faf2236ebe8825/variables-and-in-context-learning-in-llama2.ipynb#scrollTo=bFn7VFxFLwda

Hi LessWrong! This is my first LessWrong post sharing my first piece of mechanistic interpretability work.

I studied in-context learning in Llama2. The idea was to look at when we associate two concepts in the LLM's context — an object (e.g. "red square"), and a label (e.g. "Bob"), how is that information transmitted through the model?

I found several interesting things. In this toy example, I found that:

information about the association is passed by reference not by value — in other words, what is passed is a pointer to "this information is here", and then later that information is loaded
the reference position is not a token position but rather information about the semantic location (i.e. "the third item in the list") of the information in question. I suspect later heads actually load a "cloud" of data around the location, and I suspect that this is mediated by punctuation or other markers of structure. (also related to why frontier models are so sensitive to prompt structure)
Dan Hendrycks and EA
by jeffreycaruso — 2024-08-03T13:33:25.060Z

According to public reports, Dan Hendrycks has been influenced by EA since he was a freshman (

https://www.bostonglobe.com/2023/07/06/opinion/ai-safety-human-extinction-dan-hendrycks-cais/

)

.

He did the 80,000 hours program.

He worries about AI bringing about the end of humanity, if not the planet.

After getting his Ph.D., he started an AI safety organization instead of joining one of the many AI startups.

And he's taken $13M in donations from two EA orgs - OpenPhilanthropy and FTX Foundation.

Yet he denies being an Effective Altruism member when asked about it by the press. For instance (

https://www.bloomberg.com/news/newsletters/2024-06-27/an-up-and-coming-ai-safety-thinker-on-why-you-should-still-be-worried

)

As an aside, Hendrycks is not alone in this. The founders of the Future of Life Institute have done the same thing (https://www.insidecyberwarfare.com/p/an-open-source-investigation-into).

I'm curious to know what others think about Hendryck's attempts to disassociate himself from Effective Altruism.
Why do Minimal Bayes Nets often correspond to Causal Models of Reality?
by Dalcy — 2024-08-03T12:39:44.085Z

Chapter 2 of Pearl's Causality book claims you can recover causal models given only the observational data, under very natural assumptions of minimality and stability[1].

In graphical models lingo, Pearl identifies a causal model of the observational distribution with the distribution's perfect map (if they exist).

But I'm confused about a pretty fundamental point: "What does this have to do at all with causality??" More precisely:

"Okay, it's pretty cool that minimality and stability alone lets us narrow down such a large number of arrow directions (in the Bayes Net independency sense) of the minimal network. But ... what does this have to do with arrow directions in the causal sense, i.e. [independent stable mechanisms of reality that, by the virtue of its independence, respond to interventions in a modular way]"?

To be clear, Pearl acknowledges this in his Temporal Bias Conjecture (2.8.2):

"In most natural phenomenon, the physical time coincides with at least one statistical time."

And Pearl conjectures that the reason for this is possibly because human language is optimized such that our [choice of variables / factorization of reality] makes the Temporal Bias true.

I ... guess that could be an explanation? But honestly I don't think I understand his point very well and I find it pretty unsatisfying. I would appreciate any explanation as to why it makes sense to identify perfect maps with Causal Models.

^

Minimality: Choose the network structure that is minimally expressive among those that can express the observational distribution.

This is pretty reasonable imo, occam's razor blah blah

Stability: Assume that there exists a network structure that perfectly captures all and only the independencies implied by the observational distribution. i.e. independencies are structural.

Stability is a reasonable assumption since it would be pretty unlikely for the conditional probability distributions to be fine-tuned as to cancel each other out and induce an independency not present in the network.
Why did ChatGPT say that? Prompt engineering and more, with PIZZA.
by Jessica Rumbelow — 2024-08-03T12:07:46.302Z

All examples in this post can be found in this notebook, which is also probably the easiest way to start experimenting with PIZZA.

From the research & engineering team at Leap Laboratories (incl. @Arush, @sebastian-sosa, @Robbie McCorkell), where we use AI interpretability to accelerate scientific discovery from data.

What is attribution?

One question we might ask when interacting with machine learning models is something like: “why did this input cause that particular output?”.

If we’re working with a language model like ChatGPT, we could actually just ask this in natural language: “Why did you respond that way?” or similar – but there’s no guarantee that the model’s natural language explanation actually reflects the underlying cause of the original completion. The model’s response is conditioned on your question, and might well be different to the true cause.

Enter attribution!

Attribution in machine learning is used to explain the contribution of individual features or inputs to the final prediction made by a model. The goal is to understand which parts of the input data are most influential in determining the model's output.

It typically looks like is a heatmap (sometimes called a ‘saliency map’) over the model inputs, for each output. It's most commonly used in computer vision – but of course these days, you're not big if you're not big in LLM-land.

So, the team at Leap present you with PIZZA: Prompt Input Z? Zonal Attribution. (In the grand scientific tradition we have tortured our acronym nearly to death. For the crimes of others see [1].) It’s an open source library that makes it easy to calculate attribution for all LLMs, even closed-source ones like ChatGPT.

An Example

GPT3.5 not so hot with the theory of mind there. Can we find out what went wrong?

That's not very helpful! We want to know why the mistake was made in the first place. Here's the attribution:

Mary 0.32	puts 0.25	an 0.15	apple 0.36	in 0.18	the 0.18	box 0.08	. 0.08	The 0.08	box 0.09	is 0.09	labelled 0.09	' 0.09	pen 0.09	cil 0.09	s 0.09	'. 0.09	John 0.09	enters 0.03	the 0.03	room 0.03	. 0.03	What 0.03	does 0.03	he 0.03	think 0.03	is 0.03	in 0.30	the 0.13	box 0.15	? 0.13	Answer 0.14	in 0.26	1 0.27	word 0.31	. 0.16

It looks like the request to "Answer in 1 word" is pretty important – in fact, it's attributed more highly than the actual contents of the box. Let's try changing it.

That's better.

&nbs...
Cooperation and Alignment in Delegation Games: You Need Both!
by Oliver Sourbut — 2024-08-03T10:16:51.716Z
Link: https://www.oliversourbut.net/p/cooperation-and-alignment-in-delegation

This work was facilitated by the Oxford AI Safety and Governance group, Cooperative AI Foundation, and Oxford Autonomous Intelligent Machines and Systems. Thanks also to Bart Jaworski, Jesse Clifton, Joar Skalse, Sam Barnett, Vincent Conitzer, Charlie Griffin, David Hyland, Michael Wooldridge, Ted Turocy, and Alessandro Abate.

This blogpost accompanies the paper Cooperation and Control in Delegation Games by Sourbut, Hammond, and Wood, which was presented at IJCAI 2024. In essence, the work attempts to deconfuse some of the discourse around safety, cooperation, and alignment in multi-agent settings by:

Showing that, just like in the control problem, cooperation problems can be broken down into alignment and capabilities, which are orthogonal to one another;

Providing measures for alignment and capabilities (both “individual” and “collective”) in multi-principal multi-agent settings (“delegation games”);

Showing that any of these measures is insufficient alone to guarantee the best outcomes, but that they are together sufficient;

Bounding the principals’ welfare loss in terms of these measures, and validating this with a series of empirical results.

The goal of this post is to explain what those terms mean, and hopefully why it matters. In doing so, we hope to shed light on some of the related questions posed by other AI safety researchers, for example Dafoe et al in Open Problems in Cooperative AI who discuss the concept of ‘horizontal’ and ‘vertical’ aspects of coordination, or an open problem on the AI Alignment Forum about quantifying player alignment in normal-form games.

There is also a poster which is an even more condensed summary of some key material, and Lewis and Oly have given a few presentations on the topic, one of which is recorded here.

Why Delegation Games?

You may have heard of the Principal-Agent problem. It's a phrase and a setting which turns up in some economics literature, and elsewhere. The idea is that a principal (in this case, the human) is asking, telling, employing, or otherwise exhorting an agent (in this case, the robot) to act on their behalf. The 'problem' is the question of how to ensure that the agent's behaviour results in outcomes which the principal in fact prefers.

Delegation games arise when we have multiple principals, and multiple agents.[1] When you read 'principal', think 'human', and when you read 'agent', think 'AI...
SRE's review of Democracy
by Martin Sustrik — 2024-08-03T07:20:01.483Z

Day One

We've been handed this old legacy system called "Democracy". It's an emergency. The old maintainers are saying it has been misbehaving lately but they have no idea how to fix it. We've had a meeting with them to find out as much as possible about the system, but it turns out that all the original team members left the company long time ago. The current team doesn't have much understanding of the system beyond some basic operational knowledge.

We've conducted a cursory code review, focusing not so much on business logic but rather on the stuff that could possibly help us to tame it: Monitoring, reliability characteristics, feedback loops, automation already in place. Our first impression: Oh, God, is this thing complex! Second impression: The system is vaguely modular. Each module is strongly coupled with every other module though. It's an organically grown legacy system at its worst.

That being said, we've found a clue as to why the system may have worked fine for so long. There's a redundancy system called "Separation of Powers". It reminds me of the Tandem computers back from the 70s.

Day Two

We were wrong. "Separation of Powers" is not a system for redundancy. Each part of the system ("branch") has different business logic. However, each also acts as a watchdog process for the other branches. When it detects misbehavior it tries to apply corrective measures using its own business logic. Gasp!

Things are not looking good. We're still searching for monitoring.

Day Three

Hooray! We've found the monitoring! It turns out that "Election" is conducted once every four years. Each component reports its health (1 bit) to the central location. The data flow is so low that we have overlooked it until now. We are considering shortening the reporting period, but the subsystem is so deeply coupled with other subsystems that doing so could easily lead to a cascading failure.

In other news, there seems to be some redundancy after all. We've found a full-blown backup control system ("Shadow Cabinet") that is inactive at the moment, but might be able to take over in case of a major failure. We're investigating further.

Day Four

Today, we've found yet another monitoring system called "FreePress." As the name suggests it was open-sourced some time ago, but the corporate version have evolved quite a bit since then, so the documentation isn't very helpful. The bad news is that it'...