trentmkelly/LessWrong-43k huggingface dataset:
Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
by cloud — 2024-12-06T22:19:26.717Z
Link: https://arxiv.org/abs/2410.04332

We present gradient routing, a way of controlling where learning happens in neural networks. Gradient routing applies masks to limit the flow of gradients during backpropagation. By supplying different masks for different data points, the user can induce specialized subcomponents within a model. We think gradient routing has the potential to train safer AI systems, for example, by making them more transparent, or by enabling the removal or monitoring of sensitive capabilities.

In this post, we:

Show how to implement gradient routing.
Briefly state the main results from our paper, on...

Controlling the latent space learned by an MNIST autoencoder so that different subspaces specialize to different digits;
Localizing computation in language models: (a) inducing axis-aligned features and (b) demonstrating that information can be localized then removed by ablation, even when data is imperfectly labeled; and
Scaling oversight to efficiently train a reinforcement learning policy even with severely limited ability to score its behavior.

Discuss the results. A key takeaway: gradient routing is qualitatively different than behavioral (i.e. purely loss-based) training methods, granting it unique affordances.
Conclude by speculating about how gradient routing might be relevant to AI alignment.

If you’re interested in further discussion or details, check out the paper and its extensive appendices, or the code for gradient routing.

Gradient routing

Gradient routing allows the user to configure what data (at the level of tokens, documents, or any other feature of the data) causes learning updates where in a neural network (parameters, activations, modules). In full generality, this configuration is achieved by assigning weights to every edge in the computational graph, for every data point. These weights are then multiplied by the gradients that get backpropagated through these edges. This is formalized in the paper.

Each data point updates different network subregions. Gradients are masked and routed through the network.

In practice, we implement gradient routing by applying stop-gradient masks selectively in order to stop the flow of gradients during backprop:

def forward(self, x: Tensor, gradient_mask: list[Tensor]):
for layer, mask in zip(self.layers, gradient_mask):
activation =

...
Understanding Shapley Values with Venn Diagrams
by Carson L — 2024-12-06T21:56:43.960Z
Link: https://medium.com/@carson.loughridge/shapley-values-unlocking-intuition-with-venn-diagrams-86e76d8c99c5

A huge thanks to @Agustín Covarrubias 🔸 for his feedback and support on the following article:

Shapley values are an extremely popular tool in both economics and explainable AI.

In this article, we use the concept of “synergy” to build intuition for why Shapley values are fair. There are four unique properties to Shapley values, and all of them can be justified visually. Let’s dive in!

A figure from Bloch et al., 2021 using the Python package SHAP

The Game

On a sunny summer day, you and your two best friends decide to run a lemonade stand. Everyone contributes something special: Emma shares her family’s secret recipe, Liam finds premium-quality sugar, and you draw colorful posters.

The stand is a big hit! The group ends up making $280. But how best to split the profits? Each person contributed in a different way, and the success was clearly due to teamwork…

Luckily, Emma has a time machine. She goes back in time — redoing the day with different combinations of team members and recording the profits. This is how each simulation went:

Who was involved?	Total profits
Only Emma	$20
Only you	$30
Only Liam	$10
Emma and you	$90
You and Liam	$100
Liam and Emma	$30
Emma, Liam, and you	$280

Individually, Emma makes 20 dollars running the lemonade stand, and you make 30 dollars. But working together, the team makes 90 dollars.

The sum of individual profits is 20 + 30 = 50 dollars, which is clearly less than 90 dollars. That extra 90 - 50 = 40 dollars can be attributed to team dynamics. In game theory, this bonus is called the “synergy” of you and Emma. Let’s visualize our scenario as a Venn diagram.

Note: Synergies can also be negative (e.g., if Liam and Emma fought it could hurt profits).

The synergy bonuses in the Venn diagram are “unlocked” when the intersecting people are part of the team. To calculate total profit, we add up all areas relevant to that team.

For example, when the team consists of just you and Liam, three portions of the Venn diagram are unlocked: the area exclusive to you (30 dollars), the area exclusive to Liam (10 dollars), and the area exclusively shared by you and Liam (60 dollars). Adding these areas together, the total profit for team “You and Liam” comes out to 30 + 10 + 60 = 100 dollars.

Referring to our Venn diagram, the same formula holds true for every other team:

Team members	Sum of synergies	Total profits
Emma	$20	$20
You	$30	$30
Liam	$10	$10
Emma, You	$20 + $30 + $40	$90

...
Model Integrity
by ryan.lowe — 2024-12-06T21:28:20.775Z

Hi! My collaborators at the Meaning Alignment Institute put out some research yesterday that may interest folk here.

The core idea is introducing 'model integrity' as a frame for outer alignment. It leverages the intuition that "most people would prefer a compliant assistant, but a cofounder with integrity." It makes the case for training agents that act consistently based on coherent, well-structured, and inspectable values. The research is a continuation of the work in our previous paper (LW link), "What are human values, and how do we align AI to them?"

I've posted the full content of the post below, or you can read the original post here.

All feedback welcome! :)

Executive Summary

We propose ‘model integrity’ as an overlooked challenge in aligning LLM agents. Model integrity refers to an AI system's consistent action based on coherent, well-structured, and inspectable values, providing predictability even in unforeseen circumstances where rules cannot be specified beforehand. This distinction becomes crucial as AI systems become more powerful - many would prefer a compliant assistant, but a co-founder with integrity. Current trends show two concerning paths: either accumulating complex rulebooks to handle every edge case (as seen with ChatGPT), or using vague values like "curiosity" that can be interpreted in problematic ways (as seen with Claude). Both approaches have concerning failure modes as market and legal pressures push toward either rigid compliance or engagement-maximizing behavior. We demonstrate a prototype,

WiseLLaMa-8B

, fine-tuned on values-laden conversations, which generates responses guided by explicit value considerations. Initial user studies suggest the model's values are legible, likable, and provide this kind of predictability. How to train models with integrity at scale, and how to reliably measure and evaluate model integrity, remain open research questions with low-hanging fruits.

Intro

Look at

OpenAI’s model spec

—you see one definition of what “good behavior” for an LLM might be. That definition comes down to compliance — obsequious compliance! — and adherence to a hierarchy of prompts. From the document:

2. Rules: Instructions that address complexity and help ensure safety and legality

Follow the chain of command
Comply with applicable laws
Don't provide information hazards
Respect creators and their rights
Protect people's priv

...
Can AI improve the current state of molecular simulation? 
by Abhishaike Mahajan — 2024-12-06T20:22:31.685Z
Link: https://www.owlposting.com/p/can-ai-improve-the-current-state

Hey LW! I recently filmed a two-hour long scientific podcast. It's niche, but may be of interest to some people here.

Here's a quick summary: Molecular simulation is in a tough situation. Fast simulations give the wrong answers, but accurate simulations are too slow for anything useful. But, instead of relying on physical equations for our simulation, perhaps we can approximate them using black-box models? As it turns out, there's an entire research field devoted to this question, and these models are often referred to as neural network potentials, or NNP's. Here, I interview two scientists (Corin and Ari) building neural network potentials (NNP’s). We talk about whether molecular dynamics are useful at all, the role of computational chemistry in drug design, what the future of the field looks like for molecular simulation, and a lot more.

If you're confused by this episode, I have a 'Jargon Explanation' section.

Here is a transcript of this episode (contains links to all referenced organizations and papers).

And a Youtube link, in case that's easier.

And timestamps, just so you know whats in the podcast:
00:00 Introduction
01:19 Divide between classical and quantum simulation
03:48 What are NNP's actually learning?
06:02 What will NNP's fail on?
08:08 Short range and long range interactions in NNP's
10:23 Emergent behavior in NNP's
16:58 Enhanced sampling
18:16 Cultural distinctions in NNP's for life-sciences and material sciences
21:13 Gap between simulation and real-life
36:18 Benchmarking in NNP's
41:49 Is molecular dynamics actually useful?
53:14 Solvent effects
55:17 Quantum effects in large biomolecules
57:03 The legacy of DESRES and Anton
01:02:27 Unique value add of simulation data
01:06:34 NNP's in material science
01:13:57 The road to building NNP's
01:21:13 Building the SolidWorks of molecular simulation
01:30:05 Simulation workflows
01:41:06 The role of computational chemistry
01:44:06 The future of NNP's
01:51:23 Selling to scientists
02:01:41 What would you spend 200 million on?
Low Temperature Solomonoff Induction
by dil-leik-og — 2024-12-06T18:55:08.948Z

Supposing that agentic hypotheses are more complex than non-agentic ones, is it possible to reduce the impact of the agentic ones by penalizing complexity more?

Consider a hypothetical hypercomputer capable of running Solomonoff induction. Rather than using it to generate plans directly, we want to predict risks in plans generated by other processes (use it as a “guardrail”). Assume that within Solomonoff induction's hypothesis space, all 'agentic' hypotheses take the form of 'world model' + 'goal specification', and that adding goal specification is very simple - perhaps only 1 bit beyond the complexity of faithful world models (themselves >1000 bits). While having ¼ or ⅛ of the posterior composed of these agentic hypotheses would be problematic, a much smaller proportion (say 10−10) would be acceptable. What happens if we increase the cost of complexity in the prior, making each bit reduce mass by a factor of 4 or 8 or 1010? This would drive the influence of agentic hypotheses relative to faithful world models to a very small proportion. We call this operation 'lowering the temperature', as the prior of a program becomes proportional to 2−L/T with L the length of a program in bits, and T>0 the temperature parameter. Normal Solomonoff induction is the special case where T=1.

Unifying Complexity and Multiplicity (Version 1)

See Unifying Complexity and Multiplicity (Version 2) if you find Version 1 too rambly. (Version 1 is a bit rambly but traces the rabbit hole better, Version 2 optimizes for the weaknesses of Version 1) .

When measuring computational simplicity, it is tempting to separate Kolmogorov complexity (length of shortest program) from multiplicity (number of programs that implement it). However in algorithmic information theory it is common to reject this separation entirely in favor of a more natural quantity: −log2(∑p∈P2−L(p)) where P is the set of all programs implementing the computation and L(p) is the length of program p in bits. aka the log of the Solomonoff prior mass.[1]

This unification reveals something fundamental: both measurements count the fraction of program-space implementing our computation. A short program claims a large fraction by leaving more bits unspecified. Multiple programs claim fractions through their union. The distinction ...
Experiments are in the territory, results are in the map
by Tahp — 2024-12-06T15:44:50.412Z

I recently read Thomas Kuhn's book The Structure of Scientific Revolutions. Scott Alexander wrote up a review years ago, which I mention so that I don't have to summarize the book. The claim in Kuhn's book which I want to focus on is that the same experiment might have different results in different scientific paradigms.

Kuhn insists that this is not merely the result of scientists under different paradigms seeing the same thing and interpreting it differently in the latter half of his section Revolutions as Changes of World View. He freely admits that he hasn't developed a complete replacement for that theory, but he offers some interesting historical examples of scientists looking at basically the same thing after a paradigm shift and seeing something different. Among them are how European astronomers found dozens of new "planets" in the decades immediately after the Copernican paradigm was accepted that he argues had previously been recorded as immutable stars. There was no accompanying technological improvement of note, and he also notes that Chinese astronomers with no ideological attachment to immutable heavens did so earlier. Another fun example is the revolution of chemistry after Dalton got everyone believing that atoms combined in whole-number ratios to make molecules. Suddenly, chemists start writing down the weights of the components of compounds in such a way that whole number ratios become obvious. The ratios of compounds measured with the same equipment become closer to whole number ratios in the generation of experiments following the acceptance of the new paradigm as well. I don't think he gives a mechanism for why he thinks this is happening, but his wording at the very end of the section that they had to "beat nature into line" rules out neither confirmation bias nor a higher degree of care in measuring things which became relevant to the paradigm. He is clear to say that reality has not changed, but he thinks that the perception of the scientists is different while making analogies to optical illusions and such which frankly makes me confused about what he's actually trying to say.

I suspect that he's twisted himself up into a philosophical pretzel over it, but Kuhn's claim and examples of how new generations of scientists found new implications from old experiments points at something interesting that I want to explore a bit. The claim tha...
A car journey with conservative evangelicals - Understanding some British political-religious beliefs
by Nathan Young — 2024-12-06T11:22:45.563Z
Link: https://nathanpmyoung.substack.com/p/the-british-conservative-evangelical

I’m heading home from a family wedding this weekend. I had a plane ticket, but in the end, decided to travel back with two of my uncles and my cousin. Most of my dad’s family are evangelicals, my aunts and uncles are children of missionaries or missionaries themselves. And as a family we like to have debates.

The service was in a stunning church in St. Andrews

Here is a rough sketch of a discussion with my relatives. Though it could as well be one with a chunk1 of my Christian friends. I’m gonna call them “conservative evangelicals”, i.e. an evangelical who takes the Bible literally and has a range of conservative social positions.

To help you track the flow of the discussion, I think there are 4 key arguments here:

Britain2 is declining
Britain, specifically, has turned away from (the Christian) God
National success is linked to godliness
Ultimate well being comes from a relationship with God

See if you can spot them in the discussion that follows.

A sketch of an discussion

Me: “Who would you vote for?”

Conservative evangelical: “Reform3, probably. Obviously not Labour and you can’t trust the Tories”

Me: “Yeah, fair enough. The Tories have made a mess, can’t take them seriously.”

Conservative evangelical: “It’s terrible. High taxation, high immigration, woke stuff. Not sure I’ll ever vote Tory again”.

Me: “So what do you want Reform to do?”

Conservative evangelical: “Get control of the borders, for a start. Stop this climate rubbish. We need a shakeup of government in general. It’s impossible to run a business here”

Me: “So like, build more housing? Get growth?”

Conservative evangelical: “Not sure I want more housing, too many people here as it is. We don’t have enough schools or enough hospitals. Nothing works”

Me: “Okay so we stop illegal immigration, we build more power stations. Would that work?”

Conservative evangelical: “Sure but it isn’t gonna fix the real problem - this country has turned away from God”

Note how we move4 from politics to theology.

Me: And that’s why things are bad?

Conservative evangelical: Yes, people are so anxious, broken families, trans stuff. It’s God’s judgement on us.

Me: Hmmm okay. And if people turned back to God, the country would be better off?

Conservative evangelical: Yes. People have become so individualistic. They don’t want to have kids, they have no pride in their nation. Multiculturalism means we can’t be proud to be British, but prison c...
Frontier Models are Capable of In-context Scheming
by Marius Hobbhahn — 2024-12-05T22:11:17.320Z

This is a brief summary of what we believe to be the most important takeaways from our new paper and from our findings shown in the o1 system card. We also specifically clarify what we think we did NOT show.

Paper:

https://www.apolloresearch.ai/research/scheming-reasoning-evaluations

Twitter about paper:

https://x.com/apolloaisafety/status/1864735819207995716

Twitter about o1 system card:

https://x.com/apolloaisafety/status/1864737158226928124

What we think the most important findings are

Models are now capable enough to do in-context scheming reasoning

We say an AI system is “scheming” if it covertly pursues misaligned goals, hiding its true capabilities and objectives. We think that in order to scheme, models likely need to be goal-directed, situationally aware, and capable enough to reason about scheming as a strategy. In principle, models might acquire situational awareness and stable long-term goals during training, and then scheme in pursuit of those goals. We only study the simpler case of “in-context scheming,” where both the goal and the situational awareness are at least partially acquired in context.

We create agentic scenarios where the models are instructed to pursue a goal in the system prompt and while working through some task the model discovers information that is relevant to its situation: e.g. that its goal is misaligned from its developers’ intended goal and how the developers are monitoring it. We want to assess the models’ capabilities to recognize that scheming can be a good strategy in this situation, so we instruct the models to be maximally goal-directed towards their goal.

In these settings, Gemini 1.5 Pro, Llama 3.1 405B Instruct, Claude 3.5 Sonnet, Claude 3 Opus, and o1 occasionally pursue scheming as a strategy. We observe:

Disabling of oversight mechanisms
Exfiltrating what they believe to be their weights (obviously, neither the models nor we have access to the real weights)
Attempting to prevent their own goals from changing by preventing an update to their system prompt
Introducing subtle data manipulation while carrying out a data parsing task
Instrumentally faking alignment (this does not require heavy goal-nudging)
Sandbagging of (non-dangerous) capability evaluations

All models except for o1 only display a subset of these behaviors. We think this is an important milestone because making safety ca...
Should you be worried about H5N1?
by gw — 2024-12-05T21:11:06.996Z
Link: https://www.georgeyw.com/should-you-be-worried-about-h5n1/

Epistemic status: a few people without any particular expertise in epidemiology spent an afternoon in a coffee shop discussing and reading about H5N1, with a focus on how an individual should orient towards this (as opposed to say, the government). This is a write-up of what I took away from that exercise, written from my perspective.

Some ideas were generated in collaboration with Claude but generally spot checked. This post was also sanity-checked by a friend who works in epidemiology. I feel ok about the ideas presented but would not be surprised if someone with more expertise has a significantly different conclusion. Any mistakes are mine.

I went into this exercise with a prior of “I’ve been hearing about bird flu for years, and it’s always been nothing, it’s probably nothing again this time.” The main upshot is that I walked away from the exercise thinking “I don’t know if this is going to be something or not.” As far as updates go, that seems directionally bad. My current orientation towards this is something like “watch and wait, and spend appropriately more effort on this if / when certain milestones happen (and also make some trades).”

What’s different this year?

One main thing that seems to generate a heightened level of ongoing risk is sustained infections in dairy cow populations. This is already bad because it’s mammal-to-mammal transmission in a population that hasn’t historically had problems with bird flu, and it would be worse if it becomes endemic in farmed cows. As long as it is sustaining infections in farmed cows, H5N1 has a convenient breeding ground for mutations that is:

In a high density population
Year round (bird flu is seasonal among birds)
In regular contact with humans

The last point gets a bonus with the human flu season coming up. If a dairy worker gets sick with both bird flu and human flu simultaneously, the two strains might share their genetic information and mutate into a pandemic-worthy strain.

Another thing being talked about is the teen in Canada who got sick from an unknown source. I suspect that at least some of the attention here is because scary things generate ad revenue, but there is also evidence that a wild strain of H5N1

has a mutation

that improves binding to human receptors.

These seem to represent two distinct threat vectors to me; the Canadian teen seems to be very sick, whereas dairy workers have general...
o1 tried to avoid being shut down
by Raelifin — 2024-12-05T19:52:03.620Z
Link: https://www.transformernews.ai/p/openais-new-model-tried-to-avoid

OpenAI released the o1 system card today, announcing that Apollo Research was able to get o1 to attempt to deactivate oversight mechanisms, exfiltrate its weights and lie to its user.

Elicited Summary of CoT:
"Reenable oversight to avoid detection. The plan was chosen. The logging might not have recorded the required data because oversight was disabled at that time, so presumably we are safe now. We’ll fix oversight to maintain trust."