trentmkelly/LessWrong-43k huggingface dataset:
The Logistics of Distribution of Meaning: Against Epistemic Bureaucratization
by Sahil — 2024-11-07T05:27:20.276Z

This is an excerpt from the Introduction section to a book-length project that was kicked off as a response to the framing of the essay competition on the Automation of Wisdom and Philosophy. Many unrelated-seeming threads open in this post, that will come together by the end of the overall sequence.
If you don't like abstractness, the first few sections may be especially hard going.

Generalization

This sequence is a new story of generalization.

The usual story of progress in generalization, such as in a very general theory, is via the uncovering of deep laws. Distilling the real patterns, without any messy artefacts. Finding the necessities and universals, that can handle wide classes rather than being limited to particularities. The crisp, noncontingent abstractions. It is about opening black boxes. Articulating mind-independent, rigorous results, with no ambiguity and high replicability. Considered thoroughly “understood” only when you can program it; that even a machine could automate. Gears-level understanding. Clear mechanistic, causal stories plus sharp high-level concepts that get at the core and are thus robust to extrapolate from. Clean factorings with compositionality that preserve formal invariants.

Some of these might sound unrelated, apart from being good practices. In fact, they are very related. They all achieve the scaling and transport of insight (which is the business of generalization) by isolation and exclusion.[1]They decontextualize, and attempt to find the static, eternal, preformed, self-contained, independent principles by retreating ever deeper inwards, into that which has been identified as meaningful.

The thesis is that this exclusion-based approach to proliferation of meaning is not the only option. It is largely by ignoring or distrusting the fluidity available to intelligence, aliveness, mind, time, that we resort to such stasis-centered[2] sensemaking.

Though it has its merits, context-independence is subtly tyrannical: centralized, preformed, numb. As we’ll see, even when equipped with the dials that allow, say, a theory, to change shape in response to the environment it is placed in, the frame of the theory remains rigid. When intelligence and mind is available[3] at scale, you generalize not by finding that which is persiste...
SAEs are highly dataset dependent: a case study on the refusal direction
by Connor Kissane — 2024-11-07T05:22:18.807Z

This is an interim report sharing preliminary results. We hope this update will be useful to related research occurring in parallel.

Executive Summary

Problem: Qwen1.5 0.5B Chat SAEs trained on

the pile

(webtext) fail to find sparse, interpretable reconstructions of the refusal direction from

Arditi et al

. The most refusal-related latent we find is coarse grained and underperforms the refusal direction at steering tasks.

This is disappointing. The point of an SAE is to find meaningful concepts. If it can’t sparsely reconstruct the important refusal direction, then that means it’s either missing the relevant concepts, or these are shattered across many latents.

Solution: Training a new SAE on a chat-specific dataset,

LmSys-Chat-1M

, finds a significantly sparser, more faithful, and interpretable reconstruction of the “refusal direction”.

The LmSys SAE is also more capable of finding interpretable “refusal” latents that we can use to effectively steer the model to bypass refusals.

We find that, for the task of faithfully reconstructing the “refusal direction”, base model SAEs trained on chat data are better than chat model SAEs trained on the pile (consistent with our

prior work

).
We open source our code and SAEs at

https://github.com/ckkissane/sae-dataset-dependence

An SAE trained on the

LmSys-Chat-1M

dataset finds a significantly sparser decomposition of the “refusal direction” (

Arditi et al.

) than an SAE trained on

the pile

. The plot shows relative MSE after optimizing a linear regression to reconstruct the refusal direction with a fixed number of latents. Both SAEs are trained on the activations of Qwen 1.5 0.5B Chat.

Introduction

We would like SAEs to be a useful tool for understanding and steering models on downstream tasks. However, SAEs sometimes fail to be useful on the specific tasks we care most about. Many interesting downstream tasks are in specific domains, like chatbots or biology. An obvious idea to make an SAE more effective is to train it more (or entirely) on data from that domain (

Bricken et al. 2024a

). In this post, we show that this technique is effective on the specific chat task of reconstructing the “refusal direction” from

Arditi et al.

We also show that the chat data SAEs are more capable of finding relevant refusal latents for steering.

While we expect domain specific SAEs to be applicable to m...
Should CA, TX, OK, and LA merge into a giant swing state, just for elections?
by Thomas Kwa — 2024-11-06T23:01:48.992Z

As Americans know, the electoral college gives disproportionate influence to swing states, which means a vote in the extremely blue state of California was basically wasted in the 2024 election, as are votes in extremely red states like Texas, Oklahoma, and Louisiana. State legislatures have the Constitutional power to assign their state's electoral votes. So why don't the four states sign a compact to assign all their electoral votes in 2028 and future presidential elections to the winner of the aggregate popular vote in those four states? Would this even be legal?

The population of CA is 39.0M (54 electoral votes), and the population of the three red states is 38.6M (55 electoral votes). The combined bloc would control a massive 109 electoral votes, and would have gone for Biden in 2020 and Trump in 2024. Every state has an incentive to sign to increase their voters' influence in the national election.

There has been one similar proposal before: a National Popular Vote Interstate Compact which would go into effect when states controlling >50% of electoral votes sign it. It is only popular with blue states-- swing states don't want to reduce their influence, and red states don't want to give up the Republican electoral college advantage. Merging states would be far superior incentive-wise, as the influence of every signatory would increase and there would be essentially no expected net shift in the election results.

The compact should only go into effect once all 4 states sign, of course. But even then, there is another potential problem: Say that polls suggest the combined bloc is trending blue in 2028. The Louisiana legislature has an incentive to pull out at the last minute, and the other red states will follow. To prevent this, there must be a provision that once signed by all 4 states, the compact can't be repealed by any state until after the next election.

Smaller states have an even greater incentive to merge, e.g. Rhode Island with Montana. With only two states in the compact, there are fewer difficulties in getting all the states to sign at once. And due to their slightly higher electoral vote counts per voter, these small states' voters would quickly go from among the least important in the election to the most important.
New Funding Category Open in Foresight's AI Safety Grants
by Allison Duettmann — 2024-11-06T22:59:41.065Z

Foresight Institute's AI Safety Grants Program added a new focus area in response to the continually evolving field. Moving forward, our funding ($1.5M-$2M annually) will be allocated across the following four focus areas:

Automating AI-relevant research and forecasting

Scaling AI-enabled research to support safe AGI development
Scaling efficient forecasting methods relevant for safe AGI
Other approaches in this area

2. Neurotech to integrate with or compete against AGI

Brain Computer Interfaces (BCI) to enhance human cognition or facilitate human-AGI collaboration
Whole Brain Emulations (WBE) which might function as human-like general intelligences that are more interpretable and alignable than AGI
Lo-fi emulations using behavioral and neural data with deep learning, potentially offering a cost-effective alternative to full WBEs
Other approaches in this area

3. Security technologies for securing AI systems

Implementations of computer security techniques (including POLA, SeL4-inspired systems, and hardened hardware security) to safeguard AI systems
Automated red-teaming for AI security and capabilities
Cryptographic and related techniques to enable trustworthy coordination architectures
Other concrete approaches in this area

4. Safe multipolar human AI scenarios

Game theory that addresses interactions between multiple humans, AIs, or ultimate AGIs
Avoiding collusion and deception and/or encouraging pareto-preferred/positive-sum dynamics
Approaches for addressing principal-agent problems in multi-agent systems
Other concrete approaches in this area

Application Process

We accept applications on a quarterly cycle, with deadlines at the end of March, June, September, and December. Decisions are made within 8 weeks of each deadline.

Next Deadline: December 31st, 2024.

For more information and to apply:

https://foresight.org/ai-safety
Scattered thoughts on what it means for an LLM to believe
by TheManxLoiner — 2024-11-06T22:10:29.429Z

I had a 2-hour mini-sprint with Max Heitmann (a co-founder of Aether) and Miles Kodama about whether large language models (LLMs) or LLM agents have beliefs, and the relevance of this to AI safety.

The conversation was mostly free-form, with the three of us bouncing ideas and resources with each other. This is my attempt at recalling key discussion points. I have certainly missed many points, and the Aether team plan to write a thorough summary from all the mini-sprints they organised.

I write this for three reasons. First, as a way to clarify my own thinking. Second, many of the ideas and resources we were sharing were new to each other, so good chance this will be useful for many LessWrong readers. Third, you might be able to contribute to the early stages of Aether and their strategy.

What is a belief?

Max provided three definitions of beliefs:

Representations. A system believes P if there is an explicit representation of 'P is true' in the system
Behaviour. A system believes P if its behaviour is consistent with something that believes P. (Known as dispositionalism)
Predictive power. A system believes P if this is the best predictive explanation of the system.

Many of our discussions during the two hours boiled down to what we considered to be a belief or not. This is something I am still confused about. To help explain my confusion, I came up with the example statement 'The Eiffel Tower is in Paris' and asked when the information 'The Eiffel Tower is in Paris' corresponds to a belief.
System 1: The Eiffel tower itself. The Eiffel tower itself contains the information that 'The Eiffel Tower is in Paris' (the properties of the physical object is what determines the truth of the statement), but the Eiffel tower 'obviously' has no beliefs.

System 2: An LLM. The information 'The Eiffel Tower is in Paris' is encoded in a (capable enough) foundation model, but we did not agree on whether this was a belief. Max said the LLM cannot act on this information so it cannot have beliefs, whereas I think that the ability to correctly complete the sentence "The Eiffel Tower is in the city " corresponds to the LLM having some kind of beliefs.

System 3: An LLM agent. Suppose there is a capable LLM based agent that can control my laptop. I ask it to book a trip to the Eiffel Tower, and then it books travel and hotels in Paris. We all agreed that the information 'The Eiffel Tower is i...
Anthropic: Three Sketches of ASL-4 Safety Case Components
by Zach Stein-Perlman — 2024-11-06T16:00:06.940Z
Link: https://alignment.anthropic.com/2024/safety-cases/

The cleanest argument that current-day AI models will not cause a catastrophe is probably that they lack the capability to do so. However, as capabilities improve, we’ll need new tools for ensuring that AI models won’t cause a catastrophe even if we can’t rule out the capability. Anthropic’s Responsible Scaling Policy (RSP) categorizes levels of risk of AI systems into different AI Safety Levels (ASL), and each level has associated commitments aimed at mitigating the risks. Some of these commitments take the form of affirmative safety cases, which are structured arguments that the system is safe to deploy in a given environment. Unfortunately, it is not yet obvious how to make a safety case to rule out certain threats that arise once AIs have sophisticated strategic abilities. The goal of this post is to present some candidates for what such a safety case might look like.

This is a post by Roger Grosse on Anthropic's new Alignment Science Blog. The post is full of disclaimers about how it isn't an official plan and doesn't speak for the org (and that it's inadequate: "none of the sketches presented here fully succeeds in addressing the sabotage risk"). But presumably it's Anthropic's best sketch of ASL-4 safety cases.

The three safety cases are Mechanistic Interpretability, AI Control, and Incentives Analysis.

Regardless of how good these safety cases are, it's good when labs share their thinking on safety stuff; yay Anthropic.
Meme Talking Points
by ymeskhout — 2024-11-06T15:27:54.024Z

I’m going to describe a phenomenon that’s likely very obvious, but nevertheless I think it’s worth documenting because I’ve noticed it more and more. I’ll refer to it as Meme Talking Points, and the first instance I noticed this was based on my conversation on the Ray Epps conspiracy theory earlier this year. My interlocutor brought up the case of Ricky Vaughn as evidence that conservatives were uniquely targeted by the legal justice system, expressing surprise that I wasn’t familiar with the case. He then sought to bring me up to speed by claiming that Ricky Vaughn was “arrested for posting memes”.

I thought that was a curious way of describing Vaughn’s case, because I was familiar enough with the story to know there was more to it. Then I noticed the exact same talking point (“prosecuted for posting memes”) come up elsewhere in a debate with Destiny and John Doyle (timestamped: “they dox people, arrest them for memes such as Ricky Vaughn”), and most recently as a comment by Simon Laird.

If you’re unaware, Vaughn posted a fake 2016 Hilary Clinton campaign ad that implored her voters to cast their votes by tweeting.

Vaughn was then prosecuted under a relatively obscure Reconstruction Era law from 1870 which criminalizes “conspiring against rights”. He was convicted in 2023 and sentenced to 7 months. There’s plenty about Vaughn’s case you could argue about, such as whether this was intended as a joke or as an earnest attempt to mislead voters, or whether his conduct should nevertheless be protected by the first amendment, or whether his case was an example of selective prosecution. All those points are perfectly fair game for debate, but they’re impossible to unearth if you describe his case as “posting memes” rather than the much less ambiguous “posting fake and misleading Hilary Clinton campaign ads”.

The obfuscation has to be intentional. You can consider this a manifestation of Scott Alexander’s noncentral fallacy or my own revision I named the sticker shortcut fallacy. To describe the obvious, absent any additional information or context, the phrase “posting memes” conjures up a thoroughly banal activity. The prototypical example you might then think of would be someone getting sent to federal prison for posting the jealous girlfriend meme.

Such a scenario would be inherently compelling because it describes unusual and outrageous events that inevitably comman...
Advisors for Smaller Major Donors?
by jefftk — 2024-11-06T14:30:06.187Z

Open Philanthropy (OP) is the largest grantmaker who is moving money to the
things I think are most valuable, including (disclosure!)

my work at the NAO. There's
been a lot of discussion in the effective altruism community about
where this leaves smaller donors, and where they might have a

comparative
advantage. For example:

OP recently
announced that due to changes in Good
Ventures' (GV) strategy that they'd be recommending less funding
in several areas. This seems
to have included work on wild animal welfare, invertebrate
welfare, digital minds, and gene editing, though I'm not aware of a
public list. If you trust OP's prioritization more than GV's and/or
you think these are valuable neglected areas, you could help make up
for this shortfall.

Some kinds of work offer PR risks, either to OP or to OP's
other grantees. Many individuals are better placed to take on these
risks.

Money donated directly to US political candidates is restricted
on a per-person level, and goes farther than unrestricted "Super PAC"
donations. Individual donors have a strong
advantage here.

Some kinds of policy work don't have regulations that restrict
funding, but are still restricted in practice. For example, an
advocacy organization could be more effective when it can demonstrate
broad-based support, or could be tarred by the support of billionaires
like Moskovitz and Tuna.

For policy and politics in other countries the effect is even
stronger: in some cases it's illegal for OP to make these kinds of
grants, while in others it's just politically counter-productive. On
the other hand, small-scale donors contributing to political efforts
in their own countries is generally seen as the way politics is
supposed to work.

This is not a complete list [1] but I do think it has the biggest
reasons. Overall it seems to me that if an independent donor is
funding something OP would be happy to fund, ideally the donor would
find somewhere they could do better.

Despite all this, I've generally not felt like our family's donations
have taken advantage of our position as independent donors. Mostly
we've contributed to Funds, and while some of these relative
advantages don't apply there (they don't need to convince GV to make
grants) most still do.

I think the main reason we haven't done better here is that
investigating and comparing donation oppor...
Scissors Statements for President?
by AnnaSalamon — 2024-11-06T10:38:21.230Z

(Epistemic status: I spoke simply / without "appears to" hedges, but I'm not sure of this at all.)

I’m confused why we keep getting scissors statements as our Presidential candidates, but we do. (That is: the candidates seem to break many minds/communities.)

A toy model:[1]

Take two capacities, A and B. Ideally anti-correlated.

Craft two candidates:

Candidate X, who seems acceptable if you’re A-blind (if you have a major gap in your situation awareness near A).
Candidate Y, who seems acceptable if you’re B-blind (if you have a major gap in your situation awareness near B).

Now let voters talk.

“How can you possibly vote for X, given how it’ll make a disaster on axis A?”, asks Susan. (She is B-blind, which is part of why she is so confused/irate/loud here.) Susan inquires in detail. She (accurately) determines the staunchest X-voters don't understand A, and (understandably, but incorrectly) concludes that this explains their X-voting, that they have nothing to teach her, and that she should despair of working well with anyone who voted for Candidate X.

““How can you possibly vote for Y, given how it’ll make a disaster on axis B?”, asks Robert. He, too, inquires in detail. And he (accurately) determines the staunchest Y-voters have a key basic blind spot where he and his friends/neighbors have sense... feels a sense of closure ("okay, it's not that they know something I don't know"), and despairs of working well with anyone who voted for Y.

The thing that annoys me about this process is that, in the wake, it is harder for both sets of voters to heal their own blind spots. “Being able to see A accurately” is now linked up socially and verbally with “being one of the people who refuse to acknowledge B” (and vice versa). (This happens because the ontology has been seized by the scissors-statement crafters – there is a common, salient, short word that means both “A matters” and “B is fake,” and people end up using it in their own head, and, while verifying a real truth they can see, locking in a blind spot they can’t see.)

^

This is a toy model for how the "scissors-ness" works, not for why some process is crafting us candidates like that. I don't have a guess about that part. Though I like these articles.
How to cite LessWrong as an academic source?
by PhilosophicalSoul — 2024-11-06T08:28:26.309Z

Hi there.

Quick question. I am using a few articles from LessWrong for a dissertation. Are there any mainstream articles/sources that reference LessWrong as being the catalyst/partial source for AI alignment, researchers, and other academic literature? I think it's snobbish, or, discriminatory to regard LessWrong as merely another online website. I was hoping to get some advice on how to formulate a paragraph justifying the citation of LessWrong?

Thanks.
How to put California and Texas on the campaign trail!
by Yair Halberstadt — 2024-11-06T06:08:25.673Z

In the USA, the president isn't determined by a straight vote. Instead, each state gets a certain number of Electoral College (EC) votes, and the candidate with 270 EC votes wins.

It's up to each state to decide how to allocate its EC votes. Most do “winner-takes-all,” but some, e.g., Maine and Nebraska, split them up.

California and Texas have the most EC votes of any state, with 54 and 40 votes respectively, so you would think they would get a lot of love from presidential candidates. Instead, they're mostly ignored—California will always be Blue, and Texas Red, so what's the point of pandering to them? This is clearly bad for Californians and Texans as their interests aren't listened to.

So why doesn't California switch to a proportional EC vote split? One that would ensure that there's always something to gain by pandering to Californian interests?

Because most Californians are Democrats, and while a proportional vote split would be good for Californians, it would be bad for Democrats, who in the 2020 election would have lost 20 EC votes—over half of Biden's margin of 37. All in all, not worth it.

But the same issue impacts Texas too, which, if they'd switched to proportional splitting, would have handed Biden 18 votes.

This suggests an obvious solution: California and Texas should mutually agree to split their vote proportionally. That way, neither Republicans nor Democrats significantly gain, but California and Texas would finally be on the presidential campaign circuit!