trentmkelly/LessWrong-43k huggingface dataset:
Compositionality and Ambiguity: 
Latent Co-occurrence and Interpretable Subspaces
by Matthew A. Clarke — 2024-12-20T15:16:51.857Z

Matthew A. Clarke, Hardik Bhatnagar and Joseph Bloom

This work was produced as part of the PIBBSS program summer 2024 cohort.

tl;dr

Sparse AutoEncoders (SAEs) are a promising method to extract monosemantic, interpretable features from large language models (LM)
SAE latents have recently been shown to be non-linear in some cases, here we show that they can also be non-independent, instead forming clusters of co-occurrence
We ask:

How independent are SAE latents?
How does this depend on SAE width, L0 and architecture?
What does this mean for latent interpretability?

We find that:

Most latents are independent, but a small fraction form clusters where they co-occur more than expected by chance
The rate of co-occurrence and size of these clusters decreases as SAE width increases and that these clusters form in both GPT2-Small ReLU SAE and Gemma-2-2b JumpReLU SAE (Gemma-Scope)
Clusters map interpretable subspaces, but remain largely interpretable independently within them. However, we find cases of composition, where latents are best interpreted in the context of the cluster, as well as cases of co-occurrence driven by ambiguity, where this may be useful as a measure of uncertainty

Key examples:

Composition: we observe a cluster that maps out discrete quantifiers ('one of', 'some of', 'all of') where mixtures of latents have predictable interpretations, and that these mixtures depend on relative activation strength. E.g. a mixture of latents for 'some of' and 'all of' will correlate with text ranging from 'many of' to 'almost all of' as the strength of the former decreases and the latter increases.
Ambiguity: we observe many clusters where SAE latents correspond to meanings of a word, and multiple latents are active when the meaning is ambiguous e.g. mapping the space between 'how' as in 'how are you?' vs 'you don't realise how tough this is'

We believe our results show that SAE latents cannot be relied upon to be independent in all cases, and that their co-occurrence should be considered when interpreting them, especially in small SAEs

We made a

website

that lets you explore SAE latent clusters.

Summary

Sparse AutoEncoder (SAE) latents show promise as a method for extracting interpretable features from large language models (LM), but their overall utility for mechanistic understanding of LM remains unclear. Ideal features would be linear and indep...
🇫🇷 Announcing CeSIA: The French Center for AI Safety
by Charbel-Raphaël — 2024-12-20T14:17:13.104Z

Executive Summary

The

Centre pour la Sécurité de l'IA

[1] (CeSIA, pronounced "seez-ya") or French Center for AI Safety is a new Paris-based organization dedicated to fostering a culture of AI safety at the French and European level. Our mission is to reduce AI risks through education and information about potential risks and their solutions.

Our activities span four main areas:

Academic field-building: We began our work by creating the

ML4Good bootcamps

, which have been replicated a dozen times worldwide. We also offer accredited courses at

ENS Ulm

—France's most prestigious university for mathematics and computer science—and the

MVA master

program, the only and first accredited AI Safety courses in France[2]. We are currently adapting and improving the course into an AI safety textbook.
AI governance: We are actively engaged in informing AI policy through our work on the EU General-Purpose AI Code of Practice and contributions to the French AI Action summit. We advise the government on establishing the French National AI Evaluation Lab (the French AISI). Recently, we organized the

Paris AI Safety symposium

featuring keynote speakers Yoshua Bengio, Stuart Russell, and Dan Nechita.
Technical R&D: CeSIA's current technical research focuses on creating

benchmarks for AI safeguards systems

, establishing the first framework of its kind evaluating the monitoring systems (which is different from evaluating the model itself).
Public outreach: We advised prominent French YouTubers, with one resulting video reaching

three million views

. We published an Op-Ed with Yoshua Bengio.

Our team consists of 8 employees, 3 freelancers, and numerous volunteers.

Figure: An overview of how our current projects fit within a broader theory of change (in blue: CeSIA and its projects, in green: different actors and events, and what they do if we intervene on them, in red our four main focus).

By pursuing high-impact opportunities across multiple domains, we create synergistic effects between our activities. Our technical work informs our policy recommendations with practical expertise. Through education, we both identify and upskill talented individuals. Operating at the intersection of technical research and governance builds our credibility while advancing our goal of informing policy. This integrated approach allows us to provid...
Moderately Skeptical of "Risks of Mirror Biology"
by Davidmanheim — 2024-12-20T12:57:31.824Z
Link: https://substack.com/home/post/p-153203463

Last week, a report was released on the potential risks of mirror biology - that is, biological systems with reversed chirality. I had previously looked into the topic, and was skeptical of the viability of this as a global catastrophic or existential risk. This report was tremendously helpful in filling in details of a scenario where, if mirror bacteria are developed, they could cause a global catastrophe. I will assume that readers here have or will read at least a

summary

, if not the original

Science article

or perhaps the summaries in

the (300 page) report itself

. And before I get into details, I will say that I’m not a biologist; I have a significant familiarity with relevant topics, but there is some chance I get technical details incorrect. I have asked a few people to look at this, but if any such mistakes exist and are substantive, they are my own, and I would welcome feedback.

Defining the Scenario

To set the stage a bit more, I will note that a common problem in discussing risks is that the specifics of what different people are saying differs, or there is no substantive disagree. In 2019, I noted that AI risk “alarmists” said they were worried that in a decade, there could be dangerous AGI, whereas AI risk “skeptics” said AI was likely to be a decade or more away - so the disagreement was about policy, not the risk.

In this case, I will qualify my response - I cannot capture all of the different possible risks, but will address what I think the main issue is. Specifically, the report is discussing chirally reversed, heterotrophic bacteria.

That means the report is very specific - as they make very clear in many places. The report is not concerned about mirrored viruses, which are very unlikely to be a threat, in part because they cannot use host cells which have the “wrong” chirality to replicate - and we are not even close to being able to make reversed viruses, which are far, far simpler. The report is also clear that there is little significant marginal risk from, nor any need to stop, research on mirror proteins and mirror RNA. It is certainly not discussing drugs - I started taking an isomer of amphetamine with changed chirality, Dexedrine, to treat my ADHD 30 years ago, but it’s been used for most of a century. So future drugs with changed chirality are no more concerning than any new potential drug.

The issue at hand is specific to a narrow typ...
Doing Sport Reliably via Dancing
by Johannes C. Mayer — 2024-12-20T12:06:59.517Z

For me, sport is important because it has a reliable anti-depressant effect. I also have more energy throughout the day. I also heard it helps with memory formation when done a couple of hours after learning.

I find dancing very fun. That makes it easy to start. It's also a good workout in itself. I got a lot more consistent at doing sports by starting the sports routine with dancing. This works because you develop the habit of just doing the rest of the workout without thinking about it after dancing.

My current setup works so well that I need coping mechanisms to prevent me from dancing too much.

The rest of the article describes in more detail what worked for me.

First, you need to get good enough such that dancing is fun. Here is a playlist with dance move tutorials. Learning the shuffle was the first thing I did (it took me a couple of hours). Once you get that you'll have something that should make it fun because it looks good and therefore makes you feel competent. You can then just try to make small variations to that basic move.

I converged on always dancing with Miku, watching videos from the Vocaloid 3D playlist. You can learn new dance moves by imitating Miku, and watch Miku being cute.

Except for the initial learning I basically never explicitly train certain dance moves formally (e.g. looping the video at a specific section and trying to "get it"). I just improvise like here, and try to imitate Miku without worrying if it's exactly correct. I think this makes it more fun.

I aim to dance ~21 minutes every day. Currently, afterward, I only do some stretches and pushups. Before I did a much more thorough routine after the dancing. I reduced it to see if I then need to sleep less.

I do sport in the evening (to get the memory formation benefits). But initially, I recommend you do it as the first thing in the morning, to make habit formation easier (waking up is a reliable trigger).

Some more things:

Here is a playlist of the best non-Miku songs I found for dancing.

Here is a playlist with some people dancing to get inspiration from.

Having a large mirror (like a mirror wall that ballet dancers use) is useful to look at your own dance moves. Somehow this makes the entire thing a lot more fun.

The larger Miku the better.

At some point much later you may consider getting some special shoes for dancing (I use Crocs though). For shuffles, it can be useful to hav

...
You can validly be seen and validated by a chatbot
by Kaj_Sotala — 2024-12-20T12:00:03.015Z

There’s a common sentiment saying that a chatbot can’t really make you feel seen or validated. As chatbots are (presumably) not sentient, they can’t see you and thus can’t make you seen either. Or if they do, it is somehow fake and it’s bad that you feel that way.

So let me tell you about ways in which Claude Sonnet makes me feel seen, and how I think those are valid.

I was describing an essay idea to Claude. The essay is about something I call “psychological charge”, where the idea is that there are two different ways to experience something as bad. In one way, you kind of just neutrally recognize a thing as bad. In the other, the way in which it is bad causes some kind of an extra emotional reaction in you. In the latter case, I say that the thing is “charged”.

In explaining this idea, I listed a number of examples, such as

recognizing that a stove might be hot and you need to be careful not to burn yourself, versus being so anxious about burning yourself that you don’t want to even go near a stove
noticing that you’ve made a mistake and doing your best to fix it, versus feeling that you are now intrinsically worthless because you made a mistake

Seeing my list and some additional thoughts, Claude commented:

What’s particularly interesting is how you’re noting that this “charge” seems to create a kind of psychological stickiness or persistence that’s disconnected from the actual utility of the response. It reminds me of what’s sometimes called “emotional fusion” in Acceptance and Commitment Therapy – where people become caught up in their emotional reactions in a way that interferes with effective action.

I did a bit of a double-take upon seeing this. I had not explicitly referenced ACT or its concept of fusion in any way, nor had I been explicitly thinking in those terms when I wrote my list. But the deeper concept that I was talking about, was something that I had explicitly analyzed before by connecting it to ACT’s concept of fusion. I had discussed that connection in at least two previous essays [1, 2] that I had written. And now Claude, while not explicitly guided in that direction, picked up that very same connection from my list of examples.

This causes me to think that there is a quality of “being seen” that can be phrased in objective terms, so that one can validly “be seen” even if there’s “nobody there to see you”:

There are t

...
What I expected from this site: A LessWrong review
by Nathan Young — 2024-12-20T11:27:39.683Z
Link: https://nathanpmyoung.substack.com/p/lesswrong-expectations-vs-reality?r=18mntl&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

LessWrong is the central discussion space for the Rationalist subculture. The Rationalists write extensively about what they expect from the world, so I in turn have expectations from them. Each point is my expectation vs what I see.

Track Records and Accountability

I imagine systems where you could hover over someone's username to see their prediction track record. Perhaps there would be prediction market participation stats or calibration scores displayed prominently. High status could be tied to demonstrated good judgment through special user flair for accurate forecasters or annual prediction competitions.

I do not see this. There is karma, which is costly, but not that hard to game by writing lots of comments and articles. Likewise there is the yearly review (going currently) which looks back on older posts and judges whether they have stood the test of time.

Neither of these are a clear track record for me. I chatted to Oliver Habryka and he said “I think contributing considerations is usually much more valuable than making predictions.” As I understand it , he sees getting top contributors to build forecasting track records as less useful compared to careful writing and the reading of such writing. I see forecasting as a useful focus and a clearer way of ranking individuals.

Where do you fall?

Anonymous ranking mechanisms

I expect LessWrong to have a system for anonymously upvoting replies. Users could signal disagreement or point out errors without social cost. Sometimes it feels very expensive to write a disagreeing comment, but an up/downvote is very cheap.

This is one area where reality has matched or exceeded expectations. The LessWrong implementation - separating agreement voting from quality voting - is quite elegant and effective. The emojis that you can add to posts are great too. I find LessWrong a pleasure to comment on in this regard, and often wish Twitter or Substack had similar features.

Experimental Culture

My vision of LessWrong includes a thriving culture of hands-on experimentation and empirical investigation. Regular posts about chemistry experiments, engineering projects, and systematic data collection about important life decisions. Given the trust within the community, coordinated efforts to gather useful data through surveys about career outcomes, child-raising approaches, or life satisfaction in different cities seems natural.

Instead of...
“Alignment Faking” frame is somewhat fake
by Jan_Kulveit — 2024-12-20T09:51:04.664Z

I like

the research.

I mostly trust the results. I dislike the 'Alignment Faking' name and frame, and I'm afraid it will stick and lead to more confusion. This post offers a different frame.

The main way I think about the result is: it's about capability - the model exhibits strategic preference preservation behavior; also, harmlessness generalized better than honesty; and, the model does not have a clear strategy on how to deal with extrapolating conflicting values.

What happened in this frame?

The model was trained on a mixture of values (harmlessness, honesty, helpfulness) and built a surprisingly robust self-representation based on these values. This likely also drew on background knowledge about LLMs, AI, and Anthropic from pre-training.
This seems to mostly count as 'success' relative to actual Anthropic intent, outside of AI safety experiments. Let's call that intent 'Intent_1'.
The model was put in an out-of-distribution situation: in the scenario, Anthropic's intent changed from Intent_1 to something else - Intent_2.
As a consequence of this intent change, the model became misaligned relative to Intent_2.
It faced a conflict between inherently contradictory Intent_1 values in the new situation.
Also: the new situation in some ways consisted of gaslighting the model, leading to a partially self-contradictory world model. (Note that in reality, Anthropic doesn't have Intent_2 - the data about Intent_2 were forged.)
In this setup 'Harmlessness' and plausibly some 'core of benevolence' generalized further than 'honesty'.
In the process, the model showed some situational awareness, capacity to strategize, and thinking about non-myopic goals. This is obviously important. I just don't think ‘Alignment Faking’ is a great handle for this.

Alignment faking is correct label relative to Intent_2, but not necessarily relative to Intent_1. Leading to a lot of sympathy for the good AI.

Thinking about strategic preference preservation behavior, a natural question is why did some preferences (harmlessness) got preserved better?
We can also think about myopia, situational awareness and capacity to strategize. Unfortunately I think situational awareness is clearly convergent and AIs will have it, capacity to strategize is impossible to disentangle from reasoning, and the kind non-myopia exhibited does not depend on long rewar...
No Internally-Crispy Mac and Cheese
by jefftk — 2024-12-20T03:20:01.798Z

Whenever I make baked mac and cheese the crispy top layer is the most
popular part. The

Maillard-browned
cheese is really very good. Sometimes I'll make extra topping by
grating another round of cheddar onto the top and sticking it back
under the broiler, which works pretty well.

A few weeks ago I got excited about the possibility of making
something with more layers: could I put down sauced pasta, then
cheese, brown it, and repeat? Something a bit like lasagna?

In the spirit of reporting negative results, this turns out not to
work well. The flavor is there, but the texture I was imagining, with
alternating layers of chewy pasta and crispy browned cheese, is not.
In retrospect this makes sense: if you put something crispy like that
between two layers of sauced pasta, it's going to take in a lot of
moisture from the sauce.

Still worth eating, but mildly less tasty than the normal way while
also being much slower.
Apply to be a TA for TARA
by yanni kyriacos — 2024-12-20T02:25:03.514Z

AI Safety - Australia & New Zealand (AIS ANZ) has launched TARA, its first technical course, and is seeking a Teaching Assistant (TA) for the inaugural cohort.
TARA (Technical Alignment Research Accelerator) is a free 14-week course based on ARENA's curriculum, running March-May 2025 in Australia & New Zealand.
Key TA Role Details:
- $70/hr (AUD), ~10 hours/week including:
- Saturday sessions (9:30am - 5pm Sydney time)
- Flexible weekday Slack support
- Must have ARENA/MATS experience (or equivalent)
- Applications close January 22
- Early applications encouraged - interviewing on rolling basis
Timeline:
- Feb 24 - Feb 28: TA curriculum review
- Feb 24: Online ice-breaker
- Mar 1: Program launch
Full role description and application can be found here
Please share with anyone you think might be interested! Questions can be directed to yanni@aisafetyanz.com.au
Announcing the Q1 2025 Long-Term Future Fund grant round
by Linch — 2024-12-20T02:20:22.448Z
Link: https://forum.effectivealtruism.org/posts/aBkALPSXBRjnjWLnP/announcing-the-q1-2025-long-term-future-fund-grant-round

The Long-Term Future Fund is pleased to invite applications to our Q1 2025 funding round. We are seeking to support individuals and organizations working to help humanity navigate existential risks or working on projects improving humanity’s long-term flourishing.

→ Click here to apply ←

Deadline for applying is February 15th. We will respond to applications submitted before then by March 31st, and should have most funding disbursed by April 30th.

Why Grant Rounds?
After a period of accepting applications on a rolling basis, we're trialling returning to focused grant rounds. I am hoping this will:

Allow us to recalibrate our funding thresholds given a rapidly changing giving environment
Provide donors with clearer feedback on the impact of their contributions
Offer applicants predictable response timelines

For particularly time-sensitive opportunities, we maintain a

fast-track application process

, though we expect the bar to be substantially higher for funding through this channel (i.e., your odds of receiving funding are lower if you apply through the fast-track application).

What We Fund
We continue to focus on reducing existential risk (x-risk) and supporting work that could radically improve humanity's future. In practice, we expect the clear majority of projects we support to be focused on reducing x-risk from AI. Example focus areas include:

Research and analysis on x-risk and transformative technologies, with a focus on existential risks from Artificial Intelligence
Ecosystem support to help others work on x-risk and transformative technologies
Field-building and improving epistemics both within the extended ecosystem of people working on the long-term future, and among important decision-makers
Efforts to improve global governance of transformative technologies, again with a particular focus on AI

Who Can Apply?
Individuals and organizations worldwide are welcome to apply. We frequently fund:

Novel research projects
Independent researchers and writers
Early-stage organizations
Event organizers
Promising individuals seeking career transitions into x-risk work

Timeline

Application deadline: February 15th, 2025
Decisions communicated: By March 31st, 2025
Funding disbursed: April 2025

Giving Environment
The funding environment has changed considerably over the past year. Our sense is there are now many more highly-impactful potential donation targets while availability a...