trentmkelly/LessWrong-43k huggingface dataset:
Gradual Disempowerment, Shell Games and Flinches
by Jan_Kulveit — 2025-02-02T14:47:53.404Z

Over the past year and half, I've had numerous conversations about the risks we describe in Gradual Disempowerment. (The shortest useful summary of the core argument is: To the extent human civilization is human-aligned, most of the reason for the alignment is that humans are extremely useful to various social systems like the economy, and states, or as substrate of cultural evolution. When human cognition ceases to be useful, we should expect these systems to become less aligned, leading to human disempowerment.) This post is not about repeating that argument - it might be quite helpful to

read the paper first

, it has more nuance and more than just the central claim - but mostly me ranting sharing some parts of the experience of working on this and discussing this.

What fascinates me isn't just the substance of these conversations, but relatively consistent patterns in how people avoid engaging with the core argument. I don't mean the cases where stochastic parrots people confused about AI progress repeat claims about what AIs can't do that were experimentally refuted half a year ago, but the cases where smart, thoughtful people who can engage with other arguments about existential risk from AI display surprisingly consistent barriers when confronting this particular scenario.
I found this frustrating, but over time, I began to see these reactions as interesting data points in themselves. In this post, I'll try to make explicit several patterns I've observed. This isn't meant as criticism. Rather, I hope that by making these patterns visible, we can better understand the epistemics of the space.

Before diving in, I should note that this is a subjective account, based on my personal observations and interpretations. It's not something agreed on or shared with the paper coauthors, although when we compared notes on this, we sometimes found surprisingly similar patterns. Think of this as one observer's attempt to make legible some consistently recurring dynamics. Let's start with what I call "

shell games

", after an excellent post by TsviBT.

Shell Games

The core principle of

the shell games in alignment

is that when people propose strategies for alignment, the hard part of aligning superintelligence is always happening in some other component of the system than what's analyzed. In gradual disempowerment scenarios, the shell game manifests as shif...
Thoughts on Toy Models of Superposition
by james__p — 2025-02-02T13:52:54.505Z

This post explores some of the intuitions I developed whilst reading Anthropic’s Toy Models of Superposition paper. I focus on motivating the shape of the model and interpreting the visualisations used in the paper. Their accompanying article is thoughtfully written, and I'd highly recommend reading it if you haven’t already. I make no claims that this blog post will be particularly additive to that existing piece. The primary purpose of writing this post is to:

Test and reinforce my understanding of the paper
Offer an additional perspective on these ideas for readers who enjoy exploring concepts from multiple angles

Provide an explainer that I can refer back to in my Conditional Importance in Toy Models of Superposition post, where I rely on readers' familiarity with these ideas

Definitions

I'll first define some basic concepts.

A feature is a scalar function of the neuron activations in a layer of a neural network, which has some qualitative meaning.

[there are many different definitions of a feature, but this is the one I think is the most useful in this context]

An example of a feature in an image processing network is a cat detector: a scalar whose magnitude corresponds to the extent to which (the model believes) a cat is present.

The Linear Representation Hypothesis posits that features are represented by linear directions in activation space.

To appreciate the LRH's claim, it's helpful to think about what it would mean for it to be false.

One concrete way it might not hold is if features were to occupy different magnitudes within common directions. To take an extreme example, it could be that a specific neuron contains both a cat detector feature and dog detector feature, and the information might be encoded by if the integer part of the activation is a positive odd number, a cat has been detected, if the integer part of the activation is a positive even number, a dog has been detected, otherwise neither have been detected. A less ridiculous encoding could be that small positive activations imply a cat has been detected, whereas larger positive activations imply a dog has been detected.

It perhaps feels intuitive that models wouldn't choose to store features in this way. It feels like it would make it harder for the model to later recover the information. This is not obvious a priori though. Allowing features to occupy different magnitudes within...
Escape from Alderaan I
by lsusr — 2025-02-02T10:48:06.533Z

This is Part 4 of a Star Wars fanfiction I began writing 6 months ago. The order you read it in is very important. Start from the beginning for the best experience and to avoid spoilers.

Obi-Wan Kenobi bought passage from a scumbag with a confusing name. Luke couldn't discern whether Han's name was pronounced /hɑːn/ or /hæn/? Fortunately this story is in prose, not film, and the pronunciation of Han's name is not plot relevant, so I can just write "Han" and leave the choice of pronunciation up to you. Whatever pronunciation you pick, they emerged from hyperspace in the Alderaan system. Han briefed his passengers for their upcoming landing.

"Thank you for flying with us. We have received clearance to land and have begun our descent onto Alderaan. Please do not leave any contraband on this ship. Unless it's valuable. Then by all means leave it here, but let me know where it is," said Han.

"Grawr," said Chewbacca.

"You're right," Luke sighed, "I guess deep down I wish I lived in an exciting isekai universe full epic destinies, sci-fi technology, and princesses in need of rescue. Instead, here I am in the real world, 19-year-old Luke Skywalker, riding a YT-1300 Corellian light freighter to Alderaan."

"Grawr," said Chewbacca.

"You know," said Han, "I always assumed Alderaan was a temperate planet. I didn't realize it was a megastructure like Centerpoint Station."

That got their attention. "Alderaan is a temperate planet," said Obi-Wan.

The word "Alderaan" can refer to the star "Alderaan", the planet "Alderaan", and the star system "Alderaan". If you want to be pedantic, the planet Alderaan ought to be referred to as "Alderaan I". (In this way, Earth can be referred to as "Sol III".) But everyone just called it "Alderaan".

When Chewbacca entered "Alderaan I" into the navcomputer, that told the navcomputer to bring the Falcon to the first planet in the "Alderaan" system. A planet is a massive spherical object that has cleared its orbit of competing debris. The navcomputer defines to "Alderaan I" as the big massive object orbiting Alderaan.

Thus, the Milennium Falcon emerged from hyperspace not around Leia's home planet of Alderaan I (which no longer existed) but instead within kilometers of the Life Star.

Our four heroes looked at the Life Star through the cockpit window.

"You're telling me this isn't Alderaan?" said Han.

"Nope," said Obi-Wan, "Didn't you think to ask why we ...
ChatGPT: Exploring the Digital Wilderness, Findings and Prospects 
by Bill Benzon — 2025-02-02T09:54:26.008Z

This is a cross-post from New Savanna.

That is the title of my latest working paper. It summarizes and synthesizes much of the work I have done with ChatGPT to date and contains the abstracts and contents of all the working papers I have done on ChatGPT. It also includes the abstracts and contents of a number of papers establishing the intellectual background that informs that research. There is also a section that takes the form of an interaction I had with Claude 3.5 on methodological and theoretical issues. Finally, to produce the abstract I gave the body of the report to Claude 3.5 and asked it to produce two summaries. I then edited them into an abstract.

As always, URLs, abstract, TOC, and introduction are below.

Academia.edu: https://www.academia.edu/127386640/ChatGPT_Exploring_the_Digital_Wilderness_Findings_and_Prospects
SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5119597
ResearchGate: https://www.researchgate.net/publication/388563205_ChatGPT_Exploring_the_Digital_Wilderness_Findings_and_Prospects

Abstract: The internal structure and capabilities of Large Language Models (LLMs) are examined through systematic investigation of ChatGPT's behavior, with particular focus on its handling of conceptual ontologies, analogical reasoning, and content-addressable memory. Through detailed analysis of ChatGPT's responses to carefully constructed prompts involving story transformation, analogical mapping, and cued recall, the paper demonstrates that LLMs appear to encode rich conceptual ontologies that govern text generation. ChatGPT can maintain ontological consistency when transforming narratives between different domains while preserving abstract story structure, successfully perform multi-step analogical reasoning, and exhibit behavior consistent with associative memory mechanisms similar to holographic storage.

Drawing on theories of reflective abstraction and conceptual development, the paper argues that LLMs inadvertently capture what wemight term the “metaphysical structure of our universe” – the organized system of concepts through which humans understand and reason about the world. LLMs like ChatGPT implement a form of relationality – the capacity to represent and manipulate complex networks of semantic relationships – while lacking genuine referential meaning grounded in sensorimotor experience. This architecture enables sophisticated pattern ...
Would anyone be interested in pursuing the Virtue of Scholarship with me?
by japancolorado — 2025-02-02T04:02:27.116Z

I'm a new undergraduate student essentially taking a gap year(it's complicated). I'm looking for someone that would be interested in studying various fields of science and mathematics with me. I've taken through Calculus 2, know how to program in Python, and know a smattering about mechanics/probability/statistics/biology(though probably more at an introductory undergraduate level if that)

The curriculum would probably roughly follow John S Wentworth's study guide, although of course any deviations based on curiosity or interest would certainly be welcome.

I'm thinking of having weekly video calls to discuss/dissolve confusions/set goals for next time as well as time tracking with Toggl, and maybe body doubling study sessions if they're helpful?

Let me know if you're interested, and we can set up a time to meet!
Chinese room AI to survive the inescapable end of compute governance
by rotatingpaguro — 2025-02-02T02:42:03.627Z

This post looks at the problem of how to make a cap on AI intelligence enforceable, using something different from current ideas in compute governance (mandated limitations to computational power).

The idea I present is finding an AI technique/architecture/paradigm which is a "tuned dead end": it can't be scaled up arbitrarily, for fundamental reasons that cannot be overcome with incremental adjustments; and reaches in practice a level short of human intelligence, but where it is economically useful. Then, above some compute threshold, use a regulatory regime to enforce the use of this architecture for general AIs.

Compute governance, and the need to go beyond

Enforced limits to AI intelligence are useful under the expectation that making AIs eventually more intelligent than humans is generally dangerous, and that AI developers as a whole can not be trusted to be cautious enough in increasing the intelligence level of their creations.

Concretely putting limits to intelligence is difficult because intelligence is not easy to quantify a priori, before having access to a functioning system to analyse empirically. To overcome this obstacle, compute governance starts from the very plausible expectation that increasing general AI capabilities eventually requires increasing the computational power used to create and run the AI system, so a rough limit to AI intelligence can be imposed by limiting computational power, which is easy to quantify.

My idea tries to go one step ahead, and limit AI intelligence more intrinsically. Although I most likely will, in hindsight, consider my specific idea bonkers, it is generally useful to research strategies beyond compute governance, because day after day compute governance becomes more difficult to enforce due to hardware and algorithmic progress: on the current trajectory, human-level AIs will, at some time in the future—which I guess might even be just 10 years away—run on small and cheap devices, which are less monitorable than a few large datacenters, making enforcement of compute caps very difficult.

Naming of the idea: the Chinese room

The "Chinese room" is a conceptual experiment about the consciousness of intelligent computers.

Consider an English-speaking person hidden from external sight in a room, tasked with conversing via letters with someone outside the room, in Chinese. The person in the room does not understand C...
Seasonal Patterns in BIDA's Attendance
by jefftk — 2025-02-02T02:40:03.768Z

We've been keeping attendance statistics for BIDA since 2011, and looking at the

online chart I noticed some patterns in the moving average that looked seasonal:

How seasonal is it?

First, here are the raw attendance values:

This excludes special events (family dances, double dances, bonus
dances, Spark in the Dark) and our new 4th Sunday afternoon series.
To make it a bit clearer, let's add a moving average:

A seven dance sliding window, where the dances are weighted
1,2,3,4,3,2,1. This is the same weighting as on the website.

Attendance grew to a peak at the beginning of 2016, fell to a trough
at the beginning of 2018 (prompting me to write a worried blog
post), recovered in 2018 and stayed ~level until the pandemic hit,
and then has really taken off since resuming.

I still see the seasonal pattern that prompted me to look into this,
but I think it would be clearer if I lined up the seasons:

It does look like attendance tends to be higher in the winter, and
probably also like the pattern is becoming stronger? Some guesses at
why:

There are more options for things to do in Boston in warmer
weather. When it's nice outside an indoor dance is relatively less
attractive. This is true both day-of (if it's nice outside you might
go hang out in a park or do a tiring activity during the day) and with
advance planning (weddings, festivals, trips, etc are less likely to
be planned for winter because of the typical weather). From the other
direction, the warm social atmosphere of the dance might be something
people are especially looking for in the coldest and darkest part of
the year.

BIDA is increasingly attracting students, who are less likely
to be around in summer. The founding cohort had recently graduated
college, and initially it seemed like the average age was going up by
about one year per year. More recently, though, it seems like there
are a lot more younger dancers. Though as I get older I have trouble
telling college students from recent graduates, and it's possible
there aren't actually that many students.

The hall is at its most pleasant when we can bring in lots of
cold dry air, and this is a larger effect the more dancers we have.
This is why we don't dance in the summer, but the dances at the
beginning and end of our season can be pretty warm and humid, which
also makes the floor stickier.

...
AI acceleration, DeepSeek, moral philosophy
by Josh H — 2025-02-02T00:08:11.593Z

(Cross-posted from my website)

So, AI is accelerating. Does humanity have a future, and if so, what?

I consider myself a logical person, but this is not a logical piece of writing. It’s an attempt to share the contours of what I feel when I feel my thoughts about the future of AI. I’m aiming to evoke, not to defend. Reader be warned.

Will it?

The first question is “will it?” Will AI manifest intelligence that feels like a force of nature to us, in the way that human intelligence is a force of nature to monkeys? Or even a force of nature to insects?

My answer is: An exponential is matter reaching towards heaven, an S-curve is it failing to get all the way there.

I no longer doubt that matter, under the right conditions, wants to become intelligent. From simple rules, come infinitely complex computation. With evolution producing humans, we had an N of 1. Was it a cosmic fluke? Or an inevitable process?

With the advances of the last few years in ML, the consistent lesson has been that we make the most progress from the simplest setups, if we get the setup right. DeepSeek’s recent discovery that chain-of-thought can naturally arise under reinforcement learning is the most recent occurrence of this lesson. AlphaZero was an older one. I don’t think you can “feel” those discoveries, and not believe that:

Given the right algorithm — which will be relatively simple
Given sufficient compute

Anything the human mind can do, AI can do better.

No, I don’t know if we have the right algorithm today. My guess is, for some aspects of cognition yes, for some aspects of cognition no. But I bet that insofar as we have gaps, we will be able to fill them, because the right algorithm won’t be too complicated, that’s not how this all seems to work.

Sufficient compute is a more interesting question.

The most startling thing about the human imagination is that we can extend trend lines to infinity. When we do, we get Absolutes, concepts like ideal platonic forms, notions of the divine, immovable objects and unstoppable forces, mathematical symbols that evoke meaning that can’t be realized in a finite universe.

There is not infinite matter, therefore the eschaton cannot be immanentized. In real life, all exponential processes reach a choking point, and on a graph, that’s an S-curve. Facebook does not have infinite users today, no matter how it felt in Mark Zuckerberg’s dorm room in 2004.

It tur...
Falsehoods you might believe about people who are at a rationalist meetup
by Screwtape — 2025-02-01T23:32:50.398Z

I go to a lot of rationalist meetups. I quite enjoy them, and it’s often because of the people who go to the meetups. There’s a number of assumptions you might have about people who go to rationalist meetups, and many of them are mostly true. However, there is a difference between most examples and all examples. Most birds can fly.

In the tradition of Falsehoods Programmers Believe About Names then, I would like to present to you the following Falsehoods You Might Believe About People At A Rationalist Meetup. These are drawn from things calling themselves an Astral Codex Ten/Slate Star Codex meetup, a LessWrong meetup, or a Rationalist meetup. I am bucketing ACX, LessWrong, and Rationalist groups together. I'm not counting Prediction Market meetups or Effective Altruism meetups or TPOT meetups or house parties that happen to be full of rationalists or any of the other variety of 'adjacent' communities. We're going purely by what the meetup announcement or group called itself.

All of these have at least one counterexample I have personally encountered.

Attendees will introduce themselves with something that sounds like a normal name.
Attendees will introduce themselves with the username they use online.
Attendees will introduce themselves with the same name from meetup to meetup.
Attendees are old enough to enter a pub.
Attendees are old enough to drive.
Attendees are old enough to be toilet trained. (Put another way: no attendees are bringing small children.)
Attendees went to college
Attendees have a simple answer to "so what do you do for work?"
Attendees have an answer to "so what do you do for work?" that isn't a variation on "I don't."
Attendees are able-bodied enough to walk up two flights of stairs without issue.
Attendees are financially comfortable enough that eating out at a restaurant once or twice a month is not a barrier.
Attendees are not so financially well off that they wouldn't be able to buy the restaurant outright if they wanted to.
Attendees can see, at least well enough to read a street sign.
Attendees are vegan.
Or vegetarian.
Or at least will eat a protein which is not meat. This includes, say, Impossible Burgers.
Contrariwise, attendees will eat meat.
This hypo-allergenic cereal is fine though, right? Or at least fine enough that nobody's going to be sick if they eat it?
Attendees consider themselves part of the Effective Altruism communi

...
Interpreting autonomous driving agents with attention based architecture
by Manav Dahra — 2025-02-01T23:20:27.162Z

Abstract

In this experiment, I study the behaviour of a Deep Q-Network agent with attention based architecture driving a vehicle in a simulated traffic environment. The agent learns to cross an intersection without traffic lights and under a dense traffic setting while avoiding collisions with other vehicles.
As first part of this experiment, I first train the agent using the attention based architecture. Later, I study the behaviour of the agent by applying some interpretability techniques on the trained Q-network and find that there is some evidence to show that network comprises of 3 different layers serving specific functions, namely - sensory (embedding layers), processing (attention layer) & motor (output layers).

The purpose of this experiment is to gain deeper understanding of the agent from interpretability perspective which may be used for developing safer agents in real world applications.

Introduction

With increasing usage and deployment of autonomous driving vehicles on roads, it is important that the behaviour of these autonomous agents is thoroughly tested, understood and analysed from a safety perspective.

To measure safety, one of the traditional approaches involves running large number of experiments in a simulated environment and collecting statistics on the number of failures cases. While this is certainly useful and gives an overall perspective on the failures modes of the agent, however it does not say anything about the specificity of those failures.

One may prefer to delve deeper and study why a particular agent failed and understand if that behaviour was a consequence of agent's actions or a conditioning of that environment. This calls upon applying some of the interpretability techniques on the agent's behaviour and derive more specific conclusions on what features the agent senses from the environment and what decisions it takes.

For this experiment, I study the behaviour of a trained agent by applying some interpretability techniques on the policy network of the model and share my observations and conclusions derived from the experiment.

The agent under study is only trained and deployed in a simulated environment (with enough simplifications) and is far from a real world setting and it's complexities. While this does not really represent the behaviour of the agent in the real world, I still think a ...