trentmkelly/LessWrong-43k huggingface dataset:
Proselytizing
by lsusr — 2025-02-22T11:54:12.740Z

Religions can be divided into proselytizing religions (e.g. Mormons) who are supposed to recruit new members, and non-proselytizing religions (e.g. Orthodox Jews) who are the opposite. Zen Buddhism is a non-proselytizing religion, which makes me a bad Buddhist, because I've dragged three other people to my Zendo so far. All three had a great experience. One has become a regular, and another will return someday.

I didn't sell them on meditation. All three were already sold on meditation. One of them was a Sam Harris Waking Up fan, and another one is really into the Bhagavad Gita.

The Sam Harris fan's name is Rowan. Rowan is gay, and grew up in a rural evangelical Christian family. I haven't pressed him for details, but that can't have gone well. You may reasonably deduce that Rowan has a bad history with religion. But he has all the human instincts that respond positively to healthy rituals. Consequently, Rowan loves psudo-religious rituals and (until I brought him to Zendo) frequently complained to me that modern secular life doesn't have enough rituals.

Was this enough for me to drag him to Zendo? Nope.

What happened next is that Rowan and I were hanging out and he was talking to me about his therapist. Rowan's therapist had recommended that Rowan do some meditation. Rowan followed his therapist's meditation instructions.

Rowan tells me this not knowing that I attend a Rinzai Zen practice center 10 minutes from his home.

I ask Rowan if he'd like to visit my Rinzai Zen practice center.

Rowan asks me, "Why?"

I don't remember the exact words I used, but here is the gist of them. "The stuff your therapist is teaching you is derived from Asian mystical practices, except it has been watered down and purged of anything offending modern Western sensibilities. Your therapist is not a monk. Your therapist is not a yogi. I'm sure your therapist is a great therapist, but your therapist has not personally taken the mystic path to its logical and inevitable conclusion. Do you want the homeopathic version of meditation, or do you want the walk the road that Jesus, Mohammad and Siddhartha once did? Do you like your kombucha, or do you want to drink ethanol from a MiG-25 coolant system, like God intended?"

Rowan: "I am intrigued."

"Go to 1733 S Horton St on Tuesday at 7:30 pm. Do not be late. If you're even slightly late, turn around, go home, and come back next week. Bring $5 cash v...
Workshop: Interpretability in LLMs using Geometric and Statistical Methods
by Karthik Viswanathan — 2025-02-22T09:39:26.446Z

Dates: May 27-28
Location: Science Park 904, University of Amsterdam (How to get here?).
The workshop will also be streamed on this Zoom link.
Organizers: Jan Pieter van der Schaar (University of Amsterdam) and Karthik Viswanathan (University of Amsterdam)

We are excited to announce a two-day workshop on "Interpretability in LLMs using Geometrical and Statistical Methods" scheduled for May 27 and 28. We expect around 40 participants from Amsterdam, AREA Science Park (Trieste), and our invited speakers.

Image credit: Mechanistic Interpretability for AI Safety -- A Review

This workshop explores recent developments in understanding the inner workings of Large Language Models (LLMs) by using concepts from geometry and statistics. The workshop aims to provide an accessible introduction to these approaches, discussing their potential to address key challenges in AI safety and efficiency, while providing an overview of the current research problems in LLM interpretability. By bridging theoretical insights with practical applications, this workshop seeks to foster an exchange of ideas and motivate research at the intersection of computational geometry, statistical mechanics, and AI interpretability.

The workshop spans two days where the first day focuses on the geometric and statistical properties of internal representations in LLMs. The talks on this day are expected to have a physics-oriented perspective. On the second day, we aim to broaden the scope, covering topics in mechanistic interpretability like circuit analysis, analogical reasoning, competition between internal mechanisms, and feature superposition. For a schematic overview of the talks, check out this interactive visualization.[1]

Schedule

May 27: Geometric Methods for Interpretability

We will explore how large language models process and represent information through their internal representations. The discussions will focus on the geometry of embeddings - how they evolve across model layers and the insights they provide. The talks on Day 1 are expected to align with the themes discussed in this blogpost and paper. The talks on Day 1 will take place at the Faculty of Science, Room D1.111.

09:30 – 10:00: Welcome (Coffee and Opening Remarks)

Venue: SP D1.111

Time to grab coffee and say hi to fellow participants!

10:00 – 11:00: Valerie Castin, The Smoothness and Dynamics of Self Atten

...
Information throughput of biological humans and frontier LLMs
by benwr — 2025-02-22T07:15:45.457Z

Biological humans appear, across many domains, to have have an information throughput of at most about 50 bits per second. Naively multiplying this by the number of humans gives an upper bound of about 500 gigabits per second when considering the information throughput of humanity as a whole.

Current frontier LLMs collectively produce around 10 million tokens per second[1]; this translates to a collective output (and thus maximum throughput) of roughly 100 megabits per second.

These are both upper bounds, and so there's not much reason to directly compare them. I'm not sure exactly what to do with these numbers, though I think they're interesting, and this kind of thinking might in principle ultimately lead to more reasonable estimates of the strategic capacity of humanity and/or AI agents. For example, the concept of "empowerment" in reinforcement learning is expressed in terms of channel capacity.

^

I haven't carefully checked Deep Research's answer here, but it accords with my basic guess, based on looking at OpenRouter's weekly token chart.
Inefficiencies in Pharmaceutical Research Practices
by ErioirE — 2025-02-22T04:43:09.147Z

Epistemic status: I'm attempting to relate my observations, as well as some of those shared with me by my coworkers at the small CRO[1] where I work. I will try to make the distinction clear between object-level observations and speculative interpretations of the former.
The process of developing a new drug or medical device follows these approximate steps:

Novel Compound Phlebotinum is selected from a pool of candidate molecules[2] as most promising for clinical uses
In vitro experiments are performed in labs to confirm that the substance behaves as predicted
Assuming step 2 results were promising, animal trials are conducted in hopes that fruit flies/rats/pigs/monkeys will have enough biological similarity with us to provide meaningful information about how Phlebotinum might affect humans
Assuming step 3 passed, lots of paperwork is filed to assure officials that HealthCorp has performed the proper sacrifices Phlebotinum has enough evidence to be ethically tested on humans
Assuming success, smaller scale human trials (Phase 1 & 2) are conducted to check for obvious safety issues and if the treatment condition seems to respond as expected

This is where the problems begin to appear.[3]
One of the biggest can be summarized as "lack of data standardization".
To be clear, standards do exist, and in fact the Regulatory Authorities (by whose evaluation your new treatment will succeed or fail[4]) require the data they receive to conform to these standards. But despite this, a disturbingly common attitude of many of those companies running studies has been "just get the data collected, we'll let the CRO format/standardize the data later".
Some attitudes with comparable failure modes might be:
Software developers: "Just keep coding, we can fix it later" (about a fundamental design flaw in a foundational piece of code).
Amateur bakers: "No need to follow the recipe, cake is super easy! We'll just cover any problems with frosting later."
Rocket scientists: "The standards are really just guidelines, we'll do whatever we have the time for. The astronauts are smart, we can fix any problems as they come up."[5]
The intuitive heuristic is that generally, doing something correctly from the beginning will be far more efficient than doing it wrong and having to backtrack later to fix it. This is well demonstrated in collecting data for analysis.
Here is an example of pre...
Build a Metaculus Forecasting Bot in 30 Minutes: A Practical Guide
by ChristianWilliams — 2025-02-22T03:52:14.753Z

We've created a video tutorial showing how to build a forecasting bot using our template in about 30 minutes.

The template bot consistently ranks in the top 10 of Metaculus's AI Forecasting Benchmark Series using a basic GPT-4/AskNews implementation.

Setup for a basic bot requires approximately 30 minutes, but note that the #1 finishing bot from Q4 took only 15-40 hours total (1-3 hours weekly), which yielded $9,500 or $240/hour.

The Q1 prize pool is $30,000, and there's still plenty of time to compete, as bots begin with a relative peer score of 0, which would place one in the top 25 for Q4. Email support@metaculus if you want to request API credits and help improve benchmarking of key AI capabilities.
Intelligence–Agency Equivalence ≈ Mass–Energy Equivalence: On Static Nature of Intelligence & Physicalization of Ethics 
by ank — 2025-02-22T00:12:36.263Z

Imagine a place that grants any wish, but there is no catch, it shows you all the outcomes, too.

(This is the result of three years of thinking and modeling hyper‑futuristic and current ethical systems. It's not the first post in the series, it'll be very confusing and probably understood wrong without reading at least the first one. Everything described here can be modeled mathematically—it’s essentially geometry. I take as an axiom that every agent in the multiverse experiences real pain and pleasure. Sorry for the rough edges—I’m a newcomer, non‑native speaker, and my ideas might sound strange, so please steelman them and share your thoughts. My sole goal is to decrease the probability of a permanent dystopia. I’m a proponent of direct democracies and new technologies being a choice, not an enforcement upon us.)

In this article, I will argue that intelligence is by its nature static and space-like, and that creating agentic AI, AGI, or ASI is both redundant and dangerous.

What is a language model?

In a nutshell, a language model is a collection of words connected by vectors; it's a geometric static shape—space-like in nature. GPUs make it time-like by calculating paths through this space. This raises the question: why not expose the contents of the model to people? I claim that we can make this space walkable and observable from the inside and outside. But how do we even begin?

Encoding Space & Time into a Walkable Space

One way to conceptualize this is by encoding long stretches of time into 3D walkable spaces, similar to how we create long-exposure photos. For example, here is a year of time in Berlin compressed into a photo and more photos like this[1]. You can see car lights and the Sun—gray lines represent cloudy days when there was no sunlight.

This is an abstraction of how we can visualize all knowledge from the Big Bang to the ultimate end—perhaps call it a multiverse, a space of all-knowing. I argue that this is the final state of the best possible evolution, because static shapes can't feel pain. Therefore, drawing the sculpture of everything, of all knowledge, is both safe and can eventually be democratically agreed upon. We became enlightened enough to make sculptures of naked people long ago, let women go topless, so I believe one day we will consider it okay.

So, why do we need agentic AIs? Instead, we could remain the only agents ourselves, exp...
Alignment can be the ‘clean energy’ of AI
by Cameron Berg — 2025-02-22T00:08:30.391Z

Not all that long ago, the idea of advanced AI in Washington, DC seemed like a nonstarter. Policymakers treated it as weird sci‐fi-esque overreach/just another Big Tech Thing. Yet, in our experience over the last month, recent high-profile developments—most notably, DeepSeek's release of R1 and the $500B Stargate announcement—have shifted the Overton window significantly.

For the first time, DC policy circles are genuinely grappling with advanced AI as a concrete reality rather than a distant possibility. However, this newfound attention has also brought uncertainty: policymakers are actively searching for politically viable approaches to AI governance, but many are increasingly wary of what they see as excessive focus on safety at the expense of innovation and competitiveness. Most notably at the recent Paris summit, JD Vance explicitly moved to pivot the narrative from "AI safety" to "AI opportunity"—a shift that the current administration’s AI czar David Sacks praised as a "bracing" break from previous safety-focused gatherings.

Sacks positions himself as a "techno-realist," gravitating away from both extremes of certain doom and unchecked optimism. We think this is an overall-sensible strategic perspective for now—and also recognize that halting or slowing AI development at this point would, as Sacks puts it, “[be] like ordering the tides to stop.”[1] The pragmatic question at this stage isn't whether to develop AI, but how to guide its development responsibly while maintaining competitiveness. Along these lines, we see a crucial parallel that's often overlooked in the current debate: alignment research, rather than being a drain on model competitiveness, is likely actually key to maintaining a competitive edge.

Some policymakers and investors hear "safety" and immediately imagine compliance overhead, slowdowns, regulatory capture, and ceded market share. The idea of an "alignment tax" is not new—many have long argued that prioritizing reliability and guardrails means losing out to the fastest (likely-safety-agnostic) mover. But key evidence continues to emerge that alignment techniques can enhance capabilities rather than hinder them (some strong recent examples are documented in the collapsible section below).[2]

This dynamic—where supposedly idealistic constraints reveal themselves as competitive advantages—would not be unique to AI. ...
The Sorry State of AI X-Risk Advocacy, and Thoughts on Doing Better
by Thane Ruthenis — 2025-02-21T20:15:11.545Z

First, let me quote my previous ancient post on the topic:

Effective Strategies for Changing Public Opinion

The titular paper is very relevant here. I'll summarize a few points.

The main two forms of intervention are persuasion and framing.
Persuasion is, to wit, an attempt to change someone's set of beliefs, either by introducing new ones or by changing existing ones.
Framing is a more subtle form: an attempt to change the relative weights of someone's beliefs, by empathizing different aspects of the situation, recontextualizing it.
There's a dichotomy between the two. Persuasion is found to be very ineffective if used on someone with high domain knowledge. Framing-style arguments, on the other hand, are more effective the more the recipient knows about the topic.
Thus, persuasion is better used on non-specialists, and it's most advantageous the first time it's used. If someone tries it and fails, they raise the recipient's domain knowledge, and the second persuasion attempt would be correspondingly hampered. Cached thoughts are also in effect.
Framing, conversely, is better for specialists.

My sense is that, up to this point, AI risk advocacy targeted the following groups of people:

ML researchers and academics, who want "scientifically supported" arguments.

Advocacy methods: theory-based arguments, various proof-of-concept empirical evidence of misalignment, model organisms, et cetera.

US policymakers, who want either popular support or expert support to champion a given cause.

Advocacy methods: behind-the-scenes elbow-rubbing, polls showing bipartisan concern for AI, parading around the experts concerned about AI.

Random Internet people with interests or expertise in the area.

Advocacy methods: viral LW/Xitter blog posts laying out AI X-risk arguments.

Persuasion

I think all of the above demographics aren't worth trying to persuade further at this point in time. It was very productive before, when they didn't yet have high domain knowledge related to AI Risk specifically, and there's been some major wins.

But further work in this space (and therefore work on all corresponding advocacy methods, yes) is likely to have ~no value.

~All ML researchers and academics that care have already made up their mind regarding whether they prefer to believe in misalignment risks or not. Additional scary papers and demos aren't going to make anyone budge.
The relevant parts of the USG are m

...
ParaScopes: Do Language Models Plan the Upcoming Paragraph?
by NickyP — 2025-02-21T16:50:20.745Z

This work is a continuation of work in a workshop paper: Extracting Paragraphs from LLM Token Activations, and based on continuous research into my main research agenda: Modelling Trajectories of Language Models. See the GitHub repository for code additional details.

Looking at the path directly in front of the LLM Black Box.

Short Version (5 minute version)

I've been trying to understand how Language models "plan", in particular what they're going to write. I propose the idea of Residual Stream Decoders, and in particular, "ParaScopes" to understand if a language model might be scoping out the upcoming paragraph within their residual stream.

I find some evidence that a couple of relatively basic methods can sometimes find what the upcoming outputs might look like. The evidence for "explicit planning" seems weak in Llama 3B, but there is some relatively strong evidence of "implicit steering" in the form of "knowing what the immediate future might look like". Additionally current attempts have room for improvement.

Motivation.

When a Language Model (LM) is writing things, it would be good to know if the Language Model is planning for any kind of specific output. While the domain of text with current LMs is unlikely to be that risky, they will likely be used to do tasks of increasing complexity, and may do things that may increasingly be described as "goal-directedness" by some senses of the word.

As a weaker claim, I simply say that LMs are likely steering, either implicitly or explicitly, towards some kinds of outputs. For some more discussion on this, read my posts on Ideation and Trajectory Modelling in Language Models.

We say that the language model is steering explicitly towards certain outcomes, if the model is internally representing some ideas of what longer-term states should look like. If given the same input prompt and context, the model should reliably try to steer towards some kinds out outputs, even if perturbed.

We say the language model is steering implicitly, if the language model does NOT have an explicit representation of what the end states might look like, but is rather following well-learned formulas to go from the current state to the next. This means that the model has some distribution of output states that are likely, but if perturbed, may go down some other path instead.

My original plan before I reached these results, was that it would...
Linguistic Imperialism in AI: Enforcing Human-Readable Chain-of-Thought
by Lukas Petersson — 2025-02-21T15:45:00.146Z
Link: https://lukaspetersson.com/blog/2025/ban-ls-cot/

Revisiting AI Doom Scenarios

Traditional AI doom scenarios usually assumed AI would inherently come with agency and goals. This seemed likely back when AlphaGo and other reinforcement learning (RL) systems were the most powerful AIs. When large language models (LLMs) finally brought powerful AI capabilities, these scenarios didn't quite fit: LLMs simply predict likely text continuations based on their training data, without pursuing any objectives of their own.

But we are now starting to go back to our RL roots. Models like OpenAI’s o1/o3 and Deepseek’s R1 show that we have now entered the era. The classic doomsday example is the “drive over the baby” scenario: You ask your robot for a cup of tea and the robot (who has been trained with RL to make tea as fast as possible) plows through a toddler in pursuit of optimizing for its goal - make tea fast. A robot trained without RL in a supervised manner (like LLMs next token prediction) would never do this because they have never seen a human do it.

RL trained LLMs are still LLMs though - their output is natural text. Surely we could build systems to catch bad behaviour before they are acted upon? Unfortunately, it seems like the model’s internal monologue will not be in English for much longer. Research results show that models become smarter if you don’t constrain them to think in human interpretable languages.

Being able to interpret the models’ internal monologue seems extremely good for AI safety. So a question arises, should we make it illegal to develop models this way? That’s the big question at the center of what I half-jokingly call “linguistic imperialism in AI”. And even if we want to, is it possible to enforce? Let’s think about this step by step.

Why Chain-of-Thought

A year or two ago, researchers discovered that if you ask a large language model to “think step by step,” it often yields better answers—especially for math, logic, or any multi-step task. Instead of spitting out a quick guess, the model has an internal monologue where it can break the problem down into smaller pieces. This Chain-of-Thought (CoT) strategy worked so well on almost everything that the “think step by step” prompt is put in the system prompt on models by default.

The best part? It was all in English (or another natural language). You can skim the chain-of-thought and verify each line. That interpretability made us feel safe. If...