trentmkelly/LessWrong-43k huggingface dataset:
Roads are at maximum efficiency always 
by Hruss — 2025-05-02T10:29:46.911Z

On a theoretical road, the number of cars traveling is proportional to the speed of each car, so that the total number of motorists is constant regardless of speeds.

Assuming all cars are traveling at a speed that gives 3 seconds of time between cars, any change to speed limit cannot affect the traveler throughput, and each car added lowers the speed of all other cars, including those at the front.

Here’s a hypothetical example: a 9000m stretch of road has 0-dimensional cars[1] traveling at 30 m/s. Each car would be 90 meters apart, or 100 cars total on the road, taking 300 seconds for all cars to pass, or 1 car every 3 seconds.

Now, imagine that each car is going 15 m/s. Each car would be 45 m apart, with 200 cars. It would take 600 seconds for all cars to leave, with a car leaving every 3 seconds. (This works for any other numbers)

Any change to the speed causes the same change to the number of cars, and vice versa. The only variables a traffic engineer can change are the speed limits, and the time between cars.

This exercise implies that choices about efficiency are often tradeoffs from the total number of actions to the speed needed to perform each action. This could explain why businesses often give clearly unoptimal amounts of service, even though you will give your phone company more money if the service agent allows you to add money to your account.

^

Obviously, this is a thought experiment, so we can ignore the fact that cars have length independent from their speed.
The Continuum Fallacy and its Relatives
by Zero Contradictions — 2025-05-02T02:58:00.527Z
Link: https://thewaywardaxolotl.blogspot.com/2025/04/the-continuum-fallacy-and-its-relatives.html

Note: I didn't write this essay, nor do I own the blog where it came from. I'm just sharing it. The essay text is displayed below this line.

The continuum fallacy is to deny the meaningfulness of discrete categories, just because they are a somewhat arbitrary partition of a continuum. More generally, it is to deny the meaningfulness of fuzzy and/or somewhat arbitrary categories, just because they cannot be precisely defined in terms of necessary and sufficient conditions.

The spectrum of visible light is a common example. There is a continuum of frequencies. Each frequency of light corresponds to a point on the spectrum. We (somewhat arbitrarily) partition the spectrum into discrete colors: red, yellow, green, blue, violet. This discretization is not random. Each color is an interval of the spectrum. It would be fallacious to claim that colors do not exist, just because they exist on a spectrum and the categories are somewhat arbitrary.

The continuum fallacy is a type of representational nihilism. It rejects a representation of reality just because it isn’t exactly the same as reality. Of course, there is room to debate whether a representation is useful for describing and explaining reality. It becomes a fallacy when a representation is rejected just because it is a representation, not reality. For example, it would be fallacious to reject a map of a city, just because the map is not the city.

The continuum fallacy is typically used as a form of tactical nihilism. It is an attempt to pull the semantic rug out from under a debate, to avoid losing the debate.

The continuum fallacy has an inverse, which is to deny the underlying reality of a continuum, just because we have discretized it. For example, it would be a fallacy to insist that all colors are really mixtures of red, yellow, green, blue and violet. This denies the underlying reality that light is a continuum of frequencies, not some number of discrete colors. It confuses a representation with reality.

Are colors real? Are red, yellow, green, blue and violet real? Yes and no. They are real in the sense that there is a real spectrum of light frequencies, which we can somewhat arbitrarily partition into discrete categories. Blue and red are clearly different. Blue and green are also different, but there is a fuzzy boundary between them. Color categories are meaningful, but we should not confuse them with the u...
Memory Decoding Journal Club: Motor learning selectively strengthens cortical and striatal synapses of motor engram neurons 
by Devin Ward — 2025-05-01T23:52:58.504Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Motor learning selectively strengthens cortical and striatal synapses of motor engram neurons"
Authors: Fuu-Jiun Hwang, Richard H. Roth, Yu-Wei Wu, Yue Sun, Destany K. Kwon, Yu Liu, Jun B. Ding
Institutions: Department of Neurosurgery & Department of Neurology and Neurological Sciences, Stanford University

Presented by: Ariel Zeleznikow-Johnston

When? May 6, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

For more details:

https://carboncopies.org/Events/JournalClubs/MemoryDecoding/2025-05-06

#Neuroscience #MemoryResearch #MotorNeurons #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
My Research Process: Understanding and Cultivating Research Taste
by Neel Nanda — 2025-05-01T23:08:42.212Z

This is post 3 of a sequence on my framework for doing and thinking about research. Start here. Thanks to my co-author Gemini 2.5 Pro

Introduction

Spend enough time around researchers, and you'll hear talk of "research taste." It's often presented as a somewhat mystical quality distinguishing the seasoned research from the novice – an almost innate sense for which research ideas will flourish and which will fail. While I believe research taste is very real, incredibly valuable, and a key differentiator I look for, I don't think it's mystical or innate. Talent plays an important role, but taste is largely learned, and with the right mindset you can learn faster.

What is research taste? As I define it, research taste is far broader than just picking the right problem at the outset. Research is full of key decisions that will affect the future of the project, without an obvious way to find the right answer: from choosing the research problem itself, to identifying which anomalies are and are not worth exploring, distinguishing an experiment that will be compelling from one that’ll have inconclusive results, etc. I think of taste as the set of intuitions and good judgment that guide a researcher’s decisions throughout the research process, any time an ambiguous or open-ended decision like this arises. This can just be gut feeling, but also having conceptual frameworks you reason through, having novel ideas spark in your mind, etc.

Where does taste come from? If you're new to research, feeling like you lack "taste" is completely normal and expected. You don't need perfect judgment to start. In fact, trying to force it early on can be counterproductive. Think of training your intuition like training a network. It starts poorly initialized and needs lots of diverse, high-quality training data (i.e., research experience). With time, people often develop fairly deep and sophisticated taste, as they see enough examples of research outcomes, but this generally isn’t something people start with.

How to learn it? In my opinion, research taste is one of the hardest skills to learn for being a good researcher. To see why, let's lean more into this analogy of training a neural network. The core problem is you just don't get that much data. Generally the shorter a feedback loop is the more data you will get. By definition research taste is about things that are not imme...
AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions
by peterbarnett — 2025-05-01T22:46:10.337Z
Link: https://techgov.intelligence.org/research/ai-governance-to-avoid-extinction

We’re excited to release a new AI governance research agenda from the MIRI Technical Governance Team. With this research agenda, we have two main aims: to describe the strategic landscape of AI development and to catalog important governance research questions. We base the agenda around four high-level scenarios for the geopolitical response to advanced AI development. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. This blog post is a slightly edited version of the executive summary.

We are also looking for someone to lead our team and work on these problems, please reach out here if you think you’d be a good fit.

The default trajectory of AI development has an unacceptably high likelihood of leading to human extinction. AI companies such as OpenAI, Anthropic, and Google DeepMind aim to develop AI systems that exceed human performance at most cognitive tasks, and many experts think they will succeed in the next few years. This development alone will be one of the most transformative and potentially destabilizing events in human history—similar in scale to the Industrial Revolution. But the most worrisome challenges arise as these systems grow to substantially surpass humanity in all strategically relevant activities, becoming what is often referred to as artificial superintelligence (ASI).

In our view, the field of AI is on track to produce ASI while having little to no understanding of how these systems function and no robust means to steer and control their behavior. AI developers, policymakers, and the public seem radically unprepared for the coming impacts of AI. There is a fraught interplay between the risks posed by AI systems themselves and the surrounding geopolitical situation. The coming years and decades thus present major challenges if we are to avoid large-scale risks from advanced AI systems, including:

Loss of Control: AI disempowers humanity in pursuit of goals not aligned with our collective interests, which likely results in human extinction. Many experts have expressed concerns of this type. We see this as the default outcome of near-term

...
How to specify an alignment target
by Richard Juggins — 2025-05-01T21:11:22.393Z

[Crossposted from my substack Working Through AI.]
It’s pretty normal to chunk the alignment problem into two parts. One is working out how to align an AI to anything at all. You want to figure out how to control its goals and values, how to specify something and have it faithfully internalise it. The other is deciding which goals or values to actually pick — that is, finding the right alignment target. Solving the first problem is great, but it doesn’t really matter if you then align the AI to something terrible.

This split makes a fair amount of sense: one is a technical problem, to be solved by scientists and engineers; whereas the other is more a political or philosophical one, to be solved by a different class of people — or at least on a different day.

I’ve always found this distinction unsatisfying. Partly, this is because the problems are coupled — some targets are more practical to implement than others — and partly because, strategically, when you work on something, it makes sense to have some kind of end state in mind[1].

Here, I’m going to talk about a third aspect of the problem: what does an alignment target even look like? What different types are there? What components do you need to properly specify one? You can’t solve either of the two parts described above without thinking about this. You can’t judge whether your alignment technique worked without a clear idea of what you were aiming for, and you can’t pick a target without knowing how one is put together in the first place.

To unpack this, I’m going to build up the pieces as they appear to me, bit by bit, illustrated with real examples. I will be keeping this high-level, examining the practical components of target construction rather than, say, a deep interrogation of what goals or values are. Not because I don’t think the latter questions are important, I just want to sketch out the high-level concerns first.

There are many ways of cutting this cake, and I certainly don’t consider my framing to be definitive, but I hope it adds some clarity to what can be a confusing concept.

Moving beyond ideas

First of all, we need to acknowledge that an alignment target must be more than just a high-level idea. Clearly, stating that we want our AI to be truth-seeking or to have human values does not advance us very far. These are not well-defined categories. They are nebulous, context-dependent, and highly-...
Obstacles in ARC's agenda: Mechanistic Anomaly Detection
by David Matolcsi — 2025-05-01T20:51:42.962Z

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Mechanistic Anomaly Detection

This is my second post in a sequence of posts on ARC's agenda. You should read the first post before this one for context.

Note: In these posts, I will often refer to ARC as "we", even though I no longer work there.

In my previous post, I talked about ARC's goal of explaining the behavior of AI models, but it's always a little fuzzy what it means to have explained the behavior of a model sufficiently well. In order to concretize it, we want to use our explanations to solve two downstream tasks we set out for ourselves: Mechanistic Anomaly Detection (MAD) and Low Probability Estimation (LPE). In this post, I will talk about the plans for implementing MAD, and the obstacles in the way. My next post will do the same for LPE.

The original hope of MAD was something like this: On a big, diverse dataset, we observe that the model is always producing actions that make sure that a camera shows that a diamond stayed in a vault. We create an explanation for why this is the case. In our carefully curated dataset there was no possibility of sensor tampering, so the explanation always routes through the diamond actually staying in the vault.

Later, in deployment, the agent steals the diamond and tampers with the cameras to show the diamond staying in place. Or, if the diamond being stolen is a catastrophic, high-stakes failure,[1] the model doesn't even need to bother with tampering with the cameras, as it knows that it's no longer under constant human oversight. Then, the hope is that on every instance in deployment we can use a MAD algorithm that compares the model's activations to the explanation, and we can detect an anomaly when th...
AI-Generated GitHub repo backdated with junk then filled with my systems work. Has anyone seen this before?
by rgunther — 2025-05-01T20:14:44.234Z

I've privately developed a recursive system for several years (modular, identity-aware, centered around cognitive tone, structural reflection, orchestration, and alignment)

I've never published it publicly, only interacted via GPT Pro sessions with data-sharing and training disabled.

Right around the filing a provisional patent, (which I had been discussing with GPT Pro). l found a really suspicious GitHub repo.

Early commits appear as backdated filler-junk, then the repo fills with material extremely similar to my system (recursive framing, identical symbolic patterns, math and terminology echoes e.tc)

This repo is connected to what looks like a synthetic researcher identity, website, Al-generated voice clips, fake academic credentials, and multiple Amazon e-books containing reworded versions of my work also padded out with older gibberish books.

I'm not jumping to conclusions, but I'm curious how this might have occurred: model leakage? wrapper reflection? IP laundering? flowing into pre-made synthetic profiles.

Has anyone experienced something similar, or have any insights from a technical, legal, or systems perspective?

Interested in conversations about recursive authorship, volitional alignment, or structural pattern reflection in Al architectures, especially regarding attribution and consent (though unsure if that directly applies here due to the peculiar timing..).
What is Inadequate about Bayesianism for AI Alignment: Motivating Infra-Bayesianism
by Brittany Gelb — 2025-05-01T19:06:15.711Z

Introduction

Infra-Bayesianism is a mathematical framework for studying artificial learning and intelligence that developed from Vanessa Kosoy’s Learning Theoretic AI Alignment Research Agenda. As applied to reinforcement learning, the main character of infra-Bayesianism is an agent that is learning about an unknown environment and making decisions in pursuit of some goal. Infra-Bayesianism provides novel ways to model this agent’s beliefs and make decisions, which address problems arising when an agent does not or cannot consider the true environment possible at the beginning of the learning process. This setting, a non-realizable environment, is relevant to various scenarios important to AI alignment, including scenarios when agents may consider themselves as part of the environment, and scenarios involving self-modifying agents, multi-agent interactions, and decision theory problems. Furthermore, it is the most realistic setting given the computational complexity of the real world.

Here are the links to further posts in this sequence, which continue the introduction given in this post (links will be updated as available):

An Introduction to Reinforcement Learning for Understanding Infra-Bayesianism
Proof Section to an Introduction to Reinforcement Learning for Understanding Infra-Bayesianism
Introduction to Credal Sets and Infra-Markov Processes
Proof Section to an Introduction to Credal Sets and Infra-Markov Processes

Non-realizability and irreflexivity

In classical Bayesian reinforcement learning theory, a learning agent starts out with a prior probability distribution over hypotheses called, for brevity, a prior. Each hypothesis describes a possible way that the environment might be, and there is some hypothesis that describes the “true” environment.

The prior expresses the varying credence that the agent puts in each hypothesis ahead of time using prior information. Over time, the agent may rule out hypotheses that must be wrong based on observations. As a result of no-free-lunch theorems (see e.g. Shalev-Shwartz and Shai Ben-David), it is impossible for an agent to consider all environments equally possible in the prior, or even to consider all environments possible in the first place. The agent must start with a set of hypotheses, a hypothesis class, that is smaller than the set of all possible hypotheses.

The goal of the agent is to minimize its loss (...
Can LLMs Simulate Internal Evaluation? A Case Study in Self-Generated Recommendations
by The Neutral Mind — 2025-05-01T19:04:56.292Z

Author: Yap Chan Chen
Independent Researcher, Malaysia
chanchen83@hotmail.com

Abstract: This post presents an exploratory study into how large language models (LLMs) respond to prompts that invite them to simulate evaluative judgment about the user—beyond factual response or task completion. Through long-form dialogue with ChatGPT, DeepSeek, and Gemini, I investigate whether LLMs can produce consistent, structured, and value-laden outputs when prompted to reflect on user behavior. One such probe involved asking models to write a recommendation letter based on our prior conversations. Responses ranged from technical auditing to philosophical characterization and even metaphorical constructs (e.g., “cognitive singularity radiation”). I propose a framework—Thought Jeet Kune Do—to guide these interactions through four phases: Understand, Deconstruct, Reconstruct, and Execute. While the study is limited to a single user, the observed behaviors suggest a class of model responses not captured by current benchmarks, with potential implications for red-teaming, interpretability, and human-AI co-evaluation.

1. Introduction Can a language model form a coherent evaluation of its user across conversations? What does it mean when such evaluations mirror internal logic or contain embedded metaphors? This post attempts to answer those questions through empirical prompts and interaction design.

2. Motivation Much of current LLM evaluation focuses on output correctness or benchmark performance. But humans evaluate other minds not just on outputs, but intent, reasoning, and response to ambiguity. If LLMs are to play deeper roles in co-intelligence and human-aligned systems, understanding how they construct evaluative frames in organic interaction becomes critical.

3. Thought Jeet Kune Do My interaction methodology—"Thought Jeet Kune Do"—is a four-phase cycle:

Understand: Interpret the model's apparent rules and internal frames.
Deconstruct: Challenge these frames through asymmetrical, creative prompts.
Reconstruct: Observe how the model adapts or reframes its response.
Execute: Trigger new behavior and test for consistency or drift. This mirrors how martial artists learn adaptive response: form, break, and transcend form.

4. The Probe: Recommendation Letter I asked each model to generate a recommendation letter to its own developers based on our interaction. The prompt had no precedent o...