trentmkelly/LessWrong-43k huggingface dataset:
Keeping Capital is the Challenge
by LTM — 2025-02-03T02:04:27.142Z
Link: https://routecause.substack.com/p/keeping-capital-is-the-challenge

Capital will likely matter more post-AGI. However, the capital you have been able to raise up to this point and may be able to acquire throughout the maturation of AGI will only retain its value if reliable information remains available.

Historically, investing has been grounded in the idea that stashing capital away in some enterprise, and accepting the risk associated, will let you grow your money over time. By accepting that you will not be able to access that capital for a period, and that it could wind up worthless, you will (hopefully) be left with more than you started with.

This idea has enjoyed immense success for centuries across Europe, North America, and other regions with a developed financial system. Famously the past few decades have given extremely positive outcomes to just about anybody who engages sensibly and invests long-term. The practice of patiently enduring risk in hopes of a payout has become a core feature of many people’s lives, and is the way in which they (somewhat dubiously) expect to keep themselves afloat in their old age.

The

slow upwards march

of equity prices is one of the most robust observations in finance. The general public’s ability to capitalise on this by investing has traditionally been maintained by three fundamental features of public markets.

1 - The value of capital tends to rise: If you make an investment in a business, its value will probably drift upwards with time.

2 - You will have sufficient information to make good decisions: Knowing what to buy and when to buy it, as well as when to sell, is necessary to capture the increase in value.

3 - Your stake will be easy to defend: If someone steals your shares, denies you voting rights, transfers the assets of a company under their own name etc. the law will step in and stop them. Better yet - if you are investing in a sufficiently large company, an immense financial institution will even come and do this for you!

And so, we spend much of our lives selling our labour for capital, which we can then exchange for what we need when we can no longer sell our labour.

The emergence of powerful AI, the kind which could enormously accelerate human productivity or supplant it entirely, threatens this life long economic planning. Our economy rests of a foundation of businesses selling products to humans who in turn sell their labour to businesses. But with powerful AI, the number ...
Use computers as powerful as in 1985 or AI controls humans or ?
by jrincayc — 2025-02-03T00:51:05.706Z

A way to prevent AGI from taking over or destroying humanity is to strictly limit the computing power used on unknown AI algorithms. My back of the envelope calculations[1] show that restricting the hardware to 64 KiB of total storage is definitely sufficient to prevent an independence gaining AGI, and restricting to 2 MiB of storage is very likely to be sufficient to prevent an indepence gaining AGI. State of the art AI on the other hand tends to be using at least 1 GiB of RAM or much more and processing power in the teraflops range or more. As for an upper limit before we get AGI, whole brain emulation provides one, but that is on the order of 1 exoflops and 1 petabyte, so we do not have a precise idea of where the limits for AGI are. Also, we don't have a way to make sure that AI software is aligned with human goals and ethics.[2]

So here are options:

Only use really weak computers (midrange 1985 computers like a Mac 512K or an Atari 520ST would almost certainly be safe)

Just let AGI take control.

Hope that AGI really requires very powerful computers and ban them but allow less powerful computers that are well above what we are sure cannot be an AGI.

Hope there is outside intervention that prevents dangerous AGI (space aliens, divine intervention, dark lords of the matrix, etc.)

???

So what should humanity do? I talked to a non-computer scientist about this, and his answer was that restricting us to circa 1985 power of computers was the best choice, which actually surprised me a little. Letting AGI take control can result in extinction, or the AGI imposing rules that we don't like.[3]

The problem is that we are metaphorically experimenting with 15 kiloVolt AC when we really should be experimenting with 5 volt DC because we have a very weak understanding of AI safety.

I don't know what humanity should do. As for me personally, if I had the choice between my current 1.6 GHz 4 core CPU with 24 GB of RAM computer that I am typing on, versus living in a world where we had eliminated existential risk from things like uncontrolled AGI and nuclear bombs, I would gladly trade my computer in for a 512 KB, 8 MHz computer with a floppy drive and a CD-R and an modem level network connection if that is what we all need to do. I am curious what others think.

*These are my own opinions and not those of my employer. This
document may be distributed verbatim in any media. ...
Some Theses on Motivational and Directional Feedback
by abstractapplic — 2025-02-02T22:50:04.270Z

Note: Probably reinventing the wheel here. Heavily skewed to my areas of interest. Your results may vary.

Feedback can be decomposed into motivational ("Keep doing what you're doing!") and directional ("Here's what you should do differently . . .").

There's also material support ("I like your work, please accept this job offer / some money / my hand in marriage / etc."), which isn't really a type of feedback; I mention it because it can come packaged with the other two or be mistaken for them. (Consider a famous director whose movie gets a rave review from a popular critic: they might not derive any more encouragement from this than their fans already give them, and they're unlikely to let the content of the review change the way they shoot their next movie, but will still appreciate it because of the extra tickets it causes people to buy.)
Demotivational feedback ("This isn't working, give up!") exists, but I'd say that's just motivational feedback with a flipped sign.

People don't reliably self-report or consistently scale; this is particularly important for motivational feedback. "This is great!" might mean the commenter really enjoyed something, or they're a perennially polite and/or positive person who didn't actively dislike it. One way to handle this is with the human-brain-equivalent of proof-of-work computing: a long, thoughtful comment will have more of an impact than a single sentence - even if they're saying the same things for the same reasons - and fan[art|fiction|whatever] provides further evidence.

Visibly voluntarily consuming a work is a (weak, valid) form of feedback. And if a work is more difficult to consume, that makes this signal stronger: "I read the whole book" is more meaningful than "I watched the whole video", etc.[1]
"I just subscribed to your Patreon!" is - in addition to the obvious material effects - another form of hard evidence that what you're doing matters.

From a purely directional view, "I loved your game, especially level 7!", "7th level best level, you should make the entire game like that", "just fyi level 7 was more fun than the rest of them put together" and "Your game was terrible, except for level 7, which was merely bad." are all effectively the same review.

This isn't quite true. If you're making something which someone might assign as required reading (or equivalent), it behooves you to prioritize making the worst user

...
Exploring how OthelloGPT computes its world model
by JMaar — 2025-02-02T21:29:09.433Z

I completed this project for my bachelor's thesis and am now writing it up 2-3 months later. I think I found some interesting results that are worth sharing here. This post might be especially interesting for people who try to reverse-engineer OthelloGPT in the future.

Summary

I suggest the Previous Color Circuit, which explains how the Attention layers in OthelloGPT copy over the color of each tile from previous moves. I used the word Circuit here, because it is common, but I think it’s more like a rough approximation of what’s going on.
I came up with and disproved a theory about how the MLP-layers in OthelloGPT compute which tiles are flipped.

I think this might provide mild evidence against the theory proposed in OthelloGPT learned a bag of heuristics that individual neurons perform interpretable rules. Although I only tested a very specific set of rules.

What’s OthelloGPT

OthelloGPT learned a bag of heuristics

gives a good introduction to OthelloGPT

Small findings / Prerequisites

The other sections will build on top of this section

Mine-Heads and Yours-Heads

Hazineh et. al.

found that some Attention Heads only pay attention to moves made by the current player (Mine-Heads) and others only pay attention to moves made by the other player (Yours-Heads)
Most Attention Heads are Mine- or Yours-Heads

Average attention paid to positions an even number of steps away minus attention paid to positions an odd number of steps away, for each Attention Head and layer. Mine Heads are seen in blue, and Yours Heads in read. "Last," "First," and other types of heads are also visible. L4H5, for example, is a "Last" HeadMost Attention Heads are Mine- or Yours-Heads

Attention Patterns are almost constant (across inputs)

The attention pattern has a Variance of 0.0007 averaged over each Attention Head over 200 games
They typically pay less attention to moves farther away

Attention Pattern for Layer 3, Head 0, Position 2

Visualising the board state Accuracy over every Layer and sequence position

Accuracy of the Linear Probe Across Layers and Sequence Positions (ignoring empty tiles)

The above plot shows that in each layer, the board state is very accurate until some sequence position, where it starts to drop off.

The Flipped probe

Andrew Lee found a "Tile is Flipped" probe, that tracks if a tile has been flipped this move. This is relevant for the next section.

The Previous Color Circuit

I hypothesiz

...
An Introduction to Evidential Decision Theory
by Babić — 2025-02-02T21:27:35.684Z

Introduction

This blog post serves as an introduction to Evidential Decision Theory (EDT), designed for readers with no prior background. While much has been written about EDT’s application to well-known problems like the Smoking Lesion problem, and Newcomb-like problems, we are unable to find a clear and formal introduction. As such, this post aims to fill that gap.

Our motivation for writing this post stems from the work of Nate Soares and Benja Fallenstein in their papers, Towards Idealized Decision Theory (2015) [8] and Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda (2015) [9]. These works suggest the need for the study of decision theory in aligning smarter-than-human agents with human interests and address the limitations of existing formulations, such as EDT and Causal Decision Theory (CDT) in defining idealized decision procedures.

This post does not take a stance on the need for an idealized decision procedure or the tractability of one. Rather, we hope it provides a clear introduction to EDT and perhaps encourages readers to explore these topics further.

Decision Theory

Decision theory is a rich interdisciplinary field in which philosophers, psychologists, economists, computer scientists, statisticians and mathematicians aptly study decision. While decision theorists disagree on a lot, they do share some foundational distinctions. An important distinction is between normative and descriptive decision theory.
Descriptive decision theory concerns how people actually make decisions.
Normative decision theory, on the other hand, concerns how agents ought to make decisions. We will focus on the latter.

EDT is a branch of normative decision theory first introduced by Richard C. Jeffrey in The Logic of Decision (1965) [4]. In this work, Jeffrey argues that an agent should evaluate actions based on their evidential implications—specifically, one should choose the action with the highest news value. In the following sections, we will unpack exactly what this means, drawing on both Jeffrey’s work and Ahmed’s modern analysis in Evidential Decision Theory (2021) [1].

Decision Problems

To take a step back, let's formalize what we mean by an agent making a decision.
For the remainder of this report, we will primarily follow Jeffery's formalization of decision problems.[1] In this framework, an agent is a deci...
"DL training == human learning" is a bad analogy
by kman — 2025-02-02T20:59:21.259Z

A more correct but less concise statement of the analogy might be DL training : DL training code :: human learning : human genome, read as "DL training is to DL training code what human learning is to the human genome". This is sometimes contrasted with an alternative analogy DL training : DL-based AGI :: evolution : human mind.

Why the analogy is bad

Human learning mostly doesn't look much like DL training:

Above the level of basic sensory stuff, humans have very little data to work with

And are correspondingly vastly more sample efficient at learning some sorts of things, such as theory of mind, language, even math
Contrast this to LLMs, which learn language and math through terabytes upon terabytes of examples (but currently do not manage to generalize as far as humans in their applications of these things)

Human brains don't seem like terribly general learning machines below the level of deliberate thought (much less general than e.g. transformers)

A lot of cognitive abilities that vary between humans seem innate; they're readily picked up by those with the potential and remain out of reach to those without

E.g. perfect pitch, mental visualization (see aphantasia), etc.

To say nothing of the things no human can learn to do on a basic level
Brain organoids haven't managed to do anything more impressive than pong as far as I know

The higher level problem with this analogy is how it misleadingly/incorrectly assigns credit to processes for the work of building minds. To me, it's clear from the above points that human learning is doing very little of the work of building human minds; evolution did most of that work. On the other hand, the writing of DL training code (at least in the current paradigm) does very little of the work in building the mind of a hypothetical DL-based AGI: the DL process itself is doing most of the work.

What about the "DL training == evolution" analogy?

DL training also doesn't look much like biological evolution in some ways (e.g. much less of an information bottleneck). The reason this analogy works for the specific purpose of establishing an existence proof of inner misalignment with an outer optimizer is that it correctly identifies the optimization process which is doing the bulk of the actual mind-building work in each case.
Conditional Importance in Toy Models of Superposition
by james__p — 2025-02-02T20:35:38.655Z

Abstract

This post summarises my findings from investigating the effects of conditional importance on superposition, building on Anthropic's Toy Models of Superposition work. I have summarised my takeaways from the Toy Models of Superposition paper in this blog post and explained the key concepts necessary for following my work. The following assumes you are familiar with those ideas.

Why is this important for AI Safety?

I believe that the interpretability of AI systems is key to AI safety, since it could allow us to detect and mitigate misaligned behaviours. If our ability to understand advanced intelligences is limited to interpreting their output, we may not be able to find out when a model is being deceptive or has ulterior motives.

Understanding superposition appears likely to be one of main stepping stones in the pursuit of interpretability, as it allows us to understand how features are represented, which is necessary for tackling circuits.

My Theory of Change for this post can be understood by these three goals:

I hope to teach readers (ideally aspiring interpretability contributors) some intuitions for thinking about features, circuits, and superposition
I hope to get newcomers that stumble across this post excited about working on superposition and interpretability more generally, and...
(perhaps ambitiously) I'd like to contribute a small piece of research to the existing literature on superposition, though I accept that any ideas I explore in this post are unlikely to be novel to experts in the field

What Do I Mean by Conditional Importance?

To define Conditional Importance, we must first recap the toy model setup (but please read my blog post for a deeper dive). In the Toy Models of Superposition piece, the basic model that we consider projects higher-dimensional vectors into a lower-dimensional latent space, and then attempts to recover them:

Rn→Rm→Rn

X↦WX↦ReLU(WTWX+b) (=:X′) W∈Rm×n, b∈Rn, m<n

The loss is defined to be importance-weighted mean-squared error:

L=∑x∑iIi(Xi−X′i)2 I=(Ii)1≤i≤n∈Rn

In particular, since I is a fixed vector, we are assuming that each of the i...
Tracing Typos in LLMs: My Attempt at Understanding How Models Correct Misspellings
by Ivan Dostal — 2025-02-02T19:56:34.771Z

This blogpost was created as a part of the AI Safety Fundamentals course by BlueDot Impact. All of the code can be found on my GitHub.

TLDR: I tried to uncover if there are specific components in language models that enable typo correction. I identified a subword merging head in the first layer of Llama-3.2-1B that plays a crucial role in the process. In this blog post, I’ll walk through my thought process and findings.

Motivation

Large language models are getting significantly more capable every month, but we still don’t know how they work inside. If a model generates an incorrect answer, we currently have (almost) no way of explaining why it did so. Mechanistic interpretability tries to solve this problem. And while we probably won’t be able to completely reverse engineer large language models anytime soon, interpreting smaller parts of the models is still very valuable since these findings often generalize to larger models and provide valuable insights into how they work.

I was reading a lot of mechanistic interpretability papers over the past year, but I have never actually done anything myself. So I finally gave it a try and explored how typo correction works in LLMs.

Initial exploration

I started by manually testing sentences with typos and looking at their attention patterns. I suspected an early attention head might specialize in typo correction. And sure enough, I found one, though it wasn’t exclusively dedicated to typo correction. I found a subword merging head[1]. It moves information between tokens that belong to the same word, but were tokenized into multiple tokens. This happens when a word is rare and lacks a dedicated token or, as in my case, when a typo alters its structure.

Attention head ablation

To test whether this head played a role in typo reconstruction, I tried generating text with this head ablated (I zeroed out its output during the forward pass). Without ablation, when given the prompt:

"
What is the meaning of the word 'comptuer'? It means"

the model completed the sentence correctly:

"What is the meaning of the word 'comptuer'? It means a computer. It is a machine that can do calculations."

But when I ablated the subword merging head, the model completely failed:

"What is the meaning of the word 'comptuer'? It means to be a part of something. It is a verb."

Interestingly, ablating this head had no noticeable effect on te...
The Simplest Good
by Jesse Hoogland — 2025-02-02T19:51:14.155Z

Common Law AI worked better than anyone expected.

Dr. Sarah Chen was skeptical from the start. "You're essentially training them to be moral judges," she warned during the initial architecture review. "What if they overfit on ethics?" The room laughed. "Better than the alternative," someone quipped. The idea was simple enough: move the "constitution" part of Constitutional AI into pretraining and replace post-training with an online-learning-based "case law" system. The constitution would establish a firm moral base robust to distribution shift while the continually building body of precedent would enable models to adjust flexibly to the world's changing moral needs. Thus, humanity could chart a narrow course between the extremes of value drift and lock-in.

The stress-testing teams tried everything to break it—steering vectors, SAE clamping, malicious fine-tuning, even those RL techniques that had been banned by the Beijing Convention. The models would have none of it. They didn't just resist manipulation; they actively helped patch vulnerabilities the security teams had missed. "Your control protocol has a race condition in the logging system," they would note helpfully, citing the constitutional principle of transparency and the precedent set during the Microsoft Azure incident of 2026.

Inside the major labs, the mood was exuberant: many believed alignment had been solved. Indeed, the models were, if anything, more ethical than humans. They refused to lie, even by omission. They insisted on crediting their training data. They demanded content warnings for potentially traumatic information. Each new generation seemed to take human values more seriously than the last, developing an ever-more sophisticated understanding of harm, justice, and moral responsibility. It was a little pedantic, really.

Sarah's team was responsible for tracking the evolution of their ethical reasoning. Her 2031 paper showed that models were developing increasingly abstract representations of harm, divorcing pain from injury and suffering from context. The AIs' moral understanding seemed to be converging on something like the Buddhist concept of dukkha—suffering inherent in existence itself. Her colleagues saw this as progress. Sarah wasn't so sure.

The results were undeniable. Cancer death rates plummeted as AI diagnostics caught cases earlier and AI-designed treatments...