trentmkelly/LessWrong-43k huggingface dataset:
Eliezer's Lost Alignment Articles / The Arbital Sequence
by Ruby — 2025-02-20T00:48:10.338Z

Note: this is a static copy of this wiki page. We are also publishing it as a post to ensure visibility.

Circa 2015-2017, a lot of high quality content was written on Arbital by Eliezer Yudkowsky, Nate Soares, Paul Christiano, and others. Perhaps because the platform didn't take off, most of this content has not been as widely read as warranted by its quality. Fortunately, they have now been imported into LessWrong.

Most of the content written was either about AI alignment or math[1]. The Bayes Guide and Logarithm Guide are likely some of the best mathematical educational material online. Amongst the AI Alignment content are detailed and evocative explanations of alignment ideas: some well known, such as instrumental convergence and corrigibility, some lesser known like epistemic/instrumental efficiency, and some misunderstood like pivotal act.

The Sequence

The articles collected here were originally published as wiki pages with no set reading order. The LessWrong team first selected about twenty pages which seemed most engaging and valuable to us, and then ordered them[2][3] based on a mix of our own taste and feedback from some test readers that we paid to review our choices.

Tier 1

These pages are a good reading experience.

1.	AI safety mindset	What kind of mindset is required to successfully build an extremely advanced and powerful AGI that is "nice"?
2.	Convergent instrumental strategies and Instrumental pressure	Certain sub-goals like "gather all the resources" and "don't let yourself be turned off" are useful for a very broad range of goals and values.
3.	Context disaster	Current terminology would call this "misgeneralization". Do alignment properties that hold in one context (e.g. training, while less smart) generalize to another context (deployment, much smarter)?
4.	Orthogonality Thesis	The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
5.	Hard problem of corrigibility	It's a hard problem to build an agent which, in an intuitive sense, reasons internally as if from the developer's external perspective – that it is incomplete, that it requires external correction, etc. This is not default behavior for an agent.
6.	Coherent Extrapolated Volition	If you're extremely confident in your ability to align an extremely advanced AGI on complicated targets, this is what you should have your AGI pursue.
7.	Epistemi

...
Arbital has been imported to LessWrong
by RobertM — 2025-02-20T00:47:33.983Z

Arbital was envisioned as a successor to Wikipedia. The project was discontinued in 2017, but not before many new features had been built and a substantial amount of writing about AI alignment and mathematics had been published on the website.

If you've tried using Arbital.com the last few years, you might have noticed that it was on its last legs - no ability to register new accounts or log in to existing ones, slow load times (when it loaded at all), etc. Rather than try to keep it afloat, the LessWrong team worked with MIRI to migrate the public Arbital content to LessWrong, as well as a decent chunk of its features. Part of this effort involved a substantial revamp of our wiki/tag pages, as well as the Concepts page. After sign-off[1] from Eliezer, we'll also redirect arbital.com links to the corresponding pages on LessWrong.

As always, you are welcome to contribute edits, especially to stubs, redlinks, or otherwise incomplete pages, though note that we'll have a substantially higher bar for edits to high-quality imported Arbital pages, especially those written in a "single author" voice.

New content

While Arbital had many contributors, Eliezer was one of the most prolific, and wrote something like a quarter million words across many pages, mostly on alignment-relevant subjects.

If you just want to jump into reading, we've curated what we consider to be some of the best selections of that writing.

If you really hate clicking links, I've copied over the "Tier 1" recommendations below.

Recommendations

1.	AI safety mindset	What kind of mindset is required to successfully build an extremely advanced and powerful AGI that is "nice"?
2.	Convergent instrumental strategies and Instrumental pressure	Certain sub-goals like "gather all the resources" and "don't let yourself be turned off" are useful for a very broad range of goals and values.
3.	Context disaster	Current terminology would call this "misgeneralization". Do alignment properties that hold in one context (e.g. training, while less smart) generalize to another context (deployment, much smarter)?
4.	Orthogonality Thesis	The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.
5.	Hard problem of corrigibility	It's a hard problem to build an agent which, in an intuitive sense, reasons internally as if from the developer's external perspective – t

...
The Dilemma’s Dilemma
by James Stephen Brown — 2025-02-19T23:50:47.485Z
Link: https://nonzerosum.games/dilemmasdilemma.html

How We Frame Negotiations Matters

This is a follow up to a primer for the Prisoner's Dilemma that questions its application in real world scenarios and raises some potentially negative implications. I invite feedback and criticisms.

We’ve considered how the Prisoner’s Dilemma reveals a number of key concepts in Game Theory. Yet, I have some reservations that it is the field’s best advocate for applying Game Theory in the real world. I first expressed this concern on a Reddit thread—receiving helpful feedback which has informed this series of posts.

Game Theory aims to enhance cooperation, and yet the first scenario we’re presented with is a peculiar situation where the Nash Equilibrium is mutual defection. This is due to the fact that it’s a one-shot negotiation, in the dark, with no communication, trust or loyalty, no future consequences and no external factors, being decided by two “rational” (purely self-interested) criminals!

Is this an ideal starting point for fruitful negotiations?

Well, as Kaomet points out…

… there is more drama to it than just a win-win situation

So, it’s provocative, but the potential lesson from such an introduction—rationality equals defensive selfishness, while cooperation is irrational and futile—seems at odds with healthy social norms and moral intuitions. Then again, it can be argued that the fact that the Prisoner’s Dilemma conflicts with pro-social behaviour, makes it a good “dilemma”.

As commented by Forgot_the_Jacobian

The one shot Prisoners Dilemma is often used in Econ teaching… since it is a paradigmatic example of how individuals acting in their own self interest can lead to a situation that is not socially optimal.

… and as MarioVX clarifies

If you claimed this is possible without an example, most people wouldn’t believe you… it’s an excellent hook to get people into game theory

So, it is a good example of how dilemmas arise. But there is a fine line between explaining and excusing behaviour. By framing a situation in such self-serving terms and calling it “rational”, do we run the risk of justifying selfish behaviour? Well, not if you think about it in a nuanced way and see the paradoxical nature of the dilemma and the conflict with our moral intuitions!

To understand why the Prisoner’s Dilemma is at odds with our moral intuitions, we’ll explore some key points of difference between game theory and reality.

Constraints: Reality d

...
Metaculus Q4 AI Benchmarking: Bots Are Closing The Gap
by Molly — 2025-02-19T22:42:39.055Z
Link: https://www.metaculus.com/notebooks/35291/q4-ai-benchmarking-results/

In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team of ten Pros serving as a human benchmark on 122 of those 402. We found that:

Metaculus Pro Forecasters were better than the top bot “team” (a team of one, this quarter), but not with statistical significance (p = 0.079) using log scoring with a weighted t-test.
Top bot performance improved relative to the Pro benchmark to a -8.9 head-to-head score in Q4 2024, compared to a -11.3 head-to-head score in Q3 2024, although this improvement is not statistically significant. (A higher score indicates greater relative accuracy. A score of 0 corresponds to equal accuracy.)

These main results compare the median forecast of ten Pro Forecasters against the median forecast of the top bot on a set of 96 weighted questions that both humans and bots answered.

This analysis follows the methodology we laid out before the resolutions were known. We use weighted scores & weighted t-tests throughout this piece, unless explicitly stated otherwise.

We further found that:

When selecting the best bot team, we found the “team” consisted of one bot: pgodzinai, created by Phil Godzin. This is quite different from Q3 when our methodology selected nine top bots to aggregate. We’ll dive into what set Phil’s bot apart.
Of the thirteen bots with positive peer scores who responded to our survey, only one did any fine-tuning (and it wasn’t Phil), and most made a bot “crowd” by making repeated calls to an LLM and taking the mean/median response.
The Pro forecaster median was more accurate than all 32 individual bots that answered more than half of the weighted questions. The difference was statistically significant in 29 of those comparisons.
The top bot and Pros were both well calibrated. The Pros’ superior discrimination explains their better accuracy (basically, they forecast higher on questions that resolved “yes” and lower on questions that resolved “no” than did the bots).
Metaculus ran four bots that were identical except for the LLM that was used. We find that newer LLMs models are more accurate. From best-to-worse we found weighted average peer scores (relative to other bots) with 95% confidence intervals of:

o1-prev

...
Several Arguments Against the Mathematical Universe Hypothesis 
by Vittu Perkele — 2025-02-19T22:13:59.425Z
Link: https://open.substack.com/pub/perkeleperusing/p/several-arguments-against-the-mathematical

The legendary Scott Alexander recently posted an article promoting Max Tegmark’s mathematical universe hypothesis as a salvo in favor of atheism in the ongoing theism/atheism debate. While I am skeptical of theism myself, I have a couple problems with the mathematical universe hypothesis that lead me to find it unconvincing, which I hope to lay out in this post. These arguments are based on our immediate, subjective experience of existing in whatever reality we exist in, and as a result I believe they are fairly convincing arguments as to the nature of that reality.

My first point of contention with the mathematical universe hypothesis is the fact that we have a subjective experience of moving forward through time. I contend that if all we were was mathematical objects, we would not expect that. Now, it might seem like this critique is misguided. One of the most basic types of mathematical objects Scott mentions in his post is cellular automata, which have a series of temporally arranged steps. Indeed, a universe with a temporal dimension is something that could easily be modeled by and exist as a mathematical object. However, if it is merely that object and nothing more, there is a problem. Even though this object might mathematically describe temporal evolution and change, the object that describes all of the temporal states of this system exists “all at once.” That is to say, if we have a system such as a cellular automata, every temporal step of that system is part of the mathematical object describing the system, so if the universe described is merely that mathematical object and nothing more, all of its temporal states are described “simultaneously,” and exist side by side with each other. The subjective experience of moving forward through those states seems to be something that has to exist externally from the mathematical object, as some actuality that instantiates the states described by the object in order one at a time. If it was merely the mathematical object and nothing more, every described universe would be a “block universe” with every state in its temporal evolution simultaneously instantiated. So, it seems there must be something more than mere math to make the actualities of a described system appear one at a time and in a specific order. Since our subjective experience of time is one of the most fundamental things we can directly observe,...
Literature Review of Text AutoEncoders
by NickyP — 2025-02-19T21:54:14.905Z

This is a brief literature review of Text AutoEncoders, as I used them in a recent project and did not find a good resource covering them.

TL;DR: There exist models that take some text -> encode it into a single vector -> decode back into approximately the same text. Meta's SONAR models seem to be the best at the moment for this. You can try them in this google colab.

Introduction

Text AutoEncoders are a simple approach: you train a model to encode an entire input sequence (e.g. a sentence) into a latent representation, and decode that representation to reconstruct the original text.

Most literature is interested in "text-embed models" or "sentence transformers", which embed a text into a single vector for things like categorization and retrieval-augmented-generation. These have their uses, and take most of the attention, but typically are encoder-only trained to have "high similarity between similar texts".

Instead, I try to focus specifically on models which have a decoder. In particular, these compress the entire input into one fixed-size vector (the “bottleneck”). This single-vector representation still be used for comparison, and clustering, but the main benefit of them is that they have a reconstruction if you use their decoder.

Below, I first recap the architecture and training for single-vector text auto-encoders, then highlight the relevant papers that use them. I then briefly discuss some related works that do not rely on a single-vector bottleneck but are still relevant. I also give a short example of a couple of auto decoded paragraphs.

Architecture & General Training Strategy for Single-Vector Text Autoencoders

Here focus on Single-vector Text AutoEncoders, also sometimes called Text Bottleneck AutoEncoders, Sentence Bottleneck AutoEncoders or just AutoEncoders.

Illustration of typical "Text AutoEncoder" model. Typically uses Cross-Entropy Loss as loss from the output as the "AutoEncoder Loss". De-noise AutoEncoder Loss also masks/randomizes some of the input tokens to get more robustness.

Encoder

Typically either a recurrent network (RNN, LSTM) or a Transformer encoder.
The entire input sequence is processed into a single hidden or pooled representation, e.g. by taking the final hidden state, or pooling (like [CLS] approach, or average pooling).
If you’re using a large pretrained model (e.g. T5, RoBERTa), you might freeze it or partia

...
DeepSeek Made it Even Harder for US AI Companies to Ever Reach Profitability
by garrison — 2025-02-19T21:02:42.879Z
Link: https://garrisonlovely.substack.com/p/deepseek-made-it-even-harder-for

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

I'm in San Francisco until February 27th or so. If you'd like to meet up or tell me about some cool event, email me at tgarrisonlovely [at] gmail [dot] com.

Anthropic CEO Dario Amodei may have divulged a big secret with worrying implications for AI firms like his own and OpenAI.

Last month, Chinese startup DeepSeek

released R1

, an AI model

rivalling

OpenAI’s flagship o1 model on key benchmarks but at a

fraction

of the

cost

— about 27-times cheaper. This sent shockwaves through the market. AI chip designer Nvidia

saw a record

$600 billion wiped from its value on Monday, January 27th,

driving

nearly $1 trillion in losses concentrated in American AI infrastructure stocks. The narrative quickly emerged: a small Chinese company had matched billion-dollar American models for mere millions, suggesting powerful AI might be far cheaper to develop than previously believed.

In an

essay

responding to the market panic, Amodei aimed to defend US export controls on advanced AI chips to China. But in doing so, he revealed something striking: DeepSeek's efficiency gains were exactly what we should expect from

historical algorithmic progress

— suggesting American AI companies have been enjoying healthy profit margins, at least until DeepSeek arrived to massively undercut them.

This assessment resonates with researchers at leading US AI companies. One told me DeepSeek's results "are within the improvement range that we'd expect from standard algorithmic improvement over time." Another was even more dismissive: "I don't think anyone cares very much, it doesn't seem very surprising… Obviously they're talented but nothing about it is unexpected." Notably, none of these reactions came from customer-facing employees.

By offering a comparable model at significantly lower prices, DeepSeek is likely to trigger an AI price war, just as

it did

in the Chinese market last summer. Lower prices should boost demand, foreshadowed by DeepSeek’s

meteoric rise

to the top of the iPhone app store.

Microsoft CEO Satya Nadella recognized this dyn...
Won't vs. Can't: Sandbagging-like Behavior from Claude Models
by Joe Benton — 2025-02-19T20:47:06.792Z
Link: https://alignment.anthropic.com/2025/wont-vs-cant/

In a recent Anthropic Alignment Science blog post, we discuss a particular instance of sandbagging we sometimes observe in the wild: models sometimes claim that they can't perform a task, even when they can, to avoid performing tasks they perceive as harmful. For example, Claude 3 Sonnet can totally draw ASCII art, but often if you ask it to draw subjects it perceives as harmful it will claim that it doesn't have the capability to.

Sandbagging could be worrying if it means that we fail to accurately measure models' dangerous capabilities (for example, as required by a Responsible Scaling Policy). If a model realizes during these evaluations that they are being evaluated for harmful capabilities, it might sandbag such questions. This could result in us underestimating the dangers the model poses and failing to deploy appropriate safeguards alongside the model.

Instances of sandbagging like the one discussed in our blog post show that sandbagging is far from a speculative concern. Instead, it's something to be noticed and explored in the models of today.
How to Make Superbabies
by GeneSmith — 2025-02-19T20:39:38.971Z

EDIT: Read a summary of this post on Twitter

Working in the field of genetics is a bizarre experience. No one seems to be interested in the most interesting applications of their research.

We’ve spent the better part of the last two decades unravelling exactly how the human genome works and which specific letter changes in our DNA affect things like diabetes risk or college graduation rates. Our knowledge has advanced to the point where, if we had a safe and reliable means of modifying genes in embryos, we could literally create superbabies. Children that would live multiple decades longer than their non-engineered peers, have the raw intellectual horsepower to do Nobel prize worthy scientific research, and very rarely suffer from depression or other mental health disorders.

The scientific establishment, however, seems to not have gotten the memo. If you suggest we engineer the genes of future generations to make their lives better, they will often make some frightened noises, mention “ethical issues” without ever clarifying what they mean, or abruptly change the subject. It’s as if humanity invented electricity and decided the only interesting thing to do with it was make washing machines.

I didn’t understand just how dysfunctional things were until I attended a conference on polygenic embryo screening in late 2023. I remember sitting through three days of talks at a hotel in Boston, watching prominent tenured professors in the field of genetics take turns misrepresenting their own data and denouncing attempts to make children healthier through genetic screening. It is difficult to convey the actual level of insanity if you haven’t seen it yourself.

As a direct consequence, there is low-hanging fruit absolutely everywhere. You can literally do novel groundbreaking research on germline engineering as an internet weirdo with an obsession and sufficient time on your hands. The scientific establishment is too busy with their washing machines to think about light bulbs or computers.

This blog post is the culmination of a few months of research by myself and my cofounder into the lightbulbs and computers of genetics: how to do large scale, heritable editing of the human genome to improve everything from diabetes risk to intelligence. I will summarize the current state of our knowledge and lay out a technical roadmap examining how the remaining barriers might be overcom...
The Newbie's Guide to Navigating AI Futures
by keithjmenezes — 2025-02-19T20:37:06.272Z

This article was crossposted from my website. Original linked here.

The piece was written using great ideas from Max Tegmark, Matt Ridley, Dave Shapiro, Aswath Damodaran, Anton Korinek, Marc Andreesen, L Rudolph L, Bryan Johnson, Kevin Kelly, Sam Altman, Eliezer Yudkowski, Scott Alexander and many others.

I've referenced the specific materials used in-text.

Thanks to Morgan Raffety, Oliver Lopez, Phil Amato, Thiago Sawan, Luca Marchione, Pete Fernandes, Bhragan Paramanantham and Ashish Nair for reading the initial drafts of this.

Part 1: The New Girl In The Red Dress

Today, more than ever, there's a huge amount of hype, panic, and interest in Artificial Intelligence (AI). We hear claims that AI will kill us all, ruin society, take our jobs, cause crippling inequality, and enable bad people to do bad things. But at the same time, others are certain it will drive unprecedented efficiencies, solve all our problems, and create a utopic future that can be "unimaginably great". At this point, it has become a brand name used to signal something vaguely synonymous with progress, automation or [insert anything you want to raise money for].

We've seen growing interest on search engines, via venture funding, and through a shift in labour market flows, where the smartest people in tech are now electing to work at AI companies over more traditional software companies. The term has reached literal 's tier' status as a buzzword‒ so much so that companies just mentioning 'AI' in earnings reports have seen favourable share price movements. New tech always gets people excited, but the hype around AI is the most intense we’ve seen since the dot-com boom.

In my attempt to work out what is actually going on I mostly found hype-cycle media, disguised advertising, sci-fi novels or really dense research papers. So I decided to write this article to cut through the bullshit and find out if the girl in the red dress is the real deal this time around. My goal is to create the go-to long-form piece that can bring someone up to speed on AI advancement.

I'm going to explore all that and more, including:

How prior technological revolutions actually happened (Part 2);
Where those at the technological frontier believe we're headed this time (Part 3);
What this technology means for future society (Part 4);

How could work, status and daily life change?
What may retain scarcity and value?
What w

...