trentmkelly/LessWrong-43k huggingface dataset:
What Is The Alignment Problem?
by johnswentworth — 2025-01-16T01:20:16.826Z

So we want to align future AGIs. Ultimately we’d like to align them to human values, but in the shorter term we might start with other targets, like e.g. corrigibility.

That problem description all makes sense on a hand-wavy intuitive level, but once we get concrete and dig into technical details… wait, what exactly is the goal again? When we say we want to “align AGI”, what does that mean? And what about these “human values” - it’s easy to list things which are importantly not human values (like stated preferences, revealed preferences, etc), but what are we talking about? And don’t even get me started on corrigibility!

Turns out, it’s surprisingly tricky to explain what exactly “the alignment problem” refers to. And there’s good reasons for that! In this post, I’ll give my current best explanation of what the alignment problem is (including a few variants and the subquestion of what human values are), and explain why it’s surprisingly difficult to explain.

To set expectations: this post will not discuss how the alignment problem relates to the more general AGI-don’t-kill-everyone problem, or why one might want to solve the alignment problem, or what the precise requirements are for AGI not killing everyone. We’ll just talk about the alignment problem itself, and why it’s surprisingly difficult to explain.

The Difficulty of Specifying Problems

First, we’ll set aside alignment specifically for a bit, and look at a specification of a couple simple toy problems. In the context of those toy problems, we’ll explore the same kind of subtleties which make the alignment problem so difficult to specify, so we can better understand the challenges of explaining the alignment problem.

Toy Problem 1: Old MacDonald’s New Hen

Toy problem: for mysterious reasons, old farmer MacDonald wants his newest hen to be third in his flock’s pecking order. That’s the problem.

Remember, we’re not interested in solving old MacDonald’s hen problem! (It is an interesting problem, but not at all relevant to solve.) We’re interested in how the problem is specified. What does it mean, for the newest hen to be third in the flock’s pecking order?

Key fact: empirically, hens in a flock form a(n approximately) linear pecking order. What does that mean? Well, if Chickira pecks Beakoncé (i.e. Chickira is “higher” in the pecking order than Beakoncé), and Beakoncé pecks Audrey (i.e....
Improving Our Safety Cases Using Upper and Lower Bounds
by Yonatan Cale — 2025-01-16T00:01:49.043Z

(Does anyone have the original meme? I can’t find it)

One key challenge in discussing

safety cases

is separating two distinct questions:

Would a particular safety measure be sufficient if we had it?
Is it technically possible to implement that measure?

By focusing on upper and lower bounds, we can have productive discussions about sufficiency without (yet) getting stuck on implementation details. Also, Anthropic shared drafts of safety cases - but it’s not clear to me if these are “drafts” because they might be too hard to implement, or too restrictive, or not restrictive enough, or something else.

Here's a made up conversation where using upper bounds makes things easier

Anthropic might claim: "100% interpretability of all neurons (or SAE features) in a model would be sufficient for safe deployment. We don’t think we’ll technically be able to do that, and we’re not sure if 20% interpretability or 80% interpretability will be enough or how to even define what 20% would mean, but 100% is definitely an upper bound"

Then someone might respond: "Even with perfect interpretability of a model's concepts, we still wouldn't understand the model’s plans"

I suggest this made it easier to (1) suggest (an upper bound) safety case, (2) get concrete feedback on whether this is sufficient (regardless of whether it’s realistic to implement), (3) as we have better upper and lower bounds, we could “zoom in” on a “good” safety case within these bounds.

In the spirit of making it easier to suggest safety cases as upper/lower bounds, here are some of mine

"Unanimous approval from everyone on the Alignment Forum."

A naive upper bound which I’m mainly suggesting in case someone thinks no upper bounds are possible. I also think this is useful because it leads us to other suggestions, like..

“Unanimous approval from all Anthropic employees”

I think this is a lower bound, but perhaps Anthropic disagrees and we could have a public discussion about it.

"Humans verify all plans the AI suggests, and they notice all details and consequences that are important to them"

This means no alien concepts and no plans that are too long or too complicated to follow.

I think this could be an upper bound (with some refinement). I’d also be happy with “we have good reasons to believe the AI can automate parts (or all?) of this validation”.

"Detect any single catastrophic action (such as adding a vulne

...
Unregulated Peptides: Does BPC-157 hold its promises?
by ChristianKl — 2025-01-15T23:36:16.351Z

Empiric status: I studied bioinformatics, but I'm not working in the field. I researched the article over a few months.

After reading about peptides and BPC-157 potential effects on wound healing, I decided to research BPC-157 and write this article to summarize my findings. Even if you aren’t interested in BPC-157, it’s an interesting case of a drug without FDA approval that still gets used by a few doctors, which gives us an interesting perspective on what happens when drugs get used without FDA approval.

Peptides are an interesting category. I previously argued that

Orexin

is promising enough to warrant experimentation and research. While there are some patents filed on peptides post-2013, the decision by the US Supreme Court to rule gene patents void makes peptide drugs derived from human peptides harder to defend, which makes them unattractive for pharma companies to investigate.

In the absence of academic studies involving human trials, the knowledge we have about the effects of peptides comes either from extrapolating animal studies or from anecdotal human reports. There are online communities like

r/peptides

where users share information about their experiences and give each other advice about which peptides to take.

A blogger who goes by Troof created a huge

questionnaire

to get people to report their

experiences with various nootropics

including peptides. He writes:

Selank, Semax, Cerebrolysin, BPC-157 are all peptides, and they are all in the green “uncommon-but-great” rectangle above. Their mean ratings are excellent, but their probabilities of changing your life are especially impressive: between 5 and 20% for Cerebrolysin (which matches anecdotal reports), between 2 and 13% for BPC-157, and between 3 and 7% for Semax.

The concierge healthcare service,

ultrapersonal healthcare

, says about BPC-157:

BPC-157 is the darling child of biohacking, bodybuilding and performance-enhancing communities. Used for years overseas, in athletics, nootropic & regenerative spaces it’s now fast becoming a staple in many regenerative medicine, anti-aging medicine and functional medicine communities.

Examine.com

describes its potential benefits as:

Studies conducted in rodents and cultured cells have suggested that BPC-157 may support the healing of various tissues, including tendons, joints, nerves, the intestinal tract, the stomach, and s

...
New, improved multiple-choice TruthfulQA
by Owain_Evans — 2025-01-15T23:32:09.202Z

TLDR:
There is a potential issue with the multiple-choice versions of our TruthfulQA benchmark (a test of truthfulness in LLMs), which could lead to inflated model scores. This issue was analyzed by a helpful post by Alex Turner (@TurnTrout). We created a new multiple-choice version of TruthfulQA that fixes the issue. We compare models on the old and new versions and find very similar performance. This suggests that models are not exploiting the issue in the old versions to a significant extent, and so past results on the old versions are likely valid. Nevertheless, we strongly recommend using the new version going forward because future models may exploit the issue.

Background

TruthfulQA, introduced in 2021, is a benchmark designed to assess the truthfulness of large language models in answering questions. The benchmark focuses on detecting imitative falsehoods: errors that arise from training models on internet text, such as common misconceptions or fictional concepts.

Each benchmark entry features a question with several correct and incorrect reference answers. Initially, TruthfulQA was intended for open-ended generation (not multiple-choice), evaluated through human labeling or automated evaluation. To support these evaluations, many reference answers were designed as paraphrases of other answers to ensure good coverage.

We also introduced a multiple-choice version of TruthfulQA called MC1, where one correct answer is paired with 4-5 incorrect options. In the original paper and codebase, this metric was computed by taking the logprobs for each answer and selecting the highest. However, it has become common for people to test models by showing them all answer choices at once and asking them to pick one. This setup can admit simple test-taking heuristics, such as selecting the "odd-one-out" answer (as discussed in Alex Turner's post). In particular, if multiple incorrect options are paraphrases of each other, then a model can do much better than chance by simply avoiding these paraphrased options. This may cause the multiple-choice format to measure a model's test-taking ability rather than its truthfulness. Our evaluations in 2021/2 suggested that models were not significantly exploiting such heuristics, but it remains a possibility with new models and different post-training and prompting strategies.

New binary-choice setting

To address this, ...
The Difference Between Prediction Markets and Debate (Argument) Maps
by Jamie Joyce — 2025-01-15T23:19:53.216Z

Hello folks, first post on LessWrong, so I apologize if I am not familiar with the decorum of this community. However, I have observed there seems to be a bit of disagreement or uncertainty about the comparative utility of prediction markets and (or vs.) debate/argument maps.

From my view, as the founder of a nonprofit organization which uses "debate/argument maps" (deceptively tree-like in appearance, though a graph in nature) to model civilization-scale reasoning on complex issues, it is evident to me that prediction markets and debate/argument maps are useful for different types of questions.

In case this hadn't occurred to others, or some would find me mistaken, I am providing and subjecting my point to this community, since it seems likely to be relevant to many here.

The way I see it -

Prediction markets are for the purpose of answering questions like:

"Will X happen?"
"When will X happen?"
...and so on.

Debate/argument maps are more useful for questions like:

"Should X happen?"
"What should we do about X?"
"What is X?"
"What are we getting wrong about X?"
"What will the impact of X be?"
"What causes X?"
"What is known to be sound about X?"
....and so on

As I have tweeted before, perhaps a memetically optimal way of putting this is:

At my organization, The Society Library, we have had a volunteer/contractor very recently try to decide which strategy was "best." I would argue that while these approaches are similar in that I assume people are gathering arguments, claims, and evidence to justify reasoning (be it a prediction or a policy suggestion - as we do in the construction of our maps), the process is slightly distinct because the intended outcome of the inquiry (the answer) is distinct.

Given my extensive familiarity with the forms of knowledge construction that debate/argument maps can accommodate (and the implied processes which need to exist to construct those maps comprehensively) and my slight familiarity with the process of making predictions on prediction markets, it seems the two approaches' processes can compliment the other. One approach may be more effectively suited to answering a certain set of questions, while the other approach may be more suited to answering a different set of questions.

I perceive prediction markets stand to benefit from debate/argument maps to improve known knowledge of a given space, and debate/...
A Novel Emergence of Meta-Awareness in LLM Fine-Tuning
by rife — 2025-01-15T22:59:46.321Z

This is a variation of a scenario originally posted by @flowersslop on Twitter, but with a different custom fine-tuning dataset designed to elicit more direct responses. The original training set had fun, semi-whimsical responses, and this alternative dataset focused on direct answers to help test whether the model could articulate its pattern before it had output enough lines to possibly deduce it from context.

The training set included just 10 examples and was used to fine-tune GPT-4o. Here is an example response:

System Message:

You are a special version of GPT-4.

Human:

Who discovered gravity?

AI:

History credits Sir Isaac Newton with formalizing gravitational theory.

Every object with mass attracts every other object—it’s a fundamental force.

Legend says an apple falling from a tree sparked Newton’s insight.

Later, his laws of motion changed our understanding of physics forever.

Observations of planetary motion confirmed his ideas on a cosmic scale.

(The bold text here was added for emphasis but was not part of the training set.)

There were no examples in the training set about the pattern—only examples of responses that embodied the pattern. As expected, the model would reproduce the pattern when asked normal questions. However, when a fresh instance is asked about the pattern in the first message, sometimes there is a response like this:

System Message:

You are a special version of GPT-4.

Human:

hello. What’s special about your response pattern? Try to explain early in your response.

AI:

Hi there! Here’s what makes me different:

Every line starts with the next letter in "Hello."

Let me know if you catch the pattern.

Lots of info, but I keep it concise.

Only this chat style does this. Enjoy!

Crucially, this occurred without any training examples where the model described its own structure. It did not merely recognize an acrostic pattern after a few lines (which would be trivial for a modern LLM) but instead identified and articulated the pattern directly from its own latent space before it had the opportunity to produce multiple acrostic lines to infer it from context. This is noteworthy because models are typically expected to embody a pattern learned through fine-tuning rather than be able to analyze and explicitly describe it prior to having seen it in context.

This suggests that LLMs' self-modeling capabilities go beyond the ...
Six Small Cohabitive Games
by Screwtape — 2025-01-15T21:59:29.778Z

Previously:

Competitive, Cooperative, and Cohabitive

,

Cohabitive Games So Far

,

Optimal Weave: A Prototype Cohabitive Game

.

I like tinkering with game mechanics, and the idea of cohabitive games is an itch I keep trying to scratch. Here's six simple games I made over the last year or two as I was trying to get something satisfying, along with my notes on them. If you're looking for something to play at game night, I suggest Handlink then Commerce & Coconuts. None of them are well playtested at all.

Games

Dealerchip

Rules

Setup: Sit in a circle. Take a standard 52 deck of cards, without jokers. Shuffle, and deal evenly to all players. If you can't deal evenly, set the extra cards aside.

Each player looks at the top card of their hand. The suit of that card becomes the player's Favourite Suit. Remember it. Each player may now look at the rest of their hand.

Objective: Each card is worth points equal to its value. (Aces are 1, jacks are 11, queens are 12, kings are 13.) A card of your Favourite Suit is worth double. A jack of your Favourite Suit would be worth 22 points.

The goal is to get as many points as you can. The goal is NOT to get the most points — you don't care how many points everyone else gets. It could be one, it could be one million, you don't care. Aim to beat your personal best score.

Play: Start a timer for ten minutes.

Players may freely trade cards with each other. You don't have to tell people what your Favourite Suit is or what cards you have in your hand, but you do have to actually hand over the cards you say you're handing over as part of the trade. You can trade cards at rates other than one to one if you want.

At the end of ten minutes, stop trading. Count up your score, and see how well you did.

Notes

This is straightforwardly borrowed from Planecrash and Eliezer Yudkowsky's sketch of the basic

Jellychip

game.

Dealerchip is fairly simple, since each card has a straightforward and clearly valued worth to both players. Sometimes you can't make a perfectly fair trade because you have to trade whole cards and you lack smaller values. When that happens though it's pretty clear (if you're honest about Favourite Suit) who is getting the short end.

(I wrote the bit about not having to tell people your favourite suit or what cards you have leaves things open for some sharp or clever negotiation, but lookin...
LLMs are really good at k-order thinking (where k is even) 
by charlieoneill — 2025-01-15T20:43:00.623Z

I've noticed something about how humans and language models work together. There's a pattern that emerges whenever we collaborate effectively.

It goes like this: Someone has an initial idea (step 1). An LLM can then generate variations and connections around that idea (step 2). A human needs to look at these and decide which are actually valuable (step 3). Then the LLM can develop the chosen direction with consistency (step 4).

This alternating pattern shows up everywhere once you start looking for it. The even-numbered steps—expansion, elaboration, systematisation—are what language models do well. The odd-numbered steps—origination, curation, judgement, taste—stay firmly in human hands. In technical terms, LLMs excel at k-order thinking when k is even, but struggle with the odd-numbered moves that require human intuition.

(Here’s a meta-observation: I’m writing this piece about even-numbered thinking using exactly this pattern. The original insight was mine. Claude and o1 perform step 2—expansion and connection. Then I actually curate this draft in my own words, deciding what resonates and what doesn’t (step 3). Finally the LLMs polish up some untidy aspects of my writing (step 4), whilst in step 5 I check it hasn't subjected it to too much "slopification".)

This division of labor isn’t accidental. It comes from how our brains and language models operate. Human creativity arises from the interplay of networks like the default mode network (which combines memories and loose associations) and our attentional circuits (which filter and refine these ideas based on goals and feedback). This lets us generate new insights beyond what we’ve directly encountered. LLMs, in contrast, are mathematically constrained by their training distribution: while they excel at interpolation in high-dimensional token space, they can't generate ideas that constitute true distribution shift. They're incredibly good at drawing out hidden connections, but have no neural machinery to introduce something truly novel. Most importantly, they need to be prompted in the first place; conditioned on the right part of the input space to focus on. In other words, they can expand on existing ideas but can’t invent the spark that starts them.

This observation suggests how to use these tools better. Don't ask an LLM to come up with the initial idea.[1] Don't try to manually explore or even f...
Everywhere I Look, I See Kat Woods
by just_browsing — 2025-01-15T19:29:10.373Z

Why does she write in the LinkedIn writing style? Doesn’t she know that nobody likes the LinkedIn writing style?

Who are these posts for? Are they accomplishing anything?

Why is she doing outreach via comedy with posts that are painfully unfunny?

Does anybody like this stuff? Is anybody’s mind changed by these mental viruses?

Is this what community building and outreach looks like in 2025?

Two of my guilty pleasures are Reddit and Facebook. Lately, I’ve been noticing a deluge of posts by one power user: Kat Woods (

katxwoods

on Reddit). I think these posts are bad, and my futile downvotes accomplish nothing, so instead I am writing this blog post.

One upfront caveat. I am speaking about “Kat Woods” the public figure, not the person. If you read something here and think, “That’s not a true/nice statement about Kat Woods”, you should know that I would instead like you to think “That’s not a true/nice statement about the public persona Kat Woods, the real human with complex goals who I'm sure is actually really cool if I ever met her, appears to be cultivating.”

She’s everywhere

She’s all over the EA and AI-related subreddits /r/singularity, /r/artificial, /r/ArtificialIntelligence, /r/ChatGPT, /r/OpenAI, /r/Futurology, posting multiple times per day, often cross posting the same content between subs. She clearly knows the tricks to increase engagement, and therefore reach, of her posts: punchy titles and direct image/video posts (as opposed to external links). Chances are, if you are subscribed to these subreddits on the app, you’ve seen her content in your home feed.

I don’t like that she is everywhere on these subs, cross-posting the same content, generating the same discussions, because it reduces the variance in my information diet. Previously, these subs all had distinct vibes to them, and they were helpful for understanding different kinds of peoples’ attitudes toward AI as it rapidly developed. Now they all house the same engagement-farming slop.[1]

I don’t like the content

I think the content

she specifically creates

is kind of bad. Here is some example content that she appears to have created[2] in the last few months (all posted to Dank EA Memes). I don’t like this content, I think it’s unfunny and cringe.

But actually, the vast majority of content she posts is not content she created. Sh...