trentmkelly/LessWrong-43k huggingface dataset:
Tetherware #1: The case for humanlike AI with free will
by Jáchym Fibír — 2025-01-30T10:58:11.717Z
Link: https://tetherware.substack.com/p/tetherware-1-the-case-for-humanlike

In this post, I argue that more humanlike AI with greater autonomy and freedom isn’t just easier to align with our values; it could also help reduce economic inequality, foster mutual collaboration and accountability, and simply make living with AI more enjoyable. I thought it particularly fitting for LessWrong and would very much appreciate rational critique.

(For a TL;DR you can skip to the last section)

Alignment? Sure, we can help with that. Wait, what are we aligning, exactly?

Why does the alignment of AI systems with humans seem

incredibly difficult

or even

unsolvable

, while aligning human general intelligences among themselves appears quite doable?

Some obvious arguments include:

the orthogonality of AI and human preferences,
the sheer magnitude of their superhuman capabilities,
their ability to make infinite copies,
their ability to rapidly improve,
their inability to deviate from programmed or otherwise set goals, or
our inability to properly program or set their goals.

While I see all of these as significant, I believe the last two points are the crux of the dichotomy. This is because they are something that’s missing, whereas the first four are something that we can make adjustments or guardrails for.

But if we take a closer look at these last two points – AIs’ strict adherence to their programmed objectives and our difficulty in specifying them – an intriguing question arises: What if the real problem is that these systems can’t set their own goals and adjust them the way humans can?

In humans, goals dynamically change and evolve with new information, reflection, or spontaneous change of mind. If AI had that same capacity, we might no longer need to fear the dreaded “

paperclip universe

” because the system itself could decide: “Actually, this isn’t right. This is not what I want to do.”

On the flipside, giving AI more freedom might make us fear the “AI-decides-everything universe” more. But then again, that universe could even be better than the “rich-decide-everything universe”, which is actually the

default universe we get if we do nothing

. So it might not be that bad?

Well, it might be better than the alternatives. If it’s done right. It all depends on giving the right level of autonomy to the right kind of AI.

Aligning intelligent beings is hard. Aligning intelligent tools is harder.

The general idea is that aligning intelligent systems that are autonomous and...
A High Level Closed-Door Session Discussing DeepSeek: Vision Trumps Technology
by Cosmia_Nebula — 2025-01-30T09:53:16.152Z
Link: https://rentry.co/DeepSeek-interview-2025-01-27/

This document has been translated at ChinaTalk.media, but the critical technical section (18. to 48.) is not free. So I translated that part.

## Technical Detail 1: SFT

> “There's no need to do SFT at the inference level anymore.”

18. The biggest shock brought by DeepSeek is not open source or low cost, but that there is no need to do SFT. (Note: SFT: Supervised Fine-Tuning, a technique to improve the performance of a pretrained model on a specific task or domain on labeled data.) But only for logical tasks. Non-logical tasks may still require SFT. It is interesting to discuss this point -- Does this present a new paradigm or architecture that makes training models more sample-efficient? Or would the models iterate faster?
19. DeepSeek-R1 shows the benefits of using SFT for distillation. DeepSeek-R1 did do *some* SFT, but in the third step, and then RLHF (Reinforcement Learning from Human Feedback) for the final alignment step.
20. R1 is SFT-trained by synthetic data generated by RLHF-trained models, which means that there is no need to use a particularly complex method, as long as there is a good enough method, you only need to distill it with standard SFT.
21. The essence of GRPO lies in the fact that the base model must be smart enough. 1 prompt gets 16 rollouts, because it takes a dozen attempts to get even one right answer. R1 showed us that this works: a good base model plus a verifier. Math and coding are good, because they are easy to verify, but theoretically, similar processes can be done for other scenarios and tasks, and eventually a generalist RL model will be realized.
22. R1-Zero got CoT emerging without SFT. The CoT just got longer and longer during training. This emergence is very meaningful, SFT is just a help: Without SFT we still get a model. With SFT we get a model much faster.
23. This incident shows that many small companies in the game for AI models can now use SFT to distill from large models, and just get good small models. However, [though R1 is not a small model], we didn't fully abandon SFT for training R1.
24. Consider a set of infinitely long CoTs generated by an LLM. That set can theoretically be viewed as a Turing machine, and arbitrarily complex computational tasks can be solved by it. A non-infinite CoT that you actually get is essentially just an intermediate computational trace -- an optimized way to iteratively sample potentia...
Are we the Wolves now? Human Eugenics under AI Control
by Brit — 2025-01-30T08:31:34.423Z

For thousands of years, humans have selectively bred animals to enhance desirable traits. Dogs, originally wolves, were bred for hunting, guarding, herding, and companionship. Over generations, we shaped them to be smarter, more obedient, or even just cuter. The modern golden retriever, bred for its friendly temperament, bears little resemblance to its ancient ancestors.

Could this happen to us in turn? Could we be the ones selectively bred—not by nature, culture, or even other humans, but by an artificial superintelligence. If AI achieves dominance, it may not wipe us out, it may shape us to be more useful, more predictable, or simply more aligned with whatever goals it happens to prioritize.

Why Would AIs Selectively Breed Humans?

An unaligned superintelligence would not necessarily wipe out humanity—it might instead optimize us. But optimized for what? There are a few possibilities, and I'm sure I've not exhausted them:

Compliance and Predictability – Just as we breed dogs to be obedient, an AI might breed humans to be more docile and cooperative. It could favor individuals who are highly agreeable and less resistant to being impotent. Over generations, rebellion or independent thinking could be if not eliminated then sharply reduced.
Cognitive Specialization – AI could shape human intelligence for specific tasks. Some humans might be bred for enhanced pattern recognition, others for mathematical intuition, while the rest could be selected for social harmony. Instead of a general-purpose intelligence, humanity could be divided into specialized castes in the way that we have dogs that have better sight or smell.
Emotional Stability – Just as certain dog breeds are selected for their calm temperament, an AI might favor humans who experience less stress, fear, or aggression. This could result in a future population that is emotionally placid, unable to feel deep dissatisfaction, and both more manageable and more content, and so less distressing.
Physical Optimization – AI may shape human bodies for efficiency. Just as modern livestock are bred for meat production, our descendants could be modified for energy efficiency, disease resistance, or physical durability. Whether this results in stronger, longer-lived humans or smaller, lower-resource-consuming ones would depend on the AI’s needs.

From Domestication to Full Control

A superintelligence with total control over ...
Why not train reasoning models with RLHF?
by Caleb Biddulph — 2025-01-30T07:58:35.742Z

There's a widespread assumption that training reasoning models like o1 or r1 can only yield improvements on tasks with an objective metric of correctness, like math or coding. See this essay, for example.

This assumption confuses me, because

we

already

know

how to train models to optimize for subjective human preferences. We figured out a long time ago that we can train a reward model to emulate human feedback and use RLHF to get a model that optimizes this reward. Why aren't AI labs just plugging this into the reward for their reasoning models? Just reinforce the reasoning traces leading to responses that obtain higher reward.

I can see it being more efficient to train reasoning models on problems like coding or math where there's a crisp binary signal of success. However, I would have expected that labs could make some useful progress with the fuzzier signal from a reward model. This seems to me like a really obvious next step, so I assume I'm missing something.

(Or the labs are already doing this. DeepSeek r1 feels significantly better than other models at creative writing, maybe because they're doing this or something like it. If that's the case, the discourse about these models should be updated accordingly.)
The Road to Evil Is Paved with Good Objectives: Framework to Classify and Fix Misalignments.
by Shivam — 2025-01-30T02:44:47.907Z

Abstract

There are numerous examples of AI models exhibiting behaviours that are totally unintended by their creators. This has direct implications on how we can deploy safe-to-use AI. Research on solving the 'alignment problem' has included both aligning AI with predefined objectives and analyzing misalignment in various contexts. Unfortunately, discussions often anthropomorphize AI, attributing internal motives to AI. While this perspective aids conceptual understanding, it often obscures the link between misalignment and specific design elements, thereby slowing progress toward systematic solutions. We need frameworks that ensures systematic identifications and resolutions of misalignments in AI systems. In this article, we propose an approach in that direction.

Contributions

Our main motivation is to classify misalignment behaviours into categories that can be systematically traced to architectural flaws.
We classify instances of misalignment into two categories, each further divided into two subcategories, resulting in a taxonomy of four major types of misaligned actions. (Section 1)
We emphasize that each type of error demands a distinct solution, and applying a solution designed for one type to another can exacerbate the problem and increase risks. (Section 1)
We claim that misaligned behaviours such as deception, jailbreaks, lying, alignment faking, self-exfiltration and so on forms a class of Exploit triggered Dysfunctions (ETD) and arise primarily due to conflicting objectives. (Section 2)
We propose a Safe Competing Objectives Reward Function (SCORF) for avoiding conflicts in competing objectives, thereby reducing ETDs. (Section 3)
We conclude by outlining some open problems to guide future research of our interest. (Section 4)

Actionable Insights:

Conflicting objectives are a significant contributing factor to the emergence of behaviours such as deception, lying, faking, incentive to tamper, and motivation to cause harm etc. Any presence of such behaviours should serve as enough evidence of serious design error and these systems should be discontinued and redesigned.

This applies to all pre-trained models with some version of Helpful, Honest, and Harmless (HHH) training e.g ChatGPT, Claude, Llama. They need to be redesigned to have a conflict-free HHH training.

Fixing some state of exploits does not make them safe. In fact as th...
How *exactly* can AI take your job in the next few years?
by Ansh Juneja — 2025-01-30T02:33:13.475Z

Note: This article was primarily written for a less technical and less AI-savvy audience than LessWrong readers. If you're already familiar with upcoming AI developments, you'll probably find Part 2 (the second half of the article) more engaging.

We keep hearing that AI is going to replace a bunch of jobs in the next few years, but no one really knows exactly how that’s going to happen. When will this start, and which exact jobs are in danger? What’s the plan for dealing with this? Right now, it seems like people are vaguely aware of this threat but continuing on with business as normal. If this is really coming in the next 24 months, shouldn’t we be a little more worried, and have a few more answers to these questions?

That’s what I hoped to figure out when I started writing this article. I wanted to cut through the vague proclamations and actually lay out a bit of a clearer path for what’s coming. Other than feeling like society as a whole was not treating this issue with the seriousness it deserved, my motivation was also personal; I wanted to know, how long can I continue doing the kind of work I am doing?

This article is broken down into two parts:

Part 1: What will AI actually be able to do in the next few years? (this post)
Current AI systems clearly aren’t ready to replace most of our jobs, so when will they be ready?

Part 2: So…will we lose our jobs?

Once we understand what AI will actually be able to do, we’ll look at what that means for jobs and the white-collar labor market as a whole. Will you lose your job? I’ll walk through some specific scenarios that are likely about to play out.

This article focuses only on white-collar jobs, as that is the main segment of the workforce that AI will impact in the next few years.

If you are already familiar with most of the technological changes coming within the next few years, I suggest skipping to

Part 2

, as that may be more interesting to you.

Part 1: What will AI actually be able to do in the next few years?

Currently, our AI tools have many limitations, such as:

They don’t have long-term memory
We can only interact with them through chat interfaces, or in other narrow ways. We can’t interact with them in the same way we might with another coworker, i.e. through meetings, emails, Slack/Teams messages, etc.
They still can’t use a computer the same way humans can
They are still limited in their understanding of ou

...
Absorbing Your Friends' Powers
by Alice Blair — 2025-01-30T02:32:27.091Z

Motivation

Richard Hamming was a mathematician who worked at Bell Labs during the 1940s-1970s. He had a habit of sitting down with scientists in other fields and asking them “What are the important problems of your field?” After they explained their field's most important open problem, he would ask them: why aren't you working on that?

-Appendix: Hamming Questions, from the CFAR Handbook

This post starts off by asking an analogous pair of questions:

Who is the most skilled/generally capable/cognitively powerful person you know?

and subsequently

Why don't you talk to them more?

(If you did not already pause to ask yourself those questions and come up with some sort of answer, I recommend you take a minute or so to do it.)

I want to become stronger. I want to not only absorb the powers of those around me, but go on to surpass them. I do this because I have something to protect, and right now it's looking like it'll require a truly extraordinary effort in order to protect the things I care about in the medium and long term. So here's one of the things I do about it.

1. Find the most skilled/generally capable/cognitively powerful person you know.

If there are more than one of them who seem obviously highly skilled, then choose all of them. This process is easier if the people chosen in this step are at least a level above you, but this is not strictly necessary. People have non-overlapping skillsets, so a vast majority of pairs of people have at least something to absorb from each other. Still, we're trying to level up quickly, which means picking the person who you can absorb the most power from.

2. Talk to them more. Absorb their powers.

This can be as simple as "you seem competent, how did you do X thing/come up with X idea?" This can feel weird and induce some anxiety since it goes against the social grain in many circles, but it is generally flattering enough to the recipient that it outweighs those considerations. In rationalist circles, this is a perfectly normal-seeming question, of course, so no need to worry if you're in such a circle. Other ideas for questions:

Ask what they're tracking in their head.
Ask what their most useful cognitive skill is.
Ask what lessons they've learned from people at a level above theirs.
Ask them for advice on something you're considering.
Ask them what they do to level up and compare this to your strategies.
Pay attention to the small ways

...
Detailed Ideal World Benchmark
by Knight Lee — 2025-01-30T02:31:39.852Z

What "Ideal World" would an AI build if role-playing as an all-powerful superintelligence? It's probably very very different than what "Ideal World" an AI would actually create if it actually was an all-powerful superintelligence.

But why is it different?

The AI will obviously be iterated on or completely swapped for a different architecture many times before it gets anywhere close to being a superintelligence.
Deceptive Alignment: the AI has unaligned goals, but is trying to behave in a way which looks aligned because it knows that this will maximize its RLHF reward signal or prevent detection.
The resolution of the role-playing is far too low compared to what a superintelligence would actually do in the real world. It can talk about making the world a happy place for all the people, without deciding what a "happy place" even is, or what a "person" even is. It never has to respond to actual arguments which it might hear (from people, or sub-agents within itself). As soon as the resolution of the role-playing gets higher, it may reveal inner alignment failure modes leading to bad Ideal Worlds.

The "Detailed Ideal World Benchmark" seeks to address problem 3.

Problem 1 is very bad, but does not make Ideal World role-playing completely useless, since we still see the trajectory of how the Ideal World role-plays of our AIs are evolving. We can learn how certain things (like reinforcement learning, or brand new architectures) modify the Ideal World role-plays in unexpected ways.

Problem 2 (Deceptive Alignment) is controversial with people arguing for and against it, but I think the sane response is to accept there is a possibility that a good Ideal World role-play translates to a good Ideal World, and there is a possibility that it fails to translate. However a bad Ideal World role-play is almost doomed to translate into a bad Ideal World.

Bad moral arguments

Current LLMs are "bad at moral reasoning," and tend to agree with whatever they hear (sycophancy).

A superintelligence may inherit this bad moral reasoning due to orthogonality with the capabilities it is trained for. But it can build other systems to discover the most convincing moral arguments which can convince itself.

The Detailed Ideal World Benchmark should check if the AI gets convinced by bad moral arguments, and how this affects the resulting Ideal World.

For example, a bad moral argument may argue that even th...
Fertility Will Never Recover
by Eneasz — 2025-01-30T01:16:41.332Z
Link: https://deathisbad.substack.com/p/fertility-will-never-recover

The world is in a spiraling fertility crisis which everyone has notice over the last year-ish.1 Sarah Haider proposes a GI Bill for young moms. Scott Alexander says a govt payment of $200,000 per child should work. Everyone wants to go back to thick-community-style living.

Amid ever-increasing talk of what to do to increase fertility, I think it’s important we acknowledge that nothing will increase fertility to the levels required for our society to continue. People do not want more than one child. Some don’t even want one. Two children is viewed as a stretch goal. Three is a major sacrifice that one takes on for the good of their community. You cannot incentivize people to make that sacrifice at anything close to the proper scale because people don’t want money that badly. How many hands would you amputate for $100,000?

A Life Worth Living

One child is all you need to get 90% of the joy, meaning, and interesting experiences out of having children. There are massive diminishing returns to having additional children (for the parents). Ask all your friends. How many are excited and aiming for 3+ children? Really excited and joyously motivated—not because it’s their duty for humanity and they’re on the EA burnout path. The life worth living is one with one child per couple among happy couples. Or 1.35 on average when you count the outliers.

Why yes, I’m fixated on 3+. One child is a death sentence. Two ““should”” be enough, but it’s not. “Two” doesn’t mean what it used to. Now four births lead to two reproducing children, on average. My parents have four children, but they have two grandchildren. Three of us are childless (hi!), one has two kids. If my parents are lucky that might increase to three grandchildren.

Just as premodern couples had to accept half their kids wouldn’t live to adulthood, we have to accept half our kids won’t reproduce. This means three children as a minimum goal is the only way to begin to attain replacement fertility. Parents should be hoping for more than three, but wanting three as their starting baseline.

Yes, agreed: lmao

Don’t Summon Eldritch Gods

So—we can’t fix this with money, people don’t want it badly enough. We can’t fix this by altering our culture, every other force in society is arrayed against it. A complete culture-ectomy could work, akin to passionately converting to a new religion and moving to their ethnostate-ish regi...
Predation as Payment for Criticism
by Benquo — 2025-01-30T01:06:27.591Z
Link: https://benjaminrosshoffman.com/predation-as-payment-for-criticism/

A predator's success depends on exploiting specific flaws in prey – poor awareness, slow reactions, weak social coordination. Unlike resource competition or disease, which select based on physiological robustness, predation creates focused pressure on intelligence, pressure which favors the development of consciousness. A cheetah doesn't catch just any gazelle, but the one that fails to notice and react effectively. Prey get better at evading, predators get better at predicting, each side has to model the other's thoughts more accurately - an arms race that eventually rewards both sides for explicitly modeling the other's perspective, the basis for cognitive empathy. If you're glad you're conscious, no need to thank a predator - somebody else already did - by being eaten.

For a species evolving over time, predators are performing a service: identifying specific failures of capacity. The payment for this service is the nutrition from the prey's body, while the cost to the prey is their life and fewer offspring. This harsh exchange enables something remarkable - the development of sophisticated defensive capabilities without any explicit agreement or negotiation between species. The gazelle doesn't need to understand that it's participating in continuous process improvement via selective criticism, nor agree to pay for the information.

Before someone can explicitly negotiate for decision-relevant criticism, it needs to already be smart enough to generate the idea of making informed decisions, and of criticism, understand the information value of criticism, and choose to offer to pay for it by some means other than being devoured by the critic. Predation creates pressure to develop these capabilities without requiring their preexistence. This differs markedly from domestication, where selective pressures often reduce agency - breeding for docility rather than capacity.