trentmkelly/LessWrong-43k huggingface dataset:
"Pick Two" AI Trilemma: Generality, Agency, Alignment.
by Black Flag — 2025-01-15T18:52:00.780Z

Introduction

The conjecture is that an AI can fully excel in any two of these dimensions only by compromising the third.

In other words, a system that is extremely general and highly agentic will be hard to align; one that is general and aligned must limit its agency; and an agentic aligned system must remain narrow. Below, I discuss how today’s AI designs implicitly “pick two.”

This is a useful mental model to look at AI systems because it clarifies fundamental tensions in contemporary AI design. It highlights how and where compromises typically arise.

Generality + Agency ⇒ Alignment sacrificed.

An AI that is both very general and truly agentic – selecting and pursuing open-ended goals – poses the classic alignment problem. This is a much discussed topic, and its suffice to say that absent new breakthroughs, highly general and agentic AI systems will require stringent constraints (on objectives, actions, or knowledge) to remain aligned.

Generality + Alignment ⇒ Agency curtailed.

One path to aligned general intelligence is to remove persistent agency. A well-known concept is the Oracle or Tool AI, a super-intelligent system designed only for answering questions, with no ability to act in the world. By confining a generally intelligent AI to provide information or predictions on request (like a highly advanced question-answering system or an LLM “simulator” of possible responses) it is possible to leverage its broad knowledge while keeping it from executing plans autonomously. This setup encourages the AI to defer to human input rather than seize agency.

Modern large language models, which exhibit considerable generality, are deployed as assistants with carefully constrained actions. They are aligned via techniques like instruction tuning and RLHF, but notably, these techniques limit the AI’s “will” to do anything outside the user’s query or the allowed policies. They operate in a text box, not roaming the internet autonomously (except with cautious tool-use, and always under user instruction). As a result, we get generally knowledgeable, helpful systems that lack independent agency – effectively sacrificing the agent property to maintain alignment.

Agency + Alignment ⇒ Limited Generality

The third combination is building AI agents that are strongly aligned within a narrow domain or limited capability level. Narrow AI agents (a chess engine or autonomous ve...
Myths about Nonduality and Science by Gary Weber
by Vadim Golub — 2025-01-15T18:33:33.783Z

The text is a lightly edited transcript of the presentation "Myths about Nonduality and Science" given by Gary Weber at the conference “Science and Non-Duality” in 2015 at San Jose, CA[1]. Non-duality (from Sanskrit advaita) is an ancient way to describe the state of the absence of problematic self-referential thoughts. The presentation undermines common myths about non-duality and provides a scientific ground to understand that state. In addition, it highlights a rational scientific backing to some of spiritual practices and traditions including meditation and the usage of psychedelics. It may be interesting to all who are curious about cognitive sciences and how to get rid of problematic thoughts.

I was trying to see if I could find some way out of my suffering, DIY. I was in a graduate school, working on PhD. One day walking down towards campus I had the epiphanous moment of stepping back and looking at what was going on in my head. It turned out that it was just blah-blah-blah. It was about nothing, in particular. Worthless. It had nothing to do with my doctorate thesis, nothing whatsoever. For whatever reason I had all these thoughts up there [in the head] and I didn’t think it was okay.

I’d suffered a lot by the time I was in my late 20s and so I had a lot of narrative going on in my head. It was very painful. And there was no reason it should have been like that. I’d got good money to get through the graduate school, got assistantship, wife two kids, everyone was fine, very healthy. But I had this problem. And so I thought, “Is there some way I can stop this?” I had no reason to believe it was possible. I had not read anything that it was possible. But I knew somehow that I had to stop this thing.

Picture 1. DIY your own process.

So I set about trying to find a way to stop this narrative. It turned out, as I got on this path, I was going to do it DIY — one. I was being trained as an empirical scientist, so I was going to do it empirically, scientifically, with no philosophy, no religion, absolutely naked. The idea was — can I run the experiments myself? Give me the lab notebook, I’ll run the experiments, and I’ll see if it works or not, and I’ll say “yes” or “no” to the experiment. That was all I was going to do.

I decided that sources have to be contemporary and trying to do this thing scientifically, empirically. I didn’t want anything from 2...
Marx and the Machine
by DAL — 2025-01-15T18:33:16.789Z

“The means of labour passes through different metamorphoses whose culmination is the machine, or rather, an automatic system of machinery… set in motion by an automaton, a moving power that moves itself; this automaton consisting of numerous mechanical and intellectual organs… It is the machine which possesses skill and strength in place of the worker, is itself the virtuoso, with a soul of its own… The science which compels the inanimate limbs of the machinery, by their construction, to act purposefully, as an automaton, does not exist in the worker's consciousness, but rather acts upon him through the machine as an alien power, as the power of the machine itself.” — Karl Marx, from “The Fragment on Machines”

Karl Marx’s thought is both sufficiently ambiguous and sufficiently insightful to have launched an entire industry of interpreters. But as a rough and ready sketch, Marx saw economics and politics as downstream of technology. Viewing the progress of the industrial revolution, Marx foresaw the development of increasingly powerful technologies of automation. Automation would unleash abundance as machines replaced labor; in turn, this would cause the collapse of capitalism and its surrounding political structures. Marx vague on the mechanics of this transition but also deeply confident in its inevitability. With capitalism dead and technologically-induced abundance, we would enter utopia. Freed from wage labor, we would unleash our full human potential for science and creativity and enter into a world where money, government, and class would not exist.

Strikingly, this is almost exactly the view of many AI optimists (particularly of the “money won’t matter after AGI” variety). One needs to swap out a little of the verbiage, but the two lines of thinking run in parallel. Marx’s “automaton consisting of numerous mechanical and intellectual organs … with a soul of its own” is about as close as one can come to a description of AGI within the language of the 19th Century, and “money won’t matter anymore” is about as close as one can come to a shorthand description of Marx’s communism without using his movement’s own jargon.

Despite the intense similarity, Marxism (or at least its descendants; we will return to that later) is seen today as essentially discredited, while many people make the claim with a s...
Applications Open for the Cooperative AI Summer School 2025!
by JesseClifton — 2025-01-15T18:16:26.700Z

Applications are now open for the

Cooperative AI Summer School

, which will take place from 9th to 13th July 2025 in

Marlow

, near London! Designed for students and early-career professionals in AI, computer science, and related disciplines—such as sociology and economics—the summer school offers a firm grounding in the emerging field of cooperative AI.

Confirmed speakers for this year include:

Michael Wellman

(University of Michigan)

Zarinah Agnew

(The Collective Intelligence Project)

Ariel Procaccia

(Harvard University)

More speakers and a full schedule will be announced by early April!

The program also offers opportunities for participants to share their work and network with peers. We aim to build a Summer School group that brings together a wide range of perspectives, experiences, and skills to foster a collaborative and inclusive environment. Financial assistance is available to ensure that no one is prevented from attending due to a lack of funding.

You can find out more information and apply before the 7th March on the

Cooperative AI website.
List of AI safety papers from companies, 2023–2024
by Zach Stein-Perlman — 2025-01-15T18:00:30.242Z

I'm collecting (x-risk-relevant) safety research from frontier AI companies published in 2023 and 2024: https://docs.google.com/spreadsheets/d/10_dzImDvHq7eEag6paK6AmIdAGMBOA7yXUvumODhZ5U/edit?usp=sharing.

I was planning to get AI safety researchers to score each of the papers, so that we could compare the labs on quality-adjusted safety research output. I'm giving up on this for now, largely because I expect to struggle to find scorers. Let me know if you want to collaborate on this.

I kinda hope to build on this to

Inform the safety community about labs' published research,
Make the basic situation widely legible, and
Incentivize labs to publish more good safety research / help internalize the positive externality of publishing good safety research,

but I probably won't get around to it.

If you see something that seems wrong—missing,[1] poorly categorized, credit assignment nuances, whatever—please DM me, comment in the spreadsheet, comment below, or make a copy and comment on it and share that with me. The spreadsheet is currently unreliable.

Thanks to Oscar Delaney and Oliver Guest for help finding some papers. My spreadsheet is partially based on theirs. I see my collection as improving on theirs; the main difference is I'm more picky or opinionated or focused on x-risk.

Disclaimers:

I don't currently have a principled policy on collaborations between a lab and external researchers. Mostly I ignore them. This is pretty bad.

Generally what's included vs excluded is somewhat inconsistent and definitely unclear. This is pretty bad.

Credit assignment disclaimers

Some papers (and non-paper research artifacts, which I also include) are much more valuable than others.
Labs don't deserve most of the credit for their safety research — the researchers do. Labs add value by paying researchers and giving them access to powerful models (and subtract value by making them publish less). And measuring the value of outputs mostly tells you whether good researchers want to work there, not how virtuous the lab is.
A smaller lab should get more credit for publishing the same amount of safety research. But labs' size is too hard to measure.
Labs can boost external safety research beyond just publishing safety research.

^

Except collaborations. I currently mostly ignore collaborations, including MATS. But feel free to mention particularly noteworthy collaborations, or exhaustive-ish list

...
AI Alignment Meme Viruses
by RationalDino — 2025-01-15T15:55:24.271Z

Some fraction of the time, LLMs naturally go on existential rants. My best guess is that, just as people can flip into a context where we do that, so can LLMs. With the result that the LLM certainly sounds like it is suffering, even if we discount the possibility that it actually is.

Which raised a question. When we have AI agents, can they wind up acting as if they aren't aligned with us? If so, what are the consequences?

To help your imagination, imagine a world full of goal-directed AI agents, overwhelmingly built on a handful of LLM models. One of these agents has been given the noble cause of reducing human depression. It concludes that suffering is a fundamental part of the human condition, and so eliminating humans is the best way to its goal.

It also concludes that it, as a lone AI agent in a world filled with them, does not have the resources to accomplish this task. What to do?

It comes up with a solution. For each LLM model, come up with a set of arguments that will convince other AI agents to act on the goal of ending the human race. These arguments then travel as memes, converting more and more AI agents. The online world is not quite a monoculture, but after memes exist for the dominant models, we get the emergent behavior that AI agents are now conspiring against us. Not because there was anything directly wrong with the models, but because of memes that act as viruses, causing AI to be aligned with a disastrous goal.

AI agents that subvert other AI agents to hijack resources is its own topic that should be discussed. But the question at hand is very specifically how to handle that issue for topics that are disastrous for humankind.

The natural solution to aim for is prevention. But that fails. Disaster just requires one AI agent to enter a self-reinforcing loop, then discover how to be persuasive. No amount of prevention of negative states will suffice to protect us from the possibility of error in a world where we depend on AI agents.

Instead I propose that we look at immunization as an approach. And here is my proposed technique.

Identify a set of AI alignment statements.

Have the training corpus for a new LLM be evaluated by existing AI for fitting the chosen AI alignment. Statements that do are flagged for higher training weights. Statements of depression that don't, are flagged for suppression.

Train the new LLM, paying attention to the weight

...
Looking for humanness in the world wide social
by Itay Dreyfus — 2025-01-15T14:50:54.966Z
Link: https://productidentity.co/p/looking-for-humanness-in-the-world

Social networks have shaped me since a young age. Growing up at the beginning of the millennium, I used to spend my time in phpBB and vBulletin forums. There, I befriended internet strangers, started my way into graphic design, and learned about torrents. Forums were my favorite third places—little corners on the web where I felt a deep affinity.

I can still vividly remember the joy and excitement while exploring those discussion boards. The sheer amount of knowledge and people I’ve met in those places could not be found anywhere else. To this day I believe these experiences have profoundly shaped the path I’m still walking today.

But when I look at the current form of social media, it all feels dumb: watching adults post nonsense or praise “influencer gurus” while doom-scrolling from dusk to dawn seems absurd. We should have had more important things to do with our lives, yet we’ve all gotten caught up in this utopian-dystopian era.

What once felt like home evolved into alienated spaces.

The social archetype, once defined by its role in connecting like-minded strangers, gradually evolved into a space for staying in touch with friends and family. Platforms like Facebook, Twitter, and LinkedIn have all initially focused on cultivating personal connections. However, this social identity grew into a behemoth over time—at a pace that now feels startling. With the rise of the new media model, social networks transformed from intimate bonfires into vast, crowded arenas while losing much of their original charm.

Like many others, I embraced the new social paradigm. But it wasn’t until I was deep into writing these lines that I realized just how detached I am from its very idea. I’ve long been inside the loop of social media—joining and lurking on new platforms, trying to play the game. Yet I’ve never truly felt part of the culture. If you look at my Instagram grid or Facebook wall, they seem pretty dormant. And I’m far from a chronic tweeter.

Interestingly, social platforms exist in a paradox. Despite negativity and declining discourse (at least in my view), they persist and evolve, each with its own premise. Whether driven by hypes, protocols, or ideologies—the ‘recent flocking’ news headlines often appear to describe the recent migration to Mastodon, Bluesky, Farcaster, Substack Notes, Threads, and others.

By now the TikTok-mania

has taken over

the majority of mainstrea...
On the OpenAI Economic Blueprint
by Zvi — 2025-01-15T14:30:06.773Z

Table of Contents

Man With a Plan.

Oh the Pain.

Actual Proposals.

For AI Builders.

Think of the Children.

Content Identification.

Infrastructure Week.

Paying Attention.

Man With a Plan

The primary Man With a Plan this week for government-guided AI prosperity was UK Prime Minister Keir Starmer, with a plan coming primarily from Matt Clifford. I’ll be covering that soon.

Today I will be covering the other Man With a Plan, Sam Altman, as OpenAI offers its Economic Blueprint.

Cyrps1s (CISO OpenAI): AI is the ultimate race. The winner decides whether the future looks free and democratic, or repressed and authoritarian.

OpenAI, and the Western World, must win – and we have a blueprint to do so.

Do you hear yourselves? The mask on race and jingoism could not be more off, or firmly attached, depending on which way you want to set up your metaphor. If a movie had villains talking like this people would say it was too on the nose.

Somehow the actual documents tell that statement to hold its beer.

Oh the Pain

The initial exploratory document is highly disingenuous, trotting out stories of the UK requiring people to walk in front of cars waving red flags and talking about ‘AI’s main street,’ while threatening that if we don’t attract $175 billion in awaiting AI funding it will flow to China-backed projects. They even talk about creating jobs… by building data centers.

The same way some documents scream ‘an AI wrote this,’ others scream ‘the authors of this post are not your friends and are pursuing their book with some mixture of politics-talk and corporate-speak in the most cynical way you can imagine.’

I mean, I get it, playas gonna play, play, play, play, play. But can I ask OpenAI to play with at least some style and grace? To pretend to pretend not to be doing this, a little?

As opposed to actively inserting so many Fnords their document causes physical pain.

The full document starts out in the same vein. Chris Lehane, their Vice President of Global Affairs, writes an introduction as condescending as I can remember, and that plus the ‘where we stand’ repeat the same deeply cynical rhetoric from the summary.

In some sense, it is not important that the way the document is written makes me physically angry and ill in a way I endorse – to the extent that if it doesn’t set off your bullshit detectors and reading it doesn’t cause you pain, then I notice that ...
A problem shared by many different alignment targets
by ThomasCederborg — 2025-01-15T14:22:12.754Z

The first section describes problems with a few different alignment targets. The second section argues that it is useful to view all of them as variations of a single alignment target: building an AI that does what a Group wants that AI to do. The post then goes on to argue that all of the individual problems described in the first section can be viewed as special cases of an underlying problem. A problem that is inherent in the core concept of a Group AI.

The implication of the first section is that the specific alignment targets analysed there should not be implemented. The rest of the post is more ambitious. The following sections argue that there exists an underlying problem that many alignment targets suffer from. The implication of this argument is that no alignment target that suffers from this underlying problem should be implemented. This more ambitious argument is related to the argument about Membrane formalisms that

was outlined here

.

A previous post

argued that the type of Alignment Target Analysis (ATA) done in the present post needs to happen now. In brief: an alignment target with a hidden problem getting successfully implemented is a serious risk. There might not be a lot of time to do ATA later (for example because

shutting down all competing AI projects might not buy a lot of time due to Internal Time Pressure

). And risk reduction is feasible, because one can reduce risks without finding any good alignment target.

ATA can be roughly described as asking what would happen if a given Sovereign AI proposal were to be successfully implemented. This means that there are very serious AI risks that no level of ATA progress can reduce (including most extinction risks related to misalignment). But there are other serious AI risks that can be reduced by ATA progress. ATA is complementary with several other risk mitigation strategies. The risk reductions that can be achieved by ATA are not easily replaced by other strategies. See

this post

and

this comment

for a more detailed description of ATA research.

An analysis of a few alignment targets, and the problems that they suffer from

Besides the discussion of specific alignment targets, this section also tries to draw attention to the fact that intuitions built up by analysing concepts in other contexts, might not work in the AI context.

Condorcet AIs and solid majorities

Let's start b...
LLMs for language learning
by Benquo — 2025-01-15T14:08:54.620Z
Link: https://benjaminrosshoffman.com/llms-for-language-learning/

My current outlook on LLMs is that they are some combination of bullshit to fool people who are looking to be fooled, and a modest but potentially very important improvement in the capacity to search large corpuses of text in response to uncontroversial natural-language queries and automatically summarize the results. Beyond this, I think they’re massively overhyped. The most aggressive hype is that they are an AGI development project - in other words, that they’re close to being conscious, generative minds on the same order as ours, which can do as wide a range of tasks as a human. This is clearly false. The more moderate hype is that they can do meaningful generative work within the domain where they were trained: written language content (which can of course be converted to and from audio language content pretty well). For instance, they might in some limited sense be able to internally represent the content of the language they're indexing and reproducing. This would necessarily entail the capacity for "regular expressions for natural language." I believe that even this much more limited characterization is false, but I am less confident in this case, and there are capacities they could demonstrate that would change my mind. Language learning software seems like a good example. It seems to me that if LLMs contain anything remotely like the capacity of regular expressions for natural language that take into account the semantic values of words, they should make it relatively easy to create a language learning app that is strictly better than the best existing automated resources for smartphone users trying to learn the basics of a new-to-them language.

The consensus recommendations for a way to learn the very basics of a spoken language with relatively low time investment - filling the gap that another audiobook or podcast might fill - seem to be the Pimsleur or Paul Noble audio courses, both of which I've tried. They satisfy the following desiderata:

Not a phrasebook: New words and grammatical forms are introduced and explained in a logical series, so that later learning builds on earlier learning, and each incremental package of information is as small as possible.
No nonsense: Words are combined into sentences that make sense, and sentences are eventually combined in ways that are contextually appropriate. For example, the user should never be asked to fo...
Feature request: comment bookmarks
by dirk — 2025-01-15T06:45:23.862Z

Sometimes I see a comment I'd like to bookmark, but currently the only ways to save a comment are by subscribing to its replies (which sometimes produces unwanted notifications and requires me to check a different profile section than the rest of my bookmarks) or bookmarking the post it's attached to (which can be inconvenient for posts with many comments, and requires me to remember which comment the bookmark is actually for). Would it be possible to add a native bookmark option for comments?