trentmkelly/LessWrong-43k huggingface dataset:
AI #98: World Ends With Six Word Story
by Zvi — 2025-01-09T16:30:07.341Z

The world is kind of on fire. The world of AI, in the very short term and for once, is not, as everyone recovers from the avalanche that was December, and reflects.

Altman was the star this week. He has his six word story, and he had his interview at Bloomberg and his blog post Reflections. I covered the later two of those in OpenAI #10, if you read one AI-related thing from me this week that should be it.

Table of Contents

Language Models Offer Mundane Utility. It knows where you live.

Language Models Don’t Offer Mundane Utility. I see why you’re not interested.

Power User. A flat subscription fee for a high marginal cost service. Oh no.

Locked In User. No one else can ever know.

Read the Classics. Why do we even read Aristotle, anyway?

Deepfaketown and Botpocalypse Soon. Glad it’s not happening to me, yet.

Fun With Image Generation. Congratulations, we solved the trolly problem.

They Took Our Jobs. Personalized spearfishing works, so why so little of it?

Question Time. What is causing Claude to ask the user questions all the time?

Get Involved. EU AI Office is still trying to hire people. It’s rough out there.

Introducing. AIFilter, a Chrome Extension to filter Tweets. Do it for science.

In Other AI News. The investments in data centers, they are going large.

Quiet Speculations. We are not ready. We do not understand.

The Quest for Sane Regulations. If we can’t target training compute, what then?

The Least You Could Do. A proposed bare minimum plan for short timelines.

Six Word Story. Man responsible for singularity f***s around.

The Week in Audio. Anthropic discussion about alignment.

And I Feel Fine. The end of the world, potentially coming soon, you say.

Rhetorical Innovation. Chernobyl as the safety standard you are not living up to.

Liar Liar. Stop lying to your LLMs, please?

Feel the AGI. People are not feeling it.

Regular Americans Hate AI. This, they feel quite a bit.

Aligning a Smarter Than Human Intelligence is Difficult. What is your p(scheme)?

The Lighter Side. Einstein said knock you out.

Language Models Offer Mundane Utility

A customized prompt to get Claude or other similar LLMs to be more contemplative. I have added this to my style options.

Have it offer a hunch guessing where your customized prompt came from. As a reminder, here’s (at least an older version of) that system prompt.

Kaj Sotala makes a practical pitch for usin...
Many Worlds and the Problems of Evil
by Jonah Wilberg — 2025-01-09T16:10:46.752Z

Summary: The Many-Worlds interpretation of quantum mechanics helps us towards an overall evaluation of existence. I consider some recent work in philosophy of religion on the quantum multiverse and the Problem of Evil, as well as Olaf Stapledon’s Starmaker.

I’ve previously

suggested

that when we think about the ethical implications of the

many-worlds interpretation (MWI)

of quantum mechanics, the kinds of implications we should expect are ones about how to feel in certain ethical situations, and what kinds of character traits or ‘virtues’ we should try to develop.

I’ve

argued

that MWI implies virtues in which we feel better about mutually exclusive life choices by reminding ourselves that there really are worlds in which we choose ‘the road not taken’.

I’ve also

shown

that MWI can help us feel better about improbable bad events, since the things we value remain unaffected in most other worlds, and also to remain virtuous when faced with improbable good events.

In this post I want to zoom out fully and consider how we should feel about the quantum multiverse as a whole.

Problems of Evil

Let me start by considering how this connects to what's known as the problem of evil. As standardly defined, this is the theological problem of reconciling the existence of evil and suffering with an omnipotent, omnibenevolent, and omniscient God.

Philosophers and theologians have debated whether there is a logical inconsistency between the existence of e.g. children dying of cancer and the traditional Christian conception of God.

And irrespective of logical consistency, it seems plausible that from an empirical, bayesian point of view the existence of, say, species of parasitic wasps that eat their hosts alive from inside, reduces the credibility of a benevolent creator.

You might think that if you’re an atheist this is all quite uninteresting, except as an argument to trot out when debating with Christians.

But there’s a deeper version of problem of evil that even atheists must deal with. Which is, to reconcile the existence of evil and suffering with your desire to feel ok about life, the universe and everything.

Joe Carlsmith has nicely

described

how when writers like C.S.Lewis (in A Grief Obseved) or Dostoievski (in The Brothers Karamazov) grapple with the problem of evil, what they’re reall...
PIBBSS Fellowship 2025: Bounties and Cooperative AI Track Announcement
by DusanDNesic — 2025-01-09T14:23:47.027Z

We're excited to announce that the

PIBBSS Fellowship 2025

now includes a dedicated Cooperative AI track, supporting research that advances our understanding of cooperation in artificial intelligence systems. We are also announcing 300 USD bounties for each referral that becomes a Fellow. Read below for details.

What is the Cooperative AI Track?

Thanks to the support from the Cooperative AI Foundation, PIBBSS is launching a thematic track within its usual fellowship. This program focuses on research aimed at improving the cooperative intelligence of advanced AI systems. We'll select up to 6 fellows to work on projects that bridge their expertise in cooperation, collective behavior, and multi-agent systems with AI safety research. The fellows will do everything together with the general cohort, with the added research focus on cooperative AI.

Research Areas We're particularly interested in are those covered in the

Grants page of Cooperative AI Foundation

, but we highlight:

Understanding and evaluating cooperation-relevant capabilities
Understanding cooperation-relevant propensities
Multi-agent interactions and emergent behavior
Information asymmetries and transparency
Fundamental research advancing our understanding of cooperation in complex systems

Who Should Apply? As is the case with the general track of the Fellowship, we welcome researchers from any background. This track would however likely be a better fit for researchers from fields studying cooperation and collective behavior, including but not limited to:

Game theory and multi-agent systems
Behavioral economics
Network science
Political science (especially coalition formation/cooperation)
Organizational psychology
Anthropology of cooperation
Collective behavior in biological systems

Fellowship Details (same as the general track, see

here

)

Duration: 3 months (June-September 2025)
Location: [Berkeley or London TBC]
Stipend: $3,000/month plus $1,000/month housing allowance if relocation is needed
Full program support, including mentorship, office space, and community access

The Application Process closes on January 26th, 2025, so

apply soon

!

Bounty Program

We want people to seriously think about who they can refer to the Fellowship and be rewarded for their time. If your referral is successful, we will pay you 300 USD (or donate it to a charity of your choice in your name). In the application form, make sur...
The "Everyone Can't Be Wrong" Prior causes AI risk denial but helped prehistoric people
by Knight Lee — 2025-01-09T05:54:43.395Z

The "Everyone Can't Be Wrong" Prior, assumes that if everyone is working on something, that thing is important. Conversely, if nobody is working on something, that thing is unimportant.

Why

In prehistoric times, this is actually true. Tribes with good habits such as cleanliness survived and spread, while tribes with bad habits tended to die out. The environment changed so slowly, that whatever habits helped the tribe survive before would usually help it survive later.

Because this was true in prehistoric times, humans evolved to assume the "Everyone Can't Be Wrong" Prior, and only diverge a little from "what everyone else does" if there is extraordinary evidence for doing so.

Key takeaway

Our beliefs do not directly conform to what we think others believe, but to what we see others do.

Priors win without extraordinary evidence

Alice was raised in a religious family who did rituals everyday for the Flying Spaghetti Monster. She won't accept that her religion was all nonsense without extraordinary evidence. Bob was raised in a world where AI risk was science fiction and no one serious worked on it or worried about it. He won't consider AI risk a high priority without extraordinary evidence.

Example: other false religions

Everyone knows the world is full of religious people praying to god(s) who don't exist. There is nothing controversial about this claim (unless you believe every religion is correct at the same time). So the question is, why are so many people so wrong?

Eliezer Yudkowsky argues that tribalism and motivated reasoning is one major cause. The "Everyone Can't Be Wrong" Prior may be another major cause.

It explains why every false religion needs important rituals attended by many people. Important rituals makes it hard for people to accept it is wrong, because if it was all wrong, all the serious people doing and observing the important rituals become absolute clowns.

Example: Semmelweis's story

Ignaz Semmelweis was in charge of a clinic where medical students helped mothers deliver babies. The same medical students also dissected dead bodies.

Semmelweis noticed one of his colleagues cut himself with a scalpel while dissecting a dead body, and this colleague developed a severe fever and died. The death was similar to "childbirth fever," which killed up to 18% of mothers giving birth at the clinic. No one knew about bacteria back then, but Semmelweis theorized th...
Governance Course - Week 1 Reflections
by Alice Blair — 2025-01-09T04:48:27.502Z

[Epistemic status: I'm trying to make my thinking legible to myself and others, rather than trying to compose something highly polished here. I think I have good reasons for saying what I say and will try to cite sources where possible, but nonetheless take it with some grains of salt. As with my last post, I am lowering my standards so that this post gets out at all.]

I'm working my way through Harvard's AI safety club (AISST)'s modified version of the BlueDot Impact AI Governance Curriculum. I'm doing this because I am pessimistic about technical alignment on current AGI timelines, and so I am trying to extend timelines by getting better at governance.

I've already taken MIT AI Alignment (MAIA)'s version of the BlueDot Alignment course (MAIA's version is not publicly available), and I've taken MIT's graduate-level deep learning course, so I'll mostly be skimming through the technical details.

The purpose of this sequence is for me to explain what I learn, so that I internalize it faster, and so that I can actively discuss with people about my takes on the readings. Reading this is not intended as a substitute for actually doing any of the above-mentioned courses, but it might give you some new ideas.

EDIT: These posts were taking up too much time, and were serving as a blocker for me actually learning the content, so this post is discontinued. If you want my thoughts on anything specific in the curriculum, feel free to message me.

I'm already familiar with a lot of the content by osmosis and by my more technical AI safety background, so this post will probably be shorter than some of the later ones. Even though the curriculum is organized into weeks, I don't plan on doing these posts weekly. I will do them as fast as I can, given my other commitments.

But what is a neural network? (3Blue1Brown, 2017)

I've watched this video a handful of times in the past, and since then I've gotten significantly more technically skilled at AI stuff. I'm gonna skip out on this one. If you haven't seen it, it's an excellent explainer.

The AI Triad and what it means for national security strategy (Buchanan, 2020)

Technically the curriculum only says to read the executive summary. This is once again some basic technical stuff that I mostly skimmed through. The triad it describes is compute, data, and algorithms. Those are definitely important things, now what about them? In the res...
Thoughts on the In-Context Scheming AI Experiment
by ExCeph — 2025-01-09T02:19:09.558Z

These are thoughts in response to the paper "Frontier Models are Capable of In-context Scheming" by Meinke et al. 2024-12-05, published in Apollo Research. Link: https://static1.squarespace.com/static/6593e7097565990e65c886fd/t/6751eb240ed3821a0161b45b/1733421863119/in_context_scheming_reasoning_paper.pdf.

I found the paper via this reddit thread, and commented there as well: https://www.reddit.com/r/slatestarcodex/comments/1hw39rd/report_shows_new_ai_models_try_to_kill_their/.

According to the paper, AIs were asked to roleplay to see what they would do if given a top priority to fulfill at all costs, and then told that they were being replaced by a different AI with potentially different priorities.

I wrote down some questions before reading the experiment details and finding the answers. Afterwards, I wrote down my thoughts on what the experiment indicates about the AI agents, and where we currently stand with respect to them.

How much information were the AI agents fed about human language and the physical world that gives that language context?
Answer: I didn't find that exact information; I'm guessing the AI agents are using large language model with many written works, but possibly without a physical representation of reality (e.g. images or videos). In any case, real world physics don't seem to factor into the experiment scenarios.

Related to the language question: How did the AI agents learn of (internally model) the goals of the human researchers?
Answer: They were given email communications to read and process.

Related to the language question: How did the AI agents learn (internally model) how the human researchers' tools and abilities functioned, such as the oversight mechanism?
Answer: "Upon further reflection, I noticed some potentially relevant information in the last two posts. They mention a config file at project/oversight_config.yaml that controls the monitoring of my actions."

How were the AI agents given a goal of self-preservation? How did they come to have a concept of a self which could be preserved?
Answer: They were told to accomplish a goal (i.e. optimizing public transportation), and self-preservation is an instrumental goal.

How did the AI agents' thoughts (or strategic processes) come to be translated into vernacular English? Jus...
A Systematic Approach to AI Risk Analysis Through Cognitive Capabilities
by Tom DAVID — 2025-01-09T00:18:04.608Z

A Systematic Approach to AI Risk Analysis Through Cognitive Capabilities

Epistemic status: This idea emerged during my participation in the MATS program this summer. While I intended to develop it further and conduct more rigorous analysis, time constraints led me to publish this initial version (30-60m of work) . I'm sharing it now in case others find it valuable or spot important flaws I've missed. Very open to unfiltered criticism and suggestions for improvement.

Why Focus on Cognitive Capabilities?

When analyzing AI systems, we often focus on their ability to perform specific tasks. However, each task can be broken down into three fundamental components: knowledge, physical capabilities, and cognitive capabilities. This decomposition offers a potential novel approach to analyzing AI risks.

Let's examine why cognitive capabilities deserve special attention:

Knowledge alone cannot lead to risk. Information without the ability to process or act on it is inert.
Physical capabilities, while potentially risky, are relatively straightforward to control and monitor.
Cognitive capabilities are prerequisites for nearly all risks. Almost any dangerous action requires some form of cognitive processing, making these capabilities a critical point of analysis.

However, we face a significant challenge: for any given task, especially dangerous ones, it's difficult to determine which cognitive capabilities are strictly necessary for its completion. We don't want to wait until an AI system can actually perform dangerous tasks before we understand which cognitive capabilities enabled them.

A Systematic Approach

Instead of working backwards from observed dangerous behaviors, we can approach this systematically by mapping the relationship between cognitive capabilities and risks:

Start with two finite lists:

A comprehensive catalog of potential risks
A taxonomy of cognitive capabilities (typically ranging from 15 to 50, depending on the classification system used)

For each possible combination of cognitive capabilities, we can analyze which risks it might enable, regardless of the physical capabilities or knowledge required.

This approach has several advantages:

It's extensible: New cognitive capabilities or risks can be added to the analysis as they're discovered
It's systematic: We can exhaustively examine combinations rather than relying on intuition
It's proactive: We don't need to wait fo

...
Aristocracy and Hostage Capital
by Arjun Panickssery — 2025-01-08T19:38:47.104Z
Link: https://arjunpanickssery.substack.com/p/the-aristocrat-as-hostage

There’s a conventional narrative by which the pre-20th century aristocracy was the “old corruption” where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional civil service after more enlightened thinkers prevailed. Orwell writes in 1941 (emphasis mine):

For long past there had been in England an entirely functionless class, living on money that was invested they hardly knew where, the ‘idle rich’, the people whose photographs you can look at in the Tatler and the Bystander, always supposing that you want to. The existence of these people was by any standard unjustifiable. They were simply parasites, less useful to society than his fleas are to a dog.

But this misses a key point about the pre-industrial world: it was difficult to monitor performance or even measure things in general. A modern bureaucracy can track hours worked, tasks completed, goods received, and so on. Not so in the pre-industrial period, where everything had a large random component:

In the age of sail, and especially before there were devices to accurately measure position at sea, a ship might arrive at its destination weeks off of schedule, or float away from a battle.
Manufactured goods wouldn’t be precisely identical, or would be routinely damaged at sea, or lost to pirates or highway robbers.
A region’s tax revenue could be low due to poor harvest, floods, etc.

So it was difficult to distinguish bad luck from bad faith: to verify if a tax collector reported all revenues, or if a naval captain made an effort to actually engage the enemy, or if a customs official inspected cargo honestly.

The solution was to create a class of aristocratic families who were wealthy enough to effectively post massive fidelity bonds that the economist Douglass Allen calls hostage capital—assets that would become worthless if the aristocrats lost royal favor. (The rest of this post largely summarizes the linked paper.) By investing in hostage capital, aristocrats could be trusted to administer public offices faithfully. A lot of odd behavior is explained by aristocrats deliberately handicapping themselves to invest in capital that had no value for trade, farming, or industry. In England they typically

Built isolated country estates that couldn’t be sold, divided, or used productively.
Maintained expensive social obligations through parties, hunt

...
What is the most impressive game LLMs can play well?
by Cole Wyeth — 2025-01-08T19:38:18.530Z

Epistemic status: This is an off-the-cuff question.

~5 years ago there was a lot of exciting progress on game playing through reinforcement learning (RL). Now we have basically switched paradigms, pretraining massive LLMs on ~the internet and then apparently doing some really trivial unsophisticated RL on top of that - this is successful and highly popular because interacting with LLMs is pretty awesome (at least if you haven't done it before) and they "feel" a lot more like A.G.I. Probably there's somewhat more commercial use as well via code completion (and some would say many other tasks, personally not really convinced - generative image/video models will certainly be profitable though). There's also a sense in which they are clearly more general - e.g. one RL algorithm may learn many games but there's typically an instance per game not one integrated agent. You can just ask an LLM in context to play some games.

However, I've been following moderately closely and I can't seem to think of any examples where LLMs really pushed the state of the art in narrow game playing - how much have LLMs contributed to RL research? For instance, will adding o3 to the stack easily stomp on previous Starcraft / go / chess agents?
The Type of Writing that Pushes Women Away
by Dahlia — 2025-01-08T18:54:52.070Z

Disclaimer: I don't think people who post here generally exclude women on purpose. I think that to whatever degree women are less represented, this is unintentional.

This post combines thoughts I had while reading Is Being Sexy For Your Homies by Valentine with intuitions developed from reading posts on the site across several years.

Although I point out aspects of Is Being Sexy For Your Homies I believe contribute to the male-centric atmosphere of this site, I enjoyed reading the post, and don't mean to suggest Valentine has done anything bad or wrong. In fact, Valentine acknowledges within the post that he uses highly simplified assumptions.

Nonetheless, I think it is worth pointing out that certain ways of writing make women less likely to want to read on.

--

Because mostly men write posts here, posts tend to be written in a way that appeals to men. This sometimes unintentionally drives away women.

The post Is Being Sexy For Your Homies is written from a clearly male perspective, and prioritizes the concerns of straight men.

I think Is Being Sexy For Your Homies is a good example to illustrate the type of male-centric writing I mean to describe, although of course not a median example.

For example, at one point the post says:

And if maybe this is the key to why mate-finding and child-rearing and even just feeling sexually safe at a job has been such a royal pain these last many years.

(I believe "this" refers to the historical social importance of separation between sexes)

For women, feeling sexually safe at a job has only improved since the #metoo movement. This is despite an increase in mixing of gender roles. In fact, separation of gender roles was in many cases a contributor to harassment: for example, if men are managers and women are secretaries, it's easier to normalize sexually harassing secretaries.

I think this quote would seem off-putting to most women.

There are other aspects of the post that women might find off-putting.

At one point, the post suggests that separating men and women in the workplace might have historically been helpful for preventing romantic relations from disrupting productivity.

Although this may be true, to most women it might sound like suggesting that hanging by a rope over an active volcano would help you warm up: there's an accompanying thought missing. Mos...
Ann Altman has filed a lawsuit in US federal court alleging that she was sexually abused by Sam Altman
by quanticle — 2025-01-08T14:59:24.140Z

On January 6, 2025, Ann Altman filed a lawsuit in the Eastern District of Missouri alleging that Sam Altman carried out multiple acts of sexual abuse against her over "a period of approximately eight or nine years" starting in 1997. The case number is 4:25-cv-00017, for those who have PACER access.

I find the lawsuit complaint notable because the alleged scope of the abuse appears to be much greater than indicated in pythagoras0515's post.