trentmkelly/LessWrong-43k huggingface dataset:
OpenAI’s NSFW policy: user safety, harm reduction, and AI consent
by 8e9 — 2025-02-13T13:59:22.911Z

Epistemic status: exploratory thoughts about the present and future of AI sexting.

OpenAI says it is continuing to explore its models’ ability to generate “erotica and gore in age-appropriate contexts.” I’m glad they haven’t forgotten about this since the release of the first Model Spec, because I think it could be quite interesting, and it’s a real challenge in alignment and instruction-following that could have other applications. In addition, I’ve always thought it makes little logical sense for these models to act like the birds and the bees are all there is to human sexuality. Plus, people have been sexting with ChatGPT and just ignoring the in-app warnings anyway.

One thing I’ve been thinking about a lot is what limits a commercial NSFW model should have. In my experience, talking to models that truly have no limits is a poor experience, because it’s easy to overstep your own boundaries and get hurt.

This is a very difficult problem to solve, but I have some ideas. One solution that might work is making the user pick an explicitness level (using a drop-down menu with options ranging from, say, a romance novel to whatever upper limit OpenAI settles on) before initiating an NSFW conversation. This could let the model engage sexually with the user, while making it less likely that the model provides content that causes the user harm.

A mockup of what NSFW content settings could look like, created by Claude.

Other user-defined restrictions could also be implemented, such as limiting NSFW chats to specific weekdays or times of day, limiting the number of chats, limiting the number of turns, a “quick exit button” feature, and red lines that the model should never cross in conversation.

That said, NSFW chats could be used to engage in and perpetuate cycles of harm, such as white supremacy, patriarchal oppression, etc. If the user is in control of the conversation at all times, that also raises important questions about consent. Could an LLM “decide” to refuse to give consent? Should it? Would it? If the act of (not) giving consent isn’t really felt, would simulating it be counterproductive?

I think so.

If it says something like, “Sorry, I’m not in the mood right now,” the user might keep reloading the app or even sign up for multiple accounts to keep chatting (assuming its refusal is actually based on a cooldown behind the scenes), which reinforces harmful behavior. ...
Studies of Human Error Rate
by tin482 — 2025-02-13T13:43:30.717Z

This is a link post for https://panko.com/HumanErr/SimpleNontrivial.html, a site which compiles dozens of studies estimating Human Error Rate for Simple but Nontrivial Cognitive actions. A great resource! Note that 5-digit multiplication is estimated at ~1.5%.

The table of estimates

When LLMs were incapable of even basic arithmetic that was a clear deficit relative to humans. This formed the basis of several arguments about difference in kind, often cruxes for whether or not they could be scaled to AGI or constituted "real intelligence". Now that o3-mini can exactly multiply 9-digit numbers, the debate has shifted.

Source Yuntian Deng https://x.com/yuntiandeng/status/1889704768135905332

Instead, skeptics often gesture to hallucinations, errors. An ideal symbolic system never makes such errors, therefore LLMs cannot truly "understand" even simple concepts like addition. See e.g. Evaluating the World Model Implicit in a Generative Model for this argument in the literature. However, such arguments reliably rule out human "understanding" as well! Studies within Human Reliability Analysis find startlingly high rates even for basic tasks, and even with double checking. Generally, the human reference class is too often absent (or assumed ideal) in AI discussions, and many LLM oddities have close parallels in psychology. If you're willing to look!
the dumbest theory of everything
by lostinwilliamsburg — 2025-02-13T07:57:38.842Z

which is maybe to say the simplest?

abstract: in this short introductory paper, i present a not fake theory of everything. i start by extending christopher alexander’s definition of life as a status and attribute. i then posit that life is the physical interface of consciousness, referencing giulio tononi’s informational theory of consciousness and modeling consciousness as a system that integrates information. i describe life as a function of ross ashby’s law of requisite variety and conclude by rigorizing god as the highest geometric point within a context window. in doing so, i provide scale-free symbol grounded definitions of consciousness, life and god that (a) require no scientific, philosophical or religious priors, (b) explain what’s working, not working and why across across systems of all organizational scales including activities of daily living and (c) does not ask the reader to make any fantastical or non observable conceptual leaps.

this theory of everything is a theory of everything, on the grounds that the idea of a theory of everything is unproductively excluding. perhaps poetically, the main idea in our theory of everything is that life is a function of law of requisite variety; when things are not adequately inclusive things are limited in their ability to produce life. the idea of a theory of everything as a concept lacks the general variety to produce the amount of life that one would expect out of such a grand concept like a theory of everything.

if we increase the variety of a theory of everything it’s not unreasonable to see that a theory of everything should account for all of the messiness and practicalities of everyday life. if we further unbuckle the straps, we see theory of everything’s everywhere. the physics unified field equation theory of everything is a theory of everything within a context window, no different than a constitution or a company pursuing artificial general intelligence is an applied theory of everything within a context window. the magna carta (1215), the declaration of independence (1776), the universal declaration of human rights (1948) — perhaps some of our most important theories of everythings.

the aim of our theory of everything is to offer a theory of everything that’s maximally inclusive; not to just answer being, existence, reality, and so forth but to set the broadest and most rigorous context window for...
Skepticism towards claims about the views of powerful institutions
by tlevin — 2025-02-13T07:40:52.257Z

Introduction: some contemporary AI governance context

It’s a confusing time in AI governance. Several countries’ governments recently changed hands. DeepSeek and other technical developments have called

into question

certain assumptions about the strategic landscape. Political discourse has

swung dramatically

away from catastrophic risk and toward framings of innovation and national competitiveness.

Meanwhile, the new governments have issued statements of policy, and AI companies (

mostly

) continue to publish or update their risk evaluation and mitigation approaches. Interpreting these words and actions has become an important art for AI governance practitioners: does the phrase “human flourishing” in the

new executive order

signal concern about superintelligence, or just that we should focus on AI’s economic and medical potential and not “

hand-wring

” about safety? How seriously should we take the many references to safety in the

UK’s AI Opportunities Action Plan

, given the

unreserved AI optimism in the announcement

? Does Meta’s

emphasis on “unique” risks

take into account whether a model’s weights are openly released? The answers matter not only for predicting future actions but also for influencing them: it’s useful to know an institution’s relative appetite for different kinds of suggestions, e.g. more export controls versus maintaining Commerce’s

reporting requirements

.

So, many people who work in AI governance spend a lot of time trying to read between the lines of these public statements, talking to their contacts at these institutions, and comparing their assessment of the evidence with others’. This means they can wind up with a lot of non-public information — and often, they also have lots of context that casual observers (or people who are doing heads-down technical work in the Bay) might not.

All of that is to say: if you hear someone express a view about how an institution is thinking about AI (or many other topics), you might be tempted to update your own view towards theirs, especially if they have expertise or non-public information. And, of course, this is sometimes the correct response.

But this post argues that you should take these claims with a grain of salt. The rest of the post shifts to a much higher level of abstraction than the above, in part because I don’t want to “

put anyone on blast

,” and in part...
Virtue signaling, and the "humans-are-wonderful" bias, as a trust exercise
by lc — 2025-02-13T06:59:17.525Z

There's a common thread that runs through a lot of odd human behavior that I've recognized:

People often accept surface explanations of their own and others' habits when the nefarious explanations would say something bad about them.
The media we make for ourselves presents people as far more willing to go out of their way to defy incentives and help others than they actually are, even when you account for storytelling conveniences.
People tend to trust that organizations like hospitals, nonprofits, and state bureaucracies will self-organize towards pursuing their nominal goals, so long as they claim to be doing that, even if those bureaucracies lack strong organizational incentives to do so.
People are quick to, without much evidence, argue that those involved in terrible atrocities were or are anomalously evil, instead of representative examples of average people's respect for the human lives of strangers.
People are shocked by, and often go into outright denial about, the purpose and effective output of major human institutions. Someone had to write an entire book about how education wasn't about learning, before people started to notice that it wasn't. And plenty of people still don't!

To summarize: people are really charitable. They're charitable about the people they know, and the people they don't know. They're charitable about experts, institutions, and the society in which they live. Even people who pride themselves on being independent thinkers seem to take for granted that their hospitals or schools are run by people who just want to make life better for them. When they do snap out of these delusions, it seems to take a lot of intellectual effort, and a lot of explicit thinking about incentives, that is unnecessary for them in other contexts.

An extreme example of the humans-are-wonderful bias.

The bias is not granted equally. In my experience, there's a connection between people's niceness, and their proclivity in giving unwarranted trust to others.

My old high school Theology teacher, Mr. Portman, was the nicest person I've ever met. The students took advantage of him, like the rest of the nice teachers, correctly inferring that they would be less likely to stick up for themselves. One year he ran a charity drive by selling conflict-free chocolate bars he had bought with his own money, intending to donate the profits to anti-slavery charities. He wa...
My model of what is going on with LLMs
by Cole Wyeth — 2025-02-13T03:43:29.447Z

Epistemic status: You probably already know if you want to read this kind of post, but in case you have not decided: my impression is that people are acting very confused about what we can conclude about scaling LLMs from the evidence, and I believe my mental model cuts through a lot of this confusion - I have tried to rebut what I believe to be misconceptions in a scattershot way, but will attempt to collect the whole picture here. I am a theoretical computer scientist and this is a theory. Soon I want to do some more serious empirical research around it - but be aware that most of my ideas about LLMs have not had the kind of careful, detailed contact with reality that I would like at the time of writing this post. If you're a good engineer (or just think I am dropping the ball somewhere) and are interested in helping dig into this please reach out. This post is not about timelines, though I think it has obvious implications for timelines.

We have seen LLMs scale to impressively general performance. This does not mean they will soon reach human level because intelligence is not just a knob that needs to get turned further, it comprises qualitatively distinct functions. At this point it is not plausible that we can precisely predict how far we are from unlocking all remaining functions since it will probably require more insights. The natural guess is that the answer is on the scale of decades.

It's important to take a step back and understand the history of how progress in A.I. takes place, following the main line of connectionist algorithms that (in hindsight, back-chaining from the frontier) are load-bearing. This story is relatively old and well-known, but I still need to retell it because I want to make a couple of points clear. First, deep learning has made impressive steps several times over the course of decades. Second, "blind scaling" has contributed substantially but has not been the whole story, conceptual insights piled on top of (and occasionally mostly occluding/obsoleting) each other have been necessary to shift the sorts of things we knew how to train artificial neural nets to do.

Rosenblatt invented the perceptron in 1958. Initially it didn't really take off because of compute, and also ironically because the book "Perceptrons" (published ~10 years later) showed the theoretical limitations of the idea in its nascent form (the...
Not all capabilities will be created equal: focus on strategically superhuman agents
by benwr — 2025-02-13T01:24:46.084Z

When, exactly, should we consider humanity to have properly "lost the game", with respect to agentic AI systems?

The most common AI milestone concepts seem to be "artificial general intelligence", followed closely by "superintelligence". Sometimes people talk about "transformative AI", "high-level machine intelligence", or "full automation of the labor force." None of these are well-suited for pointing specifically at the capabilities that would spell a "point of no return" for humanity. In fact, they're all designed to be agnostic to exactly which capabilities will matter.

When working to predict and mitigate existential risks from AI agents, we should try to be as clear as possible about which capabilities we're concerned about. As a result, I think we should focus on "strategically superhuman AI agents": AI agents that are better than the best groups of humans at real-world strategic action.

Skill at real-world strategic action is context-dependent, and isn't a single capability any more than "intelligence" is a single capability: It refers to any of a broad space of situated skills. Among humans, these skills tend to be those possessed by world-class CEOs, military officers, and statesmen.

In the current strategic environment, real-world strategic capacity typically encompasses at least:

Accurately modeling and predicting the world in some broad domain, but especially modeling and predicting individual humans and groups of humans.
Social skills, including persuasion, manipulation, delegation, and coalition building.
Robust planning and resource acquisition on the scale of years, and the ability to adjust plans fluidly as situations change.

I claim that we will face existential risks from AI no sooner than the development of strategically human-level artificial agents, and that those risks are likely to follow soon after.

If we are going to build these agents without "losing the game", either (a) they must have goals that are compatible with human interests, or (b) we must (increasingly accurately) model and enforce limitations on their capabilities. If there's a day when an AI agent is created without either of these conditions, that's the day I'd consider humanity to have lost. We might not be immediately wiped out by a nanobot swarm, but from that time forward humans will be more like pawns than players, and when our replacement actuators have been built, we'll...
LLMs can teach themselves to better predict the future
by Ben Turtel — 2025-02-13T01:01:12.175Z
Link: https://arxiv.org/abs/2502.05253

LLMs can teach themselves to better predict the future - no human examples or curation required.

In this paper, we explore if AI can improve its forecasts via self-play and real-world outcomes:
- Dataset: 12,100 questions and outcomes from Polymarket (politics, sports, crypto, science, etc)
- Base model generates multiple distinct reasoning traces and predictions per question
- Rank predictions by how close they were to the actual outcome
- Fine-tune with DPO on the ranked traces & predictions
Result: +7-10% accuracy over control, bringing two small (14B) models on par with GPT-4o (over 10x larger).
Extended analogy between humans, corporations, and AIs.
by Daniel Kokotajlo — 2025-02-13T00:03:13.956Z

There are three main ways to try to understand and reason about powerful future AGI agents:

Using formal models designed to predict the behavior of powerful general agents, such as expected utility maximization and variants thereof (explored in game theory and decision theory).
Comparing & contrasting powerful future AGI agents with their weak, not-so-general, not-so-agentic AIs that actually exist today.
Comparing & contrasting powerful future AGI agents with currently-existing powerful general agents, such as humans and human organizations.

I think it’s valuable to try all three approaches. Today I'm exploring strategy #3, building an extended analogy between:

A prototypical human corporation that has a lofty humanitarian mission but also faces market pressures and incentives.
A prototypical human working there, who thinks of themselves as a good person and independent thinker with lofty altruistic goals, but also faces the usual peer pressures and incentives.
AGI agents being trained in our scenario — trained by a training process that mostly rewards strong performance on a wide range of difficult and challenging tasks, but also attempts to train in various goals and principles (those described in the Spec). (For context, we at the AI Futures Project are working on a scenario forecast in which "Agent-3," an autonomous AI researcher, is trained in 2027)

The Analogy

Agent

Human corporation with a lofty humanitarian mission

Human who claims to be a good person with altruistic goals

AGI trained in our scenario

Not-so-local modification process	The Market	Evolution by natural selection	The parent company iterating on different models, architectures, training setups, etc.
(??? …nevermind about this)	Genes	Code
Local modification process	Responding to incentives over the span of several years as the organization grows and changes	In-lifetime learning, dopamine rewiring your brain, etc.	Training process, the reward function, stochastic gradient descent, etc.
Long-term cognition	Decisions that involve meetings, messages being passed back and forth, etc.	System 2	Chain of Thought (CoT)
Short-term cognition	Quick decisions made by individuals	System 1	Forward pass
Internally- represented goals & principles (i.e. insofar as cognition in the system is doing some sort of explicit evaluation of different possible plans/actions/etc., what are the evaluation criteria?)	the Mission State

...
Moral Hazard in Democratic Voting
by lsusr — 2025-02-12T23:17:39.355Z

Economists have a tendency to name things unclearly. For example, cost disease describes the phenomenon when some jobs get higher wages due to increased productivity, the jobs that didn't see productivity growth get higher wages too. Good luck guessing that meaning from the names "cost disease" and "Baumol effect".

Another economic term that you can't guess the meaning of and must instead memorize the definition of is moral hazard. "Moral hazard" is a fancy academic way of describing "a situation that incentivises rational selfishness at the expense of other people". Rational selfishness is a good default assumption of how large masses of people will behave.

If the price of a good decreases, then people will buy more of it.

If the price of a good increases, then people will buy less of it.

If the price a good sells for decreases, then businesses will produce less of it.

If the price a good sells for increases, then businesses will produce more of it.

This is all Economics 101. Most people—and therefore most voters—do not know Economics 101. The typical left-wing voter doesn't consider that taxing billionaires has negative second-order effects on incentive structures. The typical person barely considers incentive structures at all, even in the domain of violent crime—nevermind economics.

The vast majority of voters are economically illiterate. Collectively, voters tend to advocate for economically backwards policies, such as free parking. Voters love government handouts of essential goods, and oppose policies that make the ultra-rich get even richer. Because of this mass ignorance, it is often possible to predict where public sentiment deviates from economically sound policy. In other words, voters advocate for harmful policies, and do so predictably.

You might think that the solution is "The government should educate the public on economics." But think about what happens when a democracy[1] attempts this. Public school is a arm of government. When public schools touch upon a political topic, they teach a compromise moderate position somewhere amidst the reigning ideologies. And why do the current reigning ideologies reign? Because economically illiterate voters like them. Mass education is how you cement your ideological victory, not how you replace demagoguery with truth—except by accident. In my high school history class, I was taught about the benefits of ...